elastic/osearch - the index rollover situation

this is for indices – see policy-mgmt for managing datastreams

warning / lessons learned

handling rollover and life-cycle of casual indices is much harder than dealing with data-streams

speed-up the policies

speed-up ilm/ism for testing to 1 minute (default is 5 minutes)

PUT _cluster/settings
{
  "transient" : {
    "plugins.index_state_management.job_interval" : "1"
  }
}

index template

osearch

create template

name        test-index-tpl
type        INDEX NOT DATA STREAM
pattern     test-index*
alias       NOTHING HERE *1
shards      1
replicas    0
adv settings    + "index.plugins.index_state_management.rollover_alias": "test-index"

elastic

{
  "template": {
    "settings": {
      "index": {
    "lifecycle": {
      "name": "delete-7d",
      "rollover_alias": "test-alias"
    }
      }
    },
    "mappings": {
    },
    "aliases": {}
  }
}

create mgmt policy

create policy – see policy-mgmt samples

    test-index-ism

prepare alias write index

do not send logs yet, prepare the write index first

use Dev Tools

PUT test-index-000001
{
  "aliases": {
    "test-index":{
      "is_write_index": true
    }
  }
}

write to the rollover alias

PUT test-alias/_doc/99
{
  "somefield": "test",
}

define an index pattern

test-alias (notice it matches test-index-000001 !)

Discover / test-alias

(check the _index field)

==> goes to test-000001

also back to index mgmt, that index shows up under the test-index-ism policy

acceptance

send some logs to the rollover alias – see elastic-stress-test

==> you should now see indices being rotated when reaching 1mb+

clean-up

• delete test-index index pattern
• delete test-index-000001 test-index-000002 indices
• delete test-index-tpl template
• delete test-index-ism policy

troubleshooting

while writing a document on the alias *1

{
    "cause": "Rollover alias [test-index] can point to multiple indices, found duplicated alias [[test-index]] in index template [test-index-tpl]",
    "message": "Failed to rollover index [index=test-index-000001]"
}

==> do not specify test-index as an alias in the template

resources

https://opster.com/guides/elasticsearch/data-architecture/index-lifecycle-policy-management/