elastic/osearch policy management (ilm/ism)

this is for datastreams – see policy-mgmt-indices for managing indices

speed-up the policies

speed-up ilm/ism for testing to 1 minute (default is 5 minutes)

    PUT _cluster/settings
    {
      "transient" : {
        "plugins.index_state_management.job_interval" : "1"
      }
    }

setup

MWE (start simple) – see policy-mgmt samples

test-ism

advanced setup – see policy-mgmt samples

note we do not include total size into the hot_state actions in view to handle any kind of shards amount – so it doesn’t matter how many shards the index has.

"min_size": "30gb",

acceptance

send some logs to the datastrean – see elastic-stress-test

resources

https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html

https://opensearch.org/docs/latest/im-plugin/ism/index/

https://opensearch.org/docs/latest/im-plugin/ism/settings/

https://opensearch.org/docs/latest/im-plugin/ism/policies/

samples

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism-tutorial.html

optimal shard size

https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/7.17/size-your-shards.html ==> 10GB to 50GB and < 200M documents

history

https://forum.opensearch.org/t/ism-with-min-size-and-noisy-neighbours/8637/3

misc

update/apply policy to indices https://opensearch.org/docs/latest/im-plugin/ism/managedindexes/

rollover parms https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover

moar

shrink https://bonsai.io/blog/reduce-number-of-shards-elasticsearch

shrink with condition https://forum.opensearch.org/t/option-to-shrink-based-on-size-of-index-shards/13808/2

troubles

https://discuss.elastic.co/t/forbidden-8-index-write-api-during-date-change-on-indexes/179319 https://stackoverflow.com/questions/55507320/logstash-cannot-index-data-to-elasticsearch-forbidden-8-index-write-api https://stackoverflow.com/questions/44383601/aws-elastic-search-forbidden-8-index-write-api-unable-to-write-to-index ==> is in warm_state instead of hot_state?

https://discuss.elastic.co/t/index-rollover-not-working/317750

https://github.com/opensearch-project/index-management/issues/413