elastic doesn’t have the same syntax e.g.
"rollover": { "max_primary_shard_size": "5gb", "max_age": "1m" }
and the mgmt policy isn’t setup at the same place – it is done with the template
{ "template": { "settings": { "index": { "lifecycle": { "name": "delete-7d", "rollover_alias": "test-alias" } } }, "mappings": { }, "aliases": {} } }
{ "policy": { "description": "rotate 1mb primary shards", "default_state": "hot_state", "states": [ { "name": "hot_state", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "rollover": { "min_primary_shard_size": "1mb" } } ] } ], "ism_template": [ { "index_patterns": [ "test-*" ] } ] } }
push some logs in a loop e.g.
while true; do curl -I --resolve tmp.nethence.com:80:127.0.0.1 tmp.nethence.com/; done
==> you should now see indices being rotated when reaching 1mb+
see tuning
note we do not include total size into the hot_state
actions in view to handle any kind of shards amount
– so it doesn’t matter how many shards the index has.
"min_size": "30gb",
https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html
https://opensearch.org/docs/latest/im-plugin/ism/index/
https://opensearch.org/docs/latest/im-plugin/ism/settings/
https://opensearch.org/docs/latest/im-plugin/ism/policies/
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html
https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism-tutorial.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/7.17/size-your-shards.html ==> 10GB to 50GB and < 200M documents
https://forum.opensearch.org/t/ism-with-min-size-and-noisy-neighbours/8637/3
update/apply policy to indices https://opensearch.org/docs/latest/im-plugin/ism/managedindexes/
rollover parms https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover
shrink https://bonsai.io/blog/reduce-number-of-shards-elasticsearch
shrink with condition https://forum.opensearch.org/t/option-to-shrink-based-on-size-of-index-shards/13808/2
https://discuss.elastic.co/t/forbidden-8-index-write-api-during-date-change-on-indexes/179319 https://stackoverflow.com/questions/55507320/logstash-cannot-index-data-to-elasticsearch-forbidden-8-index-write-api https://stackoverflow.com/questions/44383601/aws-elastic-search-forbidden-8-index-write-api-unable-to-write-to-index ==> is in warm_state instead of hot_state?
https://discuss.elastic.co/t/index-rollover-not-working/317750
https://github.com/opensearch-project/index-management/issues/413