elastic/osearch policy management (ilm/ism)

logging | elk | fluentbit

warning

elastic doesn’t have the same syntax e.g.

          "rollover": {
            "max_primary_shard_size": "5gb",
            "max_age": "1m"
          }

and the mgmt policy isn’t setup at the same place – it is done with the template

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete-7d",
          "rollover_alias": "test-alias"
        }
      }
    },
    "mappings": {
    },
    "aliases": {}
  }
}

MWE (start simple)

{
    "policy": {
        "description": "rotate 1mb primary shards",
        "default_state": "hot_state",
        "states": [
            {
                "name": "hot_state",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_primary_shard_size": "1mb"
                        }
                    }
                ]
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "test-*"
                ]
            }
        ]
    }
}

stress-test

push some logs in a loop e.g.

    while true; do curl -I --resolve tmp.nethence.com:80:127.0.0.1 tmp.nethence.com/; done

==> you should now see indices being rotated when reaching 1mb+

speed-up the policies

see tuning

advanced setup

see policy-mgmt samples

note we do not include total size into the hot_state actions in view to handle any kind of shards amount – so it doesn’t matter how many shards the index has.

                            "min_size": "30gb",

resources

https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html

https://opensearch.org/docs/latest/im-plugin/ism/index/

https://opensearch.org/docs/latest/im-plugin/ism/settings/

https://opensearch.org/docs/latest/im-plugin/ism/policies/

samples

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism.html

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ism-tutorial.html

optimal shard size

https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/7.17/size-your-shards.html ==> 10GB to 50GB and < 200M documents

history

https://forum.opensearch.org/t/ism-with-min-size-and-noisy-neighbours/8637/3

misc

update/apply policy to indices https://opensearch.org/docs/latest/im-plugin/ism/managedindexes/

rollover parms https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html https://opensearch.org/docs/latest/im-plugin/ism/policies/#rollover

moar

shrink https://bonsai.io/blog/reduce-number-of-shards-elasticsearch

shrink with condition https://forum.opensearch.org/t/option-to-shrink-based-on-size-of-index-shards/13808/2

troubles

https://discuss.elastic.co/t/forbidden-8-index-write-api-during-date-change-on-indexes/179319 https://stackoverflow.com/questions/55507320/logstash-cannot-index-data-to-elasticsearch-forbidden-8-index-write-api https://stackoverflow.com/questions/44383601/aws-elastic-search-forbidden-8-index-write-api-unable-to-write-to-index ==> is in warm_state instead of hot_state?

https://discuss.elastic.co/t/index-rollover-not-working/317750

https://github.com/opensearch-project/index-management/issues/413


HOME | GUIDES | LECTURES | LAB | SMTP HEALTH | HTML5 | CONTACT
Copyright © 2024 Pierre-Philipp Braun