elastic/osearch policy management (ilm/ism)

warning

elastic doesn’t have the same syntax e.g.

          "rollover": {
            "max_primary_shard_size": "5gb",
            "max_age": "1m"
          }

and the mgmt policy isn’t setup at the same place – it is done with the template

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "delete-7d",
          "rollover_alias": "test-alias"
        }
      }
    },
    "mappings": {
    },
    "aliases": {}
  }
}

MWE (start simple)

{
    "policy": {
        "description": "rotate 1mb primary shards",
        "default_state": "hot_state",
        "states": [
            {
                "name": "hot_state",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_primary_shard_size": "1mb"
                        }
                    }
                ]
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "test-*"
                ]
            }
        ]
    }
}

stress-test

push some logs in a loop e.g.

    while true; do curl -I --resolve tmp.nethence.com:80:127.0.0.1 tmp.nethence.com/; done

==> you should now see indices being rotated when reaching 1mb+

speed-up the policies

see tuning

advanced setup

see policy-mgmt samples

note we do not include total size into the hot_state actions in view to handle any kind of shards amount – so it doesn’t matter how many shards the index has.

                            "min_size": "30gb",

resources

https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html

https://opensearch.org/docs/latest/im-plugin/ism/index/

https://opensearch.org/docs/latest/im-plugin/ism/settings/

https://opensearch.org/docs/latest/im-plugin/ism/policies/

elastic/osearch policy management (ilm/ism)

warning

MWE (start simple)

stress-test

speed-up the policies

advanced setup

resources

samples

optimal shard size

history

misc

moar

troubles