elastic/osearch cluster tuning

general instructions

GET _cluster/settings?flat_settings=true&include_defaults=false

cluster optimization

we want one shard per node, always

    "cluster.routing.allocation.balance.shard": 0.01,
    "cluster.routing.allocation.balance.index": 99,
    "cluster.routing.allocation.balance.threshold": 0.01,
    "cluster.routing.allocation.balance.prefer_primary": true

although we keep the disk allocation decider enabled

    "cluster.routing.allocation.disk.threshold_enabled": true,
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "93%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "96%"

write optimization

reduce shards cap per node by half – one should have large shards (10-50gb and up to 200 million docs) hence a reduced amount of those

PUT _cluster/settings
{
   "persistent":{
      "cluster.max_shards_per_node": 500
   }
}

read optimization

assuming 8 pri shards and 1 replica, one should not need more than 16 shards for queries – and as we use pri shard size only policies, let’s assume no more than 4 indices – that is fine ONLY IF YOU HAVE SPECIFIC INDEX PATTERNS (if you have global index patterns walking across many indices, that won’t do)

echo $(( 16 * 4 ))
PUT /_cluster/settings
{
  "transient": {
    "action.search.shard_count.limit": 64
  }
}

query least busy replicas

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.use_adaptive_replica_selection": true
  }
}

eventually enable that fork-feature and check – by “fork” I mean that’s opensearch only, apparently.

PUT _cluster/settings
{
  "persistent": {
    "shard_indexing_pressure": {
      "enabled": true,
      "enforced": false
    }
  }
}
GET /_nodes/_local/stats/shard_indexing_pressure

policy mgmt schedule

speed-up the policies: every minute

curl -sk https://localhost:9200/_cluster/settings -u admin:PASSWORD | jq

cat <<EOF | curl -sk -X PUT -H "Content-Type: application/json" \
        "https://localhost:9200/_cluster/settings?flat_settings=true" -u admin:PASSWORD \
        -d @- | jq
{
  "persistent" : {
    "plugins.index_state_management.job_interval" : "1"
  }
},
{
  "transient" : {
    "plugins.index_state_management.job_interval" : "1"
  }
}
EOF

resources

cluster settings

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/cluster-update-settings.html

ism settings

https://opensearch.org/docs/latest/im-plugin/ism/settings/

index settings

https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/

performance

https://opensearch.org/docs/latest/tuning-your-cluster/performance/