elastic/osearch cluster tuning

overall status

checkout currently enabled settings

GET _cluster/settings?flat_settings=true

if you want to see more

GET _cluster/settings?flat_settings=true&include_defaults=true

cluster optimization

we want one shard per node, whatever happens – and I would like to add that this is what a cluster and replicas are for (not sure we need such a complicated DS algo here…)

PUT _cluster/settings
{
  "persistent":{
    "cluster.routing.allocation.balance.index": 99,
    "cluster.routing.allocation.balance.prefer_primary": true,
    "cluster.routing.allocation.balance.shard": 0.01,
    "cluster.routing.allocation.balance.threshold": 0.01
  }
}

also enforce light constant re-balance

PUT _cluster/settings?flat_settings=true
{
  "persistent" : {
    "cluster.routing.allocation.allow_rebalance": "indices_all_active",
    "cluster.routing.allocation.cluster_concurrent_rebalance": 1,
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.rebalance.enable": "all"
  }
}

write optimization

one should have large shards (10-50gb and up to 200 million docs) hence a reduced amount of shards

default is 1000 shards per node – so you could easily reduce that amount by half

PUT _cluster/settings
{
   "persistent":{
      "cluster.max_shards_per_node": 500
   }
}

read optimization

rather use per-data-stream index patterns, instead of setting action.search.shard_count.limit – that way, you will reduce the amount of data-streams and shards involved in searches

but you can set a high-limit just in case e.g.

PUT /_cluster/settings
{
  "transient": {
    "action.search.shard_count.limit": 1500
  }
}

query least busy replicas

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.use_adaptive_replica_selection": true
  }
}

eventually enable that indexing backpressure (osearch-only apparently) – in shadow mode – not sure how it goes as for enforced mode

PUT _cluster/settings
{
  "persistent": {
    "shard_indexing_pressure": {
      "enabled": true,
      "enforced": false
    }
  }
}

more

GET /_nodes/_local/stats/shard_indexing_pressure

resources

cluster settings

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/cluster-update-settings.html

index settings

https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/

read optimization

https://opensearch.org/blog/shard-indexing-backpressure-in-opensearch/

https://docs.opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/shard-indexing-settings/ ==> High-level controls

FW https://forum.opensearch.org/t/shard-indexing-backpressure/11430/2

performance

https://opensearch.org/docs/latest/tuning-your-cluster/performance/