opensearch cluster rolling upgrade

tested on debian12

upgraded v2.11.0 ==> v2.14.0

descr

proceed with coordinator nodes, then data nodes, and only the the cluster manager

define the node you want to query for status and upgrade preparation

k=https://opensearch0:9200 # coordinator node
k=https://opensearch1:9200 # data1
k=https://opensearch2:9200 # data2
k=https://opensearch3:9200 # manager node

status

curl -sk $k/_cluster/health?pretty -u admin:PASSWORD

curl -sk "$k/_cat/nodes?v&h=name,version,node.role,master" -u admin:PASSWORD | column -t

curl -sk "$k/_nodes/opensearch-c1?pretty" -u admin:PASSWORD | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-d1?pretty" -u admin:PASSWORD | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-d2?pretty" -u admin:PASSWORD | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-cluster_manager?pretty" -u admin:PASSWORD | grep version | sort -uV

upgrade preparation

cat <<EOF | curl -sk -X PUT "$k/_cluster/settings?pretty" -u admin:PASSWORD \
    -H "Content-Type: application/json" -d@-
{
    "persistent": {
        "cluster.routing.allocation.enable": "primaries"
    }
}
EOF

curl -sk -X POST "$k/_flush?pretty" -u admin:PASSWORD

upgrade

step by step (cluster manager comes last)

disable the node

systemctl stop opensearch

proceed

dpkg -i opensearch-2.14.0-linux-x64.deb
rm -f opensearch-2.14.0-linux-x64.deb

/etc/opensearch/jvm.options     I
/etc/opensearch/opensearch.yml  N

eventually upgrade the config manually (enabling quite a few plugins) then

cd /etc/opensearch/
cp opensearch.yml opensearch.yml.old
vi opensearch.yml

(update the upper part and keep node-specific settings)

chown -R opensearch:opensearch /etc/opensearch/
chmod 640 /etc/opensearch/opensearch.yml

systemctl restart opensearch

re-enable

cat <<EOF | curl -sk -X PUT "$k/_cluster/settings?pretty" -u admin:PASSWORD \
        -H "Content-Type: application/json" -d@-
{
    "persistent": {
        "cluster.routing.allocation.enable": "all"
    }
}
EOF

troubleshooting

while upgrading the coordinator node, some index cache was found, need to clean-up – double quotes missing

vi /etc/default/opensearch

OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m"

fix

export OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk
/usr/share/opensearch/bin/opensearch-node repurpose

resources

https://opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/index/

https://opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/rolling-upgrade/