opensearch cluster rolling upgrade

tested on debian12

upgraded 2.11.0 ==> 2.14.0 ==> 2.18.0

descr

proceed with coordinator nodes, then data nodes, and only then, the cluster manager

define the node you want to query for status and upgrade preparation

k=https://opensearch0:9200 # coordinator node
k=https://opensearch1:9200 # data1
k=https://opensearch2:9200 # data2
k=https://opensearch3:9200 # manager node

admin_user=admin
admin_passwd=...

status

curl -sk $k/_cluster/health?pretty -u $admin_user:$admin_passwd
curl -sk "$k/_cat/nodes?v&h=name,version,node.role,master" -u $admin_user:$admin_passwd | column -t
curl -sk "$k/_nodes/opensearch-c1?pretty" -u $admin_user:$admin_passwd | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-d1?pretty" -u $admin_user:$admin_passwd | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-d2?pretty" -u $admin_user:$admin_passwd | grep version | sort -uV
curl -sk "$k/_nodes/opensearch-cluster_manager?pretty" -u $admin_user:$admin_passwd | grep version | sort -uV

upgrade preparation

cat <<EOF | curl -sk -X PUT "$k/_cluster/settings?pretty" -u $admin_user:$admin_passwd \
    -H "Content-Type: application/json" -d@-
{
    "persistent": {
        "cluster.routing.allocation.enable": "primaries"
    }
}
EOF

curl -sk -X POST "$k/_flush?pretty" -u $admin_user:$admin_passwd

data nodes upgrade

node by node (cluster manager comes last)

eventually take the chance to also upgrade the system

apt update
apt dist-upgrade
apt autoremove --purge
apt install colordiff

grab the latest release and proceed

ver=2.18.0

dpkg -i opensearch-$ver-linux-x64.deb

# from opensearch0
scp opensearch-$ver-linux-x64.deb opensearch1:
scp opensearch-$ver-linux-x64.deb opensearch2:
scp opensearch-$ver-linux-x64.deb opensearch3:

override what you want to be stock defaults e.g.

/etc/opensearch/jvm.options     I

keep your settings and review stock changes after

/etc/opensearch/opensearch.yml  N

clean-up

rm -f opensearch-$ver-linux-x64.deb

eventually upgrade the config manually (enabling quite a few plugins) then

cd /etc/opensearch/
cp -pi opensearch.yml opensearch.yml.`date +%s`

diff -bu opensearch.yml.dist opensearch.yml.dpkg-dist

mv -f opensearch.yml.dist opensearch.yml.dist.old
mv -i opensearch.yml.dpkg-dist opensearch.yml.dist

mv -f opensearch.yml.clean opensearch.yml.clean.old
grep -vE '^#|^$' opensearch.yml.dist > opensearch.yml.clean

vi opensearch.yml

(sync/update depending on dist's diff)
(keep node-specific settings at the bottom)
(keep custom settings below that e.g.)

opensearch.experimental.feature.composite_index.star_tree.enabled: true

chown -R opensearch:opensearch /etc/opensearch/
chmod 640 /etc/opensearch/opensearch.yml*

systemctl restart opensearch
systemctl status opensearch

re-enable

cat <<EOF | curl -sk -X PUT "$k/_cluster/settings?pretty" -u $admin_user:$admin_passwd \
        -H "Content-Type: application/json" -d@-
{
    "persistent": {
        "cluster.routing.allocation.enable": "all"
    }
}
EOF

troubleshooting

while upgrading the coordinator node, some index cache was found, need to clean-up – double quotes missing

vi /etc/default/opensearch

OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m"

fix

export OPENSEARCH_JAVA_HOME=/usr/share/opensearch/jdk
/usr/share/opensearch/bin/opensearch-node repurpose

resources

https://opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/index/

https://opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/rolling-upgrade/