Setting up redis-cluster

Intro

This is not just HA. This is sharding. And to avoid the need for specific libraries and code tweaks to talk to the cluster, you can otherwise enable redis-cluster-proxy once this is in place.

Note you need to setup the cluster BEFORE you start writing in the databases.

Install

see redis-server

Initial setup

on every node

cd /etc/redis/
mv -i redis.conf redis.conf.dist
grep -vE '^[[:space:]]*#|^[[:space:]]*$' redis.conf.dist > redis.conf.clean
grep -vE '^[[:space:]]*#|^[[:space:]]*$' redis.conf.dist > redis.conf
vi redis.conf

bind 0.0.0.0

cluster-enabled yes
cluster-config-file nodes.conf
#cluster-node-timeout 5000
#appendonly yes

    # default acl user
    #requirepass SOME-PASSWORD

systemctl restart redis-server
systemctl status redis-server
systemctl enable redis-server

Cluster setup

now that the daemon is up and running on every node, let’s define the cluster memberships

configure the master nodes — not sure why, but we have to use IP addresses here

redis-cli --cluster create 10.3.3.11:6379 10.3.3.12:6379 10.3.3.13:6379
# -a PASSWORD
redis-cli cluster nodes
# -a PASSWORD
redis-cli --cluster check redis1:6379
# -a PASSWORD

add a slave to every master (here only the master node has to be ab IP)

redis-cli --cluster add-node redis5:6379 10.3.3.11:6379 \
    --cluster-slave --cluster-master-id 93724406643ce2833b6f118aa17e1467f5cf9f40

redis-cli --cluster add-node redis6:6379 10.3.3.12:6379 \
            --cluster-slave --cluster-master-id 83bacda2bc91020d2c0ea5101e6a9f640ce88a0c

redis-cli --cluster add-node redis4:6379 10.3.3.13:6379 \
            --cluster-slave --cluster-master-id 1f77b40916a407845bbfb36ec9b22dec7e19aa56

Initial test

available databases

redis-cli -h redis1 info | grep ^db
# -a PASSWORD
# | sed -rn '/^# Keyspace/,$p'

create key/value pairs that go to different shards on db0 e.g.

redis-cli -h redis1

set ref pouet
set ref2 pouet2
set ref3 pouet3

keys *

get ref
get ref2
get ref3

Maintenance

shards

re-distribute the shards

redis-cli --cluster fix redis1:6379

delete a node

redis-cli --cluster del-node redisX:6379 NODE-ID

horizontal scaling

once a new master/slave pair is added to the cluster, redistribute the shards as such

redis-cli --cluster reshard redisX:6379

Additional notes

in case you want to serve on different ports

port 6380

cluster-config-file nodes-6380.conf

pidfile /var/run/redis/redis-server-6380.pid
logfile /var/log/redis/redis-server-6380.log
dir /var/lib/redis-6380

vi ...SYSD UNIT...

systemctl daemon-reload
systemctl list-unit-files | grep redis
systemctl restart redis-server-6380.service
systemctl status redis-server-6380.service
systemctl enable redis-server-6380.service

Troubleshooting

[ERR] Not all 16384 slots are covered by nodes.

==> --cluster fix NODE-ID

WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

==> see Install

Resources

Scaling with Redis Cluster https://redis.io/docs/manual/scaling/

CLUSTER NODES https://redis.io/commands/cluster-nodes

Redis Cluster https://redis.io/presentation/Redis_Cluster.pdf

https://www.linode.com/docs/guides/how-to-install-and-configure-a-redis-cluster-on-ubuntu-1604/

troubles

https://stackoverflow.com/questions/39568561/how-to-solve-redis-cluster-waiting-for-the-cluster-to-join-issue


HOME | GUIDES | LECTURES | LAB | SMTP HEALTH | NU | HTML5
Copyright © 2022 Pierre-Philipp Braun