Linux-HA old school

tested on slackware current (oct 2021)

Requirements

slackpkg install libxml2

hostname slack1
vi /etc/HOSTNAME

slack1

vi /etc/hosts

CLUSTER-IP       slack1
CLUSTER-IP       slack2
CLUSTER-IP       slack3

Install

groupadd -g 226 haclient
useradd -u 226 -g 226 -d / -s /sbin/nologin hacluster

we’re having too much troubles with the --libdir= we’re supposed to use for proper slackware builds, since many pathes are hard-coded into the heartbeat and agents scripts. this is why we’ve made tainted packages without defining libdir at configure time.

lftp -c 'open https://lab.nethence.com/; mirror --continue --delete slackpkgs/'
cd slackpkgs/
installpkg libnet-1.1.6-x86_64-1_SBo.tgz
installpkg glue-1.0.12-x86_64-1_SBo.tgz
installpkg heartbeat-3.0.6.11-x86_64-1_SBo.tgz
installpkg agents-1.0.4-x86_64-1_SBo.tgz
echo /lib >> /etc/ld.so.conf
echo /usr/lib >> /etc/ld.so.conf
echo /usr/local/lib >> /etc/ld.so.conf
ldconfig

Cluster setup

we disable auto_failback as we don’t care on what node a given resource will live on

slack1

mv -i /etc/ha.d/authkeys /etc/ha.d/authkeys.dist
vi /etc/ha.d/authkeys

auth 1
1 sha1 SOME-PASSPHRASE-HERE

chmod 600 /etc/ha.d/authkeys
mv -i /etc/ha.d/ha.cf /etc/ha.d/ha.cf.dist
grep -vE '^#|^$' /etc/ha.d/ha.cf.dist > /etc/ha.d/ha.cf.clean
vi /etc/ha.d/ha.cf

baud        460800
logfacility none
logfile     /var/log/ha-log
#debugfile  /var/log/ha-debug
#debug      1
auto_failback   off
crm             off
use_logd        off

#bcast xenbr0
#mcast xenbr0 239.0.0.10 694 1 0
ucast xenbr0 192.168.122.11
ucast xenbr0 192.168.122.12
ucast xenbr0 192.168.122.13

warntime 4
deadtime 5
initdead 15
keepalive 1

node slack1 slack2 slack3
autojoin none

we’re now ready to setup the resources

Basic floating IP resources (x3)

ls -lF /etc/ha.d/resource.d/IPaddr
mv -i /etc/ha.d/haresources /etc/ha.d/haresources.dist
vi /etc/ha.d/haresources

slack1 192.168.122.251/24/xenbr0
slack2 192.168.122.252/24/xenbr0
slack3 192.168.122.253/24/xenbr0

DRBD resource

warning: this resource is powered by an old-school LSB/Heartbeat script

grab our custom LSB/Heartbeat script

    ls -lF /etc/ha.d/resource.d/DRBD # no exist
    wget https://pub.nethence.com/bin/haresources/DRBD.txt -O /etc/ha.d/resource.d/DRBD
    chmod +x /etc/ha.d/resource.d/DRBD

check manually that it works

    res=dnc9

    /etc/ha.d/resource.d/DRBD $res start
    /etc/ha.d/resource.d/DRBD $res monitor
    /etc/ha.d/resource.d/DRBD $res stop

add the old-school resource definition

slack1 DRBD::dnc9

THTTPD resource

warning: this resource is powered by an old-school LSB/Heartbeat script

grab our custom LSB/Heartbeat script

    ls -lF /etc/ha.d/resource.d/THTTPD # no exist
    wget https://pub.nethence.com/bin/haresources/THTTPD.txt -O /etc/ha.d/resource.d/THTTPD
    chmod +x /etc/ha.d/resource.d/THTTPD

check manually that it works

vi /etc/thttpd.conf

port=80
user=thttpd
dir=/var/www/thttpd
chroot

    conf=/etc/thttpd.conf

service goes up

    /etc/ha.d/resource.d/THTTPD $conf start; echo $? # 0
    /etc/ha.d/resource.d/THTTPD $conf start; echo $? # 0
    /etc/ha.d/resource.d/THTTPD $conf monitor; echo $? # 0

service goes down

    /etc/ha.d/resource.d/THTTPD $conf stop; echo $? # 0
    /etc/ha.d/resource.d/THTTPD $conf stop; echo $? # 0
    /etc/ha.d/resource.d/THTTPD $conf monitor; echo $? # 1

add the old-school resource definition

slack1 THTTPD::/etc/thttpd.conf

XEN guest resource

warning: this resource is powered by an old-school LSB/Heartbeat script

grab our custom LSB/Heartbeat script

    ls -lF /etc/ha.d/resource.d/XEN # no exist
    wget https://pub.nethence.com/bin/haresources/XEN.txt -O /etc/ha.d/resource.d/XEN
    chmod +x /etc/ha.d/resource.d/XEN

check manually that it works

    guest=dnc9

    /etc/ha.d/resource.d/XEN /data/guests/$guest/$guest start
    /etc/ha.d/resource.d/XEN /data/guests/$guest/$guest monitor
    /etc/ha.d/resource.d/XEN /data/guests/$guest/$guest stop

add the old-school resource definition

    vi /etc/ha.d/haresources

    slack1 XEN::/data/guests/dnc9/dnc9

ideally we should be able to do as follows, but we’re not there yet

#slack1 DRBD::dnc9 XEN::/data/guests/dnc9/dnc9

Ready to go

populate and reload

slack1

scp /etc/ha.d/authkeys /etc/ha.d/ha.cf /etc/ha.d/haresources slack2:/etc/ha.d/
scp /etc/ha.d/authkeys /etc/ha.d/ha.cf /etc/ha.d/haresources slack3:/etc/ha.d/

scp /etc/ha.d/resource.d/DRBD slack2:/etc/ha.d/resource.d/
scp /etc/ha.d/resource.d/DRBD slack3:/etc/ha.d/resource.d/

scp /etc/ha.d/resource.d/THTTPD slack2:/etc/ha.d/resource.d/
scp /etc/ha.d/resource.d/THTTPD slack3:/etc/ha.d/resource.d/
scp /etc/thttpd.conf slack2:/etc/
scp /etc/thttpd.conf slack3:/etc/

    scp /etc/ha.d/resource.d/XEN slack2:/etc/ha.d/resource.d/
    scp /etc/ha.d/resource.d/XEN slack3:/etc/ha.d/resource.d/

    dsh -e -g xen /etc/rc.d/init.d/heartbeat reload

Maintenance

slack1

slackpkg install nmap libnl
nmap -sU -p694 slack2
nmap -sU -p694 slack3

start

/etc/rc.d/init.d/heartbeat start

and enable

vi /etc/rc.d/rc.local

# self-verbose
sysctl -w kernel.core_pattern=core.%e.%p.%h.%t
sysctl -w kernel.core_uses_pid=1
/etc/rc.d/init.d/heartbeat start

status

/etc/rc.d/init.d/heartbeat status
cl_status hbstatus -v
#cl_status listnodes -p
cl_status listnodes -n
cl_status listnodes -v

cl_status listhblinks slack1
cl_status listhblinks slack2
cl_status listhblinks slack3

cl_status nodestatus slack1
cl_status nodestatus slack2
cl_status nodestatus slack3

# rscstatus slack
cl_respawn THTTPD

stop

/etc/rc.d/init.d/heartbeat stop

Additional nodes

eventually some handy symlinks

cd ~/
ln -s /etc/ha.d/ha.cf
ln -s /etc/ha.d/haresources
ln -s /usr/lib/ocf/resource.d/heartbeat

Troubleshooting

draft

using heartbeat-3.0.6 on three nodes. trying to make the PoC for e.g. THTTPD. and the failover brings TWO daemons up instead of a single one, on that three node cluster.

ERROR: Both machines own our resources!
ERROR: Both machines own foreign resources!

TODO

Resources

Heartbeat http://www.linux-ha.org/wiki/Heartbeat

http://hg.linux-ha.org/heartbeat-STABLE_3_0/

Heartbeat 3.0 / summary http://hg.linux-ha.org/heartbeat-STABLE_3_0/summary

Linux-HA repositories http://hg.linux-ha.org/

https://wiki.clusterlabs.org/wiki/Initial_Configuration

http://linux-ha.org/wiki/Releases

Linux-HA downloads http://linux-ha.org/wiki/Download

Cluster Glue http://www.linux-ha.org/wiki/Cluster_Glue

tutorials

Getting Started with Linux-HA (heartbeat) https://snrg.cs.hku.hk/srg/html/cprobe/ha/GettingStarted.html

Getting Started with Heartbeat https://www.linuxjournal.com/article/9838

Heartbeat configuration http://wiki.kolmisoft.com/index.php/Heartbeat_configuration

debian/ubuntu packages

Package: heartbeat (1:3.0.6-10build1) [universe] https://packages.ubuntu.com/focal/heartbeat

heartbeat 1:3.0.6-10 source package in Ubuntu https://launchpad.net/ubuntu/+source/heartbeat/1:3.0.6-10

Package: fence-agents (4.5.2-1) [universe] https://packages.ubuntu.com/focal/fence-agents

fence-agents 4.5.2-1 source package in Ubuntu https://launchpad.net/ubuntu/+source/fence-agents/4.5.2-1

new-school resource agants

https://github.com/ClusterLabs/resource-agents

https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/Xen

old-school resource agents

Resource Agents http://www.linux-ha.org/wiki/Resource_Agents

Haresources http://www.linux-ha.org/wiki/Haresources

Resource Agents http://www.linux-ha.org/wiki/Resource_agents

LSB Resource Agents http://www.linux-ha.org/wiki/LSB_Resource_Agents

Heartbeat Resource Agents http://www.linux-ha.org/wiki/Heartbeat_Resource_Agents

OCF Resource Agents http://www.linux-ha.org/wiki/OCF_Resource_Agents

Cluster Resource Agents http://hg.linux-ha.org/agents/

http://linux-ha.org/doc/man-pages/re-ra-Xen.html

fencing

Configuration Explained https://clusterlabs.org/pacemaker/doc/deprecated/en-US/Pacemaker/2.0/html/Pacemaker_Explained/_fence_agents.html

moar & troubles

ERROR: Both machines own our resources! https://linux-ha.linux-ha.narkive.com/trG3u4Jy/error-both-machines-own-our-resources –> warntime deadtime initdead

[Linux-HA] how can i resolve “ERROR: Both machines own our resources!” https://www.mail-archive.com/linux-ha@lists.linux-ha.org/msg08629.html

Heartbeat-3-0-958e11be8686/heartbeat/config.c https://fossies.org/linux/Heartbeat/heartbeat/config.c

Kombinasi High Availability Cluster DRBD Dengan Heartbeat Menggunakan Centos 6.4 https://anapiscan.wordpress.com/2014/07/02/kombinasi-high-availability-cluster-drbd-dengan-heartbeat-menggunakan-centos-6-4/


HOME | GUIDES | LECTURES | LAB | SMTP HEALTH | HTML5 | CONTACT
Copyright © 2024 Pierre-Philipp Braun