GFS2 FROM SCRATCH

tested on Slackware Linux current (Aug 2020)

Corosync setup

totem {
    version: 2
    secauth: off
    cluster_name: pro5s
}

nodelist {
    node {
        ring0_addr: x.x.x.x
        nodeid: 1
    }
    node {
        ring0_addr: x.x.x.x
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 1
}

logging {
    to_syslog: yes
}

replicate the conf

scp /etc/corosync/corosync.conf pro5s2:/etc/corosync/

/etc/init.d/corosync start
netstat -lntup | grep corosync

DRBD setup

See drbd. Using v8 here as:

Requirements

zcat /proc/config.gz | grep GFS2

CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_DLM=m

slackpkg install libtool

Build

#git clone https://pagure.io/gfs2-utils.git
#cd gfs2-utils/
wget https://pagure.io/gfs2-utils/archive/3.3.0/gfs2-utils-3.3.0.tar.gz
cd gfs2-utils-3.3.0/
./autogen.sh
./configure --disable-nls
make > ../make.gfs2-utils.log && echo BUILT
make install
cd ../

#git clone https://pagure.io/dlm.git
#cd dlm/
wget https://pagure.io/dlm/archive/dlm-4.0.9/dlm-dlm-4.0.9.tar.gz
cd dlm-dlm-4.0.9/
vi dlm_controld/Makefile

comment this out (don’t just switch USE_SD_NOTIFY to no, that will prevent udev to work properly afterwards)

#ifeq ($(USE_SD_NOTIFY),yes)
#       BIN_CFLAGS += $(shell $(PKG_CONFIG) --cflags libsystemd) \
#                     -DUSE_SD_NOTIFY
#       BIN_LDFLAGS += $(shell $(PKG_CONFIG) --libs libsystemd)
#endif

and proceed

make clean
make > ../dlm.log && echo BUILT
make install
cd ../

DLM

there is no quorum on a two-node cluster

mkdir /etc/dlm/
vi /etc/dlm/dlm.conf

enable_quorum = 0
enable_fencing = 0
log_debug = 1
debug_logfile = 1

ls -lF /var/log/dlm_controld/dlm_controld.log

Format once

only once from some node

drbdadm status
drbdadm primary res-data

time mkfs.gfs2 -j2 -t pro5s:data /dev/drbd0
fsck.gfs2 -fy /dev/drbd0

Ready to go

on all nodes

vi /etc/rc.d/rc.local

echo DRBD
/sbin/modprobe drbd
/sbin/modprobe drbd_transport_tcp
/etc/init.d/drbd start
/usr/local/sbin/drbdadm primary res-data
echo

echo GFS2/DLM
/etc/init.d/corosync start
/sbin/modprobe dlm
/sbin/modprobe gfs2
/bin/mount -t configfs none /sys/kernel/config
/usr/sbin/dlm_controld
sleep 2
/bin/mount -t gfs2 -o noatime,nodiratime,noquota /dev/drbd0 /data
echo

vi /etc/rc.d/rc.local_shutdown

echo GFS2/DLM DOWN
/bin/umount -R /data/
lsof | grep data1
/bin/umount -R /data1/
lsof | grep data2
/bin/umount -R /data2/
#/bin/umount -t gfs2 -a
/usr/bin/pkill dlm_controld
/sbin/rmmod gfs2 dlm
/etc/init.d/corosync stop
echo

echo DRBD DOWN
#/etc/init.d/drbd stop
/usr/local/sbin/drbdadm down all
#v9 /sbin/rmmod drbd_transport_tcp
/sbin/rmmod drbd
echo

STATUS

echo DRBD
drbdadm status
echo

echo DLM
pgrep -a dlm_controld
dlm_tool ls
dlm_tool status
echo

echo COROSYNC
/etc/init.d/corosync status
echo

echo GFS2
mount | grep gfs2
echo

Additional notes

more mount options for GFS2

rw,suid,dev,exec,nouser,async,_netdev   0 0
_netdev,noauto
acl

Note fs_passno HAS TO BE ZERO.

Troubleshooting

lsmod | grep gfs2
ls -lF /usr/lib/udev/rules.d/51-dlm.rules
ls -lF /dev/misc/dlm-control
pgrep -a dlm

drbdadm status
corosync-quorumtool

dlm_controld -D --enable_quorum_fencing 0

you are getting

umount: /data2: target is busy.

==>

fuser -vm /data2
fuser -vmMki /data2
lsof | grep data2

Resources

GFS2 https://en.wikipedia.org/wiki/GFS2

GFS Project Page https://sourceware.org/cluster/gfs/

dlm https://pagure.io/dlm

gfs2/dlm usage in cluster4 http://people.redhat.com/teigland/cluster4-gfs2-dlm.txt

How to use gfs2/dlm directly http://people.redhat.com/teigland/cluster4-dlm.txt

https://linux.die.net/man/8/gfs2_mount

https://www.thegeekdiary.com/how-to-create-a-gfs2-formatted-cluster-file-system/

https://www.admin-magazin.de/Das-Heft/2012/01/HA-Workshop-Teil-7-GFS-mit-DRBD-und-Pacemaker/(offset)/6

https://xahteiwi.eu/resources/hints-and-kinks/gfs2-pacemaker-debianubuntu/

configfs - Userspace-driven kernel object configuration. https://www.kernel.org/doc/Documentation/filesystems/configfs/configfs.txt

target is busy https://stackoverflow.com/questions/7878707/how-to-unmount-a-busy-device

rhel kernel https://stackoverflow.com/questions/41314978/can-we-git-clone-the-redhat-kernel-source-code-and-see-the-changes-made-by-them https://github.com/kernelim/linux/tree/centos8 https://git.centos.org/rpms/kernel/raw/c8s/f/SPECS/kernel.spec https://wiki.centos.org/AdditionalResources/Repositories/CentOSPlus#CentOSPlus_kernel_differences https://wiki.centos.org/HowTos/I_need_the_Kernel_Source https://wiki.centos.org/HowTos/Custom_Kernel

RPM http://centos.mirrors.proxad.net/8.2.2004/BaseOS/x86_64/os/Packages/ http://mirror.nsc.liu.se/centos-store/8.2.2004/BaseOS/ http://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/

extract RPM https://www.cyberciti.biz/tips/how-to-extract-an-rpm-package-without-installing-it.html https://en.wikipedia.org/wiki/RPM_Package_Manager

guides

Clustering with Pacemaker, DRBD and GFS2 on Bare-Metal servers in SoftLayer https://icicimov.github.io/blog/high-availability/Clustering-with-Pacemaker-DRBD-and-GFS2-on-Bare-Metal-servers-in-SoftLayer/

all gone

The OpenGFS Project (2004…) http://opengfs.sourceforge.net/

DRAFTS

SRPM

mkdir -p ~/rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros

wget http://vault.centos.org/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.19.1.el8_2.src.rpm
#wget http://mirror.nsc.liu.se/centos-store/8.2.2004/BaseOS/Source/SPackages/kernel-4.18.0-193.19.1.el8_2.src.rpm

rpm -i kernel-4.18.0-193.19.1.el8_2.src.rpm

mv ~/rpmbuild/SOURCES/linux-4.18.0-193.19.1.el8_2.tar.xz /usr/src/

CRMSH

crm ra info ocf:pacemaker:controld
crm configure property stonith-enabled=false
crm configure show

crm
configure primitive gfs ocf:heartbeat:Filesystem params device="/dev/drbd0" directory="/data" fstype="gfs2"
...

configure primitive p_dlm_controld ocf:pacemaker:controld \
  params daemon="dlm_controld.pcmk" \
  op start interval="0" timeout="90" \
  op stop interval="0" timeout="100" \
  op monitor interval="10"

PCS

pcs cluster status

pcs resource create clusterfs Filesystem device="/dev/drbd0" directory="/data" fstype="gfs2" "options=noatime" op monitor interval=10s on-fail=fence clone interleave=true

HOME | COURSES | GUIDES | BENCHMARKS | SMTP HEALTH | NU | HTML5