XEN // Bootstrap FreeBSD/ZFS from GNU/Linux

freebsd full | freebsd cloud | freebsd ext2fs | freebsd zfs

tested on a Slackware current dom0 (Sep 2020)

Requirements

for the guest, we need a kernel with ZFS built-in because we don’t have a loader here and we cannot load deliver the zfs and opensolaris modules.

we then bootstrap the guest system from the XEN host/dom0, similarly to what was done with NetBSD and Slackware. therefore dom0 also needs to be ZFS-capable already

grab latest base tarball

    mkdir -p /data/kernels/freebsd130/
    cd /data/kernels/freebsd130/
    wget https://download.freebsd.org/releases/amd64/amd64/13.0-RELEASE/base.txz
    tar xaf kernel.txz

for userland it’s better to use bsdtar

    # ubuntu
    apt install libarchive-tools

    # slackware
    slackpkg search libarchive

Guest vdisk

raw sparse file virtual disk

mkdir -p /root/guests/freebsd-zfs/lala/
cd /root/guests/freebsd-zfs/

dd if=/dev/zero of=freebsd-zfs.zfs bs=1G count=0 seek=10

File-system

the kernel wants a GPT partition (GUID a504)

cgdisk freebsd-zfs.zfs

Part. #     Size        Partition Type            Partition Name
----------------------------------------------------------------
        1007.0 KiB  free space
   1        10.0 GiB    FreeBSD ZFS               zfs0

losetup -a
losetup --find --partscan --show freebsd-zfs.zfs

#cannot work - kernel won't find it
#zpool create tank `pwd`/freebsd-zfs.zfs

ls -lF /dev/loop*
zpool create tank /dev/loop0p1
zpool list
zpool status

zfs get compression
zfs set compression=lz4 tank
zfs create -o mountpoint=`pwd`/lala/ tank/root
zfs list
zfs mount

Userland

bsdtar -xJ --numeric-owner -f /data/kernels/freebsd130/base.txz -C lala/
ls -lhF lala/

no need for a valid file-system table unless you have swap, but still it’s good to have one, even empty, to avoid complains at system startup

vi lala/etc/fstab # new file

# Device                Mountpoint      FStype  Options         Dump    Pass#
#/dev/xbd0p2        none        swap    sw              0       0

cp -pi lala/etc/ttys lala/etc/ttys.dist
vi lala/etc/ttys

#ttyv0  "/usr/libexec/getty Pc"         xterm   onifexists secure
xc0     "/usr/libexec/getty Pc"         xterm   onifexists secure

zfs unmount `pwd`/lala/
zfs unmount /tank/
zfs mount
rmdir lala/

zpool export tank
zpool list

xen guest tuning

fix the serial console

    mv -i lala/etc/ttys lala/etc/ttys.dist
    vi lala/etc/ttys

    xc0     "/usr/libexec/getty Pc"         xterm   onifexists secure

network

    vi lala/etc/rc.conf

    hostname="freebsd.localdomain"

# esp. required for the cloud image
    growfs_enable="YES"

    ifconfig_xn0="inet CIDR up"
    defaultrouter="GW-IP"
    sshd_enable=yes

    mv lala/etc/ssh/sshd_config lala/etc/ssh/sshd_config.dist
    grep -vE '^#|^$' lala/etc/ssh/sshd_config.dist > lala/etc/ssh/sshd_config
    echo permitrootlogin prohibit-password >> lala/etc/ssh/sshd_config

    mkdir lala/root/.ssh/
cat ~/.ssh/*pub
    vi lala/root/.ssh/authorized_keys

(paste your pub key)

    chmod 600 lala/root/.ssh/authorized_keys
    chmod 700 lala/root/.ssh/
    chmod 700 lala/root/

    mv -i lala/etc/hosts lala/etc/hosts.dist
    vi lala/etc/hosts

    127.0.0.1               localhost.localdomain localhost
    ::1                     localhost.localdomain localhost

    10.1.0.9                freebsd.localdomain freebsd

    vi lala/etc/resolv.conf

    nameserver 10.1.255.19
    nameserver 10.1.255.20

environment

ls -lF lala/etc/localtime
ln -s ../usr/share/zoneinfo/Europe/Moscow lala/etc/localtime

losetup -a
losetup -d /dev/loop0

Guest skeleton

dom0

ls -lhF /data/kernels/kernel-zfs
vi freebsd-zfs

    # booting from custom ZFS kernel
kernel = "/data/kernels/kernel-zfs"
extra = "vfs.root.mountfrom=zfs:tank/root"
    #extra = "vfs.root.mountfrom=zfs:zroot/ROOT/default"
type = "pvh"

memory = 7168
name = "freebsd-zfs"
vcpus = 3
disk = ['tap:tapdisk:aio:/data/guests/freebsd-zfs/freebsd-zfs.zfs,xvda,w']
vif = [ 'bridge=xenbr0, vifname=fbsdzfs.0',
    'bridge=xenbr0, vifname=fbsdzfs.1' ]

Ready to go

Now that you got fstab and ttys alright

xl create freebsd-zfs -c

TODO

check about bootfs= of a zpool
check about GELI flags indications from loader

Additional notes

zfs post-install maintenance

mount and rescue from the XEN dom0 host

    losetup -a
    losetup --find --partscan --show freebsdzfs.img
    ls -lF /dev/loop*
    fdisk -l /dev/loop0
    #/dev/loop0p3 4196352 20969471 16773120    8G FreeBSD ZFS

    #zpool import -d /dev -R `pwd`/lala/ zroot freebsdzfs-zroot
    zpool import -d /dev -R `pwd`/lala/ -a
    #-o readonly=on -f
    zpool list
    zpool status

    zfs mount
    zfs mount zroot/ROOT/default

fix and rescue

    vi lala/etc/fstab
    vi lala/etc/ttys

and when finished

    zpool export freebsdzfs-zroot
    losetup -d /dev/loop0
    ls -lF /dev/loop*

Resources

boot zfs

Custom Kernel build and ZFS prefetch https://lists.freebsd.org/pipermail/freebsd-questions/2015-February/264111.html

ZFS module parameters in Linux (OR kernel parameter in FreeBSD or zfs-in-kernel OS) https://billtian.github.io/digoal.blog/2014/06/25/03.html

Make ZFS depend on xdr.ko only. It doesn’t need kernel RPC. https://github.com/freebsd/freebsd/commit/77755bd98c42004a010540bd9b6c171a557337a2