Running NetBSD on Linux Ext2 File-System

Warning

This is not about boot strapping. This is rather on how to fully boot and run NetBSD over ext2fs. So what ever method you used to load the kernel is fine. We are taking the XEN/PV route here, for convenience.

This is PoC, just for fun. And there are mainly three ways to do this:

  1. use BSD partition d as root-device – that is MUCH EASIER but not possible on bare-metal
  2. trick around with SunOS BSD labels from util-linux
  3. cross-compile netbsd disklabel on gnu/linux and use it from there

This guide is about the second way. In short:

Requirements

Get the latest sets

mkdir -p /data/kernels/netbsd/sets/
cd /data/kernels/netbsd/sets/

#mv releasedir/amd64/binary/sets ./
#mv releasedir/amd64/binary/kernel/netbsd*.gz ./
#rm -rf releasedir/
source=http://ftp.fr.netbsd.org/pub/NetBSD-daily/HEAD/201903291230Z/amd64
for set in kern-GENERIC kern-XEN3_DOMU kern-XEN3_DOM0 modules base comp debug etc man misc tests text; do
    echo -n $source/binary/sets/$set...
    wget -q $source/binary/sets/$set.tar.xz && echo done
done; unset set
cd ../
wget $source/binary/kernel/netbsd-XEN3_DOMU.gz
wget $source/binary/kernel/netbsd-INSTALL.gz
wget $source/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz

Prepare a guest

mkdir /root/guests/netbsdextfs/
cd /root/guests/netbsdextfs/
rm -f netbsdextfs.sparse netbsdextfs.data.sparse
dd if=/dev/zero of=netbsdextfs.sparse bs=1000k count=0 seek=4000
dd if=/dev/zero of=netbsdextfs.data.sparse bs=1000k count=0 seek=4000
cat > netbsdextfs <<EOF
#kernel = "/data/kernels/netbsd/netbsd-INSTALL_XEN3_DOMU.gz"
kernel = "/data/kernels/netbsd/netbsd-XEN3_DOMU.gz"
root = "xbd0a"
extra = "-v -s"
memory = 512
name = "netbsdextfs"
vcpus = 4
disk = ['tap:tapdisk:aio:/data/guests/netbsdextfs/netbsdextfs.sparse,xvda,w',
'tap:tapdisk:aio:/data/guests/netbsdextfs/netbsdextfs.data.sparse,xvdb,w']
vif = [ 'bridge=xenbr0,vifname=netbsdextfs.0' ]
EOF

Leaving some space for the BSD labels

start clean

losetup --list | grep netbsd

prepare the DOS partitions

fdisk -v #fdisk from util-linux 2.31.1
fdisk netbsdextfs.sparse

n p 1 2048 ALL t a9 a/BOOTABLE
p w

now create some (Solaris?) BSD labels from gnu/linux. We will not use it as-is on NetBSD. We just use it temporarily, first to find and locate a place for the ext2fs, then for netbsd to actually find its root-device at sector 4096.

fdisk netbsdextfs.sparse

b
Do you want to create a BSD disklabel? [Y]es/[N]o: y
d d
n a 4096 ALL
t a a/unknown otherwise unable to mount 0/unused
p w q

#losetup --all
losetup --list
losetup --detach /dev/loop1
losetup --find --partscan --show netbsdextfs.sparse
ls -lkF /dev/loop1*
mkfs.ext2 -O^dir_index,^ext_attr /dev/loop1p5
tune2fs -l /dev/loop1p5 | grep features

should match what we support

resize_inode filetype sparse_super large_file

and proceed

mount /dev/loop1p5 /mnt
ls -alkF /mnt/
#warning you may get all kernels extracted over there, last one being DOMU
for set in /data/kernels/netbsd/sets/*.tar.xz; do
    echo -n $set...
    tar xJph --numeric-owner -f $set -C /mnt && echo done
done; unset set
cp -v /data/kernels/netbsd/*.gz /mnt/
ls -alkF /mnt/ /mnt/dev/
cd /mnt/dev/
cp -pi MAKEDEV MAKEDEV.linux
vi MAKEDEV.linux

…get rid of -F -R -r -g -u

./MAKEDEV.linux all
ls -alkF
cd /root/guests/netbsdextfs/
ls -lkF /mnt/etc/passwd*
mv /mnt/etc/passwd /mnt/etc/passwd.dist
sed '1s@/bin/sh@/bin/ksh@' /mnt/etc/passwd.dist > /mnt/etc/passwd
diff -u /mnt/etc/passwd.dist /mnt/etc/passwd

cat > /mnt/etc/fstab <<EOF
# no dump
/dev/xbd0a / ext2fs rw 0 1
#force
EOF
cat /mnt/etc/fstab

mv -i /mnt/etc/rc.conf /mnt/etc/rc.conf.dist
cat > /mnt/etc/rc.conf <<EOF
. /etc/defaults/rc.conf
rc_configured=yes
rc_silent=no # /var/run/rc.log
no_swap=yes
savecore=no

wscons=no
EOF
cat /mnt/etc/rc.conf

If you want to skip the learning curve of this guide, now is a good moment to have a look at the EMBED THE TWEAK section.

umount /mnt
fsck.ext2 /dev/loop1p5
fsck.ext2 -f /dev/loop1p5
losetup --detach /dev/loop1
ls -lkF /dev/loop1* #dos and bsd partitions are gone

reaching the single user mode as requested with extra =

cd /data/guests/netbsdextfs/
xl create netbsdextfs -c

/bin/ksh
alias ll='ls -alkF'

Supported Ext2 Features

Let us test the features that NetBSD 8.99.36 implements

newfs_ext2fs -I -O1 /dev/rxbd1d

then from the gnu/linux host

file netbsdextfs.data.sparse
tune2fs -l netbsdextfs.data.sparse
dumpe2fs netbsdextfs.data.sparse > netbsdextfs.data.sparse.dumpe2fs.dist
dumpe2fs netbsdextfs.data.sparse | grep ^Group | wc -l #31
dumpe2fs netbsdextfs.data.sparse | grep 'Backup superblock' | wc -l #7

gives

Linux rev 1.0 ext2 filesystem data, UUID=29a111d1-03f8-9048-9b95-30446b526f74 (large files)

tune2fs 1.44.1 (24-Mar-2018)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          29a111d1-03f8-9048-9b95-30446b526f74
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      resize_inode filetype sparse_super large_file
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              500960
Block count:              1000000
Reserved block count:     50000
Free blocks:              982309
Free inodes:              500949
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      244
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16160
Inode blocks per group:   505
Last mount time:          n/a
Last write time:          Sat Mar 30 16:12:01 2019
Mount count:              0
Maximum mount count:      20
Last checked:             Sat Mar 30 16:12:01 2019
Check interval:           15552000 (6 months)
Next check after:         Thu Sep 26 16:12:01 2019
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           128

Fixing Device File Ownerships and Disk Label

check that NO tmpfs /dev is there

mount
mount -vv
ls -alkF /dev/xbd0*
ls -alkF /dev/rxbd0*

notice difference on what the kernel vs disk says

disklabel xbd0
disklabel -r xbd0

we not only need this for -e/-R to work on /tmp

#mount -t tmpfs none /tmp
mount -u -o rw /dev/xbd0a /

but also for fixing device files ownerships

cd /dev/
./MAKEDEV all
cd /

craft your own (total sectors / 2048 = cylinders)

cat > xbd0c.label <<EOF99
# /dev/rxbd0:
type: ESDI
disk: xbd0
label: crafted
flags:
bytes/sector: 512
sectors/track: 2048
tracks/cylinder: 1
sectors/cylinder: 2048
cylinders: 3906
total sectors: 8000000
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

4 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:   7995904      4096 Linux Ext2 4096 0
 c:   7997952      2048     unused 0 0
 d:   8000000         0     unused 0 0
EOF99
disklabel -R xbd0 xbd0c.label
disklabel -R -r xbd0 xbd0c.label
disklabel xbd0
disklabel -r xbd0

then exit single user mode and the system boots

^D

EMBED THE TWEAK

cp -pi /mnt/etc/rc /mnt/etc/rc.dist
vi /mnt/etc/rc

...
umask 022

#CWD is /
echo -n RE-GENERATING DEVICE FILES WITH PROPER OWNERSHIPS...
cd /dev/
./MAKEDEV all && echo DONE
cd /
echo CURRENT MOUNT SITUATION IS
mount
echo -n MOUNTING /tmp AS TMPFS...
mount -t tmpfs none /tmp && echo DONE
echo MOUNT SITUATION IS NOW
mount
echo -n WRITING /tmp/xbd0c.label...
cat > /tmp/xbd0c.label <<EOF99 && echo DONE
# /dev/rxbd0:
type: ESDI
disk: xbd0
label: crafted
flags:
bytes/sector: 512
sectors/track: 2048
tracks/cylinder: 1
sectors/cylinder: 2048
cylinders: 3906
total sectors: 8000000
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0           # microseconds
track-to-track seek: 0  # microseconds
drivedata: 0 

4 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
 a:   7995904      4096 Linux Ext2 4096 0
 c:   7997952      2048     unused 0 0
 d:   8000000         0     unused 0 0
EOF99
echo -n APPLYING DISKLABEL FIXUP...
disklabel -R xbd0 /tmp/xbd0c.label && echo DONE
echo -n YET AGAIN ON THE DISK...
disklabel -R -r xbd0 /tmp/xbd0c.label && echo DONE
echo -n GETTING RID OF TMPFS...
umount -aft tmpfs && echo DONE
echo MOUNT SITUATION IS BACK TO
mount

DO NOT FORGET TO GET RID OF THOSE TWEAKS ONCE THE SYSTEM HAS BOOTED PROPERLY. YOU DO NOT WANT TO RE-WRITE THE LABELS AND RE-GENERATE DEVICE FILES AT EVERY SYSTEM STARTUP.

Troubleshooting

DISK LABELS

disk label it is on sector 1 (not 0 which is reserved for mbr, even on a DOS partition) of the a9/netbsd DOS partition

dd if=/dev/xbd0d bs=512 skip=2049 count=1 | hexdump -vC

EXT2

modstat | grep ext2fs

# fsck_ext2fs /dev/xbd0a                                                       
** /dev/rxbd0a
BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN FIRST ALTERNATE

LOOK FOR ALTERNATE SUPERBLOCKS? [yn] y


CANNOT READ: BLK 8388608
CONTINUE? [yn] ^C

reproduce with

#mkfs.ext2 -r0 netbsdextfs.data.sparse
mkfs.ext2 -O^dir_index,^ext_attr netbsdextfs.data.sparse

disklabel -e xbd1 

 d:   8000000         0 Linux Ext2   4096     0        # (Cyl.      0 -   3906*)

fixed by mlelstvsbin/fsck_ext2fs/setup.c 1.36

Traces

DOS partitions

Disk /dev/loop1: 3.8 GiB, 4096000000 bytes, 8000000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7e91a4e9

Device       Boot Start     End Sectors  Size Id Type
/dev/loop1p1 *     2048 7999999 7997952  3.8G a9 NetBSD

BSD labels

Disk /dev/loop1p1: 3.8 GiB, 4096000000 bytes, 8000000 sectors
Geometry: 1 heads, 2048 sectors/track, 3906 cylinders
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: bsd

Slice Start     End Sectors  Size Type     Fsize Bsize Cpg
a      4096 7999999 7995904  3.8G unknown      0     0   0
c      2048 7999999 7997952  3.8G unused       0     0   0
d         0 7999999 8000000  3.8G unused       0     0   0

Partition table entries are not in disk order.

TODO

check if -u can be used with MAKEDEV to just update ownerships

Resources

Troubleshooting


Nethence | Pub | Lab | Pbraun | SNE Russia | xhtml