Setting up and maintaining an FTP/HTTP/RSYNC mirror

with NetBSD 8/9

Requirements

assuming a xen guest with additional drive

newfs -O2 /dev/xbd1a
#-b 64k
vi /etc/fstab

/dev/xbd1a      /tftpboot       ffs  rw      1 2

mount /tftpboot/
df -hT

vi .profile

PKG_PATH=

. ./.profile
echo $PKG_PATH
pkg_add -v rsync screen
cp /usr/pkg/share/examples/screen/etcscreenrc /usr/pkg/etc/screenrc
ln -s /usr/pkg/etc/screenrc
vi screenrc

caption always "%-Lw%{= BW}%50>%n%f* %t%{-}%+Lw%<"

NetBSD

Get a almost-fresh copy first-shot from your closesd mirror

mkdir -p /tftpboot/pub/NetBSD/
rsync -v -Wartz --delete-after rsync://rsync.fr.NetBSD.org/NetBSD/NetBSD-9.0/ /tftpboot/pub/NetBSD/NetBSD-9.0/

then re-sync against the true master based on file size, not timestamp

rsync -Wrltvz --size-only --delete-after rsync://rsync.NetBSD.org/NetBSD/NetBSD-9.0/ /tftpboot/pub/NetBSD/NetBSD-9.0/
#--checksum

Ready to go

make sure your system is able to send emails and proceed

crontab -e

50      3       *       *       *       /root/MIRROR

vi /root/MIRROR

#!/bin/ksh

/usr/pkg/bin/rsync -Wrltvz --size-only --delete-after \
        rsync://rsync.NetBSD.org/NetBSD/NetBSD-9.0/ /tftpboot/pub/NetBSD/NetBSD-9.0/
        #--checksum

chmod +x /root/MIRROR

Additional notes

mirror providers

Slackware – https://mirrors.slackware.com/

https://mirrors.slackware.com/slackware/

wget

    wget -r -N --no-if-modified-since --relative -l 1 -nH https://lab.nethence.com/slackpkgs/
    # --mirror == -r -N -l inf --no-remove-listing
    # --progress=dot:mega --tries=0
    rm -f slackpkgs/index.html

Services

FTP – see lukemftpd HTTP – see bozohttpd RSYNC – see rsyncd

Resources

https://www.netbsd.org/mirrors/

https://www.netbsd.org/docs/mirror.html

https://download.samba.org/pub/rsync/rsyncd.conf.html