TODO

UFS2 write support experimental in Linux mainline in 2025?
This script works for not but it is just dying to die... I really just wanted to make a point that there is a lot that can be trivialized in this process and relying cd images to install anything in 2025 is just out of the question for me. I don't even want to do it for physical hardware and if I just got my shit together a little better, I don't think I would have an excuse given things like iPXE.

Take aways

I just can't really believe how little a UDM pro actually delivers in terms of being able to do simple things like a VRF. Well afaik UDM is Linux and in my experience VRF on Linux is more of an insult to injury compared to Free/OpenBSD's fib/rtable. I guess NetNS does it's job, well enough that people can use it for Docker and somewhere in the world I think somebody even uses VRF in the context of Docker (custom OCI network driver.) It's still a nightmare for the things I want to do with it and it's actually a huge part of why I've come back to BSD after... 20 years.

Usage

./script.sh BIGVM -t littletemplate and template.conf go in /vm/.templates

Configuration

A script

I made this script with Cursor, it's really meant to be a proof of concept but it will probably end up just being my goto for awhile. It does a lot, but it uses BHyve-vm to minimize some of the work, I am keeping this script in /vm/create_vm.sh for now. To summarize what this does though:

Fetches RootFS for Void Linux if it doesn't exist in /mnt/pub
Clones the v6.14 branch of the Linux kernel to /mnt/pub (if it hasn't been yet)
Creates the VM
Sets the associated zvol mode to geom
Creates an EFI partition table and EFI filesystem
Creates a UFS filesystem FreeBSD doesn't really have a NewFS for much else, I'll get to how this works in a moment. ZFS in ZFS didn't seem like a very good idea, though and I couldn't figure out how to probably because you can't. ZFS has been a bit of a learning curve for me but I'm starting to really like it.
Extracts the RootFS
Mounts /dev /proc and /sys
Loop mounts the Linux kernel source from /mnt/pub to the chroot
Does a chroot into a linux chroot because Linux compatibility layer (I love that)
Builds the kernel if it's not already built
Creates a startup.nsh because we're really pushing our luck here already okay, you're not going to get GRUB and efibootmgr to play nice in a chroot under FreeBSD's Linux compatibility layer. If I'm being honest with you I could care less; the Linux kernel is capable of executing as an EFI executable itself (EFI STUB) and honestly I'm kinda mad at myself for not realizing that I could do this with startup.nsh sooner. If you are fortunate enough to have EFIShell on your system, then you don't even need a bootloader, you can just do it this way!
Sets a user and some boring stuff, unmounts everything and cleans up:

#!/usr/bin/env bash

set -e

# Function to handle cleanup on exit
cleanup() {
    local exit_code=$?
    echo "Cleaning up..."
    
    # First try to exit chroot if we're in it
    if [ "$(pwd)" = "/mnt/void-install" ]; then
        cd /vm || cd /
    fi
    
    # Wait a moment for any processes to finish
    sleep 1
    
    # Unmount in reverse order of mounting, with retries
    for mount_point in "/mnt/void-install/usr/src/linux" \
                      "/mnt/void-install/proc" \
                      "/mnt/void-install/sys" \
                      "/mnt/void-install/dev" \
                      "/mnt/void-install/boot/efi" \
                      "/mnt/void-install"; do
        if mount | grep -q "$mount_point"; then
            echo "Unmounting $mount_point..."
            for i in {1..3}; do
                if umount "$mount_point" 2>/dev/null; then
                    break
                fi
                echo "Retry $i: Waiting for $mount_point to be unmountable..."
                sleep 2
            done
        fi
    done
    
    # If we failed and VM exists, destroy it
    if [ $exit_code -ne 0 ] && [ -n "$VM_NAME" ]; then
        echo "Installation failed for VM '${VM_NAME}' with template '${VM_TEMPLATE}', destroying VM..."
        vm destroy -f "$VM_NAME" || echo "Warning: Failed to destroy VM ${VM_NAME}"
    fi
    
    exit $exit_code
}

# Set up trap to call cleanup function on script exit
trap cleanup EXIT

# Check if nmdm kernel module is loaded, if not load it
if ! kldstat -m nmdm > /dev/null 2>&1; then
    echo "Loading nmdm kernel module..."
    kldload nmdm || { echo "Failed to load nmdm kernel module"; exit 1; }
fi

# Parse command line arguments
VM_TEMPLATE="void" # Default template

usage() {
    echo "Usage: $0 <vm_name> [options]"
    echo "Options:"
    echo "  -t, --template <template>   Specify the VM template to use (default: void)"
    echo "  -h, --help                  Show this help message"
    echo ""
    echo "Examples:"
    echo "  $0 my-vm                    # Create VM 'my-vm' with default 'void' template"
    echo "  $0 my-vm -t freebsd         # Create VM 'my-vm' with 'freebsd' template"
    echo "  $0 my-vm --template debian  # Create VM 'my-vm' with 'debian' template"
    exit 1
}

# Check if at least one argument is provided
if [ $# -lt 1 ]; then
    usage
fi

VM_NAME="$1"
shift

# Parse remaining arguments
while [ $# -gt 0 ]; do
    case "$1" in
        -t|--template)
            VM_TEMPLATE="$2"
            shift 2
            ;;
        -h|--help)
            usage
            ;;
        *)
            echo "Unknown option: $1"
            usage
            ;;
    esac
done

echo "Creating VM with name: ${VM_NAME}, template: ${VM_TEMPLATE}"

# Create the VM
vm create -t ${VM_TEMPLATE} ${VM_NAME} || { echo "Failed to create VM"; exit 1; }

# Prepare zvol
zfs set volmode=geom storage/vm/${VM_NAME}/disk0 || { echo "Failed to set zvol mode"; exit 1; }

# Partition disk
gpart create -s gpt /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to create GPT partition table"; exit 1; }
gpart add -t efi -s 256M /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to add EFI partition"; exit 1; }
gpart add -t freebsd-ufs /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to add UFS partition"; exit 1; }

# Format partitions
newfs_msdos /dev/zvol/storage/vm/${VM_NAME}/disk0p1 || { echo "Failed to format EFI partition"; exit 1; }
newfs /dev/zvol/storage/vm/${VM_NAME}/disk0p2 || { echo "Failed to format UFS partition"; exit 1; }

# Create mount points and mount partitions
mkdir -p /mnt/void-install || { echo "Failed to create mount point"; exit 1; }
mount /dev/zvol/storage/vm/${VM_NAME}/disk0p2 /mnt/void-install || { echo "Failed to mount root partition"; exit 1; }
mkdir -p /mnt/void-install/boot/efi || { echo "Failed to create EFI mount point"; exit 1; }
mount -t msdosfs /dev/zvol/storage/vm/${VM_NAME}/disk0p1 /mnt/void-install/boot/efi || { echo "Failed to mount EFI partition"; exit 1; }

# Check if rootfs archive exists in /mnt/pub, if not download it
if [ ! -f "/mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz" ]; then
    mkdir -p /mnt/pub || { echo "Failed to create /mnt/pub directory"; exit 1; }
    wget -O /mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz https://repo-default.voidlinux.org/live/current/void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to download rootfs"; exit 1; }
fi

# Download and extract rootfs
cd /mnt/void-install || { echo "Failed to change to mount directory"; exit 1; }

cp /mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz . || { echo "Failed to copy rootfs archive"; exit 1; }
bsdtar -x --no-xattrs -f void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to extract rootfs"; exit 1; }
rm void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to remove rootfs archive"; exit 1; }

# Prepare chroot
mount -t linprocfs linprocfs /mnt/void-install/proc || { echo "Failed to mount proc"; exit 1; }
mount -t linsysfs linsysfs /mnt/void-install/sys || { echo "Failed to mount sys"; exit 1; }
mount -t devfs devfs /mnt/void-install/dev || { echo "Failed to mount dev"; exit 1; }
cp /etc/resolv.conf /mnt/void-install/etc/ || { echo "Failed to copy resolv.conf"; exit 1; }
echo "135.181.160.58 repo-default.voidlinux.org" >> /mnt/void-install/etc/hosts || { echo "Failed to update hosts file"; exit 1; }

# Create /etc/fstab for the VM
cat << EOF > /mnt/void-install/etc/fstab || { echo "Failed to create fstab"; exit 1; }
/dev/vda2 / ufs defaults 0 1
/dev/vda1 /boot/efi msdos defaults 0 2
EOF

# Create script to run inside chroot
cat << EOF > /mnt/void-install/setup.sh || { echo "Failed to create setup script"; exit 1; }
#!/bin/bash

set -e

# VM template used for creation
VM_TEMPLATE="${VM_TEMPLATE}"
VM_NAME="${VM_NAME}"

xbps-pkgdb -m hold linux
xbps-pkgdb -m hold linux-headers
xbps-install --force -Suy
xbps-install --force -y gcc make flex bison libelf rsync elfutils-devel bc openssl-devel perl base-system xz \
tar xz bc elfutils-devel flex gmp-devel kmod libmpc-devel pkg-config openssl-devel perl uboot-mkimage \
cpio pahole python3 zstd

# Print VM info
echo "Setting up VM '${VM_NAME}' created with template '${VM_TEMPLATE}'"

cd /usr/src/linux

if [ ! -f ".config" ]; then
    make mrproper
    make clean
    make defconfig
    make kvm_guest.config

    scripts/config -e CONFIG_EFI_STUB
    scripts/config -e CONFIG_HZ_100
    scripts/config -d CONFIG_HZ_250
    scripts/config -d CONFIG_HZ_300
    scripts/config -d CONFIG_HZ_1000
    scripts/config -d CONFIG_NO_HZ_IDLE
    scripts/config -d CONFIG_NO_HZ
    scripts/config -e CONFIG_PREEMPT
    scripts/config -e CONFIG_BOOT_CONFIG
    scripts/config -e CONFIG_HZ_PERIODIC
    scripts/config -e CONFIG_MSDOS_FS
    scripts/config -e CONFIG_VFAT_FS
    scripts/config -e CONFIG_EXFAT_FS
    scripts/config -e CONFIG_UFS_FS
    scripts/config -e CONFIG_UFS_FS_WRITE
    scripts/config -e CONFIG_SCSI_VIRTIO
    scripts/config -e CONFIG_DRM_VIRTIO_GPU
    scripts/config -e CONFIG_HW_RANDOM_VIRTIO
    scripts/config -e CONFIG_NFS_FS
    scripts/config -e CONFIG_NFS_V2
    scripts/config -e CONFIG_NFS_V3
    scripts/config -e CONFIG_NFS_V3_ACL
    scripts/config -e CONFIG_NFS_V4
    scripts/config -e CONFIG_NFS_SWAP
    scripts/config -e CONFIG_NFS_V4_1
    scripts/config -e CONFIG_NFS_V4_2
    scripts/config -e CONFIG_HAVE_KERNEL_GZIP
    scripts/config -d CONFIG_HAVE_KERNEL_BZIP2
    scripts/config -d CONFIG_HAVE_KERNEL_LZMA
    scripts/config -d CONFIG_HAVE_KERNEL_XZ
    scripts/config -d CONFIG_HAVE_KERNEL_LZO
    scripts/config -d CONFIG_HAVE_KERNEL_LZ4
    scripts/config -d CONFIG_HAVE_KERNEL_ZSTD
    scripts/config -e CONFIG_KERNEL_GZIP
    scripts/config -d CONFIG_KERNEL_BZIP2
    scripts/config -d CONFIG_KERNEL_LZMA
    scripts/config -d CONFIG_KERNEL_XZ
    scripts/config -d CONFIG_KERNEL_LZO
    scripts/config -d CONFIG_KERNEL_LZ4
    scripts/config -d CONFIG_KERNEL_ZSTD
    scripts/config -m CONFIG_NET_KEY
    scripts/config -m CONFIG_NET_HANDSHAKE_KUNIT_TEST
    scripts/config -m CONFIG_NET_IPIP
    scripts/config -m CONFIG_NET_IPGRE_DEMUX
    scripts/config -m CONFIG_NET_IP_TUNNEL
    scripts/config -m CONFIG_NET_IPGRE
    scripts/config -m CONFIG_NET_IPVTI
    scripts/config -m CONFIG_NET_UDP_TUNNEL
    scripts/config -m CONFIG_NET_FOU
    scripts/config -m CONFIG_NET_DSA
    scripts/config -m CONFIG_NET_DSA_TAG_NONE
    scripts/config -m CONFIG_NET_DSA_TAG_AR9331
    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_COMMON
    scripts/config -m CONFIG_NET_DSA_TAG_BRCM
    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_LEGACY
    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_PREPEND
    scripts/config -m CONFIG_NET_DSA_TAG_HELLCREEK
    scripts/config -m CONFIG_NET_DSA_TAG_GSWIP
    scripts/config -m CONFIG_NET_DSA_TAG_DSA_COMMON
    scripts/config -m CONFIG_NET_DSA_TAG_DSA
    scripts/config -m CONFIG_NET_DSA_TAG_EDSA
    scripts/config -m CONFIG_NET_DSA_TAG_MTK
    scripts/config -m CONFIG_NET_DSA_TAG_KSZ
    scripts/config -m CONFIG_NET_DSA_TAG_OCELOT
    scripts/config -m CONFIG_NET_DSA_TAG_OCELOT_8021Q
    scripts/config -m CONFIG_NET_DSA_TAG_QCA
    scripts/config -m CONFIG_NET_DSA_TAG_RTL4_A
    scripts/config -m CONFIG_NET_DSA_TAG_RTL8_4
    scripts/config -m CONFIG_NET_DSA_TAG_RZN1_A5PSW
    scripts/config -m CONFIG_NET_DSA_TAG_LAN9303
    scripts/config -m CONFIG_NET_DSA_TAG_SJA1105
    scripts/config -m CONFIG_NET_DSA_TAG_TRAILER
    scripts/config -m CONFIG_NET_DSA_TAG_VSC73XX_8021Q
    scripts/config -m CONFIG_NET_DSA_TAG_XRS700X
    scripts/config -m CONFIG_NET_SCH_HTB
    scripts/config -m CONFIG_NET_SCH_HFSC
    scripts/config -m CONFIG_NET_SCH_PRIO
    scripts/config -m CONFIG_NET_SCH_MULTIQ
    scripts/config -m CONFIG_NET_SCH_RED
    scripts/config -m CONFIG_NET_SCH_SFB
    scripts/config -m CONFIG_NET_SCH_SFQ
    scripts/config -m CONFIG_NET_SCH_TEQL
    scripts/config -m CONFIG_NET_SCH_TBF
    scripts/config -m CONFIG_NET_SCH_CBS
    scripts/config -m CONFIG_NET_SCH_ETF
    scripts/config -m CONFIG_NET_SCH_MQPRIO_LIB
    scripts/config -m CONFIG_NET_SCH_TAPRIO
    scripts/config -m CONFIG_NET_SCH_GRED
    scripts/config -m CONFIG_NET_SCH_NETEM
    scripts/config -m CONFIG_NET_SCH_DRR
    scripts/config -m CONFIG_NET_SCH_MQPRIO
    scripts/config -m CONFIG_NET_SCH_SKBPRIO
    scripts/config -m CONFIG_NET_SCH_CHOKE
    scripts/config -m CONFIG_NET_SCH_QFQ
    scripts/config -m CONFIG_NET_SCH_CODEL
    scripts/config -m CONFIG_NET_SCH_FQ_CODEL
    scripts/config -m CONFIG_NET_SCH_CAKE
    scripts/config -m CONFIG_NET_SCH_FQ
    scripts/config -m CONFIG_NET_SCH_HHF
    scripts/config -m CONFIG_NET_SCH_PIE
    scripts/config -m CONFIG_NET_SCH_FQ_PIE
    scripts/config -m CONFIG_NET_SCH_INGRESS
    scripts/config -m CONFIG_NET_SCH_PLUG
    scripts/config -m CONFIG_NET_SCH_ETS
    scripts/config -m CONFIG_NET_CLS_BASIC
    scripts/config -m CONFIG_NET_CLS_ROUTE4
    scripts/config -m CONFIG_NET_CLS_FW
    scripts/config -m CONFIG_NET_CLS_U32
    scripts/config -m CONFIG_NET_CLS_FLOW
    scripts/config -m CONFIG_NET_CLS_CGROUP
    scripts/config -m CONFIG_NET_CLS_BPF
    scripts/config -m CONFIG_NET_CLS_FLOWER
    scripts/config -m CONFIG_NET_CLS_MATCHALL
    scripts/config -m CONFIG_NET_EMATCH_CMP
    scripts/config -m CONFIG_NET_EMATCH_NBYTE
    scripts/config -m CONFIG_NET_EMATCH_U32
    scripts/config -m CONFIG_NET_EMATCH_META
    scripts/config -m CONFIG_NET_EMATCH_TEXT
    scripts/config -m CONFIG_NET_EMATCH_CANID
    scripts/config -m CONFIG_NET_EMATCH_IPSET
    scripts/config -m CONFIG_NET_EMATCH_IPT
    scripts/config -m CONFIG_NET_ACT_POLICE
    scripts/config -m CONFIG_NET_ACT_GACT
    scripts/config -m CONFIG_NET_ACT_MIRRED
    scripts/config -m CONFIG_NET_ACT_SAMPLE
    scripts/config -m CONFIG_NET_ACT_NAT
    scripts/config -m CONFIG_NET_ACT_PEDIT
    scripts/config -m CONFIG_NET_ACT_SIMP
    scripts/config -m CONFIG_NET_ACT_SKBEDIT
    scripts/config -m CONFIG_NET_ACT_CSUM
    scripts/config -m CONFIG_NET_ACT_MPLS
    scripts/config -m CONFIG_NET_ACT_VLAN
    scripts/config -m CONFIG_NET_ACT_BPF
    scripts/config -m CONFIG_NET_ACT_CONNMARK
    scripts/config -m CONFIG_NET_ACT_CTINFO
    scripts/config -m CONFIG_NET_ACT_SKBMOD
    scripts/config -m CONFIG_NET_ACT_IFE
    scripts/config -m CONFIG_NET_ACT_TUNNEL_KEY
    scripts/config -m CONFIG_NET_ACT_CT
    scripts/config -m CONFIG_NET_ACT_GATE
    scripts/config -m CONFIG_NET_IFE_SKBMARK
    scripts/config -m CONFIG_NET_IFE_SKBPRIO
    scripts/config -m CONFIG_NET_IFE_SKBTCINDEX
    scripts/config -m CONFIG_NET_MPLS_GSO
    scripts/config -m CONFIG_NET_NSH
    scripts/config -m CONFIG_NET_PKTGEN
    scripts/config -m CONFIG_NET_DROP_MONITOR
    scripts/config -m CONFIG_NET_9P
    scripts/config -m CONFIG_NET_9P_FD
    scripts/config -m CONFIG_NET_9P_VIRTIO
    scripts/config -m CONFIG_NET_9P_XEN
    scripts/config -m CONFIG_NET_9P_USBG
    scripts/config -m CONFIG_NET_9P_RDMA
    scripts/config -m CONFIG_NET_IFE
    scripts/config -m CONFIG_NET_SELFTESTS
    scripts/config -m CONFIG_NET_TEST
    scripts/config -m CONFIG_NET_TEAM
    scripts/config -m CONFIG_NET_TEAM_MODE_BROADCAST
    scripts/config -m CONFIG_NET_TEAM_MODE_ROUNDROBIN
    scripts/config -m CONFIG_NET_TEAM_MODE_RANDOM
    scripts/config -m CONFIG_NET_TEAM_MODE_ACTIVEBACKUP
    scripts/config -m CONFIG_NET_TEAM_MODE_LOADBALANCE
    scripts/config -m CONFIG_NET_VRF
    scripts/config -m CONFIG_NET_DSA_BCM_SF2
    scripts/config -m CONFIG_NET_DSA_LOOP
    scripts/config -m CONFIG_NET_DSA_HIRSCHMANN_HELLCREEK
    scripts/config -m CONFIG_NET_DSA_LANTIQ_GSWIP
    scripts/config -m CONFIG_NET_DSA_MT7530
    scripts/config -m CONFIG_NET_DSA_MT7530_MDIO
    scripts/config -m CONFIG_NET_DSA_MT7530_MMIO
    scripts/config -m CONFIG_NET_DSA_MV88E6060
    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ_COMMON
    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C
    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ_SPI
    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI
    scripts/config -m CONFIG_NET_DSA_MV88E6XXX
    scripts/config -m CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB
    scripts/config -m CONFIG_NET_DSA_MSCC_OCELOT_EXT
    scripts/config -m CONFIG_NET_DSA_MSCC_FELIX
    scripts/config -m CONFIG_NET_DSA_MSCC_SEVILLE
    scripts/config -m CONFIG_NET_DSA_AR9331
    scripts/config -m CONFIG_NET_DSA_QCA8K
    scripts/config -m CONFIG_NET_DSA_SJA1105
    scripts/config -m CONFIG_NET_DSA_XRS700X
    scripts/config -m CONFIG_NET_DSA_XRS700X_I2C
    scripts/config -m CONFIG_NET_DSA_XRS700X_MDIO
    scripts/config -m CONFIG_NET_DSA_REALTEK
    scripts/config -m CONFIG_NET_DSA_REALTEK_RTL8365MB
    scripts/config -m CONFIG_NET_DSA_REALTEK_RTL8366RB
    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303
    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303_I2C
    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303_MDIO
    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX
    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX_SPI
    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM
    scripts/config -m CONFIG_NET_XGENE
    scripts/config -m CONFIG_NET_XGENE_V2
    scripts/config -m CONFIG_NET_CALXEDA_XGMAC
    scripts/config -m CONFIG_NET_AIROHA
    scripts/config -m CONFIG_NET_MEDIATEK_SOC
    scripts/config -m CONFIG_NET_MEDIATEK_STAR_EMAC
    scripts/config -m CONFIG_NET_FAILOVER
    scripts/config -m CONFIG_NETCONSOLE
    scripts/config -m CONFIG_NETDEV_ADDR_LIST_TEST
    scripts/config -m CONFIG_NETDEV_NOTIFIER_ERROR_INJECT
    scripts/config -m CONFIG_NETDEVSIM
    scripts/config -m CONFIG_NETFILTER_NETLINK
    scripts/config -m CONFIG_NETFILTER_NETLINK_HOOK
    scripts/config -m CONFIG_NETFILTER_NETLINK_ACCT
    scripts/config -m CONFIG_NETFILTER_NETLINK_QUEUE
    scripts/config -m CONFIG_NETFILTER_NETLINK_LOG
    scripts/config -m CONFIG_NETFILTER_NETLINK_OSF
    scripts/config -m CONFIG_NETFILTER_CONNCOUNT
    scripts/config -m CONFIG_NETFILTER_SYNPROXY
    scripts/config -m CONFIG_NETFILTER_XTABLES
    scripts/config -m CONFIG_NETFILTER_XT_MARK
    scripts/config -m CONFIG_NETFILTER_XT_CONNMARK
    scripts/config -m CONFIG_NETFILTER_XT_SET
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_AUDIT
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CHECKSUM
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CLASSIFY
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CONNMARK
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CONNSECMARK
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CT
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_DSCP
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_HL
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_HMARK
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_IDLETIMER
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_LED
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_LOG
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_MARK
    scripts/config -m CONFIG_NETFILTER_XT_NAT
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NETMAP
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NFLOG
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NFQUEUE
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NOTRACK
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_RATEEST
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_REDIRECT
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_MASQUERADE
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TEE
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TPROXY
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TRACE
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_SECMARK
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TCPMSS
    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ADDRTYPE
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_BPF
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CGROUP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CLUSTER
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_COMMENT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNBYTES
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNLABEL
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNLIMIT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNMARK
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNTRACK
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CPU
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DCCP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DEVGROUP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DSCP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ECN
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ESP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HASHLIMIT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HELPER
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HL
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPCOMP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPRANGE
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPVS
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_L2TP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_LENGTH
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_LIMIT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MAC
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MARK
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MULTIPORT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_NFACCT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_OSF
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_OWNER
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_POLICY
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_PHYSDEV
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_PKTTYPE
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_QUOTA
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_RATEEST
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_REALM
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_RECENT
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_SCTP
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_SOCKET
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STATE
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STATISTIC
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STRING
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_TCPMSS
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_TIME
    scripts/config -m CONFIG_NETFILTER_XT_MATCH_U32
    scripts/config -m CONFIG_NETFS_SUPPORT
    scripts/config -m CONFIG_NETLINK_DIAG
    scripts/config -m CONFIG_NETROM
    scripts/config -m CONFIG_NF_CONNTRACK
    scripts/config -m CONFIG_NF_LOG_SYSLOG
    scripts/config -m CONFIG_NF_CONNTRACK_AMANDA
    scripts/config -m CONFIG_NF_CONNTRACK_FTP
    scripts/config -m CONFIG_NF_CONNTRACK_H323
    scripts/config -m CONFIG_NF_CONNTRACK_IRC
    scripts/config -m CONFIG_NF_CONNTRACK_BROADCAST
    scripts/config -m CONFIG_NF_CONNTRACK_NETBIOS_NS
    scripts/config -m CONFIG_NF_CONNTRACK_SNMP
    scripts/config -m CONFIG_NF_CONNTRACK_PPTP
    scripts/config -m CONFIG_NF_CONNTRACK_SANE
    scripts/config -m CONFIG_NF_CONNTRACK_SIP
    scripts/config -m CONFIG_NF_CONNTRACK_TFTP
    scripts/config -m CONFIG_NF_CT_NETLINK
    scripts/config -m CONFIG_NF_CT_NETLINK_TIMEOUT
    scripts/config -m CONFIG_NF_CT_NETLINK_HELPER
    scripts/config -m CONFIG_NF_NAT
    scripts/config -m CONFIG_NF_NAT_AMANDA
    scripts/config -m CONFIG_NF_NAT_FTP
    scripts/config -m CONFIG_NF_NAT_IRC
    scripts/config -m CONFIG_NF_NAT_SIP
    scripts/config -m CONFIG_NF_NAT_TFTP
    scripts/config -m CONFIG_NF_TABLES
    scripts/config -m CONFIG_NF_DUP_NETDEV
    scripts/config -m CONFIG_NF_FLOW_TABLE_INET
    scripts/config -m CONFIG_NF_FLOW_TABLE
    scripts/config -m CONFIG_NF_DEFRAG_IPV4
    scripts/config -m CONFIG_NF_SOCKET_IPV4
    scripts/config -m CONFIG_NF_TPROXY_IPV4
    scripts/config -m CONFIG_NF_DUP_IPV4
    scripts/config -m CONFIG_NF_LOG_ARP
    scripts/config -m CONFIG_NF_LOG_IPV4
    scripts/config -m CONFIG_NF_REJECT_IPV4
    scripts/config -m CONFIG_NF_NAT_SNMP_BASIC
    scripts/config -m CONFIG_NF_NAT_PPTP
    scripts/config -m CONFIG_NF_NAT_H323
    scripts/config -m CONFIG_NF_SOCKET_IPV6
    scripts/config -m CONFIG_NF_TPROXY_IPV6
    scripts/config -m CONFIG_NF_DUP_IPV6
    scripts/config -m CONFIG_NF_REJECT_IPV6
    scripts/config -m CONFIG_NF_LOG_IPV6
    scripts/config -m CONFIG_NF_DEFRAG_IPV6
    scripts/config -m CONFIG_NF_TABLES_BRIDGE
    scripts/config -m CONFIG_NF_CONNTRACK_BRIDGE
    scripts/config -m CONFIG_INET_AH
    scripts/config -m CONFIG_INET_ESP
    scripts/config -m CONFIG_INET_ESP_OFFLOAD
    scripts/config -m CONFIG_INET_IPCOMP
    scripts/config -m CONFIG_INET_XFRM_TUNNEL
    scripts/config -m CONFIG_INET_TUNNEL
    scripts/config -m CONFIG_INET_DIAG
    scripts/config -m CONFIG_INET_TCP_DIAG
    scripts/config -m CONFIG_INET_UDP_DIAG
    scripts/config -m CONFIG_INET_RAW_DIAG
    scripts/config -m CONFIG_INET_MPTCP_DIAG
    scripts/config -m CONFIG_INET_DCCP_DIAG
    scripts/config -m CONFIG_INET_SCTP_DIAG
    scripts/config -m CONFIG_INET6_AH
    scripts/config -m CONFIG_INET6_ESP
    scripts/config -m CONFIG_INET6_ESP_OFFLOAD
    scripts/config -m CONFIG_INET6_IPCOMP
    scripts/config -m CONFIG_INET6_XFRM_TUNNEL
    scripts/config -m CONFIG_INET6_TUNNEL
    scripts/config -m CONFIG_IPVLAN
    scripts/config -m CONFIG_IPVTAP
    scripts/config -d CONFIG_VLAN_8021Q
    scripts/config -m CONFIG_VLAN_8021Q
    scripts/config -d CONFIG_VLAN_8021Q_GVRP
    scripts/config -e CONFIG_VLAN_8021Q_GVRP
    scripts/config -d CONFIG_VLAN_8021Q_MVRP
    scripts/config -e CONFIG_VLAN_8021Q_MVRP
    scripts/config -d CONFIG_VETH
    scripts/config -m CONFIG_VETH
    scripts/config -d CONFIG_TUN
    scripts/config -m CONFIG_TUN
    scripts/config -d CONFIG_TUN_VNET_CROSS_LE
    scripts/config -e CONFIG_TUN_VNET_CROSS_LE
    scripts/config -d CONFIG_TAP
    scripts/config -m CONFIG_TAP
    scripts/config -d CONFIG_XFRM
    scripts/config -e CONFIG_XFRM
    scripts/config -d CONFIG_XFRM_OFFLOAD
    scripts/config -e CONFIG_XFRM_OFFLOAD
    scripts/config -d CONFIG_XFRM_ALGO
    scripts/config -m CONFIG_XFRM_ALGO
    scripts/config -d CONFIG_XFRM_USER
    scripts/config -m CONFIG_XFRM_USER
    scripts/config -d CONFIG_XFRM_USER_COMPAT
    scripts/config -m CONFIG_XFRM_USER_COMPAT
    scripts/config -d CONFIG_XFRM_INTERFACE
    scripts/config -m CONFIG_XFRM_INTERFACE
    scripts/config -d CONFIG_XFRM_SUB_POLICY
    scripts/config -e CONFIG_XFRM_SUB_POLICY
    scripts/config -d CONFIG_XFRM_MIGRATE
    scripts/config -e CONFIG_XFRM_MIGRATE
    scripts/config -d CONFIG_XFRM_STATISTICS
    scripts/config -e CONFIG_XFRM_STATISTICS
    scripts/config -d CONFIG_XFRM_AH
    scripts/config -m CONFIG_XFRM_AH
    scripts/config -d CONFIG_XFRM_ESP
    scripts/config -m CONFIG_XFRM_ESP
    scripts/config -d CONFIG_XFRM_IPCOMP
    scripts/config -m CONFIG_XFRM_IPCOMP
    scripts/config -d CONFIG_XFRM_IPTFS
    scripts/config -m CONFIG_XFRM_IPTFS
    scripts/config -d CONFIG_XFRM_ESPINTCP
    scripts/config -e CONFIG_XFRM_ESPINTCP
    scripts/config -d CONFIG_MPTCP
    scripts/config -e CONFIG_MPTCP
    scripts/config -d CONFIG_MPTCP_KUNIT_TEST
    scripts/config -m CONFIG_MPTCP_KUNIT_TEST
    scripts/config -d CONFIG_PPTP
    scripts/config -m CONFIG_PPTP
    scripts/config -d CONFIG_PPPOE
    scripts/config -m CONFIG_PPPOE
    scripts/config -d CONFIG_PPPOE_HASH_BITS_4
    scripts/config -e CONFIG_PPPOE_HASH_BITS_4
    scripts/config -d CONFIG_PPPOE_HASH_BITS
    scripts/config --set-val CONFIG_PPPOE_HASH_BITS 4
    scripts/config -d CONFIG_PPP
    scripts/config -m CONFIG_PPP
    scripts/config -d CONFIG_PPP_BSDCOMP
    scripts/config -m CONFIG_PPP_BSDCOMP
    scripts/config -d CONFIG_PPP_DEFLATE
    scripts/config -m CONFIG_PPP_DEFLATE
    scripts/config -d CONFIG_PPP_FILTER
    scripts/config -e CONFIG_PPP_FILTER
    scripts/config -d CONFIG_PPP_MPPE
    scripts/config -m CONFIG_PPP_MPPE
    scripts/config -d CONFIG_PPP_MULTILINK
    scripts/config -e CONFIG_PPP_MULTILINK
    scripts/config -d CONFIG_PPP_ASYNC
    scripts/config -m CONFIG_PPP_ASYNC
    scripts/config -d CONFIG_PPP_SYNC_TTY
    scripts/config -m CONFIG_PPP_SYNC_TTY
    scripts/config -d CONFIG_GENEVE
    scripts/config -m CONFIG_GENEVE
    scripts/config -d CONFIG_L2TP
    scripts/config -m CONFIG_L2TP
    scripts/config -d CONFIG_L2TP_DEBUGFS
    scripts/config -m CONFIG_L2TP_DEBUGFS
    scripts/config -d CONFIG_L2TP_V3
    scripts/config -e CONFIG_L2TP_V3
    scripts/config -d CONFIG_L2TP_IP
    scripts/config -m CONFIG_L2TP_IP
    scripts/config -d CONFIG_L2TP_ETH
    scripts/config -m CONFIG_L2TP_ETH
    scripts/config -d CONFIG_VXLAN
    scripts/config -m CONFIG_VXLAN
    make mod2yesconfig

    make -j$(nproc) bzImage
    make -j$(nproc) modules
fi

make modules_install
make headers_install

mkdir -p /boot/efi/efi/boot
cp arch/x86_64/boot/bzImage /boot/efi/efi/boot/vmlinuz

# Create startup.nsh for EFI boot
cat << 'EOFNSH' > /boot/efi/efi/boot/startup.nsh
fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs
EOFNSH
chmod +x /boot/efi/efi/boot/startup.nsh

# Basic system configuration
echo "${VM_NAME}" > /etc/hostname
ln -s /etc/sv/agetty-ttyS0 /etc/runit/runsvdir/default

# Install sudo
xbps-install -y sudo

# Create a user with sudo permissions
useradd -m -G wheel -s /bin/bash admin
usermod -U admin
passwd -d admin

# Add the user to sudoers with ALL/ALL privileges
echo "admin ALL=(ALL) ALL" > /etc/sudoers.d/admin
chmod 440 /etc/sudoers.d/admin

EOF

# Make the script executable
chmod +x /mnt/void-install/setup.sh || { echo "Failed to make setup script executable"; exit 1; }

# Check if Linux source exists in /mnt/pub, if not clone it
if [ ! -d "/mnt/pub/linux" ]; then
    mkdir -p /mnt/pub || { echo "Failed to create /mnt/pub directory"; exit 1; }
    git clone --single-branch --branch v6.14 https://github.com/torvalds/linux.git /mnt/pub/linux || { echo "Failed to clone Linux source"; exit 1; }
fi

# Mount Linux source to VM using nullfs instead of copying
mkdir -p /mnt/void-install/usr/src/linux || { echo "Failed to create Linux source mount point"; exit 1; }
mount_nullfs /mnt/pub/linux /mnt/void-install/usr/src/linux || { echo "Failed to mount Linux source"; exit 1; }

# Run the script inside chroot and ensure we exit properly
cd /vm || { echo "Failed to change to /vm directory"; exit 1; }
chroot /mnt/void-install /setup.sh || { echo "Failed to run setup script in chroot"; exit 1; }

echo "Installation complete for VM '${VM_NAME}' with template '${VM_TEMPLATE}'."
echo "Start the VM with: vm start -f ${VM_NAME}"

void.conf

This is the BHyve-vm template file, you can file the reference for it here: https://github.com/churchers/vm-bhyve/blob/master/sample-templates/config.sample

# /vm/.templates/void.conf

loader="uefi"
cpu="1"
cpu_sockets="1"
cpu_cores="1"
cpu_threads="1"
memory=512M
disk0_type="virtio-blk"
disk0_dev="zvol"
disk0_name="disk0"
disk0_size="8G"
loader="bhyveload"
comports="com1"
graphics="no"
xhci_mouse="no"
wired_memory="yes"
network0_type="virtio-net"
network0_switch="bridge0"
network0_span="yes"

rc.conf

# Enable routing
gateway_enable="YES"
ipv6_gateway_enable="YES"
# Physical interface
ifconfig_ix1="inet 192.168.1.128 netmask 255.255.255.0"

# Virtual interfaces for VM networking
cloned_interfaces="bridge0 epair0 tap0"
ifconfig_bridge0="fib 8 up"
ifconfig_bridge0_aliases="addm epair0b addm tap0"
ifconfig_epair0a="192.0.0.0/31 fib 0 up"
ifconfig_epair0b="192.0.0.1/31 fib 8 up"
ifconfig_tap0="198.18.0.1/23 fib 8 up"

# Static routes (combined)
static_routes="net1 net2 default_fib8 null_private_fib8"
route_net1="-net 192.168.1.0/24 192.168.1.1"
route_net2="default 192.168.1.1"
route_default_fib8="-fib 8 default 192.0.0.0"
route_null_private_fib8="-fib 8 -net 192.168.0.0/16 -reject"
route_fib0_to_fib8="-net 198.18.0.0/23 192.0.0.1"

pf.conf

# 192.168.0.0/16 is null-routed on FIB 8, but this should prevent NAT traversal too while NAT is enabled
nat on ix1 inet from 198.18.0.0/23 to !192.168.0.0/16 -> ix1

Review

Interfaces

ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 98:b7:85:1e:de:4d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 98:b7:85:1e:de:4e
        inet 192.168.1.128 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 3c:ec:ef:56:eb:5c
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
        ether 3c:ec:ef:56:eb:5d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:9b:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        groups: bridge
        fib: 8
        nd6 options=9<PERFORMNUD,IFDISABLED>
epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:77:28:21:0a
        inet 192.0.0.0 netmask 0xfffffffe broadcast 255.255.255.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:40:77:28:21:0b
        inet 192.0.0.1 netmask 0xfffffffe broadcast 255.255.255.255
        groups: epair
        fib: 8
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tap0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vmnet/SWARM1/0/bridge0
        options=80000<LINKSTATE>
        ether 58:9c:fc:00:1d:5d
        inet 198.18.0.1 netmask 0xfffffe00 broadcast 198.18.1.255
        groups: tap vm-port
        fib: 8
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 8159

Routes

➜  /etc netstat -r
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            unifi.localdomain  UGS             ix1
localhost          link#5             UH              lo0
192.0.0.0          link#5             UHS             lo0
192.0.0.0/31       link#7             U           epair0a
192.168.1.0/24     link#2             U               ix1
192.168.1.128      link#5             UHS             lo0
198.18.0.0/23      192.0.0.1          UGS         epair0a

Internet6:
Destination        Gateway            Flags         Netif Expire
::/96              link#5             URS             lo0
localhost          link#5             UHS             lo0
::ffff:0.0.0.0/96  link#5             URS             lo0
fe80::%lo0/10      link#5             URS             lo0
fe80::%lo0/64      link#5             U               lo0
fe80::1%lo0        link#5             UHS             lo0
ff02::/16          link#5             URS             lo0
➜  /etc netstat -r -F 8
Routing tables (fib: 8)

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.0.0.0          UGS         epair0b
localhost          link#5             UHS             lo0
192.0.0.0/31       link#8             U           epair0b
192.0.0.1          link#5             UHS             lo0
192.168.0.0/16     link#5             URS             lo0
198.18.0.0/23      link#9             U              tap0
198.18.0.1         link#5             UHS             lo0

Internet6:
Destination        Gateway            Flags         Netif Expire
::/96              link#5             URS             lo0
localhost          link#5             UHS             lo0
::ffff:0.0.0.0/96  link#5             URS             lo0
fe80::%lo0/10      link#5             URS             lo0
ff02::/16          link#5             URS             lo0
➜  /etc

Guest

➜  /etc vm list        
NAME    DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
SWARM1  default    uefi    4    3128M   -    No    Running (8159)
➜  /etc 

bash-5.2# uname -a
Linux SWARM1 6.14.0 #1 SMP PREEMPT_DYNAMIC Tue Apr 15 22:20:22 UTC 2025 x86_64 GNU/Linux
bash-5.2# 

bash-5.2# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 58:9c:fc:0d:4a:eb brd ff:ff:ff:ff:ff:ff
    inet 198.18.0.2/23 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::5a9c:fcff:fe0d:4aeb/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

bash-5.2# ping 192.0.0.0
PING 192.0.0.0 (192.0.0.0) 56(84) bytes of data.
64 bytes from 192.0.0.0: icmp_seq=1 ttl=64 time=0.171 ms

--- 192.0.0.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 0.171/0.171/0.171/0.000 ms

bash-5.2# ping 4.2.2.2
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
64 bytes from 4.2.2.2: icmp_seq=1 ttl=55 time=29.7 ms

--- 4.2.2.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 29.661/29.661/29.661/0.000 ms

bash-5.2# traceroute 192.168.1.1
traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets
 1  198.18.0.1 (198.18.0.1)  0.223 ms  0.150 ms  0.171 ms
 2  198.18.0.1 (198.18.0.1)  0.091 ms !H  0.130 ms !H  0.135 ms !H
bash-5.2#

Boot

After giving up on net boot, it drops to UEFI shell, which searches for startup.nsh:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (BHYVE, 0x00010000)
Mapping table
      FS0: Alias(s):HD0b:;BLK1:
          PciRoot(0x0)/Pci(0x4,0x0)/HD(1,GPT,26B76E50-1900-11F0-A1CB-98B7851EDE4D,0x40,0x80000)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x4,0x0)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x4,0x0)/HD(2,GPT,26BA3334-1900-11F0-A1CB-98B7851EDE4D,0x80040,0xF7FF80)


Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs
[    0.000000] Linux version 6.14.0 (root@stelleri.netcrave.network) (gcc (GCC) 13.2.0, GNU ld (GNU Binutils) 2.41) #1 SMP PREEMPT_DYNAMIC Mon Apr 14 06:08:58 UTC 2025
[    0.000000] Command line: fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs

Takeaways

I think that's it:

The configuration persists across reboots,
it's reproducible and if I could just get the rest of the house behind a VRF similar to how bridge0 is setup on FIB 8:

(west) <home network>---VGW---<core router (north to internet)>---HV---<FIB 8> (east)

and there is a role in here for VLAN as well, but it would make more sense as VGW and HV to exist on separate VLAN, and really I don't have any other layer 2 needs in mind. Ideally:

(west) <home network>---VGW1---<core router (north to internet)>---VGW2---HV---<FIB 8> (east)

The TLDR is VRF is just better and VLANs are (still useful.) The responsibility of null routing 192.168.0.0/16 would go to a VGW2 that is part of the core router rather than being the responsibility of HV because separating these concerns would ensure that in the case that a BHyve guest is no longer a reliably secure sandbox then the home network would still remain unreachable unless the core router is compromised.

There's nothing about this that says you can't use VLANs galore either. I think I may have made this work similarly with Libvirt by offloading the networking configuration to system-networkd once but I don't remember it being something I was particularly proud of. That's not to say that this necessarily is, that remains to be seen. To say the least, I just like it and it because the whole setup just makes a lot of sense to me without being too much or too little. I'm pretty happy with how it seems to have turned out though.

Need help ?

Feel free to shoot me an e-mail!