TODO

  • UFS2 write support experimental in Linux mainline in 2025?
  • This script works for not but it is just dying to die… I really just wanted to make a point that there is a lot that can be trivialized in this process and relying cd images to install anything in 2025 is just out of the question for me. I don’t even want to do it for physical hardware and if I just got my shit together a little better, I don’t think I would have an excuse given things like iPXE.

Take aways

I just can’t really believe how little a UDM pro actually delivers in terms of being able to do simple things like a VRF. Well afaik UDM is Linux and in my experience VRF on Linux is more of an insult to injury compared to Free/OpenBSD’s fib/rtable. I guess NetNS does it’s job, well enough that people can use it for Docker and somewhere in the world I think somebody even uses VRF in the context of Docker (custom OCI network driver.) It’s still a nightmare for the things I want to do with it and it’s actually a huge part of why I’ve come back to BSD after… 20 years.

Usage

./script.sh BIGVM -t littletemplate and template.conf go in /vm/.templates

Configuration

A script

I made this script with Cursor, it’s really meant to be a proof of concept but it will probably end up just being my goto for awhile. It does a lot, but it uses BHyve-vm to minimize some of the work, I am keeping this script in /vm/create_vm.sh for now. To summarize what this does though:

  • Fetches RootFS for Void Linux if it doesn’t exist in /mnt/pub
  • Clones the v6.14 branch of the Linux kernel to /mnt/pub (if it hasn’t been yet)
  • Creates the VM
  • Sets the associated zvol mode to geom
  • Creates an EFI partition table and EFI filesystem
  • Creates a UFS filesystem FreeBSD doesn’t really have a NewFS for much else, I’ll get to how this works in a moment. ZFS in ZFS didn’t seem like a very good idea, though and I couldn’t figure out how to probably because you can’t. ZFS has been a bit of a learning curve for me but I’m starting to really like it.
  • Extracts the RootFS
  • Mounts /dev /proc and /sys
  • Loop mounts the Linux kernel source from /mnt/pub to the chroot
  • Does a chroot into a linux chroot because Linux compatibility layer (I love that)
  • Builds the kernel if it’s not already built
  • Creates a startup.nsh because we’re really pushing our luck here already okay, you’re not going to get GRUB and efibootmgr to play nice in a chroot under FreeBSD’s Linux compatibility layer. If I’m being honest with you I could care less; the Linux kernel is capable of executing as an EFI executable itself (EFI STUB) and honestly I’m kinda mad at myself for not realizing that I could do this with startup.nsh sooner. If you are fortunate enough to have EFIShell on your system, then you don’t even need a bootloader, you can just do it this way!
  • Sets a user and some boring stuff, unmounts everything and cleans up:
  1#!/usr/bin/env bash
  2
  3set -e
  4
  5# Function to handle cleanup on exit
  6cleanup() {
  7    local exit_code=$?
  8    echo "Cleaning up..."
  9    
 10    # First try to exit chroot if we're in it
 11    if [ "$(pwd)" = "/mnt/void-install" ]; then
 12        cd /vm || cd /
 13    fi
 14    
 15    # Wait a moment for any processes to finish
 16    sleep 1
 17    
 18    # Unmount in reverse order of mounting, with retries
 19    for mount_point in "/mnt/void-install/usr/src/linux" \
 20                      "/mnt/void-install/proc" \
 21                      "/mnt/void-install/sys" \
 22                      "/mnt/void-install/dev" \
 23                      "/mnt/void-install/boot/efi" \
 24                      "/mnt/void-install"; do
 25        if mount | grep -q "$mount_point"; then
 26            echo "Unmounting $mount_point..."
 27            for i in {1..3}; do
 28                if umount "$mount_point" 2>/dev/null; then
 29                    break
 30                fi
 31                echo "Retry $i: Waiting for $mount_point to be unmountable..."
 32                sleep 2
 33            done
 34        fi
 35    done
 36    
 37    # If we failed and VM exists, destroy it
 38    if [ $exit_code -ne 0 ] && [ -n "$VM_NAME" ]; then
 39        echo "Installation failed for VM '${VM_NAME}' with template '${VM_TEMPLATE}', destroying VM..."
 40        vm destroy -f "$VM_NAME" || echo "Warning: Failed to destroy VM ${VM_NAME}"
 41    fi
 42    
 43    exit $exit_code
 44}
 45
 46# Set up trap to call cleanup function on script exit
 47trap cleanup EXIT
 48
 49# Check if nmdm kernel module is loaded, if not load it
 50if ! kldstat -m nmdm > /dev/null 2>&1; then
 51    echo "Loading nmdm kernel module..."
 52    kldload nmdm || { echo "Failed to load nmdm kernel module"; exit 1; }
 53fi
 54
 55# Parse command line arguments
 56VM_TEMPLATE="void" # Default template
 57
 58usage() {
 59    echo "Usage: $0 <vm_name> [options]"
 60    echo "Options:"
 61    echo "  -t, --template <template>   Specify the VM template to use (default: void)"
 62    echo "  -h, --help                  Show this help message"
 63    echo ""
 64    echo "Examples:"
 65    echo "  $0 my-vm                    # Create VM 'my-vm' with default 'void' template"
 66    echo "  $0 my-vm -t freebsd         # Create VM 'my-vm' with 'freebsd' template"
 67    echo "  $0 my-vm --template debian  # Create VM 'my-vm' with 'debian' template"
 68    exit 1
 69}
 70
 71# Check if at least one argument is provided
 72if [ $# -lt 1 ]; then
 73    usage
 74fi
 75
 76VM_NAME="$1"
 77shift
 78
 79# Parse remaining arguments
 80while [ $# -gt 0 ]; do
 81    case "$1" in
 82        -t|--template)
 83            VM_TEMPLATE="$2"
 84            shift 2
 85            ;;
 86        -h|--help)
 87            usage
 88            ;;
 89        *)
 90            echo "Unknown option: $1"
 91            usage
 92            ;;
 93    esac
 94done
 95
 96echo "Creating VM with name: ${VM_NAME}, template: ${VM_TEMPLATE}"
 97
 98# Create the VM
 99vm create -t ${VM_TEMPLATE} ${VM_NAME} || { echo "Failed to create VM"; exit 1; }
100
101# Prepare zvol
102zfs set volmode=geom storage/vm/${VM_NAME}/disk0 || { echo "Failed to set zvol mode"; exit 1; }
103
104# Partition disk
105gpart create -s gpt /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to create GPT partition table"; exit 1; }
106gpart add -t efi -s 256M /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to add EFI partition"; exit 1; }
107gpart add -t freebsd-ufs /dev/zvol/storage/vm/${VM_NAME}/disk0 || { echo "Failed to add UFS partition"; exit 1; }
108
109# Format partitions
110newfs_msdos /dev/zvol/storage/vm/${VM_NAME}/disk0p1 || { echo "Failed to format EFI partition"; exit 1; }
111newfs /dev/zvol/storage/vm/${VM_NAME}/disk0p2 || { echo "Failed to format UFS partition"; exit 1; }
112
113# Create mount points and mount partitions
114mkdir -p /mnt/void-install || { echo "Failed to create mount point"; exit 1; }
115mount /dev/zvol/storage/vm/${VM_NAME}/disk0p2 /mnt/void-install || { echo "Failed to mount root partition"; exit 1; }
116mkdir -p /mnt/void-install/boot/efi || { echo "Failed to create EFI mount point"; exit 1; }
117mount -t msdosfs /dev/zvol/storage/vm/${VM_NAME}/disk0p1 /mnt/void-install/boot/efi || { echo "Failed to mount EFI partition"; exit 1; }
118
119# Check if rootfs archive exists in /mnt/pub, if not download it
120if [ ! -f "/mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz" ]; then
121    mkdir -p /mnt/pub || { echo "Failed to create /mnt/pub directory"; exit 1; }
122    wget -O /mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz https://repo-default.voidlinux.org/live/current/void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to download rootfs"; exit 1; }
123fi
124
125# Download and extract rootfs
126cd /mnt/void-install || { echo "Failed to change to mount directory"; exit 1; }
127
128cp /mnt/pub/void-x86_64-musl-ROOTFS-20250202.tar.xz . || { echo "Failed to copy rootfs archive"; exit 1; }
129bsdtar -x --no-xattrs -f void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to extract rootfs"; exit 1; }
130rm void-x86_64-musl-ROOTFS-20250202.tar.xz || { echo "Failed to remove rootfs archive"; exit 1; }
131
132# Prepare chroot
133mount -t linprocfs linprocfs /mnt/void-install/proc || { echo "Failed to mount proc"; exit 1; }
134mount -t linsysfs linsysfs /mnt/void-install/sys || { echo "Failed to mount sys"; exit 1; }
135mount -t devfs devfs /mnt/void-install/dev || { echo "Failed to mount dev"; exit 1; }
136cp /etc/resolv.conf /mnt/void-install/etc/ || { echo "Failed to copy resolv.conf"; exit 1; }
137echo "135.181.160.58 repo-default.voidlinux.org" >> /mnt/void-install/etc/hosts || { echo "Failed to update hosts file"; exit 1; }
138
139# Create /etc/fstab for the VM
140cat << EOF > /mnt/void-install/etc/fstab || { echo "Failed to create fstab"; exit 1; }
141/dev/vda2 / ufs defaults 0 1
142/dev/vda1 /boot/efi msdos defaults 0 2
143EOF
144
145# Create script to run inside chroot
146cat << EOF > /mnt/void-install/setup.sh || { echo "Failed to create setup script"; exit 1; }
147#!/bin/bash
148
149set -e
150
151# VM template used for creation
152VM_TEMPLATE="${VM_TEMPLATE}"
153VM_NAME="${VM_NAME}"
154
155xbps-pkgdb -m hold linux
156xbps-pkgdb -m hold linux-headers
157xbps-install --force -Suy
158xbps-install --force -y gcc make flex bison libelf rsync elfutils-devel bc openssl-devel perl base-system xz \
159tar xz bc elfutils-devel flex gmp-devel kmod libmpc-devel pkg-config openssl-devel perl uboot-mkimage \
160cpio pahole python3 zstd
161
162# Print VM info
163echo "Setting up VM '${VM_NAME}' created with template '${VM_TEMPLATE}'"
164
165cd /usr/src/linux
166
167if [ ! -f ".config" ]; then
168    make mrproper
169    make clean
170    make defconfig
171    make kvm_guest.config
172
173    scripts/config -e CONFIG_EFI_STUB
174    scripts/config -e CONFIG_HZ_100
175    scripts/config -d CONFIG_HZ_250
176    scripts/config -d CONFIG_HZ_300
177    scripts/config -d CONFIG_HZ_1000
178    scripts/config -d CONFIG_NO_HZ_IDLE
179    scripts/config -d CONFIG_NO_HZ
180    scripts/config -e CONFIG_PREEMPT
181    scripts/config -e CONFIG_BOOT_CONFIG
182    scripts/config -e CONFIG_HZ_PERIODIC
183    scripts/config -e CONFIG_MSDOS_FS
184    scripts/config -e CONFIG_VFAT_FS
185    scripts/config -e CONFIG_EXFAT_FS
186    scripts/config -e CONFIG_UFS_FS
187    scripts/config -e CONFIG_UFS_FS_WRITE
188    scripts/config -e CONFIG_SCSI_VIRTIO
189    scripts/config -e CONFIG_DRM_VIRTIO_GPU
190    scripts/config -e CONFIG_HW_RANDOM_VIRTIO
191    scripts/config -e CONFIG_NFS_FS
192    scripts/config -e CONFIG_NFS_V2
193    scripts/config -e CONFIG_NFS_V3
194    scripts/config -e CONFIG_NFS_V3_ACL
195    scripts/config -e CONFIG_NFS_V4
196    scripts/config -e CONFIG_NFS_SWAP
197    scripts/config -e CONFIG_NFS_V4_1
198    scripts/config -e CONFIG_NFS_V4_2
199    scripts/config -e CONFIG_HAVE_KERNEL_GZIP
200    scripts/config -d CONFIG_HAVE_KERNEL_BZIP2
201    scripts/config -d CONFIG_HAVE_KERNEL_LZMA
202    scripts/config -d CONFIG_HAVE_KERNEL_XZ
203    scripts/config -d CONFIG_HAVE_KERNEL_LZO
204    scripts/config -d CONFIG_HAVE_KERNEL_LZ4
205    scripts/config -d CONFIG_HAVE_KERNEL_ZSTD
206    scripts/config -e CONFIG_KERNEL_GZIP
207    scripts/config -d CONFIG_KERNEL_BZIP2
208    scripts/config -d CONFIG_KERNEL_LZMA
209    scripts/config -d CONFIG_KERNEL_XZ
210    scripts/config -d CONFIG_KERNEL_LZO
211    scripts/config -d CONFIG_KERNEL_LZ4
212    scripts/config -d CONFIG_KERNEL_ZSTD
213    scripts/config -m CONFIG_NET_KEY
214    scripts/config -m CONFIG_NET_HANDSHAKE_KUNIT_TEST
215    scripts/config -m CONFIG_NET_IPIP
216    scripts/config -m CONFIG_NET_IPGRE_DEMUX
217    scripts/config -m CONFIG_NET_IP_TUNNEL
218    scripts/config -m CONFIG_NET_IPGRE
219    scripts/config -m CONFIG_NET_IPVTI
220    scripts/config -m CONFIG_NET_UDP_TUNNEL
221    scripts/config -m CONFIG_NET_FOU
222    scripts/config -m CONFIG_NET_DSA
223    scripts/config -m CONFIG_NET_DSA_TAG_NONE
224    scripts/config -m CONFIG_NET_DSA_TAG_AR9331
225    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_COMMON
226    scripts/config -m CONFIG_NET_DSA_TAG_BRCM
227    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_LEGACY
228    scripts/config -m CONFIG_NET_DSA_TAG_BRCM_PREPEND
229    scripts/config -m CONFIG_NET_DSA_TAG_HELLCREEK
230    scripts/config -m CONFIG_NET_DSA_TAG_GSWIP
231    scripts/config -m CONFIG_NET_DSA_TAG_DSA_COMMON
232    scripts/config -m CONFIG_NET_DSA_TAG_DSA
233    scripts/config -m CONFIG_NET_DSA_TAG_EDSA
234    scripts/config -m CONFIG_NET_DSA_TAG_MTK
235    scripts/config -m CONFIG_NET_DSA_TAG_KSZ
236    scripts/config -m CONFIG_NET_DSA_TAG_OCELOT
237    scripts/config -m CONFIG_NET_DSA_TAG_OCELOT_8021Q
238    scripts/config -m CONFIG_NET_DSA_TAG_QCA
239    scripts/config -m CONFIG_NET_DSA_TAG_RTL4_A
240    scripts/config -m CONFIG_NET_DSA_TAG_RTL8_4
241    scripts/config -m CONFIG_NET_DSA_TAG_RZN1_A5PSW
242    scripts/config -m CONFIG_NET_DSA_TAG_LAN9303
243    scripts/config -m CONFIG_NET_DSA_TAG_SJA1105
244    scripts/config -m CONFIG_NET_DSA_TAG_TRAILER
245    scripts/config -m CONFIG_NET_DSA_TAG_VSC73XX_8021Q
246    scripts/config -m CONFIG_NET_DSA_TAG_XRS700X
247    scripts/config -m CONFIG_NET_SCH_HTB
248    scripts/config -m CONFIG_NET_SCH_HFSC
249    scripts/config -m CONFIG_NET_SCH_PRIO
250    scripts/config -m CONFIG_NET_SCH_MULTIQ
251    scripts/config -m CONFIG_NET_SCH_RED
252    scripts/config -m CONFIG_NET_SCH_SFB
253    scripts/config -m CONFIG_NET_SCH_SFQ
254    scripts/config -m CONFIG_NET_SCH_TEQL
255    scripts/config -m CONFIG_NET_SCH_TBF
256    scripts/config -m CONFIG_NET_SCH_CBS
257    scripts/config -m CONFIG_NET_SCH_ETF
258    scripts/config -m CONFIG_NET_SCH_MQPRIO_LIB
259    scripts/config -m CONFIG_NET_SCH_TAPRIO
260    scripts/config -m CONFIG_NET_SCH_GRED
261    scripts/config -m CONFIG_NET_SCH_NETEM
262    scripts/config -m CONFIG_NET_SCH_DRR
263    scripts/config -m CONFIG_NET_SCH_MQPRIO
264    scripts/config -m CONFIG_NET_SCH_SKBPRIO
265    scripts/config -m CONFIG_NET_SCH_CHOKE
266    scripts/config -m CONFIG_NET_SCH_QFQ
267    scripts/config -m CONFIG_NET_SCH_CODEL
268    scripts/config -m CONFIG_NET_SCH_FQ_CODEL
269    scripts/config -m CONFIG_NET_SCH_CAKE
270    scripts/config -m CONFIG_NET_SCH_FQ
271    scripts/config -m CONFIG_NET_SCH_HHF
272    scripts/config -m CONFIG_NET_SCH_PIE
273    scripts/config -m CONFIG_NET_SCH_FQ_PIE
274    scripts/config -m CONFIG_NET_SCH_INGRESS
275    scripts/config -m CONFIG_NET_SCH_PLUG
276    scripts/config -m CONFIG_NET_SCH_ETS
277    scripts/config -m CONFIG_NET_CLS_BASIC
278    scripts/config -m CONFIG_NET_CLS_ROUTE4
279    scripts/config -m CONFIG_NET_CLS_FW
280    scripts/config -m CONFIG_NET_CLS_U32
281    scripts/config -m CONFIG_NET_CLS_FLOW
282    scripts/config -m CONFIG_NET_CLS_CGROUP
283    scripts/config -m CONFIG_NET_CLS_BPF
284    scripts/config -m CONFIG_NET_CLS_FLOWER
285    scripts/config -m CONFIG_NET_CLS_MATCHALL
286    scripts/config -m CONFIG_NET_EMATCH_CMP
287    scripts/config -m CONFIG_NET_EMATCH_NBYTE
288    scripts/config -m CONFIG_NET_EMATCH_U32
289    scripts/config -m CONFIG_NET_EMATCH_META
290    scripts/config -m CONFIG_NET_EMATCH_TEXT
291    scripts/config -m CONFIG_NET_EMATCH_CANID
292    scripts/config -m CONFIG_NET_EMATCH_IPSET
293    scripts/config -m CONFIG_NET_EMATCH_IPT
294    scripts/config -m CONFIG_NET_ACT_POLICE
295    scripts/config -m CONFIG_NET_ACT_GACT
296    scripts/config -m CONFIG_NET_ACT_MIRRED
297    scripts/config -m CONFIG_NET_ACT_SAMPLE
298    scripts/config -m CONFIG_NET_ACT_NAT
299    scripts/config -m CONFIG_NET_ACT_PEDIT
300    scripts/config -m CONFIG_NET_ACT_SIMP
301    scripts/config -m CONFIG_NET_ACT_SKBEDIT
302    scripts/config -m CONFIG_NET_ACT_CSUM
303    scripts/config -m CONFIG_NET_ACT_MPLS
304    scripts/config -m CONFIG_NET_ACT_VLAN
305    scripts/config -m CONFIG_NET_ACT_BPF
306    scripts/config -m CONFIG_NET_ACT_CONNMARK
307    scripts/config -m CONFIG_NET_ACT_CTINFO
308    scripts/config -m CONFIG_NET_ACT_SKBMOD
309    scripts/config -m CONFIG_NET_ACT_IFE
310    scripts/config -m CONFIG_NET_ACT_TUNNEL_KEY
311    scripts/config -m CONFIG_NET_ACT_CT
312    scripts/config -m CONFIG_NET_ACT_GATE
313    scripts/config -m CONFIG_NET_IFE_SKBMARK
314    scripts/config -m CONFIG_NET_IFE_SKBPRIO
315    scripts/config -m CONFIG_NET_IFE_SKBTCINDEX
316    scripts/config -m CONFIG_NET_MPLS_GSO
317    scripts/config -m CONFIG_NET_NSH
318    scripts/config -m CONFIG_NET_PKTGEN
319    scripts/config -m CONFIG_NET_DROP_MONITOR
320    scripts/config -m CONFIG_NET_9P
321    scripts/config -m CONFIG_NET_9P_FD
322    scripts/config -m CONFIG_NET_9P_VIRTIO
323    scripts/config -m CONFIG_NET_9P_XEN
324    scripts/config -m CONFIG_NET_9P_USBG
325    scripts/config -m CONFIG_NET_9P_RDMA
326    scripts/config -m CONFIG_NET_IFE
327    scripts/config -m CONFIG_NET_SELFTESTS
328    scripts/config -m CONFIG_NET_TEST
329    scripts/config -m CONFIG_NET_TEAM
330    scripts/config -m CONFIG_NET_TEAM_MODE_BROADCAST
331    scripts/config -m CONFIG_NET_TEAM_MODE_ROUNDROBIN
332    scripts/config -m CONFIG_NET_TEAM_MODE_RANDOM
333    scripts/config -m CONFIG_NET_TEAM_MODE_ACTIVEBACKUP
334    scripts/config -m CONFIG_NET_TEAM_MODE_LOADBALANCE
335    scripts/config -m CONFIG_NET_VRF
336    scripts/config -m CONFIG_NET_DSA_BCM_SF2
337    scripts/config -m CONFIG_NET_DSA_LOOP
338    scripts/config -m CONFIG_NET_DSA_HIRSCHMANN_HELLCREEK
339    scripts/config -m CONFIG_NET_DSA_LANTIQ_GSWIP
340    scripts/config -m CONFIG_NET_DSA_MT7530
341    scripts/config -m CONFIG_NET_DSA_MT7530_MDIO
342    scripts/config -m CONFIG_NET_DSA_MT7530_MMIO
343    scripts/config -m CONFIG_NET_DSA_MV88E6060
344    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ_COMMON
345    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ9477_I2C
346    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ_SPI
347    scripts/config -m CONFIG_NET_DSA_MICROCHIP_KSZ8863_SMI
348    scripts/config -m CONFIG_NET_DSA_MV88E6XXX
349    scripts/config -m CONFIG_NET_DSA_MSCC_FELIX_DSA_LIB
350    scripts/config -m CONFIG_NET_DSA_MSCC_OCELOT_EXT
351    scripts/config -m CONFIG_NET_DSA_MSCC_FELIX
352    scripts/config -m CONFIG_NET_DSA_MSCC_SEVILLE
353    scripts/config -m CONFIG_NET_DSA_AR9331
354    scripts/config -m CONFIG_NET_DSA_QCA8K
355    scripts/config -m CONFIG_NET_DSA_SJA1105
356    scripts/config -m CONFIG_NET_DSA_XRS700X
357    scripts/config -m CONFIG_NET_DSA_XRS700X_I2C
358    scripts/config -m CONFIG_NET_DSA_XRS700X_MDIO
359    scripts/config -m CONFIG_NET_DSA_REALTEK
360    scripts/config -m CONFIG_NET_DSA_REALTEK_RTL8365MB
361    scripts/config -m CONFIG_NET_DSA_REALTEK_RTL8366RB
362    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303
363    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303_I2C
364    scripts/config -m CONFIG_NET_DSA_SMSC_LAN9303_MDIO
365    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX
366    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX_SPI
367    scripts/config -m CONFIG_NET_DSA_VITESSE_VSC73XX_PLATFORM
368    scripts/config -m CONFIG_NET_XGENE
369    scripts/config -m CONFIG_NET_XGENE_V2
370    scripts/config -m CONFIG_NET_CALXEDA_XGMAC
371    scripts/config -m CONFIG_NET_AIROHA
372    scripts/config -m CONFIG_NET_MEDIATEK_SOC
373    scripts/config -m CONFIG_NET_MEDIATEK_STAR_EMAC
374    scripts/config -m CONFIG_NET_FAILOVER
375    scripts/config -m CONFIG_NETCONSOLE
376    scripts/config -m CONFIG_NETDEV_ADDR_LIST_TEST
377    scripts/config -m CONFIG_NETDEV_NOTIFIER_ERROR_INJECT
378    scripts/config -m CONFIG_NETDEVSIM
379    scripts/config -m CONFIG_NETFILTER_NETLINK
380    scripts/config -m CONFIG_NETFILTER_NETLINK_HOOK
381    scripts/config -m CONFIG_NETFILTER_NETLINK_ACCT
382    scripts/config -m CONFIG_NETFILTER_NETLINK_QUEUE
383    scripts/config -m CONFIG_NETFILTER_NETLINK_LOG
384    scripts/config -m CONFIG_NETFILTER_NETLINK_OSF
385    scripts/config -m CONFIG_NETFILTER_CONNCOUNT
386    scripts/config -m CONFIG_NETFILTER_SYNPROXY
387    scripts/config -m CONFIG_NETFILTER_XTABLES
388    scripts/config -m CONFIG_NETFILTER_XT_MARK
389    scripts/config -m CONFIG_NETFILTER_XT_CONNMARK
390    scripts/config -m CONFIG_NETFILTER_XT_SET
391    scripts/config -m CONFIG_NETFILTER_XT_TARGET_AUDIT
392    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CHECKSUM
393    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CLASSIFY
394    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CONNMARK
395    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CONNSECMARK
396    scripts/config -m CONFIG_NETFILTER_XT_TARGET_CT
397    scripts/config -m CONFIG_NETFILTER_XT_TARGET_DSCP
398    scripts/config -m CONFIG_NETFILTER_XT_TARGET_HL
399    scripts/config -m CONFIG_NETFILTER_XT_TARGET_HMARK
400    scripts/config -m CONFIG_NETFILTER_XT_TARGET_IDLETIMER
401    scripts/config -m CONFIG_NETFILTER_XT_TARGET_LED
402    scripts/config -m CONFIG_NETFILTER_XT_TARGET_LOG
403    scripts/config -m CONFIG_NETFILTER_XT_TARGET_MARK
404    scripts/config -m CONFIG_NETFILTER_XT_NAT
405    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NETMAP
406    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NFLOG
407    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NFQUEUE
408    scripts/config -m CONFIG_NETFILTER_XT_TARGET_NOTRACK
409    scripts/config -m CONFIG_NETFILTER_XT_TARGET_RATEEST
410    scripts/config -m CONFIG_NETFILTER_XT_TARGET_REDIRECT
411    scripts/config -m CONFIG_NETFILTER_XT_TARGET_MASQUERADE
412    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TEE
413    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TPROXY
414    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TRACE
415    scripts/config -m CONFIG_NETFILTER_XT_TARGET_SECMARK
416    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TCPMSS
417    scripts/config -m CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP
418    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ADDRTYPE
419    scripts/config -m CONFIG_NETFILTER_XT_MATCH_BPF
420    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CGROUP
421    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CLUSTER
422    scripts/config -m CONFIG_NETFILTER_XT_MATCH_COMMENT
423    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNBYTES
424    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNLABEL
425    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNLIMIT
426    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNMARK
427    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CONNTRACK
428    scripts/config -m CONFIG_NETFILTER_XT_MATCH_CPU
429    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DCCP
430    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DEVGROUP
431    scripts/config -m CONFIG_NETFILTER_XT_MATCH_DSCP
432    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ECN
433    scripts/config -m CONFIG_NETFILTER_XT_MATCH_ESP
434    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HASHLIMIT
435    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HELPER
436    scripts/config -m CONFIG_NETFILTER_XT_MATCH_HL
437    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPCOMP
438    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPRANGE
439    scripts/config -m CONFIG_NETFILTER_XT_MATCH_IPVS
440    scripts/config -m CONFIG_NETFILTER_XT_MATCH_L2TP
441    scripts/config -m CONFIG_NETFILTER_XT_MATCH_LENGTH
442    scripts/config -m CONFIG_NETFILTER_XT_MATCH_LIMIT
443    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MAC
444    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MARK
445    scripts/config -m CONFIG_NETFILTER_XT_MATCH_MULTIPORT
446    scripts/config -m CONFIG_NETFILTER_XT_MATCH_NFACCT
447    scripts/config -m CONFIG_NETFILTER_XT_MATCH_OSF
448    scripts/config -m CONFIG_NETFILTER_XT_MATCH_OWNER
449    scripts/config -m CONFIG_NETFILTER_XT_MATCH_POLICY
450    scripts/config -m CONFIG_NETFILTER_XT_MATCH_PHYSDEV
451    scripts/config -m CONFIG_NETFILTER_XT_MATCH_PKTTYPE
452    scripts/config -m CONFIG_NETFILTER_XT_MATCH_QUOTA
453    scripts/config -m CONFIG_NETFILTER_XT_MATCH_RATEEST
454    scripts/config -m CONFIG_NETFILTER_XT_MATCH_REALM
455    scripts/config -m CONFIG_NETFILTER_XT_MATCH_RECENT
456    scripts/config -m CONFIG_NETFILTER_XT_MATCH_SCTP
457    scripts/config -m CONFIG_NETFILTER_XT_MATCH_SOCKET
458    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STATE
459    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STATISTIC
460    scripts/config -m CONFIG_NETFILTER_XT_MATCH_STRING
461    scripts/config -m CONFIG_NETFILTER_XT_MATCH_TCPMSS
462    scripts/config -m CONFIG_NETFILTER_XT_MATCH_TIME
463    scripts/config -m CONFIG_NETFILTER_XT_MATCH_U32
464    scripts/config -m CONFIG_NETFS_SUPPORT
465    scripts/config -m CONFIG_NETLINK_DIAG
466    scripts/config -m CONFIG_NETROM
467    scripts/config -m CONFIG_NF_CONNTRACK
468    scripts/config -m CONFIG_NF_LOG_SYSLOG
469    scripts/config -m CONFIG_NF_CONNTRACK_AMANDA
470    scripts/config -m CONFIG_NF_CONNTRACK_FTP
471    scripts/config -m CONFIG_NF_CONNTRACK_H323
472    scripts/config -m CONFIG_NF_CONNTRACK_IRC
473    scripts/config -m CONFIG_NF_CONNTRACK_BROADCAST
474    scripts/config -m CONFIG_NF_CONNTRACK_NETBIOS_NS
475    scripts/config -m CONFIG_NF_CONNTRACK_SNMP
476    scripts/config -m CONFIG_NF_CONNTRACK_PPTP
477    scripts/config -m CONFIG_NF_CONNTRACK_SANE
478    scripts/config -m CONFIG_NF_CONNTRACK_SIP
479    scripts/config -m CONFIG_NF_CONNTRACK_TFTP
480    scripts/config -m CONFIG_NF_CT_NETLINK
481    scripts/config -m CONFIG_NF_CT_NETLINK_TIMEOUT
482    scripts/config -m CONFIG_NF_CT_NETLINK_HELPER
483    scripts/config -m CONFIG_NF_NAT
484    scripts/config -m CONFIG_NF_NAT_AMANDA
485    scripts/config -m CONFIG_NF_NAT_FTP
486    scripts/config -m CONFIG_NF_NAT_IRC
487    scripts/config -m CONFIG_NF_NAT_SIP
488    scripts/config -m CONFIG_NF_NAT_TFTP
489    scripts/config -m CONFIG_NF_TABLES
490    scripts/config -m CONFIG_NF_DUP_NETDEV
491    scripts/config -m CONFIG_NF_FLOW_TABLE_INET
492    scripts/config -m CONFIG_NF_FLOW_TABLE
493    scripts/config -m CONFIG_NF_DEFRAG_IPV4
494    scripts/config -m CONFIG_NF_SOCKET_IPV4
495    scripts/config -m CONFIG_NF_TPROXY_IPV4
496    scripts/config -m CONFIG_NF_DUP_IPV4
497    scripts/config -m CONFIG_NF_LOG_ARP
498    scripts/config -m CONFIG_NF_LOG_IPV4
499    scripts/config -m CONFIG_NF_REJECT_IPV4
500    scripts/config -m CONFIG_NF_NAT_SNMP_BASIC
501    scripts/config -m CONFIG_NF_NAT_PPTP
502    scripts/config -m CONFIG_NF_NAT_H323
503    scripts/config -m CONFIG_NF_SOCKET_IPV6
504    scripts/config -m CONFIG_NF_TPROXY_IPV6
505    scripts/config -m CONFIG_NF_DUP_IPV6
506    scripts/config -m CONFIG_NF_REJECT_IPV6
507    scripts/config -m CONFIG_NF_LOG_IPV6
508    scripts/config -m CONFIG_NF_DEFRAG_IPV6
509    scripts/config -m CONFIG_NF_TABLES_BRIDGE
510    scripts/config -m CONFIG_NF_CONNTRACK_BRIDGE
511    scripts/config -m CONFIG_INET_AH
512    scripts/config -m CONFIG_INET_ESP
513    scripts/config -m CONFIG_INET_ESP_OFFLOAD
514    scripts/config -m CONFIG_INET_IPCOMP
515    scripts/config -m CONFIG_INET_XFRM_TUNNEL
516    scripts/config -m CONFIG_INET_TUNNEL
517    scripts/config -m CONFIG_INET_DIAG
518    scripts/config -m CONFIG_INET_TCP_DIAG
519    scripts/config -m CONFIG_INET_UDP_DIAG
520    scripts/config -m CONFIG_INET_RAW_DIAG
521    scripts/config -m CONFIG_INET_MPTCP_DIAG
522    scripts/config -m CONFIG_INET_DCCP_DIAG
523    scripts/config -m CONFIG_INET_SCTP_DIAG
524    scripts/config -m CONFIG_INET6_AH
525    scripts/config -m CONFIG_INET6_ESP
526    scripts/config -m CONFIG_INET6_ESP_OFFLOAD
527    scripts/config -m CONFIG_INET6_IPCOMP
528    scripts/config -m CONFIG_INET6_XFRM_TUNNEL
529    scripts/config -m CONFIG_INET6_TUNNEL
530    scripts/config -m CONFIG_IPVLAN
531    scripts/config -m CONFIG_IPVTAP
532    scripts/config -d CONFIG_VLAN_8021Q
533    scripts/config -m CONFIG_VLAN_8021Q
534    scripts/config -d CONFIG_VLAN_8021Q_GVRP
535    scripts/config -e CONFIG_VLAN_8021Q_GVRP
536    scripts/config -d CONFIG_VLAN_8021Q_MVRP
537    scripts/config -e CONFIG_VLAN_8021Q_MVRP
538    scripts/config -d CONFIG_VETH
539    scripts/config -m CONFIG_VETH
540    scripts/config -d CONFIG_TUN
541    scripts/config -m CONFIG_TUN
542    scripts/config -d CONFIG_TUN_VNET_CROSS_LE
543    scripts/config -e CONFIG_TUN_VNET_CROSS_LE
544    scripts/config -d CONFIG_TAP
545    scripts/config -m CONFIG_TAP
546    scripts/config -d CONFIG_XFRM
547    scripts/config -e CONFIG_XFRM
548    scripts/config -d CONFIG_XFRM_OFFLOAD
549    scripts/config -e CONFIG_XFRM_OFFLOAD
550    scripts/config -d CONFIG_XFRM_ALGO
551    scripts/config -m CONFIG_XFRM_ALGO
552    scripts/config -d CONFIG_XFRM_USER
553    scripts/config -m CONFIG_XFRM_USER
554    scripts/config -d CONFIG_XFRM_USER_COMPAT
555    scripts/config -m CONFIG_XFRM_USER_COMPAT
556    scripts/config -d CONFIG_XFRM_INTERFACE
557    scripts/config -m CONFIG_XFRM_INTERFACE
558    scripts/config -d CONFIG_XFRM_SUB_POLICY
559    scripts/config -e CONFIG_XFRM_SUB_POLICY
560    scripts/config -d CONFIG_XFRM_MIGRATE
561    scripts/config -e CONFIG_XFRM_MIGRATE
562    scripts/config -d CONFIG_XFRM_STATISTICS
563    scripts/config -e CONFIG_XFRM_STATISTICS
564    scripts/config -d CONFIG_XFRM_AH
565    scripts/config -m CONFIG_XFRM_AH
566    scripts/config -d CONFIG_XFRM_ESP
567    scripts/config -m CONFIG_XFRM_ESP
568    scripts/config -d CONFIG_XFRM_IPCOMP
569    scripts/config -m CONFIG_XFRM_IPCOMP
570    scripts/config -d CONFIG_XFRM_IPTFS
571    scripts/config -m CONFIG_XFRM_IPTFS
572    scripts/config -d CONFIG_XFRM_ESPINTCP
573    scripts/config -e CONFIG_XFRM_ESPINTCP
574    scripts/config -d CONFIG_MPTCP
575    scripts/config -e CONFIG_MPTCP
576    scripts/config -d CONFIG_MPTCP_KUNIT_TEST
577    scripts/config -m CONFIG_MPTCP_KUNIT_TEST
578    scripts/config -d CONFIG_PPTP
579    scripts/config -m CONFIG_PPTP
580    scripts/config -d CONFIG_PPPOE
581    scripts/config -m CONFIG_PPPOE
582    scripts/config -d CONFIG_PPPOE_HASH_BITS_4
583    scripts/config -e CONFIG_PPPOE_HASH_BITS_4
584    scripts/config -d CONFIG_PPPOE_HASH_BITS
585    scripts/config --set-val CONFIG_PPPOE_HASH_BITS 4
586    scripts/config -d CONFIG_PPP
587    scripts/config -m CONFIG_PPP
588    scripts/config -d CONFIG_PPP_BSDCOMP
589    scripts/config -m CONFIG_PPP_BSDCOMP
590    scripts/config -d CONFIG_PPP_DEFLATE
591    scripts/config -m CONFIG_PPP_DEFLATE
592    scripts/config -d CONFIG_PPP_FILTER
593    scripts/config -e CONFIG_PPP_FILTER
594    scripts/config -d CONFIG_PPP_MPPE
595    scripts/config -m CONFIG_PPP_MPPE
596    scripts/config -d CONFIG_PPP_MULTILINK
597    scripts/config -e CONFIG_PPP_MULTILINK
598    scripts/config -d CONFIG_PPP_ASYNC
599    scripts/config -m CONFIG_PPP_ASYNC
600    scripts/config -d CONFIG_PPP_SYNC_TTY
601    scripts/config -m CONFIG_PPP_SYNC_TTY
602    scripts/config -d CONFIG_GENEVE
603    scripts/config -m CONFIG_GENEVE
604    scripts/config -d CONFIG_L2TP
605    scripts/config -m CONFIG_L2TP
606    scripts/config -d CONFIG_L2TP_DEBUGFS
607    scripts/config -m CONFIG_L2TP_DEBUGFS
608    scripts/config -d CONFIG_L2TP_V3
609    scripts/config -e CONFIG_L2TP_V3
610    scripts/config -d CONFIG_L2TP_IP
611    scripts/config -m CONFIG_L2TP_IP
612    scripts/config -d CONFIG_L2TP_ETH
613    scripts/config -m CONFIG_L2TP_ETH
614    scripts/config -d CONFIG_VXLAN
615    scripts/config -m CONFIG_VXLAN
616    make mod2yesconfig
617
618    make -j$(nproc) bzImage
619    make -j$(nproc) modules
620fi
621
622make modules_install
623make headers_install
624
625mkdir -p /boot/efi/efi/boot
626cp arch/x86_64/boot/bzImage /boot/efi/efi/boot/vmlinuz
627
628# Create startup.nsh for EFI boot
629cat << 'EOFNSH' > /boot/efi/efi/boot/startup.nsh
630fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs
631EOFNSH
632chmod +x /boot/efi/efi/boot/startup.nsh
633
634# Basic system configuration
635echo "${VM_NAME}" > /etc/hostname
636ln -s /etc/sv/agetty-ttyS0 /etc/runit/runsvdir/default
637
638# Install sudo
639xbps-install -y sudo
640
641# Create a user with sudo permissions
642useradd -m -G wheel -s /bin/bash admin
643usermod -U admin
644passwd -d admin
645
646# Add the user to sudoers with ALL/ALL privileges
647echo "admin ALL=(ALL) ALL" > /etc/sudoers.d/admin
648chmod 440 /etc/sudoers.d/admin
649
650EOF
651
652# Make the script executable
653chmod +x /mnt/void-install/setup.sh || { echo "Failed to make setup script executable"; exit 1; }
654
655# Check if Linux source exists in /mnt/pub, if not clone it
656if [ ! -d "/mnt/pub/linux" ]; then
657    mkdir -p /mnt/pub || { echo "Failed to create /mnt/pub directory"; exit 1; }
658    git clone --single-branch --branch v6.14 https://github.com/torvalds/linux.git /mnt/pub/linux || { echo "Failed to clone Linux source"; exit 1; }
659fi
660
661# Mount Linux source to VM using nullfs instead of copying
662mkdir -p /mnt/void-install/usr/src/linux || { echo "Failed to create Linux source mount point"; exit 1; }
663mount_nullfs /mnt/pub/linux /mnt/void-install/usr/src/linux || { echo "Failed to mount Linux source"; exit 1; }
664
665# Run the script inside chroot and ensure we exit properly
666cd /vm || { echo "Failed to change to /vm directory"; exit 1; }
667chroot /mnt/void-install /setup.sh || { echo "Failed to run setup script in chroot"; exit 1; }
668
669echo "Installation complete for VM '${VM_NAME}' with template '${VM_TEMPLATE}'."
670echo "Start the VM with: vm start -f ${VM_NAME}"

void.conf

This is the BHyve-vm template file, you can file the reference for it here: https://github.com/churchers/vm-bhyve/blob/master/sample-templates/config.sample

 1# /vm/.templates/void.conf
 2
 3loader="uefi"
 4cpu="1"
 5cpu_sockets="1"
 6cpu_cores="1"
 7cpu_threads="1"
 8memory=512M
 9disk0_type="virtio-blk"
10disk0_dev="zvol"
11disk0_name="disk0"
12disk0_size="8G"
13loader="bhyveload"
14comports="com1"
15graphics="no"
16xhci_mouse="no"
17wired_memory="yes"
18network0_type="virtio-net"
19network0_switch="bridge0"
20network0_span="yes"

rc.conf 

 1# Enable routing
 2gateway_enable="YES"
 3ipv6_gateway_enable="YES"
 4# Physical interface
 5ifconfig_ix1="inet 192.168.1.128 netmask 255.255.255.0"
 6
 7# Virtual interfaces for VM networking
 8cloned_interfaces="bridge0 epair0 tap0"
 9ifconfig_bridge0="fib 8 up"
10ifconfig_bridge0_aliases="addm epair0b addm tap0"
11ifconfig_epair0a="192.0.0.0/31 fib 0 up"
12ifconfig_epair0b="192.0.0.1/31 fib 8 up"
13ifconfig_tap0="198.18.0.1/23 fib 8 up"
14
15# Static routes (combined)
16static_routes="net1 net2 default_fib8 null_private_fib8"
17route_net1="-net 192.168.1.0/24 192.168.1.1"
18route_net2="default 192.168.1.1"
19route_default_fib8="-fib 8 default 192.0.0.0"
20route_null_private_fib8="-fib 8 -net 192.168.0.0/16 -reject"
21route_fib0_to_fib8="-net 198.18.0.0/23 192.0.0.1"

pf.conf 

1# 192.168.0.0/16 is null-routed on FIB 8, but this should prevent NAT traversal too while NAT is enabled
2nat on ix1 inet from 198.18.0.0/23 to !192.168.0.0/16 -> ix1

Review

Interfaces 

 1ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
 2        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
 3        ether 98:b7:85:1e:de:4d
 4        media: Ethernet autoselect
 5        status: no carrier
 6        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 7ix1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
 8        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
 9        ether 98:b7:85:1e:de:4e
10        inet 192.168.1.128 netmask 0xffffff00 broadcast 192.168.1.255
11        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
12        status: active
13        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
14igb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
15        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
16        ether 3c:ec:ef:56:eb:5c
17        media: Ethernet autoselect
18        status: no carrier
19        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
20igb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
21        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG>
22        ether 3c:ec:ef:56:eb:5d
23        media: Ethernet autoselect
24        status: no carrier
25        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
26lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
27        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
28        inet 127.0.0.1 netmask 0xff000000
29        inet6 ::1 prefixlen 128
30        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
31        groups: lo
32        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
33bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
34        options=0
35        ether 58:9c:fc:10:9b:00
36        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
37        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
38        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
39        groups: bridge
40        fib: 8
41        nd6 options=9<PERFORMNUD,IFDISABLED>
42epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
43        options=8<VLAN_MTU>
44        ether 02:40:77:28:21:0a
45        inet 192.0.0.0 netmask 0xfffffffe broadcast 255.255.255.255
46        groups: epair
47        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
48        status: active
49        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
50epair0b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
51        options=8<VLAN_MTU>
52        ether 02:40:77:28:21:0b
53        inet 192.0.0.1 netmask 0xfffffffe broadcast 255.255.255.255
54        groups: epair
55        fib: 8
56        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
57        status: active
58        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
59tap0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
60        description: vmnet/SWARM1/0/bridge0
61        options=80000<LINKSTATE>
62        ether 58:9c:fc:00:1d:5d
63        inet 198.18.0.1 netmask 0xfffffe00 broadcast 198.18.1.255
64        groups: tap vm-port
65        fib: 8
66        media: Ethernet 1000baseT <full-duplex>
67        status: active
68        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
69        Opened by PID 8159

Routes 

 1➜  /etc netstat -r
 2Routing tables
 3
 4Internet:
 5Destination        Gateway            Flags         Netif Expire
 6default            unifi.localdomain  UGS             ix1
 7localhost          link#5             UH              lo0
 8192.0.0.0          link#5             UHS             lo0
 9192.0.0.0/31       link#7             U           epair0a
10192.168.1.0/24     link#2             U               ix1
11192.168.1.128      link#5             UHS             lo0
12198.18.0.0/23      192.0.0.1          UGS         epair0a
13
14Internet6:
15Destination        Gateway            Flags         Netif Expire
16::/96              link#5             URS             lo0
17localhost          link#5             UHS             lo0
18::ffff:0.0.0.0/96  link#5             URS             lo0
19fe80::%lo0/10      link#5             URS             lo0
20fe80::%lo0/64      link#5             U               lo0
21fe80::1%lo0        link#5             UHS             lo0
22ff02::/16          link#5             URS             lo0
23➜  /etc netstat -r -F 8
24Routing tables (fib: 8)
25
26Internet:
27Destination        Gateway            Flags         Netif Expire
28default            192.0.0.0          UGS         epair0b
29localhost          link#5             UHS             lo0
30192.0.0.0/31       link#8             U           epair0b
31192.0.0.1          link#5             UHS             lo0
32192.168.0.0/16     link#5             URS             lo0
33198.18.0.0/23      link#9             U              tap0
34198.18.0.1         link#5             UHS             lo0
35
36Internet6:
37Destination        Gateway            Flags         Netif Expire
38::/96              link#5             URS             lo0
39localhost          link#5             UHS             lo0
40::ffff:0.0.0.0/96  link#5             URS             lo0
41fe80::%lo0/10      link#5             URS             lo0
42ff02::/16          link#5             URS             lo0
43➜  /etc

Guest 

 1➜  /etc vm list        
 2NAME    DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
 3SWARM1  default    uefi    4    3128M   -    No    Running (8159)
 4➜  /etc 
 5
 6bash-5.2# uname -a
 7Linux SWARM1 6.14.0 #1 SMP PREEMPT_DYNAMIC Tue Apr 15 22:20:22 UTC 2025 x86_64 GNU/Linux
 8bash-5.2# 
 9
10bash-5.2# ip addr show dev eth0
112: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
12    link/ether 58:9c:fc:0d:4a:eb brd ff:ff:ff:ff:ff:ff
13    inet 198.18.0.2/23 scope global eth0
14       valid_lft forever preferred_lft forever
15    inet6 fe80::5a9c:fcff:fe0d:4aeb/64 scope link proto kernel_ll 
16       valid_lft forever preferred_lft forever
17
18bash-5.2# ping 192.0.0.0
19PING 192.0.0.0 (192.0.0.0) 56(84) bytes of data.
2064 bytes from 192.0.0.0: icmp_seq=1 ttl=64 time=0.171 ms
21
22--- 192.0.0.0 ping statistics ---
231 packets transmitted, 1 received, 0% packet loss, time 2ms
24rtt min/avg/max/mdev = 0.171/0.171/0.171/0.000 ms
25
26bash-5.2# ping 4.2.2.2
27PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
2864 bytes from 4.2.2.2: icmp_seq=1 ttl=55 time=29.7 ms
29
30--- 4.2.2.2 ping statistics ---
311 packets transmitted, 1 received, 0% packet loss, time 0ms
32rtt min/avg/max/mdev = 29.661/29.661/29.661/0.000 ms
33
34bash-5.2# traceroute 192.168.1.1
35traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets
36 1  198.18.0.1 (198.18.0.1)  0.223 ms  0.150 ms  0.171 ms
37 2  198.18.0.1 (198.18.0.1)  0.091 ms !H  0.130 ms !H  0.135 ms !H
38bash-5.2#

Boot

After giving up on net boot, it drops to UEFI shell, which searches for startup.nsh:

 1UEFI Interactive Shell v2.2
 2EDK II
 3UEFI v2.70 (BHYVE, 0x00010000)
 4Mapping table
 5      FS0: Alias(s):HD0b:;BLK1:
 6          PciRoot(0x0)/Pci(0x4,0x0)/HD(1,GPT,26B76E50-1900-11F0-A1CB-98B7851EDE4D,0x40,0x80000)
 7     BLK0: Alias(s):
 8          PciRoot(0x0)/Pci(0x4,0x0)
 9     BLK2: Alias(s):
10          PciRoot(0x0)/Pci(0x4,0x0)/HD(2,GPT,26BA3334-1900-11F0-A1CB-98B7851EDE4D,0x80040,0xF7FF80)
11
12
13Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
14Shell> fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs
15[    0.000000] Linux version 6.14.0 (root@stelleri.netcrave.network) (gcc (GCC) 13.2.0, GNU ld (GNU Binutils) 2.41) #1 SMP PREEMPT_DYNAMIC Mon Apr 14 06:08:58 UTC 2025
16[    0.000000] Command line: fs0:\efi\boot\vmlinuz console=ttyS0 root=/dev/vda2 rootflags=ufstype=ufs2 rootfstype=ufs

Takeaways

I think that’s it:

  • The configuration persists across reboots,
  • it’s reproducible and if I could just get the rest of the house behind a VRF similar to how bridge0 is setup on FIB 8:
1(west) <home network>---VGW---<core router (north to internet)>---HV---<FIB 8> (east)

and there is a role in here for VLAN as well, but it would make more sense as VGW and HV to exist on separate VLAN, and really I don’t have any other layer 2 needs in mind. Ideally:

1(west) <home network>---VGW1---<core router (north to internet)>---VGW2---HV---<FIB 8> (east)

The TLDR is VRF is just better and VLANs are (still useful.) The responsibility of null routing 192.168.0.0/16 would go to a VGW2 that is part of the core router rather than being the responsibility of HV because separating these concerns would ensure that in the case that a BHyve guest is no longer a reliably secure sandbox then the home network would still remain unreachable unless the core router is compromised.

There’s nothing about this that says you can’t use VLANs galore either. I think I may have made this work similarly with Libvirt by offloading the networking configuration to system-networkd once but I don’t remember it being something I was particularly proud of. That’s not to say that this necessarily is, that remains to be seen. To say the least, I just like it and it because the whole setup just makes a lot of sense to me without being too much or too little. I’m pretty happy with how it seems to have turned out though.

Need help ?

Feel free to shoot me an e-mail!