FreeBSD bhyve hypervisor to run OpenBSD virtual machines

    

Because I’m not 100% satisfied with my OmniOS bhyve experiment for running OpenBSD virtual machines, I’m giving it a try on a stock FreeBSD 14. And as usual, I’ll write down how I did it in case you are interested too :)

What happens with Illumos?

I have network issues while running OpenBSD on the bhyve implementation of OmniOS. They happen a bit less on a ThinkPad A485 with Realtek 8168 than on a Partaker H2 with Intel I211 but they still do. The reproductible test is quite simple, from another machine on a LAN, send or get quite a lot of data to or from the virtual machine.

# dd if=/dev/urandom of=TEST bs=1m count=1024
# while true; do scp TEST openbsdhvm:/home/ ; sleep 1; done

Everything works like a charm (~460Mbps) for a moment. Then, when about 400GB have been transferred, the VM network stack starts to woe. Connectivity goes up and down, SSH connexion gets long delay… The VM is not useable anymore as a connected asset.

Yet, using the VM from the console is responsive as usual ; and the Illumos host also doesn’t show signs of illness. The top command reveals no load, interrupt or hungry process. Everything appears as if the VM was doing nothing. I couldn’t find any information in the logs of the guest or the host. After the VM is rebooted, everything starts working again normally ; until is breaks again.

If you have ideas, don’t hesitate to ping me :)

FreeBSD installation

Read Chapter 2. Installing FreeBSD .

Grab an install media. I used FreeBSD-14.0-RELEASE-amd64-memstick.img. None of the ISO or IMG file worked while with my Ventoy USB key.

Install FreeBSD as usual. I used a ZFS root on a single disk and choose to enable all security features.

FreeBSD first-boot configuration

Connect to the server using SSH, deploy the public key for the unprivileged user and root. While I’m there, I like to setup a forward(5) file to get email informations properly.

$ mkdir ~/.ssh && cat > ~/.ssh/authorized_keys
(...)
^D
$ echo changeme@example > ~/.forward

$ su -
# mkdir ~/.ssh && cat > ~/.ssh/authorized_keys
(...)
^D
# echo "changeme" > ~/.forward

# vi /etc/ssh/sshd_config
(...)
PermitRootLogin prohibit-password
PasswordAuthentication no

# service sshd restart

Yes, I’m allowing SSH root access 😱. But this server is not exposed on the Internet. So if someone tries to gain a root access, I’ll have other things to take care of.

During installation, I use DHCP. To switch static IP, simply run a few commands:

# sysrc ifconfig_em0="ether 00:00:5E:00:53:76"
# sysrc ifconfig_em0="inet 192.0.2.76 netmask 255.255.255.0"
# sysrc ifconfig_em0_ipv6="inet6 2001:db8::76 prefixlen 32"

# sysrc defaultrouter="192.0.2.1"
# sysrc ipv6_defaultrouter="2001:db8::1"

# cat > /etc/resolv.conf
search home.arpa
nameserver 192.0.2.1
nameserver 2001:db8::1

# nohup sh -c 'service netif restart && service routing restart'

Enable and configure pf(4)

# service pf enable
# service pflog enable

# kldload pf
# sysrc kld_list+=pf

# vi /etc/pf.conf
ext_if="igb0"
block in
pass in on $ext_if proto tcp to port ssh
pass out modulate state

# pfctl -f /etc/pf.conf -n
# pfctl -e -f /etc/pf.conf
# service pflog start

Update FreeBSD

# freebsd-update fetch
# freebsd-update install
# reboot

bhyve hypervisor: using native tools

Have a look at the Virtualization Handbook section , there are loads of information there.

Enable bhyve is as simple as loading the proper kernel module.

# kldload vmm
# sysrc kld_list+=vmm

Every virtual machines will have its own tap(4) device to access network. Creating a bridge(4) and adding the tap interfaces as members enables network connection between them. Keep the bridge as-is and all VM are isolated. Add a physical interface to the bridge and the VMs will be able to appear on the LAN.

For the purpose of this article, two VMs will be created and gain direct access to my LAN resources.

# ifconfig tap0 create
# ifconfig tap1 create
# sysctl -w net.link.tap.up_on_open=1
# ifconfig bridge0 create
# ifconfig bridge0 addm igb0 addm tap0 addm tap1
# ifconfig bridge0 up

The sysctl variable ensures the tunnel devices will be marked up when the control device is opened.

To persist this configuration, files have to be modified.

# echo "net.link.tap.up_on_open=1" >> /etc/sysctl.conf
# sysrc cloned_interfaces="bridge0 tap0 tap1"
# sysrc ifconfig_bridge0="addm igb0 addm tap0 addm tap1"

Because I liked how OmniOS managed its VM, I’ll do the same kind of thing: have a dataset to host the virtual machines stuff and using Zvol as virtual disks - instead of using image files.

# zfs create -o mountpoint=/guests tank/guests
# zfs create tank/guests/iso

The server is now ready to launch virtual machines.

Run a FreeBSD guest

Grab the FreeBSD 14 installation image:

# cd /guests/iso
# fetch https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso
# cd -

Create a virtual disk:

# zfs create -o volmode=dev -V 16G tank/guests/freebsd

Install FreeBSD using the example script:

# sh /usr/share/examples/bhyve/vmrun.sh                     \
  -c 2 -m 1024m -t tap0 -d /dev/zvol/tank/guests/freebsd    \
  -i -I /guests/iso/FreeBSD-14.0-RELEASE-amd64-bootonly.iso \
  freebsd

The console can be set to “xterm” if you wish too.

In this network configuration, the VM can benefit from the LAN DHCP server.

At the end of the installation, selecting “Shutdown” will turn the VM off, stop the script and get you back to the hypervisor prompt.

Run the FreeBSD virtual machine using the example script:

# sh /usr/share/examples/bhyve/vmrun.sh                  \
  -c 2 -m 1024m -t tap0 -d /dev/zvol/tank/guests/freebsd \
  freebsd

The script runs in the foreground and connects to the VM console. To stop the VM from the console, simply use halt -p. To start the VM again, run the script again.

Run an OpenBSD guest

To boot non-FreeBSD guests, you need some extras material. In this particular case, a UEFI (and/or legacy BIOS) firmware is needed.

# pkg info bhyve-firmware

OpenBSD 7.4 installation ISO doesn’t support UEFI boot. This will change for 7.5. As for now, I’ll be using the installation IMG. One could boot OpenBSD using legacy BIOS. But I got used to using UEFI so lets go this way.

Also, I don’t run OpenBSD guests with a graphical console. I’m using the serial console emulation.

Grab the OpenBSD 7.4 installation image:

# cd /guests/iso
# fetch https://cdn.openbsd.org/pub/OpenBSD/7.4/amd64/miniroot74.img
# cd -

Create a virtual disk:

# zfs create -o volmode=dev -V 16G tank/guests/openbsd

Install OpenBSD:

# bhyve -A -D -H -P -S -u -w -c 2 -m 4G                   \
  -s 0,amd_hostbridge                                     \
  -s 3,virtio-blk,/dev/zvol/tank/guests/openbsd           \
  -s 4,ahci-hd,/guests/iso/miniroot74.img                 \
  -s 10,virtio-net,tap1,mac=00:00:5E:00:53:99             \
  -s 20,virtio-rnd                                        \
  -s 31,lpc -l com1,stdio                                 \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  openbsd

(...)
probing: pc0 com0 com1 mem[640K 3048M 16M 3M 1024M]                         
disk: hd0 hd1*                 
>> OpenBSD/amd64 BOOTX64 3.65
boot> set tty com0                  
switching console to com0                                                   
>> OpenBSD/amd64 BOOTX64 3.65                                               
boot>                                 
cannot open hd0a:/etc/random.seed: No such file or directory
booting hd0a:/bsd: 3969732+1655808+3886664+0+708608
[109+444888+297417]=0xa76798                                                                        
entry point at 0x1001000                                                    
Copyright (c) 1982, 1986, 1989, 1991, 1993                                  
        The Regents of the University of California.  All rights
reserved.                                                                              
Copyright (c) 1995-2023 OpenBSD. All rights reserved.
https://www.OpenBSD.org
                                                                                                                                                        
OpenBSD 7.4 (RAMDISK_CD) #1322: Tue Oct 10 09:07:38 MDT 2023                                                                                            
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/RAMDISK_CD
real mem = 4254203904 (4057MB)                                              
avail mem = 4121255936 (3930MB)                                             
random: good seed from bootblocks                                           
mainbus0 at root                                                            
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xbfbcf000 (12 entries)
bios0: vendor BHYVE version "14.0" date 10/17/2021                          
bios0: FreeBSD BHYVE
(...)

Don’t forget to use set tty com0 while in the OpenBSD bootloader. Then proceed to installation as usual. Use (G)PT as the formating layout ; eventhough the installer will complain a bit. When done, (H)alt the system and quit the console.

Running the VM will then be as simple as:

# bhyve -A -D -H -P -S -u -w -c 2 -m 4G                   \
  -s 0,amd_hostbridge                                     \
  -s 3,virtio-blk,/dev/zvol/tank/guests/openbsd           \
  -s 10,virtio-net,tap1,mac=00:00:5E:00:53:99             \
  -s 20,virtio-rnd                                        \
  -s 31,lpc -l com1,stdio                                 \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  openbsd

It can also be run with the example script:

# sh /usr/share/examples/bhyve/vmrun.sh              \
  -c 2 -m 4G -t tap1 -C stdio                        \
  -d /dev/zvol/tank/guests/openbsd                   \
  -E -f /usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
   openbsd

bhyve hypervisor: using vm-bhyve

Running stock tools is my preferred way of doing things. But the need to use tmux or such to maintain the VM up and running doesn’t fit me well - too many vmd(8)/vmctl(8) and vmadm(1M)/zadm(1) habbits.

There are various tools available in the packages repo. After a quick doc review, I went for vm-bhyve .

Note: if you already configured bhyve using the previous section, you should probably revert the network configuration to avoid weird behaviours. Same goes with the VM dataset if you’ll use the same.

Enable bhyve is as simple as loading the proper kernel module.

# kldload vmm
# sysrc kld_list+=vmm

Following the “Quick-Start” directions, setting it up is fast.

# pkg install vm-bhyve

# sysrc vm_enable="YES"
# sysrc vm_dir="zfs:tank/guests"

# vm init
# cp /usr/local/share/examples/vm-bhyve/* /guests/.templates/

Once again, I’m going for the network configuration where all my VMs have access to the LAN.

# vm switch create homelan
# vm switch add homelan igb0

Using vm persists configuration in the configured dataset.

# cat /guests/.config/system.conf 
switch_list="homelan"
type_homelan="standard"
ports_homelan="igb0"

Run a FreeBSD guest

Grab the FreeBSD 14 installation ISO:

# vm iso https://download.freebsd.org/releases/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-amd64-bootonly.iso

Create the virtual machine:

# vm create -t freebsd-zvol -s 16G -m 1024m -c 2 freebsd
# sed -i -e 's/"public"/"homelan"/' /guests/freebsd/freebsd.conf

Install FreeBSD:

# vm install -f freebsd FreeBSD-14.0-RELEASE-amd64-bootonly.iso

When installation is finished, (S)hutdown turns the VM off and the vm script gets you back to the hypervisor prompt.

The vm will be run in background. To access to the console, a dedicated command is required.

# vm start freebsd
Starting freebsd
  * found guest in /guests/freebsd
  * booting...

# vm list
NAME     DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO  STATE
freebsd  default    bhyveload  2    1024m   -    No    Running (40698)

# vm console freebsd

The VM console can be quitted using ~~. or ~ + Ctrl-D

Run an OpenBSD guest

To boot non-FreeBSD guests, you need some extras material. In this particular case, a UEFI (and/or legacy BIOS) firmware is needed. In case you have not done it previsously, install the firmware.

# pkg info bhyve-firmware

OpenBSD 7.4 installation ISO doesn’t support UEFI boot. This will change for 7.5. As for now, I’ll be using the installation IMG. One could boot OpenBSD using legacy BIOS. But I got used to using UEFI so lets go this way.

Also, I don’t run OpenBSD guests with a graphical console. I’m using the serial console emulation.

Grab the OpenBSD 7.4 installation image:

# vm iso https://cdn.openbsd.org/pub/OpenBSD/7.4/amd64/miniroot74.img

The original OpenBSD template uses grub and image file. The grub configuration is not recommended nowadays. And I want to use ZFS volumes. So create a custom OpenBSD template:

# vi /guests/.templates/openbsd.conf
loader="uefi"
cpu=1
memory=512M
network0_type="virtio-net"
network0_switch="homelan"
disk0_type="virtio-blk"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk1_type="ahci-hd"
disk1_name="../.iso/miniroot74.img"
bhyve_options="-A -D -H -P -S -u -w"
virt_random="yes"

Create the virtual machine:

# vm create -t openbsd -s 16G -m 1024m -c 2 openbsd

Install OpenBSD, telling the boot loader to use the console:

# vm start -f openbsd
probing: pc0 com0 com1 mem[640K 1000M 16M 3M]
disk: hd0 hd1*
>> OpenBSD/amd64 BOOTX64 3.65
boot> set tty com0
switching console to com0
>> OpenBSD/amd64 BOOTX64 3.65
boot> <ENTER>
(...)

Don’t forget to use set tty com0 while in the OpenBSD bootloader. Then proceed to installation as usual. Use (G)PT as the formating layout ; eventhough the installer will complain a bit. When done, (H)alt the system and quit the console.

Remove the installation media definition and start the VM using the simple vm command:

# vm configure openbsd
:g/^disk1/d
:wq

# vm start openbsd
# vm console openbsd

The VM console can be quitted using ~~. or ~ + Ctrl-D.

Create and deploy OpenBSD images

Creating virtual machine using the same base can be tiresome. And this task can be automated using several manners. The same way I did on OmniOS , I decided to use images using vm-bhyve.

You can create an OpenBSD instance using the previous steps. I like to modified the installation by configuring a few standard sthings (like SSH public keys, sshd and smtpd configuration) and have the new VM delete all traces of previous installation and upgrade to the latest syspatches. This is not mandatory though.

# vm create -t openbsd -s 16G -m 1024m -c 2 openbsd74

# vm start -f openbsd74
(...)
boot> set tty com0
(...)
Exit to (S)hell, (H)alt or (R)eboot? [reboot] s
To boot the new system, enter 'reboot' at the command prompt.

openbsd74# chroot /mnt /bin/ksh

# echo "ssh-ed25519 (...)" > /root/.ssh/authorized_keys
# TERM=vt220 vi /etc/ssh/sshd_config

# echo change_me@example > /root/.forward

# TERM=vt220 vi /etc/mail/smtpd.conf
# echo "(...)" > /etc/mail/secrets
# chown root:_smtpd /etc/mail/secrets
# chmod 0640 /etc/mail/secrets

# cp /etc/examples/doas.conf /etc/
# TERM=vt220 vi /etc/doas.conf

# cat >> /etc/rc.firsttime
echo "************************************************************************"
echo "This system was build from a template."
echo -n "System hostname? (short form, e.g. 'foo') "
read _hostname
/usr/bin/sed -E -i "s/openbsd74/$_hostname/g" /etc/myname
/bin/rm /etc/ssh/ssh_host*
echo "Applying syspatches..."
/usr/sbin/syspatch
echo "Updating packages..."
/usr/sbin/pkg_add -u
echo "Rebooting in 5 seconds..."
/bin/sleep 5
/sbin/shutdown -r now
^D

# exit
# halt -p
syncing disks... done

The operating system has halted.
Please press any key to reboot.

Remove the installer reference and create the OpenBSD template image:

# vm configure openbsd74
:g/^disk1/d
:wq

# vm image create -d "OpenBSD 7.4/amd64" openbsd74
Creating a compressed image, this may take some time...
Image of openbsd74 created with UUID 60817e2e-b268-11ee-9aeb-009027e529f7

Deploy a new VM from an image. Sizing and/or network and/or disk configuration can be modified using the configure action.

# vm image list
UUID                                  NAME       CREATED
DESCRIPTION
60817e2e-b268-11ee-9aeb-009027e529f7  openbsd74  Sun Jan 14 00:06:26 CET
2024  OpenBSD 7.4/amd64

# vm image provision 60817e2e-b268-11ee-9aeb-009027e529f7 puffy
Unpacking guest image, this may take some time...

# vm configure puffy

# vm start puffy
# vm console puffy

The template source can be either deleted or kept. Depending on what you prefer.

# vm destroy openbsd74

Start VM automatically

vm-bhyve uses the “startall” command at boot time. Definition on which VM to start and how many time to wait before starting the next one is done via environment variables.

# sysrc vm_list+="puffy"
# sysrc vm_dealy="5"

If you wonder, the OpenBSD dmesg output is available here .

And that’s all for now!