FreeBSD 9 booting from a 3 disks ZFS

I already managed to boot FreeBSD from ZFS ; a single disk ZFS pool. Now, let’s see how to boot FreeBSD-9-RELEASE from a ZFS pool made of 3 disks using RAIDZ configuration.

I used the FreeBSD-9.0-RELEASE-amd64-dvd1.iso CD-ROM image.

Introduction

In this example, I’m using a virtual machine running on ESXi v5, with 2 vCPU, 4GB of RAM and 3x 8GB virtual disk attached on a “LSI Logic Parallel” virtual adapter. The “Guest Operating System” is configured as “Other / FreeBSD (64-bit)”.

That said, boot the ISO and select the “Live CD” option when prompted. The CD/DVD is mounted read-only and I’m using bits in /tmp to alter the filesystem:

# mkdir /tmp/boot # mount_unionfs /tmp/boot /boot

I do the stuff using the Bourne shell, as I know it better that C-shell, and verify that the VM does know about my virtual disks:

# exec /bin/sh # sysctl kern.disks kern.disks: cd0 da2 da1 da0

Preparing the disks

ZFS requires a GPT configuration. Configure parts of those disks as swap space and the rest as a ZFS pool:

# for DISK in 0 1 2; do gpart create -s gpt da${DISK}; done da0 created da1 created da2 created # for DISK in 0 1 2; do > gpart add -s 128k -t freebsd-boot -l boot${DISK} da${DISK} > gpart add -s 2G -t freebsd-swap -l swap${DISK} da${DISK} > gpart add -t freebsd-zfs -l disk${DISK} da${DISK} > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da${DISK} > done da0p1 added da0p2 added da0p3 added bootcode written to da0 da1p1 added da1p2 added da1p3 added bootcode written to da1 da2p1 added da2p2 added da2p3 added bootcode written to da2

The swap spaces will be known as swap0, swap1, swap2.
The ZFS will be composed of disk0, disk1, disk2.

I can loose one disk of this set and still be able to boot from the two remaining ones.

Configuring ZFS

Load the ZFS related modules and create a RAIDZ with the 3 available disks:

# kldload zfs.ko # zpool create -f -m none tank raidz /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2 # zpool status tank pool: tank state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 gpt/disk2 ONLINE 0 0 0 errors: No known data errors

Configure general parameters and create the various datasets you want:

# zfs set checksum=fletcher4 tank # zfs create -o mountpoint=/media -o compression=off -o dedup=sha256,verify tank/freebsd9 # zpool set bootfs=tank/freebsd9 tank # zfs create -o compression=gzip tank/freebsd9/usr # zfs create -o compression=gzip tank/freebsd9/var

I’m setting checksum=fletcher4 to ensure better integrity checking. I’m setting up deduplication from the root of my datasets and add compression to datasets where most data will be.

Note that you can’t, AFAIK, boot from a compressed dataset. This is why tank/freebsd9 isn’t compressed at all. Also note that installing the system under freebsd9 is used to ease ZFS sharing and snapshot managing. This way, I can have tank/freebsd8 and tank/freebsd10 on the same ZFS pool and boot from any of them.

Install FreeBSD

The installation is only a matter of untarring archives:

# cd /usr/freebsd-dist # for FILE in base.txz doc.txz kernel.txz lib32.txz; do > tar --unlink -xpJf ${FILE} -C /media/ > done

A few more steps are required to end the configuration:

# cd / # chroot /media /bin/sh # cat >> /etc/rc.conf zfs_enable="YES" hostname="zfreebsd.tumfatig.net" # cat >> /boot/loader.conf zfs_load=YES vfs.root.mountfrom="zfs:tank/freebsd9" ^D # cat >> /etc/fstab /dev/gpt/swap0 none swap sw 0 0 /dev/gpt/swap1 none swap sw 0 0 /dev/gpt/swap2 none swap sw 0 0 ^D # passwd root Changing local password for root New Password: Retype New Password: # tzsetup (...) # cd /etc/mail; make aliases /usr/sbin/sendmail -bi -OAliasFile=/etc/mail/aliases /etc/mail/aliases: 28 aliases, longest 10 bytes, 283 bytes total chmod 0640 /etc/mail/aliases.db # exit

Before exiting chroot, you may want to run sysinstall to “Configure” various aspect of the system ; like network configuration, daemon to be started, user management… If you created users to log in, you can install the user’s SSH key now.

Final settings

Update the ZFS pool cache and copy it to the FreeBSD instance:

# cd / # zpool export tank && zpool import tank # cp -p /boot/zfs/zpool.cache /media/boot/zfs/

Unmount the filesystem and set the mounpoints:

# zfs unmount -a # zfs set mountpoint=legacy tank/freebsd9 # zfs set mountpoint=/usr tank/freebsd9/usr # zfs set mountpoint=/var tank/freebsd9/var

Cross your fingers and reboot.

Feedback

So far, the installation is working great.

The only strange thing is the long booting time. It takes about a minute or two to get to the boot loader. Not sure if this is related to FreeBSD or to the VMware configuration. Since the OS is booted, it seems to run at full speed.

Here’s a bit of number from ZFS:

# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT tank 17.9G 906M 17.0G 4% 1.01x ONLINE - # zfs list -r -o name,used,available,referenced,mountpoint,compressratio NAME USED AVAIL REFER MOUNTPOINT RATIO tank 610M 11.1G 40.0K none 1.74x tank/freebsd9 596M 11.1G 345M legacy 1.74x tank/freebsd9/usr 250M 11.1G 250M /usr 2.85x tank/freebsd9/var 1.72M 11.1G 1.72M /var 3.96x

Sources

Here’s a few docs I studied while experimenting:

  • Installing FreeBSD Root on ZFS (Mirror) using GPT ;
  • HOWTO: FreeBSD ZFS Madness, from vermaden ;
  • FreeBSD ZFS Tuning Guide

1 Comment

  • Victor M. Sat, 08 Dec 2012 01:57:25 +0000 Reply

    Thank you very much! It works! )) freebsd 9.0_i386 on virtualbox. I will try it at work on real pc. If I could make NLE on freebsd like Final Cut Pro or beter with huge amount of a/v effects and net renderer I will die in peace! ))) Thank you. In FreeBSD we trust!

Leave a Reply

%d bloggers like this: