Install FreeBSD on a ZFS pool that’s mirrored using GPT partitions 1

Here’s my notes for ZFS root installation with FreeBSD 8.1. (amd64 on my system)

Boot up the FreeBSD install disk, you’ll need one with the Fixit system.  In the examples listed, I had 2 disks, ad4 and ad6.  Yours may be different, replace as necessary.

1. Add the modules on boot, hit “6” at the BTX loader menu, then enter these lines at the “OK” prompt.

load ahci.ko
load opensolaris.ko
load zfs.ko


2. At the sysinstall menu, go to Fixit.

3. Create the GPT disks and partitions.

  • Delete any partions already on the disk, and destroy. (only if needed)  On my disks, I had 2 partitions created from another OS installation.  “-i 2” indicates the partition number.  You may need to add a third or more.
    gpart delete -i 2 ad4
    gpart delete -i 1 ad4
    gpart delete -i 2 ad6
    gpart delete -i 1 ad6
    gpart destroy ad4
    gpart destroy ad6 
  • Create the gpt disk.
    gpart create -s gpt ad4
    gpart create -s gpt ad6 
  • (NOTE: “gpart show” and “gpart show –l” will display the partitions.
  • You may want to layout different partitions, but we’ll just create a boot, swap, and zfs partition here, with GPT labels.  (swap is better left off ZFS for crash dumps)
    gpart add -s 128 -t freebsd-boot -l boot0 ad4
    gpart add -s 128 -t freebsd-boot -l boot1 ad6
    gpart add -s 8388608 -t freebsd-swap -l swap0 ad4
    gpart add -s 8388608 -t freebsd-swap -l swap1 ad6
    gpart add -t freebsd-zfs -l disk0 ad4
    gpart add -t freebsd-zfs -l disk1 ad6

    Next, add the bootcode to the disk.

    gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad4
    gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad6

4. OK, now we’ve got some GTP disks setup for a mirror and boot. 
IMPORTANT!!!  Make sure you don’t skip this step, create the /boot/zfs directory for the zpool.cache.

mkdir /boot/zfs


5. Now we can start the ZFS fun! On the FreeBSD wiki and on other sites I used as reference, there may be listed other options and settings for your ZFS layout.  Here, I am going to keep it very simple, but I’ll list out my options later.

Also, you may prefer different names or even separate location/directory/ZFS filesystem for your root pool. My preference is to call the root pool “rpool” and place the FreeBSD system in the ROOT filesystem.  (looks like this: rpool/ROOT)  In addition, my preference is to place “/home” outside of the ROOT fs.  Which allows is to separate system from data and management of snapshots is more flexible. But this is totally a preference thing.  One of the beautiful things about ZFS is the flexibility and ease of use.

  • Let’s create the pool. Notice the gpt/disk# items, which correspond to your gpt labels. Also, we’ll disable a mountpoint on rpool.
    zpool create rpool mirror gpt/disk0 gpt/disk1
    zfs set mountpoint=none rpool

  • Add the ROOT filesystem.  Again, here you may prefer to add all ZFS filesystems for places like /usr, /var, /tmp, and other system locations.  I don’t recommend it, but in this example we’ll keep it all in one filesystem, except /home.
    zfs create -o mountpoint=/mnt rpool/ROOT
    zfs create -o mountpoint=/mnt/home rpool/home

    (OPTIONAL file systems)

    zfs create rpool/ROOT/usr
    zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/ports
    zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/distfiles
    zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/packages
    zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/src
    zfs create rpool/ROOT/var
    zfs create -o exec=on -o setuid=off rpool/ROOT/tmp

6. Install the base system.  We’ll be intsalling the system into the /mnt directory for now, but we’ll change those mounts later.

  • cd /dist/8.1*
    export DESTDIR=/mnt
    for dir in base catpages dict doc games info lib32 manpages ports; do (cd $dir ; ./ ; done
    cd src ; ./ all
    cd ../kernels ; ./ generic
    cd /mnt/boot ; cp -Rlp GENERIC/* /mnt/boot/kernel/

7. Edit /mnt/boot/loader.conf and add these lines.


8. Edit /mnt/etc/rc.conf and add these lines. I have re0 as my network interface, your’s may be different.


9. Edit /mnt/etc/fstab and add these lines. Setup your swap.

/dev/gpt/swap0 none	swap	sw	0	0
/dev/gpt/swap1 none	swap	sw	0	0


10. Copy the zpool.cache.  IMPORTANT!!!!! Don’t forget this step!!

cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache


11. Export this…

export 	LD_LIBRARY_PATH=/dist/lib


12. Unmount ZFS filesystems and set the correct mountpoints for the new root to boot. If you setup other/optional zfs filesystems, then you’ll want to set their correct mount points, too.

zfs umount -a
zfs set mountpoint=legacy rpool/ROOT
zfs set mountpoint=/home rpool/home


13.  Set the bootfs property of the pool to rpool/ROOT

zpool set bootfs=rpool/ROOT rpool


That’s about it!  Well, actually, you’ll still need to do a lot of post install stuff.  (see the wiki (set passwd, time zone, etc))  This is only a very base load of FreeBSD, root doesn’t even have a password. 

Keep in mind, at the time I wrote this, there is a bug in the zfs boot loader, where you can only boot from the first disk in the mirror.  Kind of reduces the usefulness of a root mirror, that’s for sure.  However, there is a patch that you can compile a new zfs loader with.  There’s a procedure to do so, and I’ll write up a how-to on that sooner than later.

Here are the site’s I used as reference, as well as the forum post on the root mirror boot issue.