Here’s my notes for ZFS root installation with FreeBSD 8.1. (amd64 on my system)

Boot up the FreeBSD install disk, you’ll need one with the Fixit system.  In the examples listed, I had 2 disks, ad4 and ad6.  Yours may be different, replace as necessary.

1. Add the modules on boot, hit “6″ at the BTX loader menu, then enter these lines at the “OK” prompt.

load ahci.ko
load opensolaris.ko
load zfs.ko
boot 

 

2. At the sysinstall menu, go to Fixit.

3. Create the GPT disks and partitions.

• Delete any partions already on the disk, and destroy. (only if needed)  On my disks, I had 2 partitions created from another OS installation.  “-i 2” indicates the partition number.  You may need to add a third or more.

  gpart delete -i 2 ad4
  gpart delete -i 1 ad4
  gpart delete -i 2 ad6
  gpart delete -i 1 ad6
  gpart destroy ad4
  gpart destroy ad6 

• Create the gpt disk.

  gpart create -s gpt ad4
  gpart create -s gpt ad6 

• (NOTE: “gpart show” and “gpart show –l” will display the partitions.
• You may want to layout different partitions, but we’ll just create a boot, swap, and zfs partition here, with GPT labels.  (swap is better left off ZFS for crash dumps)

  gpart add -s 128 -t freebsd-boot -l boot0 ad4
  gpart add -s 128 -t freebsd-boot -l boot1 ad6
  gpart add -s 8388608 -t freebsd-swap -l swap0 ad4
  gpart add -s 8388608 -t freebsd-swap -l swap1 ad6
  gpart add -t freebsd-zfs -l disk0 ad4
  gpart add -t freebsd-zfs -l disk1 ad6
  

• Next, add the bootcode to the disk.

  gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad4
  gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad6
  

4. OK, now we’ve got some GTP disks setup for a mirror and boot. 
IMPORTANT!!!  Make sure you don’t skip this step, create the /boot/zfs directory for the zpool.cache.

mkdir /boot/zfs

 

5. Now we can start the ZFS fun! On the FreeBSD wiki and on other sites I used as reference, there may be listed other options and settings for your ZFS layout.  Here, I am going to keep it very simple, but I’ll list out my options later.

Also, you may prefer different names or even separate location/directory/ZFS filesystem for your root pool. My preference is to call the root pool “rpool” and place the FreeBSD system in the ROOT filesystem.  (looks like this: rpool/ROOT)  In addition, my preference is to place “/home” outside of the ROOT fs.  Which allows is to separate system from data and management of snapshots is more flexible. But this is totally a preference thing.  One of the beautiful things about ZFS is the flexibility and ease of use.

• Let’s create the pool. Notice the gpt/disk# items, which correspond to your gpt labels. Also, we’ll disable a mountpoint on rpool.

  zpool create rpool mirror gpt/disk0 gpt/disk1
  zfs set mountpoint=none rpool
  

• Add the ROOT filesystem.  Again, here you may prefer to add all ZFS filesystems for places like /usr, /var, /tmp, and other system locations.  I don’t recommend it, but in this example we’ll keep it all in one filesystem, except /home.

  zfs create -o mountpoint=/mnt rpool/ROOT
  zfs create -o mountpoint=/mnt/home rpool/home
  

  (OPTIONAL file systems)

  zfs create rpool/ROOT/usr
  zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/ports
  zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/distfiles
  zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/packages
  zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/src
  zfs create rpool/ROOT/var
  zfs create -o exec=on -o setuid=off rpool/ROOT/tmp
  

6. Install the base system.  We’ll be intsalling the system into the /mnt directory for now, but we’ll change those mounts later.

• cd /dist/8.1*
  export DESTDIR=/mnt
  for dir in base catpages dict doc games info lib32 manpages ports; do (cd $dir ; ./install.sh) ; done
  cd src ; ./install.sh all
  cd ../kernels ; ./install.sh generic
  cd /mnt/boot ; cp -Rlp GENERIC/* /mnt/boot/kernel/
  

7. Edit /mnt/boot/loader.conf and add these lines.

ahci_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:rpool/ROOT"

8. Edit /mnt/etc/rc.conf and add these lines. I have re0 as my network interface, your’s may be different.

zfs_enable="YES"
ifconfig_re0="DHCP"
hostname="systemname"

9. Edit /mnt/etc/fstab and add these lines. Setup your swap.

/dev/gpt/swap0 none	swap	sw	0	0
/dev/gpt/swap1 none	swap	sw	0	0

 

10. Copy the zpool.cache.  IMPORTANT!!!!! Don’t forget this step!!

cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache

 

11. Export this…

export 	LD_LIBRARY_PATH=/dist/lib

 

12. Unmount ZFS filesystems and set the correct mountpoints for the new root to boot. If you setup other/optional zfs filesystems, then you’ll want to set their correct mount points, too.

zfs umount -a
zfs set mountpoint=legacy rpool/ROOT
zfs set mountpoint=/home rpool/home

 

13.  Set the bootfs property of the pool to rpool/ROOT

zpool set bootfs=rpool/ROOT rpool

 

That’s about it!  Well, actually, you’ll still need to do a lot of post install stuff.  (see the wiki (set passwd, time zone, etc))  This is only a very base load of FreeBSD, root doesn’t even have a password. 

Keep in mind, at the time I wrote this, there is a bug in the zfs boot loader, where you can only boot from the first disk in the mirror.  Kind of reduces the usefulness of a root mirror, that’s for sure.  However, there is a patch that you can compile a new zfs loader with.  There’s a procedure to do so, and I’ll write up a how-to on that sooner than later.

Here are the site’s I used as reference, as well as the forum post on the root mirror boot issue.

http://www.b0rken.org/freebsd/zfs.html

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror

http://forums.freebsd.org/showthread.php?p=95482

I setup my share right, followed all the instructions, set the /etc/pam.conf entry.  Couldn’t get into my share, and my password wouldn’t work.

I found that you can test your PAM mapping like so:

smbutil login –c username

But that gave me:

Keychain entry not found.

 

Ok, had to do a little searching…  found out that you can SET THE STORED PASSWORD FOR SMB!  Duh!  Why didn’t I think of that in the first place?  LOL!

All you need to do is this:

smbutil login username@server

It will ask you for a password and suddenly your login from windows will work! Nice!

Keep in mind, that is assuming your server is setup in workgroup mode, joined to your workgroup not the domain.  I think you would just specify “username@netbiosdomain” in that case.

Check the /etc/pam.conf has the smb line:

   other    password required    pam_smb_passwd.so.1    nowarn

If not, add that to the end of the file.  If you, check /var/smb/smbpasswd you should also see a line for your account with a hashed password listed.  In my case, I had my username and an ID listed, but not hashed password.  I didn’t have the line in pam.conf.  After I added it, I had to run the passwd username command on my account, which generated the smb password and inserted it into the smbpasswd file. Then the error about password not being correct went away and I could get in from Windows.  (Windows 7 too! With the “NTLMv2 if negotiated” option in policy.)

Notes to self:

Install your system to a PARTITION, NOT the whole drive.  I seem to remember reading somewhere that this was required.

My system had 2 disks.  rpool was setup on disk 0.  (those are ZERO’s not O’s.)

c8t0d0s0  (disk 0 with rpool)
c8t1d0s0   (disk 1 we want to mirror)

After installed and booted into new system…  (as root)

format
(choose your second disk, should be 1 and in this sys it was c8t1d0)
fdisk
(here, choose Y to select the 100% Solaris partion)
exit to save changes

Now do…

prtvtoc /dev/rdsk/c8t0d0s0 | fmthard -s - /dev/rdsk/c8t1d0s0

(notice the “rdsk”, it’s not “dsk”, and it includes the slices.  If you installed on a full drive, I don’t think you see the “s0″ slices)

zpool attach -f rpool c8t0d0s0 c8t1d0s0

And last setup grub on the second disk…

installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c8t1d0s0

All done!  Make note that you use “rdsk” on the prtvtoc command.  I kept getting “must be raw device” errors because I missed the “r” and had only /dev/dsk in the drive path.  Also, on first attempts, I somehow messed up my second drive partitions and labels and it was EFI labeled.  Nothing worked when it was EFI, and I couldn’t remove it either.  Since I did this initially in a VMware guest,  I just deleted the disk and added a new one.  But that’s important to note, do not do anything to that drive except run “format” and then use the “fdisk” command on the new drive.  It will come up and say something like you need a 100% Solaris partition, just say Y and exit that to save.  THEN… you can do the prtvtoc thing to copy the partition table to the new drive.

This is just another one of those things that didn’t make any sense and only partially does now. At least NOW I know there is more at play here than the simple solutions in Samba using create mask and create directory mask. In Linux, that’s how I would get around the issues of Windows directory permissions running on a Linux SMB share.

Now, I am learning to do things the OpenSolaris way. I am loving OpenSolaris and ZFS! However, coming from a Linux and Windows “way of life”, there are some differences that just aren’t clear. What kills me is, I try the RTFM thing, and somehow completely miss that one little thing that makes it all work. Off topic, but an example, coming from Linux, I would just type “su” and get root access. In OpenSolaris, that won’t work. Neither will “pfexec su”, nor “sudo su”. Then one day, after dealing with it for a week or so, I stumble upon a post where someone in an unrelated sample script typed “pfexec su – root”. There ya go! Argh!

Anyway, back on the ZFS/CIFS/ACL thing. It was driving me nuts that I couldn’t figure it out. I wanted a folder with this setup:
/pool/sharefs – owner:greg – group:domusers
greg and domusers should have full control and all folders under “sharefs” should inherit that.

So under linux/samba, that’s where I would do like “create mask = 770″ or simlar, and “force create group = domusers”. Something like that, can’t remember exactly. made it simple actually. It always wrote files with the right perms and ownership and other people in that group could read/write just fine.

Problem is, you can’t get very specific about who get’s what, where, and you can’t use more than one group. Well, sure enough, there’s a thing called “ACL” that handles that stuff now. It’s been around for a while now, but I never even heard of it until I started using OpenSolaris. I like how it seems to be more compatible with the way Windows handles ACL’s. What I don’t like is, it’s confusing. I get the NTFS/Share perms in Windows, been doing that a long time now. The CIFS/ZFS ACL thing kind of makes sense, and it will “click” at some point the more I use it.

After spending hours on this, I reached a point where I had to figure it out. Here’s what I did.

On the ZFS file system, create it normally for SMB access. Then I changed some properties for aclinherit and aclmode. Change those to “passthrough”:
zfs set -o aclinherit=passthrough -o aclmode=passthrough pool/sharefs

Then chmod/chown. OH! That’s another thing. You need to use /bin/chmod and /bin/ls! Not just type: chmod … That wont work. In OpenSolaris the default path points to /usr/gnu/bin/chmod, which doesn’t have the “A” or “V” options to set/view ACL’s. That was another thing that DROVE ME CRAZY!!! I read the man pages and manuals and docs online and I didn’t catch anything that said, “Hey, there are different versions of chmod and ls here!” I can’t believe the time wasting here! Back to the point, do this to put your own default perms on:

/bin/chmod 2774 /pool/sharefs
(I actually am not positive that is needed, but I think it set group as inheritable)

/bin/chmod -R A- /pool/sharefs
(that will wipe out the current perms)

/bin/chmod -R A=owner@:full_set:fd:allow /pool/sharefs
(resets perms with only that acl)

/bin/chmod -R A+group@:full_set:fd:allow /pool/sharefs
(that appends the group perms, full control)

/bin/chmod -R A+everyone@:read_set:fd:allow /pool/sharefs
(above appends everyone read access)

In all the above that will preset INHERITABLE permissions for the subdirectories.  Notice above there is one with “A=” on it?  That will reset the perms and set only that perm.  So I guess you may not even need the previous line for “A-” to reset.  (I am just learning here ya know!)

It looks as if that makes a little sense now.  You can view the current ACL’s like so:  ”/bin/ls -V /pool/sharefs”

In my case, I might want to add another user or group:

/bin/chmod -R A+user:stacy:full_set:fd:allow /pool/sharefs
/bin/chmod -R A+group:othergroup:full_set:fd:allow /pool/sharefs
/bin/chmod -R A+group:yetanothergroup:read_set:fd:allow /pool/sharefs

So with this setup I can now open the share on the server and create a file or folder with inherited permissions.  It does, however, save my username as a new owner, so keep that in mind.  But if the group stays in there with “domusers” as full read/write access, I am happy.

Well, now I get it just a little and it makes more sense compared to Windows ACL’s.  I didn’t go over any share specifics and authentication issues, this was just ACL’s!  I still have to RTFM my way around that for a while.  Next project, join OpenSolaris to a Windows domain.  (Which, BTW, does not work in NT Domain style connections, you have to use Active Directory.)