Qwest / Centurylink DSL Web site timeouts, hangups, or failures while running a firewall or router with static IP addresses.

Recently I setup the Qwest DSL service in my home. It worked great, but after a couple days I started to get hang ups and timeouts or hangups while browsing the web. Speed tests still showed the correct speeds and rebooting all my hardware didn’t solve the issue.

I run PFSense as my internet firewall. (check it out, http://pfsense.org ) I love PFSense, it gives me very fine grained control over just about everything, even simple traffic speed limiters for certain parts of my LAN. (like my kid’s systems, I don’t want them eating my my bandwidth with Youtube!) In the past, I’ve had some issues with my client networks and firewalls while using Qwest DSL and PFSense. I found almost ZERO help on Google searches, which I find surprising, because I can’t be the only one with this setup. But, to get to the point, you need to setup the Qwest modem advanced options to use Dynamic Routing. I use version 2.

If I plug in and connect directly, I do not get the connectivity issues, so I knew something with the PFSense was, to put it mildly, not being cooperative. Setting to Dynamic Routing fixed this issue on 3 client networks AND on my recent install at home.

Note, too, that all these networks have static IP’s or static blocks. I tried setting to use transparent bridging, which didn’t help. But one time I setup PPPoE directly on PFSense, and that did help. In the end, the only way I could reliably run PFSense on Qwest DSL was to disable NAT on the modem, setup Dynamic Routing, and purchase static IP address(s). Keep in mind, you need to use the “Static IP” setup from Qwest and NOT run transparent bridging, like I assumed. Read their docs, there’s a special setup for this in the Quick Setup section of their modem firmware.

Also, I had this issue on most of the later model modems and firmware, but NOT on the oldest Actiontek modems running old firmware. (like the 701′s) But on the newer Qwest firmware (with the blue background and preschool-style-coloring :) I had to enable Dynamic Routing. Also, I have the newer Zyxel Q1000Z modem now, same issue.

I have no understanding of why this happens. It doesn’t make any sense to me. Although, just to throw an idea out there, maybe it has to do with the way the modems manage hops from the external destinations.  From what I read, dynamic routing has something to do with maintaining the hops between routers online.  Maybe, since using the modem with static IP’s basically sets it into bridging mode, it incorrectly maintains that hop information, or at the very least it doesn’t identify itself correctly.  So what may happen is some routers out there get flaky and don’t respond well with your bridged modem by the time they communicate with your firewall.  Some do fine though, which would explain why some sites fail and some don’t.  I don’t believe PFSense is doing any dynamic routing protocol work, at all.  It’s just firewalling my LAN, right?  So all I can assume is, since the Qwest modem is in between me and the rest of the internet, IT has something to do with that communication breakdown using the dynamic routing.  Of course, I really don’t know what I am talking about and making complete assumptions!  But hey, its just an idea. Maybe someone who know’s more than I do can shed some light on it. :)

I have CrashPlan running on a couple FreeBSD servers now, and I thought I’d write up a quick startup rc script.
Here’s what we’ll do. This assumes you’ve installed CrashPlan in the default directory, which requires the “/compat/linux/bin/bash” environment.

1. Create a startup script in /etc/rc.d called “crashplan”. Put the contents below in the script.

#!/bin/sh

load_rc_config ${name}
run_rc_command "${1}"


2. Make it executable:

chmod 755 /etc/rc.d/crashplan


3. Edit your /etc/rc.conf file and add:

crashplan_enable="YES"


Test by running :

/etc/rc.d/crashplan start


If all goes well, when you restart the system it should bring up CrashPlan for you.

In case you haven’t seen it, here’s the install how-to for Crashplan on FreeBSD.

http://www.1stbyte.com/2011/01/26/how-to-install-crashplan-on-freebsd/

This took a while to solve, its not exactly user-friendly.  I have a FreeBSD server running VirtualBox virtual machines in headless mode, and I need to do everything using VBoxManage command line. (CLI)  In one case I wanted to remove the attached DVD ISO image from my machine. They don’t have a “storagedetach” or “removehd” command, you have to use the “storageattach” command, with some extra info.

List the VM info to see the port and device attached.
VBoxManage showvminfo vmMaridia
(My VM name is vmMaridia)

You will see stuff like so:

Storage Controller Name (0): maridiasata
Storage Controller Type (0): IntelAhci
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0): 30
Storage Controller Port Count (0): 30
Storage Controller Name (1): maridiaide
Storage Controller Type (1): PIIX4
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1): 2
Storage Controller Port Count (1): 2
maridiasata (0, 0): /vm/sys/maridia/mir1.vdi (UUID: fd813be8-dd38-4784-a374-d96263aae1ba)
maridiasata (1, 0): /vm/sys/maridia/mir2.vdi (UUID: dab57578-4de5-4bef-91ee-232c56eea671)
maridiasata (3, 0): /vm/sys/maridia/mardia-bu.vdi (UUID: 6ac3c6aa-ae5b-4d57-8f30-24e59d2ef381)
maridiasata (4, 0): /vm/sys/maridia/share-sata.vdi (UUID: d16a4847-95fb-43bd-b8b4-c7d9e38d9946)
maridiaide (0, 0): /vm/iso/sysresc156.iso (UUID: 10cbebd7-e058-4c8e-9487-52e7776d9935)


So, in my system, I wanted to remove from maridiaide controller, the port 0, device 0. Which is the sysresc156.iso.
To remove the DVD:

VBoxManage storageattach vmMaridia --storagectl maridiaide --port 0 --device 0 --medium none


The trick was, you just specify “none” as the medium when you want to remove it. Now when you run “showvminfo” on that VM, you’ll see no drive attached, in fact no maridiaide controller listed. (although, I do think it is registered still?)

Been using FreeBSD 8 and 8.1 for over a year now, love it.  Have root on the ZFS mirror, works great!  However, I’ve struggled to find a good way to set the /boot/loader.conf variables for kmem_size and zfs.arc_max. Everything I tried didn’t seem to work, so I ended up leaving all that out and just let the default “auto-tuning” to its thing.  But whenever I tried to copy large files, like 20+ GB files, it would crash the server.  It would give me that “kmem_map too small” kernel panic.

In addition, I have a GB network and fairly new 500GB drives in my array, which, are not high-end, speedy drives, but should operate fairly well.  I was getting network transfer speeds of about 45MB/s to 55MB/s, varying.  But usually managed to stick to about 50 fairly regularly.  I am not using an scientific method of measuring this, only the Windows 7 transfer window.  So it’s not something I’d rely on, but it was close enough to see *about* what we’re doing.

I finally did some reading, and found that setting my vfs.zfs.arc_max to about one-half my RAM, minus 512MB, and setting vm.kmem_size to 1.5x my RAM, it would stop giving me the kmem_map panic.  I found a wide range of suggestions out there, but in general, there were more suggestions to set the kmem_size higher, and keep the arc_max lower than half the ram.   Also, I found the general thought was you would be best having 8G of RAM in the first place and auto-tuning seemed to work best.

For my server, running 8.1-RELEASE, I have an i3 cpu on an  Intel server board, with 4G RAM.  My ZFS is a mirror on the Root, plus my main storage using 6 drives, 500G  ea. (varying makes), in a RAIDz2 array. So I have no UFS, only ZFS.  Once I set those two parameters, I jumped from about 50MB/s to 72MB/s.  And my 48G transfer test of files moved over no prob.  FINALLY!! I love it!

Here’s my relevant loader.conf variables:

vm.kmem_size_max="8G"
vm.kmem_size="6G"
vfs.zfs.arc_min="1024M"
vfs.zfs.arc_max="1500M"
vfs.zfs.prefetch_disable=0

Also, I have two 16G swap partitions. (I know, overkill, but I wanted to pre-allocate those on my mirror and leave it for when I upgrade the RAM)

I am sure someone out there with a better understanding would recommend better values, but, it works good enough for me and so far its’ stable.

By the way this server also shares lots of media, VirtualBox VM’s, and also some Postgres and Mysql with Apache/PHP.  It’s not under any load, just a few of us access this system, but we do read/write large files from time to time.

I hope this helps others out there!  Good luck!

A how-to for installing Crashplan backup software on FreeBSD, my favorite server software with my favorite backup software.

We owe Kim Scarborough and Aaron Baff for this information, BIG TIME! Thank you, Thank you, Thank you, Kim and Arron for your help in making this possible!

Kims original HOW-TO is here:
http://kim.scarborough.chicago.il.us/do/nerd/tips/crashplan

I have no contact or reference info for Aaron, exept, here’s the original CrashPlan forum thread with our conversations:
https://crashplan.zendesk.com/entries/112498

All credit for this really belongs to Arron and Kim! I only wanted to record my own notes and share my experience here.

Greg’s How-to: How to install Crashplan on FreeBSD!

Note, this install is on a FreeBSD 8.1-RELEASE system. Kim installed on FreeBSD 7. Also, this doesn’t seem to matter for installation purposes, but my system also runs with ZFS on a GPT root mirror, with other ZFS pools on the system as well using RAIDz2. (where I will store my Crashplan data)

Below I mention “install some/port”, which is not intended to be a literal command. I’ll leave the details of installing ports to you, which you can find online on other sites.

1. install emulators/linux_base-f10
Note: I had to find glibc-common online and manually load into distfiles when I first tried to install on 8.1.
Update 3/30/11: As of 3/30, and running on FreeBSD 8.2, I did not need to find glibc manually, it just loaded first try. Maybe the first time I tried the mirrors were unavailable?

1.b Check /etc/rc.conf
After you install the Linux base packages, check /etc/rc.conf that linux is enabled and add it if not.
linux_enable="YES"

2. Add to fstab and mount:
linproc /compat/linux/proc linprocfs rw 0 0

Then…
mount linproc

Update 3/30/11 FreeBSD 8.2: I wasn’t able to mount linproc at this point. I rebooted the system, then it mounted. So, reboot first, then add this to your fstab and make sure it mounts before you reboot.

3. install sysutils/linux-procps

Update 3/30/11 FreeBSD 8.2: In the next steps 3, 4, 5, 6, 7, I needed to add the following to my /etc/make.conf file. Note that the “f10″ is NOT in quotes. Once this was added, these extra ports installed fine.

OVERRIDE_LINUX_BASE_PORT=f10
OVERRIDE_LINUX_NONBASE_PORTS=f10
LINUX_DIST_SUFFIX=f10


4. install /usr/ports/textproc/linux-f10-expat

5. install /usr/ports/x11-fonts/linux-f10-fontconfig

6. install /usr/ports/x11/linux-f10-xorg-libs

7. install /usr/ports/java/linux-sun-jre16
note: you’ll need to manually find and download the jre-6u3 file from sun/oracle.

You should have these installed. (or similar, maybe other versions)

linux-f10-expat-2.0.1 = up-to-date with port
linux-f10-fontconfig-2.6.0 = up-to-date with port
linux-f10-xorg-libs-7.4_1 = up-to-date with port
linux-procps-3.2.5 = up-to-date with port
linux-sun-jre-1.6.0.03 = up-to-date with port
linux_base-f10-10_3 = up-to-date with port

8. change to linux bash, as root:
/compat/linux/bin/bash

Note that this puts you into the linux compat environment. If you go looking for /usr/local/bin/crashplan, it won’t be there unless you are in the linux compat environment first. The actual location is here:
/compat/linux/usr/local/crashplan

9. Install Crashplan.
go to the Crashplan-install folder (where you extracted it) and start the ./install.sh script. Use all defaults and finish the script.

10. Edit run.conf. In the /usr/local/crashplan/bin folder,
edit the run.conf file. There are two long lines defining
Java statup parameters. Add the following to the SVR line.

-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider

That should go within the quotes!

11. Try starting the service:
/usr/local/crashplan/bin/CrashPlanEngine start

You’ll need to start this manually (from within linux bash), and will not start at boot.
(until one of us writes an rc script)
Update 3/30/11: Here’s an rc.d startup script how to for Crashplan:

12. Use the “headless” install connection per the Crashplan FAQ’s online to connect from another Linux/Windows/Mac system to manage the service.

Good luck!!

Here’s my notes for ZFS root installation with FreeBSD 8.1. (amd64 on my system)

Boot up the FreeBSD install disk, you’ll need one with the Fixit system.  In the examples listed, I had 2 disks, ad4 and ad6.  Yours may be different, replace as necessary.

1. Add the modules on boot, hit “6″ at the BTX loader menu, then enter these lines at the “OK” prompt.

load ahci.ko
load opensolaris.ko
load zfs.ko
boot 

2. At the sysinstall menu, go to Fixit.

3. Create the GPT disks and partitions.

• Delete any partions already on the disk, and destroy. (only if needed)  On my disks, I had 2 partitions created from another OS installation.  “-i 2” indicates the partition number.  You may need to add a third or more.

  gpart delete -i 2 ad4
  gpart delete -i 1 ad4
  gpart delete -i 2 ad6
  gpart delete -i 1 ad6
  gpart destroy ad4
  gpart destroy ad6 

• Create the gpt disk.

  gpart create -s gpt ad4
  gpart create -s gpt ad6 

• (NOTE: “gpart show” and “gpart show –l” will display the partitions.
• You may want to layout different partitions, but we’ll just create a boot, swap, and zfs partition here, with GPT labels.  (swap is better left off ZFS for crash dumps)

  gpart add -s 128 -t freebsd-boot -l boot0 ad4
  gpart add -s 128 -t freebsd-boot -l boot1 ad6
  gpart add -s 8388608 -t freebsd-swap -l swap0 ad4
  gpart add -s 8388608 -t freebsd-swap -l swap1 ad6
  gpart add -t freebsd-zfs -l disk0 ad4
  gpart add -t freebsd-zfs -l disk1 ad6
  

• Next, add the bootcode to the disk.

  gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad4
  gpart bootcode -b /dist/boot/pmbr -p /dist/boot/gptzfsboot -i 1 ad6
  

4. OK, now we’ve got some GTP disks setup for a mirror and boot. 
IMPORTANT!!!  Make sure you don’t skip this step, create the /boot/zfs directory for the zpool.cache.

mkdir /boot/zfs

5. Now we can start the ZFS fun! On the FreeBSD wiki and on other sites I used as reference, there may be listed other options and settings for your ZFS layout.  Here, I am going to keep it very simple, but I’ll list out my options later.

Also, you may prefer different names or even separate location/directory/ZFS filesystem for your root pool. My preference is to call the root pool “rpool” and place the FreeBSD system in the ROOT filesystem.  (looks like this: rpool/ROOT)  In addition, my preference is to place “/home” outside of the ROOT fs.  Which allows is to separate system from data and management of snapshots is more flexible. But this is totally a preference thing.  One of the beautiful things about ZFS is the flexibility and ease of use.

• Let’s create the pool. Notice the gpt/disk# items, which correspond to your gpt labels. Also, we’ll disable a mountpoint on rpool.

  zpool create rpool mirror gpt/disk0 gpt/disk1
  zfs set mountpoint=none rpool
  

• Add the ROOT filesystem.  Again, here you may prefer to add all ZFS filesystems for places like /usr, /var, /tmp, and other system locations.  I don’t recommend it, but in this example we’ll keep it all in one filesystem, except /home.

  zfs create -o mountpoint=/mnt rpool/ROOT
  zfs create -o mountpoint=/mnt/home rpool/home
  

  (OPTIONAL file systems)

  zfs create rpool/ROOT/usr
  zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/ports
  zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/distfiles
  zfs create -o compression=off -o exec=off -o setuid=off rpool/ROOT/usr/ports/packages
  zfs create -o compression=lzjb -o setuid=off rpool/ROOT/usr/src
  zfs create rpool/ROOT/var
  zfs create -o exec=on -o setuid=off rpool/ROOT/tmp
  

6. Install the base system.  We’ll be intsalling the system into the /mnt directory for now, but we’ll change those mounts later.

• cd /dist/8.1*
  export DESTDIR=/mnt
  for dir in base catpages dict doc games info lib32 manpages ports; do (cd $dir ; ./install.sh) ; done
  cd src ; ./install.sh all
  cd ../kernels ; ./install.sh generic
  cd /mnt/boot ; cp -Rlp GENERIC/* /mnt/boot/kernel/
  

7. Edit /mnt/boot/loader.conf and add these lines.

ahci_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:rpool/ROOT"

8. Edit /mnt/etc/rc.conf and add these lines. I have re0 as my network interface, your’s may be different.

zfs_enable="YES"
ifconfig_re0="DHCP"
hostname="systemname"

9. Edit /mnt/etc/fstab and add these lines. Setup your swap.

/dev/gpt/swap0 none	swap	sw	0	0
/dev/gpt/swap1 none	swap	sw	0	0

10. Copy the zpool.cache.  IMPORTANT!!!!! Don’t forget this step!!

cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache

11. Export this…

export 	LD_LIBRARY_PATH=/dist/lib

12. Unmount ZFS filesystems and set the correct mountpoints for the new root to boot. If you setup other/optional zfs filesystems, then you’ll want to set their correct mount points, too.

zfs umount -a
zfs set mountpoint=legacy rpool/ROOT
zfs set mountpoint=/home rpool/home

13.  Set the bootfs property of the pool to rpool/ROOT

zpool set bootfs=rpool/ROOT rpool

That’s about it!  Well, actually, you’ll still need to do a lot of post install stuff.  (see the wiki (set passwd, time zone, etc))  This is only a very base load of FreeBSD, root doesn’t even have a password. 

Keep in mind, at the time I wrote this, there is a bug in the zfs boot loader, where you can only boot from the first disk in the mirror.  Kind of reduces the usefulness of a root mirror, that’s for sure.  However, there is a patch that you can compile a new zfs loader with.  There’s a procedure to do so, and I’ll write up a how-to on that sooner than later.

Here are the site’s I used as reference, as well as the forum post on the root mirror boot issue.

http://www.b0rken.org/freebsd/zfs.html

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror

http://forums.freebsd.org/showthread.php?p=95482

Tried to manually setup my disks in order to install root FreeBSD on a ZFS mirror, but following the how-to’s available, I couldn’t get pass the first command.

gpart create -s gpt ad0

geom 'ad0': File exists

I tried deleting the partitions/slices, even doing a “dd of=/dev/null” on my drives.  But I keep getting this file exists error, which is totally less-than-helpful.  Google didn’t give me much help either, but it did lead me to other info which helped.

So… delete the partitions:

gpart delete -i 2 ad0
gpart delete -i 1 ad0

In the above, I only had 2 partitions, so you may need to add a line for “-i 3” if you had a third, fourth, etc.  Also, in my case I was setting up a mirror on ZFS, so I repeated all these for both drives. (ad0 and ad2, for my system)

After that, there was a ONE EXTRA thing that was needed before the “gpart create” command would work.  You need to “destroy” it!

gpart destroy ad0

I was finally able to runt he gpart create normally after that. If you had drives setup previously from another system, this was how I removed any partition info on them.

These are the sites I used as reference.

http://forums.freebsd.org/showthread.php?t=9777

http://www.b0rken.org/freebsd/zfs.html

http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/Mirror

I only need to update this once in a while, but twice now I’ve forgotten what to run after an update to /etc/login.conf.  Run this:

cap_mkdb /etc/login.conf