Thursday, November 27, 2014

Hot to set up pfSense software raid in 2.1.5-RELEASE (amd64)

Here is an example of how to create a software RAID1 in pfSense 2.1.5.

I created a lab in Virtualbox with two 8GB thin provisioned disks and installed pfSense. "pfsense.vdi "and "pfsense2.vdi"

During the install, I chose "1," to boot with the default settings.
The initialization screen defaults to the LiveCD installer. Skip that and press "I" to install directly.
Accept the default settings for the Video and Keymap
Then we want to choose "Setup GEOM Mirror"
Confirm the selection
Now we choose the Primary disk and press enter
Choose the Mirror disk and press enter.
Verify no errors exist. Press Enter.
Choose the Custom Install
Then we choose the mirror/pfSenseMirror we just created.
Format the disk
Use the default disk geometry (just tab to "Use this Geometry")
Format the mirror/pfSenseMirror
Choose Partition Disk
Accept and Create the default settings
Choose "Yes, partition mirror/pfSenseMirror"
Press "OK"
Now, we want to uncheck "Install Bootblock" and make sure "Packet mode" is unchecked as well.
Accept and install
Press "OK"
Choose the default partition slice.
Confirm "OK", then "OK" again
Choose the defaults for the subpartitions (tab to "Accept and Create")
Once the install writes to the mirror, choose "Symmetric multiprocessing kernel", unless you are creating a headless RS232 serial-only interface.
Eject the virtual CD and Reboot.
Once the system reboots, configure pfSense like normal.

We now have a RAID1 mirror of the disks. We can now test booting by removing either of the virtual disks and booting pfSense. In the lab, I've removed the primary disk "pfsense.vdi" and it's booting off the mirror "pfsense2.vdi"

pfSense lacks notification (by default) on a degraded RAID mirror. You can manually check the status of the disk health by going into the console and typing "gmirror status". You can also see the status of the mirror when I shut down the VM (at about the 58 second mark):
GEOM_MIRROR: Device pfSenseMirror destroyed.

To mimic rebuilding a disk, in my lab I created a new volume called "pfsense3.vdi" and made it a blank 8GB, thin provisioned disk to match what I was replacing.

To rebuild the disk, I first checked the status of the disks, "gmirror status"
I destroyed the mirror "gmirror forget pfSenseMirror"
Now, "gmirror status" shows COMPLETE (with just one disk, ad0)
I looked at which disks were present, "atacontrol list" and saw ad1 available and not part of the mirror. This is the new blank disk we want to become part of the mirror.
Inserted it into the mirror with "gmirror insert pfSenseMirror /dev/ad1"
It will start rebuilding. We can check the sync status with "gmirror status" again.
Once complete, you will get the message:
GEOM_MIRROR: Device pfSenseMirror: rebuilding provider ad1 finished.
This will take some time in a normal install. This VM was installed over a SSD on a blank install. Expect some time for it to synchronize.
"gmirror status" should now show us both ado and ad1 as ACTIVE