MegaRAID ZFS
Sep 6, 2016

ZFS has been available on Linux for quite some time but this is the first time I’ve used it on Linux. Unfortunately the setup I plan to implement is not ideal as I am using a RAID controller and not an HBA adapter.

HBA Adapters (no affiliate links)

  1. LSI 9211-8i (needs to be flashed to IT mode)
  2. LSI 9207-8i
  3. LSI 9311-8i
    • I cannot find a place to buy this other than eBay.

MegaRAID Configuration

This step can be skipped if your HBA adapter supports JBOD out of the box.

This neat trick creates a RAID0 with a single disk per array. This will allow you to create a zpool across all of the disks instead of a “single disk” zpool.

  1. Delete the old configuration.
    • megacli -CfgLdDel L0 -a0
  2. Create the RAID0 arrays.
    • megacli -CfgEachDskRaid0 -a0

or

It is possible to create a JBOD with certain MegaRAID controllers. This is still not ideal since it will not be as reliable as an HBA adapter.

I had initially configured each disk as a single disk RAID0 and it did manage to find the pool after converting to a “real” JBOD but it seems like the data was corrupted.

  1. Delete the old configuration.
    • megacli -CfgLdDel L0 -a0
  2. Enable JBOD.
    • megacli AdpSetProp EnableJBOD 1 -a0
  3. Mark the drives as GOOD (may return an error if they are already GOOD).
    • megacli -PDMakeGood -PhysDrv[252:0,252:1,252:2,252:3] -Force -a0
  4. Create the JBOD.
    • megacli -PDMakeJBOD -PhysDrv[252:0,252:1,252:2,252:3] -a0

It’s probably a good idea to reboot now.

ZFS

I have 4 disks connected to the RAID controller (originally a RAID6), so I’ll be creating a raidz2 pool which leaves me with the same amount of available space.

  1. Find the device IDs for your new RAID0 devices.
    • ls -l /dev/disk/by-id
  2. Create the zpool. If you have advanced format drives (4096 byte sectors vs. 512 byte sectors) prepend the -o ashift=12 argument to the zpool name. 
    • zpool create -f -m /mnt/local local raidz2 scsi-3600605b005142c301f61e9920e0b1492 scsi-3600605b005142c301f61e9930e156757 scsi-3600605b005142c301f61e9940e208ddd scsi-3600605b005142c301f61e9940e2b5578
  3. Turn on relatime
    • zfs set relatime=on local
  4. Turn on lz4 compression (lz4 is extremely fast).
    • zfs set compression=lz4 local
  5. Enable ZFS automatic start. 
    • systemctl enable zfs.target
systemctl start zfs.target

If you did everything correctly you should see that the zpool is mounted and “ONLINE”.

local on /mnt/local type zfs (rw,relatime,xattr,noacl)

zpool status local

pool: local
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Sep  6 21:00:33 2016
config:

  NAME                                        STATE     READ WRITE CKSUM
  local                                       ONLINE       0     0     0
	raidz2-0                                  ONLINE       0     0     0
	  scsi-3600605b005142c301f61e9920e0b1492  ONLINE       0     0     0
	  scsi-3600605b005142c301f61e9930e156757  ONLINE       0     0     0
	  scsi-3600605b005142c301f61e9940e208ddd  ONLINE       0     0     0
	  scsi-3600605b005142c301f61e9940e2b5578  ONLINE       0     0     0

Scrubs

Data scrubbing is very important! Scrubbing checks the integrity of data blocks and replaces them with good copies if errors are found. You should do this at least once per month. The cronjob below will run once per week; which is probably a bit unnecessary.

  1. crontab -e
    • 0 0 * * 0 zpool scrub local

Creating zvols

zvols are similar to logical volumes (or lvs) and can be used for QEMU/KVM.

zfs create -V 20G local/NAME

Disk /dev/zd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 8192 bytes
I/O size (minimum/optimal): 8192 bytes / 8192 bytes

Creating Datasets

Datasets are similar to directories except with far more configurable options (quotas, individual properties like: compression options, etc).

zfs create local/NAME

Deleting a Pool

Do not do this unless you want to destroy all of your data!

zpool remove NAME
linux Related
Comments