ZFS on Linux machine

Beavertaill Cactus [Wikipedia]

Beavertaill Cactus [Wikipedia]

Here is my ZFS on Linux story, and some of you might have seen these pictures when I started this project last year: I recycled an old Athlon system from work and placed an 35 Watt AMD A2 processor with 8GB 1600 ram on an Asus mobo in it. I name my home systems after after cacti, and after I installed Ubuntu 12.04 on it, I named this one Beavertail.bitratchet.net.

My previous experiences with storage systems involved them dying from heat. So I decided I would avoid full sized drives and stick with laptop drives and boot off an SSD. I have the SSD partitioned with a /boot, root, and two more partitions for ZIL and L2ARC. The bulk of the storage is a mix of 750GB Hitachi and 500GB Toshiba laptop hard drives, 16 total.  I have lost two drives in this system, which I would label “normal drive attrition.” Boot drive is a 128GB OCZ Vertex 2.


Half the drives are on the bottom, and half are on top. At work I have access to gobs of full-height card brackets and this is what I built drive cages out of.

To get all the drives wired up, I started with a bunch of 1x and 2x PCIe sata expanders and used up all my mobo sata ports, but by the time I got to about 12 drives, I only had a PCI slot left, so had to use that. When looking at my disk utilization in iostat -Nx and dstat --disk-util it was plainly clear that I had a swath of drives underperforming and they were all connected to the slowest PCI controller.

Supermicro HBA 8-port SAS controllers

Supermicro HBA 8-port SAS controllers

I saved up and remedied that by purchasing two SuperMicro SAS HBA’s with Marvel chipsets. They are only 3G SATA (equivalent) but they each control eight drives, and they do so consistently. They take 8x PCIe lanes, and that’s great use for the two 16x PCIe slots on the mobo.

02:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev c3)
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

It took me a while to find out my network bandwidth issues. The problem was my motherboard: it has an onboard Realtek chipset. It would max out at 500Mbps download and 250Mbps upload…and very often wedge the system. I got a PCIe 1x Instell card and I got a good clean 955Mbps both ways out of that with one iperf stream, and 985+Mpbs with two iperf streams. To actually achieve this, I needed to put an Intel nic in my workstation as well. (My switch is a 16-port unmanaged Zyxel).

picture of drives

Eight drives on top

I am able to push close to full network capacity to Beavertail. As you can see, the results speak for themselves: the screenie below shows iftop displaying better than 880Mps and I saw it grab 910Mbps during this backup. Clearly part of the success is having a Samsung 840EVO in my laptop, but having a stripe of four zvols clearly allows plenty of IO headroom.

screen capture of iftop

910Mbps transfer from laptop to NAS.


Here are some other nerdy stats, mostly on how my drives are arranged:

 > zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 4h43m with 0 errors on Sat Sep  6 00:13:22 2014

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9PDPC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9SBBC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G6GMGC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G95REC  ONLINE       0     0     0
          raidz1-1                                      ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9LH9C  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G95JPC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G6LUDC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G5PXYC  ONLINE       0     0     0
          raidz1-2                                      ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_X3EJSVUOS            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_X3EJSVUNS            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT11T            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT17T            ONLINE       0     0     0
          raidz1-3                                      ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT12T            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT13T            ONLINE       0     0     2
            ata-TOSHIBA_MQ01ABD050_933PTT14T            ONLINE       0     0     2
            ata-TOSHIBA_MQ01ABD050_933PTT0ZT            ONLINE       0     0     0
          ata-OCZ-AGILITY4_OCZ-77Z13FI634825PNW-part5   ONLINE       0     0     0
          ata-OCZ-AGILITY4_OCZ-77Z13FI634825PNW-part6   ONLINE       0     0     0

errors: No known data errors

And to finish up, this system has withstood a series of in-place Ubuntu upgrades. It is now running 14.04. My advice on this, and Linux kernels is this:

  • Do not rush to install new mainline kernels, you have to wait for dkms and spl libraries to sync up with mainline and to send out PPA updates through the ubuntu-zfs ppa.
  • If you do a dist-upgrade and reboot, and your zpool does not return on reboot, this is easily fixed by doing a ubuntu-zfs reinstall: apt-get install --reinstall ubuntu-zfs. This will re-link your kernel modules and you should be good to go.

Like I said, this has been working for four releases of Ubuntu for me, along with replacing controllers anbd drives. My only complaint is that doing sequences of small file operations in it tends to bring the speed down a lot (or has, have not recreated on 14.04 yet). But for streaming large files, I get massive throughput…which is great for my large photo collection!

How Low-Income Commuters View Cycling – CityLab

This is where our society needs to seriously loose the brainwashing:

Even increasingly popular car-sharing was not satisfying for respondents; when asked if they would “rather share a car through a program like Zipcar or Car2Go than own my own vehicle,” 35 percent strongly disagreed. Also, 32 percent strongly disagreed with “I want a lifestyle where I don’t need to own a car.”

How Low-Income Commuters View Cycling – CityLab.

When will we get true civilian Internet? — “Internet’s Own Boy” Briefly Knocked Off YouTube With Bogus DMCA Claim – Slashdot

This is why we need copyright reform, and we need to invest in truly civilian Internet spaces. Things like youtube are entirely taken for granted but they live at nothing more than the whim of corporations, with no actual rights of free speech on them.

The whole Google and “right to be forgotten” drama is another case where civilian government is unable to form a basis for a search engine for a search engine ruled by civilian law. We act as if Google is a utility…and while it might be a de-facto utility, it is not, and it is ephemeral and as temporary as its stock price lives well.

“Internet’s Own Boy” Briefly Knocked Off YouTube With Bogus DMCA Claim – Slashdot.

Why Bikes Make Smart People Say Dumb Things — Medium

This so telling about our society:

We’ve been conditioned since infancy to ignore most of these fatalities, along with the behaviors that cause them. If you’re a typical American, your first experience of speeding was while strapped into a car seat, and you rode past half a dozen fatal accident scenes before speaking your first complete sentence.


Keep that Laptop from Racing thru Battery

Linux and Laptops: a long, old story. However, one that with a bit homework, helps get the most out of your battery. Using powertop, you can see your energy usage profile and your device power settings. The downside to powertop is that you cannot "export as shell script." (Seriously, y u no export?) To get these power savings on boot, I wrote my own script.

  1 #!/bin/bash
  2 /usr/sbin/rfkill block 0 #bt
  3 /usr/sbin/rfkill block 1 #wifi
  4 /usr/sbin/rfkill block 2 #bt
  5 for i in /sys/class/scsi_host/host*/link_power_management_policy
  6 do
  7     echo "min_power" > $i
  8 done
  9 echo 0 > /proc/sys/kernel/nmi_watchdog
 10 echo 1 > /sys/module/snd_hda_intel/parameters/power_save
 11 echo 60000 > /proc/sys/vm/dirty_writeback_centisecs
 13 find /sys/devices/ \
 14 | grep 'power/control' \
 15 | while read i
 16 do
 17     echo -n "$i@"
 18     cat $i 2>/dev/nulll
 19 done \
 20 | grep -v auto \
 21 | awk -F@ '{print $1}' > /tmp/sleepy.tmp
 23 for i in `cat /tmp/sleepy.tmp`
 24 do
 25     echo "auto" > $i
 26 done
 28 for i in `find /sys -name autosuspend -exec echo {} \;`
 29 do
 30     echo "1" > $i
 31 done
 33 find /sys/devices/pci0000:00 -type f -iname autosuspend_delay_ms \
 34    -exec echo "1000" > {} \;
 36 for i in /sys/bus/usb/devices/usb*/power/autosuspend_delay_ms
 37 do
 38     echo "1000" > $i
 39 done
 41 for i in /sys/bus/usb/devices/usb*/power/autosuspend
 42 do
 43     echo "1" > $i
 44 done
 46 for i in /sys/bus/usb/devices/*/power/autosuspend
 47 do
 48     echo "1" > $i
 49 done
 51 webcamDev=$( lsusb | grep -i webcam | awk '{print $2 "-" $4 }' | tr -d ':0' )
 52 if [ ! -z "$webcamDev" ]
 53 then
 54     echo 1 > /sys/bus/usb/devices/$webcamDev/power/autosuspend
 55     echo auto > /sys/bus/usb/devices/$webcamDev/power/level
 56     echo 1000 > /sys/bus/usb/devices/$webcamDev/power/autosuspend_delay_ms
 57 fi
 59 # eof

The best way to run this script is to open two windows, one running powertop, the other for this script. After adjusting this script, type r in the powertop window to refresh the device stats. I was able to take my T60 from 30.1W down to somewhere between 19.8 – 16.1W. Unfortunately, I haven’t seen that 16.1W reading since.

I hope this serves as a good starting point for your own laptop power savings! And not just laptops, but this script can also be used for home NAS or media center or other small servers where you need thermal management, too.

Thoughts on my GSII

Having a mobile phone is kind of like holding an open beer and not sipping. However, if I wanted to take a sip out of this phone, it would repeatedly place the pull tab over the mouth of the can, the brew would slosh up when the can was not being held, the beer would mysteriously pause after the first tip, then it would run backwards back up into the can and fall forward again. Repeatedly.

The screen on this phone seems always to think some of the lower buttons are getting tapped. This might be due to the case I put on the phone, but I did my best to carve a bunch off. Google voice jumps up most annoyingly as soon as I start to do something.

And battery life is atrocious. I it seems like the actual phone life is closer to 100 minutes judging from how quickly I see the battery tick down from 100% even if I have wifi and mobile data off. Checking email on it is frustrating and it brings to mind how much I have to refine my gmail filters because I do not want to see donation requests on my phone, that seems like a complete waste of my battery, data and attention.

Having a flip-phone was actually more relaxing.