Category commuting

Build OpenZFS on Ubuntu 16.04 from git

I have to import a zpool from one Ubuntu workstation with recent features to a new Ubuntu workstation. The new workstation being a fresh Ubuntu 16.04 Server install. It only has ubuntu-mate-desktop and build-essentials installed. Below is an aggregation of some of the apt install commands I preformed to get things going:

apt install dkms
apt install automake autoconf
apt install uuid-dev
sudo apt install libblkid-dev
sudo apt install -y libattr1-dev
apt install libnvpair1linux

This should get to to the point where you can do these commands:

$ git clone https://github.com/zfsonlinux/spl
$ ./configure && make -j13 && make install
$ git clone https://github.com/zfsonlinux/zfs
$ ./configure && make -j13 && make install
$ update-grub2

Emulating Web Browsers

Here’s a tedious task for you to consider. When you log-in to your favorite website these days, you’re creating a JSON document and posting it to the service you’re logging into. Long gone are the days when you just posted simple form parameters from a whole post. I work on an emulation platform: one of our features is to use Perl to emulate hundreds of users logging into a captive portal. This requires an economy of memory and time: creating hundreds of “firefox -p ~/.cache/firefox/xzf30d.userprofile” profile sessions is clearly not:

  1. memory efficient
  2. bound to specific network interfaces
  3. time efficient
  4. scriptable

So we use Perl. This requires reading through the F12 -> Networking tab of your browser’s debugging window and emulating the AJAX post to login. Fun once. Wouldn’t want to live there.

customer-tedium

Adding a bicycle bell

The bike bell I had was way to narrow for the pipe on my bike. So luckily I had a small scrap of pipe that I sanded down and pressed into hole of the larger pipe.

image

image

image

image

This was my Hack Your World moment.  I drilled a hole in the handle of the furniture clamp and tightened it with a screw driver as a cheater bar.

image

image

Ubuntu 15.10 and ZFS

Screenshot-root@cholla:~

Some quick thots on doing this for my workstation:

  1. I have six 2TB drives in raid 10 zfs pool, and they would not import to 15.10 because 15.10 ships with (or tries to) zfs 0.6.4.2
  2. I decided on /boot, swap, /, mdadm partitions for OS install
  3. needed to do 15.10 server cmdline install for mdadm raid setup
  4. glad to not have attempted zfs-on-root for this distro
  5. setup three extra partitions on my two 120GB SSDS, using them for
    1. tank zil
    2. tank l2arc
    3. home pool (second pool named homer :-)
  6. Do not attempt to use PPA ubuntu/zfs-stable anymore, 15.10 will not accept it and it WILL mess with your zfsutils-linux recommended install.
  7. Somehow it did end up installing zfs-fuse. Somehow trying to install spl-dkms and zfs-dkms and uninstalling zfsutils-linux apt-get chose it. Why?
  8. I purged zfsutils, zfs/spl-dkms and did git clones on github/zfsonlinux/{spl,zfs}
  9. All of this required starting off with build-essential, autotools, automake, auto… and libuuid and … stuff. Not difficult to chase down.
  10. ./autoconfig.sh, ./configure && make -j10 && make install for spl and zfs
  11. updated /etc/rc.local to modprobe spl zfs, zpool import tank; zpool import homer; zfs mount tank ; zfs mount homer

I am able to reboot and import without pool version warnings.

Why did I move off 14.04.x? I really want to do video editing for kid videos and all the video packages for 14.04 are way ancient.

Also:

  1. get first server install working
  2. install lubuntu-desktop
  3. replace /etc/default/grub hidden -> false
  4. default/grub -> replace “splash quiet” with “nofb”
  5. once LXDE displays, then I do a “apt-get install mate-desktop-*” which seems to work just fine.
  6. Why? lubuntu-desktop flawlessly sets up Xorg dependencies and gives me a desktop the first time without messing around wondering why mate-desktop didn’t.

Merry Xmas!

Network Cards Suck

At least, on consumer motherboards.  I have a 4ghz Haswell on a Asus z97 mobo. Wanted to reboot it to check on fan profiles.  Realized it was not letting me into uefi BIOS and when it would get there my mouse was all locked up. Now I’m in a situation where I need to make sure my workstation actually works correctly.   Cold boot doesn’t help. Different BIOS key F2 doesn’t help. Pull power cord and pull bios power jumper. Didn’t help. Finally pulled the two Intel i840 cards.

Bounced into the BIOS straight away and nothing was amiss.

Anyhow, I think the system should run cooler now.

ZFS on Linux machine

Beavertaill Cactus [Wikipedia]

Beavertaill Cactus [Wikipedia]


Here is my ZFS on Linux story, and some of you might have seen these pictures when I started this project last year: I recycled an old Athlon system from work and placed an 35 Watt AMD A2 processor with 8GB 1600 ram on an Asus mobo in it. I name my home systems after after cacti, and after I installed Ubuntu 12.04 on it, I named this one Beavertail.bitratchet.net.

My previous experiences with storage systems involved them dying from heat. So I decided I would avoid full sized drives and stick with laptop drives and boot off an SSD. I have the SSD partitioned with a /boot, root, and two more partitions for ZIL and L2ARC. The bulk of the storage is a mix of 750GB Hitachi and 500GB Toshiba laptop hard drives, 16 total.  I have lost two drives in this system, which I would label “normal drive attrition.” Boot drive is a 128GB OCZ Vertex 2.

image

Half the drives are on the bottom, and half are on top. At work I have access to gobs of full-height card brackets and this is what I built drive cages out of.

To get all the drives wired up, I started with a bunch of 1x and 2x PCIe sata expanders and used up all my mobo sata ports, but by the time I got to about 12 drives, I only had a PCI slot left, so had to use that. When looking at my disk utilization in iostat -Nx and dstat --disk-util it was plainly clear that I had a swath of drives underperforming and they were all connected to the slowest PCI controller.

Supermicro HBA 8-port SAS controllers

Supermicro HBA 8-port SAS controllers

I saved up and remedied that by purchasing two SuperMicro SAS HBA’s with Marvel chipsets. They are only 3G SATA (equivalent) but they each control eight drives, and they do so consistently. They take 8x PCIe lanes, and that’s great use for the two 16x PCIe slots on the mobo.

02:00.0 RAID bus controller: Marvell Technology Group Ltd. 88SE9485 SAS/SATA 6Gb/s controller (rev c3)
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

It took me a while to find out my network bandwidth issues. The problem was my motherboard: it has an onboard Realtek chipset. It would max out at 500Mbps download and 250Mbps upload…and very often wedge the system. I got a PCIe 1x Instell card and I got a good clean 955Mbps both ways out of that with one iperf stream, and 985+Mpbs with two iperf streams. To actually achieve this, I needed to put an Intel nic in my workstation as well. (My switch is a 16-port unmanaged Zyxel).

picture of drives

Eight drives on top

I am able to push close to full network capacity to Beavertail. As you can see, the results speak for themselves: the screenie below shows iftop displaying better than 880Mps and I saw it grab 910Mbps during this backup. Clearly part of the success is having a Samsung 840EVO in my laptop, but having a stripe of four zvols clearly allows plenty of IO headroom.

screen capture of iftop

910Mbps transfer from laptop to NAS.

 

Here are some other nerdy stats, mostly on how my drives are arranged:

 > zpool status -v
  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 4h43m with 0 errors on Sat Sep  6 00:13:22 2014
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9PDPC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9SBBC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G6GMGC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G95REC  ONLINE       0     0     0
          raidz1-1                                      ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G9LH9C  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G95JPC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G6LUDC  ONLINE       0     0     0
            ata-Hitachi_HTS547575A9E384_J2190059G5PXYC  ONLINE       0     0     0
          raidz1-2                                      ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_X3EJSVUOS            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_X3EJSVUNS            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT11T            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT17T            ONLINE       0     0     0
          raidz1-3                                      ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT12T            ONLINE       0     0     0
            ata-TOSHIBA_MQ01ABD050_933PTT13T            ONLINE       0     0     2
            ata-TOSHIBA_MQ01ABD050_933PTT14T            ONLINE       0     0     2
            ata-TOSHIBA_MQ01ABD050_933PTT0ZT            ONLINE       0     0     0
        logs
          ata-OCZ-AGILITY4_OCZ-77Z13FI634825PNW-part5   ONLINE       0     0     0
        cache
          ata-OCZ-AGILITY4_OCZ-77Z13FI634825PNW-part6   ONLINE       0     0     0

errors: No known data errors

And to finish up, this system has withstood a series of in-place Ubuntu upgrades. It is now running 14.04. My advice on this, and Linux kernels is this:

  • Do not rush to install new mainline kernels, you have to wait for dkms and spl libraries to sync up with mainline and to send out PPA updates through the ubuntu-zfs ppa.
  • If you do a dist-upgrade and reboot, and your zpool does not return on reboot, this is easily fixed by doing a ubuntu-zfs reinstall: apt-get install --reinstall ubuntu-zfs. This will re-link your kernel modules and you should be good to go.

Like I said, this has been working for four releases of Ubuntu for me, along with replacing controllers anbd drives. My only complaint is that doing sequences of small file operations in it tends to bring the speed down a lot (or has, have not recreated on 14.04 yet). But for streaming large files, I get massive throughput…which is great for my large photo collection!

How Low-Income Commuters View Cycling – CityLab

This is where our society needs to seriously loose the brainwashing:

Even increasingly popular car-sharing was not satisfying for respondents; when asked if they would “rather share a car through a program like Zipcar or Car2Go than own my own vehicle,” 35 percent strongly disagreed. Also, 32 percent strongly disagreed with “I want a lifestyle where I don’t need to own a car.”

How Low-Income Commuters View Cycling – CityLab.