Crazy Times with zxfer

by

I’ve started using zxfer that @AllanJude referred me to recently. It does a nice job. My main difficulty was how to get it to work efficiently over the 10Mbps that’s my effective DSL speed.

First, I made a copy of zxfer (zmxfer) that incorporates mbuffer. This is a crude hack, but helps me ensure that I’m getting around the mysterious hanging transmits I have previously seen sending zfs to zfs. Mbuffer seems to smooth this out well.

$LZFS send -i "$copyprev" "$copysrc" \| \
/usr/local/bin/mbuffer -q -s 128k -m 128M \
| /usr/local/bin/mbuffer -q -s 128k -m 128M \
| $RZFS receive $option_F "$copydest" \
|| { echo "Error when zfs send/receiving."; beep; exit 1; }

My off-site transfer script ssh’s to the primary backup server, queries a list of zfs filesystems to replicate and copies that back:

#~/bin/bash
CMDLIST=/tmp/zxfer_cmds.txt
XFPRE=/tmp/zxfer_batch_
SK=.ssh/backup_dsa
rm -f /tmp/zxfer_cmds*
if [ `ls /tmp/xfer-* 2>/dev/null | wc -l` -gt 0 ] ; then
   echo "Previous transfer in progress, bye."
   exit 1
fi
ssh -i $SK juno ./mk_fs_list.sh || \
   ( echo "Crap, didn't generate file-system list, bye."; exit 1 )
scp -i $SK juno:/tmp/vol_list /tmp || \
   ( echo "Crap, didn't copy file-system list, bye."; exit 1 )

We need to turn that list of filesystems into actual transfer commands. I create a file that full of the commands to execute later:

while read FS ; do
   [ -z "$FS" ] && continue;
   PFS=`dirname $FS`
   if [ "$PFS" == "." ] ; then 
      PFS=tank
   else
      PFS="tank/$PFS"
   fi
   echo "[ ! -f /tmp/stop-xfer ] && sudo zmxfer -dFPsv \
 -O \"-i .ssh/backup_dsa ctbu@juno sudo \" \
 -N tank/$FS $PFS"
done < /tmp/vol_list > $CMDLIST

You might think, “what a lot of sudo!” It’s good practice. I have dedicated a backup user to do this instead of root. I’ve configured the necessary sudoers file entries to make this work.

TIP: disable requiretty in sudoers [S.O.]

We want to increase the parallelism of these zfs transfers as much as possible. The time it takes to transfer zero-length snapshots in serial is prohibitive.

L=`wc -l < $CMDLIST`
Q=$[ $[ $L + 8 ] / 8 ]
split -l $Q $CMDLIST $XFPRE

Now we run these in screen, partly because ssh and sudo and mbuffer all tend to get a bit grouchy if they can’t agree on if the really need a tty or not…and mostly because I want to keep tabs on where any transfer hangups are. This keeps script output collated. First we test for and fire up a detached screen as necessary:

screen -ls xfer | fgrep -q '.xfer' || screen -dmS xfer
sleep 1

And then we fill the screen with some commands. (We need to have a .screenrc that defines eight screens.)

i=0
for x in $XFPRE* ; do
   echo "rm /tmp/xfer-$i" >> $x
   cmd="touch /tmp/xfer-$i"
   screen -S xfer -p$i -X stuff $"$cmd\n"
   screen -S xfer -p$i -X stuff $"time bash -x $x\n"
   i=$[ $i + 1 ]
done

Once this pxfer-lists.sh script of mine is run, you can connect to the screen using:

screen -S xfer -x

And watch the scripts do their stuff. (That stuff command is actually a true screen directive: stuff $crap into terminal $p.)

I’ve been able to get my transfer time down from 140 minutes to about 14 minutes. Also many of the backups I started transferring I figured out how to reduce in scope by stopping hourly snapshots on file systems that don’t require them.

Advertisements