I’ve started using zxfer that @AllanJude referred me to recently. It does a nice job. My main difficulty was how to get it to work efficiently over the 10Mbps that’s my effective DSL speed.
First, I made a copy of zxfer (zmxfer) that incorporates mbuffer. This is a crude hack, but helps me ensure that I’m getting around the mysterious hanging transmits I have previously seen sending zfs to zfs. Mbuffer seems to smooth this out well.
$LZFS send -i "$copyprev" "$copysrc" \| \ /usr/local/bin/mbuffer -q -s 128k -m 128M \ | /usr/local/bin/mbuffer -q -s 128k -m 128M \ | $RZFS receive $option_F "$copydest" \ || { echo "Error when zfs send/receiving."; beep; exit 1; }
My off-site transfer script ssh’s to the primary backup server, queries a list of zfs filesystems to replicate and copies that back:
#~/bin/bash CMDLIST=/tmp/zxfer_cmds.txt XFPRE=/tmp/zxfer_batch_ SK=.ssh/backup_dsa rm -f /tmp/zxfer_cmds* if [ `ls /tmp/xfer-* 2>/dev/null | wc -l` -gt 0 ] ; then echo "Previous transfer in progress, bye." exit 1 fi ssh -i $SK juno ./mk_fs_list.sh || \ ( echo "Crap, didn't generate file-system list, bye."; exit 1 ) scp -i $SK juno:/tmp/vol_list /tmp || \ ( echo "Crap, didn't copy file-system list, bye."; exit 1 )
We need to turn that list of filesystems into actual transfer commands. I create a file that full of the commands to execute later:
while read FS ; do [ -z "$FS" ] && continue; PFS=`dirname $FS` if [ "$PFS" == "." ] ; then PFS=tank else PFS="tank/$PFS" fi echo "[ ! -f /tmp/stop-xfer ] && sudo zmxfer -dFPsv \ -O \"-i .ssh/backup_dsa ctbu@juno sudo \" \ -N tank/$FS $PFS" done < /tmp/vol_list > $CMDLIST
You might think, “what a lot of sudo!” It’s good practice. I have dedicated a backup user to do this instead of root. I’ve configured the necessary sudoers file entries to make this work.
TIP: disable requiretty in sudoers [S.O.]
We want to increase the parallelism of these zfs transfers as much as possible. The time it takes to transfer zero-length snapshots in serial is prohibitive.
L=`wc -l < $CMDLIST` Q=$[ $[ $L + 8 ] / 8 ] split -l $Q $CMDLIST $XFPRE
Now we run these in screen, partly because ssh and sudo and mbuffer all tend to get a bit grouchy if they can’t agree on if the really need a tty or not…and mostly because I want to keep tabs on where any transfer hangups are. This keeps script output collated. First we test for and fire up a detached screen as necessary:
screen -ls xfer | fgrep -q '.xfer' || screen -dmS xfer sleep 1
And then we fill the screen with some commands. (We need to have a .screenrc that defines eight screens.)
i=0 for x in $XFPRE* ; do echo "rm /tmp/xfer-$i" >> $x cmd="touch /tmp/xfer-$i" screen -S xfer -p$i -X stuff $"$cmd\n" screen -S xfer -p$i -X stuff $"time bash -x $x\n" i=$[ $i + 1 ] done
Once this pxfer-lists.sh script of mine is run, you can connect to the screen using:
screen -S xfer -x
And watch the scripts do their stuff. (That stuff command is actually a true screen directive: stuff $crap into terminal $p.)
I’ve been able to get my transfer time down from 140 minutes to about 14 minutes. Also many of the backups I started transferring I figured out how to reduce in scope by stopping hourly snapshots on file systems that don’t require them.