Robocopy Notes

Install cmder: It’s the nicest shell I’ve seen for windows. Run your console as Administrator. Otherwise you can’t use the /B backup switch. Also rember you need to do a net use command as administrator.

Before you robocopy stuff, setup a dedicated drive letter. The drive letter is only available to the logged in session. So if you have drive p: for Bob, and then you boost your console to Administrator–no more drive p: ! So dont use the users drive mappings: create admin drive mappings.

net use p: \\nas02\backup\ "secret" /user:bob /persistent:yes

Remember to type the password with “double quotes” and not ‘single quotes’. If you type single quotes you may as well be typing capital Xes: they become part of your password.

The net use command to see if you already have a drive share. Close any File Explorer windows open to that server becuase that’s equivalent of have a net use $d /user:anonymous open at the same time, and windows wont cooperate. Mount the directory

There are a lot of switches. We’ll assume a C:\Users directory.

C:\Users\bob> mkdir c:\temp
C:\Users\bob> cd C:\Users
C:\Users> robocopy bob P:\bu-bob\ /mir /ZB /FFT /XA:SH /W:5 /R:2 /dcopy:T ^
 /XJ /XD "Temp*" "cache2" "temporary internet files" "*cache*" /NFL

First try the command without the /LOG switch. The command goes faster with the LOG turned on, do that later. /XF is a pattern to exclude files. Example log option: /LOG:C:\temp\bu.txt. The /NFL will show directories. not files.

The /MT flag is useful, but it prohibits logging, not available on Vista. The /XJ flag should be default, but sadly–no. Juntion points create these really frustrating backup path loops. Use /XJ!

Advertisements

Crazy Times with zxfer

I’ve started using zxfer that @AllanJude referred me to recently. It does a nice job. My main difficulty was how to get it to work efficiently over the 10Mbps that’s my effective DSL speed.

First, I made a copy of zxfer (zmxfer) that incorporates mbuffer. This is a crude hack, but helps me ensure that I’m getting around the mysterious hanging transmits I have previously seen sending zfs to zfs. Mbuffer seems to smooth this out well.

$LZFS send -i "$copyprev" "$copysrc" \| \
/usr/local/bin/mbuffer -q -s 128k -m 128M \
| /usr/local/bin/mbuffer -q -s 128k -m 128M \
| $RZFS receive $option_F "$copydest" \
|| { echo "Error when zfs send/receiving."; beep; exit 1; }

My off-site transfer script ssh’s to the primary backup server, queries a list of zfs filesystems to replicate and copies that back:

#~/bin/bash
CMDLIST=/tmp/zxfer_cmds.txt
XFPRE=/tmp/zxfer_batch_
SK=.ssh/backup_dsa
rm -f /tmp/zxfer_cmds*
if [ `ls /tmp/xfer-* 2>/dev/null | wc -l` -gt 0 ] ; then
   echo "Previous transfer in progress, bye."
   exit 1
fi
ssh -i $SK juno ./mk_fs_list.sh || \
   ( echo "Crap, didn't generate file-system list, bye."; exit 1 )
scp -i $SK juno:/tmp/vol_list /tmp || \
   ( echo "Crap, didn't copy file-system list, bye."; exit 1 )

We need to turn that list of filesystems into actual transfer commands. I create a file that full of the commands to execute later:

while read FS ; do
   [ -z "$FS" ] && continue;
   PFS=`dirname $FS`
   if [ "$PFS" == "." ] ; then 
      PFS=tank
   else
      PFS="tank/$PFS"
   fi
   echo "[ ! -f /tmp/stop-xfer ] && sudo zmxfer -dFPsv \
 -O \"-i .ssh/backup_dsa ctbu@juno sudo \" \
 -N tank/$FS $PFS"
done < /tmp/vol_list > $CMDLIST

You might think, “what a lot of sudo!” It’s good practice. I have dedicated a backup user to do this instead of root. I’ve configured the necessary sudoers file entries to make this work.

TIP: disable requiretty in sudoers [S.O.]

We want to increase the parallelism of these zfs transfers as much as possible. The time it takes to transfer zero-length snapshots in serial is prohibitive.

L=`wc -l < $CMDLIST`
Q=$[ $[ $L + 8 ] / 8 ]
split -l $Q $CMDLIST $XFPRE

Now we run these in screen, partly because ssh and sudo and mbuffer all tend to get a bit grouchy if they can’t agree on if the really need a tty or not…and mostly because I want to keep tabs on where any transfer hangups are. This keeps script output collated. First we test for and fire up a detached screen as necessary:

screen -ls xfer | fgrep -q '.xfer' || screen -dmS xfer
sleep 1

And then we fill the screen with some commands. (We need to have a .screenrc that defines eight screens.)

i=0
for x in $XFPRE* ; do
   echo "rm /tmp/xfer-$i" >> $x
   cmd="touch /tmp/xfer-$i"
   screen -S xfer -p$i -X stuff $"$cmd\n"
   screen -S xfer -p$i -X stuff $"time bash -x $x\n"
   i=$[ $i + 1 ]
done

Once this pxfer-lists.sh script of mine is run, you can connect to the screen using:

screen -S xfer -x

And watch the scripts do their stuff. (That stuff command is actually a true screen directive: stuff $crap into terminal $p.)

I’ve been able to get my transfer time down from 140 minutes to about 14 minutes. Also many of the backups I started transferring I figured out how to reduce in scope by stopping hourly snapshots on file systems that don’t require them.

Time Machine on a network drive

Helping a friend setup time machine backups to his new NAS device. Reasonable howto so far.

Time Machine on a network drive.

Unfortunately, when you see some of the examples of how to get ‘computer name’ and ‘drive name,’ you might not get familysomething.local as your hostname, it might be ‘familysomething.routerbrand.com.’ Why familysomething would gain the home routers brand domain name seems like a travesty. I hate that kind of tinkering with dhcp settings. And also, when you want to ping your nas, chances are it does not register it’s hostname, so you cannot ping the name reported in the Apple Finder.

Backups: Using `find` Across a Panalopy of Directories

Linux Backups logo

Linux Backups

I love using the find command. In DOS, find is like grep. In Linux, find is the most powerful recursive DOS dir /s or Linux ls -r command you could ever put your saddle on.

One of the things you can do with find is to avoid directories, using the -prune switch. Like so:

find /usr/local -type d -a \( -name jre1.6.0_38 -prune -o -type d -print \)

Yeah, put your bike helmet on if you keep reading. That spat out a ton of gook. But was I lying? Well, grep out everything but what we should have pruned:

find /usr/local -type d -a \( -name jre1.6.0_38 -prune -o -type d -print \) | grep jre1.6

What if you have a series of subdirectories you want to include, but you cannot write enough -prune switches for them? This is a problem I frequently have. For instance, how do you exclude all your Firefox Cache directories, especially if you have multiple profiles? Great question.

I’d first use find to find all the directories I

do want to backup:

find /home/jed -maxdepth 4 -type d > /tmp/dirlist

Then you grep out things you really don’t want:

egrep -i "/cache|/Trash" /tmp/dirlist > /tmp/avoid

Then parse it into things you do want to find to avoid:

cat /tmp/avoid | while read F ; do echo " -path $F -o " ; done > /tmp/avoid2 ;
echo "-path ./asdf" >> /tmp/avoid2

Now we can refresh our list of directories to descend:

find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -print

If we want to turn that right into files, modify the last print statement to find files:

find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -type f -print

Now if you want to find the files more recently created than your last backup in /home/backup/monday.tgz, try this:

find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -type f -newer /home/backup/monday.tgz -print

Is that enough to make you cry? Chin up, think of all the disk space you’re saving, and how much faster a specific backup can occur. This means you can run backups every 15 minutes.

Backups: Sorting Through a Restore

Linux Backups logo

Linux Backups

When you need to be rough with your data–change a bunch of files at once…and you might not do it right the first time. Or you need to recover something that grew a few bad sectors and you only have a bits of your file left. Do you have to restore ALL your work? How do you see where the changes are?

Let’s step thru a partial restore using tar and diff. For example, say I have a code directory that I’ve damaged with a regular expression. My work is in /home/work, my backups are in /home/backups.

I “untar” the backup in a temporary directory next to the work directory like so:

$ cd /home/
$ mkdir restore
$ cd restore
$ tar xzf /home/backup/monday.tgz
$ cd /home

With the two directory trees next to each other, finding the differences is easy. The -r switch for diff tell us to search an entire directory tree, and the -q switch tells diff to display a brief description of just file names. (Diff will work on binary files, so this command will also work on graphics and audio files, just don’t forget the -q).

$ diff -qr /home/work /home/restore
Files /home/work/lib/BugCatcherHelper.java and /home/restore/lib/BugCatcherHelper.java differ
Files /home/work/lib/BugCatcher.java and /home/restore/lib/BugCatcher.java differ

This example is obviously contrived, and often what source control should be used for, but not everyone uses source control.