Happy People at LinuxFest Northwest
For the first time in 9 years I decided to take my camera to the after-party. I found a lot of happy people. If you like the picture enough, let me know. None of these have been through post, so I could clean up your portrait a bit. Great to see you all there! I look forward to 2014!
Backups: Using `find` Across a Panalopy of Directories
Linux Backups
I love using the find command. In DOS, find is like grep. In Linux, find is the most powerful recursive DOS dir /s or Linux ls -r command you could ever put your saddle on.
One of the things you can do with find is to avoid directories, using the -prune switch. Like so:
find /usr/local -type d -a \( -name jre1.6.0_38 -prune -o -type d -print \)
Yeah, put your bike helmet on if you keep reading. That spat out a ton of gook. But was I lying? Well, grep out everything but what we should have pruned:
find /usr/local -type d -a \( -name jre1.6.0_38 -prune -o -type d -print \) | grep jre1.6
What if you have a series of subdirectories you want to include, but you cannot write enough -prune switches for them? This is a problem I frequently have. For instance, how do you exclude all your Firefox Cache directories, especially if you have multiple profiles? Great question.
I’d first use find to find all the directories I
do want to backup:
find /home/jed -maxdepth 4 -type d > /tmp/dirlist
Then you grep out things you really don’t want:
egrep -i "/cache|/Trash" /tmp/dirlist > /tmp/avoid
Then parse it into things you do want to find to avoid:
cat /tmp/avoid | while read F ; do echo " -path $F -o " ; done > /tmp/avoid2 ; echo "-path ./asdf" >> /tmp/avoid2
Now we can refresh our list of directories to descend:
find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -print
If we want to turn that right into files, modify the last print statement to find files:
find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -type f -print
Now if you want to find the files more recently created than your last backup in /home/backup/monday.tgz, try this:
find . -xdev -depth -type d \( `cat /tmp/avoid2` \) -prune -o -type f -newer /home/backup/monday.tgz -print
Is that enough to make you cry? Chin up, think of all the disk space you’re saving, and how much faster a specific backup can occur. This means you can run backups every 15 minutes.
Backups: Sorting Through a Restore
Linux Backups
When you need to be rough with your data–change a bunch of files at once…and you might not do it right the first time. Or you need to recover something that grew a few bad sectors and you only have a bits of your file left. Do you have to restore ALL your work? How do you see where the changes are?
Let’s step thru a partial restore using tar and diff. For example, say I have a code directory that I’ve damaged with a regular expression. My work is in /home/work, my backups are in /home/backups.
I “untar” the backup in a temporary directory next to the work directory like so:
$ cd /home/ $ mkdir restore $ cd restore $ tar xzf /home/backup/monday.tgz $ cd /home
With the two directory trees next to each other, finding the differences is easy. The -r switch for diff tell us to search an entire directory tree, and the -q switch tells diff to display a brief description of just file names. (Diff will work on binary files, so this command will also work on graphics and audio files, just don’t forget the -q).
$ diff -qr /home/work /home/restore Files /home/work/lib/BugCatcherHelper.java and /home/restore/lib/BugCatcherHelper.java differ Files /home/work/lib/BugCatcher.java and /home/restore/lib/BugCatcher.java differ
This example is obviously contrived, and often what source control should be used for, but not everyone uses source control.
Linux Photography: Basic Darktable Tutorial
I know why my first few minutes with Darktable seemed so frustrating–they were all me scrubbing this modal interface looking for things I thought all should be in a menubar. But there are no menubars. While DT has quite a bit of keyboard shortcuts (not discussed today) Those were no help because you have to study the Settings dialog Shortcuts tab…no quick to get started with when you’re used to mundane office software. Darktable is as different from GIMP as GIMP is from Photoshop. Is DT similar to PhotoShop? You tell me.
- select photos in light table mode
- enter darkroom mode
- select an action group “basic”
- enable a module 
- reset a module using the furthest right “standby” icon
- spot selects are often a rectangle
- return to lighttable mode
- turn down stars of photos you don’t want
- expand right sidebar
- select group, select all
- where will it save?
- export selected group with the Export button at the bottom of the column. This will batch-export all the photos on your light-table screen.
Certainly this is nothing like the GIMP. You do have to scrub the interfaces to find all the gritty little features, but it is more batch-oriented and possibly a cleaner work-flow.
Backups: one quick file backup alias
When you have a file you need to edit and you have the foresight to think, “whoa, make a copy before I destroy…” you often copy hulk.txt to hulk.txt.old (that’s using the minimum of keystrokes:
cp hul[tab][tab] hul[tab][tab].old[enter].
Linux Backups
Well, a week later, what do you rename your next .old file? .old2? No time to put this folder into revision control? Thought so. You can inspect that last modified time on your file with stat. Experiment with this first:
echo `stat hulk.txt | awk '/Modify:/ {print $2}'`
(*snrk* did I just get you use use Awk? OMG!)
So how does that help…more precisely, you’re asking how do I add that to a backup file name? One of many ways, and I will show you the method with least typing: use an in-place shell exansion.
cp hulk.txt .hulk.txt.`stat hulk.txt | awk '/Modify:/ {print $2}'`
STOP. What wee character did I just sneek into that filename? Hold on, first write it up in an alias so you can reuse it:
alias bu="cp hulk.txt .hulk.txt.\`stat hulk.txt | awk '/Modify:/ {print $2}'\`"
Right, the backslashes (or ‘hacks’ as I nic them) keep your statement from actually evaluating the command as soon as it’s defined. The backtick is the same as saying “bash -e …stuff...”. Anyhow, now type bu and you can backup hulk.txt again. Now type ‘ls’ and see where your backup is.
No file? And no error? Oh, right the period before name hides it (sneeky). This means the next time we accidentally do a “rm *” (which often appears when you say “rm * .old” — Computer, stop, replay with magnification: rm__*__.old ). You need a good-old-fasioned:
ls -a
It’s hiding. Let’s finish up here with your alias, properly written:
alias bu="\`cp $1 .$1.\`stat $1 | awk '/Modify:/ {print $2}'\`"
Can we do it without that crazy awk? Sure:
alias bu="\`cp $1 .$1.\`stat $1 --printf %Y '\`"
Now go make a backup…right now!
Lightroom vs Darktable [Tutorial Geek]
Here is a more in-depth comparison of features and processes available in Darktable and Lightroom.
Linux Photo Processing
Example of GIMP
At LinuxFest Northwest 2013 April 28/29, I will be giving a talk comparing the GIMP and Darktable. These are two very powerful photo manipulation tools.
I am particularly looking at two programs that provide a strong post-processing capability. You use them for different purposes and how they are used is quit different as well. There will be a few more posts on each of these. (What about digiKam? Honestly, I don’t know anyone who uses it, so it didn’t immediately come to mind. For all-around photo-management, digiKam is certainly worthwhile. I won’t speak against it.)
The GNU Image Manipulation Program (GIMP)
Largely, the GIMP is what many people might think of a Photoshop for Linux. Many would strongly disagree–commonly what you hear is this: Gimp is nothing like Photoshop. I think people will agree on this: if you need retouching, layer compositing, text, and pixel-pencil drawing, your choices are pretty likely going to be some version of Photoshop or the GIMP. If you dont want to pay for a copy of Photoshop but want to produce layered screen graphics or high-res graphics for printing, here’s your tool. (And while GIMP has some vector tools for pathings, it is not a vector drawing program–see Inkscape for that).
Darktable
Example of Darktable
If you have used Lightroom, (a semi-pro and above level raw photo organizer and post-process workflow program), Darktable shall fill an analagous role. Darktable is has no intentions of being a drawing program. Color control is Darktables primary focus. It’s internals operate on color as 32-bit floating point values, which is mighty accurate. However, this means it wants a 64-bit computer with at least 2 gigs of ram. WIth it you can run through a batch of photos imported from your SD card, pick a few 4- and 5-star photos, isolate that set, apply color correction and “make snapshots” of them to jpg or png images.
Next
I look forward to writing out a few examples comparing and constrasting how GIMP and Darktable are used. Linux and digital photography are getting along quite well these days, and I look forward to helping you get a leg up on these two programs!
Backups: using tar and find
If you are familiar with zip files, they are the DOS version of tar files (tar = Tape Archive). The tar utility is totally intended for storing backups. A quick way to backup your home directory is:
cd /home ; tar -cvf home-jed.tar ./jed
You might see that command grab a whole lot of stuff you don’t want to keep, including all your Firefox cache files and your Trash files. Also that archive is uncompressed. Lets get it compressed as much as we can, first, that’s easy:
tar -cvjf home-jed.tbz2 ./jed
Next, we can build a list of files we want to backup using find. Please don’t try and avoid the find command, once you begin to understand it, life in Linux really can improve. On our first try, we will pair it down with fgrep (simple grep) to exclude our Firefox .Cache directory.
cd home find jed/.mozilla/firefox \ | fgrep -v '.default/Cache' \ > /tmp/jed.txt
And following that, avoiding our trash can:
find jed/.mozilla/firefox \ | fgrep -v '.default/Cache' \ | fgrep -v '.local/share/Trash' \ > /tmp/jed.txt
Now think about why we want to use pipe operators in that second find command. Would it be easier as two commands both appending to /tmp/jed.txt? (Think about the overlap and duplication that results.)
If we wanted to use that file to guide tar, we change our tar command like so:
tar cvjf ./jed.tbz2 -T /tmp/jed.txt
In order to make regular backups a regularity, we need to make them pertinent and economical (of time and of space). We often do not want to back up ephemeral files that are byproducts of our work. If you program, you will have ready examples on your own drive: .a, .o, .out, .class code files often do not need to be kept if you make them several times a day.
Consider the example below. With it we can backup the substantive slice of our code tree to another drive on our system. We avoid the ephemeral files. We also chose to backup our code separately from the rest of our home directory. By doing this we can schedule code tree backups every hour, and schedule our home tree backups just once a day.
#!/bin/bash
function CodeSnap() {
local now=`date +%Y-%m-%d.%H%M`
local arcnom="/mnt/backup/code.$now.tbz2"
local flist="/tmp/code.$now.txt"
find ~/code -type f -a\
\( -name '*.xml' \
-o -name '*.java' \
-o -name '*.properties' \
-o -name '*.php' \
-o -name '*.pl' \
-o -name '*.conf' \
-o -name '*.pm' \
-o -name '*.c' \
-o -name '*.h' \
-o -name '*sh' \
-o -name '[Mm]ake*' \
\) > $flist
find ~/Documents -type f -a\
\( -name '*.php' \
-o -name '*.pl' \
-o -name '*.conf' \
-o -name '*.pm' \
\) >> $flist
tar vcjf $arcnom -T $flist
}
##
## Copyright (C) 2013, Jed Reynolds
## Free for non commercial use.
##
CodeSnap
Questions? I hope! You just saw a full strength, professional level bash script. If you don’t have questions, show me your script.
Backups: outline
Here’s some basic programs and techniques I’ll be covering about backups.
- tar
- rsync
- find
- date
- how to write “now” using date
- how to find files newer than your last backup
- all this will be done in bash
Backups: Using rsync and find.
Linux Backups
I’ll start with an out-of-order post showing rsync and find. If you don’t know [[WTF]] I’m talking about, do comment, but stick with me. This could save you hundreds or thousands of dollars by avoiding purchasing a separate piece of backup software or proprietary solution.
1 #!/bin/bash 2 ## 3 ## backup script, (C) 2013, Jed Reynolds 4 ## 5 source ./bu-rsync.sh 6 export DEST="latitude" 7 EXCLUDES="XX.gvfs PP.cache PPCache PPTrash" 8 bu_steps home/jreynolds/ / $DEST/home-jreynolds JUST_FILES $EXCLUDES 9 bu_steps home/jreynolds/ / $DEST/home-jreynolds $EXCLUDES 10 bu_steps home/liam/ / $DEST/home-liam JUST_FILES $EXCLUDES 11 bu_steps home/liam/ / $DEST/home-liam $EXCLUDES 12 bu_steps home/Music / $MDEST SIZEONLY 13 bu_steps home/Pictures / $PDEST SIZEONLY 14 15 EXCLUDES="" 16 bu_steps home/candela/btbits/x64_btbits/client / $DEST/home-candela/client \ 17 'XX*.class' 18 bu_steps home/candela/btbits/x64_btbits/server / $DEST/home-candela/server \ 19 'XX*.o' 20 bu_steps home/candela/btbits/x64_btbits/tools / $DEST/home-candela/btbits/x64_btbits/tools \ 21 'XX*.o' 22 bu_steps home/candela/btbits/x64_btbits/3plibs / $DEST/home-candela/3plibs \ 23 'XX*.o' 24 bu_steps home/candela/btbits/x64_btbits/html / $DEST/home-candela/html 25 bu_steps etc / $DEST/etc 26 bu_steps usr/local / $DEST/usr-local \ 27 SIZE_ONLY 28 ## 29 ## Free for non-commercial use. No warrany or support offered. 30 ##
That was the config file. It wont do anything without the library of functions, below:
1 #!/bin/bash
2 ##
3 ## backup script library, (C) 2013, Jed Reynolds
4 ## Free for non-commercial use. No warrany or support offered.
5 ##
6 BU_HOST=beavertail
7 SSH_OP="-i /home/jreynolds/.ssh/beavertail_dsa"
8 LSYNC="rsync "
9 export REALM=tank
10 RHOST="backup@$BU_HOST"
11 RSYNC="rsync --progress -rlpt --copy-unsafe-links "
12 Y=`date +%Y`
13 DEST="latitude"
14 PDEST="pictures"
15 MDEST="music"
16 SDEST="softlib"
17 VDEST="VMs"
18 RSYNC_PASSWORD="m........"
19 RSYNC_PASSWD="m........"
20 FLIST="/tmp/bu-list"
21 MK_SNAP="000-mksnap-000"
22 export RSYNC_PASSWORD RSYNC_PASSWD
23
24 function fail() {
25 local msg=${1:-"unknown cause"}
26 echo " -- $msg --"
27 exit 1
28 }
29 function ping_gw() {
30 local default_gw=$(ip r | grep default | cut -d ' ' -f 3)
31 ping_host $default_gw
32 return $?
33 }
34 function ping_host() {
35 [ -z "$1" ] && echo "ping_host cannot ping no host, bye" && exit 1
36 ping -q -w 1 -c 1 $1 > /dev/null && return 0 || return 1
37 }
38 function bu_steps() {
39 ping_gw || fail "no gateway"
40 ping_host $BU_HOST || fail "no ping to $BU_HOST"
41 local f_others=""
42 local r_others=""
43 local dir_a=$1; shift
44 local sit_on=$1; shift
45 local dir_b=$1; shift
46 [ "${dir_b:0:1}" != "/" ] && dir_b="$dir_b"
47 local xcld=""
48 local prun=""
49 local minusv=""
50 local flist="/tmp/flist.txt"
51 local RECT=".recent"
52 while(( "$#" )); do
53 [ "$1" == "SIZEONLY" ] && r_others="$r_others --size-only "
54 [ "$1" == "JUST_FILES" ] && f_others="-maxdepth 1 -type f $f_others "
55 [ "$1" == "JUST_FILES" ] && RECT=".recent_files"
56 if [[ $1 == XX* ]] ; then
57 [ ! -z "$xlcd" ] && xcld="$xcld -o"
58 [ ! -z "$minusv" ] && minusv="${minusv}|"
59 xcld="$xcld ${1/XX/-name }"
60 minuxv="${minusv}${1/XX/}"
61 fi
62 if [[ $1 == PP* ]] ; then
63 [ ! -z "$prun" ] && prun="$prun -o"
64 [ ! -z "$minusv" ] && minusv="${minusv}|"
65 prun="$prun ${1/PP/-name } -prune"
66 minuxv="${minusv}${1/XX/}"
67 fi
68 shift;
69 done
70 [ ! -z "$prun" ] && prun="-a ( ! $prun )"
71 [ ! -z "$xcld" ] && xcld="-a ( ! $xcld )"
72 [ ! -z "$minusv" ] && minusv="| grep -v ($minusv)"
73
74 local mk_remot=0
75 $LSYNC ${RHOST}::${REALM}/${dir_b} || mk_remot=1
76 if [ $mk_remot -eq 1 ] ; then
77 ssh jreynolds@$BU_HOST "sudo mkdir -p /$REALM/$dir_b && sudo chmod 777 /$REALM/$dir_b " || fail "Could not make dir in /$REALM/$dir_b"
78 fi
79 cd $sit_on
80
81 # Find the .recent file, or create a recent file of Jan 1, 1990
82 if [ ! -f $dir_a/$RECT ] ; then
83 echo "creating ${sit_on}${dir_a}/$RECT that will pull Everything!"
84 touch -d "01 Jan 1990" ${sit_on}${dir_a}/$RECT || fail "permission denied in ${sit_on}${dir_a}, bye"
85 fi
86 cat /dev/null > $flist
87 if [ ! -d ${sit_on}${dir_a} ] ; then
88 echo "** ${sit_on}${dir_a} not found, skipping **"
89 return 0
90 else
91 FND_CMD="find $dir_a $f_others -type f -a -newer ${sit_on}${dir_a}/$RECT $xcld $prun"
92 echo "$FND_CMD $minusv > $flist"
93 $FND_CMD $minusv > $flist || fail "unable to complete find command"
94 fi
95 local fil_ct=`wc -l $flist | cut -d' ' -f1`
96 if [ $fil_ct -lt 1 ] ; then
97 echo "Skipping $dir_b, no files found"
98 else
99 echo "== $RSYNC $r_others --files-from=$flist $sit_on $RHOST::$REALM/$dir_b/ =="
100 $RSYNC $r_others -v --files-from=$flist $sit_on $RHOST::$REALM/$dir_b/ || fail "Rsync failed, bad paths?"
101 touch_mksnap $dir_b
102 touch $dir_a/$RECT
103 fi
104 }
105 function check_nas_ready() {
106 ssh $SSH_OP jreynolds@$BU_HOST "~/bin/mount_up"
107 }
108 function touch_mksnap() {
109 ssh $SSH_OP jreynolds@$BU_HOST "sudo touch /$REALM/$1/$MK_SNAP"
110 }
111
112 [ `id -u` == "0" ] || fail "Do not run but as root, bye."
113 ping_gw || fail "no gateway"
114 ping_host $BU_HOST || fail "no ping to $BU_HOST"
115 check_nas_ready || fail "NAS not prepared, cannot continue."
116
117 [ -f $DEST/NOT-MOUNTED ] && mount $DEST
118 [ -f $PDEST/NOT-MOUNTED ] && mount $PDEST
119 [ -f $MDEST/NOT-MOUNTED ] && mount $MDEST
120
121 [ -f $DEST/NOT-MOUNTED ] && fail "Remote system not mounted to $DEST, bye."
122 [ -f $MDEST/NOT-MOUNTED ] && fail "Remote system not mounted to $MDEST, bye."
123 [ -f $PDEST/NOT-MOUNTED ] && fail "Remote system not mounted to $PDEST, bye."
124 #
2013 Bike Picnics
Bike Picnic with the kids. BYOF. This year: cold lunches. I plan on one picnic each in May, June, July, Aug.
Nanopaper and Flexible buckytube transistors?
Nanopaper and Flexible buckytube transistors?
I have been always quite excited to hear about ways of making e-paper. This is rather a kind of complment to that, this is a microcellulose nanopaper that has organic circuitry (buckytubes) adhered to it. Not only is this not a display device yet, it is not a anything close to generaly manufacturable. From the comments:
I’ve read the fine article, and it might not be immediately clear from the summary that the breakthrough here is not the transistor per se – the important step was in using the “nanopaper” (which is tech that is in fact NOT 2KA old).
And while the nanopaper may be biodegradable, I am wondering about the carbon nanotubes they are printing on top (as conductors). While the toxicity of carbon nanotubes is still being studied, there are good indications that they might behave similar to asbestos fibres. So not something you would necessarily want to throw on your compost heap.
Minotaur Rock Star, Bellingham 2012-05-05 [wallpaper]
This guy (or gal) was rockin’ the scene at the 2012 Procession of the Species.
minotaur @ bellingham procession of species 1920×1500 2012-05-05
Remember Spring? 2012-04-08 Grasses [wallpaper]
Park St Grass 2012-04-08
Even though it’s been colder than the 30s in Bellingham for many winters, riding a bike through the cold doesn’t make me sweat. I cooked up a pleasant Easter picture for your desktop.
Building faster with Netbeans and SSDs
The build directories that Netbeans uses are pounded a lot, and chances are you have enough ram to leverage tmpfs. Let do it! Start with our /etc/rc.local file:
echo deadline > /sys/block/sda/queue/scheduler echo 1 > /sys/block/sda/queue/iosched/fifo_batch D=/home/jreynolds/.cache/netbeans rm -rf $D/* mount $D chown -R jreynolds:jreynolds $D D=/home/jreynolds/NetBeansProjects/MyProject/build rm -rf $D/* mount $D # dont furgit the ram directory wants to be regularly deleted by netbeans mkdir $D/ram chown -R jreynolds:jreynolds $D D=/home/jreynolds/build/build-lib/ rm -rf $D/* mount $D mkdir $D/jar mkdir $D/classes chown -R jreynolds:jreynolds $D exit 0
Next we make our /etc/fstab match:
none /var/tmp tmpfs defaults,noatime 0 0 none /tmp tmpfs defaults,noatime 0 0 none /home/jreynolds/.cache/chromium tmpfs noatime,noexec 0 0 none /home/jreynolds/.cache/netbeans tmpfs noauto,noatime 0 0 none /home/jreynolds/NetBeansProjects/MyProject/build tmpfs defaults,noatime,noauto 0 0 none /home/jreynolds/.mozilla/firefox/0m31s0ag.default/Cache tmpfs defaults,noatime,noexec 0 0 none /home/jreynolds/build/myproject-lib tmpfs noauto,noatime 0 0
Biplane, Bellingham Air Museum
2012-11-09-flight-museum-biplane-01-1920×1285
2012-11-09-flight-museum-biplane-02-1920×1285
Ellie at OMSI 2012-08-27
Ellie at OMSI 2011-08-27 1920×1440
Improve SSD performance on Linux
I had little idea that you could so easily change the disk scheduler in linux. I’m actually rather relieved I can do it in grub, or even live. I’ll have to do this on my workstations with SSDs.
Four Tweaks for Using Linux with Solid State Drives | Tombuntu.
Eloise, Ravenna Park, 2012-04-03
I cooked this one to emphasize her magic.
Ellie, Ravenna Park, 2012-04-03
Experiement: contrast, pattern, texture
Interesting to play with the ‘newsprint’ filter and grain merge.
