Category F/LOSS

Adding Lots of Storage Pools to Libvirt

I do not enjoy having subdirectories involved for storing files in libvert. The virt-manager interface is just way too brutal about how you manually add storage pools. After much ranting, I wrote a bash script to add these directories for my VM disk images.

  1 #!/bin/bash
  2 
  3 set -e
  4 #set -x
  5 existing_pools=()
  6 export existing_pools
  7 
  8 function add_this_dir() {
  9     local -n xisting_pools="$1"
 10     if [ ! -d "$2" ]; then
 11       echo "add_this_dir: $2 is not a directory"
 12       return
 13     fi
 14     local shortname=`basename $2`
 15     if [[ " ${xisting_pools[@]} " =~ " $2 " ]]; then
 16       echo "$2 already exists"
 17       sudo virsh pool-start "$shortname"  &>/dev/null ||:
 18       sudo virsh pool-autostart "$shortname"  &>/dev/null ||:
 19       return
 20     fi
 21     if [[ " ${xisting_pools[@]} " =~ " $shortname " ]]; then
 22       echo "$shortname already exists"
 23       sudo virsh pool-start "$shortname" &>/dev/null ||:
 24       sudo virsh pool-autostart "$shortname" &>/dev/null  ||:
 25       return
 26     fi
 27     sudo virsh pool-define-as --name $shortname --type dir --target "$2" --source-path "$2"
 28     sleep 1
 29     sudo virsh pool-start "$shortname" &>/dev/null ||:
 30     sudo virsh pool-autostart "$shortname"  &>/dev/null ||:
 31     sleep 1
 32 }
 33 
 34 while read L; do
 35   if [[ x$L = x ]]; then continue; fi
 36   hunks=($L) 
 37   existing_pools+=("${hunks[0]}")
 38 done < <(sudo virsh pool-list --all | grep -v -e Autostart -e '----' )
 39 
 40 echo "You have these existing pools defined: "
 41 echo "%%${existing_pools[@]}%%"
 42 
 43 while read D; do
 44   add_this_dir existing_pools "/tank/VMs/$D"
 45 done < <(ls /tank/VMs) 
 46 while read D; do
 47   add_this_dir existing_pools "/tank/softlib/iso/$D"
 48 done < <(ls /tank/softlib/iso)

 

OpenVPN easy-rsa notes

Get the recent easy-rsa scripts:

# git clone openvpn/easy-rsa
git checkout v3.0.6
# copy easy-rsa directory to /etc/openvpn/server
cd /etc/openvpn/server/easy-rsa
./easyrsa init-pki
./easyrsa build-ca # this will need a password
./easyrsa gen-dh
./easyrsa gen-req servername.com nopass
./easyrsa sign-req server servername.com # requires ca passwd from above

Now you can edit your server/server.conf file and fire it up with

systemctl enable openvpn@server
systemctl start openvpn@server

and watch journalctl while you do that.

Generating a client:

./easyrsa gen-req clientname.com nopass
./easyrsa sign-req client clientname.com #requires ca passwd

Example conf file looks like:

client
proto udp
dev tun
remote support.foo.net 1194
keepaliave 10 120
keysize 256
cipher AES-256-CBC
 verb 3
compress lz4-v2
key ...
cert ...
ca ...

Ubuntu 18.04 Terminal Boot

Here are a series of commands to get Ubuntu 18.04 to boot into terminal mode, with various extras on how to get an automatic menu on boot up.

Skipping Graphical Boot

If you want to skip the graphical login screen, hit [Shift] or [Esc] before you see the grub menu to get to the grub menu. Add these features to the linux command:
systemd.unit=multi-user.target
Then hit Ctrl-X.

Changing the Default Boot Target

Become root. In /lib/systemd/system, change the default.target symlink:

# rm default.target; ln -s multi-user.target default.target
# systemctl daemon-reload

Checking the Filesystem Every Boot

If you do the first command above with a semicolon, you can still use tab-completion. Next, we go to /etc/default and update the grub settings:

# cd /etc/default
# vim grub
Change GRUB_CMDLINE_LINUX_DEFAULT to this value:
"fsck.mode=force fsck.repair=yes"

Run update-grub2:
# update-grub2

Reinforce this behavior by using tune2fs to make each file system run a check each boot. What file systems are you running?

# lsblk -o NAME,MOUNTPOINT # will produce output kinda like:
sda   
  sda1  /boot
  sda2  /
  sda3 [SWAP]
  sda4 /home

Running these command will make sda1, sda2, sda4 all check every mount:

# tune2fs -c1 /dev/sda1
# tune2fs -c1 /dev/sda2
# tune2fs -c1 /dev/sda4

Reboot:
# reboot

That shouldn’t take too long. You have a tty login now.

Creating an Automatic Menu

I’m disabling a few things:

systemctl disable snapd.service wpa_supplicant.service unattended-upgrades.service cups-browserd.service cups.service
systemctl daemon-reload

There will be lots of snaps you don’t want:

snap list --all | awk '/gnome|gtk/{print $1, $2}' | while read snapname snaprevision; do snap remove "$snapname" --revision="$snaprevision"; done
This didn't work well, maybe snap remove "$snapname" is enough
You are logged in on tty1 by default. (I don't know why tty0 exists.) Following this guide, create this directory:
# cd /etc/systemd/system
# mkdir getty@tty1.service.d
# cd getty@tty1.service.d
# vim override.conf
[Service]
ExecStart=
ExecStart=-/root/onboot.bash
StandardInput=tty
StandardOutput=tty
# vim /root/onboot.bash

#!/bin/bash
echo "This is a sound recorder appliance. Hit a key to start recording."
RECORDING=0
while true; do
  read -sn1 KEY
  if [[ $RECORDING = 0 ]]; then
    RECORDING=1
    echo "Now recording"
    /root/start-recording.bash
  else
    RECORDING=0
    echo "Recording stopped"
    /root/stop-recording.bash
  fi
done

 

# chmod +x /root/onboot.bash
# systemd daemon-reload
# reboot

All you have to do then is record things with the start-recording.bash and stop-recording.bash scripts.

FreeNAS: Installing Dovecot

Various notes on installing dovecot on FreeNAS 11. I understand this has no dovecot security applied. This is a tutorial for a LAN lab environment.

    1. If the FreeNAS is a VM, make sure the virtual network adapter permits permiscuous mode. This allows jails to network.
    2. Create dataset.
    3. Adjust Jails Setting, disable DHCP fn-jails
    4. Create a jail with an IP address and allow.raw_sockets=true. This allows ping. Make sure that VIMAGE is unselected.fn-jail1fn-jail2
    5. Add storage to jail. Both /usr/ports and /mnt/pool/jails/foo.
    6. Install vim:
      1. jls
      2. sudo jexec foo sh
      3. # cd /usr/ports/editors/vim
      4. make install
    7. Update pkg metadata for jail
      1. # pkg update
    8. Install screen, dovecot from package
      1. # pkg install dovecot
      2. # pkg install screen
    9. Edit dovecot stuff
    10. Message from dovecot-2.2.33.2_2:
---------------------------------------------------------------------
You must create the configuration files yourself. Copy them over
to /usr/local/etc/dovecot and edit them as desired:

cp -R /usr/local/etc/dovecot/example-config/* \
/usr/local/etc/dovecot

The default configuration includes IMAP and POP3 services, will
authenticate users agains the system's passwd file, and will use
the default /var/mail/$USER mbox files.

Next, enable dovecot in /etc/rc.conf:
dovecot_enable="YES"
---------------------------------------------------------------------
To avoid a risk of mailbox corruption, do not enable the
security.bsd.see_other_uids or .see_other_guids sysctls if Dovecot
is storing mail for multiple concurrent users (PR 218392).
---------------------------------------------------------------------
If you want to be able to search within attachments using the
decode2text plugin, you'll need to install textproc/catdoc, and
one of graphics/xpdf or graphics/poppler-utils.
---------------------------------------------------------------------
    11. We’ll skip the imap search features for now
    12. Let’s create a user inside this jail
      1. adduser, nologin, use a password
    13. Verify /var/mail/kathy exists
    14. Check in on QuickConfiguration
    15. in /usr/local/etc/dovecot…
      1. should be using conf.d/auth-system
      2. conf.d/10-auth: disable_plaintext = no
    16. /etc/pam.d
      1. create dovecot:
      2. auth    required        pam_unix.so
account required        pam_unix.so
    17. conf.d/10-mail.conf : mail_location = maildir:~/Maildir
    18. 10-master.conf: comment out pop3
    19. 10-ssl.conf : ssl=no
    20. in ../dovecot.conf: remove ‘::’ as interface to listen on
    21. # service dovecot restart
    22. Now attempt to hit it with thunderbird, use configs like this:fn-tbird-imap

Next is Outlook

Follow these directions on adding a IMAP account to Outlook.

If it’s really old, microsoft suggests this article.

 

Make New Human Language for Open Source

I was listening to Changelog episode 242 and when James Long describes an typical yet ideal experience of posting libre code to Github as an unexpected burden, it jumped out to me that Github, Google Code, Sourceforge, and a dozen other libre code hosting sites can step forward to change that burden by altering the human language around posting a new project.

An earnest developer wants to be known for creative thinking and their thesis is a code dump. Yet the environment of coders is young, scrappy and hungry: if it’s useful, the originator is bombarded by the obligations of maintenance and interaction of the audience.

If things that were experiments, proofs of concept, or insights, were not subject to maintenance requests, what would happen? Would the authors gain a kind of reprieve from obligation to participate? Should services like GitHub develop a code diary publishing form such that these ideas can be read in terse form but not pulled?

What form should appropriate feedback take? Comments, or a higher bar, such as patches? Or should we prohibit code in order to reduce obligation and encourage a longer form conversation: a code diary reply post?

The anxious among us immediately worry that anything synthesized even as pseudocode is somehow salient knowledge: anyone reading and not replying might be copying and posting an app while the original author is doing his best to maintain a healthy sleep schedule. Is this anxiety actually unfounded, though? Unlikely.

My point is really to focus on the how we can better steward notions of experiments, proofs of concept, and insights in a libre coding realm. The notion of having a one shot idea is perfectly valid for some people: they publish a proof of concept, something larger than a gist, yet beneath the definition of library or application. Maybe those notions can be automatically published only a markdown, or PDF. Or maybe they cannot be pulled or cloned. Or maybe the appropriate mechanism is for the publishing platform to synthesize an animation of the usage and help the author produce a technical blog post?

What other language distinctions have I not touched on that might be pertinent? Are there other situations that could be considered? As a parent challenged with compulsive coding urges, I love this topic. Chime in!

Updated Bash CPU Meter

In August, I did a post on a pretty simple bash cpu meter. This one is still Intel-only, but is records the range of frequencies used during a report.

#!/bin/bash
COLWID=17
MAXTURBO=4200
function find_lines {
   local i=0
   local j=0
   while read line; do
      if [[ $line =~ cpu\ MHz ]]; then
         cpu[i]=$j
         ((i++))
      fi
      ((j++))
   done < /proc/cpuinfo
}
function get_mhz() {
   mhz=()
   local cpulines=()
   local line hunks m L i c
   for i in `seq 1 15`; do
      c=0;
      readarray cpulines < /proc/cpuinfo
      for L in "${cpu[@]}"; do
          line="${cpulines[$L]}"
          hunks=($line)
          m=${hunks[3]}
          mhz[c]+="${m%.*} "
          ((c++))
       done
       sleep 0.1s
    done
}
# main
find_lines
while [[ 1 = 1 ]]; do
    COLS=`tput cols`
    mhz=()
    get_mhz
    cpunum=0
    for A in ${cpu[@]}; do
       lowest=0
       highest=0
       for H in ${mhz[$cpunum]}; do
         (( $H > $highest)) && highest=$H
         (( $lowest == 0    )) && lowest=$H
         (( $H < $lowest )) && lowest=$H
      done      
      outline=""      
      bars=$(( ($lowest * ($COLS-$COLWID) )/$MAXTURBO))
      for (( L=1; L<=$bars; L++ )); do
         outline=$outline"-"
      done
      bars=$(( (((1+$highest)-$lowest) * ($COLS-$COLWID))/$MAXTURBO))
      for (( L=1; L<=$bars; L++ )); do
         outline=$outline"="
      done
      d=$(($cpunum+9900))
      echo "${d##99} $lowest-$highest $outline"
      ((cpunum++))
   done
   echo ""
   sleep 0.1
done
#

LZOP is my friend

I’ve been doing a lot of disk cloning lately, working up instructions for duplicating a “sysprep” style Fedora image for the LANforge product. Now that live CDs tend to allow live installation of packages, I can boot one, plugin in a stick with the archive of my raw image, and install pv and lzop and I’m off to the races.

First you need to destroy whatever was on /dev/sda first of course:

wipefs -af /dev/sda

Install your utilities:

dnf install -y pv lzop

Then throw on your image:

pv -pet -B $((1024*1024*1024)) /media/images/frozen-sda.raw.lzop \
| lzop -dc | dd iflag=fullblock oflag=direct bs=1M of=/dev/sda

I think this is super cool. After that gets thrown on the drive, you need to run parted on it to correct any partition glitches. Fsck the paritions after that.

 

Beware the Hubris of FreeBSD

I hear a lot of very proud talk from various BSD zealots, mostly about how if you want better uptime/throughput/correctness, you should drop Linux like a rock crawling with centipedes and pick up the shinny golden nugget of FreeBSD. This will not work for me.

While Linux might be “only good enough” in their eyes, in my eyes its been better than windows for over 20 years. I hear quiet, off-hand mumbling about the state of graphics drivers and laptop support on FreeBSD. I hear they just recently got preliminary support for UEFI booting.

Various interviews with FreeBSD proponents often start with “oh, I started with Linux in the early nineties and it was a trash fire and I loved FreeBSD 4.x and haven’t looked back.” And “I hear Linux lacks proper jail support” and “Linux has a broken security model” and “SystemD is going to be the death of Linux.” Are these actually helpful points of view?

Not everyone is suffering under Linux. I’ve been putting Linux under punishing workloads for decades now and the important bit of wisdom I want to remind you of is this:

You cannot support something you are not familiar with.

Linux still makes a great desktop, server and embedded system. It’s got great tools. It runs zillions of servers and there is not a mass exodus to FreeBSD. To give a BSD or Linux box the kind of uptime, performance or security you want takes years of experience and knowledge of the subtleties of the platform. If you’re actually considering switching, start by doing a pilot project: build an evaluation stack and see if your workflow matches up to it. Bet you a buck the first thing you find is that paths, utilities and configuration defaults are going to get in your way for months until you build up your mental tree.

FreeBSD zealots behave just as much like Linux zealots behave to Windows users. Often haughty, rude and dismissive. Let’s none of us be that way.