/ backup

HashBackup Example #1 - Files

I learn by example. So why not post some of my own examples?
In this post I will show how I backup my Windows 10 laptop using HashBackup every 20 minutes with notifications and logs sent to Discord. Because HashBackup is a Linux program, I'm using Bash on Windows (WSL) to execute it.

Note: All snippets for this post are available on GitHub

What is HashBackup?

A backup tool. Check it out at hashbackup.com

What am I backing up?

A lot of miscellaneous things.

  • Dropbox synced files
  • Most things in C:\Users\jh
  • A few things in AppData
  • Whatever else that's important

What isn't backed up?

This backup set doesn't include virtual machines. That requires a more complex setup that I do with a separate HB instance. I'll make a post about that later.

Scripting

HashBackup is a good tool, but for my use I've done a lot of my own scripting to automate it.

I keep all of my computer-sepecific backup scripts in a Git repo at /home/jh/do.
2017-08-21_17-01-30_ConEmu64
There are a few shared scripts that I keep in another repo at /home/jh/hbscripts.
2017-08-21_17-04-05_ConEmu64

Main backup script

This script is pretty neat. It gets the list of paths I want to backup from do/list/backup-files.txt and generates a temporary script to run the hb command.

do/backup-files.sh:

#!/bin/bash
RUN="$HOME/do/run-files.sh"
echo '#!/bin/bash' > $RUN
echo 'hb backup --no-ino -c $HOME/hb/files \' >> $RUN
echo `xargs -i echo \"{}\" < $HOME/do/list/backup-files.txt` >> $RUN
/bin/bash $RUN
rm $RUN

Here is the generated run-files.sh script:

#!/bin/bash
hb backup --no-ino -c $HOME/hb/files \
"/mnt/c/Dropbox" "/mnt/c/Programs"

Note: there are many more paths in my actual script than in the above example. But you get the gist.

Notifications

I like to keep an eye on backups, but I dislike using email for notifications.
Discord has a handy feature called Webhooks which can post messages in a channel using a single curl request.

I made a handy little script to send a message to my #backups channel.
discord.sh:

#!/bin/bash
LOG_PATH=$1
DISCORD_USERNAME=$2
if [[ $# -eq 0 ]]; then
     echo "First argument is LOG_PATH (required), second is DISCORD_USERNAME (optional)."
     exit 1
 fi
# Upload log
UUID=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 40 | head -n 1)
DATE=$(date +%Y-%m-%d_%H-%M)
NAME="$DATE"_$UUID.log
rclone copyto $LOG_PATH "b2:my-logs/Backup-$HOSTNAME/$NAME"
msg="\"Backup done! It is $DATE nl_https://txt.jha.sh/logs/Backup-$HOSTNAME/$NAME\""
msg=$(echo $msg | sed 's/nl_/\\n/g')

# Send to Discord
data="{\"username\":\"$DISCORD_USERNAME\",\"content\":$msg}"
echo $data | curl -X POST https://discordapp.com/api/webhooks/<HOOK_ID> [email protected]

This uploads logs and sends them to my #backups Discord channel.
2017-08-22_08-47-20_Discord

Here is my simple wrapper for backup-files and discord scripts.
do/notify/backup-files.sh:

#!/bin/bash
LOG_PATH=$HOME/log/hb-files.log
$HOME/do/backup-files.sh > $LOG_PATH 2>&1
$HOME/hbscripts/discord.sh $LOG_PATH "Laptop Files$1"

Automatic Backups

Under Linux it's easy to set up automatic backups using Cron:
0 * * * * $HOME/do/notify/backup-files.sh

But I'm using Bash on Windows (WSL) which doesn't support Cron jobs.
That won't stop me though!
do/auto-backup.sh

#!/bin/bash
while $true; do
    time $HOME/do/notify/backup-files.sh " | Auto-Backup"
    sleep 20m
done

This will run my backup every 20 minutes. But there is one annoying problem: it needs to be started every time I turn on my laptop!
That's where the Windows Task Scheduler comes in.
2017-08-22_09-27-29_mmc
This task is scheduled to run at login.
2017-08-22_09-28-02_mmc
Now I don't need to remember my backups as I'm waking up in the morning.
Running the backup every 20 minutes is a bit overkill, but sometimes overkill is more fun (backups are supposed to be fun right?)

HashBackup Config Backups

If I lost the critical HashBackup config files like key.conf then I'd be unable to restore my data.
To backup all HB backup sets I execute hbscripts/config-backup.sh

It's broken up into three small scripts:
hbscripts/config-backup.sh:

#!/bin/bash
$HOME/hbscripts/create-hb-config-backup.sh ~/hb
mv $HOME/$HOSTNAME.hashbackup.conf/* /mnt/c/NOBACKUP/$HOSTNAME.hashbackup.conf
rmdir $HOME/$HOSTNAME.hashbackup.conf

hbscripts/create-hb-config-backup.sh:

#!/bin/bash
$(dirname $0)/gen-configs.sh $1
rsync -aPL --include='*/' --include='*.conf' --exclude='*' $1 $HOME/$HOSTNAME.hashbackup.conf
rsync -aP $HOME/do $HOME/$HOSTNAME.hashbackup.conf --exclude='.git'
rsync -aP $HOME/hbscripts $HOME/$HOSTNAME.hashbackup.conf --exclude='.git'

hbscripts/gen-configs.sh:

#!/bin/bash
for f in `find $1/* -maxdepth 0`; do hb config -c $f > $f/config.conf 2>&1; done

In short, these put all config files in C:\NOBACKUP\laptop.hashbackup.conf.
Next I zip them up and upload them to 1Password.

HashBackup Config

Here is the output of hb config -c ~/hb/files

arc-size-limit 16MB
audit-commands
backup-linux-attrs False
cache-size-limit 5GB
copy-executable True
dbid 92b3-8bee (read-only)
dbrev 28 (read-only)
dedup-mem 8GB
disable-commands
enable-commands
hfs-compress False
no-backup-ext
no-backup-tag
no-compress-ext
no-dedup-ext jpg,png
pack-age-days 30
pack-bytes-free 1MB
pack-percent-free 50
pack-remote-archives False
remote-update normal
simulated-backup False

arc-size-limit is set to 16MB to reduce max cache size when restoring for heavily deduped data.
If I was using a destination that supported selective downloads like B2 then I'd opt for larger arc files.
cache-size-limit is set at 5GB so that I'm rarely waiting on destinations during backups.

HB Destinations

When I first started using HB, I only used Google Drive as the destination and no local cache (cache-size-limit=0).
After a while it became apparent how slow it can be when restoring.
Now I use both Google Drive and a USB hard drive. This makes restores much faster.

Most of the time I disable Google Drive and opt for local backups only. That way during the day backups are really fast. I enable Google Drive every other night or so and hb dest sync the last few backups up to Google Drive.

Here is my dest.conf:

destname passport
type dir
dir /mnt/s/NOBACKUP/HashBackup/Laptop.Files

destname gdrive
type shell
workers 5
run python /home/jh/bin/rclone.py --args "--retries 5" --destname gdrive --backupdir /home/jh/hb/files --clonedir gd:Backup/HashBackup/Laptop.Files --command

Note how Google Drive uses type shell because HB only supports GDrive via Rclone.

Switching between local and remote destinations

To make it really easy to toggle remotes I made a couple of aliases in my .zsh_aliases:

alias hblocal='cp ~/hb/files/local.dest.conf ~/hb/files/dest.conf'
alias hbremote='cp ~/hb/files/remote.dest.conf ~/hb/files/dest.conf'

remote.dest.conf contains remotes gdrive and passport.
local.dest.conf contains only the passport remote.

That's it!

How do you backup your files? Let me know in the comments below.

HashBackup Example #1 - Files
Share this