Backing Up VM Images to Wasabi (or S3, B2, etc.)
May 20, 2020
My current setup is a RAIDz2 with VM images stored in the raw img format. I backup important individual files using rsync, but sometimes I wish I had a copy of each VM so I can easily restore it if something goes terribly wrong. Wasabi is probably the best option for this because everyone else charges you to download your data which can get very expensive if you have large VM images.
- ZFS pool name -
local - VM path -
/mnt/local/vm - Cloned snapshot path -
/mnt/local/clone
You will need to install these packages: zstd mbuffer s3cmd
Each time this script is run it will destroy the previous snapshot for the day of week it was taken on. The snapshot is then cloned so each file can be compressed and uploaded using s3cmd. The clone is then destroyed so there are no hanging snapshots.
#!/bin/bash
ZFS_BIN=$(which zfs || echo /sbin/zfs)
BUCKET="YOUR-BUCKET"
DOW=$(date +"%a")
SNAPSHOT="local@backup-$DOW"
VM="local/vm@backup-$DOW"
$ZFS_BIN destroy local/clone
$ZFS_BIN destroy -r $SNAPSHOT
$ZFS_BIN snapshot -r $SNAPSHOT
$ZFS_BIN clone $VM local/clone
s3cmd put --recursive /etc/libvirt/qemu/ s3://$BUCKET/
for filename in /mnt/local/clone/*.*; do
pzstd -c -1 $filename | mbuffer -q -m 100M | s3cmd put --multipart-chunk-size-mb=100 - s3://$BUCKET/$(basename "$filename").zst
done
$ZFS_BIN destroy local/clone
To decompress:
zstd -d <path>