Backups
Deprecated
Backup strategy for Bertram.
Table of Contents
Idea
The following things need backup:
- The “data” aka. the
/storageraid 1. - Dockers persistent volumes as they hold all important (e.g. configuration) files for the services.
- The Databases. They need special treatment to get persistent backups (database dumps)
- (OPTIONAL) The whole root file system, as it is not “protected” by a raid setup. To easily recover from a boot drive failure it is a good idea to back it up completely
The general idea is to utilize the LVM and BTRFS snapshot capabilities to create a snapshot and then to back up from said snapshot while the system can continue running. So basically: stop service → take and mount snapshot → resume service → run backup on snapshot → unmount and delete snapshot For the volume backups we can copy them to the storage to profit from the raid as well.
For maintenance and clarity reasons it may be best to separate all these different backups. So one backup config for
- The data storage
- The persistent volumes
- The root (and boot) directory
Making, mounting and deleting snapshots
BTRFS
Reference: BTRFS wiki
Create a snapshot of /storage: Note: As subvolumes can only be created on btrfs formatted partitions we can not create a snapshot at e.g. /snapshot and it has to be inside the mount point of /storage.
sudo btrfs subvolume snapshot -r /storage /storage/snapshots/SNAPSHOT_NAME
(A snapshot name could be storage_snapshot_`date +'%Y%m%d'`) It will automatically be mounted and available under /snapshots/storage_snapshot_SNAPSHOTNAME.
Delete Snapshots:
sudo btrfs subvolume delete /storage/snapshots/SNAPSHOT_NAME
LVM2
Reference: LVM Snapshots and Red Hat docs
The current setup looks like this:
$ sudo vgdisplay
--- Volume group ---
VG Name fedora
[...]
VG Size <236.89 GiB
PE Size 4.00 MiB
Total PE 60643
Alloc PE / Size 16128 / 63.00 GiB
Free PE / Size 44515 / <173.89 GiB
VG UUID u36edF-ZFbo-ZzvR-D6Wb-VgyN-DEb3-cXdbtE
$ sudo lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root fedora -wi-ao---- 63.00g
So currently the root partition with the operating system and the persistent volumes takes lives inside a 63GB big logical volume in the fedora volume group. We can now create snapshots by creating a second logical “snapshot” volume. (Note: It probably does not even need to be of similar size and can instead be rather small - like 10% of the original size - as the snapshot actually only contains data that has changed since the creation of the snapshot. But as we still have plenty of disk space available we can make it bigger than it needs to be.)
Create a new logical snapshot volume:
sudo lvcreate --size 63G --snapshot --name root_snap /dev/fedora/root
Create mount point for snapshots:
sudo mkdir /mnt/root_snap
Mount snapshot volume:
sudo mount -o nouuid /dev/fedora/root_snap /mnt/root_snap/
Unmount snapshot before removal:
sudo umount /mnt/root_snap/
Remove snapshot volume:
sudo lvremove -y /dev/fedora/root_snap
Dump databases
Dumb database (e.g. nextcloud database):
sudo docker exec -d mariadb-nextcloud mysqldump --single-transaction --default-character-set=utf8mb4 -h localhost -u ncbertram -p'DATABASE_PASSWORD' nextcloud > /path/to/backup/nextcloud-sqlbkp_`date +"%Y%m%d"`.dump
Scripts
From the above we can create scripts to run before and after backups. See Scripts.
Install and setup Kopia
Reference: Kopia docs
Install
- Install GPG signing key:
sudo rpm --import https://kopia.io/signing-key - Install repository:
cat <<EOF | sudo tee /etc/yum.repos.d/kopia.repo [Kopia] name=Kopia baseurl=http://packages.kopia.io/rpm/stable/\$basearch/ gpgcheck=1 enabled=1 gpgkey=https://kopia.io/signing-key EOF - Install Kopia
sudo dnf install -y kopia
Setup
- Create and connect to a remote (sftp) repository
- Create a passwordless ssh key:
ssh-keygen -t ed25519 -f $HOME/.ssh/bertram_backup_ed25519 -C bertram@bertram - Create repository (substitute
$REPO_PASS)sudo kopia repository create sftp \ --description="Bertram backup remote repository (SFTP)" \ --password=$REPO_PASS \ --host=<backup server domain> \ --path=/bertram_backup/kopia \ --username=porto \ --port=2242 \ --keyfile=/home/bertram/.ssh/bertram_backup_ed25519 \ --known-hosts=/home/bertram/.ssh/known_hosts \ --enable-actions - run validation test:
sudo kopia repository validate-provider
- Create a passwordless ssh key:
- Set backup (repository) policies
- check default policies:
$ sudo kopia policy show --global Policy for (global): Retention: Annual snapshots: 3 (defined for this target) Monthly snapshots: 24 (defined for this target) Weekly snapshots: 4 (defined for this target) Daily snapshots: 7 (defined for this target) Hourly snapshots: 48 (defined for this target) Latest snapshots: 10 (defined for this target) Ignore identical snapshots: false (defined for this target) Files policy: Ignore cache directories: true (defined for this target) No ignore rules: Read ignore rules from files: (defined for this target) .kopiaignore Scan one filesystem only: false (defined for this target) Error handling policy: Ignore file read errors: false (defined for this target) Ignore directory read errors: false (defined for this target) Ignore unknown types: true (defined for this target) Scheduling policy: Scheduled snapshots: None. Manual snapshot: false (defined for this target) Uploads: Max parallel snapshots (server/UI): 1 (defined for this target) Max parallel file reads: - (defined for this target) Parallel upload above size: 2 GiB (defined for this target) Compression disabled. No actions defined. Logging details (0-none, 10-maximum): Directory snapshotted: 5 (defined for this target) Directory ignored: 5 (defined for this target) Entry snapshotted: 0 (defined for this target) Entry ignored: 5 (defined for this target) Entry cache hit: 0 (defined for this target) Entry cache miss: 0 (defined for this target) - set useful global policies (compression and retention):
sudo kopia policy set --global --compression=zstd --keep-latest=6 --keep-hourly=0 --keep-daily=7 --keep-weekly=4 --keep-monthly=12 --keep-annual=2
- check default policies:
- Define actions See Scripts
- storage actions:
sudo kopia policy set /storage --before-folder-action /scripts/before_storage_backup.sh --action-command-mode=essential sudo kopia policy set /storage --after-folder-action /scripts/after_storage_backup.sh - volume actions:
sudo kopia policy set /volume1 --before-folder-action /scripts/before_volume_backup.sh --action-command-mode=essential sudo kopia policy set /volume1 --after-folder-action /scripts/after_volume_backup.sh
- storage actions:
- Create first (manual) snapshot
sudo kopia snapshot create /volume1 sudo kopia snapshot create /storage - Schedule snapshots via
crontabs(sudo crontab -e)0 */6 * * * kopia snapshot create /storage /volume1 >/dev/null 2>&1Creates a snapshot every six hours.
- Connect/ Disconnect
- For disconnection from the currently connected repository run:
sudo kopia repository disconnect - To connect again to our created repository run
sudo kopia repository connect sftp \ --description="Bertram backup remote repository (SFTP)" \ --host=<backup server domain> \ --path=/bertram_backup/kopia \ --username=porto \ --port=2242 \ --keyfile=/home/bertram/.ssh/bertram_backup_ed25519 \ --known-hosts=/home/bertram/.ssh/known_hosts \ --enable-actionsand enter the repository password.
- For disconnection from the currently connected repository run:
(work in progress) Kopia Server
[!Note] Does not work. A Server can be started to control Kopia from a browser:
sudo kopia server --tls-generate-cert --tls-generate-cert-valid-days=1 --tls-generate-cert-name=192.168.178.26 --address=192.168.178.26:51515 --random-password --random-server-control-password --shutdown-on-stdin