Borgmatic / Bash CI / git-crypt and key management

Borgmatic / Bash CI / git-crypt and key management

Borgmatic Backups for security
“Backups are important for information security. Underestimated, often forgotten, but key to resilience and compliance alike.” - every sane tech expert Data is more valuable than ever. And hard disks are just consumables. In the old days (early 2010s), I had setups with Bacula Backup. It worked well, but restore

A couple of side notes in the following. I updated the blog post on borgmatic.

Borg != rdiff-backup

At first I saw a more or less updated variant of rdiff-backup. Everybody knows Rsync style backups. So why use something else?

It turns out you can stream Borg backups in chunks. Meaning: you don't need to mirror large shares and diff these by filename, atime , ctime , or other types of file metadata. You can use chunks. That has a couple of advantages, including better diffing and easier streaming of data.

Another thing I want to add is that key management becomes a concern. Key management refers to these encryption keys you need to use to protect the confidentiality of the backups. What do you do with the keys?

git-crypt

You can add them to Git(Hub). If you use git-crypt. This can encrypt defined keys (in a folder ./keys) symmetrically with a key file.

git-crypt gets setup via .gitattributes:

keys/** filter=git-crypt diff=git-crypt
**/.hetzner_password filter=git-crypt diff=git-crypt
**/*.key filter=git-crypt diff=git-crypt
.gitattributes !filter !diff
  1. keys/** - individual Borg repo keys
  2. .hetzner_password - storage box PW for unattended backups
  3. *.key should not exist because keys belong in the key folder. But you never know...

git will automatically encrypt the files. These keys don't change frequently, so it's not a big issue if the keys cannot get diffed.

Bash / borgmatic YAML files / cron

You can create

  • separate bash files per job, which will individually use a yaml file for borgmatic
  • in the borgmatic yaml you specifiy the actions like docker compose down etc.
# Borgmatic configuration for Ghost Blog: Osroad
# Location: /home/marius/ghost-osroad

# Local directories to back up
source_directories:
    - /home/marius/ghost-osroad

# Hetzner Storage Box repository configuration
repositories:
    - path: ssh://***.your-storagebox.de:23/./ghost-osroad-backups
      label: hetzner-storage-box
      encryption: repokey-blake2

# SSH connection settings
ssh_command: sshpass -f /home/marius/borgmatic/keys/.hetzner_password ssh -p 23 -o StrictHostKeyChecking=no

# Encryption passphrase
encryption_passcommand: cat /home/marius/borgmatic/keys/.hetzner_password

# Exclude patterns
exclude_patterns:
    - '*.pyc'
    - '*/.cache'
    - '*/node_modules'
    - '*/.git'

exclude_caches: true

# Retention policy
keep_secondly: 3

# Compression
compression: lz4

# Archive naming
archive_name_format: 'ghost-osroad-{now:%Y-%m-%d-%H%M%S}'

# Consistency checks
checks:
    - name: repository
      frequency: 2 weeks
    - name: archives
      frequency: 1 month

# Command hooks
commands:
    # Stop containers before backup to ensure DB consistency
    - before: repository
      when:
          - create
      run:
          - echo "Stopping Ghost containers..."
          - cd /home/marius/ghost-osroad && docker compose -f compose.yml down

    # Restart containers after backup
    - after: repository
      when:
          - create
      run:
          - echo "Starting Ghost containers..."
          - cd /home/marius/ghost-osroad && docker compose -f compose.yml up -d

    # Ensure restart on failure
    - after: repository
      when:
          - create
      states:
          - fail
      run:
          - echo "Backup failed. Restarting Ghost containers..."
          - cd /home/marius/ghost-osroad && docker compose -f compose.yml up -d || true

retries: 3
retry_wait: 300
progress: true
statistics: true
verbosity: 1

Then you trigger the job via Bash:

#!/bin/bash
#
# Backup script for Ghost Blog: Osroad
# Usage: ./backup-ghost-osroad.sh
#

# Configuration
PROJECT_DIR="/home/marius/borgmatic"
CONFIG_FILE="${PROJECT_DIR}/configs/ghost-osroad.yaml"
LOG_DIR="${PROJECT_DIR}/logs"
LOG_FILE="${LOG_DIR}/ghost-osroad-$(date +%Y%m%d-%H%M%S).log"
BORGMATIC_BIN="/root/.local/bin/borgmatic"

# Ensure log directory exists
mkdir -p "${LOG_DIR}"

echo "Starting Ghost Osroad Backup..."
echo "Config: ${CONFIG_FILE}"
echo "Log: ${LOG_FILE}"

# Run borgmatic
sudo "${BORGMATIC_BIN}" create \
    --config "${CONFIG_FILE}" \
    --verbosity 1 \
    --stats \
    2>&1 | tee -a "${LOG_FILE}"

EXIT_CODE=${PIPESTATUS[0]}

if [ $EXIT_CODE -eq 0 ]; then
    echo "✅ Backup finished successfully."

    # Prune old backups
    echo "Pruning old archives..."
    sudo "${BORGMATIC_BIN}" prune --config "${CONFIG_FILE}" >> "${LOG_FILE}" 2>&1

    # Compact repository
    echo "Compacting repository..."
    sudo "${BORGMATIC_BIN}" compact --config "${CONFIG_FILE}" >> "${LOG_FILE}" 2>&1
else
    echo "❌ Backup FAILED. Check logs: ${LOG_FILE}"
fi

exit $EXIT_CODE
  • This is simple, scalable, and you can streamline this across different docker setups.
  • Finally, you can mount the repo via SSHfs / fuse and selectively restore files.

Summary

  • you can track backup scripts in Git, and you should consider git-crypt. Just store the key in a password manager or elsewhere
  • borgmatic works with docker compose
  • Borg allows streamable backups without 1:1 storage allocation
    • you can have a slim backup console

You can also backup SaaS apps with scripts and use Borgmatic in step 2:

Atlassian Confluence Migration / Exit Plan
The world is turning, and Atlassian’s products have turned into a centerpiece of many internal business tools. An exit strategy is both a compliance requirement and a way to maintain flexibility. No one wants a vendor lock-in. Confluence Markdown Exporter GitHub - Spenhouet/confluence-markdown-exporter: Export Atlassian Confluence pages as markdown

Share This Article