flowchart TB accTitle: NutsNews Backend Backup And Restore Baseline accDescr { This documents the backup and restore policy for ramideltoro/nutsnews-backend before any production backend state is stored on 65.75.201.18. } A["NutsNews Backend Backup And Restore Baseline"] --> B["NUTSNEWS_BACKEND_BACKUP_RESTORE.md"] B --> C["Auto-generated placeholder diagram"]

NutsNews Backend Backup And Restore Baseline

This documents the backup and restore policy for ramideltoro/nutsnews-backend before any production backend state is stored on 65.75.201.18.

Current State

The backend host has no deployed app runtime, upload storage, or production backend app state yet. It does have host baseline, Caddy, dashboard, backup-status state, and a private PostgreSQL restore/failover target covered by the service-aware backup matrix. The worker-uplift RabbitMQ broker is now provisioned as durable transport; its live message store is intentionally excluded from normal Restic snapshots, while non-secret topology/config and RabbitMQ recovery evidence are included.

The backend repo source of truth is:

docs/backend-backup-service-matrix.json

Policy

Backups must survive VPS loss. Provider snapshots are supplemental only and must not be the sole recovery mechanism for application data, database state, credentials, or operational evidence.

Scope

Data class	Initial owner	Backup requirement
Backend app code/config	GitHub repositories	Git remotes are source of truth
Runtime env/secrets	GitHub Environment secrets or documented secret store	Secret names documented; values excluded from restic
Host and reverse proxy config	Backend repo through Ansible plus restic evidence	Recreate from repo and protected apply; restore selected state as needed
Ops dashboard/status metadata	Backend host collectors	Backed up for incident evidence and exposed in dashboard/reporting
Application uploads/local state	Future backend issue	Off-server backups required before production use
PostgreSQL data	Backend issue #13	Host state is covered by service-aware restic paths; staging Supabase restore drills prove database restore readiness before production use
RabbitMQ broker	Backend worker-uplift runtime	Non-secret config, topology, Compose, and recovery evidence are backed up; live `/var/lib/nutsnews/rabbitmq` message-store snapshots require a stopped or quiesced broker and are excluded from normal Restic jobs
Logs	Host retention plus future off-server policy	Retain enough for troubleshooting without unbounded raw logs

Initial Retention Baseline

Until a workload-specific issue chooses different values:

Backup type	Retention
Daily	14
Weekly	8
Monthly	12
Yearly	2

Restore Test Gate

Before production traffic or production data depends on this backend host:

Restore the latest backup to an isolated path or non-production host.
Verify ownership and permissions.
For database backups, run the backend PostgreSQL failover drill in restore-staging mode and verify /var/lib/nutsnews/postgres/status.json.
Confirm the restored data satisfies the documented RPO/RTO.
Record snapshot ID, restore target, validation commands, and result in the relevant issue or PR.

Secret Boundary

Backup credentials live in the production-backend GitHub Environment. Do not commit backup credentials, repository passwords, provider keys, database dumps, or restore artifacts.

Required secret names:

RESTIC_REPOSITORY
RESTIC_PASSWORD
provider credentials for NUTSNEWS_BACKEND_RESTIC_PROVIDER, currently AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for s3

The protected apply writes a root-only systemd environment file. Status JSON is world-readable for observability and contains no secret values.

GitOps Components

The backend protected apply installs:

/usr/local/sbin/nutsnews-backup
/etc/nutsnews-backup/service-matrix.json
/etc/nutsnews-backup/restic.env with mode 0600
/var/lib/nutsnews/backups/
nutsnews-backup.service and .timer
nutsnews-backup-verify.service and .timer
nutsnews-restore-drill.service and .timer

Manual runs use the fixed Backend Backup Maintenance workflow with only these actions:

status
backup
verify
restore-drill

Mutating actions require confirm_target=backend.nutsnews.com and the production-backend approval gate.

Status Files

The runner writes:

File	Meaning
`/var/lib/nutsnews/backups/last-backup.json`	latest backup freshness, snapshot id, included paths, quota status
`/var/lib/nutsnews/backups/last-verification.json`	latest restic check result
`/var/lib/nutsnews/backups/last-restore-verification.json`	lightweight restore-drill result
`/var/lib/nutsnews/postgres/status.json`	PostgreSQL restore/failover readiness and latest staging restore drill result
`/var/lib/nutsnews/rabbitmq-recovery/last-definition-export.json`	latest sanitized RabbitMQ definition export result
`/var/lib/nutsnews/rabbitmq-recovery/last-clean-rebuild-drill.json`	latest disposable empty-broker rebuild drill result
`/var/lib/nutsnews/rabbitmq-recovery/last-stopped-volume-restore-drill.json`	latest disposable stopped-volume restore drill result

The health report and loopback-only ops dashboard expose backup failure, stale backup, unverified latest snapshot, and storage/quota warning as separate signals.

PostgreSQL-specific restore flow is documented in NutsNews Backend PostgreSQL Failover Target. RabbitMQ-specific rebuild, stopped-volume restore, and upgrade procedures are documented in Worker-Uplift RabbitMQ Recovery.

Recovery Order

Provision or recover the host.
Apply baseline configuration through the protected backend workflow.
Restore secrets through the documented secret store.
Restore stateful data from encrypted off-server backups.
For RabbitMQ, rebuild from the pinned image, config, topology, and protected credentials unless a stopped/quiesced broker snapshot was deliberately captured for message-store restore.
Run workload-specific integrity checks.
Reconcile RabbitMQ queue gaps from backend PostgreSQL outbox and reconciliation state.
Verify health endpoints, dashboard status, and logs.
Route traffic only after verification passes.

Primary diagram