The previous redaction (${PG_URL%%@*@}) never matched when the URL has a
single @, so the full DSN (user:password@host) went to the log/journal.
Now: scheme + ***:***@ + hostport/db only.
Found during the point-3 E2E test; the leaked value in that transcript
was the already-public test DB credential (itest probe DB).
/usr/local/bin/cayce-check-backup-freshness (from
deploy/backup/check-freshness.sh): exits 1 if the newest
/var/backups/cayce/YYYY-MM-DD/ dir has mtime older than 26h, or if no
backups exist while the age recipient IS configured (backups expected).
Info-exit-0 when backups have never been configured. Not attached to a
timer (only cayce-backup.timer was authorized); documented for manual
or cron use.
Verified: fresh dir => OK 0h exit 0; dir with 2-day mtime => ALERT 48h
exit 1; no-dir + recipient => ALERT; no-dir + no-recipient => INFO.
RESTORE.md documents the exact from-zero recovery on a fresh VPS:
download+verify (sha256 over encrypted artifacts, no private key needed
on the pull side), decrypt on the operator Mac, Postgres pg_restore,
Gitea unzip+migrate, Caddy/systemd/.env config restore, repo re-clone,
backup pipeline reinstall with the SAME age recipient, and a
post-restore checklist.
E2E test executed on the VPS (no production touched — cayce-ops has no
production DB; a probe DB was created and dropped):
- manifest.sha256: 3/3 artifacts OK (verified WITHOUT any private key)
- age decrypt roundtrip with a throwaway keypair (destroyed afterwards)
- pg_restore TOC valid; full restore into a scratch DB: 5000/5000 rows
- gitea zip valid, config tar valid (34 entries)
- all test material removed: probe DB dropped, throwaway key deleted,
recipient file removed, test backup deleted (was encrypted to the
destroyed throwaway key)
System user backup-pull created on the VPS (no sudo, locked password).
authorized_keys restricted with restrict + forced command
rrsync -ro /var/backups/cayce. Verified with a since-deleted throwaway
key: no shell, no arbitrary commands, no sftp/scp, no reads outside
/var/backups/cayce, no writes; the positive rsync pull works.
authorized_keys is left EMPTY — the operator adds their real key with
the exact line documented in deploy/backup/backup-pull-setup.md.
deploy/backup/make-backup.sh (installed as /usr/local/bin/cayce-make-backup):
- pg_dump custom-format of the cayce-ops DB (URL from env or .env,
SQLAlchemy driver qualifier stripped, credentials never logged)
- gitea dump (repos + sqlite + attachments), live; consistency notes in RESTORE.md
- config tar: Caddyfile(+d), gitea app.ini + internal_token, cayce-* units, .env
- EACH artifact encrypted with age BEFORE leaving the tmpdir; recipient
read from /etc/cayce-backup/age-recipient.pub (public key only). The
private age key never touches this server.
- plaintext exists ONLY in a mktemp dir removed by an EXIT trap; a partial
day directory is removed on failure; any step failure => exit != 0 =>
journald entry
- manifest.sha256 over the encrypted artifacts (pull side verifies
integrity WITHOUT the age private key)
- destination /var/backups/cayce/YYYY-MM-DD/ (750 root:backup-pull, 640)
- retention: 7 daily dirs, pruned by mtime
systemd: cayce-backup.service (oneshot, hardened: ProtectSystem=strict,
NoNewPrivileges, ProtectHome=read-only) + cayce-backup.timer (daily
03:15 UTC, Persistent). Timer ENABLED on the VPS per operator instruction;
runs fail loudly (exit 1, clear journald message) until the age recipient
is provided — that is the designed not-configured state.
cayce