Backup and restore

Back up the database and application data, keep encrypted copies off the server, and recover to a snapshot or a moment in time.

Source checkout backups

./scripts/backup.sh

This dumps Postgres (pg_dump) and archives data/ into backups/<stamp>/. Backup directories have mode 0700, and both artifacts have mode 0600, including on reused paths. A missing data/ or PREA_BACKUP_SKIP_HOMES=1 produces an empty archive that restore treats as omitted homes. Database or archive errors fail the backup. Discard the output directory of any failed run.

Restore

For backups created by scripts/backup.sh, use an empty prea database in the development Compose stack. The script validates and extracts the homes archive in a clean sibling directory before touching the database, then stops the API, worker, and supervisor. It rejects a data/ path that is a symlink or a non-directory. Stop bot computers and any other processes that write to this database or tree before running the script.

The SQL import runs in one transaction and stops on the first error, including conflicts with existing tables. An invalid archive or failed import leaves the current data tree unchanged. After SQL succeeds, the script moves the current tree to data.rollback.<random>/data, prints that path, and replaces data/ with the saved tree. Files absent from the snapshot do not remain in the active tree. A missing homes archive or an empty archive from omitted homes leaves data/ unchanged. An archive containing an empty data/ directory replaces it with an empty tree.

Application services start only after the import and tree replacement succeed. Once writer shutdown starts, a later failure or interruption makes the script attempt to stop the API, worker, and supervisor again. This includes services that started before final startup failed. The script preserves the original failure status even if cleanup fails. If Docker cannot stop the services, stop them before recovery.

A failure after the SQL commit cannot roll back the database automatically. Leave the application stopped, preserve both trees, and recover a matching database before moving the rollback tree back to data/. Keep a separate database backup from before recovery if you need that rollback. This script does not consume the production snapshot’s custom-format prea.dump or appdata.tgz.

./scripts/restore.sh backups/<stamp>

Production backups

For the single-VM production layout, install infra/compose/backup-prod.sh as /usr/local/sbin/prea-backup and enable the supplied prea-backup.timer. It creates a verified Postgres custom-format dump plus an application-data archive under /var/backups/prea, with mode 0600. It keeps up to PREA_BACKUP_KEEP verified snapshots (default 3), bounded by PREA_BACKUP_MAX_GB (default 50 GiB), and reserves PREA_BACKUP_MIN_FREE_GB (default 5 GiB) of free disk space. Before starting, it removes abandoned partial snapshots and excess history, then checks capacity by scaling only the previous appdata archive by appdata source growth, plus the previous database dump size. Snapshots without source-size metadata use the raw source size for their initial estimate. It preserves the newest verified snapshot when capacity is insufficient, caps output file sizes, and removes a failed run’s partial snapshot. Local snapshots alone do not survive losing the host; encrypt them and copy them off-site as described below.

The scheduled backup uses /srv/prea by default. For another deployment directory, set PREA_DEPLOY_DIR=/absolute/path/to/checkout in a root-owned /etc/prea/backup.env (mode 0600). The service reads this optional file on each run; the script uses the selected checkout’s .env and production Compose file. If the stack was started with a custom -p, set the same COMPOSE_PROJECT_NAME in that file. Set the retention and space settings there too. When updating an existing installation, reinstall the script and units, then run systemctl daemon-reload; pulling the repository does not update /usr/local/sbin copies.

For a dedicated backup disk, mount it persistently and bind its backups directory onto /var/backups in /etc/fstab. Add x-systemd.requires-mounts-for=/mnt/your-backup-volume to the bind mount options. Set PREA_BACKUP_REQUIRE_VOLUME=1 and add a service drop-in containing RequiresMountsFor=/var/backups and AssertPathIsMountPoint=/var/backups under [Unit]. The script then refuses to fall back to root storage if the mount is missing or points at the root filesystem.

Encrypted and off-site backups

Backups are encrypted with age to public recipients only, so the host can write backups it cannot read. Generate the key pair on a machine you control, not on the server, and keep the identity file (the private key) offline, ideally with a second recipient held by another operator:

age-keygen -o prea-backup.key   # prints "Public key: age1..."

harden-host.sh installs age; on other hosts run apt-get install age. Then set the public keys in /etc/prea/backup.env, separated by spaces, or point at a recipients file with one key per line:

PREA_BACKUP_AGE_RECIPIENTS=age1exampleprimaryrecipient age1examplerecoveryrecipient
# or: PREA_BACKUP_AGE_RECIPIENTS_FILE=/etc/prea/backup-recipients.txt

With recipients set, each snapshot holds prea.dump.age, appdata.tgz.age, SOURCE_SIZE_KB, and SHA256SUMS. pg_dump and tar stream straight into age, so plaintext never touches disk. Verification needs no private key: pg_restore --list and tar -t read the same plaintext stream while it is encrypted, each file must carry an age header, and the checksums cover the encrypted files.

Without recipients the backup stays plaintext as before, logs a warning, and the host check alerts that the latest backup is not encrypted. After the first encrypted backup, a run without recipients fails instead of quietly writing plaintext. Set PREA_BACKUP_ALLOW_PLAINTEXT=1 only to accept plaintext on purpose; it also silences the host-check alert.

For an off-site copy, install rclone and set PREA_BACKUP_OFFSITE_REMOTE to any rclone remote and path, such as an S3-compatible bucket or an SFTP server. Configure the remote with rclone’s own settings in the same file, for example:

PREA_BACKUP_OFFSITE_REMOTE=offsite:prea-backups
PREA_BACKUP_OFFSITE_KEEP=14
RCLONE_CONFIG_OFFSITE_TYPE=s3
RCLONE_CONFIG_OFFSITE_PROVIDER=Other
RCLONE_CONFIG_OFFSITE_ENDPOINT=https://s3.example.com
RCLONE_CONFIG_OFFSITE_ACCESS_KEY_ID=replace-me
RCLONE_CONFIG_OFFSITE_SECRET_ACCESS_KEY=replace-me

After each verified local snapshot, the encrypted files are uploaded to <remote>/<stamp>/, checked with rclone check, and only then given SHA256SUMS, which marks the copy complete. Off-site copies require encryption. The remote keeps the newest PREA_BACKUP_OFFSITE_KEEP complete snapshots (default 14), independent of PREA_BACKUP_KEEP; only stamp-named directories are removed. Set it to 0 to leave deletion to the bucket’s lifecycle rules, so the credentials need list, read and write but no delete permission. A failed upload keeps the local snapshot but fails the service, so the backup alert fires.

Restoring an encrypted backup

Run these where the identity file lives, never leaving it on the server. Fetch a snapshot with rclone copy offsite:prea-backups/<stamp> ./<stamp> or scp, then run the drill, which checks the checksums and decrypts each file into pg_restore --list and tar -t without writing plaintext (it needs age, pg_restore of the server’s major version or newer, which is 18 for the production Compose file, and tar):

bash infra/compose/backup-drill.sh ./<stamp> prea-backup.key

Never extract appdata over the active volume. Files created after the snapshot would survive and mix with the restored database. Extract into a clean sibling directory, then preserve the old tree before replacing it. Pause automatic deploys and the updater during recovery. Stop any other processes that write to this database or volume. Keep the application stopped if any step fails.

Run these commands in one Bash session on the machine with the identity file. Replace the snapshot and volume names first. VOLUME must name the volume the API currently mounts at /data, which can differ when the stack uses another project name.

set -euo pipefail
umask 077
COMPOSE="sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml"
VOLUME=prea-prod_appdata

ssh deploy@app.example.com "cd /srv/prea && $COMPOSE stop api worker web"
ROLLBACK_DUMP="$(mktemp "$PWD/before-restore.dump.XXXXXX")"
if ! ssh deploy@app.example.com \
  "cd /srv/prea && $COMPOSE exec -T postgres sh -c 'pg_dump -Fc -U \"\$POSTGRES_USER\" \"\$POSTGRES_DB\"'" \
  > "$ROLLBACK_DUMP"; then
  printf 'Pre-restore dump failed. Do not use the partial file at %s. Keep the application stopped.\n' "$ROLLBACK_DUMP" >&2
  exit 1
fi

APPDATA="$(ssh deploy@app.example.com "sudo docker volume inspect -f '{{.Mountpoint}}' '$VOLUME'")"
RESTORE="$(ssh deploy@app.example.com "sudo mktemp -d '${APPDATA%/*}/prea-restore.XXXXXX'")"
ssh deploy@app.example.com "sudo mkdir '$RESTORE/staged'"
printf 'Record this rollback pair:\nDatabase dump: %s\nData tree on deploy@app.example.com: %s/previous\n' \
  "$ROLLBACK_DUMP" "$RESTORE"
age -d -i prea-backup.key <stamp>/appdata.tgz.age | ssh deploy@app.example.com \
  "sudo tar -xzf - --numeric-owner -C '$RESTORE/staged'"

age -d -i prea-backup.key <stamp>/prea.dump.age | ssh deploy@app.example.com \
  "cd /srv/prea && $COMPOSE exec -T postgres sh -c 'pg_restore --clean --if-exists --no-owner --exit-on-error --single-transaction -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\"'"

ssh deploy@app.example.com \
  "sudo mv -- '$APPDATA' '$RESTORE/previous' && sudo mv -- '$RESTORE/staged' '$APPDATA'"
ssh deploy@app.example.com \
  "cd /srv/prea && $COMPOSE up -d --force-recreate --wait --pull never api worker web"

The archive must extract successfully before the SQL import starts. The clean staging directory and the rollback tree sit beside the volume’s active data directory, on the same filesystem. Recreating the application containers makes them mount the replacement directory. --single-transaction rolls back SQL errors, but it cannot include the directory switch. After the import commits, a failed switch or startup can leave the database and active tree out of sync. If startup fails, stop api worker web again, including containers that did start.

$ROLLBACK_DUMP names a private, plaintext rollback dump. mktemp creates a new mode-0600 file for each attempt without overwriting before-restore.dump, earlier rollback dumps, or existing symlink targets. If the dump fails, the commands stop with the application still stopped. The printed partial file is not usable rollback data. Preserve earlier rollback pairs and fix the dump failure before starting another recovery attempt.

Record the printed database dump path together with its matching remote $RESTORE/previous tree path. Keep that pair until you have checked the restored application. Do not remove the Docker volume while you need the rollback tree. For rollback after SQL commits, keep the application stopped and restore the recorded $ROLLBACK_DUMP into a freshly created empty database. Move the failed active tree to a separate path for diagnosis, then move that attempt’s recorded $RESTORE/previous tree back to $APPDATA and recreate the application containers. If the first directory move failed, the old tree is still at $APPDATA. Inspect both paths before moving anything. Switching the tree alone does not roll back the database.

Plaintext snapshots use cat in place of age -d and the filenames prea.dump and appdata.tgz. Use the same clean-directory procedure.

To restore into a new stack, create the prea_app role before the procedure above. The dump grants to it, and on a new stack only the migration step would create it, so pg_restore stops without it. Do not run the migrations first: tables newer than the snapshot would block the restore. The procedure’s final start applies them.

ssh deploy@app.example.com "cd /srv/prea && $COMPOSE up -d --wait postgres"
ssh deploy@app.example.com "cd /srv/prea && $COMPOSE exec -T postgres sh -c 'psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"CREATE ROLE prea_app NOLOGIN NOSUPERUSER NOBYPASSRLS NOCREATEDB NOCREATEROLE\"'"

Point-in-time recovery

The nightly backup can lose up to a day. Point-in-time recovery streams the database’s write-ahead log (WAL) to S3-compatible object storage and takes a base backup every 24 hours, so the database can be restored to any moment in the last PITR_RETAIN_DAYS (default 7) and a lost server costs about a minute of writes. It covers the database only; keep the nightly backup for application data. It is off by default and needs the production Compose file with Postgres 18.

The optional pitr service streams WAL with pg_receivewal over the internal data network, using a replication-only login and a replication slot, and uploads from its own network, so Postgres keeps no internet access. Postgres closes a WAL segment at least every minute while anything is written (archive_timeout). Each segment and base backup is encrypted with age to public recipients before it leaves the container, as the nightly backup is, so the server cannot read its own backups. You can use the same recipients.

The slot keeps WAL that has not reached the sidecar, so a sidecar restart leaves no gap. Disk use stays bounded if uploads stall: the sidecar holds at most PITR_MAX_WAL_GB (default 4) of compressed WAL waiting to upload, then pauses streaming, and Postgres keeps at most that much again for the slot (max_slot_wal_keep_size). Beyond that Postgres drops the slot’s WAL, recovery has a gap, and the sidecar takes a new base backup once it reconnects.

Use a bucket of its own. Its credentials need to list, read, write and delete objects in it; deletes only remove backups older than the retention window. With PITR_RETAIN_DAYS=0 nothing is deleted, so no delete permission is needed and the bucket’s lifecycle rules handle expiry. Add to .env:

COMPOSE_PROFILES=pitr
PITR_DATABASE_PASSWORD=replace-with-openssl-rand-hex-24
PITR_BUCKET=example-prea-pitr
PITR_ENDPOINT=https://s3.example.com
PITR_REGION=example-region
PITR_ACCESS_KEY_ID=replace-me
PITR_SECRET_ACCESS_KEY=replace-me
PITR_AGE_RECIPIENTS=age1exampleprimaryrecipient age1examplerecoveryrecipient
# Optional: PITR_PREFIX (a key prefix), PITR_PATH_STYLE=true (path-style bucket addressing),
# PITR_RETAIN_DAYS, PITR_MAX_WAL_GB.

COMPOSE_PROFILES is comma-separated; with the updater it is updater,pitr. Leave PITR_ENDPOINT unset for AWS S3. Then turn it on from the checkout:

sudo infra/compose/pitr.sh enable

This creates the prea_pitr replication login, allows it in pg_hba.conf, sets archive_timeout and max_slot_wal_keep_size, and reloads Postgres without restarting it. The sidecar then takes its first base backup. Run enable again after changing PITR_DATABASE_PASSWORD or PITR_MAX_WAL_GB, or after moving Postgres to a new cluster. Uploads are grouped by the cluster’s system identifier, so a new cluster never overwrites an old one’s WAL.

The sidecar’s health check fails when an upload has waited over ten minutes, streaming is paused, nothing is streaming from the slot, or the last base backup is over 30 hours old. With PREA_MONITOR_COMPOSE_PROJECT set, the host check alerts on it. For details:

sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml exec pitr prea-pitr health

Restores run on any machine with Docker and this checkout. The identity comes on stdin and stays in memory inside the restore container. Prove the latest point restores, into a scratch volume that is removed afterwards. Run it after enabling, after changing the PITR scripts, and periodically:

sudo infra/compose/pitr.sh drill < prea-backup.key

Away from the server, set PITR_ENV_FILE to a file holding only the PITR_ storage settings. To recover production to a moment, stop everything that writes to the database and restore into its volume. The current cluster is moved aside inside the volume, not deleted. The time is read as UTC unless it names a zone:

COMPOSE="sudo docker compose --env-file .env -f infra/compose/docker-compose.prod.yml"
ssh deploy@app.example.com "cd /srv/prea && $COMPOSE stop api worker web pitr postgres"
ssh deploy@app.example.com \
  "cd /srv/prea && sudo infra/compose/pitr.sh restore '2026-01-31 09:41:00+00' prea-prod_postgres" \
  < prea-backup.key
ssh deploy@app.example.com "cd /srv/prea && $COMPOSE up -d --wait"

The restore uses the newest base backup that finished before the target, replays WAL up to it, and promotes the result, which starts a new timeline; the sidecar takes a fresh base backup for it. Pause CI deploys during recovery. Once the restored database checks out, remove the old cluster: $COMPOSE run --rm --no-deps --entrypoint sh postgres -c 'rm -rf "$PGDATA".before-restore-*'.

To turn it off, remove pitr from COMPOSE_PROFILES, then run sudo infra/compose/pitr.sh disable. It removes the sidecar, the slot (so Postgres stops keeping WAL for it), the login and the settings. Uploaded backups stay in the bucket.

Moving an account between installs

To move from Preá Cloud to your own install (or back), use Settings → Export data on the old install and Import data on the new one. The export is a zip of every space you belong to: bots, chats, files, memory, routines, skills, bot workspace files and, in spaces you own, the board. Content is decrypted inside the zip, so keep it private. Importing creates new spaces you own. Routines arrive switched off. Credentials and connections (model keys, apps, MCP servers, messaging) stay behind, so connect them again on the new install. Workspace files over 8 MiB are left out, the export must stay under 4 GiB, and the upload must finish within the server’s request timeout (5 minutes by default in Node).