Backup and Import
tamarackdb-backup keeps a copy of an instance’s events. tamarackdb-init --import builds a new database from the
history of another event store. To copy a TamarackDB instance as is, copy its database file while the server is
stopped.
Backup
tamarackdb-backup does one catch-up run and exits. Schedule it with cron or a systemd timer. Each run picks up where
the last one stopped, so a missed run is not a problem.
Settings
The settings live under a [backup] section, in their own file or in the server’s (see
Configuration).
tamarackdb-backup --default-config > backup-config.toml
tamarackdb-backup --config backup-config.toml| Key | Environment variable | Default | What it sets |
|---|---|---|---|
sourceUrl | TAMARACKDB_BACKUP_SOURCE_URL | none | The http:// or https:// address of the instance to copy |
sourceSocket | TAMARACKDB_BACKUP_SOURCE_SOCKET | none | The unix socket of an instance on the same host |
sourceTokenFile | TAMARACKDB_BACKUP_SOURCE_TOKEN_FILE | none | The file holding the source’s Bearer token, when the source has authTokenFile set |
dataDir | TAMARACKDB_BACKUP_DATA_DIR | backup | The directory the backup files go to. Never the server’s own dataDir |
pageLimit | TAMARACKDB_BACKUP_PAGE_LIMIT | 10000 | Events read per page. At most the source’s maxEventsPerPage |
- Set exactly one of
sourceUrlandsourceSocket. - The token file holds the token alone, like the server’s (see Security). It
must be readable by the backup’s user only (
chmod 600).
The source
- On the same host: set
sourceSocketto the server’s socket. The backup’s user must be allowed by the server’ssocketMode(see Security). - On another host: set
sourceUrlto thehttps://address of a reverse proxy in front of the server, withauthTokenFileset, and the same token insourceTokenFile(see Security).
How a run works
- The backup file is
tamarackdb-backup.sqlite, indataDir. - A run copies the events after the last one already in the file, page by page. Each event keeps its
sequenceand itstime. - Each page is written as soon as it’s read. A run that fails keeps the pages already written, and the next run resumes from there.
- A run fails, with a non-zero exit code and the error on stderr, when a page is cut short, or when a page takes more than 5 minutes.
- A backup file holds events only. Projections are left out: they can be rebuilt from events.
- The backup file is a regular TamarackDB database.
Scheduling
A cron entry:
*/5 * * * * /usr/local/bin/tamarackdb-backup --config /etc/tamarackdb/backup-config.toml >> /var/log/tamarackdb-backup.log 2>&1Or a systemd timer:
# /etc/systemd/system/tamarackdb-backup.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/tamarackdb-backup --config /etc/tamarackdb/backup-config.toml# /etc/systemd/system/tamarackdb-backup.timer
[Timer]
OnCalendar=*:0/5
Persistent=true
[Install]
WantedBy=timers.targetRestoring
Serve the backup file as the new instance.
Wait for the running backup to finish, and stop scheduling new ones.
Copy the file into a new data directory, under the name
tamarackdb.sqlite, owned by the server’s user:sudo install -d -o tamarackdb -g tamarackdb -m 700 /path/to/new-data sudo install -o tamarackdb -g tamarackdb -m 600 /path/to/backup/tamarackdb-backup.sqlite /path/to/new-data/tamarackdb.sqliteIf the backup was made with an older major version, migrate it (see Upgrade).
Start
tamarackdb-serverwithdataDirset to that directory.With the application stopped, restart every projector from the beginning. Start the application once the projections are rebuilt.
Import
An application that moves from another event store brings its history once, before it goes live:
sudo -u tamarackdb /usr/local/bin/tamarackdb-init --data-dir /path/to/data --import events.ndjsontamarackdb-init: created /path/to/data/tamarackdb.sqlite, imported 1000000 event(s), sequence 1 to 1000000- The import always creates a new database. It never adds to an existing one.
- Each event keeps its
sequenceand itstime. The firstsequencecan be 1 or more, and each next one is the one before plus 1. - A line that isn’t exactly an event stops the import, with its line number. So does an event whose type or tags break the rules for names and values.
- The import is all or nothing: when it stops, no database is left.
maxEventSizedoesn’t apply.- Compare the count and the first and last
sequencewith the source before starting the server. Then rebuild every projection.
The dump
One event per line, in NDJSON, in the shape of an event read with QUERY /events. Each line holds exactly these six
fields:
sequence: an integer, 1 or more.time: exactly the form2024-03-01T09:12:44.000000Z, in UTC, with six digits after the second.type: a non-empty string.identifiersandmetadata: objects,{}when empty, at most 20 entries each (see Concepts).payload: a string,""when empty.
{"sequence":1,"time":"2024-03-01T09:12:44.000000Z","type":"order-placed","identifiers":{"orderId":"o-1"},"metadata":{"tenantId":"acme"},"payload":"..."}