Backup and Restore

For operators planning backups, and for whoever is holding the pager when a restore actually happens. A restore is a workflow-level event, not just a database one — flows re-run, completed work rewinds, and side effects fire again. Know that before you need to.

What to Back Up

The shard databases, and nothing else. Dwarf keeps no state outside SQL: no local disk, no broker, no external service, no in-memory state that matters across a restart. A replica is disposable — everything durable is in the tables. So your backup strategy is entirely your database’s, and the engine adds no step to it.

Two things worth confirming rather than assuming:

  • Every shard is backed up, on the same schedule. A shard is an independent database and nothing reconciles them.
  • The migration ledger travels with the data. Dwarf records which migrations have run in a table in each shard database; a restore that produces data without it will attempt to re-create existing tables.

Restore All Shards to the Same Point, or None

A flow lives entirely on one shard, and subgraph trees, continuations and forks are all shard-local. So a partial restore does not corrupt individual flows — nothing is split across a boundary.

What it does produce is worse in a quieter way: a fleet where some tenants are days behind others, with no engine-level signal that this is the case. Every flow looks internally consistent. Nothing alarms. You find out from customers.

If you must restore one shard, treat the divergence as the incident: know which tenants sat on that shard (Query.FairnessKey is normally the tenant, and Query.Shard scopes a listing to one shard), and decide deliberately what happens to their work.

What a Restore Does to Flows

Three consequences, all of which follow from flows being durable and leases being wall-clock timestamps.

Every step that was running at backup time re-dispatches immediately. Leases in the restored data are all in the past, so they read as expired. Expect a burst of dwarf_steps_recovered_total and a corresponding burst of duplicate side effects the moment the fleet comes up. A step caught between completing and creating its successor does not re-run: the recovery sweep resumes it from its recorded output as the fleet comes up.

Work that completed after the backup point is rewound. Those flows return to whatever state they were in and run forward again from there. A flow that finished, and whose completion your application already acted on, will finish a second time.

Flows deleted after the backup point come back. Delete, Purge and DeleteOnCompletion removals are ordinary row deletions, so a restore resurrects them. If you deleted data in response to an erasure request, a restore reinstates it — that is a compliance concern, not just an operational one, and it is worth knowing which of your backups still contain a record you were asked to remove.

Before You Restore

  1. Stop the fleet. Restoring underneath running replicas mixes restored rows with live ones and produces states nothing in the engine expects. A replica also holds what it has read in memory, so one that keeps running through a restore can serve a flow data from after the restore point.
  2. Know your window. Everything between the backup point and now is lost or will re-run. Which of your tasks are safe to run twice? That question is answered by whether they’re idempotent (Writing Tasks → Idempotency), and the answer is the same one that governs ordinary crash recovery — a restore is just a much larger dose of it.
  3. Expect side effects to re-fire. Tasks with an idempotency key keyed on StepKey() survive this correctly, because a restored step keeps its key. Tasks without one will repeat their effect.
  4. Consider whether to restore at all. A flow that failed is kept for the archive’s retention, and one that stalled stays open until it is terminated, by you or by SetFlowExpiration, and is then archived like any other, so either is still present to fork. Recovering specific flows with Fork is often narrower and safer than rewinding every tenant to a point in the past.

After a Restore

Expect, and do not be alarmed by:

  • A burst of dwarf_steps_recovered_total as expired leases re-dispatch. This is the designed path.
  • A backlog spike, since everything that was in flight becomes runnable at once. dwarf_steps_dispatch_lag_seconds will climb and then drain.
  • Phantom peers. The replica registry is restored with the rest of the database, and a restored entry is as old as the backup. A replica stops counting 40 seconds after its last heartbeat, so entries from an older backup shrink no connection pool. They still slow handoffs: a replica taking over a shard’s dispatching waits about a second for each stale entry that joined before it, and unless replica identities are pinned with SetEngineID, every restored entry did. With k restored entries, each handoff during the fleet’s start-up takes about k seconds, until the engine deletes the entries after five minutes of healthy registry reads. From a backup taken less than 40 seconds before the restore, the entries also count as live replicas until they age out, shrinking every live replica’s connection pool meanwhile. To avoid both, clear the registry before starting the fleet: with the fleet still stopped, run DELETE FROM dwarf_peers; on every shard. Each replica registers itself again as it starts. See the Runbook if it does not resolve on its own.

What should not happen: dwarf_steps_write_failed_total, dwarf_steps_unwedged_total or dwarf_flows_orphaned_total moving. Those are alarms in a restored fleet exactly as in a healthy one — see the Runbook.

Failing Over to a Standby That Lost Writes

Promoting an asynchronous standby is a restore to a few seconds ago, done while the fleet is running. The writes the standby had not received are gone, and everything in What a restore does to flows applies to them. Restart every replica after such a failover, for the same reason the fleet is stopped for a restore: a replica that kept running holds data from before the failover in memory. A synchronous standby loses no committed write, so failing over to one needs no restart.

SQLite

A SQLite shard is its file, together with the -wal file beside it while the engine runs. A copy of the file taken while it is being written can be torn, or miss commits still in the -wal. Back it up live with either:

  • VACUUM INTO '/backups/dwarf_1.db', one statement that writes a consistent copy of the whole database, compacted;
  • SQLite’s online backup API (the sqlite3 shell’s .backup command), which copies page by page while writers continue.

To restore, stop the replica, delete the -wal and -shm files, put the backup in place of the file, and start the replica. A leftover -wal belongs to the database it was written against, and SQLite would apply it to the restored file. With several shards, restore every file from the same point, as for any dialect.

Test the Restore, Not Just the Backup

The engine-specific thing worth rehearsing is not whether the bytes come back — your database vendor has that covered — but what your workflows do when they re-run. Restore into a staging environment, bring a fleet up against it, and watch which side effects fire a second time. That is the part no database-level test exercises, and it is the part that will surprise you.

If a rehearsal shows duplicate side effects you cannot tolerate, the fix is idempotency keys in the affected tasks, not a change to how you back up.

Next: Data Handling & Security.

Backward Resharding Data Handling & Security Forward