Operating Dwarf on Dwarf

Production Checklist

Mon, 01 Jan 0001 00:00:00 +0000

Production Checklist

For developers and operators about to run Dwarf for real. Every item links to the guide that explains it. Most defaults are safe; what follows is the set that is not defaulted, or where the default is deliberately conservative and wrong for production.

Database

PostgreSQL, unless you have a reason. It needs no isolation-level tuning and runs fan-out deadlock-free at any concurrency. → Deployment
SQLite only for one replica at a modest flow rate, on a local-disk file with the recommended DSN (journal_mode(WAL), synchronous(NORMAL), auto_vacuum(FULL)) if flows must survive a restart, a retention that fits the disk, and live backups with VACUUM INTO; in memory they last only as long as the process. → Deployment
MySQL/MariaDB only: transaction-isolation = READ-COMMITTED, innodb_autoinc_lock_mode = 2 with binlog_format = ROW. Without the first, concurrent flow creation deadlocks on gap locks. → Runbook
MySQL/MariaDB only: interpolateParams=true in the DSN. Without it every statement with arguments costs two round trips, roughly doubling each step’s. → Deployment
SQL Server only: READ_COMMITTED_SNAPSHOT ON.
Every replica connects to a shard as one database user, allowed to create and drop tables in the engine’s schema. The engine keeps the archive in a table per day, creating and dropping them as it runs. → Deployment
Disk IOPS sized to the write rate, not just to capacity. Cloud providers often provision IOPS by disk size, and a throttled disk shows up as throughput collapse with everything apparently idle. → Deployment
PostgreSQL 16 and older: autovacuum_work_mem set to 1 GB. At a 64 MB default, vacuum on the step table rescans every index each time its memory fills, and falls behind. → Deployment
PostgreSQL: checkpoints sized for the write load. max_wal_size about 2 × peak WAL MB/s × 60 s, checkpoint_timeout 15 minutes. At the 1.5 GB default a busy shard checkpoints every few seconds and ran 30% slower with 8 KB payloads. → Deployment
ANALYZE/autovacuum tuned for dwarf_steps. It is a high-churn queue table whose row counts swing by orders of magnitude, and stale statistics can flip the selection query from an index scan to a sequential one. → Runbook
No statement, transaction or idle-in-transaction timeout on the engine’s database, its role or its connection string. That means PostgreSQL’s statement_timeout, transaction_timeout and idle_in_transaction_session_timeout, and MySQL’s max_execution_time. General database guidance often recommends them; leave them unset here.
- Some of the engine’s background queries grow with the backlog, so no value is safe for every backlog. A timeout that cuts them off can stop a database’s work from being dispatched at all.
- An idle-in-transaction timeout ends transactions while the engine is busiest, and the work is retried.
Long-running queries stay off the shard databases: analytics, reporting, ad hoc sessions. While a query runs for minutes, PostgreSQL cannot clean up the engine’s queue table behind it, and dispatch slows until it ends. A read replica with hot_standby_feedback on holds cleanup back the same way, and so does a lagging replication slot. → Runbook

Engine Configuration

Connection budgets suit each database. The default (250, rising to 500 for a distant database, never more than half its max_connections) suits databases of 4-32 vCPU. Set MaxConnections for a larger one or one shared with other heavy users, and set Weight on shards of unequal size so new flows follow capacity. → Deployment
The shard index-to-DSN map is identical on every replica and stable across restarts. A flow key encodes its shard, so a flow created on a shard a peer does not know about is unroutable there. → Resharding
SetEngineID pinned if your platform restarts replicas in place. Otherwise a crash-looping replica leaves a stale registry row per restart, and one that ranked as a shard’s dispatcher holds that slot until it ages out, leaving the shard fewer replicas running it. Must be unique among concurrently-live replicas. → Upgrading
GOMEMLIMIT set on engine hosts, a little below the memory the process may use. The engine keeps a tenth of it for large state fields, and without it Go’s garbage collector can let the heap reach about twice its live size. → Deployment
Logger, meter provider and tracer provider injected. All three default to no-ops, so an engine with nothing injected is silent — including its error logs. → Observability
Two engine replicas per shard, each at half the database’s vCPU (two 8-vCPU replicas per 32-vCPU database), so either can carry the shard alone. → Deployment

Lifecycle

Shutdown is called on termination, and the platform’s termination grace period exceeds your largest TimeBudget. Graceful shutdown waits for in-flight tasks; if the platform kills the process first, those steps are abandoned and their tasks re-run elsewhere. → Deployment
Deploys replace replicas one at a time, letting the fleet settle between steps, so pools re-divide cleanly rather than transiently over-connecting. → Upgrading
Dwarf version bumps are planned as a maintenance window, not a rolling deploy — v0.x makes no compatibility promise. → Upgrading

Observability

These five page someone. Each is documented as zero in a healthy engine: dwarf_steps_unwedged_total, dwarf_flows_orphaned_total, dwarf_steps_write_failed_total, dwarf_flows_reaped_total{reason="archive_failed"} and dwarf_flows_expire_failed_total. → Runbook
dwarf_steps_dispatch_lag_seconds is dashboarded — it is the honest backlog signal, better than a queue count.
Per-shard gauges aggregate with max per shard, not sum. dwarf_steps_dispatch_lag_seconds, dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds describe a shard, and more than one replica can report the same one; summing across replicas overstates the backlog and makes the age readings meaningless. → Observability
Someone knows where the runbook is before the first alarm, not after.

Data and Retention

The retention is set to what you must keep. Closed flows are archived and kept for SetRetention days (30 by default), then dropped a day at a time; open flows are never removed by the engine, unless they expire (below). → Data Handling
Abandoned flows are bounded, if they can occur. A flow left interrupted, or polling forever, stays in the live tables until it closes. SetFlowExpiration terminates any flow older than it. → Expiring Flows That Never End
The archive’s disk is sized for the retention: a day’s closed flows times the retention, per shard.
Backups cover every shard on the same schedule, and a restore has been rehearsed against a staging fleet — the part worth testing is what your workflows do when they re-run. → Backup and Restore
ListOpen and ListClosed (and their Query.Search) are gated by credential in your host. The engine has no notion of a caller, and the lists are the operations that return flow keys wholesale — which are capabilities. → Data Handling
Quotas enforced before Create. Beyond refusing any single value over 256 MB, the engine bounds nothing: not initial state size, not baggage, not fan-out width, not subgraph depth. → Data Handling, and a worked front door that enforces a quota in Building a Host
A flow’s final state fits in 256 MB. State can grow past it while the flow runs, but the final state holds every field at once. → Data Handling
MySQL/MariaDB: max_allowed_packet is at least 512M. → Deployment

Workflows and Tasks

Tasks are idempotent, keyed on f.StepKey() where the downstream supports it. Execution is at-least-once and two attempts can overlap. → Writing Tasks
Tasks respect their context deadline. A task that overruns its TimeBudget loses its lease to a peer and runs twice concurrently.
No PII in error text or terminate/cancel reasons. Error text is returned by both the lists and History — the readers that otherwise expose no state payloads — and the terminate or cancel reason by the lists. Both are substring-searchable. → Writing Tasks
Large integers read with typed accessors, or carried as strings. Beyond ±2^53 storage is exact and GetInt and when expressions are exact, but an untyped read rounds. And binary data base64-encoded — a NUL byte is rejected by PostgreSQL and accepted by everything else. → Writing Tasks
Every field whose writes should combine has an explicit reducer. There is no inference from a field’s name — an unwired sumTotal silently takes replace, not add. → Building Graphs
Flow deadlines your workflows rely on are in author space. An interrupted flow waits indefinitely unless SetFlowExpiration is set, and that terminates without running any of your code. → Writing Tasks

First Week

Watch dwarf_peer_replicas match your actual replica count.
Watch dwarf_steps_recovered_total — a steady trickle usually means tasks are overrunning their budget, or that your termination grace period is cutting drains short. Neither is broken, but both are worth knowing. → Runbook
Confirm the live tables stay level under load, by watching table size rather than disk usage — PostgreSQL holds its high-water mark. → Runbook
Re-read the sizing rules against what you actually observe. The published numbers assume cheap in-process tasks; a host dispatching over a network transport pays more engine CPU per step.

Next: Deployment.

Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Deployment

For operators standing Dwarf up in production, and for developers doing the wiring. It covers choosing and tuning a database, declaring shards, connection pools, workers, and running multiple replicas. Once it is running, Operating Dwarf takes over.

Configuration

All configuration is set with Set* methods, each returning an error. They split by whether the knob can change on a running engine: the live one (SetTimeBudget) takes effect immediately, even after Startup; the construction-time-only ones (SetShard, SetWorkers, SetStateCacheSize, SetMaxPriority, SetRetention, SetFlowExpiration, and the dependency-injection setters below) are rejected if called after Startup.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Observability

For developers and operators. Developers inject the three providers; operators read what comes out. This is the full instrument catalogue and how to interpret each one. For which of them to alert on, see Operating Dwarf; for what to do when one fires, the Runbook.

Dwarf emits structured logs, OpenTelemetry metrics, and distributed traces. All three go through standard providers you inject — the engine owns no exporter and adds no network connections. With nothing injected, everything degrades to a no-op, so unconfigured and test use pays nothing.

Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

Upgrading

For operators deploying an application that embeds Dwarf. It covers two different operations that get confused with each other — redeploying your own code, and moving to a new version of Dwarf — plus the compatibility obligation that outlives both: a long-running flow dispatches tasks from a graph frozen weeks ago.

Dwarf Is v0.x, and That Is the Headline

There is no backward-compatibility promise yet. The API and the database schema are both still evolving, and a v0.x release may change either — including by editing a migration an earlier release already shipped, which a database created by that earlier release does not pick up. Version skew between replicas is not tested and not supported.

Resharding

Mon, 01 Jan 0001 00:00:00 +0000

Resharding

For operators changing the set of database shards behind a running fleet. Read Deployment → Sharding first for what shards are and how many to run; this page is about changing the set after flows already exist.

The Constraint

A flow’s shard is part of its identity and never changes. Every flow key is {shard}-{hour}-{id}-{token}, so the shard index is encoded in the handle your callers hold, and it is what routes every subsequent operation. Everything derived from a flow stays on its shard for life: subgraph children, continuations, and forks are all shard-pinned.

Backup and Restore

Mon, 01 Jan 0001 00:00:00 +0000

Backup and Restore

For operators planning backups, and for whoever is holding the pager when a restore actually happens. A restore is a workflow-level event, not just a database one — flows re-run, completed work rewinds, and side effects fire again. Know that before you need to.

What to Back Up

The shard databases, and nothing else. Dwarf keeps no state outside SQL: no local disk, no broker, no external service, no in-memory state that matters across a restart. A replica is disposable — everything durable is in the tables. So your backup strategy is entirely your database’s, and the engine adds no step to it.

Data Handling & Security

Mon, 01 Jan 0001 00:00:00 +0000

Data Handling & Security

For developers and operators who have to answer for what the engine stores: a compliance review, a deletion request, a decision about who may hold a flow key. It covers what lands in the database, which of it is reachable by free-text search, what telemetry carries, and how deletion actually works.

What the Engine Stores

Everything durable is in the SQL databases you provide. There is no other store, no local disk state, and no external service. Two tables carry data you supply.

Inspecting and Repairing a Flow

Mon, 01 Jan 0001 00:00:00 +0000

Inspecting and Repairing a Flow

For operators and developers looking into a flow that failed, stalled or did something unexpected, and fixing it. The Runbook tells you when to do this; this page shows how.

The example is a payout flow with two tasks, Validate and Pay, where Pay rejects amounts over a limit. A flow was created with an amount of 5,000, and it failed.

1. Find the Flow

ListClosed finds closed flows by status, workflow, task, tenant and free text, within a window on when each was archived; ListOpen does the same for flows still running or interrupted:

Runbook

Mon, 01 Jan 0001 00:00:00 +0000

Runbook

For operators responding to a live incident. Each entry is a symptom you can observe from a dashboard or an alert, followed by how to confirm it and what to do. For the orientation — what the engine repairs on its own, and on what schedule — see Operating Dwarf. Timings quoted here are the engine’s defaults.

Before anything else: check whether it is already being handled. Most faults are absorbed. A database outage shorter than a step’s lease produces latency and no errors at all; a dead replica’s work is re-dispatched within the flow’s time budget plus about a minute and a half, including a step it had finished but not yet advanced. If the symptom is “things are slow” rather than “a counter fired,” start at Backlog is growing and resist restarting replicas — a restart discards in-flight work that was about to complete and adds a phantom replica to the registry.