Dwarf Documentation on Dwarf

Building Graphs

Mon, 01 Jan 0001 00:00:00 +0000

Building Graphs

For developers defining workflows. This is the graph-authoring API: the tasks, transitions, conditions and reducers that decide what runs when. Writing the task bodies themselves is Writing Tasks.

A workflow.Graph is the definition of a workflow. You build one in code and hand it to the engine via your host’s LoadGraph. This guide covers the full graph-authoring API.

Tasks

A graph is a set of tasks connected by transitions. Bind a task with SetURL, giving a task name (its identity in the graph) and a URL (the address your host’s ExecuteTask resolves to reach it):

Cloud Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Cloud Benchmarks

For operators sizing a deployment. This page answers how much one database serves, what adding shards and engine replicas does, and what the engine itself costs. Every number was measured, and the configuration that produced it is stated beside it.

The Short Version

What it delivers

One 32 vCPU PostgreSQL database runs 18,900–24,300 steps/s with the archive on, 1.6–2.1 billion steps a day, at the engine’s shipped defaults and with the database’s checkpoints sized as recommended. The range is the instance: the slower one spent 22% more database CPU per step and took 25% longer per log flush. Check yours (Method Notes).
Payload is paid in write-ahead log. With 8 KB of JSON per step the same database runs ~10,300 steps/s, with 64 KB ~2,800 (~176 MB/s of payload). Every flush of the log costs about 0.8 ms plus its bytes, so the bytes a step logs set the rate.
Size the database’s checkpoints. With 8 KB payloads, a 20 GB max_wal_size ran 42% more steps/s than the 1.5 GB default.
32 vCPU is the most economical database size, $0.036–0.047 per million steps at full load. Smaller databases run out of CPU; larger ones are held back by the write-ahead log’s flush, so past 32 vCPU you add shards.
The engine is cheap. A shard’s work takes about 5 engine cores. Two 8 vCPU engine hosts per 32 vCPU database serve it at about a third of their CPU each, and either can carry it alone.
Shards add throughput linearly. Four 32 vCPU shards ran 91,800 steps/s, 3.8 times one shard, each shard at close to what one shard runs alone.
No flow failed in any run. In the 243 runs behind this page, across 40 sessions, no flow errored and no step was left stuck. The engine recovered a crashed worker’s step only in the runs that killed engines on purpose.
Failover takes about 40 seconds and loses nothing. With every dispatcher killed at once, dispatch paused for ~40 s, standby replicas took over, and every flow completed.
Replicas add availability, not cost. One, two or four replicas on a shard deliver the same throughput within 6%. Each shard is served by two of them, and any others stand by to take over.
Long tasks cost no throughput, and mixed durations share one engine. With 8-second tasks the engine delivered the full offered rate, and a mix of 5 ms, 200 ms and 8 s tasks ran at the latency each class predicts. 10,000 concurrent 60-second tasks ran on under one engine core.
Large state works up to the database column’s limit. 250 MB per step completed cleanly, at ~45 MB/s for one flow and ~135 MB/s for four.
A day of varied load ran without falling behind. 4.5 hours of mixed shapes, payloads, task durations and priorities, with bursts above capacity, a rolling restart and a database restart: flows completed at the rate they arrived, bursts cost only the lowest priorities a wait, and the live tables levelled off at 10 GB (A Day of Varied Load).
A growing archive costs nothing measurable. A 32 vCPU database ran saturated until its archive held 100 million step rows, and ran at its empty rate the whole way, while the live tables stayed at ~3 GB. Archived reads took 7–20 ms at every size (Database Volume).

What it means for sizing

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Getting Started

For developers meeting Dwarf for the first time. It installs the module, explains the two callbacks you must implement, and gets a flow running against the in-process test harness — no database required.

Install

go get github.com/microbus-io/dwarf

Dwarf requires Go 1.27.3+ and a SQL database for production use (PostgreSQL, MySQL/MariaDB, or SQL Server). For tests and local experiments it uses SQLite in-memory automatically — no database to set up.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Glossary

For everyone. Each term in one sentence, with a link to where it is explained in full.

Archive. Where a closed flow moves about a minute after it closes, readable by its key and listed by ListClosed until the retention drops it. Driving Flows

Baggage. Context your application attaches to a flow, such as the acting user, that every task can read but no task can change. Writing Tasks

Production Checklist

Mon, 01 Jan 0001 00:00:00 +0000

Production Checklist

For developers and operators about to run Dwarf for real. Every item links to the guide that explains it. Most defaults are safe; what follows is the set that is not defaulted, or where the default is deliberately conservative and wrong for production.

Database

PostgreSQL, unless you have a reason. It needs no isolation-level tuning and runs fan-out deadlock-free at any concurrency. → Deployment
SQLite only for one replica at a modest flow rate, on a local-disk file with the recommended DSN (journal_mode(WAL), synchronous(NORMAL), auto_vacuum(FULL)) if flows must survive a restart, a retention that fits the disk, and live backups with VACUUM INTO; in memory they last only as long as the process. → Deployment
MySQL/MariaDB only: transaction-isolation = READ-COMMITTED, innodb_autoinc_lock_mode = 2 with binlog_format = ROW. Without the first, concurrent flow creation deadlocks on gap locks. → Runbook
MySQL/MariaDB only: interpolateParams=true in the DSN. Without it every statement with arguments costs two round trips, roughly doubling each step’s. → Deployment
SQL Server only: READ_COMMITTED_SNAPSHOT ON.
Every replica connects to a shard as one database user, allowed to create and drop tables in the engine’s schema. The engine keeps the archive in a table per day, creating and dropping them as it runs. → Deployment
Disk IOPS sized to the write rate, not just to capacity. Cloud providers often provision IOPS by disk size, and a throttled disk shows up as throughput collapse with everything apparently idle. → Deployment
PostgreSQL 16 and older: autovacuum_work_mem set to 1 GB. At a 64 MB default, vacuum on the step table rescans every index each time its memory fills, and falls behind. → Deployment
PostgreSQL: checkpoints sized for the write load. max_wal_size about 2 × peak WAL MB/s × 60 s, checkpoint_timeout 15 minutes. At the 1.5 GB default a busy shard checkpoints every few seconds and ran 30% slower with 8 KB payloads. → Deployment
ANALYZE/autovacuum tuned for dwarf_steps. It is a high-churn queue table whose row counts swing by orders of magnitude, and stale statistics can flip the selection query from an index scan to a sequential one. → Runbook
No statement, transaction or idle-in-transaction timeout on the engine’s database, its role or its connection string. That means PostgreSQL’s statement_timeout, transaction_timeout and idle_in_transaction_session_timeout, and MySQL’s max_execution_time. General database guidance often recommends them; leave them unset here.
- Some of the engine’s background queries grow with the backlog, so no value is safe for every backlog. A timeout that cuts them off can stop a database’s work from being dispatched at all.
- An idle-in-transaction timeout ends transactions while the engine is busiest, and the work is retried.
Long-running queries stay off the shard databases: analytics, reporting, ad hoc sessions. While a query runs for minutes, PostgreSQL cannot clean up the engine’s queue table behind it, and dispatch slows until it ends. A read replica with hot_standby_feedback on holds cleanup back the same way, and so does a lagging replication slot. → Runbook

Engine Configuration

Connection budgets suit each database. The default (250, rising to 500 for a distant database, never more than half its max_connections) suits databases of 4-32 vCPU. Set MaxConnections for a larger one or one shared with other heavy users, and set Weight on shards of unequal size so new flows follow capacity. → Deployment
The shard index-to-DSN map is identical on every replica and stable across restarts. A flow key encodes its shard, so a flow created on a shard a peer does not know about is unroutable there. → Resharding
SetEngineID pinned if your platform restarts replicas in place. Otherwise a crash-looping replica leaves a stale registry row per restart, and one that ranked as a shard’s dispatcher holds that slot until it ages out, leaving the shard fewer replicas running it. Must be unique among concurrently-live replicas. → Upgrading
GOMEMLIMIT set on engine hosts, a little below the memory the process may use. The engine keeps a tenth of it for large state fields, and without it Go’s garbage collector can let the heap reach about twice its live size. → Deployment
Logger, meter provider and tracer provider injected. All three default to no-ops, so an engine with nothing injected is silent — including its error logs. → Observability
Two engine replicas per shard, each at half the database’s vCPU (two 8-vCPU replicas per 32-vCPU database), so either can carry the shard alone. → Deployment

Lifecycle

Shutdown is called on termination, and the platform’s termination grace period exceeds your largest TimeBudget. Graceful shutdown waits for in-flight tasks; if the platform kills the process first, those steps are abandoned and their tasks re-run elsewhere. → Deployment
Deploys replace replicas one at a time, letting the fleet settle between steps, so pools re-divide cleanly rather than transiently over-connecting. → Upgrading
Dwarf version bumps are planned as a maintenance window, not a rolling deploy — v0.x makes no compatibility promise. → Upgrading

Observability

These five page someone. Each is documented as zero in a healthy engine: dwarf_steps_unwedged_total, dwarf_flows_orphaned_total, dwarf_steps_write_failed_total, dwarf_flows_reaped_total{reason="archive_failed"} and dwarf_flows_expire_failed_total. → Runbook
dwarf_steps_dispatch_lag_seconds is dashboarded — it is the honest backlog signal, better than a queue count.
Per-shard gauges aggregate with max per shard, not sum. dwarf_steps_dispatch_lag_seconds, dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds describe a shard, and more than one replica can report the same one; summing across replicas overstates the backlog and makes the age readings meaningless. → Observability
Someone knows where the runbook is before the first alarm, not after.

Data and Retention

The retention is set to what you must keep. Closed flows are archived and kept for SetRetention days (30 by default), then dropped a day at a time; open flows are never removed by the engine, unless they expire (below). → Data Handling
Abandoned flows are bounded, if they can occur. A flow left interrupted, or polling forever, stays in the live tables until it closes. SetFlowExpiration terminates any flow older than it. → Expiring Flows That Never End
The archive’s disk is sized for the retention: a day’s closed flows times the retention, per shard.
Backups cover every shard on the same schedule, and a restore has been rehearsed against a staging fleet — the part worth testing is what your workflows do when they re-run. → Backup and Restore
ListOpen and ListClosed (and their Query.Search) are gated by credential in your host. The engine has no notion of a caller, and the lists are the operations that return flow keys wholesale — which are capabilities. → Data Handling
Quotas enforced before Create. Beyond refusing any single value over 256 MB, the engine bounds nothing: not initial state size, not baggage, not fan-out width, not subgraph depth. → Data Handling, and a worked front door that enforces a quota in Building a Host
A flow’s final state fits in 256 MB. State can grow past it while the flow runs, but the final state holds every field at once. → Data Handling
MySQL/MariaDB: max_allowed_packet is at least 512M. → Deployment

Workflows and Tasks

Tasks are idempotent, keyed on f.StepKey() where the downstream supports it. Execution is at-least-once and two attempts can overlap. → Writing Tasks
Tasks respect their context deadline. A task that overruns its TimeBudget loses its lease to a peer and runs twice concurrently.
No PII in error text or terminate/cancel reasons. Error text is returned by both the lists and History — the readers that otherwise expose no state payloads — and the terminate or cancel reason by the lists. Both are substring-searchable. → Writing Tasks
Large integers read with typed accessors, or carried as strings. Beyond ±2^53 storage is exact and GetInt and when expressions are exact, but an untyped read rounds. And binary data base64-encoded — a NUL byte is rejected by PostgreSQL and accepted by everything else. → Writing Tasks
Every field whose writes should combine has an explicit reducer. There is no inference from a field’s name — an unwired sumTotal silently takes replace, not add. → Building Graphs
Flow deadlines your workflows rely on are in author space. An interrupted flow waits indefinitely unless SetFlowExpiration is set, and that terminates without running any of your code. → Writing Tasks

First Week

Watch dwarf_peer_replicas match your actual replica count.
Watch dwarf_steps_recovered_total — a steady trickle usually means tasks are overrunning their budget, or that your termination grace period is cutting drains short. Neither is broken, but both are worth knowing. → Runbook
Confirm the live tables stay level under load, by watching table size rather than disk usage — PostgreSQL holds its high-water mark. → Runbook
Re-read the sizing rules against what you actually observe. The published numbers assume cheap in-process tasks; a host dispatching over a network transport pays more engine CPU per step.

Next: Deployment.

v0.14.0

Mon, 01 Jan 0001 00:00:00 +0000

v0.14.0 — A Daily Archive for Closed Flows, Lists by State, Continue by Key

Upcoming release: these notes may change before it is published on GitHub Releases.

Closed flows now move to an archive about a minute after they close, kept by the day for a retention you set and dropped a day at a time, so the live tables that every step reads hold the open flows and little else. An archived flow reads by its key as before. The lists are named by a flow’s state - ListOpen and ListClosed - and SetFlowExpiration can end flows that would otherwise never close. This release edits migrations that earlier releases shipped, so existing databases cannot be upgraded in place: start v0.14.0 on fresh databases (see Upgrading). Nothing detects an old database: Startup succeeds on one, and ListOpen then fails on every call.

A Day of Varied Load

Mon, 01 Jan 0001 00:00:00 +0000

A Day of Varied Load

For operators deciding whether one database carries their traffic through a day, not just at a steady rate. Four and a half hours of mixed flows on one PostgreSQL database, with bursts above its capacity, a rolling restart of the engines and a restart of the database itself. The steady-rate measurements are in Cloud Benchmarks, whose database and settings this run shares.

Concepts

Mon, 01 Jan 0001 00:00:00 +0000

Concepts

For everyone — developers and operators alike. This is the shared vocabulary the rest of the documentation assumes. Read it once, before the guides.

Five terms run through all of Dwarf: graph, task, flow, step and reducer. This page defines them. What happens at runtime (which process runs what, and how crashes are recovered) is covered next, in How Dwarf Works.

Graph

A graph is the definition of a workflow: which tasks exist, how they connect, and how parallel branches merge. It’s a directed graph of tasks and transitions between them, with an entry point and optional per-field reducers. You build one in code with workflow.NewGraph and the Add* methods; it carries a display name, and your host’s LoadGraph resolves it by a workflow URL you choose. Graphs are reusable templates — many flows run the same graph.

Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Deployment

For operators standing Dwarf up in production, and for developers doing the wiring. It covers choosing and tuning a database, declaring shards, connection pools, workers, and running multiple replicas. Once it is running, Operating Dwarf takes over.

Configuration

All configuration is set with Set* methods, each returning an error. They split by whether the knob can change on a running engine: the live one (SetTimeBudget) takes effect immediately, even after Startup; the construction-time-only ones (SetShard, SetWorkers, SetStateCacheSize, SetMaxPriority, SetRetention, SetFlowExpiration, and the dependency-injection setters below) are rejected if called after Startup.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

FAQ

For everyone. Short answers, each with a link to the full explanation.

Does Dwarf Need Its Own Server?

No. Dwarf is a library that runs inside your application. The only infrastructure is the SQL database it stores flows in, and replicas of your application coordinate through that database. See How Dwarf Works.

Do Tasks Have to Be Written in Go?

No. The engine and its host are Go, but the host can reach tasks written in any language, for example over HTTP with JSON in and out. See Tasks in Other Languages.

v0.13.0

Mon, 01 Jan 0001 00:00:00 +0000

v0.13.0 — The Creation Hour in Every Flow and Step Key

Released 2026-10-31. GitHub release.

Flow and step keys carry the flow’s creation hour, and a step key names its flow. This is groundwork for retaining ended flows cheaply. This release edits migrations that earlier releases shipped, so existing databases cannot be upgraded in place: start v0.13.0 on fresh databases (see Upgrading).

Breaking Changes

Flow keys are {shard}-{hour}-{flowID}-{token}. {hour} is the flow’s creation hour, in whole hours since 2000-01-01 00:00 UTC, taken from the database clock. This applies everywhere a flow key appears:

Writing Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Writing Tasks

For developers writing the code that runs inside a workflow. It covers the workflow.Flow carrier a task receives: reading and writing state, the control signals, and handling failure. The graph that decides which task runs is Building Graphs.

A task is a function your host’s ExecuteTask runs. It receives a *workflow.Flow — the carrier holding the step’s state and control signals — does its work, writes outputs back onto the flow, and returns. This guide covers the Flow API a task uses.

Building a Host

Mon, 01 Jan 0001 00:00:00 +0000

Building a Host

For developers putting Dwarf behind a real application. It turns Access Control Is the Host’s Job into code. Read Getting Started first for the two-method Host interface.

The engine has no notion of a caller: no authentication, no authorization, no tenants. Every access decision belongs to the layer you build around it. This page builds that layer for a multi-tenant service, as one type, App, that is three things at once:

Coming from Temporal or LangGraph

Mon, 01 Jan 0001 00:00:00 +0000

Coming from Temporal or LangGraph

For developers who know Temporal or LangGraph and want to find their footing in Dwarf quickly.

From Temporal

Temporal	Dwarf	Notes
Workflow definition	Graph	A declarative graph of tasks and transitions instead of imperative code, so nothing is replayed and there are no determinism constraints.
Workflow execution	Flow
Activity	Task	A plain function, reached by URL through your host.
Worker	Your application, with the engine embedded	There is no separate Temporal Service to run.
Task queue	Your host	The host decides how each task URL is reached.
Retry policy	`f.Retry`	Called by the task, which decides what is retryable.
Timer	`f.Sleep`
Signal	`Resume`	Delivers input to a task waiting in `f.Interrupt`.
Query	`Snapshot`	Reads a flow’s status and state.
Child workflow	Subgraph
Continue-as-new	`Continue`	Starts a new flow from a completed flow’s final state. Your application keeps the chain of flow keys.
Versioning and patching	Frozen graphs	Each flow keeps the graph it started with. See Graph Evolution.
Visibility and search attributes	`ListOpen` and `ListClosed`	Filter by status, workflow, task, fairness key, time and free text. Closed flows are listed from the archive, kept for the retention.
Task queue priority and fairness	Priority and fairness keys

The main shift is where orchestration logic lives. In Temporal it is code that the server replays; in Dwarf it is a graph, with loops, conditions, fan-out and computed jumps, and anything more complex goes in a task, which is plain code. See Building Graphs.

How Dwarf Works

Mon, 01 Jan 0001 00:00:00 +0000

How Dwarf Works

For everyone who wants to know what actually happens at runtime: which process runs what, where the state lives, and what happens when something crashes. The terms defined here are used throughout the guides.

Dwarf is a library that runs inside your application process. There is no Dwarf server, broker or control plane: every copy of your application embeds the engine, and the copies coordinate only through the SQL databases they share.

Observability

Mon, 01 Jan 0001 00:00:00 +0000

Observability

For developers and operators. Developers inject the three providers; operators read what comes out. This is the full instrument catalogue and how to interpret each one. For which of them to alert on, see Operating Dwarf; for what to do when one fires, the Runbook.

Dwarf emits structured logs, OpenTelemetry metrics, and distributed traces. All three go through standard providers you inject — the engine owns no exporter and adds no network connections. With nothing injected, everything degrades to a no-op, so unconfigured and test use pays nothing.

Sizing & Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Sizing & Tuning

For operators who want to know how the engine sizes itself, or who need to override it. You don’t need this page to run Dwarf: the decisions are in Deployment, and the engine applies these numbers for you. The measurements behind them are in the Cloud Benchmarks.

Connection Pool

Each shard’s connection budget is a constant adjusted for distance and capped by the server: 250 connections up to 0.25 ms of round-trip time, rising evenly to 500 at 1 ms, and never more than half the server’s own max_connections (a fixed 8 on SQLite). The budget is shared by the shard’s two dispatching replicas, about 125 connections each at the default. The rules for overriding it are in Deployment.

v0.12.0

Mon, 01 Jan 0001 00:00:00 +0000

v0.12.0 — Reducers That Fold Everywhere, History Stored Once, Priority Bands That Spill

Released 2026-10-23. GitHub release.

v0.12.0 makes a reduced field mean the same thing wherever a task writes it. Every write to a field with a reducer is a contribution, folded through the reducer at every hand-off: the next step, the fan-in, the final state and Continue. A task writes only what it adds, and the outcome no longer depends on the graph’s shape. An agent loop’s history is stored once: a field under append, or a list a task keeps rewriting but only extends, is kept as references to the rows that hold its elements rather than as the whole history at every step. Priorities are now five bands that spill, so a better band no longer starves a worse one while workers sit idle. Beneath that, the engine does fewer round trips per step, a shard is dispatched by two replicas at a budgeted pool, and a step whose transition was lost to a crash is replayed rather than stranded.

Dialect Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Benchmarks

For developers and operators comparing SQL dialects or shard counts on their own hardware. These are single-host laptop-scale numbers — good for relative comparison, not for capacity planning. For absolute numbers and how to size a deployment, see Cloud Benchmarks.

The benchmarks in fixtures in the Dwarf repository measure end-to-end flow throughput and latency through the public API (Run = create + await) against whatever SQL dialect SEQUEL_TESTING_DSN points at. They measure what code review cannot tell you: how many flows and steps per second each dialect sustains at each shard count, and the dispatch-latency percentiles.

Tasks in Other Languages

Mon, 01 Jan 0001 00:00:00 +0000

Tasks in Other Languages

For developers whose tasks are, or need to be, written in a language other than Go. Read Writing Tasks first: everything there about state, outputs and errors still applies, only the transport changes.

The Dwarf engine is written in Go, and so is the host that embeds it. Your tasks don’t have to be. The host decides how each task is reached, so it can just as well call a service written in Python, TypeScript, Java or anything else, over HTTP or any other transport. Graphs are JSON too (workflow.Graph marshals to and from JSON), so LoadGraph can fetch them from a service in any language as well.

Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

Upgrading

For operators deploying an application that embeds Dwarf. It covers two different operations that get confused with each other — redeploying your own code, and moving to a new version of Dwarf — plus the compatibility obligation that outlives both: a long-running flow dispatches tasks from a graph frozen weeks ago.

Dwarf Is v0.x, and That Is the Headline

There is no backward-compatibility promise yet. The API and the database schema are both still evolving, and a v0.x release may change either — including by editing a migration an earlier release already shipped, which a database created by that earlier release does not pick up. Version skew between replicas is not tested and not supported.

v0.11.0

Mon, 01 Jan 0001 00:00:00 +0000

v0.11.0 — Graceful Cancel, Budget-Bound Shard Connections, Measured Pool Sizing

Released 2026-09-28. GitHub release.

v0.11.0 gives a flow a way to stop gracefully. The immediate, unconditional stop is renamed Terminate, and Cancel now delivers the cancellation to every step in progress as an error on its node’s onError transition, so a workflow can compensate, clean up, or carry on. The flow ends cancelled, failed, or completed, depending on what its handlers do. Beneath that, a shard’s connection budget no longer grows with the fleet. A few replicas per shard dispatch it and split its budget, and every other replica holds a two-connection reader pool there, so a host scaled wide for its own traffic can no longer over-connect the database. Pool sizes come from measured sweeps on six database sizes instead of a model, sleeping steps get an index of their own so they stop slowing every scheduling pass, and graph validation rejects field names that a when expression could never reference.

Fan-Out & Subgraphs

Mon, 01 Jan 0001 00:00:00 +0000

Fan-Out & Subgraphs

For developers building workflows that branch. Both features here are graph-authoring plus task-side API, so read Building Graphs and Writing Tasks first.

Two features let a workflow do more than a straight line: fan-out/fan-in runs work in parallel and merges the results, and subgraphs call another workflow as a child.

Fan-Out

A fan-out runs several branches concurrently from one task. There are two kinds.

Resharding

Mon, 01 Jan 0001 00:00:00 +0000

Resharding

For operators changing the set of database shards behind a running fleet. Read Deployment → Sharding first for what shards are and how many to run; this page is about changing the set after flows already exist.

The Constraint

A flow’s shard is part of its identity and never changes. Every flow key is {shard}-{hour}-{id}-{token}, so the shard index is encoded in the handle your callers hold, and it is what routes every subsequent operation. Everything derived from a flow stays on its shard for life: subgraph children, continuations, and forks are all shard-pinned.

v0.10.5

Mon, 01 Jan 0001 00:00:00 +0000

v0.10.5 — Update Dependencies

Released 2026-08-03. GitHub release.

Update dependencies to latest, including Sequel v1.11.1 which changes the sequel_pool_wait_count and sequel_pool_wait_duration_seconds gauges to counters and renames them appropriately.

Backup and Restore

Mon, 01 Jan 0001 00:00:00 +0000

Backup and Restore

For operators planning backups, and for whoever is holding the pager when a restore actually happens. A restore is a workflow-level event, not just a database one — flows re-run, completed work rewinds, and side effects fire again. Know that before you need to.

What to Back Up

The shard databases, and nothing else. Dwarf keeps no state outside SQL: no local disk, no broker, no external service, no in-memory state that matters across a restart. A replica is disposable — everything durable is in the tables. So your backup strategy is entirely your database’s, and the engine adds no step to it.

Driving Flows

Mon, 01 Jan 0001 00:00:00 +0000

Driving Flows

For developers integrating Dwarf into an application. This is the reference for the calls your code makes against a running engine — starting flows, reading them back, resuming and terminating them, and managing retention. It assumes you have an engine configured; standing one up in production is Operating Dwarf.

Every interaction with a flow is a method on *engine.Engine. They fall into a few groups: creating & running, inspecting, pausing & resuming, terminating & recovering, continuing, and retention. Flows are addressed by their flow key ({shard}-{hour}-{flowID}-{token}), returned at creation; steps by their step key ({shard}-{hour}-{flowID}-{stepID}-{token}), which names its flow too.

v0.10.3

Mon, 01 Jan 0001 00:00:00 +0000

v0.10.3 — A Fan-Out Node Can Route to END Instead of Fanning Out

Released 2026-08-03. GitHub release.

Fixes a non-terminating loop. A node that both fans out over an array and carries a withGoto/onError edge to END — the shape a task-driven loop takes, “fan out over what is pending, or end the flow when there is nothing left to do” — now ends the flow when the task steers to END.

Data Handling & Security

Mon, 01 Jan 0001 00:00:00 +0000

Data Handling & Security

For developers and operators who have to answer for what the engine stores: a compliance review, a deletion request, a decision about who may hold a flow key. It covers what lands in the database, which of it is reachable by free-text search, what telemetry carries, and how deletion actually works.

What the Engine Stores

Everything durable is in the SQL databases you provide. There is no other store, no local disk state, and no external service. Two tables carry data you supply.

Detecting Completion

Mon, 01 Jan 0001 00:00:00 +0000

Detecting Flow Completion

For developers whose application needs to know when a flow finished — to answer an HTTP request, send a notification, or trigger downstream work. It explains the three ways to learn a flow’s outcome and when to use each.

There are three ways to learn a flow’s outcome, and they suit different situations:

Await — block a live caller until the flow stops, then read the outcome.
Poll — Await with a bounded wait: on a ctx timeout it returns the flow’s current outcome instead of an error, so a caller whose own deadline is shorter than the flow can answer now and ask again.
Orchestration — model the follow-up work as tasks inside a workflow that calls the real work as a subgraph, so the reaction to success or failure is itself durable.

Pick Await when a caller is standing by and can wait for the result. Pick Poll when the caller is bounded by its own deadline — an HTTP handler backing a status endpoint — but the flow is not. Pick orchestration when the reaction must happen reliably regardless of who is (or isn’t) waiting — a push notification, a downstream call, a compensation.

v0.10.2

Mon, 01 Jan 0001 00:00:00 +0000

v0.10.2 — NewEngineUnderTest Takes a Test Name, Not a testing.TB

Released 2026-08-03. GitHub release.

Test-support change only. engine.NewEngineUnderTest is now keyed by a plain test name instead of a testing.TB, so a host can stand up an isolated engine under its own harness — one that has no *testing.T to hand over. A name was all the constructor ever needed from one.

No schema migrations. No change to dispatch, scheduling, recovery, pool sizing, or anything else on the production path.

Inspecting and Repairing a Flow

Mon, 01 Jan 0001 00:00:00 +0000

Inspecting and Repairing a Flow

For operators and developers looking into a flow that failed, stalled or did something unexpected, and fixing it. The Runbook tells you when to do this; this page shows how.

The example is a payout flow with two tasks, Validate and Pay, where Pay rejects amounts over a limit. A flow was created with an amount of 5,000, and it failed.

1. Find the Flow

ListClosed finds closed flows by status, workflow, task, tenant and free text, within a window on when each was archived; ListOpen does the same for flows still running or interrupted:

Scheduling

Mon, 01 Jan 0001 00:00:00 +0000

Scheduling

For developers and operators. Developers set priority, fairness and time budgets per flow; operators need the same model to read a backlog. The behaviour described here is what the Runbook assumes you know.

Dwarf decides which pending step runs next, and how long each task may run. Most of this is automatic; this guide explains the knobs you control. How a step survives a crashed worker is covered in How Dwarf Works, and how a task retries a transient failure in Writing Tasks.

v0.10.0

Mon, 01 Jan 0001 00:00:00 +0000

v0.10.0 — Replicas Coordinate Through the Database, and Size Themselves

Released 2026-08-02. GitHub release.

This release rebuilds how a Dwarf fleet finds work, divides resources, and carries state. Both the API and the database schema change. Dwarf is v0.x, so upgrading is a maintenance window, not a rolling deploy — see docs/upgrading.md.

Breaking Changes

Host loses SignalPeers. The engine now sends nothing to its peers at all; hosts implement two methods, LoadGraph and ExecuteTask. Three signal kinds existed here and all three were deleted after measurement — a per-step work doorbell, a per-flow stop broadcast, and a fleet-membership nudge — because in each case the poll the engine already runs converges faster than the message.
SetShard(ShardSpec{Index, DSN, VirtualCPUs}) replaces SetDSN / SetNumShards / SetWorkersPerConn. You declare facts about each shard’s server; the engine derives the tuning. New SetEngineID pins a replica identity so restarts don’t inflate the observed fleet count.
workflow.State is a type, not a map[string]any — on FlowOutcome.State, Flow.Snapshot(), baggage, and interrupt/subgraph payloads. It holds each field as raw JSON and decodes on read. Flow.Delete → Flow.Del; Flow.SetState, Flow.Transform, MergeState, Graph.HasFanIn and the Duration() helpers are gone; Render() no longer returns an error; typed getters panic on a mistyped read instead of quietly returning a zero value.
NewEngineUnderTest(t) / SetTestName replace RunInTest / SetInTest.
Schema: a new dwarf_peers registry, new step columns (state_refs, engine_id), fixed-width CHAR tokens and statuses widened to VARCHAR, and SQL Server JSON payloads stored as VARBINARY — which ends a silent corruption of non-ASCII payloads on that dialect. The baseline migrations are edited in place, so a database created by v0.9.x cannot be migrated forward — recreate it.

Coordination Is the Database, and Nothing Else

Replicas discover each other through a dwarf_peers registry read per shard, not through the host’s transport. From one reading a replica derives two different numbers with deliberately opposite risk postures: how many ways to divide a shard’s connection budget (over-count and you collapse a database) and which replicas demonstrably dispatch work (over-count and you strand it). A replica that goes blind holds its last good reading rather than acting on silence.

Runbook

Mon, 01 Jan 0001 00:00:00 +0000

Runbook

For operators responding to a live incident. Each entry is a symptom you can observe from a dashboard or an alert, followed by how to confirm it and what to do. For the orientation — what the engine repairs on its own, and on what schedule — see Operating Dwarf. Timings quoted here are the engine’s defaults.

Before anything else: check whether it is already being handled. Most faults are absorbed. A database outage shorter than a step’s lease produces latency and no errors at all; a dead replica’s work is re-dispatched within the flow’s time budget plus about a minute and a half, including a step it had finished but not yet advanced. If the symptom is “things are slow” rather than “a counter fired,” start at Backlog is growing and resist restarting replicas — a restart discards in-flight work that was about to complete and adds a phantom replica to the registry.

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Testing

For developers writing tests for their own workflows. It covers the in-process harness Dwarf ships so your test suite can run real flows with no database and no transport.

Dwarf ships an in-process test harness so you can exercise real workflows — scheduling, fan-out, subgraphs, interrupts, retries — with no database to set up and no transport to stand up.

The Two Pieces

engine.NewEngineUnderTest(name) constructs an engine wired for testing: an isolated SQLite in-memory database (per name, and per shard for multi-shard tests), auto-dropped, with migrations run at Startup — no DSN. It takes a plain name rather than a testing.TB, so a harness with no *testing.T to hand over reaches it the same way a Go test does; pass t.Name() from a test. You call Startup(ctx) yourself, so any Set* configuration goes in between, and you own teardown: defer eng.Shutdown(ctx). The name is the isolation key — give several engines the same name to share one database, or distinct names to isolate them. It logs to stderr at Error for CI clues (DWARF_TEST_LOG_LEVEL overrides — info/debug for the flow-status play-by-play, silent/off to quiet it); a benchmark or fuzz target silences it outright with SetLogger(slog.New(slog.DiscardHandler)), since per-iteration logging would swamp the measurement.
hosts.Mux is an in-memory implementation of the Host interface. Register graphs and task functions on it, then register it with SetHost(mux); it implements LoadGraph and ExecuteTask from those registrations.

func TestCheckout(t *testing.T) {
 ctx := context.Background()
 mux := hosts.NewMux()

 g := workflow.NewGraph("Checkout")
 g.SetURL("Reserve", "inventory.reserve")
 g.SetURL("Charge", "billing.charge")
 g.AddTransition("Reserve", "Charge")
 g.AddTransition("Charge", workflow.End)
 mux.HandleGraph("checkout", g)

 mux.HandleTask("inventory.reserve", func(ctx context.Context, f *workflow.Flow) error {
 f.SetBool("reserved", true)
 return nil
 })
 mux.HandleTask("billing.charge", func(ctx context.Context, f *workflow.Flow) error {
 f.SetString("receipt", "r-123")
 return nil
 })

 eng := dwarf.NewEngineUnderTest(t.Name())
 defer eng.Shutdown(ctx)
 eng.SetHost(mux)
 eng.Startup(ctx)

 _, out, err := eng.Run(ctx, "checkout", map[string]any{"sku": "ABC"}, nil)
 if err != nil {
 t.Fatal(err)
 }
 if out.Status != workflow.StatusCompleted {
 t.Fatalf("status %s", out.Status)
 }
 if receipt := out.State.GetString("receipt"); receipt != "r-123" {
 t.Errorf("receipt %q", receipt)
 }
}

Mux.HandleGraph(workflowURL, graph) registers a graph under the URL you pass to Create or Run; HandleTask(taskURL, handler) registers a task under the URL bound with SetURL. The handler signature is the same func(ctx, *workflow.Flow) error you write in production.

v0.9.5

Mon, 01 Jan 0001 00:00:00 +0000

v0.9.5 — Seamster Extraction and Graph.Validate Race Fix

Released 2026-07-07. GitHub release.

This release fixes a concurrency bug in graph validation and moves the engine’s test-only instrumentation into a standalone module. Both live in one commit (0c113c3).

Fixed

Data race in workflow.Graph.Validate. Validate() built the fan-out to fan-in convergence map as a side effect stored on the graph. Because a graph is loaded once and shared across concurrent Create calls, two validations (or a validation racing a MarshalJSON) wrote and read that field at the same time, which the race detector flagged and which could corrupt the map. Validate() is now pure over the graph’s immutable definition and stores nothing on the graph.

Changed

Fan-in routing is now derived at dispatch, not persisted. The fan-out to fan-in map is a pure function of the graph structure, so the engine computes it per flow from the frozen graph definition (new internal package internal/faninmap) and caches it beside the parsed graph. It is no longer written into the stored graph JSON, so a workflow graph carries only its author-defined shape over the wire.

Patterns

Mon, 01 Jan 0001 00:00:00 +0000

Patterns

For developers looking for how to express a common need. Each recipe is short and links to the guide that covers its building blocks.

An LLM Agent Loop

An agent thinks, calls tools, and repeats until it has an answer. That is a loop in the graph: Think decides, and a switch either runs the tools or ends the flow.

g := workflow.NewGraph("Agent")
g.SetURL("Think", "https://agent/think")
g.SetURL("UseTools", "https://agent/use-tools")
g.AddTransitionSwitch("Think", "UseTools", "hasToolCalls")
g.AddTransitionSwitch("Think", workflow.End, "!hasToolCalls")
g.AddTransition("UseTools", "Think")

graph LR
 classDef task fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1
 classDef term fill:#e5f4f3,color:#434343,stroke:#32a7c1
 linkStyle default stroke:#32a7c1

 t0["Think"]:::task
 t0_switch{"switch"}:::term
 t1["UseTools"]:::task
 t1 --> t0
 t0 --> t0_switch
 t0_switch -->|"hasToolCalls"| t1
 _start(( )):::term --> t0
 t0_switch -->|"!hasToolCalls"| _end(( )):::term

Every turn of the loop is a durable step, so a crash resumes the agent where it was. For a conversation, end the flow when the agent has answered, and start the next turn with Continue: it carries the previous turn’s final state (the conversation so far) into a new flow. Your application keeps the conversation as a chain of flow keys, and continues the key the last Continue returned. To run several tool calls in parallel, have Think emit them as an array and fan out with forEach.

v0.9.4

Mon, 01 Jan 0001 00:00:00 +0000

v0.9.4 — Maintenance Release. No API or Behavior Changes to the Engine

Released 2026-07-06. GitHub release.

Fixes

Test stability: Fixed a MySQL-timing flake in TestFault_PollSizingErr. The faultPollSizingErr fault is now armed sticky rather than one-shot, so the background timerLoop’s own poll can no longer consume the fault before the test’s explicit poll — the poll reliably clamps to the near-term error-retry interval instead of occasionally sleeping the full maxPollInterval.

Dependencies

Bumped github.com/microbus-io/sequel v1.10.6 → v1.10.7.
Bumped golang.org/x/text v0.38.0 → v0.39.0 (indirect).

Docs

Removed the stale Go Report Card badge from README.md.

v0.9.3

Mon, 01 Jan 0001 00:00:00 +0000

v0.9.3 — Poll Operation

Released 2026-07-05. GitHub release.

The engine now exposes Poll(ctx, flowKey) alongside Await. Like Await, it blocks up to the context deadline for a flow to stop, but where Await treats a deadline expiring before the flow stops as a timeout error, Poll instead returns the flow’s current outcome — non-terminal, with Stopped() reporting false — and no error. This makes it the natural fit for bridging an open-ended durable flow to a bounded request (for example, answering an HTTP long-poll within its time budget and re-polling on the next request) without conflating “not done yet” with a real failure. The returned FlowOutcome gains a Stopped() helper to distinguish a terminal/interrupted outcome from one still running. As always, the underlying flow keeps running on the engine independent of the caller; a ctx timeout never tears it down.