How Dwarf Works

For everyone who wants to know what actually happens at runtime: which process runs what, where the state lives, and what happens when something crashes. The terms defined here are used throughout the guides.

Dwarf is a library that runs inside your application process. There is no Dwarf server, broker or control plane: every copy of your application embeds the engine, and the copies coordinate only through the SQL databases they share.

Architecture

%%{init: {'themeVariables': {'lineColor': '#32a7c1', 'clusterBkg': '#f4f2ef', 'clusterBorder': '#c1cccc'}}}%%
graph LR
    classDef node fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1
    classDef ext fill:#e5f4f3,color:#434343,stroke:#32a7c1

    subgraph R1["Your app, replica 1"]
        E1["Dwarf engine"]:::node --> H1["Your host<br/>LoadGraph / ExecuteTask"]:::node
    end
    subgraph R2["Your app, replica 2"]
        E2["Dwarf engine"]:::node --> H2["Your host"]:::node
    end
    H1 --> F["Local function"]:::ext
    H1 --> P["Python service<br/>over HTTP"]:::ext
    H2 --> Q["Message bus or RPC"]:::ext
    E1 <--> S1[("Shard 1<br/>PostgreSQL")]:::ext
    E1 <--> S2[("Shard 2<br/>PostgreSQL")]:::ext
    E2 <--> S1
    E2 <--> S2

There is nothing between the replicas: they see each other’s work, outcomes and membership only through the shard databases.

The Pieces

  • Host. The code you write to connect the engine to your world. It has two methods: LoadGraph returns a graph by its workflow URL, and ExecuteTask runs one task by its URL. How a task is reached is up to the host: a local function call, an RPC, a message on a bus, or an HTTP call to a service in another language. See Getting Started.
  • Engine. The Dwarf library inside your process. It decides which step runs next, calls your host to run it, and records the outcome in the database.
  • Replica. One running copy of your application, with its own engine. Run several for availability and CPU; they need no configuration to find each other. See Running Multiple Replicas.
  • Shard. One SQL database holding a share of the flows. Each flow lives on one shard for its whole life, and its flow key begins with the shard’s number. Add shards to scale write throughput. See Sharding.
  • Worker. A lightweight thread (a goroutine) inside the engine that claims a step, calls your host, and records the result. The engine grows workers on demand, so there is nothing to size. See Workers.
  • Dispatcher. A replica chosen, per shard, to run that shard’s steps. With several replicas, each shard picks two of them as dispatchers, so its connection budget is shared rather than multiplied by the fleet. See Dispatchers.

A Step’s Life

Every task execution is a step, and every step goes through the same cycle:

  1. Pending. The step is created in the database, with its input state, when the previous step finishes (or when the flow is created).
  2. Claimed. A worker claims the step in a single database statement, so no two workers can hold it. The claim is time-limited: it expires after the task’s time budget plus 30 seconds, so a step whose worker disappears doesn’t stay claimed forever. The operator pages and metrics call this time-limited claim a lease.
  3. Running. The worker calls your host’s ExecuteTask. While the task runs, the worker holds no database connection.
  4. Recorded. The worker writes the task’s output and, in the same transaction, creates the next steps the graph calls for.

Because the outcome is in the database before the next step starts, a flow can stop and resume at any step boundary, on any replica.

Crash Recovery

If a replica dies while one of its workers is running a task, the step’s lease eventually lapses, and a recovery sweep that runs every minute hands the step to another replica, which runs the task again:

%%{init: {'themeVariables': {'actorBkg': '#32a7c1', 'actorTextColor': '#f4f2ef', 'actorBorder': '#32a7c1', 'signalColor': '#434343', 'noteBkgColor': '#e5f4f3', 'noteBorderColor': '#32a7c1', 'noteTextColor': '#434343'}}}%%
sequenceDiagram
    participant A as Replica A
    participant DB as Shard database
    participant B as Replica B
    A->>DB: Claim the step (lease = time budget + 30s)
    A->>A: ExecuteTask
    Note over A: Crashes mid-task
    Note over DB: The lease expires
    B->>DB: Recovery sweep (every minute) returns the step to pending
    B->>DB: Claim the step (same step key)
    B->>B: ExecuteTask (the task runs again)
    B->>DB: Record the outcome, create the next step

So a task whose worker crashed runs again within its time budget, plus 30 seconds, plus up to a minute. Steps that were pending but not yet claimed are picked up by the surviving replicas within seconds.

A narrower case is repaired without re-running anything: if a replica dies after recording a step’s outcome but before creating the next step, the sweep resumes the flow from the recorded outcome, and the task does not run again. What the engine repairs on its own, and how fast, is summarized in Operating Dwarf.

A clean shutdown avoids all of this: Shutdown lets running tasks finish before the process exits. See Shutting Down.

At-Least-Once Execution

Crash recovery means a task can run more than once. In normal operation each task runs once, but after a crash, a lost database connection or a task that overruns its time budget, it runs again. Every durable execution engine works this way; the alternative would be to lose work.

So write tasks to be idempotent: running one twice should have the same effect as running it once. f.StepKey() gives each step a stable key that is the same on every attempt, which makes this easy: pass it to the downstream as an idempotency key, or record it alongside the side effect. See Idempotency.

Parking: Interrupt and Subgraph

A flow can wait for something without holding a worker. This is called parking:

  • Interrupt. A task calls f.Interrupt to wait for outside input, such as a human’s approval. The flow becomes interrupted and waits, for minutes or weeks, until your application calls Resume with the input. The task then runs again, and this time f.Interrupt returns the input.
  • Subgraph. A task calls f.Subgraph to run another workflow as a child flow. The parent step waits until the child finishes, then runs again with the child’s result.
%%{init: {'themeVariables': {'actorBkg': '#32a7c1', 'actorTextColor': '#f4f2ef', 'actorBorder': '#32a7c1', 'signalColor': '#434343', 'noteBkgColor': '#e5f4f3', 'noteBorderColor': '#32a7c1', 'noteTextColor': '#434343'}}}%%
sequenceDiagram
    participant C as Your application
    participant E as Engine
    participant T as Task
    C->>E: Create the flow
    E->>T: ExecuteTask
    T->>E: f.Interrupt(question) yields, and the task returns
    E-->>C: The flow is interrupted, with the question
    Note over E: Parked: no worker is held
    C->>E: Resume(flowKey, answer)
    E->>T: ExecuteTask (the same step runs again)
    T->>E: f.Interrupt returns the answer, and the task finishes
    E-->>C: The flow continues to the next step

Both are covered in Writing Tasks, and subgraphs in Fan-Out & Subgraphs.

Next: Building Workflows.

Backward Concepts Building Workflows Forward