v0.12.0 — Reducers That Fold Everywhere, History Stored Once, Priority Bands That Spill

Released 2026-10-23. GitHub release.

v0.12.0 makes a reduced field mean the same thing wherever a task writes it. Every write to a field with a reducer is a contribution, folded through the reducer at every hand-off: the next step, the fan-in, the final state and Continue. A task writes only what it adds, and the outcome no longer depends on the graph’s shape. An agent loop’s history is stored once: a field under append, or a list a task keeps rewriting but only extends, is kept as references to the rows that hold its elements rather than as the whole history at every step. Priorities are now five bands that spill, so a better band no longer starves a worse one while workers sit idle. Beneath that, the engine does fewer round trips per step, a shard is dispatched by two replicas at a budgeted pool, and a step whose transition was lost to a crash is replayed rather than stranded.

This release breaks the schema, the API and the wire format, so upgrading from v0.11.0 needs a maintenance window and a fresh database or hand-applied schema changes. Read “Upgrading from v0.11.0” first.

Highlights

• Reducers fold at every hand-off. A write to a reduced field is a contribution everywhere; Overwrite and Del replace. (Behavior change - see below.)
• An agent loop stores each message once. A 100-step loop of 1 KB messages stored ~37x less under append, and ~33x less when a task rewrites a growing list each step, on PostgreSQL.
• Priority bands that spill. Priorities run 1 to 5 by default (SetMaxPriority widens it up to 100), the default is 3, and capacity a better band leaves unused goes to the next one. (Breaking - see below.)
• Fewer round trips per step. The transition after a task runs as one pipelined batch on PostgreSQL, a flow is created in one batch, and the claim returns what the dispatch needs.
• Two dispatchers per shard, on a budgeted pool. Each shard’s connections are a budget of 250-500 by distance to the database, clamped by the server, and split between the two replicas that dispatch it. ShardSpec.MaxConnections is the lever.
• A lost transition is replayed. A replica that dies between a step’s completion and its transition no longer leaves the flow stranded; the step’s recorded outcome is committed without running the task again.
• A 256 MB ceiling on every value, refused with a 413 instead of failing deep inside a write.
• Many API renames and removals ahead of v1.0. (Breaking - see “API changes”.)

Upgrading from v0.11.0

Plan a maintenance window. Three things change at once: the schema, how stored writes to reduced fields are read, and how replicas share a shard. None of them can be rolled through one replica at a time.

1. Drain In-Flight Flows

Let every flow finish, or Terminate it, before the upgrade. In v0.11.0 a step’s write to a reduced field outside a fan-out branch was the field’s whole value; v0.12.0 reads every stored write as a contribution and folds it in. Anywhere a v0.11.0 step’s stored output is folded again, it would count twice: an interrupted step resumed, a pending Retry, a Subgraph re-entry, a replay, a cohort still converging, a tail merged into the final state. Terminal flows are unaffected; a Continue from one folds its final state the new way, which is correct.

2. Start from a Fresh Database, or Apply the Schema Changes by Hand

The migration scripts were edited in place, so a v0.11.0 database does not receive these changes on startup. A fresh database gets all of them. On an existing one, apply them during the drained window:

Columns added. Without them the engine cannot run:

• dwarf_flows.payload_ver: without it every Create fails.
• dwarf_flows.graph_hash, NOT NULL with no default. A hand-applied ALTER TABLE ... ADD on a populated table needs DEFAULT '', which the engine reads as “no hash” and falls back to reading the graph.
• dwarf_steps.advancing and the advance_* columns, state_reduced and payload_deferrals: without payload_deferrals every claim fails.
• dwarf_peers.draining and dwarf_peers.joined_at: without draining every registry read fails, and each replica dispatches every shard at the full budget.

Use the per-dialect definitions in the migration scripts for the ALTER TABLE ... ADD statements.

Column renamed: dwarf_flows.step_id is stopped_step_id. On an older database every List, Create and claim fails with “no such column”. Rename it after the drain, since a running flow’s old value is not a stopped step: ALTER TABLE dwarf_flows RENAME COLUMN step_id TO stopped_step_id; on SQL Server EXEC sp_rename 'dwarf_flows.step_id', 'stopped_step_id', 'COLUMN'.

Column type changed: dwarf_steps.subgraph_error is a JSON column (JSON, JSONB, VARBINARY(MAX) or TEXT by dialect, like subgraph_result), holding {} for no error. Alter it to the type the migration script declares.

PostgreSQL time columns are TIMESTAMP, not TIMESTAMPTZ. An older database keeps TIMESTAMPTZ, which on a server or role whose time zone is not UTC stores every time shifted by the offset (a flow.Retry give-up horizon then never expires). After the drain, alter each time column with ALTER TABLE ... ALTER COLUMN <col> TYPE TIMESTAMP USING <col> AT TIME ZONE 'UTC', or keep the server in UTC.

Rebuild idx_dwarf_steps_due on PostgreSQL, SQL Server and SQLite: drop it and recreate it from the migration script. Its key and its predicate changed. An older database keeps the old index, which the new scheduling statements no longer match, so they silently fall back to reading every pending row. MySQL’s is unchanged.

Indexes removed. An older database keeps them; they are harmless but written on every step:

• DROP INDEX idx_dwarf_steps_saturation
• DROP INDEX idx_dwarf_steps_status (... ON dwarf_steps on MySQL and SQL Server)

PostgreSQL storage setting: dwarf_flows no longer sets a fillfactor. A table created during this release cycle may carry fillfactor = 80, which is harmless; ALTER TABLE dwarf_flows RESET (fillfactor) reverts it for new pages.

3. Priorities Run 1 to 5

The default priority is 3, the midpoint, and SetDefaultPriority is gone. SetMaxPriority(n), at most 100 and set before Startup, widens the range; Create and Run reject a priority above it. Lower numbers still run first.

Work stored above the range is served after every level in it. That includes every flow created at the old default of 100: in flight if it was not drained, and anything a Continue, Fork or subgraph inherits from a finished one, since inheritance keeps the stored priority. No migration remaps it. To bring old threads back into range, run UPDATE dwarf_flows SET priority=3 WHERE priority>5, and the same on dwarf_steps, during the window.

4. Upgrade Every Replica, and the Host, Together

Mixed versions disagree on which replicas dispatch a shard and how its pool is sized, on how stored state references are read, and on the flow’s JSON form. In particular, a v0.11.0 replica reads v0.12.0’s list and forEach element references as no references at all: silent data loss. From this release on, a replica fails a step it cannot read rather than dropping data, and a replica that meets a flow in a newer payload format defers it to a newer replica.

Reducers Fold at Every Hand-Off

A write to a field with a reducer is a contribution, folded through the reducer at the next step, the fan-in, the final state and Continue. A read returns the combined value.

• A task writes only its contribution. f.Set("messages", []any{m}) under append adds m. A read-modify-write (Set(k, append(Get(k), m))) adds the history again.
• The result does not depend on graph shape. A step that changes nothing changes nothing, and wrapping a step in a one-item forEach gives the same result.
• Flow.Overwrite replaces a reduced field instead of contributing to it, and Del is Overwrite(k, nil). Both are refused on a reduced field inside a fan-out branch, where a reset has no defined meaning for the other branches.
• A step’s runs compose. On Retry, a resumed Interrupt or a Subgraph re-entry, the task sees its earlier runs’ writes and adds to them. Write after Interrupt or Subgraph, not before.
• A bare nil contributes nothing to a reduced field.
• Nested fan-outs: an inner cohort’s work reaches the outer fan-in and the final state. Before, it could be lost.
• A Fork at an inner fan-in hands on its cohort’s original part, whatever the overrides.
• Folds keep numbers exact, including integers past 2^53, and union compares numbers by value, so 1, 1.0 and a database’s respelling of the same number are one element.

An Agent Loop Stores Each Message Once

A field under append, and a list with no reducer that a task rewrites each step but only extends, are stored as references to the rows holding their elements, instead of the whole history at every step. A 100-step loop of 1 KB messages measured ~37x less storage under append and ~33x less for the rewrite, on PostgreSQL. A forEach branch references its element inside the array rather than storing a second copy.

dwarf_state_write_bytes gained column=state_refs, where those references are written; such a workload’s state series falls to near zero.

Priority Bands That Spill

The scheduler fills the best due band first, then gives whatever capacity remains to the next, and so on. A sparse but continuous better band therefore no longer starves every worse band while workers sit idle. Priority stays strict over capacity: worse work gets only what better work leaves unused.

Throughput and Dispatch

• Fewer round trips per step. The transition after a task is one pipelined batch on PostgreSQL, a single-successor transition is one batch, a flow is created in one batch, and the claim returns the flow row. Successors are inserted with multi-row inserts, and a flow-row lock no longer writes a row version on PostgreSQL and MySQL.
• Less allocation per step, and a graph cache that parses each graph once per replica.
• Two dispatchers per shard. The dispatchers are a window over the replicas in the order they joined, so the role spreads evenly across shards. A replica losing its dispatcher slot keeps its share of the pool until the steps it already claimed finish, and a draining replica hands its work to its peers.
• Connection budget. A shard’s budget is a constant of 250-500 connections, ramped by the round trip to the database and clamped by the server’s max_connections, shared by its dispatchers. ShardSpec.MaxConnections sets it; ShardSpec.Weight (new) weights flow placement across shards.
• Placement. A new flow goes to a shard the creating replica dispatches, when there is one. Any ShardSpec.Weight other than 1 switches the fleet to weighted placement over every non-cordoned shard.
• Scheduling at scale. The scheduling queries read only what each band needs, and the background sweeps no longer read the whole step table at once.

Reliability

• A lost transition is replayed. A replica that dies between a step’s completion and its transition used to leave the flow running with nothing to run. The completion now records what the transition needs, and recovery commits it without running the task again.
• Expired leases are recovered every minute (was 5), so a crashed replica’s claimed steps run again within the step’s TimeBudget + 30 s + 1 min. A dead worker’s completed step is recovered on the same bound.
• Contention on the completion write is retried, never the task.
• Shutdown no longer waits for background maintenance in flight; an abandoned pass is redone by a peer or the next start.
• Payload format. A flow records the format its state is stored in. A replica that meets a newer format answers 501 on the reads that cannot wait (Snapshot, Step, Fork) and defers the step to a newer replica.
• Unreadable rows fail loudly. A step whose stored state, changes or references cannot be read fails, and Snapshot and Step return the error, instead of carrying on with an empty value.

Limits

• Every value over 256 MB is refused: a 413 from Create, Run, Continue, Resume and Fork. A step whose output is too large fails at once, and a final state over the limit fails the flow with an empty final state. MySQL and MariaDB need max_allowed_packet of at least 512M, since a step row holds its state and its changes, each up to 256 MB.
• A Terminate or Cancel reason over 4 KB, or holding a NUL or invalid UTF-8, is a 400.
• Stored error messages are capped at 16 KB, with invalid UTF-8 replaced.
• Create returns a 400 for a FlowOptions.Priority beyond 32 bits or a TimeBudget over just under 24.8 days, and SetTimeBudget returns a 400 over that ceiling.

API Changes

Reducers and the carrier:

• workflow.Changes (contributions and overwrites) is a step’s writes. State.Apply and ApplyAll take it, and Changes.Compose combines two. Changes.Additions and AdditionSuffix are new.
• Removed: State.Merge, MergeReduce, MergeReduceAll, DelNils.
• RawFlow.RawChanges and SetRawChanges take Changes; RawFlow.SetReducers, RetryDelay and SetInBranch are new.
• A field name ending in ! or + cannot be written.
• Flow.SetChanges(source) drops its snapshot parameter and sets every field the source marshals, skipping empty omitempty/omitzero fields. A host’s generated wrappers drop the Snapshot() call.
• Flow.Get returns (ok bool, err error) (was error). Flow.ParseState is Parse.
• GotoRequested, RetryRequested, SleepRequested, InterruptRequested and SubgraphRequested moved from Flow to RawFlow.

Renames:

• Graph.SetEndpoint is SetURL.
• workflow.Node and Graph.Nodes() are Task and Tasks(); workflow.END is End.
• Transition’s WithGoto, OnError and Switch booleans are one Kind ("", TransitionSwitch, TransitionGoto, TransitionError; JSON kind). Stored graphs with the booleans still decode.
• FlowStep.SubWorkflowURL and SubWorkflowName are SubgraphURL and SubgraphName.
• FlowSummary.TaskName and Query.TaskName are StoppedTaskName, and the meaning changes: the task the flow stopped at, empty for a running flow, so the filter matches no running flow.
• ShardSummary.Shard is Index, and LatencyMs int is Latency time.Duration.
• engine.TestProxy and NewTestProxy are hosts.Mux and hosts.NewMux, in a new package: HandleGraph(workflowURL, graph), HandleTask(taskURL, func).
• Engine.FlowsTerminated is FlowsEnded.

Added:

• Engine.SetStateCacheSize, Engine.SetMaxPriority.
• FlowStep.SubgraphKey: History names a subgraph caller’s child flow.
• ShardSpec.MaxConnections and ShardSpec.Weight.

Removed:

• Engine.SetDefaultPriority.
• Engine.HistoryMermaid: render History with workflow.NewFlowRenderer(steps).WithLinks("step").Render().
• FlowRenderer.WithTitle and GraphRenderer.WithTitleLabel: no renderer draws a title.
• engine.TaskHandler: HandleTask takes a plain func.
• Engine.SetMaxOpenConns: the engine derives each shard’s connection budget, and ShardSpec.MaxConnections sets it instead, for example lower on a database shared with other workloads.
• Engine.DB, Engine.Seams and the exported fault, checkpoint and variable seam constants.
• Engine.SetRefillInterval.
• workflow.TerminalStatuses: use IsTerminalStatus.
• ShardSpec.VirtualCPUs: pools no longer follow vCPUs.
• ShardSummary.Steps and Flows: ShardInfo reports health only.

Wire and Serialized Formats

• The flow’s JSON, which a host exchanges with a remote task, gains reducers (the graph’s) and inBranch. changes names an overwritten field with a ! suffix: {"log": [...], "summary!": "x", "gone!": null}. On a re-run it may also hold an addition recorded by an earlier run, under a + suffix: {"history+": [m]}.
• FlowStep’s JSON drops stepID, predecessorID and successorID; the edges are predecessorKey and successorKey, as step keys. subgraphURL, subgraphName and stoppedTaskName follow the renames above.
• ShardSummary’s JSON has index and latency (nanoseconds).

Metrics and Operations

• Renamed: dwarf_flows_terminated is dwarf_flows_ended. Dashboards and alerts must change.
• New labels: dwarf_flows_started and dwarf_flows_ended have a priority label, so more series; a View can drop it.
• New: dwarf_operation_seconds (to see a slow shard), dwarf_state_cache_bytes, dwarf_steps_dispatch_lag_seconds, dwarf_shard_dispatching.
• dwarf_steps_pending and dwarf_steps_oldest_pending_age_seconds are reported every 30 s by one leading replica per shard. They can be absent for up to 30 s at a handoff; aggregate them with max per shard, never sum.
• No gauge queries the database at collection.
• Memory: the state cache defaults to a tenth of the memory limit (GOMEMLIMIT, else the cgroup, else 1 GiB), and without GOMEMLIMIT the heap can reach about twice that, so an upgraded replica’s resident memory grows. SetStateCacheSize overrides it.
• Tracing: with no OpenTelemetry SDK installed by Startup, the engine traces through a no-op and never picks up a provider installed later. Install the tracer before Startup.
• PostgreSQL sessions are pinned to UTC, so stored times are right whatever the server’s or role’s time zone.

New operating guidance in the docs:

• Set autovacuum_work_mem = 1GB on PostgreSQL.
• Leave statement_timeout, transaction_timeout, idle_in_transaction_session_timeout and MySQL’s max_execution_time unset for the engine’s database, and keep long-running queries off it.
• After a failover to an asynchronous standby, or a restore, restart every replica.

Fixes

• Terminate of an ended flow returns nil (was 409). Construction-time configuration errors are 400 or 409 (were 500). Resume with data that is not JSON-marshalable is a 400 (was silently dropped).
• Await and Run timeouts are 408s that satisfy errors.Is(err, context.DeadlineExceeded), also when the caller’s context had no deadline. An Await racing a Resume stays parked instead of returning a 408 while the flow is still running.
• List and Purge reject a negative OlderThan or NewerThan with a 400, and Purge accepts every List filter.
• SQL Server: an interval past ~24.8 days no longer overflows. Before, a List or Purge age filter of 30 days failed, and a flow.Sleep or Retry backoff that long failed its flow.
• SQL Server: an operation over a few thousand flows or steps no longer exceeds the 2,100-parameter limit.
• Long tasks no longer slow Create and the step hand-off.
• Retry returns false for a negative initialDelay or maxIntervalDelay, or a multiplier that is negative, NaN or infinite. A backoff with no maximum saturates rather than overflowing, and the engine fails a step whose retry delay comes out negative.
• A flow or step key parses only in the spelling the engine issues: 01-5-tok or 1-+5-tok is a 404.
• Snapshot and Resume follow the interrupt the flow row records, so they agree with StoppedTaskName when several siblings interrupt.
• During a deletion window, History of a subgraph child’s key returns 404 with its root.
• Fork of a flow that a Delete or Purge stamped mid-call is a 409, and of one already removed a 404 (was a 500).
• Purge no longer marks a root that a Resume revived after it was selected and that has since completed.
• Terminate, Delete and Purge stop a flow whose rows hold an unresolvable state reference or an unparseable graph, with an empty final state. They used to retry forever.
• A step covered by Cancel whose changes, with the error added, exceed the payload limit fails on the size.
• A failed child’s error reaches its flow.Subgraph caller whole, status code and properties included.
• dwarf_steps_executed{interrupted} no longer counts an interrupt fenced away by a lost lease, and dwarf_flows_ended counts only the flows a Terminate itself ended.