Scheduling

For developers and operators. Developers set priority, fairness and time budgets per flow; operators need the same model to read a backlog. The behaviour described here is what the Runbook assumes you know.

Dwarf decides which pending step runs next, and how long each task may run. Most of this is automatic; this guide explains the knobs you control. How a step survives a crashed worker is covered in How Dwarf Works, and how a task retries a transient failure in Writing Tasks.

Priority

Priority arbitrates between flows competing for workers. It’s a property of the flow, set once at create and immutable for its life:

eng.Create(ctx, "report", state, &workflow.FlowOptions{Priority: 5})

Priorities run from 1 to the engine’s maximum, 5 by default and configurable up to 100 with SetMaxPriority, and lower numbers run first. An unset priority (0) takes the midpoint of the range, 3 by default. Create and Run reject a priority outside the range. Use the same maximum on every replica sharing a database. Work that carries a priority beyond a replica’s range (a flow created before the range existed, or by a replica configured with a larger one) still runs there, after all of its own levels. Priority is strict over capacity: lower-priority work gets only the capacity higher-priority work leaves unused. Once the engine observes due priority-1 work, priority-4 work is selected only for the workers priority-1 work cannot keep busy, so a light stream of urgent flows never idles the engine while a backlog waits below it.

The guarantee is over observed work, and observation is not instantaneous: the scheduler periodically scans for due work, well under a second apart. When priority-1 work arrives, it takes a scan or two before every part of a sharded cluster has observed it, and a small amount of priority-4 work may start in that window. With several replicas, each one serves its own share of the due work first, so priority-4 work can start on one replica while priority-1 work waits a few scans for another, after which any replica takes it ahead of its own lower-priority work. These effects are bounded by the scan interval, not by the size of the backlog: strict ordering is what you get between arrivals separated by more than a scan or two, which is the normal case.

Priority is never preemptive: anything already running continues to completion. The worker pool grows on demand, so newly arrived higher-priority work normally starts as soon as it is observed; only when every worker the engine may run is busy with lower-priority tasks does it wait for one of them to finish. Where that can happen and urgent latency matters, keep lower-priority tasks short (split a long job into steps) so a worker frees up often. There is no aging — a stream of high-priority flows that keeps every worker busy will starve lower bands by design, so reserve the top bands for genuinely urgent work. To see it happening, chart dwarf_steps_oldest_pending_age_seconds by priority: a lower band whose age climbs while dwarf_steps_dispatch_lag_seconds stays low is being held back by better-priority work, not by a slow engine (Operations).

Step order within a flow is dictated by the graph, not by priority.

Fairness

Within a priority band, fairness prevents one busy tenant from monopolizing the workers. Tag flows with a FairnessKey (typically the tenant) and an optional FairnessWeight:

eng.Create(ctx, "import", state, &workflow.FlowOptions{
    FairnessKey:    "tenant-42",
    FairnessWeight: 4, // 4x the dispatch share of a weight-1 key
})

The scheduler dispatches in proportion to weight across the distinct keys within each priority band — a weight-4 key gets ~4x the share of a weight-1 key, independent of how much backlog each has. If flows sharing a key carry different weights, the key dispatches at the lowest weight among its due work, so give one key one weight. A weight is at most 1,048,576 (2^20); use a fraction for a smaller share. A key is at most 256 ASCII letters, digits, -, _ and ., and is case-sensitive: Tenant-42 and tenant-42 are two keys. Within a single key, dispatch is oldest-first (FIFO). Both FairnessKey and FairnessWeight are immutable for the flow’s life (changing them mid-run would be a self-promotion vector).

Taken together, the scheduler picks the next step like this:

graph LR
    classDef node fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1
    classDef ask fill:#e5f4f3,color:#434343,stroke:#32a7c1
    linkStyle default stroke:#32a7c1
    due["Due steps"]:::ask --> band["Lowest priority<br/>number first"]:::node
    band --> keys["Weighted share<br/>across fairness keys"]:::node
    keys --> fifo["Oldest first<br/>within a key"]:::node
    fifo --> w["Next free worker"]:::ask

Errors and Retries

The engine never inspects status codes or error text. Any error a task returns is terminal for that attempt: the engine follows the graph’s onError transition if one is declared, and otherwise fails the flow. There is no rate-limit or unavailability signal to wrap an error with, which keeps the engine independent of any transport. A task that wants to back off on a transient failure, such as an HTTP 429 or 503, detects it itself and calls f.Retry, as shown in Handling Transient Failures.

Time Budgets

Every ExecuteTask call runs under a context deadline: the task’s time budget. The engine-wide default is set with SetTimeBudget (2 minutes unless changed), and a flow can set its own with FlowOptions.TimeBudget:

eng.Create(ctx, "transcribe", state, &workflow.FlowOptions{
    TimeBudget: 15 * time.Minute, // every task in this flow gets up to 15 minutes
})

The flow’s budget applies to each of its tasks, not to the flow as a whole, and its subgraphs inherit it. It is fixed when the flow is created, and is at most just under 24.8 days (a 32-bit count of milliseconds). A step whose replica crashes is not run again until its budget has passed, so a task that waits for days should Sleep or Interrupt rather than hold one long call. Your executor must honor ctx cancellation; for a tighter bound on a particular call, shorten the context inside the task. A flow deadline is the workflow’s own: implement one in your own code with a CreatedAt() guard, as shown in Timestamps. SetFlowExpiration is an operator’s backstop against flows that never end, and terminates them without running any task (Expiring Flows That Never End).

The budget also bounds how long a step stays claimed, which is how the engine notices that a task’s worker crashed: see Crash Recovery.

Workers

Per-replica concurrency needs no configuration: the engine sizes a resident worker set from each shard’s connection budget and grows it on demand, up to a ceiling that keeps crash recovery reliable. A worker blocked in an ExecuteTask call holds no database connection, so a workload of long tasks grows to fit its own concurrency. SetWorkers(n) is an expert override — mainly for bounding memory, since each in-flight step carries its state map — and pins the maximum rather than the resident set. See Deployment → Workers.

Crash Recovery

A step a crashed worker was running is recovered on its own: it is run again once its claim expires (the time budget plus 30 seconds), within the flow’s time budget plus about a minute and a half, and a step whose outcome was recorded but whose next step was not is picked up from that record without running its task again. Execution is therefore at least once, so make tasks idempotent. The timings, and what an operator sees, are in Deployment → Crash Recovery.

Next: Testing.