Inspecting and Repairing a Flow

For operators and developers looking into a flow that failed, stalled or did something unexpected, and fixing it. The Runbook tells you when to do this; this page shows how.

The example is a payout flow with two tasks, Validate and Pay, where Pay rejects amounts over a limit. A flow was created with an amount of 5,000, and it failed.

1. Find the Flow

ListClosed finds closed flows by status, workflow, task, tenant and free text, within a window on when each was archived; ListOpen does the same for flows still running or interrupted:

failed, err := eng.ListClosed(ctx, workflow.Query{
    Status:   workflow.StatusFailed,
    FromTime: time.Now().Add(-24 * time.Hour),
    ToTime:   time.Now(),
    Limit:    100,
})
for _, f := range failed {
    fmt.Println(f.FlowKey, f.WorkflowName, f.StoppedTaskName, f.Error)
}

Each summary names the task where the flow stopped and its error, here Pay and amount over the limit.

  • A flow appears in ListClosed about a minute after it closes, when it moves to the archive. Until then it is in neither list, but History, Step and Fork already work on it.
  • There is no pagination. To see more, narrow the filters or the window, as Listing Flows describes. The window may reach at most 31 days.
  • A closed flow stays for the retention, 30 days by default (SetRetention). Repair it before then: once its day is dropped from the archive, the flow is gone.

In a multi-tenant service, scope the lists the way Building a Host does, since they return flow keys, and a key grants control of its flow.

2. Read Its History

History returns the flow’s steps in order, with each one’s task, status, error, attempt count and timings:

steps, err := eng.History(ctx, flowKey)
for _, s := range steps {
    fmt.Println(s.StepDepth, s.TaskName, s.Status, s.Error)
}
// 1 Validate completed
// 2 Pay failed amount over the limit

History leaves out each step’s state on purpose, so a console built on the lists and History shows the shape of a flow and its failures without exposing payloads. A step inside a subgraph carries the child’s steps in SubHistory.

3. Draw It

workflow.NewFlowRenderer turns a history into a Mermaid chart, with each step colored by its status:

chart := workflow.NewFlowRenderer(steps).WithLinks("step").Render()

For the failed payout, it renders this (shown here without the links):

flowchart TD
    classDef completed fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1
    classDef running fill:#32a7c1,color:#f4f2ef,stroke:#32a7c1,stroke-dasharray:4 2
    classDef pending fill:#e5f4f3,color:#434343,stroke:#434343
    classDef failed fill:#f15922,color:#f4f2ef,stroke:#f15922
    classDef terminated fill:#f15922,color:#f4f2ef,stroke:#f15922
    classDef cancelled fill:#f15922,color:#f4f2ef,stroke:#f15922
    classDef interrupted fill:#ed2e92,color:#f4f2ef,stroke:#ed2e92
    classDef term fill:#e5f4f3,color:#434343,stroke:#434343
    linkStyle default stroke:#32a7c1

    _start((" ")):::term
    _end((" ")):::term
    s1["Validate
1ms"]:::completed
    s2["Pay
0ms"]:::failed
    s1 -- "0ms" --> s2
    _start --> s1
    s2 --> _end

WithLinks("step") makes every step clickable: a click navigates to ?step=<stepKey>, so a page that shows the chart can read that parameter and open the step, as in the next section. Leave it out for a static chart. Subgraph steps render as nested boxes, and an interrupted flow shows the waiting step highlighted.

4. Look Inside a Step

Step returns one step in full, including the state the task saw and the changes it wrote:

step, err := eng.Step(ctx, stepKey)
fmt.Println(step.State.GetInt("amount")) // 5000: the input that made Pay fail

This is the one read that exposes a step’s payloads, so give it the same access check as Snapshot; see Data Handling & Security.

5. Repair It with Fork

A flow that has ended is never re-run in place. Fork creates a new flow from a chosen step, re-runs from there, and can override state on the way, which is how you get past a bad input:

fixedKey, err := eng.Fork(ctx, failedStepKey, map[string]any{"amount": 900})
outcome, err := eng.Await(ctx, fixedKey)
// outcome.Status == "completed"

The fork keeps the original’s scheduling and baggage, and re-runs only from the step you chose: Validate is not repeated here. The original flow stays as it was, for the record.

  • Choosing the step. Fork from the step that failed to retry it with corrected state, or from an earlier one to redo more. Any recorded step works, including one inside a subgraph.
  • A flow that is still running can’t be forked until it ends. If it is stuck, stop it with Terminate first.
  • Side effects. A forked step runs its task again, so the task must be idempotent like any other.

The same steps repair a flow the engine reports as stranded: see dwarf_flows_orphaned Is Non-Zero.

Next: Runbook.

Backward Data Handling & Security Runbook Forward