Composing durable work: steps, parallel fan-in, and sagas
The agent loop journals its own model turns and tool calls. But the same durable substrate is exposed directly, so you can wrap your own work in the same at-most-once, crash-safe, independently-provable guarantee. This is the "Option B" authoring model: write plain Go control flow (if / for / functions), and name the operations that must survive a crash. There is no graph DSL; the graph is a derived output (see Debugging RenderMermaid).
Three primitives cover the common shapes, all methods of the *Journal over a Store (extension points):
Journal.Step[T]: one named durable step.Journal.Parallel[T]/Task[T]: durable fan-out/fan-in.WithSaga+CompensatedFunc: transactional agents with reverse-order rollback.
Journal.Step[T]: one named durable operation
func (j *Journal) Step[T any](ctx context.Context, runID, name string,
fn func(context.Context) (T, error), opts ...StepOption) (T, error)Step runs fn as a named durable step keyed by (runID, name) and returns its typed result. On resume, a completed step returns its recorded result without re-running fn. It is the building block the agent loop itself is made of, exposed for your own orchestration.
A step is a side effect unless you say otherwise, and it gets the same guarantee as a tool call: it runs at most once. An attempt marker is journaled before fn runs, so if the process dies after fn's effect and before its result is recorded, the resumed step returns *OutcomeUnknown instead of running fn again; ResolveHalt (with the halt's Ref()) records the confirmed outcome. The same holds when fn returns an error, since a failed call may still have taken effect. A step cancelled (or whose store fails) after its marker is written and before fn is called does not call fn, and records that the attempt did not start, so the next call runs fn under a new marker instead of halting; only a process that dies in that gap leaves a halt. A step that is safe to re-run declares it with WithSafety, and then simply re-runs after a crash or an error:
inv, err := journal.Step(ctx, runID, "fetch-invoice",
func(ctx context.Context) (Invoice, error) { return billing.Lookup(ctx, id) },
agent.WithSafety(agent.Safety{ReadOnly: true}))
res, err := journal.Step(ctx, runID, "reserve", // at most once; halts on an unknown outcome
func(ctx context.Context) (Reservation, error) { return inventory.Reserve(ctx, sku) })A side-effect step must not pause. If fn of a step that is not retry-safe returns a pause (Interrupt, Sleep, Await, a pending approval, a halt from a sub-agent it drives), Step returns an ErrConfig error, not the pause, and the step's marker stays, so its next attempt halts: fn may have done something before it paused, and running it again could repeat that. Put the pause in a retry-safe step of its own, before or after the side effect. This is the same rule a tool follows.
Step is a generic method of *Journal (Go 1.27 methods can have type parameters). The result is journaled as a StepValue record, so it shows up in RenderMermaid as step: <name> and is independently provable via audit.ProveStep (see Audit). name must be unique within the run: a second Step with the same (runID, name) returns the first one's recorded result. It shares the run's journal with the engine's own records, so a name that starts with a prefix the engine reserves for them (@, run:, tool:, attempt:, approval:, approval-tally:, approval-pending:, approval-pause:, approval-identity:, approval-status:, approval-exec-hold:, approval-request:, signal:, await-timeout:, await-resolved:, timer:, interrupt:, chan:, chanack:, turn/, start/, from/, audit:, slot:, cap-intent:, cap-reserve:, cap-release:, gate-reason:, and a plan flow's node:, switch:, flow:; see agent.IsReservedStepName) is ErrConfig, for Step and for a Parallel task alike, and so is an empty name. Inside the body of a plan flow's node, a Step for the flow's run is recorded under the node's key (node:<node>:step:<name>, per loop iteration in a loop body), so its name need be unique only within the node; see Plan flows.
The same memoized journal is the idempotency guard the Messaging webhook pattern uses (through Run and SendOnce) to make a redelivered inbound event replay instead of re-fire.
Journal.Parallel[T] / Task[T]: durable fan-in
type Task[T any] struct {
Name string
Fn func(context.Context) (T, error)
Safety Safety // as WithSafety: the zero value is a side effect
}
func (j *Journal) Parallel[T any](ctx context.Context, runID string,
tasks []Task[T], opts ...ParallelOption) ([]T, error)Parallel runs each Task concurrently, each as its own durable Step, and returns the results in task order (not completion order). It is the durable, auditable fan-in that a compliance pipeline wants: run several independent checks at once (sanctions, credit, fraud), each crash-safe, each result committed to the journal and provable on its own, then aggregate.
- Each task's
Nameis its durable memoization key within the run, so it must be unique across the tasks in one call; a repeated name returnsErrConfigbefore any task runs. - Each task is a
Stepwith the task'sSafety: mark checks and lookupsReadOnly, and leave a side effect at the zero value so it runs at most once. - All tasks run even if some fail, so a failed check never hides the others. The returned error joins every task's error (
errors.Join) and isnilonly if all succeeded. Succeeded tasks are memoized on resume; a failedReadOnlytask re-runs, and a failed side effect halts. maxConcurrencycaps in-flight tasks;<= 0means one goroutine per task.
checks := []agent.Task[CheckResult]{
{Name: "sanctions_check", Fn: runSanctions, Safety: agent.Safety{ReadOnly: true}},
{Name: "pep_check", Fn: runPEP, Safety: agent.Safety{ReadOnly: true}},
{Name: "adverse_media_check", Fn: runAdverseMedia, Safety: agent.Safety{ReadOnly: true}},
}
results, err := journal.Parallel(ctx, runID, checks) // add agent.WithMaxConcurrency(n) to cap tasks in flightThis is deliberately a thin primitive over the journal, not a graph engine. Dynamic, model-driven routing stays in plain Go and sub-agents; Parallel covers the static fan-out/fan-in that a governed workflow's "parallel checks, then decide" stage is made of. The full worked flow (parallel durable checks, then a governed decision, then an offline proof) is examples/govern/compliance.
Related but distinct: the agent loop already runs a single turn's tool calls concurrently (bounded by the
WithMaxConcurrencyoption).Parallelis for fan-out you author yourself outside a model turn. See the parallel-tool concurrency notes in KNOWN-LIMITATIONS.md.
Sagas: transactional agents with reverse-order compensation
A saga is the sequential/hierarchical transactional tier: "charged the card and booked the flight, then failed on the hotel, so cleanly refund and cancel." A tool declares how to undo its write with the Compensator port (extension points); CompensatedFunc builds a typed tool that carries both the forward action and its undo:
book := agent.MustCompensatedFunc("book_flight", "book a flight", func(ctx context.Context, in BookArgs) (Booking, error) { return airline.Book(ctx, in) },
func(ctx context.Context, in BookArgs, out Booking) error { return airline.Cancel(ctx, out.PNR) })Run the agent with the WithSaga() run option. If a step fails after earlier compensatable writes succeeded, the run rolls those writes back in reverse order (recursing through sub-agent trees) and returns *SagaAborted:
_, err := a.Run(ctx, runID, agent.UserText(input), agent.WithSaga())
var aborted *agent.SagaAborted
if errors.As(err, &aborted) {
// aborted.Cause is why it rolled back; the completed writes were compensated.
}WithSaga() works the same with Stream and the other entry points, and the *Result is returned with the error (see Debugging). SagaAborted unwraps to its cause, so errors.Is against a sentinel still works.
How it survives a crash
The abort is derived from the journal: a saga step failure writes a durable StepSagaFail record, so a crash at any point resumes correctly. On re-entry a recorded failure sends the run straight to rollback, and each compensation is itself a durable memoized step, so it runs at-most-once if it completes. This split is the precise contract, proven adversarially in saga_dst_test.go (see Testing):
- The forward non-idempotent effect is at-most-once (halt on unknown outcome).
- Compensators are at-least-once: memoized so they run once if they complete, but a crash mid-compensation re-runs them. This is why a
Compensatemust be idempotent. - A
CompensatedFuncundo sees the arguments the forward call decoded. It decodes the call's recorded arguments with the same strict decoder the forward call used. A call journaled before tool arguments decoded strictly may hold arguments onlyencoding/jsonaccepts (a case variant, an unknown name); that call decoded them withencoding/json, so compensation does too. - Undo sees the arguments the tool accepted, after tool middleware. When a middleware changes a compensable call's arguments in a saga, the arguments the tool receives are journaled before it runs, and compensation undoes those (a charge rewritten from 5 to 500 is refunded 500, even when its outcome was resolved with
ResolveHalt). A rollback that re-runs a retry-safe call to learn its result runs it through the same middleware. Unchanged arguments journal nothing. A journal written before this has no such record, and compensation there uses the model's arguments. A middleware must rewrite a retry-safe compensable call's arguments the same way every time, since the first record is kept.
The boundaries (read these before relying on it)
These are stated in full in KNOWN-LIMITATIONS.md; in brief:
- A saga step must be atomic. The failing step itself is not compensated (there is no recorded result to drive
Compensate), so a forward step must not leave a partial external side effect before returning an error. Make forward steps all-or-nothing or idempotent. - Unknown-outcome resume halts, it does not auto-roll-back. If a non-retriable step crashes after its attempt marker but before any result, the run returns
*OutcomeUnknown: a human decides, because you cannot safely roll back a step that may have committed. - Rollback covers calls the abort cut off. Rollback walks every tool call the model made, not only those that returned. A call of the failing turn that had not started when the step failed never starts. A side effect that started but has no recorded outcome (a sibling the failure cancelled mid-call) stops the rollback with a
*OutcomeUnknowninSagaAborted.CompensateErr; resolve it withResolveHaltand drive the run again to finish the rollback. A retry-safe call with a compensator is run again to learn its result, then undone; if the re-run is refused without reaching its tool (anaudit.AttenuatingSubAgentdelegation's grant has expired), the call is listed inSagaAborted.UnknownOutcomeand the rollback goes on. A sub-agent call is rolled back into whether or not it returned, and whatever its result: an error result (its sub-run failed in a run that is not a saga) does not say its sub-run made no write. A sub-agent whose own saga failed rolled itself back first, and if that rollback stopped (a crash, an unknown outcome, a failing compensator) the parent's stops there too and resumes it on the next drive, soSagaAbortedcovers the whole tree. A completed write with no compensator, idempotent or not, is listed inSagaAborted.Uncompensated. So is any call whose tool is no longer registered when the rollback runs (its compensator and safety are unknown), and one of those with an attempt marker and no result stops the rollback with a*OutcomeUnknown. - Programmatic sub-runs roll back with their call. A tool that runs another agent itself, under
RunInfo.SubRunFor(name), starts a sub-run of its call. In a saga the parent's journal links each one before it records anything, and the rollback walks them, latest first, after the call itself, as it walks a sub-agent's. Declare the agent each one runs with, so the rollback can compensate its writes:agent.Func(name, desc, fn, agent.WithSafety(safety), agent.WithSubRuns(func(name string) *agent.Agent { return child })). With none declared, the sub-run's writes are listed inSagaAborted.Uncompensated, as is a declared agent the rollback cannot use (one on another store, or aWithSubRunsfunction that panics), with the reason. The call's own compensation runs before its sub-runs are walked. A sub-run in a saga's tree must journal to the saga's store (Runrefuses another withErrConfig), and a plain run started from a saga's call links its own sub-runs too. Start the sub-run withWithSaga()to keep it a saga (whether a call is in a saga is its own run's flag,RunInfo.Saga), and start it within the call:SubRunForfrom a goroutine that outlives its call isErrConfig. A sub-run still running after its call returned may write after the rollback walked it; such a write is not undone. - A failed step with an unknown outcome is reported, not undone. A retry-safe step that fails with
ErrToolOutcomeUnknown, or returns an error after itsWithTimeoutdeadline, may have committed before it was cut off. Its failure record carriesoutcome_unknown, and the rollback lists it inSagaAborted.UnknownOutcome(through sub-agent trees) without running its compensator on a result it never returned. Check each one against its downstream system. - A delegation is undone under its own authority. A rollback into an
audit.AttenuatingSubAgent's sub-run rebinds the child grant and identity the sub-run journaled, so a compensation never runs under authority the delegation did not grant. A saga whose delegations were minted under several grants (the root grant was rotated between drives) rolls back with all of them bound:audit.WithGrantfor the acting one,audit.WithRollbackGrantsfor the earlier ones (see the delegation guide). - A saga resumes as a saga. A run's first drive records whether it runs as a saga, and every later drive runs it that way:
RunwithoutWithSaga()drives a saga as the saga it is, andWithSaga()on a run that is not a saga isErrConfig. - A reconciled failure aborts. A crash between a step's failure and its record leaves the step with an unknown outcome. Resolving it with
ResolveHaltandOutcome{IsError: true}records a failed step, and the next drive rolls back as it would have had the failure been recorded. (A human's denial of a gated call is not a failure: the model reacts to it, as outside a saga.) - A step of unknown outcome holds its turn. While a sub-agent step halts on an unknown outcome, no later call of the same turn starts: the step may prove to have failed, and the saga must not run a step a run that never crashed would not have reached.
- Compensation is hierarchical, not concurrent. Rollback recurses through a sub-agent tree (one causal order). Truly concurrent agents mutating shared state out of order need the provable convergence of the governance tier (Governance), not a saga.
SubAgent(name, description, sub, opts...) (or MustSubAgent) composes agents into that durable tree (pass agent.WithApproval to have the parent wait for a human before it delegates): give the sub-agent the same journal as the parent for a unified journal, and a crash anywhere in the tree resumes the whole tree precisely (completed sub-agents reused, the in-flight one resumed, and OutcomeUnknown / ApprovalPending / SagaAborted from deep in the tree propagating up). A pause raised inside a sub-agent surfaces from the parent's own Run (match it with AsPause or errors.As); resolve it against the sub-run's journal, Paused().RunID (with the halt's Ref(), or the approval's ToolUseID), then re-run the run named by Paused().RootRunID, the top-level run, with the root agent to resume down the path. An OutcomeUnknown or ApprovalPending from a sub-agent propagates whatever the calling tool's safety; any other pause (InterruptPending, SignalPending, TimerPending) needs a retry-safe calling tool, as it does anywhere. A Sleep in a sub-agent schedules its wake (Wake.RunID the sub-run, Wake.RootRunID the root) for the root run. When several calls of one turn pause, the run returns an OutcomeUnknown ahead of any other pause (the run cannot go on until it is resolved, whatever else is answered), and otherwise the first call's pause. A call that loses its answer (ErrToolOutcomeUnknown) is held the same way as a halt: its siblings in flight finish. A storage failure inside a sub-agent, or a lost answer in it, is not the sub-agent's answer: the parent records nothing for the call and returns the error, and resuming re-enters the sub-run.
A sub-run's ID is agent.SubRunID(parentRunID, toolUseID): the parent's run ID, >, and the tool-use ID encoded so that no ID a model sends can name another call's sub-run. A session's journal and turn runs are named the same way, "<session id>>@session", "<session id>>@turn/<n>" and "<session id>>@event/<encoded key>". A top-level run ID therefore may not contain > (Run, Stream and the rest return ErrConfig), so none can name a sub-run or a session's run; / is fine (tenant/123). agent.IsSubRun and agent.IsSessionRun tell them apart. Recover skips sub-runs, which their root's re-run resumes, and session runs, which the session resumes when the turn's message is sent again.
A run ID that starts with @cap/ is reserved as well, for the cap ledger runs the engine keeps in the store beside your runs, and so is @stop, the kill switch's control run. Run, Stream, Session and the rest return ErrConfig for either. agent.IsReservedRun reports a reserved run (agent.IsCapRun only a cap ledger), and Recover, bide view and the other passes over a store's runs skip them.
Clearing an OutcomeUnknown: ResolveHalt
An OutcomeUnknown is deliberately terminal until a human confirms the real outcome: the runtime cannot know whether the non-idempotent side effect (a charge, a send) actually committed. Once you have verified it out of band, ResolveHalt is the sanctioned escape. It records the missing result under the halted operation's key (the same journal key the loop or Step uses), so a re-run proceeds past the halt instead of halting again:
if halt, ok := errors.AsType[*agent.OutcomeUnknown](err); ok {
// operator confirmed the charge did go through
_ = agent.ResolveHalt(ctx, journal, halt.Ref(), agent.Outcome{Result: "charged (confirmed)"})
res, err := a.Run(ctx, halt.RootRunID, agent.UserText(input)) // resumes past the halt
if err == nil {
msg = res.Message
}
}The recorded result notes when it was written (Record.RecordedAt, recorded_at): the resolving process's clock, agent.WithClock if you pass it, else the wall clock (the same clock the call's claim marker and WithMinHaltAge use; the context's clock is not read). The resolver signs nothing, so the time is a claim anchored only by the journal's Merkle trail once a signed head covers the record. An auditor reads it to tell when a notified halt was resolved (bide view shows it on the timeline and in the verdict, bide audit verify-evidence beside the tool call). It is informational: no drive reads the time.
halt.Ref() names the run, the operation (Op: a tool call, OpTool, with its tool-use ID, or a Step, OpStep, with its name), and the halt's Cause. Pass the result value an actual call would have returned, and IsError: true if the verified outcome was a failure the model should react to (in a saga, a failed step, which rolls the saga back). It is idempotent (the first result for an operation wins), so a retry or a racing driver records it at most once. A tool call's result and a step's are separate journal records, so it refuses an operation that only the other kind attempted. It resolves only an operation that halted, one with a live attempt marker: an operation never attempted, or whose attempts are recorded as not started (the next drive re-attempts it), is refused with agent.ErrNoLiveAttempt, so a resolution cannot pre-record an effect that never ran. A plan flow's node halts as a Step under its node key; resolve it with Flow.ResolveHalt, which also checks the value against the node's output type.
ResolveHalt will not resolve an effect a driver may still be running. HaltContended means another driver of the same run won the claim while this one ran (a node that took over after a lease lapsed); HaltCrashed means the halting driver knew of no live claimant, which is not proof that none is running. So, whatever the cause:
- On a store that leases runs (
MemStore,store/sqlite,store/postgres; found through aJournaland through store wrappers, asagent.Capabilityfinds it),ResolveHalttakes the root run's lease while it resolves and returns*HaltInFlightwhile a driver holds it. Only drivers that lease the run (Lease,Recover,RecoverLoop) are seen; a plainRunholds no lease. - On a store that cannot (a custom store with no
Leaser), it requiresWithMinHaltAge, so the halt is resolved only once no driver can still be running it. - On either kind of store, it then claims the attempt after the live one, under a claim of its own, before it records the outcome, and returns
*HaltInFlightif a driver holds that claim already: a process that could not record that its claim never started can void the live attempt after the check and run the effect as the next attempt (a plainRun, which the lease does not see, included), and a resolution must not override it. The resolution's claim stays in the journal as an attempt marker of the operation. If recording the outcome fails, that claim stays live (the outcome may have been recorded all the same): the operation halts until it is resolved again, and the next resolution claims the attempt after the resolution's own (withWithMinHaltAge, measured from that attempt). - A
HaltContendedhalt always requiresWithMinHaltAge. - Where
WithMinHaltAgestands in for the live-driver check (aHaltContendedhalt, or a store with noLeaser), the age must cover the tool's real worst-case run time, not itsToolSpec.Timeout(WithTimeout): a result the tool returns after its deadline is still recorded, and a tool that ignores cancellation can run past the timeout, so an age equal to the timeout can resolve a halt while the call is still running. The lease check on a store that leases runs is not affected. WithoutLiveDriverCheck()skips all of these, for an operator who knows no driver is running (every worker stopped).
An operation that already has a different outcome (an earlier resolution, or the result the live driver recorded) is refused with *HaltAlreadyResolved (ErrAlreadyResolved, which wraps ErrConfig), which reports the outcome that stands; the same outcome again returns nil. A HaltRef built by hand must state its cause.
Resolving without a human: reconcilers
The verifier does not have to be a person. Most systems that lack an idempotency key still leave a queryable record (a sent-message id, a row, a log line), so a reconciler can query that record, decide, and call ResolveHalt itself. Two options make that safe:
WithMinHaltAge(d)refuses to resolve a halt younger thand, measured from the live attempt marker (OutcomeUnknown.AttemptedAt; an attempt recorded as never started does not count) to now. A provider's record can lag the send by seconds, so a reconciler that queries too early reads "absent" and re-fires the exact effect the halt prevents. A minimum age keeps "unknown" unknown until the record has had time to appear; too soon returns*HaltTooYoung, so the reconciler waits and retries. A marker with no usable timestamp (none, or a zero or negative one, which the engine never writes) isErrConfig: the age cannot be known, so the halt is not resolved blind. A marker stamped in the future counts as too young.Outcome.Evidence(or theWithEvidence(v)option) records the resolution as reconciled and stores what was read to decide, signed alongside the outcome (Record.Reconciled/Record.Evidence). The journal is the audit record, so a reconciled outcome that did not say it was reconciled, and on what basis, would be a hole in the very thing the trail exists to protect. With it, a later reader tells a reconciled step from a clean one and re-checks the evidence.
if halt, ok := errors.AsType[*agent.OutcomeUnknown](err); ok {
sent, record := providerSays(halt) // query the system of record
if sent {
_ = agent.ResolveHalt(ctx, journal, halt.Ref(),
agent.Outcome{Result: "sent (reconciled)", Evidence: record}, // the basis, signed with the outcome
agent.WithMinHaltAge(30*time.Second)) // do not decide before the record can settle
}
}The human stays the fallback for the genuinely unknowable case, not the default.
See it against Stripe test mode: examples/stripe-refund kills a refund agent after its request is sent, halts on restart, and reconciles by listing the payment intent's refunds and matching the step's key in their metadata. It resolves with the refund as evidence, and exactly one refund exists. It leaves the step halted when Stripe has no such refund.
Durable timers: Sleep / WaitUntil and the Waker
An agent often has to wait: for a deadline, a cool-off, a scheduled follow-up. (Timers need an agent run: a plan flow node cannot call Sleep or WaitUntil today.) Sleep(ctx, name, d) and WaitUntil(ctx, name, until) make that wait durable. Called from inside a retry-safe tool, they journal the wake time once (at-most-once by name) and pause the run with *TimerPending, the same durable pause as Interrupt. Because the wake time is journaled on the first call and memoized, a resumed or crash-recovered run waits to the same absolute instant rather than restarting the clock; no goroutine is held blocked across the wait.
wait := agent.MustFunc("cooldown", "wait before retrying", func(ctx context.Context, _ struct{}) (string, error) {
if err := agent.Sleep(ctx, "cooldown", time.Hour); err != nil {
return "", err // *TimerPending propagates out of Run; the run is paused durably
}
return "resumed", nil
}, agent.WithSafety(agent.Safety{ReadOnly: true}))Re-invoking Run with the same runID at or after the wake time resumes past the Sleep. What re-invokes it is a Waker, the time-driven sibling of the inbound event trigger in Messaging: the SDK provides the durable, at-most-once timer and its resume safety, and the trigger is pluggable. Give the agent one with the agent.WithWaker(w) option and Sleep registers its wake automatically, through Schedule(ctx, agent.Wake{RunID, RootRunID, Name, FireAt}). A Schedule that fails fails the run with an error wrapping ErrStorage instead of pausing it (a paused run with no wake registered might never wake), and the tool call records nothing, so re-driving the run (RecoverLoop does, on its next pass) reaches the Sleep again and schedules again. MemWaker is the reference in-process implementation:
ctx, cancel := context.WithCancel(ctx)
var a *agent.Agent
w := agent.NewMemWaker(func(ctx context.Context, runID string) error {
_, err := a.Resume(ctx, runID) // resume from the journaled input and options; may sleep again
return err
})
a, err := agent.New(model, journal, agent.WithTools(tools...), agent.WithWaker(w))
if err != nil {
return err
}
done := w.Start(ctx, time.Second, nil) // tick: resume every run whose timer is due
// On shutdown: stop the loop and wait for it (including any Fire in flight) before closing the
// store the resumed runs write to.
cancel()
<-done
store.Close() // e.g. a store/sqlite or store/postgres StoreStart returns a channel that closes once the loop has stopped, after any in-flight Fire has returned. Waiting on it before closing the store keeps a resume from writing to a closed store.
Boundaries: MemWaker is a local-dev default, not a durable scheduler. Its in-memory timer set is lost on process exit, so the wake times must also live in the journal (they do), and a restarted deployment rebuilds pending wakes by scanning runs or hands the trigger to an external scheduler (cron, a queue). Tests give the agent a clock with the agent.WithClock option to advance time deterministically. This is the piece that makes an always-on ambient agent turnkey: a durable wait plus a trigger, with at-most-once and crash-resume intact across the wait.