Untrusted content: provenance marks and the taint policy
An agent that moves money reads text it does not control: a customer's email, a web page, a ticket, a third-party tool's result. That text can tell the model to act. bide does not try to detect that. It bounds what an agent can do after it reads such text: a run that read untrusted content will need a human for any side effect, and amounts will be capped. This page covers the first part: provenance marks, which record in the journal what a run read, and the taint policy, which makes a run that read marked content need an escalation gate for its side effects. Both bound consequences; neither prevents prompt injection (limits, known limitations).
The declarations and the policy are experimental (stability).
Declaring what is untrusted
Only the host declares provenance. The model never does, and nothing in a result can.
read := agent.MustFunc("read_email", "read an email", func(ctx context.Context, in struct{ ID string }) (string, error) {
return "...", nil
}, agent.Untrusted("inbound-email"))
a, err := agent.New(model, j, agent.WithTools(read),
agent.WithRetrieval(retriever, 3, agent.RetrievalUntrusted("kb")))
if err != nil {
return err
}
// The run's own input came from a customer: say so.
_, err = a.Run(ctx, "run-1", agent.UserText("refund order 4417"), agent.WithUntrustedInput("customer-chat"))agent.Untrusted(source)on a tool: every result the tool returns is recorded with the source name (Safety.Untrustedof the result record: the normal result, an error, a denial and a saga's failure record). Split a tool whose results are sometimes trusted into two tools.agent.WithUntrustedInput(source)on a run: the input is untrusted.agent.WithTrustedInput(reason)says the input is the host's own. Either is journaled inrun:start's settings; a later drive that passes another isErrConfig, and the two cannot be combined.agent.RetrievalUntrusted(source)on a retrieval layer: the mark is recorded with the layer's@retrieval/<n>step.agent.TaintExempt(reason)declares a tool whose calls need no escalation gate in a tainted run. It requires a reason, is part of the declarations digest, and is recorded on thegate-reasonrecord of every call it let through, so the audit shows each call that ran because of it. It does nothing without a taint policy.agent.Trusted(reason)on a tool (experimental): the tool's results are the host's own (a table the host writes, a service it runs), and why. The reason is required and printable, it is part of the declarations digest, and a trusted tool writes no mark. It exists so that a policy can tell a tool someone classified from one nobody did (seeRequireProvenancebelow). A tool cannot be bothUntrustedandTrusted;ToolSpec.Trustedis the same declaration, and a wrapper that embeds the inner tool or implementsUnwrapmust carry it likeUntrusted.agent.RaisesSafety(reason)on a wrapper (experimental): a wrapper that embeds a tool or implementsUnwrapmay not declare aSafetysafer than a tool inside it (side effect, thenIdempotent, thenReadOnly), because the agent acts on the wrapper'sSafety: aReadOnlywrapper over a side-effecting tool is not gated by the taint policy, and its call is re-run on resume.agent.Newrefuses it withErrConfig, naming the wrapper, the tool and both levels, unless the wrapper declaresRaisesSafety(reason)(the reason follows the rules ofTrustedand is part of the declarations digest). With it, the wrapper is treated by the level it declared, so use it only when the wrapper does make the call safer.ToolSpec.RaisesSafetyis the same declaration.
A source given through agent.WithSafety(agent.Safety{Untrusted: "..."}) is the same declaration, never ignored; giving two different sources (this and the option) is ErrConfig at registration.
A source name is a short printable string (at most 128 bytes). A tool that wraps an Untrusted tool must declare it too. Registration refuses a wrapper that embeds the inner tool or implements Unwrap and does not carry the declaration; it does not see a wrapper that does neither (the same holds for an approval gate, a timeout or caps), so the declaration is the wrapper author's duty there. A wrapper that declares Trusted over a tool that declares Untrusted would drop the taint, and is refused the same way (the error names both); one that drops an inner Trusted is only unclassified, which RequireProvenance refuses.
What is recorded, and what is not
Marks are recorded whether or not the agent has a taint policy, so the journal shows what a run read before a policy exists. A run that states nothing records nothing: its records are byte for byte as before, and older binaries ignore the new members.
Not covered: a result ResolveHalt injects (it has no access to the registered tools; for a plan flow node see Plan flows), and any text a tool or system prompt injects that the host did not declare.
The taint policy
escalation := &agent.ApprovalPolicy{Need: 2, Approvers: []string{"risk-1", "risk-2"}, ExcludePrincipal: true}
a, err := agent.New(model, j, agent.WithTools(read, refund),
agent.WithApproverVerifiers(verifiers),
agent.WithTaintPolicy(agent.TaintPolicy{Gate: escalation}))
if err != nil {
return err
}
_, err = a.Run(ctx, "run-1", agent.UserText("refund order 4417"))agent.WithTaintPolicy(agent.TaintPolicy{Gate: policy}) makes the agent enforce the policy on the runs it starts. After a model turn has been shown content marked untrusted, every call of a tool that can have a side effect, requested by a later turn, needs the escalation gate policy.Gate (an ordinary ApprovalPolicy: for a bank, an m-of-n of signed decisions with ExcludePrincipal). With Refuse: true and no Gate the call is not run and not asked; the model reads an error result.
- Both gates apply. If the tool has its own gate, the call runs only when that gate has passed and the escalation gate has passed. There is no "weaker wins" rule.
- The escalation gate is a second instance on the call, under the subject
<call>#taint(a tool-use ID with a#is refused,ErrToolUseIDInvalid). Its approvers decide it and nobody else: a decision signed for the call does not count on the instance, and the reverse, and an approver of the tool's own gate is not an approver of the escalation gate unless the policy names them.bide approve,bide approve -list,bide view, the approvals intake and the notifier show it as a gate of its own, with the call's tool and arguments. - The gate that decides it is the one its pause note records, never the tool's live gate or the process's current policy. The approvals intake serves an escalation subject only when its
gate-reasonrecord agrees with its pause note; a journal that disagrees is not served. - The reason is shown, not signed. The engine records why it held the call (
gate-reason:<call>): the sources of untrusted content its turn read, with where and at which journal position.bide approve -list, the decision screen,bide viewand the notifier'sapproval.whyshow it for the#taintinstance (a cap reason, for a call a cap rule holds, shows the rule and this call's figures as of the ask). It is the application's record: no gate reads it and the signed decision does not bind it. Thebide approve -listview of a held call (a sample: an inbound email asked for a $4,800 refund, and the agent'srefundcall waits for two of ops, finance and risk, not the customer the run acts for):

The same call on the run page of bide view, with "why this gate" open. The reason is the application's record and is not part of what an approver signs:

- An approval clears nothing. It is a decision on one call. The next side-effecting call is gated again, and no operation clears taint on a run.
- Not gated: a
ReadOnlytool, a tool declaredTaintExempt, a call requested in the same turn as the read (the model had not yet seen the content), and a saga's rollback. - The policy is a property of the run. A run that started with no policy is never gated for its marks, even after the host adds a policy. A run that started under a policy records its digest in
run:start; a process with no policy that drives it refuses its side-effecting calls (ErrConfig), and a process whose policy has another digest holds a call that was asked under the old one (ErrConfig). Roll the whole fleet before enabling a policy. - The
gate-reasonrecord says why the call was held: the sources and positions of the marks and the escalation policy's digest. It is evidence for the approver and the auditor, and never authority: the engine recomputes the reasons and enforces the union, so a record can add a requirement and never remove one. SeparateApprovers(experimental) makes the second look a second person's. Without it nothing stops one approver approving both a call and its escalation instance, so the intended second look can be one person. Withagent.TaintPolicy{..., SeparateApprovers: true}, the instance's tally leaves out the approvers whose decisions the call's own gate counted (the terminal tally the own gate recorded: approvers and deniers alike), the gate records them asexcluded_by_sibling, and when too few approvers are left to reach the Gate'sNeed, the instance is denied, not left waiting. It is checked atNewtoo: a tool that can be escalated (it can have a side effect, and is notTaintExempt) with an own gate must have an m-of-n one whose approvers share no ID with the policy's Gate (compared as written, ignoring ASCII case, before any requester exclusion), orNewisErrConfignaming the tool and the shared IDs; aSingleApprovalown gate, or aSingleApprovalorRefusepolicy, isErrConfigwith it set, since a single decision has no approver ID to keep apart. The own gate is never recounted because of the instance. It is in the declarations digest only when set, andbide doctor -diffreports its removal as weakened. The approvals intake refuses a decision by an approver the own gate counted,bide approverefuses it and lists them as unable to approve, the run viewer shows them excluded, and the notifier does not ask them.bide audit verify-approvals -separate-approverswith-callthe instance (andaudit.VerifyApprovalswithApprovalPolicy.SeparatedFrom) recounts an instance with the same exclusion, from the own gate's tally that the evidence carries. It works on approver IDs, not people: two IDs of one person are not told apart, and in a certificate-bound gate the decider is the approver name, not the certificate's subject.RequireProvenance(experimental) leaves nothing unclassified by omission. Withagent.TaintPolicy{..., RequireProvenance: true}, a tool that declares neitherUntrustednorTrusted(TaintExemptdoes not count) isErrConfigatagent.New, naming the tool, and so is a tool that declares both. A newRun,SendorSendOncethat states neitherWithUntrustedInputnorWithTrustedInputisErrConfigat its first drive and writes nothing; a sub-agent's run (its input is the parent's call) and a resume (the input was stated when the run started) are not asked. The setting is part of the declarations digest only when it is on, so an existing policy's digest does not change. It declares; it does not verify: aTrustedreason is the host's word, and a tool that returns outside text underTrustedtaints nothing.- Plan flows take the same policy, with a rule of their own (see Plan flows). Children and sessions inherit taint, as follows.
Children, sub-agents and sessions
Taint follows the content, not the run that read it.
- Down. A sub-agent started by a call of a tainted turn starts tainted: its
run:startrecords the marks the calling turn was shown (ext.taint_inherited, one entry per source, sorted), so its first turn is gated as the parent's would have been. The marks pass exactly when the parent run is under taint; a child of a run that is not under taint inherits nothing. A later drive of the child that finds another list isErrConfig. The child need not share the parent's store. - Up. If a run the call drove ended with any mark, whether it read one itself or started tainted, the parent's
tool_resultfor the call carries the marksubagent:<tool name>(cut to 128 bytes on a character boundary), and the parent's next turn is tainted. This covers a sub-agent tool and a tool that starts a programmatic sub-run (WithSubRuns,RunInfo.SubRunFor) and returns its answer. Without it a tainted child's answer would launder what it read. The child's journal is read from the child's own store. If it cannot be read, a call that succeeded records nothing and stops with a storage error (a resume reads again), and a call that failed is marked, since its error text may carry what the child read. - Along a session. A session keeps the questions and answers of its turns, so the answer of a tainted turn reaches the next turn as the assistant's own words. A turn that ended with marks records them in the session journal (
taint/turn/<n>, the union of all marks so far), and every later turn starts with the union of all the session's taint records, so a record can add marks and never remove any. A clean turn after a tainted one does not clear it: the later turn inherits the marks and ends with them. The turns of a session before a policy joined are not retro-marked. - A tool that runs an agent on a context of its own (a fresh
context.Background(), a run ID that is not a sub-run of the call) is an ordinary tool that returns content from elsewhere: bide cannot see the run, so the answer is not marked. DeclareUntrustedon such a tool. - A tool that calls a sub-agent is itself a call: unless it is
ReadOnlyorTaintExempt, a tainted turn needs the escalation gate to delegate. Attenuation (a narrower grant, delegation) is separate and does not clear taint. - Not covered: a saga's rollback re-run of a sub-agent, and the answer a
ResolveHaltinjects.
Plan flows
A plan flow has no model turn to taint, so it has a rule of its own, the run-level one (design). It is experimental, like the rest of the policy, and Go only: a config cannot give a flow a policy.
- Declare. A node says what its results carry with
plan.Untrusted(source)(a Step, Model, Join or Tool block; theuntrustedkey of a config node adds a source and never removes one; a tool's ownUntrustedis taken),plan.Trusted(reason)orplan.TaintExempt(reason).Builder.WithTaintPolicy(orplan.WithLoadedTaintPolicyfor a flow loaded from a config) gives the flow anagent.TaintPolicy: an escalationGate, orRefuse, and optionallyRequireProvenance(every node declaresUntrustedorTrusted, and a new run states where its input came from). A declaration without a policy gates nothing, as for an agent. - The rule. What a node has read is the run's journal in order, not an edge: the sources of the nodes whose results are recorded before it, the flow's input when
WithUntrustedInputnames it, the marks the run inherited (below), and the marks a run its nodes started brought up. A node that can have a side effect (notReadOnly), notTaintExempt, and run after any of those needs the policy's escalation gate on a second subject,node:<name>:taint(node:iter:<n>:<name>:taintin a loop body, so a loop asks once per iteration), after its own gate and before its attempt claim; underRefuseit is refused instead (plan.ErrTaintRefused, no node record, attempt or approval request; the run stays unfinished untilagent.Cancel). An approval clears nothing: the next such node is gated again. A node does not read itself, nor the marks its own child brought up. - It is a property of the run. The run records the policy, the exempt nodes and the declared sources in
run:start; a drive under other declarations (a weaker policy included) isErrConfig, naming both digests. Roll the whole fleet before enabling a policy, and cancel a run, or restore its declarations, to change them in flight. A flow with no policy that drives a run that started under one runs only nodes that cannot have a side effect.Flow.Conformrecomputes each node's obligation from the journal and reports a node that ran without its gate. - Nested flows. A flow a node's body starts inside a run under taint (or a tool of a tainted agent that runs one) inherits the status and the marks the node had read, as an agent child does (
ext.taint_inherited, one entry per source). With a policy of its own it enforces it; with none it may run only nodes that cannot have a side effect, and a flow with such a node is refused (ErrConfig) before it writes anything. It must use its parent's journal. - Up. A nested flow sends the marks it learned to the node that started it, as
taint-mark:records in the parent's journal, so the nodes after that node are gated: the sources its own nodes declare, and what its own nested flows brought up, not the marks it inherited. It sends after each node's record, again when a drive replays the node, and for a run that finished, so a crash does not lose a mark.ResolveHaltof a node folds in the marks of the flows the node started (taint-child:records name them), then records the operator's result. - Agents and flows across. An agent a node's body runs hands the marks it read itself up to the node, before the node is recorded. A flow run as a tool of an agent marks the tool's result
subflow:<tool>when it ended with a mark (its own, or inherited), so the agent's next side effect is gated. AResolveHaltof a node that drove an agent cannot read the agent's marks (its tools' declarations are not in the journal): it marks the nodesubagent:<node name>, which over-taints, on that path only. A flow with no policy records the sources its nodes declare inrun:start(taint_sources, only when it declares some), so it marks what it reads too; they gate nothing in it, and a drive by a flow that declares other sources isErrConfig. A run that began outside taint sends them up too when a tainted node's body drives it. - Separated approvers. A flow policy with
SeparateApproversis held to the same rule:Buildrefuses a node that can be escalated whose own gate is one decision, or shares an approver ID with the policy's Gate, and the node's instance (node:<name>:taint) leaves out the approvers its own gate counted. - Not built: a
Trustedblock that brings nothing up (a node that declares its results trusted still takes the marks its nested flows and agents bring up).
Reviewing the declarations
Taint and cap declarations are Go. Nothing at runtime changes them, so a change is a code change, and the review of it needs to see the declarations, not the diff of the Go around them. The agent dumps its declarations as a document, and bide doctor reports on one and diffs two.
doc, err := a.Declarations()
if err != nil {
return err
}
_, err = w.Write(doc)
return errCommit the document next to the code (a test that writes it, and fails when it differs, keeps it current), then:
bide doctor -declarations declarations.json
bide doctor -diff main/declarations.json declarations.json- The document (
bide.declarations.v1) holds the taint policy (the escalation gate, or refuse), and every registered tool with whether it can have a side effect, its own gate, itsUntrustedsource, itsTaintExemptreason, itsTrustedreason, itsRaisesSafetyreason, whether it isIdempotent(a side effect that is retry-safe; the document carriestracks_idempotentto say so), its cap rules, and the policy'srequire_provenance. Itsdigestis thetaint_policy_digestthe runs of that agent record inrun:start, and the reader recomputes it from the document: a document whose digest is not the digest of what it declares is refused, and so is one with a member this version does not know (a document from a newer version). - A flow's declarations are read from the same two commands. The file is the
plan.RegistryDescriptionof version 2 that a program writes from itsRegistry(see Describing a Registry), with the flow's policy set fromplan.DescribeTaintPolicy; the reader tells the two kinds apart, and refuses to compare one with the other. It has no digest.-declarationslists a side-effecting block when the description gives no policy, each block with a source and no policy, each exempt block and the notes on the gate, as for an agent.-diffapplies the same rules to blocks: a removedUntrustedsource, a newTaintExempt, a gate lowered orrequire_provenanceturned off isWEAKENED. A policy that was not given to a description is not "no policy", so when either description gives none the diff reports one weakened line that the policies cannot be compared. A description of version 1 carries no taint declarations:-declarationssays taint is not known, and-diffreports one weakened line when either side lacks them, so regenerate the baseline once. The description cannot show a run's input provenance, which is a run option. -declarationsprints what a reader should look at: a side-effecting tool in an agent with no taint policy, anUntrustedtool in an agent with no policy (the marks are recorded and nothing is gated), eachTaintExempttool with its reason, each tool that declaresRaisesSafetywith its reason, each cap-carrying tool, and an escalation gate that is one decision, needs fewer than two, or does not exclude the requester. These are notes, not failures: the exit status is 0 for a document that reads. When the policy does not require provenance, each tool that declares neitherUntrustednorTrustedis listed as unclassified: it may be trusted or forgotten, and the declaration tells them apart.-diff old newneeds both documents from the same generation: a dump from beforetracks_idempotent(v0.24.0 and earlier) says nothing about which tools areIdempotent, so when one is, the diff reports one weakened line asking you to regenerate the baseline with this version (regenerate it once after upgrading), and an olderbide doctorrefuses a new document. Otherwise it lists every change asWEAKENED,changedorstrengthened, and exits 1 if anything is weakened, so a pipeline can compare a branch againstmain. Weakened: a removedUntrusted(also when aTrustedreplaces it);require_provenanceturned off (turned on is strengthened); a new or rewordedTaintExempt; a newRaisesSafety(removing it is strengthened, rewording it is changed); any tool that becomes safer to re-run (side effect toIdempotent, or either toReadOnly), since a resume then re-runs its calls (the move back is strengthened); the policy removed, orrefuseturned into a gate; a lowerneed; a requester no longer excluded; an m-of-n gate turned into one decision; a raised cap limit; a removed cap rule; a renamed cumulative rule (its name is its ledger, so the window starts again from zero); a refuse turned into a gate; a window of a day where it was an hour; a rule that applies to fewer tools; a key added to a cumulative rule (one shared total becomes one per key); a tool that becomesReadOnly(taint stops gating its calls); an approver removed or replaced in a gate, even whenneedis still met (a rotation is a change a person must approve; an approver only added is strengthened); and, because they cannot be compared and a pipeline should stop, a cap rule that changes kind (per call and cumulative), currency, exponent or key, and cap lines that do not parse on either side. Anything the report cannot judge as safe isWEAKENED. Changed, for a person to read and not stopping the pipeline: a longer or shorter gate timeout, another untrusted source name, aTrusteddeclared, removed or reworded, another amount member, the same approvers in another order (the order is part of the digest). Cap lines are compared by their meaning, through the cap grammar, not as text.- The digest and the diff must agree. Different digests with no difference in the documents mean the digest covers something the document does not carry; equal digests with a difference in what the digest covers mean the reverse. Either is an error (exit 1), not a report.
- Strict reading. The document is read so that what the text shows is what it holds: a duplicate member, a member spelled in another case, an unknown member, data after the document and a document over 16 MiB are refused, and so is a gate no valid agent has (an approver listed twice, which would count twice toward
need, aneedoutside the approvers, no approvers). - What it cannot see. The report is only as good as the dump. It does not show what happens at run time: a sub-agent tool, or a tool that starts a programmatic sub-run, marks its answer when a run it drove read untrusted content (
subagent:<tool>), and a child inherits its parent's marks, so a delegating tool taints its caller whatever the dump says. A tool's side-effect flag and its own gate are in the document but not in its digest. A tool whose wrapper hides its spec, a tool that runs an agent on a context of its own, and anything the host does not register with the agent are not in it. A difference in a tool's side-effect flag or its own gate is listed as "not in the digest".
What this does not do
This bounds what an injected request can do. It does not detect or prevent one.
- A tool that returns outside text and is not declared untrusted taints nothing. The marks are as good as the declarations.
- A human who approves a convincing injected request is a successful attack.
- A
ReadOnlytool is not gated. A "read-only" fetch whose URL carries stolen data is an exfiltration channel; declare a tool that can reach an attacker-chosen destination as a side effect. - The final answer of a tainted run goes to whoever reads it. Taint gates tools, not text.
- A tainted run that writes to a memory or index the next run trusts launders the content.
- Models that read the same hostile text can all be steered; taint does not look inside a model's words.
- Text the host injects outside the journal (a system prompt, a model middleware) is invisible to the marks.
- An input is trusted unless the entry point passes
WithUntrustedInput. - Amount caps, a separate control, bound magnitude and are not part of this policy.
- Plan flows have a rule of their own with the same policy type; see Plan flows.
The approval volume of a deployment whose agents read outside text and write will rise. That is the intended effect; TaintExempt is the relief valve, and it is a code change visible in the declarations digest.