Model adapters: configuration and multimodal input
The Model port is a single method, Stream(ctx, Request) (*Stream, error) (extension points). Three reference adapters ship, all in the zero-dep core module (net/http + stdlib, no provider SDK, enforced by architecture_test.go):
| Adapter | Package | Wire API | Covers |
|---|---|---|---|
| Anthropic (native) | model/anthropic | Messages API (/v1/messages) | Claude, extended thinking with signatures preserved |
| OpenAI-compatible | model/openai | Chat Completions (/chat/completions) | OpenAI, and any compatible endpoint via WithBaseURL (Ollama, DeepSeek, Groq, OpenRouter, vLLM, Azure, xAI, ...) |
| Gemini (native) | model/gemini | streamGenerateContent (SSE) | Google AI Studio, and Vertex AI / a proxy via WithBaseURL |
Construction and options
Every adapter is New(apiKey string, opts ...Option) *Model. No adapter reads an environment variable: the API key is always passed explicitly to New, so key management is the caller's (pull it from your secret store, not from os.Getenv inside the SDK). The runnable examples/smoke shows the pattern (os.Getenv("OPENROUTER_API_KEY") in the caller, passed to openai.New).
Shared options (all three adapters): WithModel(id), WithMaxTokens(n), WithBaseURL(u), WithHTTPClient(c). Defaults if an option is not set:
| Adapter | Default model | Default max tokens | Default base URL |
|---|---|---|---|
anthropic | claude-sonnet-4-6 | 4096 | https://api.anthropic.com |
openai | gpt-4o | provider default (unset) | https://api.openai.com/v1 |
gemini | gemini-2.0-flash | provider default (unset) | https://generativelanguage.googleapis.com |
Adapter-specific options:
anthropic.WithPromptCache()enables Anthropic prompt caching (see below).openai.WithStrictSchema()turns on OpenAI structured-output strict mode for tool argument schemas ("strict": trueon each function). Use it when the endpoint is genuine OpenAI, which many "OpenAI-compatible" endpoints do not fully implement. It is off by default for compatibility. Note this flag governs only tool schemas: a native typed run (RunTypedwithagent.WithOutputMode(agent.OutputNative)) always emits a strictresponse_formatJSON schema regardless, so the native typed-output path does not need it. In strict mode an optional field (a pointer oromitempty) is required but nullable, so the model sendsnullfor it. A type strict mode cannot express (a map, a recursive type, or an untyped field such asanyorjson.RawMessage) fails the request with anErrConfigerror wrappingschema.ErrStrictUnsupported, rather than being sent as a schema that changes what the model may answer.
The Gemini adapter translates tool argument schemas and a native typed run's response schema to the OpenAPI subset Gemini reads (schema.Gemini): an optional field becomes nullable, a closed object drops additionalProperties: false, and keywords Gemini lacks that only annotate or further constrain a value are dropped. A tool with no arguments declares no parameters (Gemini rejects an object with no properties, so a tool whose schema is a bare {"type":"object"} is declared the same way). A schema the subset cannot express (a map, an interface{} or json.RawMessage field, a recursive type, $ref, oneOf, ...) fails the request with an ErrConfig error wrapping schema.ErrGeminiUnsupported that names the tool and the location.
Whatever the adapter, a Func tool decodes the arguments the model sends strictly, against the same fields its schema describes: a missing required field (or null for one whose schema does not admit null), an unknown or case-variant name, a duplicate name, trailing data, invalid UTF-8, or an escaped lone surrogate is an ErrToolArgs tool error the model reads and corrects. This matters most where the provider does not enforce the schema (Anthropic, and OpenAI without strict mode). RunTyped's final_answer and the JSON that a native typed run decodes are read the same way.
WithBaseURL is how one adapter reaches many providers. For OpenAI-compatible endpoints, set the base URL and the model, e.g. openai.New("", openai.WithBaseURL("http://localhost:11434/v1"), openai.WithModel("llama3")) for Ollama; the OpenAI adapter only sets the auth header when the API key is non-empty, so a keyless local endpoint works with an empty key. For Gemini on Vertex AI or behind a proxy, override the host with WithBaseURL; the request path (/v1beta/models/{model}:streamGenerateContent) is appended to it.
Sampling parameters, per provider
Generation controls are provider-neutral and set once on the agent (WithSampling, see the README). Each adapter maps what it supports onto its wire format and silently drops the rest, so a field one provider lacks is not an error:
agent.Sampling field | anthropic | openai | gemini |
|---|---|---|---|
MaxTokens | max_tokens | max_completion_tokens or max_tokens (below) | maxOutputTokens |
Temperature | temperature | temperature | temperature |
TopP | top_p | top_p | topP |
Stop | stop_sequences | stop | stopSequences |
Seed | dropped (no seed param) | seed | dropped (no seed param) |
The OpenAI adapter sends the token limit as max_completion_tokens on OpenAI's own endpoint (api.openai.com, where max_tokens is deprecated and the o-series and gpt-5 reasoning models reject it) and for an OpenAI reasoning model id (o1, o3-mini, gpt-5, ..., also behind a vendor/ prefix) on any endpoint; other OpenAI-compatible servers get max_tokens, which is what they implement. openai.WithMaxCompletionTokens(bool) forces the choice, for example for Azure OpenAI serving a reasoning model under a deployment name.
A request-level MaxTokens overrides the adapter's construction-time default. Because the fields are pointers, an explicit Temperature(0) is distinct from unset (which uses the provider default).
Finish reasons
Each adapter maps its provider's reason for ending a turn onto agent.Finish.Reason (an agent.FinishReason) in one neutral vocabulary, keeps the provider's own value in agent.Finish.Raw, and the core decides from the reason whether the turn is the model's answer:
| Neutral reason | Anthropic stop_reason | OpenAI finish_reason | Gemini finishReason | Result |
|---|---|---|---|---|
agent.FinishStop (stop) | end_turn, stop_sequence | stop | STOP | the message |
agent.FinishToolUse (tool_use) | tool_use | tool_calls, function_call | STOP with a call | the message; agent.ErrStreamProtocol if it carries no call |
agent.FinishLength (length) | max_tokens, model_context_window_exceeded | length | MAX_TOKENS (with or without a call) | agent.ErrOutputTruncated |
agent.FinishFiltered (filtered) | refusal | content_filter | SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII, IMAGE_SAFETY | agent.ErrOutputFiltered |
A reason an adapter does not map (Anthropic pause_turn, Gemini MALFORMED_FUNCTION_CALL or OTHER, one a provider adds later) reaches the core unchanged and fails the turn with agent.ErrStreamProtocol: the core does not guess what it means. The first-party adapters never report an empty reason: an OpenAI-compatible server that sends [DONE] with no finish_reason, or an Anthropic message_delta with no stop_reason, is agent.FinishStop with an empty Raw. An empty reason from a custom Model that does not report one counts as a stop. A Gemini stream with no finishReason stopped partway and is agent.ErrIncompleteResponse.
Each adapter also implements agent.Describer: Describe() returns an agent.ModelInfo naming the provider, the model ID, and whether the adapter honours Request.ResponseFormat. agent.ModelInfoOf(m) reads it through any wrapper that offers an Unwrap() agent.Model method.
The reason never decides whether tools run; the calls the turn carries do. A turn that calls a tool runs it whatever its reason says (OpenAI reports stop for a call made under a forced tool_choice), and a tool_use turn that carries no call is agent.ErrStreamProtocol, since the calls it was for were lost.
A turn that is cut off or filtered is not journaled, so a run never records half an answer as its final one. All three errors wrap agent.ErrModel, so middleware.Retryable retries them; a turn cut off at the token limit usually ends the same way again, so raise MaxTokens for it.
What a model turn journals
The agent's model handler returns an agent.ModelResponse: the Message, its Usage, the neutral Finish reason and the provider's RawFinish. An empty reason (a custom Model that does not report one) is recorded as agent.FinishStop. A response a middleware builds itself (a fallback, a cache) is checked the same way: FinishLength and FinishFiltered are the errors above, and an unknown reason is agent.ErrStreamProtocol.
Each model record (agent.Record of kind StepModel) journals, beside the message and usage, the turn's metadata. The record holds it in Record.ModelTurn (a *agent.ModelTurn, nil when the record carries none) and you read it through the nil-safe accessors Finish(), RawFinish(), Model(), PromptDigest() and ToolsDigest(); the journal encoding writes each as a member of the record:
FinishandRawFinish, whichagent.Replayreproduces;Model, theagent.ModelInfoof the model that answered (nilwhen it does not describe itself, or when a middleware built the response), so a run that fell back to another provider shows which turn went where;PromptDigestandToolsDigest, SHA-256 digests (agent.PromptDigest,agent.ToolsDigest) of the system prompt and the tool set that turn was sent, after middleware. Agent-level defaults stay live across a redeploy; the digests let an auditor tell which instructions and tools each answer was given.
To call a model outside an agent through the same checks and middleware, use agent.CallModel(ctx, model, req, mw...).
Prompt caching and usage accounting
An agent loop resends a large constant prefix (system prompt + tool schemas) every turn.
- Anthropic (
WithPromptCache()): placescache_control: {type: ephemeral}breakpoints on the last tool definition (caching the whole constant tool prefix) and the system block, so repeat turns bill the prefix at the cache-read rate. - OpenAI: caches prefixes automatically, no flag needed.
- Gemini: no explicit prompt-cache flag in this adapter.
Cache effectiveness surfaces cross-provider in agent.Usage, so the run's token budget and cost accounting see the real numbers:
CacheReadTokens: Anthropiccache_read_input_tokens; OpenAIprompt_tokens_details.cached_tokens; Gemini's cached-content tokens.CacheWriteTokens: Anthropiccache_creation_input_tokens. OpenAI and Gemini do not report a separate cache-write count, so this stays zero for them.InputTokens: the input tokens that were neither read from nor written to the cache. Anthropic reports these directly asinput_tokens. OpenAI'sprompt_tokensand Gemini'spromptTokenCountinclude the cached tokens, so those adapters subtract them.
The counts are disjoint, so each token is billed once at its own rate and the same call costs the same on every provider. Usage.TotalInputTokens() is the whole prompt and Usage.TotalTokens() adds the output.
Error surfacing (shared across all three adapters)
Every adapter maps HTTP failures onto the framework's typed errors so a retry middleware can classify them (retry, in the tour):
The provider's error object ({"error":{...}}, the same envelope on all three) is parsed, so the error carries the provider's own message.
- Quota or credit used up returns
*agent.APIErrorwrappingagent.ErrQuotaExhausted, whatever the status: HTTP 402, an error type or code ofinsufficient_quota,billing_hard_limit_reached, orbilling_not_active(OpenAI, which sends these as 429),billing_error(Anthropic), or a GeminiRESOURCE_EXHAUSTEDwhose quota violation is per day. No wait lifts it, somiddleware.Retryabledoes not retry it. - Any other HTTP 429 returns
*agent.RateLimited{RetryAfter, Message}.RetryAfteris theRetry-Afterheader (seconds or an HTTP date) or, without one, Gemini'sRetryInfo.retryDelay;0if neither.middleware.Retryhonors the hint. - Any other non-2xx returns
*agent.APIError{StatusCode, Message, Type, Code, Body}, somiddleware.Retryablecan tell a transient failure (5xx, 408) from a terminal one (most 4xx: auth, validation). - An error sent partway through a stream (an OpenAI or Gemini data line holding an error object, an Anthropic
errorevent) ends the turn with an error carrying the provider's message: classified by its status when it has one (Gemini), asErrQuotaExhaustedor*RateLimitedwhen its type or code says so, and otherwise as a plainErrModelthatRetryableretries. - A stream line over
provider.MaxSSELine(32 MiB, packagemodel/provider), or a whole reply over the adapter's response cap, fails withagent.ErrResponseTooLarge, which the same request would hit again, so it is not retried. The response cap isprovider.DefaultMaxResponseBytes(32 MiB) unless the adapter is built withWithMaxResponseBytes(n); it counts every byte of the streamed reply, so a reply of many small events cannot grow without limit, and a line is part of the reply, so a cap below 32 MiB bounds every line too. Any other failed stream read wrapsErrModel. - All of these wrap
agent.ErrModel, soerrors.Is(err, agent.ErrModel)holds either way (see the errors section of the README).
Error text is bounded whatever the endpoint sends: at most 64KB of a failed response is read, and the body, message, type, and code an error carries are each cut to 8KB (a cut body ends in ...(truncated)), so a broken endpoint, or one that echoes the prompt back, cannot put megabytes into memory, logs, or traces.
Tool names and tool choice
Each adapter checks a request's tools and tool choice against the provider's rules before sending it, and refuses a setup the provider would answer with a 400 as an ErrConfig that says what is wrong:
- A tool name must match
^[a-zA-Z0-9_-]{1,64}$(OpenAI, Anthropic) or^[a-zA-Z_][a-zA-Z0-9_.:-]{0,63}$(Gemini), and no two tools may share a name. - A
ToolChoicemode must be"",auto,none,required, ortool.requiredandtoolneed declared tools, andtoolmust name one of them. autoornonewith no tools declared is already met, so no tool choice is sent (the providers reject one with no tools).
Each adapter declares its rules through agent.ToolRules (ToolNameRule, RequiresToolsForRequired), so agent.New and Agent.With refuse a tool name the agent's model cannot take when the agent is built, not on every run; a model that declares no rules is not checked. Two tools with one name, an unknown mode, and tool naming a tool the agent lacks are refused for any model. required on an agent with no tools of its own is checked at the run, since RunTyped supplies an answer tool: a run with nothing to call fails with ErrConfig before it opens the journal.
Tool-call IDs
The agent keys each tool call's result and journal step by its tool-use ID, so every call in a run needs its own. Anthropic and OpenAI issue one per call. Gemini usually sends none: the Gemini adapter keeps an id when Gemini sends one and otherwise makes up a random call_<24 hex digits>, which carries nothing from the tool name or the call's position.
The agent checks every live model turn whatever the adapter: a tool call with no ID, an ID already used earlier in the conversation, one repeated within the turn, or one that is not valid UTF-8 fails the turn with agent.ErrToolUseIDReused (wrapping ErrProtocol and ErrModel) instead of passing the call off as already done. An ID that holds a # fails the turn with agent.ErrToolUseIDInvalid (also wrapping ErrProtocol and ErrModel): the engine reserves # in an ID, to derive a second approval subject from a call's ID. The shipped adapters pass the provider's ID through and none of their IDs holds one. The check runs below middleware, so middleware.Retry retries either, and the rejected turn is never journaled. Resuming a run from an existing journal does not re-check its recorded turns, so a journal that holds a # ID from before the check still resumes. agent.Replay used as the model of a new run is different: it hands the recorded turns back as that run's live model turns, so a recorded # ID fails the new run with ErrToolUseIDInvalid (the same check, fail closed).
Any other ID is safe, whatever characters it has or however long it is. The journal keys derived from an ID (agent.ToolResultStep, the call's attempt marker, approvals, compensation) and a sub-agent's run ID (agent.SubRunID) carry it encoded after a prefix of their own: bytes outside [A-Za-z0-9._-] become %XX, and an ID whose encoding would pass 96 bytes becomes ~ and its SHA-256. An ID such as @llm/1, run:complete or a/b therefore cannot name one of the run's own steps, another call's key, or another call's sub-run.
Multimodal input (images)
User messages carry mixed content parts, so an image can accompany text on a user turn. Build a multimodal user message with agent.UserParts:
img, _ := os.ReadFile("chart.png")
msg := agent.UserParts(
agent.Text{Text: "What trend does this chart show?"},
agent.ImageData("image/png", img), // raw bytes; the adapter base64-encodes it
)
// agent.ImageURL("https://...") references a hosted image instead.agent.Image sets exactly one of Data (raw bytes, plus Mime) or URL (a hosted image). All three adapters translate it to their native form (Anthropic base64 image source, OpenAI image-URL / data-URI content, Gemini inlineData / fileData). Gemini's fileData requires a MIME type: for an image by URL it is Image.Mime, or else the type the URL's file extension names; with neither, the request fails with ErrConfig. This is input-only: models emit text, reasoning, and tool calls, never images, so nothing produces an Image on the response path. Audio and video input are not modeled (see KNOWN-LIMITATIONS.md).
Extended thinking (Anthropic)
The Anthropic adapter preserves extended-thinking signatures: a Reasoning part carries the provider's opaque Signature, which is echoed back on later turns. Dropping it corrupts thinking + tool use, which is why message content is typed parts rather than a flat string. Each thinking block is its own Reasoning part with its own signature, in the order the model produced them, and a redacted_thinking block is kept as a Reasoning part whose Redacted field holds its encrypted data, sent back unchanged. A Reasoning part with neither a signature nor redacted data (reasoning from another provider) is not sent to Anthropic, which would reject it.
Other Anthropic request details: empty or whitespace-only text blocks are left out (Anthropic rejects them), system text from several parts or turns is joined with a blank line, and ToolChoice{Mode: "none"} is sent as {"type": "none"} with the tools still declared (Anthropic requires the declarations whenever the history holds tool_use blocks).
The OpenAI and Gemini adapters take the opposite, provider-correct stance: they do not send a prior Reasoning part back on an assistant-input turn (the providers reject it, and there is no stable signature to echo), so request-side Reasoning parts are dropped rather than sent with an invalid token. The OpenAI adapter still surfaces inbound reasoning it receives (reasoning_content from DeepSeek / Ollama and similar) as a ReasoningDelta on the stream, and the Gemini adapter surfaces a part Gemini flags thought the same way.
Gemini's thinking models attach a thoughtSignature to a functionCall part and reject the next turn unless it comes back on that part. The adapter keeps it on the call (agent.ToolUse.Signature, journaled with the call) and sends it back with the functionCall. Signatures Gemini puts on text parts are not kept; Gemini does not require them.
GCF tool-result encoding (opt-in)
Tool results are sent to the model as JSON by default. For structured output you can switch the model-facing encoding to GCF, the Graph Compact Format: a compact wire format for structured data that is more token-efficient and better comprehended by frontier models than JSON. Switching cuts the prompt tokens spent echoing tool output back each turn. It is opt-in per adapter and changes only what the model reads: the journal and the audit trail keep the canonical JSON form, so durability and proofs are unaffected.
Wire it in with any adapter's WithToolResultCodec:
import (
"github.com/bide-ai/bide/model/openai"
"github.com/bide-ai/bide/codec/gcf"
)
model := openai.New(apiKey, openai.WithToolResultCodec(gcf.New()))The same option exists on the Anthropic and Gemini adapters. GCF lives in its own module (github.com/bide-ai/bide/codec/gcf), so the core takes no GCF dependency unless you opt in. Best paired with current frontier models; weaker or older models may comprehend the compact form less reliably, so treat it as a per-deployment tuning knob rather than a default. If encoding a particular result ever fails, the adapter falls back to the JSON form for that result.
What is not built
- No native Bedrock adapter (reach Bedrock-hosted models through an OpenAI-compatible proxy).
- Settings are agent-level, not per-
Run(see KNOWN-LIMITATIONS.md).