Skip to content

Package eval ​

Go module github.com/bide-ai/bide; all packages.

github.com/bide-ai/bide/eval · pkg.go.dev · Stability: experimental

Package eval is a statistical evaluation harness for agents.

Constants

  • const ReportFormat = "bide.eval.report.v2": ReportFormat is the format tag Run stamps on every Report (Report.Format).

Variables

  • var ErrFormat: ErrFormat is a Report whose Format is not ReportFormat: one from another layout, or a zero Report that no Run produced.
  • var ErrInconclusive: ErrInconclusive is Gate's verdict when a metric could not be compared, or nothing was.
  • var ErrRegression: ErrRegression is Gate's verdict when a metric regressed significantly.

Functions

  • func HashCases(cases []Case) string: HashCases returns a stable lowercase-hex sha256 over the case set, in order.
  • func RequiredRuns(baselineRate, minDetectableDrop, alpha, power float64) (int, error): RequiredRuns returns the per-arm sample size needed to detect a drop of minDetectableDrop from baselineRate in a two-proportion comparison at the given alpha (two-sided) and power, using the standard normal-approximation formula.

type Case ​

go
type Case struct {
	Name  string
	Input string
	Want  any
	Tags  []string
}

Case is one evaluation input plus optional expectations that metrics interpret.

type CaseReport ​

go
type CaseReport struct {
	Name string `json:"name"`

	Hash    string                `json:"hash"`
	Metrics map[string]MetricStat `json:"metrics"`
}

CaseReport holds one case's per-metric stats over its RunsPerCase executions.

type CompareOption ​

go
type CompareOption interface {
	// contains filtered or unexported methods
}

CompareOption configures Compare.

Constructors and functions

  • func WithCaseSetMismatchAllowed() CompareOption: WithCaseSetMismatchAllowed lets Compare compare two reports whose case sets differ (Provenance.CaseSetHash), over only the cases both ran: a case is shared when a CaseReport of each report has the same Hash, so a case whose input or expectation changed is not shared.
  • func WithUnscoredTolerance(frac float64) CompareOption: WithUnscoredTolerance sets the largest share of a metric's runs, in each report, that may be unscored for Compare to still compare it: frac is Unscored/(Scored+Unscored) and must be in [0, 1].

type Comparison ​

go
type Comparison struct {
	Metrics []MetricComparison `json:"metrics"`
}

Comparison is the full set of per-metric comparisons between two Reports, over the metrics present in either.

Constructors and functions

  • func Compare(old, new Report, opts ...CompareOption) (Comparison, error): Compare pairs each metric in either report's Overall stats and tests whether its pass rate changed.

Methods

  • func (c Comparison) Gate() error: Gate is the comparison as a CI gate: nil only when at least one metric was compared and every metric is DirectionFlat or DirectionImprovement.
  • func (c Comparison) String() string: String renders a comparison as a readable table: rate delta, raw and adjusted p-values, and the significance direction per metric, or why a metric is inconclusive.

type Metric ​

go
type Metric struct {
	Name string
	Fn   func(ctx context.Context, c Case, out RunOutput) (bool, error)
	// contains filtered or unexported fields
}

Metric scores a single run of a case.

Constructors and functions

  • func CalledTool(name string) Metric: CalledTool passes iff the run called a tool with this name at least once.
  • func Contains(substr string) Metric: Contains passes iff the final message text contains substr.
  • func Custom(name string, fn func(ctx context.Context, c Case, out RunOutput) (bool, error)) Metric: Custom builds a metric from a name and a predicate with Metric.Fn's contract: an error leaves the run unscored.
  • func GovernanceHeld(name string, compliant func(ctx context.Context, out RunOutput) (bool, error)) Metric: GovernanceHeld passes iff compliant(ctx, out) returns true, and leaves the run unscored when it returns an error (the governance log it reads could not be read, for example).
  • func Judge(name string, m agent.Model, rubric string) Metric: Judge is an LLM-as-judge metric: it asks a model whether the output satisfies rubric, passing iff the judge's reply, less surrounding whitespace, is exactly PASS.
  • func Matches(re *regexp.Regexp) Metric: Matches passes iff the run did not error and the final message text matches re.
  • func MaxSteps(n int) Metric: MaxSteps passes iff the run took at most n model turns (a guard against looping/thrashing).
  • func NoError() Metric: NoError passes iff the run did not error.
  • func ToolOrder(names ...string) Metric: ToolOrder passes iff the named tools were each called, in the given relative order (as a subsequence of the actual call order).

type MetricComparison ​

go
type MetricComparison struct {
	Metric      string          `json:"metric"`
	OldRate     float64         `json:"old_rate"`
	NewRate     float64         `json:"new_rate"`
	RateDelta   float64         `json:"rate_delta"`
	OldPasses   int             `json:"old_passes"`
	OldScored   int             `json:"old_scored"`
	OldUnscored int             `json:"old_unscored"`
	NewPasses   int             `json:"new_passes"`
	NewScored   int             `json:"new_scored"`
	NewUnscored int             `json:"new_unscored"`
	PValue      float64         `json:"p_value"`
	AdjustedP   float64         `json:"adjusted_p"`
	Significant bool            `json:"significant"`
	Direction   MetricDirection `json:"direction"`
	Note        string          `json:"note,omitempty"`
}

MetricComparison is one metric's before/after comparison between two Reports.

type MetricDirection ​

go
type MetricDirection string

MetricDirection is the reading of one metric's change between two Reports.

Constants

  • const DirectionRegression MetricDirection = "regression": DirectionRegression is a significant drop in pass rate.
  • const DirectionImprovement MetricDirection = "improvement": DirectionImprovement is a significant rise in pass rate.
  • const DirectionFlat MetricDirection = "flat": DirectionFlat is a change that is not significant (or no change).
  • const DirectionInconclusive MetricDirection = "inconclusive": DirectionInconclusive is a metric that was not compared (MetricComparison.Note says why).

type MetricStat ​

go
type MetricStat struct {
	Passes   int     `json:"passes"`
	Scored   int     `json:"scored"`
	Unscored int     `json:"unscored"`
	Rate     float64 `json:"rate"`
	CILow    float64 `json:"ci_low"`
	CIHigh   float64 `json:"ci_high"`
}

MetricStat is a metric's pass count over some runs, with a 95% Wilson score confidence interval on the true pass rate.

type Options ​

go
type Options struct {
	Runs        int
	Concurrency int

	Provenance Provenance
}

Options configures a run.

type Provenance ​

go
type Provenance struct {
	ModelID     string    `json:"model_id"`
	Temperature float64   `json:"temperature"`
	Seed        int64     `json:"seed"`
	Timestamp   time.Time `json:"timestamp"`
	CaseSetHash string    `json:"case_set_hash"`
}

Provenance records the conditions under which an evaluation ran, so a Report can be reproduced and audited: which model version, at what temperature and seed, when, and a hash of the exact case set scored.

type Report ​

go
type Report struct {
	Format      string                           `json:"format"`
	RunsPerCase int                              `json:"runs_per_case"`
	TotalRuns   int                              `json:"total_runs"`
	Cases       []CaseReport                     `json:"cases"`
	Overall     map[string]MetricStat            `json:"overall"`
	ByTag       map[string]map[string]MetricStat `json:"by_tag,omitempty"`
	LatencyP50  time.Duration                    `json:"latency_p50"`
	LatencyP95  time.Duration                    `json:"latency_p95"`
	Provenance  Provenance                       `json:"provenance"`
}

Report is the statistical result of an evaluation: per-case and aggregate pass rates with confidence intervals, plus latency percentiles.

Constructors and functions

  • func Run(ctx context.Context, run RunFunc, cases []Case, metrics []Metric, opts Options) (Report, error): Run executes each case Runs times through run, scores every execution with each metric, and returns a Report of pass rates (with Wilson confidence intervals) and latency percentiles.

Methods

  • func (r Report) String() string: String renders the report as a readable table with confidence intervals and latency: passes over scored runs, and the unscored count where there is one.

type RunFunc ​

go
type RunFunc func(ctx context.Context, input string) RunOutput

RunFunc executes one case input and returns its output.

Constructors and functions

  • func AgentRunner(a *agent.Agent, store *agent.Journal, runIDPrefix string) (RunFunc, error): AgentRunner wraps an Agent as a RunFunc, giving each run a unique runID so evaluation runs are independent, durable, and auditable.

type RunOutput ​

go
type RunOutput struct {
	Final    agent.Message
	Err      error
	RunID    string
	Trace    Trajectory
	TraceErr error
}

RunOutput is the result of executing one case: the final message, any error, the runID, and the trajectory (populated by AgentRunner; empty for a bare RunFunc).

type Trajectory ​

go
type Trajectory struct {
	Steps     int
	ToolCalls []string
	Records   []agent.Record
}

Trajectory is the agent's behavior on one run, reconstructed from the durable journal: how many model turns it took and which tools it called, in order, plus the raw records for custom metrics.

Constructors and functions

  • func TrajectoryFrom(records []agent.Record) Trajectory: TrajectoryFrom reconstructs a Trajectory from a run's journal records.

Apache-2.0 licensed.