Package eval
Go module github.com/bide-ai/bide; all packages.
github.com/bide-ai/bide/eval · pkg.go.dev · Stability: experimental
Package eval is a statistical evaluation harness for agents.
Constants
const ReportFormat = "bide.eval.report.v2": ReportFormat is the format tag Run stamps on every Report (Report.Format).
Variables
var ErrFormat: ErrFormat is a Report whose Format is not ReportFormat: one from another layout, or a zero Report that no Run produced.var ErrInconclusive: ErrInconclusive is Gate's verdict when a metric could not be compared, or nothing was.var ErrRegression: ErrRegression is Gate's verdict when a metric regressed significantly.
Functions
func HashCases(cases []Case) string: HashCases returns a stable lowercase-hex sha256 over the case set, in order.func RequiredRuns(baselineRate, minDetectableDrop, alpha, power float64) (int, error): RequiredRuns returns the per-arm sample size needed to detect a drop of minDetectableDrop from baselineRate in a two-proportion comparison at the given alpha (two-sided) and power, using the standard normal-approximation formula.
type Case
type Case struct {
Name string
Input string
Want any
Tags []string
}Case is one evaluation input plus optional expectations that metrics interpret.
type CaseReport
type CaseReport struct {
Name string `json:"name"`
Hash string `json:"hash"`
Metrics map[string]MetricStat `json:"metrics"`
}CaseReport holds one case's per-metric stats over its RunsPerCase executions.
type CompareOption
type CompareOption interface {
// contains filtered or unexported methods
}CompareOption configures Compare.
Constructors and functions
func WithCaseSetMismatchAllowed() CompareOption: WithCaseSetMismatchAllowed lets Compare compare two reports whose case sets differ (Provenance.CaseSetHash), over only the cases both ran: a case is shared when a CaseReport of each report has the same Hash, so a case whose input or expectation changed is not shared.func WithUnscoredTolerance(frac float64) CompareOption: WithUnscoredTolerance sets the largest share of a metric's runs, in each report, that may be unscored for Compare to still compare it: frac is Unscored/(Scored+Unscored) and must be in [0, 1].
type Comparison
type Comparison struct {
Metrics []MetricComparison `json:"metrics"`
}Comparison is the full set of per-metric comparisons between two Reports, over the metrics present in either.
Constructors and functions
func Compare(old, new Report, opts ...CompareOption) (Comparison, error): Compare pairs each metric in either report's Overall stats and tests whether its pass rate changed.
Methods
func (c Comparison) Gate() error: Gate is the comparison as a CI gate: nil only when at least one metric was compared and every metric is DirectionFlat or DirectionImprovement.func (c Comparison) String() string: String renders a comparison as a readable table: rate delta, raw and adjusted p-values, and the significance direction per metric, or why a metric is inconclusive.
type Metric
type Metric struct {
Name string
Fn func(ctx context.Context, c Case, out RunOutput) (bool, error)
// contains filtered or unexported fields
}Metric scores a single run of a case.
Constructors and functions
func CalledTool(name string) Metric: CalledTool passes iff the run called a tool with this name at least once.func Contains(substr string) Metric: Contains passes iff the final message text contains substr.func Custom(name string, fn func(ctx context.Context, c Case, out RunOutput) (bool, error)) Metric: Custom builds a metric from a name and a predicate with Metric.Fn's contract: an error leaves the run unscored.func GovernanceHeld(name string, compliant func(ctx context.Context, out RunOutput) (bool, error)) Metric: GovernanceHeld passes iff compliant(ctx, out) returns true, and leaves the run unscored when it returns an error (the governance log it reads could not be read, for example).func Judge(name string, m agent.Model, rubric string) Metric: Judge is an LLM-as-judge metric: it asks a model whether the output satisfies rubric, passing iff the judge's reply, less surrounding whitespace, is exactly PASS.func Matches(re *regexp.Regexp) Metric: Matches passes iff the run did not error and the final message text matches re.func MaxSteps(n int) Metric: MaxSteps passes iff the run took at most n model turns (a guard against looping/thrashing).func NoError() Metric: NoError passes iff the run did not error.func ToolOrder(names ...string) Metric: ToolOrder passes iff the named tools were each called, in the given relative order (as a subsequence of the actual call order).
type MetricComparison
type MetricComparison struct {
Metric string `json:"metric"`
OldRate float64 `json:"old_rate"`
NewRate float64 `json:"new_rate"`
RateDelta float64 `json:"rate_delta"`
OldPasses int `json:"old_passes"`
OldScored int `json:"old_scored"`
OldUnscored int `json:"old_unscored"`
NewPasses int `json:"new_passes"`
NewScored int `json:"new_scored"`
NewUnscored int `json:"new_unscored"`
PValue float64 `json:"p_value"`
AdjustedP float64 `json:"adjusted_p"`
Significant bool `json:"significant"`
Direction MetricDirection `json:"direction"`
Note string `json:"note,omitempty"`
}MetricComparison is one metric's before/after comparison between two Reports.
type MetricDirection
type MetricDirection stringMetricDirection is the reading of one metric's change between two Reports.
Constants
const DirectionRegression MetricDirection = "regression": DirectionRegression is a significant drop in pass rate.const DirectionImprovement MetricDirection = "improvement": DirectionImprovement is a significant rise in pass rate.const DirectionFlat MetricDirection = "flat": DirectionFlat is a change that is not significant (or no change).const DirectionInconclusive MetricDirection = "inconclusive": DirectionInconclusive is a metric that was not compared (MetricComparison.Note says why).
type MetricStat
type MetricStat struct {
Passes int `json:"passes"`
Scored int `json:"scored"`
Unscored int `json:"unscored"`
Rate float64 `json:"rate"`
CILow float64 `json:"ci_low"`
CIHigh float64 `json:"ci_high"`
}MetricStat is a metric's pass count over some runs, with a 95% Wilson score confidence interval on the true pass rate.
type Options
type Options struct {
Runs int
Concurrency int
Provenance Provenance
}Options configures a run.
type Provenance
type Provenance struct {
ModelID string `json:"model_id"`
Temperature float64 `json:"temperature"`
Seed int64 `json:"seed"`
Timestamp time.Time `json:"timestamp"`
CaseSetHash string `json:"case_set_hash"`
}Provenance records the conditions under which an evaluation ran, so a Report can be reproduced and audited: which model version, at what temperature and seed, when, and a hash of the exact case set scored.
type Report
type Report struct {
Format string `json:"format"`
RunsPerCase int `json:"runs_per_case"`
TotalRuns int `json:"total_runs"`
Cases []CaseReport `json:"cases"`
Overall map[string]MetricStat `json:"overall"`
ByTag map[string]map[string]MetricStat `json:"by_tag,omitempty"`
LatencyP50 time.Duration `json:"latency_p50"`
LatencyP95 time.Duration `json:"latency_p95"`
Provenance Provenance `json:"provenance"`
}Report is the statistical result of an evaluation: per-case and aggregate pass rates with confidence intervals, plus latency percentiles.
Constructors and functions
func Run(ctx context.Context, run RunFunc, cases []Case, metrics []Metric, opts Options) (Report, error): Run executes each case Runs times through run, scores every execution with each metric, and returns a Report of pass rates (with Wilson confidence intervals) and latency percentiles.
Methods
func (r Report) String() string: String renders the report as a readable table with confidence intervals and latency: passes over scored runs, and the unscored count where there is one.
type RunFunc
type RunFunc func(ctx context.Context, input string) RunOutputRunFunc executes one case input and returns its output.
Constructors and functions
func AgentRunner(a *agent.Agent, store *agent.Journal, runIDPrefix string) (RunFunc, error): AgentRunner wraps an Agent as a RunFunc, giving each run a unique runID so evaluation runs are independent, durable, and auditable.
type RunOutput
type RunOutput struct {
Final agent.Message
Err error
RunID string
Trace Trajectory
TraceErr error
}RunOutput is the result of executing one case: the final message, any error, the runID, and the trajectory (populated by AgentRunner; empty for a bare RunFunc).
type Trajectory
type Trajectory struct {
Steps int
ToolCalls []string
Records []agent.Record
}Trajectory is the agent's behavior on one run, reconstructed from the durable journal: how many model turns it took and which tools it called, in order, plus the raw records for custom metrics.
Constructors and functions
func TrajectoryFrom(records []agent.Record) Trajectory: TrajectoryFrom reconstructs a Trajectory from a run's journal records.