Proposal (rationale)
The current corpus contains deterministic model substitutes, structured-output validators, traces, and tests, but it does not read an evaluation system as the subject. That leaves an important gap: how a task definition, agent trajectory, sandbox, scorer, evidence log, aggregation, retry, and reproduction contract fit together.
Inspect AI is selected because its public source is explicitly organized around language-model evaluations and includes tool-using, multi-turn and sandboxed execution. The extraction asks what an evaluation result actually proves, what evidence survives the run, and where model-graded or human judgement enters a result that may otherwise look mechanical.
Decision
proposed — 2026-08-15. Captured as the evaluation-system child of
WI-018, after WI-019 defines the track template.
Plan
Requirements
- Pin repository SHA, licence, date, measured scale, and the exact evaluation packages and tests read.
- Trace one agentic evaluation from dataset/sample construction through solver execution, model and tool calls, sandbox interaction, scoring, aggregation, and durable result log.
- State which inputs and environment details are sufficient to reproduce a run and which external model or infrastructure behaviours remain outside the record.
- Distinguish deterministic scoring, model-graded scoring, human review, and aggregation; state the claim boundary of each.
- Trace failure, retry, cancellation, partial result, and resume semantics where present.
- Map findings to gate, audit, evidence, bounded-loop, and sandbox candidates without editing the catalogue.
Impacts
- One evaluation/optimization extraction and an index row under WI-019’s layout.
- Candidate evidence for the distinction between execution traces, audit inputs, gates, scores, and conclusions; WI-028 decides catalogue consequences.
Approach
Choose one small evaluation that uses an agent loop and a sandbox, then follow its actual objects and stored artifacts end to end. Use built-in components rather than a third-party evaluation so the read boundary remains in one pinned repository. Treat the presence of many evaluations as an inventory claim to measure, not a proxy for framework quality.
Acceptance criteria / tests
- Snapshot and read boundary make every implementation and count claim reproducible.
- One evaluation is traced from sample input to stored result and aggregate score.
- The extraction states what the log can replay, audit, or merely describe, with failure paths included.
- Every scorer type inspected has an explicit claim boundary; a numeric result is not presented as truth.
- The sandbox trust boundary and external dependencies are traced or bounded as not found.
- Opinion is labelled; catalogue candidates are deferred to WI-028; gates and site links pass.
Out of scope
- Running expensive model evaluations or comparing model performance.
- Surveying the external evaluation collection as a second corpus.
- Security certification of sandbox backends.
- Catalogue, module, or CLI changes.
Execution
Not started.
Review
Not started.