DSPy — metric-driven agent program optimization

This extraction answers WI-027’s bounded question: what can be claimed when an agent-capable program is evaluated by a metric and an optimizer changes its prompt state. It follows the evaluation/optimization template. The selected path is DSPy’s ReAct module plus BootstrapFewShot; Evaluate and module saving establish the score and artifact boundaries. This is not a benchmark, provider comparison, or claim that an optimized program is generally better.

Snapshot and read boundary

Measured — The source is the public stanfordnlp/dspy repository at commit 80b118e52cb1f143a0d80d84685572000c59639e, read 2026-08-15. The checkout reports 3.3.0-26-g80b118e5; pyproject.toml declares package version 3.3.0, Python >=3.10, <3.15, and the dependency set including litellm, diskcache, cloudpickle, and gepa.

Measured — The bounded checkout measurement was:

git -C C:\Temp\rungs-follow-on-20260815\dspy ls-files
  563 tracked files
   279 Python files
   169 Markdown/MDX files
   135 test paths
    25 dspy/teleprompt paths
    26 dspy/predict paths

These counts describe paths at the pin. They do not measure model quality, provider availability, token cost, or the number of programs that use any optimizer.

Documented — The pinned LICENSE is MIT. The read set is deliberately narrow: pyproject.toml; dspy/predict/react.py; dspy/teleprompt/bootstrap.py; dspy/evaluate/evaluate.py; dspy/primitives/module.py and dspy/primitives/base_module.py; dspy/utils/dummies.py; and the executable tests tests/teleprompt/test_bootstrap.py and tests/evaluate/test_evaluate.py.

Executed — On 2026-08-15, Python 3.13.3 ran py -3.13 -m compileall -q over those four implementation files and two test files; the exit code was 0. The checkout had no pytest module, so the test suite was read but not presented as executed evidence. No external model, provider, benchmark, or network endpoint was called.

E1. Evaluation contract

Runtime loop: an agent program before optimization

Implemented — ReAct converts a signature into a reasoning/action predictor and adds a finish tool. forward loops for max_iters (default 20 in the implementation), records thought, tool_name, tool_args, and observation fields, executes the selected callable, and then runs a separate extraction predictor to produce the declared outputs (constructor, loop).

Implemented — A tool exception is converted to an error string in the trajectory and the loop can continue. An invalid tool selection or a context-window exception ends the action loop before extraction; context overflow gets up to three truncation attempts, removing the oldest four trajectory keys per attempt (failure path, truncation).

Boundary — The trajectory is an explicit output of ReAct, but it is not automatically a durable trace, approval record, or tool-side transaction log. A tool may have applied an external side effect before returning an error; the module does not compensate for it. These are implementation consequences of this path, not guarantees about every DSPy module or tool.

Optimization loop: BootstrapFewShot

Implemented — compile(student, teacher, trainset) first creates a reset student and a deep copy teacher. If needed, it gives the teacher up to max_labeled_demos labeled examples, checks that student and teacher have matching predictor structure, and then bootstraps examples (compile preparation).

Implemented — For each training example, _bootstrap makes at most max_rounds attempts and stops after max_bootstrapped_demos successful examples. Each attempt executes the teacher with a fresh trace; rounds after the first copy the LM with rollout_id=round_idx and temperature=1.0 to bypass caches. The metric receives (example, prediction, trace). A numeric metric is accepted when it meets metric_threshold; a truthy metric is accepted when no threshold is supplied (bootstrap loop).

Implemented — A successful trace is converted into augmented dspy.Example demonstrations for the matching predictor. Unsuccessful training examples become validation, are shuffled with random.Random(0), and are sampled as additional labeled demos. The returned student receives those demos and _compiled=True; BootstrapFewShot changes prompt examples, not model weights or the ReAct source (training result).

Implemented — The pinned tests make the contract executable in principle: test_bootstrap_effectiveness checks the compiled flag, selected demo, and prediction using DummyLM(follow_examples=True); test_error_handling_during_bootstrap checks propagation at max_errors=1; and test_validation_set_usage checks that the leftover examples are placed in the compiled demos (tests). The extraction read these tests and syntax-checked them, but did not run pytest.

Evaluation, artifact, and provenance boundaries

Implemented — Evaluate runs program(**example.inputs()), calls the metric, and uses a ParallelExecutor with configurable num_threads, max_errors, and progress settings. It keeps one result row per devset example, substitutes dspy.Prediction() plus failure_score for a failed row, and computes 100 * sum(scores) / len(devset). It can serialize the row table to CSV or JSON when the caller supplies a path (evaluation path).

Implemented — The evaluation tests cover exact-match success and failure, one- and multi-thread execution, cancellation, callbacks, and JSON/CSV output using DummyLM; they establish those bounded behaviours, not model-provider reliability or score quality (test file).

Implemented — Module.save has two materially different artifacts. State-only JSON/PKL stores dump_state() plus dependency metadata and must be loaded into a compatible program. Program saving creates a directory containing program.pkl and metadata.json; metadata records Python, DSPy, and cloudpickle versions. Program loading requires allow_pickle=True and warns that untrusted pickle can execute code. JSON is therefore the safer state interchange, while pickle is the architecture-preserving but code-executing boundary (save, load).

Boundary — A serialized compiled program does not by itself contain the dataset, metric source, provider response history, tool side effects, external files, credentials, or a held-out score. Dependency metadata helps detect drift; it does not make an LM provider deterministic.

Reproducibility, leakage, and cost

Counter-evidence — The strongest leakage risk is structural: BootstrapFewShot uses the same caller-supplied trainset to generate teacher traces and to form the leftover validation pool, then adds both successful traces and sampled leftovers to the compiled student. The optimizer has no required independent holdout and no automatic leakage detector. Any improvement claim must therefore be restricted to a separately versioned evaluation set, metric, model/provider, and run.

Counter-evidence — Evaluate aggregates arbitrary metric values as a percentage and reports no uncertainty. A score can be reproducible yet invalid for the task, or can hide per-example failures; the retained result rows are necessary evidence, not proof of metric validity.

Failure and recovery matrix

Candidate pattern consequences (deferred to WI-028)

No catalogue, module, or CLI file changed. WI-021 owns Inspect AI’s evaluation evidence, and WI-028 owns cross-track reconciliation and any catalogue decision.