Rules accumulate, and each new one changes the prompt every session runs under. A regression run replays the whole eval set — protected wins first — against the current rule set and classifies every case against its captured baseline. It is the standing answer to “did the rule we just learned break something that used to work?”
Running from code
report = await harness.run_regression()
report.clean # True when nothing regressed
report.improved, report.unchanged, report.regressed, report.skipped
for result in report.results:
print(result.case_id, result.kind, result.status, result.deltas)
baseline_scores decide the classification:
| Status | Condition |
|---|
regressed | Any shared metric dropped by rule_regress_margin or more. Regression dominates — one bad delta flags the case even if another metric improved. |
improved | No regression, and some metric rose by rule_promote_margin or more. |
unchanged | Every delta within the margins. |
skipped | Not replayable, the replay failed/timed out, or the replayed session produced no comparable scores — each with an explicit reason. |
- Wins replay first — they are the guard; with
regression_sample (or sample=) limiting the run, wins get the budget before failures.
- Replays are sequential and time-bounded (
replay_timeout_s): regression re-runs your agent, so the harness won’t multiply LLM cost or violate framework thread-safety by parallelizing it.
- Every run journals a
regression event with the per-case results — the cross-run record that a rule change was checked.
- No replay function wired? One clear warning, every case
skipped, report.clean stays true (skips are honesty, not failures) — never an exception.
The operator CLI
The same run ships as a console script over the env-configured workspace (HARNESS_ROOT, HARNESS_CLI_BINARY, …):
# Point at your replay function (an async callable, imported as module:attr):
pandaprobe-harness-eval --replay myapp.replay:replay
# Inspect the corpus without replaying anything:
pandaprobe-harness-eval --list
# Sample the first N cases (wins first); emit the full JSON report:
pandaprobe-harness-eval --replay myapp.replay:replay --sample 20 --json
Regression run: 2 case(s), 2 replayed — improved 1, unchanged 1, regressed 0, skipped 0
[win] c-0dada84f8e unchanged (agent_consistency +0.00, agent_reliability +0.00) -> s-replay-1
[failure] c-f41f46c825 improved (agent_consistency +0.48, agent_reliability +0.62) -> s-replay-1
CLEAN
0 when the run is clean, 1 on regressions or setup failure — drop it into a cron job or CI schedule and treat a non-zero exit as “a rule change needs attention.”
A good operating rhythm: enable capture_eval_cases in production, curate a handful of win cases for your critical flows, and run pandaprobe-harness-eval nightly. The journal’s regression events become an audit trail of every rule set the corpus was checked against.
