Regression runs - PandaProbe

Rules accumulate, and each new one changes the prompt every session runs under. A regression run replays the whole eval set — protected wins first — against the current rule set and classifies every case against its captured baseline. It is the standing answer to “did the rule we just learned break something that used to work?”

Running from code

report = await harness.run_regression()

report.clean          # True when nothing regressed
report.improved, report.unchanged, report.regressed, report.skipped
for result in report.results:
    print(result.case_id, result.kind, result.status, result.deltas)

Each case replays through your replay function with the full current rules in context, the new session is scored, and per-metric deltas against baseline_scores decide the classification:

Status	Condition
`regressed`	Any shared metric dropped by `rule_regress_margin` or more. Regression dominates — one bad delta flags the case even if another metric improved.
`improved`	No regression, and some metric rose by `rule_promote_margin` or more.
`unchanged`	Every delta within the margins.
`skipped`	Not replayable, the replay failed/timed out, or the replayed session produced no comparable scores — each with an explicit `reason`.

Semantics worth knowing:

Wins replay first — they are the guard; with regression_sample (or sample=) limiting the run, wins get the budget before failures.
Replays are sequential and time-bounded (replay_timeout_s): regression re-runs your agent, so the harness won’t multiply LLM cost or violate framework thread-safety by parallelizing it.
Every run journals a regression event with the per-case results — the cross-run record that a rule change was checked.
No replay function wired? One clear warning, every case skipped, report.clean stays true (skips are honesty, not failures) — never an exception.

The operator CLI

The same run ships as a console script over the env-configured workspace (HARNESS_ROOT, HARNESS_CLI_BINARY, …):

# Point at your replay function (an async callable, imported as module:attr):
pandaprobe-harness-eval --replay myapp.replay:replay

# Inspect the corpus without replaying anything:
pandaprobe-harness-eval --list

# Sample the first N cases (wins first); emit the full JSON report:
pandaprobe-harness-eval --replay myapp.replay:replay --sample 20 --json

Regression run: 2 case(s), 2 replayed — improved 1, unchanged 1, regressed 0, skipped 0
  [win] c-0dada84f8e unchanged (agent_consistency +0.00, agent_reliability +0.00) -> s-replay-1
  [failure] c-f41f46c825 improved (agent_consistency +0.48, agent_reliability +0.62) -> s-replay-1
CLEAN

Exit code 0 when the run is clean, 1 on regressions or setup failure — drop it into a cron job or CI schedule and treat a non-zero exit as “a rule change needs attention.”

A good operating rhythm: enable capture_eval_cases in production, curate a handful of win cases for your critical flows, and run pandaprobe-harness-eval nightly. The journal’s regression events become an audit trail of every rule set the corpus was checked against.

​Running from code

​The operator CLI

Running from code

The operator CLI