> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pandaprobe.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Regression runs

> Did a new rule break an old win? Replay the eval set against the current rules

Rules accumulate, and each new one changes the prompt every session runs under. A **regression run** replays the whole eval set — protected wins first — against the *current* rule set and classifies every case against its captured baseline. It is the standing answer to "did the rule we just learned break something that used to work?"

## Running from code

```python theme={null}
report = await harness.run_regression()

report.clean          # True when nothing regressed
report.improved, report.unchanged, report.regressed, report.skipped
for result in report.results:
    print(result.case_id, result.kind, result.status, result.deltas)
```

Each case replays through your [replay function](/harness/closed-loop/eval-set-and-replay#the-replay-function) with the full current rules in context, the new session is scored, and per-metric deltas against `baseline_scores` decide the classification:

| Status      | Condition                                                                                                                                        |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `regressed` | Any shared metric dropped by `rule_regress_margin` or more. Regression dominates — one bad delta flags the case even if another metric improved. |
| `improved`  | No regression, and some metric rose by `rule_promote_margin` or more.                                                                            |
| `unchanged` | Every delta within the margins.                                                                                                                  |
| `skipped`   | Not replayable, the replay failed/timed out, or the replayed session produced no comparable scores — each with an explicit `reason`.             |

Semantics worth knowing:

* **Wins replay first** — they are the guard; with `regression_sample` (or `sample=`) limiting the run, wins get the budget before failures.
* **Replays are sequential and time-bounded** (`replay_timeout_s`): regression re-runs *your agent*, so the harness won't multiply LLM cost or violate framework thread-safety by parallelizing it.
* Every run journals a `regression` event with the per-case results — the cross-run record that a rule change was checked.
* **No replay function wired?** One clear warning, every case `skipped`, `report.clean` stays `true` (skips are honesty, not failures) — never an exception.

## The operator CLI

The same run ships as a console script over the env-configured workspace (`HARNESS_ROOT`, `HARNESS_CLI_BINARY`, …):

```bash theme={null}
# Point at your replay function (an async callable, imported as module:attr):
pandaprobe-harness-eval --replay myapp.replay:replay

# Inspect the corpus without replaying anything:
pandaprobe-harness-eval --list

# Sample the first N cases (wins first); emit the full JSON report:
pandaprobe-harness-eval --replay myapp.replay:replay --sample 20 --json
```

```text theme={null}
Regression run: 2 case(s), 2 replayed — improved 1, unchanged 1, regressed 0, skipped 0
  [win] c-0dada84f8e unchanged (agent_consistency +0.00, agent_reliability +0.00) -> s-replay-1
  [failure] c-f41f46c825 improved (agent_consistency +0.48, agent_reliability +0.62) -> s-replay-1
CLEAN
```

Exit code `0` when the run is clean, `1` on regressions or setup failure — drop it into a cron job or CI schedule and treat a non-zero exit as "a rule change needs attention."

<Tip>
  A good operating rhythm: enable `capture_eval_cases` in production, curate a handful of `win` cases for your critical flows, and run `pandaprobe-harness-eval` nightly. The journal's `regression` events become an audit trail of every rule set the corpus was checked against.
</Tip>