Eval set & replay

The eval set is the harness’s corpus of scenarios worth re-running: failures to fix and wins to protect. It is the shared substrate of the closed loop — rule validation replays matching failures to vet candidates, and regression runs replay everything to catch a new rule breaking an old win.

Eval cases

One JSON file per case under <HARNESS_ROOT>/evalset/:

Field	Purpose
`id`, `created_at`, `session_id`	Identity and which session the scenario came from.
`kind`	`failure` (something to fix) or `win` (something to protect).
`signature`	The condition labels copied from the spawning notice — how validation finds cases matching a candidate rule.
`baseline_scores`	`{metric: value}` at capture time — the comparison point for every replay.
`replay_input`	The payload your replay function needs to re-run the scenario. `null` = not replayable (still useful as a calibration label).
`notes`	The notice summary, sanitized.

Capturing failures

Turn on capture and every breach-severity notice records the session as a failure case:

from pandaprobe_harness import Harness, HarnessConfig

harness = Harness.create(HarnessConfig.from_env(capture_eval_cases=True))

What qualifies. breach severity means Tier 2 confirmed a below-threshold step — a surgical, diagnosed failure. A Tier-1-only trajectory fire is an advisory trend notice and is deliberately not captured: a stall the step-level metrics can’t corroborate is worth telling the agent about, but not worth training a rule on. needs_human doesn’t qualify either — it is a rate alarm, not a failure.

Capture is opt-in (capture_eval_cases, default false) because cases store session-derived data — the signature, scores, and whatever your turn payloads carry — under the workspace.

Where does replay_input come from? From the turn payload’s end_state when your loop or adapter provides one. The facade’s bare harness.turn(...) scope sends an empty payload, so in that setup attach inputs explicitly:

harness.evalset.attach_input(case_id, {"task": "charge $42 for order 1017"})

There is deliberately no agent-facing eval-set tool: curating the corpus is an operator concern, and the agent’s turns are better spent on the task. Attach from code, or supply an end_state in the turn payload and it becomes the replay input automatically.

Protecting wins

Capture known-good scenarios as win cases — these are what regression runs guard:

harness.evalset.capture(
    session_id="s-good-1",
    kind="win",
    signature=("healthy",),
    baseline_scores={"task_completion": 0.92, "tool_correctness": 0.95},
    replay_input={"task": "verified payment flow"},
)

Corpus management is deliberately conservative: cases dedup per (session, signature, kind); at eval_case_max (default 200) the oldest failures evict first; win cases are never auto-evicted — if the corpus is all wins, capture refuses loudly rather than dropping one.

The replay function

The platform is passive — it scores traces that already exist. To learn what would happen under a new rule set, the harness must re-run your agent, and only you know how to do that:

from uuid import uuid4
import pandaprobe
from pandaprobe_harness import EvalCase, Harness, ReplayContext

async def replay(case: EvalCase, context: ReplayContext) -> str:
    """Re-run my agent on the captured input; return the NEW session id."""
    session_id = f"replay-{case.id}-{uuid4().hex[:6]}"
    with pandaprobe.session(session_id):                 # traces land under the new session
        await my_agent_step(
            str(context) + MY_PROMPT,                    # capability preamble, no rule text
            tools=list(context.task_tools.specs()),      # how the candidate is discovered
            task=case.replay_input,
        )
    return session_id

harness = Harness.create(replay=replay)

The contract, precisely:

Input: the EvalCase and a ReplayContext. The context is a str subclass — so an older callback that concatenates it still works — but the string is a capability-only preamble with no rule text. Rules are reached through context.task_tools, and during candidate validation only the candidate under test is discoverable there.
The candidate must be read. Attach context.task_tools and let the replayed agent use them. A replay that never reads the candidate produces no conclusive verdict, because whatever moved, the rule under test did not move it.
Output: a new session id whose traces the harness can score — it re-scores the run’s last trace on the Tier 1 + Tier 2 metrics. Never reuse the original session.
Behavior: each invocation is awaited sequentially and bounded by replay_timeout_s; exceptions and timeouts degrade to “inconclusive” — they never crash validation or a regression run.
Shared environments: if your replay must first acquire a single environment server, container, or device, call context.mark_execution_started() once you have it. Queueing is then charged to replay_env_wait_timeout_s and the execution budget starts fresh — otherwise a replay can “time out” having run nothing, and be recorded as inconclusive evidence about a rule that never executed.

Be honest with yourself about this dependency. Without a replay function, ReplayValidator and regression runs cannot execute: candidate validation falls back to forward trials (slower, statistical — announced once in the log and journal), and run_regression reports every case as skipped with one clear warning. The harness never silently pretends it replayed something.

Inspecting the corpus

harness.evalset.cases()                              # all, oldest first
harness.evalset.cases(kind="win")
harness.evalset.matching(("breach:tool_correctness",))    # newest matching failures

Or from the operator side: pandaprobe-harness-eval --list.

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

Eval cases

Capturing failures

Protecting wins

The replay function

Inspecting the corpus

​Eval cases

​Capturing failures

​Protecting wins

​The replay function

​Inspecting the corpus

Eval cases

Capturing failures

Protecting wins

The replay function

Inspecting the corpus