Outcome verifier

The harness’s metrics are judged proxies for success. If you already know what success means — a golden dataset, a benchmark grader, a business rule, a database assertion — you can hand the harness that oracle directly, and it becomes the signal that decides which rules get trusted.

A verifier is the strongest evidence the harness can use. Where a judged metric estimates whether the agent did well, a verifier knows. It is optional, and worth wiring whenever it is possible at all.

The contract

from pandaprobe_harness import Harness

def grade(session_id: str, end_state: dict) -> float | None:
    """Score this turn's outcome in [0, 1] — or None if it can't be judged yet."""
    task_id = end_state.get("task_id")
    if task_id is None:
        return None
    return my_benchmark.score(task_id)      # e.g. 0.0 … 1.0

harness = Harness.create(verifier=grade)

Signature: (session_id, end_state) -> float | bool | None. The end_state is whatever your loop passed in the turn payload, so put the task identity in there.
May be sync or async — the harness adapts either.
Returns: a float in [0, 1] (higher is better), a bool, or None.
None is a normal answer, not an error: a grader that can only score a finished task returns None on every intermediate turn. The harness treats it as “no usable verdict” and carries on.
A broken verifier cannot break your loop. Exceptions are caught and logged; the turn proceeds without an outcome score. Values are clamped into range.

What it produces

A verdict becomes a synthetic outcome_correct metric on the turn’s report, at tier 0 — which means it breaches on its absolute value (outcome_threshold, default 0.9), unlike the Tier-1 trajectory metrics. It runs in parallel with the tier ladder rather than after it, because it depends only on the turn payload. On the barrier path that matters: its latency would otherwise stack on top of the evaluation’s.

Why it changes promotion

When a candidate rule is validated by replay, the harness has to pick which metric’s delta decides the verdict. It chooses the most authoritative metric that the replay actually produced, in trust order:

outcome_correct — your verifier’s verdict on the replayed run.
The rule’s own declared metric.
The metric named in the triggering signature.

So with a verifier wired, a rule is promoted because the task outcome improved, not because a judge’s proxy moved. Without one, the harness falls back down the list.

A wired verifier that has no verdict for a particular case does not veto promotion. The target is chosen per case from the deltas that actually arrived — so a grader that can only score some tasks helps on those and stays out of the way on the rest. Treating an absent verdict as a failed target would veto promotion for every ungradeable case, which is why the fallback is per case rather than global.

Prefer a continuous score

A pass/fail flag that is almost always 0 is not a usable signal. If your grader can emit partial credit — fraction of subgoals met, fraction of assertions passed, normalized reward — emit that. Binary success on hard tasks is almost never positive, so it separates nothing: the same non-discriminating-signal trap the trajectory gate exists to avoid.

# Weak: on a benchmark where the agent passes 15% of tasks, this is 0 nearly always.
def grade(session_id, end_state):
    return my_benchmark.passed(end_state["task_id"])

# Strong: partial credit discriminates between "close" and "hopeless".
def grade(session_id, end_state):
    result = my_benchmark.evaluate(end_state["task_id"])
    return result.subgoals_met / result.subgoals_total

Passing the task identity through

The verifier only sees what your loop puts in end_state, so include whatever your grader keys on:

harness.on_turn_end({
    "session_id": session_id,
    "turn_index": turn_index,
    "end_state": {"task_id": task_id, "benchmark": "appworld"},
})

The same end_state is what a captured eval case stores as its replay input, so one payload serves both the verifier and replay.

The facade’s bare harness.turn(session_id) scope sends an empty end_state. If you want a verifier, call on_turn_end with a payload — or have your framework adapter supply one.

Limits

It does not replace the tiers. outcome_correct tells you whether the turn succeeded; Tier 2 tells you which step went wrong, which is what a useful rule is written about. Both feed the same report.
It is not a gate on your agent. Like everything else on the producing side, it is observational: it shapes notices and promotion decisions, never the agent’s control flow.

Configuration

Knob	Default	Meaning
`outcome_threshold`	`0.9`	The absolute floor below which `outcome_correct` breaches. High on purpose — a verifier’s verdict is trusted.

Running an A/B study or a benchmark? A verifier is the difference between measuring whether the harness helped and guessing. Wire the benchmark’s own grader, and prefer its continuous reward over its pass/fail flag.

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

The contract

What it produces

Why it changes promotion

Prefer a continuous score

Passing the task identity through

Limits

Configuration

​The contract

​What it produces

​Why it changes promotion

​Prefer a continuous score

​Passing the task identity through

​Limits

​Configuration

The contract

What it produces

Why it changes promotion

Prefer a continuous score

Passing the task identity through

Limits

Configuration