> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pandaprobe.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Outcome verifier

> Ground truth instead of a proxy: wire your own grader and let it decide promotion

The harness's metrics are judged proxies for success. If you already **know** what success means — a golden dataset, a benchmark grader, a business rule, a database assertion — you can hand the harness that oracle directly, and it becomes the signal that decides which rules get trusted.

<Note>
  A verifier is the strongest evidence the harness can use. Where a judged metric *estimates* whether the agent did well, a verifier knows. It is optional, and worth wiring whenever it is possible at all.
</Note>

## The contract

```python theme={null}
from pandaprobe_harness import Harness

def grade(session_id: str, end_state: dict) -> float | None:
    """Score this turn's outcome in [0, 1] — or None if it can't be judged yet."""
    task_id = end_state.get("task_id")
    if task_id is None:
        return None
    return my_benchmark.score(task_id)      # e.g. 0.0 … 1.0

harness = Harness.create(verifier=grade)
```

* **Signature:** `(session_id, end_state) -> float | bool | None`. The `end_state` is whatever your loop passed in the turn payload, so put the task identity in there.
* **May be sync or async** — the harness adapts either.
* **Returns:** a float in `[0, 1]` (higher is better), a bool, or `None`.
* **`None` is a normal answer**, not an error: a grader that can only score a *finished* task returns `None` on every intermediate turn. The harness treats it as "no usable verdict" and carries on.
* **A broken verifier cannot break your loop.** Exceptions are caught and logged; the turn proceeds without an outcome score. Values are clamped into range.

## What it produces

A verdict becomes a synthetic **`outcome_correct`** metric on the turn's report, at **tier 0** — which means it breaches on its absolute value (`outcome_threshold`, default `0.9`), unlike the Tier-1 trajectory metrics.

It runs *in parallel* with the tier ladder rather than after it, because it depends only on the turn payload. On the [barrier](/harness/loop/evaluation#the-per-turn-barrier) path that matters: its latency would otherwise stack on top of the evaluation's.

## Why it changes promotion

When a candidate rule is validated by replay, the harness has to pick **which metric's delta decides the verdict**. It chooses the most authoritative metric that the replay actually produced, in trust order:

1. **`outcome_correct`** — your verifier's verdict on the replayed run.
2. The rule's own declared `metric`.
3. The metric named in the triggering signature.

So with a verifier wired, a rule is promoted because the *task outcome* improved, not because a judge's proxy moved. Without one, the harness falls back down the list.

<Note>
  A wired verifier that has **no verdict** for a particular case does not veto promotion. The target is chosen per case from the deltas that actually arrived — so a grader that can only score some tasks helps on those and stays out of the way on the rest. Treating an absent verdict as a failed target would veto promotion for every ungradeable case, which is why the fallback is per case rather than global.
</Note>

## Prefer a continuous score

<Warning>
  **A pass/fail flag that is almost always `0` is not a usable signal.** If your grader can emit partial credit — fraction of subgoals met, fraction of assertions passed, normalized reward — emit that. Binary success on hard tasks is almost never positive, so it separates nothing: the same non-discriminating-signal trap the [trajectory gate](/harness/loop/evaluation#the-trajectory-gate) exists to avoid.
</Warning>

```python theme={null}
# Weak: on a benchmark where the agent passes 15% of tasks, this is 0 nearly always.
def grade(session_id, end_state):
    return my_benchmark.passed(end_state["task_id"])

# Strong: partial credit discriminates between "close" and "hopeless".
def grade(session_id, end_state):
    result = my_benchmark.evaluate(end_state["task_id"])
    return result.subgoals_met / result.subgoals_total
```

## Passing the task identity through

The verifier only sees what your loop puts in `end_state`, so include whatever your grader keys on:

```python theme={null}
harness.on_turn_end({
    "session_id": session_id,
    "turn_index": turn_index,
    "end_state": {"task_id": task_id, "benchmark": "appworld"},
})
```

The same `end_state` is what a captured [eval case](/harness/closed-loop/eval-set-and-replay) stores as its replay input, so one payload serves both the verifier and replay.

<Note>
  The facade's bare `harness.turn(session_id)` scope sends an empty `end_state`. If you want a verifier, call `on_turn_end` with a payload — or have your framework adapter supply one.
</Note>

## Limits

* **It does not replace the tiers.** `outcome_correct` tells you *whether* the turn succeeded; Tier 2 tells you *which step* went wrong, which is what a useful rule is written about. Both feed the same report.
* **It is not a gate on your agent.** Like everything else on the producing side, it is observational: it shapes notices and promotion decisions, never the agent's control flow.

## Configuration

| Knob                | Default | Meaning                                                                                                       |
| ------------------- | ------- | ------------------------------------------------------------------------------------------------------------- |
| `outcome_threshold` | `0.9`   | The absolute floor below which `outcome_correct` breaches. High on purpose — a verifier's verdict is trusted. |

<Tip>
  Running an A/B study or a benchmark? A verifier is the difference between measuring whether the harness helped and *guessing*. Wire the benchmark's own grader, and prefer its continuous reward over its pass/fail flag.
</Tip>