Calibration

Tier-2 breaches, outcome_correct breaches, and regression classification all key off one predicate: score below threshold. The default (0.5) is a starting point, not a measurement. pandaprobe-harness-calibrate is the out-of-band diagnostic that tells you whether a breach actually corresponds to a real failure on your traffic, and what threshold would serve you better.

What to calibrate. Calibration reports on whatever metrics your workspace has scores for — task_completion, coherence, tool_correctness, and friends. Aim it at the metrics whose threshold actually decides something — the Tier 2 metrics and outcome_correct. Tier-1 metrics never breach on their absolute value (their verdict is the trajectory gate), so their threshold is only an input to gate_target; their score distribution is still worth reading, because that is how you pick a sensible gate_target and gate_drop.

Calibration is an operator tool, not a runtime gate — it never enters the healing loop. It reads scores, computes statistics, and prints a report; acting on it (setting a metric’s floor in the thresholds map, say) is your call.

With ground-truth labels

If you can label sessions as failed/ok — even a few dozen — you get the full picture:

pandaprobe-harness-calibrate --labels labels.json

Accepted label formats:

JSON dict
JSON list
CSV
Eval-set proxy

{ "s-001": true, "s-002": false, "s-003": true }

[
  { "session_id": "s-001", "failed": true },
  { "session_id": "s-002", "failed": false }
]

session_id,failed
s-001,true
s-002,0

pandaprobe-harness-calibrate --from-evalset

Zero labeling effort: captured failure cases count as failed, win cases as ok.

Per metric, the report gives you:

Precision / recall / F1 of the breach predicate at your configured threshold, with the confusion matrix.
A threshold sweep (0.05 … 0.95) with the confusion at every point.
Two recommendations: the F1-maximizing threshold, and the lowest threshold reaching a target precision (--target-precision, default 0.9) with non-zero recall.

tool_correctness  (configured threshold 0.50, 12 scores)
  distribution: min 0.20  median 0.62  mean 0.59  max 0.90  stdev 0.23
  histogram [0.0..1.0): 0 0 1 2 1 1 2 2 2 1
  at threshold 0.50: precision 0.80  recall 1.00  F1 0.89  (tp 4 fp 1 fn 0 tn 7 over 12 labeled)
  best F1 1.00 at threshold 0.50
  precision ≥ 0.90 first at threshold 0.35

Without labels

No labels yet? The unsupervised report is still enough to pick a threshold sanely: the score distribution (min/median/mean/max/stdev + a 10-bucket histogram), the breach count at every candidate threshold, and inter-metric agreement — the fraction of fully-scored sessions where two metrics agree on breach vs. no-breach (low agreement usually means one threshold is mis-set with respect to the other).

Where the scores come from

Two local sources, merged with precedence and each degrading independently:

The local history store — state/score_history.json, the per-(session, metric) series the trajectory gate maintains (highest precedence).
The eval set — captured cases’ baseline_scores.

Calibration therefore works fully offline against a workspace that has seen traffic. Trace score-list records from the platform are not used because they expose a trace_id, not the session_id needed to join session labels. If neither local source yields anything, the CLI exits 1 and names both paths.

Library use

The same machinery is importable for notebooks and pipelines:

from pandaprobe_harness import HarnessConfig, calibrate

report = calibrate(
    scores,                      # {session_id: {metric: value}}
    config=HarnessConfig.from_env(),
    labels=labels,               # {session_id: failed} or None
    target_precision=0.9,
)
print(report.render_text())      # or report.to_json()

pandaprobe-harness-calibrate --json emits the identical structure from the command line. A self-contained walkthrough lives at examples/misc/calibration_demo.py in the repository.

Pair calibration with shadow mode: run with observe_only=true for a week, label a sample of sessions, calibrate, set your Tier-2 thresholds and gate_target — then let the agent act.

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

With ground-truth labels

Without labels

Where the scores come from

Library use

​With ground-truth labels

​Without labels

​Where the scores come from

​Library use

With ground-truth labels

Without labels

Where the scores come from

Library use