Cost, reliability & scale

Controlling evaluation cost

The three-tier ladder is itself the primary cost control: the expensive metrics only run when the cheap ones say something is wrong.

Tier	When it runs	Typical share of turns
1 — `task_completion`, `coherence`	Every turn, one batched run over all new traces	100%
2 — `tool_correctness`, `argument_correctness`	Only when the trajectory gate opens, last trace only	Small
3 — planning metrics	Only when Tier 2 confirms, and only with `enable_tier3=true`	Off by default

Tier 1 is deliberately cheap — coherence is embedding-based rather than an LLM judge — so the always-on rung is the least expensive one. Tier 3 is opt-in because every LLM-judge metric reads the whole trace, so cost scales with metric count.

Controlling repair cost

The repair agent is the loop’s second model consumer, and it only runs when a notice was posted — so the tier ladder gates it too. It is bounded on four axes:

Bound	Default	Effect
`repair_max_turns`	`6`	Model rounds per episode.
`repair_max_tokens`	`4096`	Output tokens across the episode.
`repair_timeout_s`	`60`	Wall clock per episode.
Episode coalescing	—	Related notices from one turn become one episode, so three findings cost one diagnosis rather than three.

RepairResult.usage reports normalized input/output tokens and cost whenever the provider supplies them, so repair spend is measurable per episode rather than inferred. Repair also makes at most one episode per settled turn.

Point repair_model at a cheaper model than your task model if diagnosis quality allows. It reads bounded evidence and writes one short rule — a smaller model is often sufficient, and the setting is independent of whatever your agent runs on.

Validation is the third consumer, and the least predictable: each replay re-runs your agent. validation_round_budget_s bounds replay work per round, and regression_sample bounds a regression run. Four independent dials bound the spend on top of that:

Control	Effect
`enable_tier3`	Leave it `false` unless you need the planning diagnosis.
`session_min_eval_interval_s`	Per-session rate limit between eval launches.
`max_concurrent_evals`	Global concurrency cap across all sessions.
`max_evals_per_run`	Hard per-process budget; once exhausted, further evaluations are skipped (loudly: one warning + a journal `skip` event).

Two structural savers:

Supersede — a newer turn cancels the session’s in-flight evaluation; you never pay for a stale turn’s scores.
One batched run per turn — Tier 1 covers every new trace in a single create + poll, not one run per trace.

eval_sample_every > 1 bounds cost by skipping turns, which thins the series the trajectory gate needs. It is a legitimate dial, but it trades away detection sensitivity in a way the others don’t. Reach for enable_tier3=false and max_evals_per_run first.

Replay costs are yours (a replay re-runs your agent), which is why validation replays at most a handful of cases per candidate, regression runs are sequential and sampleable (regression_sample), and every replay invocation is time-bounded (replay_timeout_s).

Latency and the barrier

The barrier is the one place the harness deliberately makes your loop wait. It buys in-session healing; the cost is per-turn latency bounded by barrier_timeout_s (default 180s).

Setting	Behavior
`harness.turn(session_id)`	Fire-and-forget. Zero added latency; the agent may run several turns ahead of its own diagnosis.
`harness.turn(session_id, settle=True)`	Waits for the turn’s evaluation, notice persistence, and one bounded repair attempt. A rule learned now is a provisional candidate, readable next turn. It does not wait for a validation round.
Barrier expiry	`timed_out=True`, the work continues detached. A slow platform costs latency, never correctness.

If per-turn latency is unacceptable in your product but you still want in-session healing, settle on a subset of turns — every Nth turn, or only after a tool-heavy turn. The gate still gets its series from Tier 1 on every turn; the barrier only controls when the agent finds out.

Degradation ladder

The harness’s core invariant: nothing on the harness’s side may break or stall your agent. Every failure mode has a defined, observable degradation:

Failure	Behavior
CLI missing / unauthenticated	The memoized health check fails once → one warning, a journal `health` event, all evaluations skipped. The agent keeps running un-harnessed.
Transient platform errors, trace-ingestion lag	Bounded retries with backoff; then the turn’s scores resolve as pending (never a false breach — a pending score can’t alert).
No new traces this turn	Tier 1 is skipped with a log line. An empty trace listing is only retried while the session has never yet produced a trace, so a healthy quiet turn doesn’t burn the backoff budget.
Persistent CLI errors mid-run	The evaluation degrades to pending scores; the error is logged, the loop continues.
Barrier budget expires	`SettleResult.timed_out`; work continues detached.
Verifier raises	Caught and logged; the turn proceeds with no `outcome_correct` score. A broken oracle cannot break a turn.
Repair model errors, timeout, or a malformed tool call	`SettleResult.repair` carries the status and `error_category`. Nothing is acknowledged, the notice stays pending, and a later turn can retry it. The developer task never fails.
Repair provider unreachable	The episode ends `failed`; evaluation, notices, and validation continue unaffected.
Replay hangs or raises	Bounded by `replay_timeout_s`; the case is inconclusive (validation) or skipped (regression) with the reason recorded.
Replay never reaches its environment	Bounded by `replay_env_wait_timeout_s`; recorded as `env_wait_timeout` rather than as evidence against the candidate.
Validation engine failure	Caught and logged inside its detached task; notices still post, turns still evaluate.
Notice storm	The circuit breaker escalates to one `needs_human` and suppresses further posting until the window drains.
Corrupt workspace files	All JSON parsing is forgiving — unreadable records are skipped, never fatal.

Concurrency model

on_turn_end is synchronous and cheap: parse, gate, schedule — it returns before any I/O.
Evaluations run as detached tasks under a global semaphore; candidate validation is a single-flight background round, bounded by validation_round_budget_s with forward-trial fallback for candidates replay cannot reach; blocking file I/O runs on the thread pool.
The trajectory gate’s fold is history-dependent, so a whole turn’s traces are folded in order — in one thread hop and one history write per trace, not one per metric.
All workspace stores are lock-guarded with atomic writes (unique temp file + rename) and append-only logs — one workspace safely serves many concurrent sessions, and readers never observe a half-written file.
Per-session bookkeeping is bounded (a few thousand sessions) with oldest-first eviction, so long-lived processes don’t grow without limit. Evicting a session also drops its trace seen-set.

refresh(session_id), refresh_all(), and drain_validation(timeout=...) are bounded joins (drain_timeout_s by default; drain_validation returns whether it finished, and settle_validation(timeout=...) is the phase-boundary call) for tests and explicit callers — correctness never depends on calling them. settle() is the one wait that is part of the design; hook.pending_sessions is the read-only view for host-side phase barriers.

Scaling out

Score history is local by default, which means a session handled by two replicas has two partial series. The history source is a small Protocol, so a shared remote store can replace the local JSON file without touching anything else — that is the supported fix for replica fan-out.

The gate’s state (running peak, stall counter) lives in the same state/score_history.json entry as the score series itself — no second file, no second lock — so it inherits whatever durability that store has.

For the workspace itself, give each replica its own HARNESS_ROOT or mount a shared volume: all stores are multi-session safe within a process, and cross-process safety rests on atomic renames and append-only files.

Operating recommendations

Settle per turn, not per task. This is the one integration mistake that silently disables detection — the gate needs a series. See the barrier.
Start in shadow mode (observe_only=true) and calibrate your Tier-2 thresholds and gate_target against real traffic.
Tune gate_window to your task horizon — long-horizon agents need a wider window than short interactive turns.
Wire an outcome verifier if you have any ground truth at all. It is the strongest promotion signal available, and it costs nothing per turn.
Turn on capture (capture_eval_cases=true) so the closed loop has scenarios to replay, and curate a few protected win cases.
Wire a replay function — it upgrades rule validation from statistical to counterfactual and unlocks regression runs.
Schedule pandaprobe-harness-eval (nightly, or after prompt changes) and alert on a non-zero exit.
Call settle_validation() at phase boundaries — before archiving a workspace, snapshotting a ruleset, or reporting results. Waiting on evaluations alone can freeze a candidate that had already earned promotion as permanently provisional.
Watch the journal — rule_promote / rule_retire tell you what is being learned; validation_verdict’s pending_reason tells you why something is not decided yet; needs_human notices tell you when to step in.

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

Cost, reliability & scale

Controlling evaluation cost

Controlling repair cost

Latency and the barrier

Degradation ladder

Concurrency model

Scaling out

Operating recommendations

​Controlling evaluation cost

​Controlling repair cost

​Latency and the barrier

​Degradation ladder

​Concurrency model

​Scaling out

​Operating recommendations

Controlling evaluation cost

Controlling repair cost

Latency and the barrier

Degradation ladder

Concurrency model

Scaling out

Operating recommendations