Cost, reliability & scale

Controlling evaluation cost

Every evaluated turn is one platform eval run. Four independent dials bound the spend:

Control	Effect
`eval_sample_every`	Evaluate every Nth turn per session (`1` = every turn).
`session_min_eval_interval_s`	Per-session rate limit between eval launches.
`max_concurrent_evals`	Global concurrency cap across all sessions.
`max_evals_per_run`	Hard per-process budget; once exhausted, further evaluations are skipped (loudly: one warning + a journal `skip` event).

Two structural savers on top:

Supersede — a newer turn cancels the session’s in-flight evaluation; you never pay for a stale turn’s scores.
One batched run per turn — both metrics ride a single eval run, not one each.

Replay costs are yours (a replay re-runs your agent), which is why validation replays at most a handful of cases per candidate, regression runs are sequential and sampleable (regression_sample), and every replay invocation is time-bounded (replay_timeout_s).

Degradation ladder

The harness’s core invariant: nothing on the harness’s side may break or stall your agent. Every failure mode has a defined, observable degradation:

Failure	Behavior
CLI missing / unauthenticated	The memoized health check fails once → one warning, a journal `health` event, all evaluations skipped. The agent keeps running un-harnessed.
Transient platform errors, trace-ingestion lag	Bounded retries with backoff; then the turn’s scores resolve as pending (never a false breach — a pending score can’t alert).
Persistent CLI errors mid-run	The evaluation degrades to pending scores; the error is logged, the loop continues.
Replay hangs or raises	Bounded by `replay_timeout_s`; the case is inconclusive (validation) or skipped (regression) with the reason recorded.
Validation engine failure	Caught and logged inside its detached task; notices still post, turns still evaluate.
Notice storm	The circuit breaker escalates to one `needs_human` and suppresses further posting until the window drains.
Corrupt workspace files	All JSON parsing is forgiving — unreadable records are skipped, never fatal.

Concurrency model

on_turn_end is synchronous and cheap: parse, gate, schedule — it returns before any I/O.
Evaluations run as detached tasks under a global semaphore; candidate validation is a single-flight background round; blocking file I/O runs on the thread pool.
All workspace stores are lock-guarded with atomic writes (unique temp file + rename) and append-only logs — one workspace safely serves many concurrent sessions, and readers never observe a half-written file.
Per-session bookkeeping is bounded (a few thousand sessions), with oldest-first eviction, so long-lived processes don’t grow without limit.

refresh(session_id), refresh_all(), and drain_validation() are bounded joins (drain_timeout_s) for tests and explicit callers — correctness never depends on calling them.

Scaling out

Horizontally-scaled replicas share trend state through the platform: with hydrate_history_from_backend=true, the hook seeds the local EWMA history once per session from pandaprobe evals scores list, so baselines survive process restarts and replica fan-out. The history source is a small Protocol — a shared remote store can replace the local JSON file without touching anything else. For the workspace itself, give each replica its own HARNESS_ROOT or mount a shared volume: all stores are multi-session safe within a process, and cross-process safety rests on atomic renames and append-only files.

Operating recommendations

Start in shadow mode (observe_only=true) and calibrate your thresholds against labeled traffic.
Turn on capture (capture_eval_cases=true) so the closed loop has scenarios to replay, and curate a few protected win cases.
Wire a replay function — it upgrades rule validation from statistical to counterfactual and unlocks regression runs.
Schedule pandaprobe-harness-eval (nightly, or after prompt changes) and alert on a non-zero exit.
Watch the journal — rule_promote / rule_retire events tell you what your agent is learning; needs_human notices tell you when to step in.

​Controlling evaluation cost

​Degradation ladder

​Concurrency model

​Scaling out

​Operating recommendations

Controlling evaluation cost

Degradation ladder

Concurrency model

Scaling out

Operating recommendations