Controlling evaluation cost
Every evaluated turn is one platform eval run. Four independent dials bound the spend:| Control | Effect |
|---|---|
eval_sample_every | Evaluate every Nth turn per session (1 = every turn). |
session_min_eval_interval_s | Per-session rate limit between eval launches. |
max_concurrent_evals | Global concurrency cap across all sessions. |
max_evals_per_run | Hard per-process budget; once exhausted, further evaluations are skipped (loudly: one warning + a journal skip event). |
- Supersede — a newer turn cancels the session’s in-flight evaluation; you never pay for a stale turn’s scores.
- One batched run per turn — both metrics ride a single eval run, not one each.
regression_sample), and every replay invocation is time-bounded (replay_timeout_s).
Degradation ladder
The harness’s core invariant: nothing on the harness’s side may break or stall your agent. Every failure mode has a defined, observable degradation:| Failure | Behavior |
|---|---|
| CLI missing / unauthenticated | The memoized health check fails once → one warning, a journal health event, all evaluations skipped. The agent keeps running un-harnessed. |
| Transient platform errors, trace-ingestion lag | Bounded retries with backoff; then the turn’s scores resolve as pending (never a false breach — a pending score can’t alert). |
| Persistent CLI errors mid-run | The evaluation degrades to pending scores; the error is logged, the loop continues. |
| Replay hangs or raises | Bounded by replay_timeout_s; the case is inconclusive (validation) or skipped (regression) with the reason recorded. |
| Validation engine failure | Caught and logged inside its detached task; notices still post, turns still evaluate. |
| Notice storm | The circuit breaker escalates to one needs_human and suppresses further posting until the window drains. |
| Corrupt workspace files | All JSON parsing is forgiving — unreadable records are skipped, never fatal. |
Concurrency model
on_turn_endis synchronous and cheap: parse, gate, schedule — it returns before any I/O.- Evaluations run as detached tasks under a global semaphore; candidate validation is a single-flight background round; blocking file I/O runs on the thread pool.
- All workspace stores are lock-guarded with atomic writes (unique temp file + rename) and append-only logs — one workspace safely serves many concurrent sessions, and readers never observe a half-written file.
- Per-session bookkeeping is bounded (a few thousand sessions), with oldest-first eviction, so long-lived processes don’t grow without limit.
refresh(session_id), refresh_all(), and drain_validation() are bounded joins (drain_timeout_s) for tests and explicit callers — correctness never depends on calling them.
Scaling out
Horizontally-scaled replicas share trend state through the platform: withhydrate_history_from_backend=true, the hook seeds the local EWMA history once per session from pandaprobe evals scores list, so baselines survive process restarts and replica fan-out. The history source is a small Protocol — a shared remote store can replace the local JSON file without touching anything else.
For the workspace itself, give each replica its own HARNESS_ROOT or mount a shared volume: all stores are multi-session safe within a process, and cross-process safety rests on atomic renames and append-only files.
Operating recommendations
- Start in shadow mode (
observe_only=true) and calibrate your thresholds against labeled traffic. - Turn on capture (
capture_eval_cases=true) so the closed loop has scenarios to replay, and curate a few protectedwincases. - Wire a replay function — it upgrades rule validation from statistical to counterfactual and unlocks regression runs.
- Schedule
pandaprobe-harness-eval(nightly, or after prompt changes) and alert on a non-zero exit. - Watch the journal —
rule_promote/rule_retireevents tell you what your agent is learning;needs_humannotices tell you when to step in.