> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pandaprobe.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cost, reliability & scale

> Controlling evaluation spend, degrading gracefully, and running many sessions

## Controlling evaluation cost

Every evaluated turn is one platform eval run. Four independent dials bound the spend:

| Control                       | Effect                                                                                                                   |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `eval_sample_every`           | Evaluate every Nth turn per session (`1` = every turn).                                                                  |
| `session_min_eval_interval_s` | Per-session rate limit between eval launches.                                                                            |
| `max_concurrent_evals`        | Global concurrency cap across all sessions.                                                                              |
| `max_evals_per_run`           | Hard per-process budget; once exhausted, further evaluations are skipped (loudly: one warning + a journal `skip` event). |

Two structural savers on top:

* **Supersede** — a newer turn cancels the session's in-flight evaluation; you never pay for a stale turn's scores.
* **One batched run per turn** — both metrics ride a single eval run, not one each.

Replay costs are yours (a replay re-runs *your agent*), which is why validation replays at most a handful of cases per candidate, regression runs are sequential and sampleable (`regression_sample`), and every replay invocation is time-bounded (`replay_timeout_s`).

## Degradation ladder

The harness's core invariant: **nothing on the harness's side may break or stall your agent.** Every failure mode has a defined, observable degradation:

| Failure                                        | Behavior                                                                                                                                     |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| CLI missing / unauthenticated                  | The memoized health check fails once → one warning, a journal `health` event, all evaluations skipped. The agent keeps running un-harnessed. |
| Transient platform errors, trace-ingestion lag | Bounded retries with backoff; then the turn's scores resolve as *pending* (never a false breach — a pending score can't alert).              |
| Persistent CLI errors mid-run                  | The evaluation degrades to pending scores; the error is logged, the loop continues.                                                          |
| Replay hangs or raises                         | Bounded by `replay_timeout_s`; the case is *inconclusive* (validation) or *skipped* (regression) with the reason recorded.                   |
| Validation engine failure                      | Caught and logged inside its detached task; notices still post, turns still evaluate.                                                        |
| Notice storm                                   | The circuit breaker escalates to one `needs_human` and suppresses further posting until the window drains.                                   |
| Corrupt workspace files                        | All JSON parsing is forgiving — unreadable records are skipped, never fatal.                                                                 |

## Concurrency model

* `on_turn_end` is synchronous and cheap: parse, gate, schedule — it returns before any I/O.
* Evaluations run as detached tasks under a global semaphore; candidate validation is a single-flight background round; blocking file I/O runs on the thread pool.
* All workspace stores are lock-guarded with atomic writes (unique temp file + rename) and append-only logs — one workspace safely serves many concurrent sessions, and readers never observe a half-written file.
* Per-session bookkeeping is bounded (a few thousand sessions), with oldest-first eviction, so long-lived processes don't grow without limit.

`refresh(session_id)`, `refresh_all()`, and `drain_validation()` are bounded joins (`drain_timeout_s`) for tests and explicit callers — correctness never depends on calling them.

## Scaling out

Horizontally-scaled replicas share trend state through the platform: with `hydrate_history_from_backend=true`, the hook seeds the local EWMA history once per session from `pandaprobe evals scores list`, so baselines survive process restarts and replica fan-out. The history source is a small Protocol — a shared remote store can replace the local JSON file without touching anything else.

For the workspace itself, give each replica its own `HARNESS_ROOT` or mount a shared volume: all stores are multi-session safe within a process, and cross-process safety rests on atomic renames and append-only files.

## Operating recommendations

1. **Start in shadow mode** (`observe_only=true`) and [calibrate](/harness/closed-loop/calibration) your thresholds against labeled traffic.
2. **Turn on capture** (`capture_eval_cases=true`) so the closed loop has scenarios to replay, and curate a few protected `win` cases.
3. **Wire a replay function** — it upgrades rule validation from statistical to counterfactual and unlocks [regression runs](/harness/closed-loop/regression-runs).
4. **Schedule `pandaprobe-harness-eval`** (nightly, or after prompt changes) and alert on a non-zero exit.
5. **Watch the journal** — `rule_promote` / `rule_retire` events tell you what your agent is learning; `needs_human` notices tell you when to step in.