Configuration

All configuration lives on one frozen HarnessConfig object. Build it from the environment (most knobs have a HARNESS_* variable) with explicit overrides winning:

from pandaprobe_harness import HarnessConfig

config = HarnessConfig.from_env(capture_eval_cases=True, gate_window=3)
harness = Harness.create(config)

repair_model is required. Harness.create() raises ValueError unless you set it (or HARNESS_REPAIR_MODEL), because the repair agent makes its own model calls and no billable default is ever selected for you. The one exception is observe_only=True, which evaluates and journals without mutating anything.

Not every field has an environment variable. Set the rest programmatically: thresholds (a per-metric threshold map), the per-tier metric tuples, and the fine-grained gate knobs. Derived workspace paths (rules.md, rules/, rules.jsonl, scope_metadata.json, mailbox/, journal.jsonl, …) are computed from HARNESS_ROOT and cannot be overridden individually.

Workspace & CLI

Variable	Default	Meaning
`HARNESS_ROOT`	`/harness`	Workspace root (mailbox, journal, rules subtree, traces, eval set, state).
`HARNESS_CLI_BINARY`	`pandaprobe`	The CLI binary — every platform call goes through it.
`HARNESS_CLI_TIMEOUT_S`	`30`	Per-invocation subprocess timeout.

The repair agent

The repair agent is package-owned: PandaProbe supplies the prompt, the capability set, and the episode lifecycle. You supply the model and its bounds.

Variable	Default	Meaning
`HARNESS_REPAIR_MODEL`	(none)	Required unless `observe_only`. A LiteLLM identifier — `openai/…`, `anthropic/…`, `bedrock/anthropic.…`, `vertex_ai/…`.
`HARNESS_REPAIR_TIMEOUT_S`	`60`	Wall-clock bound on one repair episode. On expiry the notice stays pending and recoverable.
`HARNESS_REPAIR_MAX_TURNS`	`6`	Model rounds per episode. Six lets a provider that emits one tool call per round complete read → inspect → search → add → acknowledge.
`HARNESS_REPAIR_MAX_TOKENS`	`4096`	Output-token budget across the episode.
`HARNESS_REPAIR_TEMPERATURE`	(provider default)	Forwarded only when set.
`HARNESS_REPAIR_REASONING_EFFORT`	`"none"`	Current OpenAI reasoning models require this to use function tools on the wrapped chat-completions path. Forwarded only when LiteLLM reports model support.
`HARNESS_TRACE_REPAIR_AGENT`	`false`	Export a `pandaprobe` trace per repair run. Task tracing is unchanged either way.
`HARNESS_DOMAIN_POLICY`	(none)	A bounded statement of authorized domain behavior, passed to diagnosis as untrusted data. It never expands the repair agent’s tools.

Repair model calls go through PandaProbe’s official LiteLLM wrapper, so the provider credentials for repair_model must be present in the environment — separate from the PANDAPROBE_* platform credentials below.

The trigger

Variable	Default	Meaning
`HARNESS_ENABLE_TIER3`	`false`	Run the Tier-3 planning metrics to enrich a confirmed Tier-2 breach. Opt-in: every LLM-judge metric reads the whole trace.
(no env var) `trace_metrics_tier1`	`("task_completion", "coherence")`	Always-on trajectory metrics.
(no env var) `trace_metrics_tier2`	`("tool_correctness", "argument_correctness")`	Surgical step-level metrics, run on the last trace once Tier 1 breaches.
(no env var) `trace_metrics_tier3`	`("plan_adherence", "plan_quality", "step_efficiency")`	Explanatory only; never a breach source.
(no env var) `trace_list_limit`	`50`	Page size for `traces list` per turn.

The trajectory gate

A Tier-1 metric breaches on the shape of its series, never on its absolute value.

Variable	Default	Meaning
`HARNESS_GATE_WINDOW`	`5`	Traces without a new high before a stall fires. The primary knob.
(no env var) `gate_target`	`0.5`	The score a session is considered to be climbing toward. A stall requires the session’s peak to still be below this, so a session that reached the target can plateau freely.
(no env var) `gate_gain`	`0.02`	Improvement over the running peak that counts as progress and resets the window.
(no env var) `gate_drop`	`0.15`	Fall from the running peak that counts as a regression (independent of `gate_target`).

Long-horizon tasks want a larger gate_window; short interactive turns want a smaller one. Read your Tier-1 score distribution from calibration before moving gate_target or gate_drop.

Evaluation & polling

Variable	Default	Meaning
`HARNESS_POLL_INTERVAL_S`	`1.0`	Delay between eval-run score polls.
`HARNESS_POLL_MAX_ATTEMPTS`	`20`	Poll budget per run.
`HARNESS_EVAL_RETRY_ATTEMPTS`	`3`	Retries for transiently unscoreable sessions (trace-ingestion lag).
`HARNESS_EVAL_RETRY_BACKOFF_S`	`1.0`	Linear backoff base between retries.
`HARNESS_BARRIER_TIMEOUT_S`	`180`	The per-turn barrier’s budget. Deliberately generous and separate from the drain budget.
`HARNESS_DRAIN_TIMEOUT_S`	`15`	Best-effort default for `refresh` / `refresh_all` / `drain_validation`. Pass an explicit `timeout=` at a phase boundary — 15 s is far short of a replay-bound validation round.

Thresholds

Variable	Default	Meaning
(no env var) `outcome_threshold`	`0.9`	Floor for the verifier’s `outcome_correct` metric. High on purpose — a verifier’s verdict is trusted.
(no env var) `thresholds`	`{}`	Per-metric absolute floor overrides, by metric name. The way to calibrate a Tier-2 metric.

Metrics deliberately have no built-in per-metric threshold table: they all sit at 0.5, and the platform reports its own threshold in each score’s metadata, which the evaluator prefers when no local override exists. A table of identical values would only be a second place to spell a metric name. Remember that a Tier-1 metric’s floor is not a breach source at all — its verdict comes from the trajectory gate, so gate_target is the knob that matters there.

Noticing

Variable	Default	Meaning
`HARNESS_ALERT_COOLDOWN_TURNS`	`0`	Re-notice cooldown for persisting conditions (`0` = only on new conditions).
`HARNESS_ENRICH_FLAGGED_TRACES`	`false`	Fetch the worst flagged trace’s TOOL spans into the notice dump.
`HARNESS_OBSERVE_ONLY`	`false`	Shadow mode: evaluate + journal, never post notices.
`HARNESS_CIRCUIT_BREAKER_MAX_NOTICES`	`5`	Notices per window before escalating to `needs_human` (`0` = off).
`HARNESS_CIRCUIT_BREAKER_WINDOW_S`	`600`	The breaker window.

Cost & sampling

Variable	Default	Meaning
`HARNESS_EVAL_SAMPLE_EVERY`	`1`	Evaluate every Nth turn per session (`1` = every turn).
`HARNESS_SESSION_MIN_EVAL_INTERVAL_S`	`0`	Per-session rate limit between eval launches.
`HARNESS_MAX_CONCURRENT_EVALS`	`4`	Global concurrency cap across sessions.
`HARNESS_MAX_EVALS_PER_RUN`	`0`	Hard per-process eval budget (`0` = unlimited).

eval_sample_every > 1 and the trajectory gate work against each other: skipping turns thins the series the gate needs to detect a stall. Prefer bounding cost with enable_tier3=false (the default) and max_evals_per_run, which don’t distort the trajectory.

Rules & retrieval

Variable	Default	Meaning
`HARNESS_MAX_ACTIVE_RULES`	`0`	Optional operator cap on live rules (active + candidate). `0` means unlimited — deterministic novelty suppression, not an arbitrary count, is what controls proliferation.
`HARNESS_SANITIZE_MAX_LEN`	`2000`	Length cap in the sanitization trust boundary.
`HARNESS_RULE_RETRIEVAL`	`true`	Narrow the active rules inside a rendered scope file when a caller passes a query. Neither this nor `topk` affects the task-facing read/search/list tools, which never preselect on the agent’s behalf.
`HARNESS_RULES_CONTEXT_TOPK`	`8`	Active rules kept per query when retrieval applies.

Validation & the closed loop

Variable	Default	Meaning
`HARNESS_RULE_VALIDATION`	`true`	The candidate lifecycle. Mutating `Harness` construction requires `true`, so a repair-authored rule can never skip the evidence gate; `false` exists only for low-level `RulesStore` use in operator tooling.
`HARNESS_RULE_TRIAL_MIN_SESSIONS`	`5`	Forward trial: distinct sessions observed before a verdict.
`HARNESS_RULE_PROMOTE_MARGIN`	`0.05`	Minimum targeted-metric improvement to promote.
`HARNESS_RULE_REGRESS_MARGIN`	`0.05`	Maximum tolerated drop before a case counts as regressed.
`HARNESS_REPLAY_TIMEOUT_S`	`300`	Hard bound on one replay’s execution (hung replays degrade, never wedge).
`HARNESS_REPLAY_ENV_WAIT_TIMEOUT_S`	`0`	Extra grace for a replay that must first acquire a shared environment. A host signals arrival with `ReplayContext.mark_execution_started()`, and the execution budget above then starts fresh. `0` keeps the single-budget behavior.
`HARNESS_VALIDATION_ROUND_BUDGET_S`	`0`	Wall-clock bound on replay work in one validation round. Past it, remaining candidates still get a verdict — the cheap forward-trial one. `0` is unlimited.
`HARNESS_CAPTURE_EVAL_CASES`	`false`	Capture `breach`-severity sessions as replayable eval cases.
`HARNESS_EVAL_CASE_MAX`	`200`	Eval-set cap; oldest failures evict first, wins never.
`HARNESS_REGRESSION_SAMPLE`	`0`	Cases replayed per regression run (`0` = all).

Where a replay must queue for a shared resource — one environment server, one container, one device — leave replay_env_wait_timeout_s at 0 only if your replay never waits. Otherwise queueing is charged to the execution budget, and the resulting timeout is indistinguishable from a hung agent: the case is recorded as inconclusive evidence about a rule that never actually ran.

Robustness & scale

Variable	Default	Meaning
`HARNESS_HEALTH_CHECK`	`true`	Startup CLI/auth verification (degrades gracefully on failure).

Useful derived helpers

config.tier_metrics(1)          # ("task_completion", "coherence")
config.tier_metrics(3)          # () while enable_tier3 is False
config.replay_metrics()         # Tier 1 + Tier 2 — what a replay is re-scored on
config.threshold_for("tool_correctness")
config.rules_scope_file("global")   # <root>/rules/global.md
config.rules_file                   # <root>/rules.md — the generated task-facing guide
config.rules_store_file             # <root>/rules.jsonl — the source of truth

rules_scope_file() takes a scope already normalized by pandaprobe_harness.workspace.scopes.normalize_scope, because the value becomes a filename. That module also exports GLOBAL_SCOPE, SCOPED_SCOPE, and RESERVED_SCOPES — one definition shared by the store, the mailbox, and the turn hook.

Platform credentials

The pandaprobe subprocess inherits your environment, so the standard SDK/CLI variables apply unchanged: PANDAPROBE_API_KEY, PANDAPROBE_PROJECT_NAME, and PANDAPROBE_ENDPOINT for self-hosting. The harness itself never reads them — authentication is entirely the CLI’s concern. The repair agent is separate: its model calls go through PandaProbe’s LiteLLM wrapper, so whatever provider credentials repair_model needs (OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS or Vertex settings) must also be present.

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

Workspace & CLI

The repair agent

The trigger

The trajectory gate

Evaluation & polling

Thresholds

Noticing

Cost & sampling

Rules & retrieval

Validation & the closed loop

Robustness & scale

Useful derived helpers

Platform credentials

​Workspace & CLI

​The repair agent

​The trigger

​The trajectory gate

​Evaluation & polling

​Thresholds

​Noticing

​Cost & sampling

​Rules & retrieval

​Validation & the closed loop

​Robustness & scale

​Useful derived helpers

​Platform credentials

Workspace & CLI

The repair agent

The trigger

The trajectory gate

Evaluation & polling

Thresholds

Noticing

Cost & sampling

Rules & retrieval

Validation & the closed loop

Robustness & scale

Useful derived helpers

Platform credentials