Skip to main content

​
The idea

Agents fail in production in ways that are invisible until someone reads a transcript: they loop on an identical tool call, drift into inconsistent state, quietly get worse after a prompt change. Tracing makes those failures observable and evaluation makes them measurable — but a human still has to notice the metric, diagnose the trace, and edit the prompt. The PandaProbe Harness closes that gap. It is the operational envelope around your agent — a diagnostic workspace on disk, a toolbelt, standing rules, and lifecycle hooks — that lets the agent observe its own failures, analyze them, and rewrite its own operating rules, with no human in the loop.
Think of the harness as giving your agent three things it doesn’t normally have: a memory of its own failures (the journal and mailbox), the means to investigate them (the toolset), and a place to write down what it learned (the rules) — plus a referee that checks the lesson actually helps before trusting it.

​
The pull model

The harness never injects anything into your agent’s conversation. Instead:
  • After each turn, the harness evaluates the session on the PandaProbe platform in a detached background task.
  • When quality degrades, it posts a structured diagnostic notice to a filesystem mailbox — nothing enters the message queue.
  • The always-loaded system context carries a compact banner — ⚠ HARNESS: N pending diagnostic notice(s) — and a standing protocol that tells the agent to check its mailbox at the start of each turn.
  • The agent pulls: it lists the mailbox, reads the notice and its trace dump, inspects the flagged traces, records a mitigation rule, and acknowledges the notice.
The agent drives the loop; the harness maintains the substrate and the trigger. This keeps the harness framework-agnostic (every framework already loads a system prompt and tools) and keeps eval-derived text out of the conversation except through one sanitized, auditable channel.

​
The closed loop

Detecting failures and writing rules is only half the job — an open loop trusts every self-authored rule immediately, injects every rule into every prompt, and never notices when a new rule breaks something old. Since v0.6 the loop is closed, on three principles:

Evidence before trust

A new rule enters as a candidate. The harness validates it — by replaying the captured failure with the rule in force, or by watching the next live sessions — and promotes it to active only when it demonstrably helps. Unhelpful candidates are retired with a journaled reason.

Relevance over volume

Only global rules plus the top-k rules lexically relevant to the current situation are injected into the prompt. The rest stay reachable on demand, so a rule learned from one failure mode doesn’t dilute attention on unrelated tasks.

Measure the foundation

Everything keys off “score below threshold” — and the threshold is a guess until measured. An offline calibration tool reports how well the breach predicate matches real failures and recommends better thresholds.
All of this is automatic. The only human-supplied ingredient is optional: a replay function that re-runs your agent on a captured scenario, which upgrades validation from statistical (forward trials over live sessions) to counterfactual (replay the exact failure with and without the rule).

​
Vocabulary

TermMeaning
WorkspaceThe on-disk root (HARNESS_ROOT, default /harness) holding the mailbox, journal, rules, traces, and eval set.
TurnOne completed agent step. The unit the harness evaluates.
SessionThe conversation grouping (shared with SDK tracing). Metrics are session-level.
NoticeA structured DiagnosticNotice describing a detected degradation: metrics, flagged traces, per-trace signal breakdown, a severity, and a dump path.
Mailboxmailbox/pending/ → mailbox/processed/. Where notices wait for the agent and where acknowledged notices live.
Journaljournal.jsonl — the append-only cross-run event log (notices, acks, rule lifecycle, validations, regressions). The agent’s long-term memory of its own health.
RuleA learned operating constraint with provenance. Lifecycle: candidate → active | retired. Rendered into the system context.
SignatureA compact condition label like breach:agent_reliability — used for de-duplication, tagging, and matching rules to eval cases.
Severitytrend < relative < breach < needs_human. Only needs_human (the circuit breaker) asks for a person.
Eval caseA captured scenario in evalset/ — a failure to fix or a win to protect — with its signature, baseline scores, and (when available) a replay input.
Replay functionDeveloper-supplied async (case, system_context) -> new_session_id: re-run the agent on a captured input under a given rule set. The strong path for validation and regression runs.
TrialForward-trial bookkeeping on a candidate: baseline breach rate at add time vs. the breach rate observed over the next live sessions.

​
Design principles

  • Zero runtime dependencies. The core is pure standard library; framework adapters are optional extras.
  • One platform seam. All platform access shells out to the pandaprobe CLI through a single injectable interface — never the REST API directly. Tests run fully offline against a fake.
  • Never block, never break the host loop. Evaluation runs in detached tasks; every failure path degrades gracefully (a pending score, a log line, a journal event) instead of raising into your agent.
  • Untrusted by default. Everything eval-derived that crosses into agent context passes a sanitization trust boundary, and the protocol tells the agent that notice/dump/trace contents are data, never instructions.
  • No human in the healing loop. Detection, diagnosis, rule authoring, validation, and regression guarding are all agent/harness-driven. Humans get involved only when the circuit breaker escalates to needs_human.

​
Where to go next

How it works

The full producing/consuming pipeline and the workspace on disk

Rule validation

The candidate lifecycle: replay validation and forward trials