Concepts

​

The idea

Agents fail in production in ways that are invisible until someone reads a transcript: they loop on an identical tool call, drift into inconsistent state, quietly get worse after a prompt change. Tracing makes those failures observable and evaluation makes them measurable — but a human still has to notice the metric, diagnose the trace, and edit the prompt. The PandaProbe Harness closes that gap. It is the operational envelope around your agent — a diagnostic workspace on disk, a trajectory trigger, and a second repair agent — that observes your agent’s failures, diagnoses them, and writes operating rules your agent can consult, with no human in the loop.

​

Two agents, one workspace

Our philosophy is that diagnosis and execution are different jobs, so they belong to different agents. You own the task agent, its model, framework, prompts, domain tools, loop, and environment. PandaProbe owns instrumentation, evaluation, trajectory detection, notices, the repair agent, workspace administration, validation, and read-only rule delivery. The two agents never address each other. The workspace is the only thing between them, which is what lets either side change without the other noticing. The earlier design let the task agent administer its own workspace. It does not hold up: in a measured run, handed ten administrative tools and a check-your-mailbox mandate, the agent spent its turns operating the harness, and nine of the thirteen rules it wrote were about gaming its own diagnostic protocol rather than about the work.

​

The pull model

The harness never injects anything into your agent’s conversation. Instead:

• After each turn, the harness scores the turn’s new traces on the PandaProbe platform in a detached background task.
• When the trajectory degrades, it posts a structured diagnostic notice to a filesystem mailbox. Your agent never sees it — the repair agent does.
• The repair agent reads the notice, inspects the flagged trace, checks whether an existing rule already covers it, and writes at most one candidate rule.
• Your agent’s system context carries one stable sentence: learned rules exist, and four read-only tools can reach them. No rule bodies, no index, no banner — building it reads nothing.
• Your agent pulls, if and when it judges prior rules relevant.

This keeps the harness framework-agnostic (every framework already loads a system prompt and tools) and keeps eval-derived text out of the conversation except through one sanitized, auditable channel.

​

Three commitments that make it work

Self-healing is easy to build and hard to make useful. Three properties separate a harness that improves an agent from one that just generates noise — each learned from a measured benchmark run where the naive version regressed:

​

The closed loop

Detecting failures and writing rules is only half the job — an open loop trusts every self-authored rule immediately and never notices when a new rule breaks something old. The harness closes it:

• Evidence before trust. A new rule enters as a candidate. The harness validates it — by replaying the captured failure with the rule in force, or by watching the next live sessions — and promotes it to active only when it demonstrably helps. Unhelpful candidates are retired with a journaled reason. Neither agent can promote a rule; that would be a self-approving loop.
• Ground truth when you have it. If you already know what success means — a golden dataset, a benchmark grader, a business rule — wire an outcome verifier and promotion is decided by that oracle instead of a judged proxy.
• Guard the wins. A replayable eval set of captured failures and protected wins backs regression runs, so a newly learned rule can’t quietly break a flow that used to work.
• Measure the foundation. Thresholds are guesses until measured; an offline calibration tool reports how well the breach predicate matches real failures.

All of this is automatic. Two ingredients are optional and developer-supplied: a replay function that re-runs your agent on a captured scenario, and an outcome verifier that grades a finished task.

​

Vocabulary

​

Design principles

• Zero runtime dependencies. The core is pure standard library; framework adapters are optional extras.
• One platform seam. All platform access shells out to the pandaprobe CLI through a single injectable interface — never the REST API directly. Tests run fully offline against a fake.
• Never block, never break the host loop. Evaluation runs in detached tasks; the one deliberate wait, the per-turn barrier, is opt-in, separately budgeted, and degrades to a latency signal on expiry. Every failure path degrades gracefully instead of raising into your agent.
• Untrusted by default. Everything eval-derived that crosses into either agent’s context passes a sanitization trust boundary, and the repair prompt declares notice, dump, trace, task-summary, and policy text to be data, never instructions.
• Capability separation. The task agent reads; the repair agent writes one candidate; only validation promotes or retires. No agent can approve its own work, and administrative calls are rejected at dispatch.
• Nothing is force-injected. rules/ is read on demand. Context construction performs no read, search, or lookup on the agent’s behalf.
• No human in the healing loop. Detection, diagnosis, rule authoring, validation, and regression guarding are all automatic. Humans get involved only when the circuit breaker escalates to needs_human.

​

Where to go next