Quickstart

The PandaProbe Harness wraps any PandaProbe-instrumented agent in a self-healing envelope: it scores your agent’s trajectory as it works, and when progress stalls or a step goes wrong, a separate repair agent diagnoses the failure and writes an operating rule — which the harness then proves before trusting.

Before you begin, make sure you have:

A PandaProbe account. Sign up at app.pandaprobe.com.
A PandaProbe API key + project name.
The pandaprobe CLI installed and authenticated — the harness reaches the platform exclusively through it.
An agent traced with the PandaProbe SDK, so its turns produce scoreable traces.
A model for the repair agent, plus its provider credentials. It is a real LLM call, and no default is chosen for you.

Install the package

pip install pandaprobe-harness

uv add pandaprobe-harness

The core has zero runtime dependencies. Framework adapters are optional extras, e.g. pip install "pandaprobe-harness[langgraph]".

Install and authenticate the CLI

curl -fsSL https://cli.pandaprobe.com/install.sh | sh
export PANDAPROBE_API_KEY="your-api-key"
export PANDAPROBE_PROJECT_NAME="your-project-name"

Every platform call the harness makes — listing traces, running evaluations, polling scores — shells out to pandaprobe. It never talks to the REST API directly.

Pick a workspace root

The harness maintains its diagnostic workspace (mailbox, journal, rules, eval cases) on disk. The default is /harness; point it anywhere writable:

export HARNESS_ROOT="./harness"

Choose a repair model

Diagnosis runs on a separate, package-owned repair agent, so it needs a model. Any LiteLLM identifier works, and that provider’s own credentials must be present:

export HARNESS_REPAIR_MODEL="openai/gpt-..."

Harness.create() raises ValueError without this — no potentially billable model is ever selected for you. To evaluate without mutating anything, use observe_only=True instead.

Wire the harness into your agent loop

Any custom loop integrates in a handful of lines — no adapter required:

from pandaprobe_harness import Harness

harness = Harness.create()          # provisions the workspace, wires everything,
                                    # runs the startup health check

# 1. A stable one-sentence preamble: learned rules exist, and four read-only
#    tools can reach them. No rule bodies, no index — building it reads nothing.
system_prompt = harness.system_context(session_id) + MY_PROMPT

# 2. Optionally let your agent consult those rules (four read-only operations).
tools = my_tools + list(harness.task_tools.specs())

# 3. Delimit turns. `settle=True` waits for this turn's evaluation and one
#    bounded repair attempt, so a rule learned now is readable next turn.
async def one_turn(session_id: str, user_input: str) -> str:
    async with harness.turn(session_id, settle=True):
        return await my_agent_step(system_prompt, tools, user_input)

Delimit turns, not whole tasks. The trajectory gate needs a series of scores per session to detect a stall; hooking the harness once per task gives it one sample and nothing can ever fire.

Using LangGraph, CrewAI, or another supported framework? A Harness.for_<framework>() factory wires turn detection for you — see Framework adapters.

Watch a healing cycle

Nothing else is required — the loop is fully automatic:

Each turn, the harness scores the new traces on task_completion and coherence. Low scores early are fine; what it watches is the trajectory.
The trajectory stalls (no progress across gate_window turns) or regresses (a drop from the session’s best). That opens the gate.
Tier 2 runs on the latest trace only — tool_correctness, argument_correctness — to find which step was wrong.
Tier 2 confirms it, so a breach notice is posted. (Had Tier 2 come back clean, it would be an advisory trend notice instead.)
The repair agent picks it up: it reads the notice, inspects the flagged trace, checks whether an existing rule already covers this, then writes at most one rule — an unproven candidate — and chooses which scope it belongs in. Your task agent is not involved and loses no turns to this.
The harness validates the candidate (by replaying the captured failure, or by watching the next live sessions) and promotes it to active — or retires it. Neither agent can promote its own rule.
The validated rule is readable by your agent, on demand, in every future run.

Inspect what happened at any time:

harness.rules.active()                       # validated rules
harness.rules.candidates()                   # rules still under evaluation
harness.rules.read_scope("global")           # a rule file, as the agent sees it
harness.mailbox.pending()                    # unresolved notices (operator view)
harness.journal.recent()                     # the cross-run event log

Or read the repair outcome straight off settlement:

settlement = await harness.settle(session_id)
if settlement.repair is not None:
    print(settlement.repair.status)            # "candidate_added" | "no_proposal" | ...
    print(settlement.repair.selected_scope)    # where the rule was filed

Optional, and worth it

Two developer-supplied seams turn a good loop into a rigorous one:

harness = Harness.create(
    replay=my_replay_fn,      # re-run a captured failure → counterfactual validation
    verifier=my_grader,       # ground truth → promotion decided by the real outcome
)

A replay function upgrades rule validation from statistical to counterfactual and unlocks regression runs.
An outcome verifier — anything that can score a finished task — becomes the metric that decides promotion, instead of a judged proxy.

Try it offline first

The repository ships runnable, fully-offline demos — no credentials, no network:

git clone https://github.com/chirpz-ai/pandaprobe-harness && cd pandaprobe-harness
uv sync --group dev
uv run python examples/misc/offline_self_heal.py       # notice → repair episode → candidate
uv run python examples/misc/closed_loop_self_heal.py   # candidate → validate → promote

These use a deterministic completion fake in place of a repair model, so they need no credentials and no repair-model spend.

What’s next?

Concepts

What self-healing means here, and the ideas behind the closed loop

The repair agent

The repair agent: episodes, capabilities, and scope selection

Framework adapters

LangGraph, LangChain, DeepAgents, CrewAI, Claude Agent SDK, OpenAI Agents

Get Started

The Self-Heal Loop

Closing the Loop

Integrations

Reference

Optional, and worth it

Try it offline first

What’s next?

Concepts

The repair agent

Framework adapters

​Optional, and worth it

​Try it offline first

​What’s next?

Concepts

The repair agent

Framework adapters

Optional, and worth it

Try it offline first

What’s next?