Harness

​

The repair agent moves into the package

developer task agent → task trace → evaluation/gate → notice
                                                ↓
                                    PandaProbe's repair agent
                                                ↓
                                     candidate rule
                                                ↓
developer task agent ── read-only tools ──→ shared workspace

​

Added

• ManagedRepairAgent — a package-owned, bounded, provider-neutral tool loop over PandaProbe’s official LiteLLM wrapper. It owns the repair prompt, the capability set, and the episode lifecycle; hosts supply only model transport. Configured by repair_model, repair_timeout_s, repair_max_turns (default 6), repair_max_tokens, repair_temperature, repair_reasoning_effort (default "none", which current OpenAI reasoning models require for function tools on the wrapped chat-completions path), trace_repair_agent, and domain_policy.
• TaskToolset — the only tool surface intended for a task agent: harness_rules_read, harness_rules_search, harness_rules_list, and harness_rule_status. All read-only. Dispatch rejects every administrative or mutating call, including hallucinated ones.
• Structured repair outcomes. SettleResult.repair returns a RepairResult carrying status, task/repair session IDs, episode and notice IDs, recommended and selected scope, considered/existing/candidate rule IDs, suppression reason, model turns and tool calls, normalized usage, and an error category. Repair failure or timeout never fails the developer task.
• Repair episodes. Related same-turn notices whose trace or signature evidence overlaps are coalesced into one episode, resolved atomically. An episode may create at most one candidate. Timeout, cancellation, or failure acknowledges nothing.
• Model-driven rule scoping. The repair agent decides where a rule is filed, from the failure evidence it already holds — part of the existing repair call, not an extra model round. global is the default for broadly reusable rules; a concise contextual name (an application, workflow, or domain seen in the evidence) is preferred when the rule belongs to that context; scoped is the fallback when no meaningful stable name can be determined. Custom names are an open catalog, normalized only for filename safety.
• RuleScopeHint and task_summary on TurnContext — bounded, optional host metadata that informs the scope decision without dictating it. A host’s own label for itself (a benchmark or integration name) is rejected as a scope, because it says where the agent ran rather than what failed.
• Harness.settle_validation(timeout=...), Harness.validation_pending, and a timeout= argument on drain_validation(), which now returns whether it actually drained. Call settle_validation() at a phase boundary before snapshotting or reporting a ruleset.
• validation_round_budget_s and replay_env_wait_timeout_s, plus ReplayContext.mark_execution_started(). The first bounds replay work per validation round; the second keeps time spent queueing for a shared environment out of the replay execution budget.
• Validation telemetry. New journal events validation_round_started, validation_candidate_started, validation_replay_case, validation_verdict, and validation_round_finished, with a closed-set pending_reason (trial_in_progress, no_matching_replayable_case, replay_inconclusive, candidate_not_exercised, env_wait_timeout, round_budget_exhausted). rule_retire now carries structured evidence — the per-case deltas that decided it.

​

Changed

• BREAKING (workspace layout): the generated task-facing guide is <harness_root>/rules.md, beside the rules.jsonl store and rules/ scope files it indexes. It is regenerated from rules.jsonl on every rule mutation and at startup, so it is a derived view with no authority — delete it and it comes back.
• BREAKING (default scope): a new rule defaults to global, not scoped, and a task-facing read with no scope argument defaults to global. scoped is now a deliberate verdict — “specific, but no better name” — rather than a catch-all for silence. Existing global, scoped, and custom scope files keep working; no rule is migrated, moved, or duplicated.
• BREAKING: DiagnosticNotice.recommended_scope and RepairAssignment.recommended_scope are str | None. None means “no host recommendation”, which is deliberately distinct from recommending the default.
• Every candidate now reaches a verdict. Replay is still the strong path, but a bounded round falls back to the cheap forward trial for candidates it cannot reach, and rounds rotate which candidate replays first. Previously a promotable candidate could accumulate observations indefinitely and never be asked.
• Replay verdicts are attributable. A replay sees only the candidate under test — active rules stay visible as the baseline — and a replay that never read the candidate cannot produce a conclusive verdict. On a failure case only the candidate’s own target metric or the authoritative outcome score can retire it; a protected win case still retires on any metric. In one measured run, 44% of retirements had fired on a metric the rule never claimed.
• harness_rule_add rejects a path-shaped scope instead of slugifying it, and validates metric against the evaluator’s metric registry. A rule whose metric matches no signature reads as “never breached”, which can invert its own verdict — so the repair model is told to correct it rather than having it silently persisted.
• Task-facing context and the rules.md template say learned rules, not “optional learned guidance”, and state plainly that PandaProbe does not automatically insert rule contents. No behavior change: nothing was ever injected.
• settle() waits for evaluation, notice persistence, and one bounded repair attempt. It deliberately does not wait for a validation round, because a replay can need the very resource the current turn holds.

​

Removed

• BREAKING: the task-agent self-administration architecture. Task agents no longer receive mailbox, trace-inspection, notice-acknowledgement, rule-write, rule-retirement, validation, or regression capabilities.
• BREAKING: HarnessToolset, Harness.toolset, OP_SCHEMAS, build_toolset_from_env, and the pandaprobe-harness-agent companion CLI. Use Harness.task_tools; the task schema is TASK_OP_SCHEMAS, and repair schemas and dispatch stay package-internal.
• BREAKING: Harness.shell. Task agents no longer receive a harness-owned administrative shell.
• BREAKING: the no-argument Harness.system_context() / startup_context() and the two-argument compose_system_preamble(rules, mailbox). All now require a task session ID.
• BREAKING: HarnessConfig.concurrent_eval and HARNESS_CONCURRENT_EVAL — one eval run has covered every metric since v0.8, and nothing read the flag.
• BREAKING: the "legacy" Resolution.kind and the pre-v0.8 severity: "relative" alias. Every resolution now names what happened; an unrecognized kind reads as no_proposal, and an unknown severity reads as breach rather than silently de-escalating.

​

One trigger, one detection pipeline

​

Removed

• Public exports: TrendDetector, TrendVerdict, SIGNAL_NAMES, and EwmaState.
• Public members: Metric.RELIABILITY, Metric.CONSISTENCY, and MetricEvaluator.evaluate_turn().
• Configuration fields: trigger_mode, session_metrics, signal_weights, reliability_threshold, consistency_threshold, eval_reliability, eval_consistency, enable_trend, ewma_fast_span, ewma_slow_span, trend_margin_cross, trend_min_samples, adaptive_threshold, adaptive_margin_drop, percentile_window, percentile_floor, and hydrate_history_from_backend.
• Environment variables: HARNESS_TRIGGER_MODE, HARNESS_ENABLE_TREND, HARNESS_ADAPTIVE_THRESHOLD, HARNESS_RELIABILITY_THRESHOLD, HARNESS_EWMA_FAST_SPAN, HARNESS_ADAPTIVE_MARGIN_DROP, HARNESS_CONSISTENCY_THRESHOLD, HARNESS_EWMA_SLOW_SPAN, HARNESS_PERCENTILE_WINDOW, HARNESS_EVAL_RELIABILITY, HARNESS_TREND_MARGIN_CROSS, HARNESS_PERCENTILE_FLOOR, HARNESS_EVAL_CONSISTENCY, HARNESS_TREND_MIN_SAMPLES, and HARNESS_HYDRATE_HISTORY_FROM_BACKEND.

​

Changed

• Score history stores only trace series and trajectory-gate state. v0.7 files carrying EWMA state still load, and obsolete state is dropped on the next write.
• The HistorySource extension point remains and now exposes the gate’s atomic record_gated() operation. Use a shared implementation for replica fan-out; backend session-score hydration is gone.
• Legacy notices with severity: "relative" load as advisory trend notices, preventing a silent escalation to breach.
• Calibration reads local trace history and eval-set baselines. Platform trace score-list records are not used because they do not carry the session id required to join session labels.

​

The “signal that discriminates” release

​

Added

• Trace-level three-tier trigger (trigger_mode="trace", the new default). Tier 1 (task_completion, coherence) scores every trace; Tier 2 (tool_correctness, argument_correctness) runs on the last trace only and only once Tier 1 breaches; Tier 3 (planning/efficiency, opt-in via enable_tier3) only enriches a confirmed Tier-2 breach. New modules evaluation/traces.py (TraceLocator — trace discovery, which did not exist before) and hook/tiers.py (TierRunner).
• Trajectory gate (evaluation/trajectory.py). A Tier-1 metric breaches on the shape of its series — a stall (no gain toward gate_target across gate_window traces) or a regression (a gate_drop fall from the running peak) — and any real gain resets the window, so a healthy climbing session never breaches. A Tier-1 score’s absolute floor is deliberately not a breach: an agent three steps into a task has legitimately not finished it.
• Per-turn await barrier: Harness.settle(session_id) / harness.turn(session_id, settle=True), on its own generous barrier_timeout_s. This is what makes healing take effect within a session, and it is the precondition for the gate having a series at all.
• Optional outcome verifier: Harness.create(..., verifier=...). A developer-supplied (session_id, end_state) -> float | bool | None oracle emits a synthetic outcome_correct score that drives breaches and, when present, decides promotion. Prefer a continuous score: a pass/fail flag that is almost always 0 discriminates no better than the metrics this release replaces.
• PandaHarnessHook.pending_sessions, so host-side phase barriers no longer need the private task map.

​

Changed

• BREAKING (workspace layout): harness_rules.md is now a skill root — protocol, tool list, and a generated References index, with no rule text. Rules live in a new rules/ subtree (global.md, scoped.md, and any <topic>.md the agent creates), and Rule gains a free-form scope. Existing rules.jsonl records migrate on read: untagged → global, tagged → scoped, preserving v0.6’s meaning.
• BREAKING (behavior): the system context no longer injects rule bodies. The agent pulls them with the new harness_rules_read. Retrieval is now keyed on scope rather than on whether a rule happens to carry tags — which previously made any rule added without a notice a permanent global by accident.
• BREAKING (toolset): 14 tools → 10. Removed harness_history (the gate and the notice now carry the trajectory), harness_journal, harness_reflect, harness_evalset_list, harness_evalset_attach. HarnessToolset no longer takes history= or evalset= — nothing left reads them.
• BREAKING (signature): Harness.system_context(), PandaHarnessHook.startup_context() and compose_system_preamble() no longer take a task_hint. It existed to pre-select which rules to inline, and no rules are inlined now.
• Tier-1 scoring issues one platform run per turn covering every new trace, and the trajectory gate folds a whole trace’s metrics in one history write instead of two per metric.
• Trace listings no longer burn the retry budget on a warm session: an empty listing is only retried while the session has never yet produced a trace.
• A wired verifier that has no verdict for a task no longer vetoes promotion. The target metric is now chosen per case from the deltas that actually arrived, in trust order (outcome_correct → the rule’s metric → the triggering signature).
• Notices carry the triggering metric’s trace_id, tier, and a coarse scope_hint (global for a trajectory fire, scoped for a step-level breach) that defaults harness_rule_add’s scope.
• Rule validation and regression runs re-score a replay on the trace metrics (Tier 1 + Tier 2) against its last trace, not the session composites.
• MetricEvaluator gained evaluate_trace / score_last_trace; --signal-weights is now sent only on the session path, where the platform actually accepts it.
• Severity mapping is deliberate: a Tier-1-only fire is advisory (trend, no eval case captured), while a confirmed Tier-2 breach is a breach — so only surgical, diagnosed failures become promotable eval cases.

​

The “closed loop” release

​

Changed

• BREAKING (behavior): harness_rule_add now records a candidate rule, not an active one. Candidates still render into the system context (under a clearly-labeled “Provisional rules (under evaluation)” section — a rule must be in force to be measurable) and are promoted to active only after a validator shows they help: ReplayValidator (replays the captured failing scenario through a developer-supplied replay function; promotes iff the targeted metric improves past rule_promote_margin with no case regressing past rule_regress_margin) or, when no replay function is wired, ForwardTrialValidator (compares the breach rate over the next rule_trial_min_sessions live sessions against the baseline captured at add time). Unfavorable candidates are retired with a journaled reason. Set HARNESS_RULE_VALIDATION=false to restore the v0.5 add→active behavior.
• Rule retrieval is task-conditioned by default: the system preamble renders global (untagged) rules plus the top-rules_context_topk rules lexically relevant to the pending notices and an optional system_context(task_hint=...) — not the full set. Everything else stays reachable via harness_rules_search / harness_rules_list. Set HARNESS_RULE_RETRIEVAL=false to restore render-everything.
• harness_rule_retire now retires candidates as well as active rules and records a reason; the rule dedup/cap count the whole live set (active + candidate).
• harness_reflect additionally returns candidate_rules and recent_validations so the reflection cycle learns which kinds of rules survive validation.

​

Added

• Rule lifecycle (candidate → active | retired) with auto-derived retrieval tags and full trial bookkeeping (TrialState: baseline vs. trial breach rates, observed/breached sessions, replay attempts, verdict).
• Validation engine driven automatically from the hook on every handled report — single-flight, time-bounded (replay_timeout_s), and incapable of blocking or crashing the host loop. New journal events: rule_promote, rule_retire (with reason), validation, evalset_capture, regression.
• Replayable regression eval-set: breach notices capture the session as a replayable failure case (opt-in via capture_eval_cases); known-good sessions can be captured as protected win cases (never auto-evicted). The ReplayFn seam — async (case, system_context) -> new_session_id — is how the harness re-runs your agent; wire it via Harness.create(..., replay=...).
• harness.run_regression() + the pandaprobe-harness-eval CLI: replay the eval set (wins first) against the current rule set and classify each case improved / unchanged / regressed vs. baseline. Without a replay function it degrades to one clear warning and all-skipped results.
• Metric calibration (pandaprobe-harness-calibrate + a calibrate() library API): precision/recall/F1 of the breach predicate, a confusion matrix, and a threshold sweep with labels (JSON/CSV or eval-set proxies); score distribution, histogram, sweep, and inter-metric agreement without.
• Five new toolset operations (9 → 14): harness_rule_status, harness_rules_search, harness_rules_list, harness_evalset_list, harness_evalset_attach.
• New facade surface: Harness.create(..., replay=) on every factory, harness.evalset, run_regression(), validate_candidates(), drain_validation(), system_context(task_hint=...).
• Ten new config knobs, all mirrored as HARNESS_* env vars: rule_validation, rule_trial_min_sessions, rule_promote_margin, rule_regress_margin, replay_timeout_s, capture_eval_cases, eval_case_max, regression_sample, rule_retrieval, rules_context_topk.

​

Fixed

• Rule.from_json no longer coerces unknown statuses to active — a persisted candidate round-trips instead of silently self-promoting across restarts.

​

The “pull model” release

​

Added

• Workspace substrate: Mailbox with DiagnosticNotice records (mailbox/pending/ → mailbox/processed/), an append-only Journal (journal.jsonl), and a RulesStore (rules.jsonl) with provenance, dedup, a rule cap, and per-rule effectiveness tracking.
• HarnessToolset exposing 9 agent-facing operations, the sandbox-allow-listed pandaprobe-harness-agent companion CLI, and native tool registrations for the supported frameworks.
• Harness facade with zero-adapter turn() / run_turn() entry points.
• Cost/latency controls: per-session eval sampling, per-session rate limiting, a global concurrency cap, and a hard per-process eval budget.
• observe_only shadow mode; a circuit breaker escalating notice storms to a single needs_human; a startup health check with graceful degraded mode; backend history hydration for horizontally-scaled agents; sandbox hardening (env scoping, argv deny rules); and a sanitization trust boundary for all eval-derived text entering agent context.

​

Security

• Mailbox rejects notice ids that are not a single safe path component, closing a workspace path-traversal vector.
• The restricted shell catches mid-path traversal, and argv deny rules match subcommands as ordered subsequences (leading global flags no longer bypass them).

​

Removed

• BREAKING — the push-model alert-injection surface is gone (inject_alert, consume_*, startup_messages, drain_pending, append_rule, and friends). Delivery is mailbox + toolset, always pull.

• Async, supersede-cancelling evaluation loop with EWMA trend detection, adaptive (relative) thresholds, and per-signature alert cooldowns.
• Single batched eval run per turn covering all active session metrics, with eventual-consistency retries and bounded run polling.
• Framework adapter suite: LangGraph, LangChain, DeepAgents, CrewAI, Claude Agent SDK, and OpenAI Agents.

• Initial public harness: the pandaprobe CLI subprocess seam, a turn-end evaluation hook with absolute score thresholds, trace dumps under traces/, harness_rules.md, and the Dockerised diagnostic sandbox with a restricted shell.