Skip to main content
A proposed rule is a hypothesis. Validation compares the same task with and without the change, checks known successes for regressions, and decides whether to publish it. Without a verifier and suitable cases, proposals remain unaccepted. The model cannot approve its own proposal through reflect_bash.

Supply three things

  1. Reproducible cases: failures to improve and at least one known successful case to protect.
  2. Fresh business state: a sandbox, fixture, or reset environment for every replay arm.
  3. An independent verifier: check the actual refund ledger, test result, database state, or other outcome; do not accept the agent’s claim as the result.
Reflect supplies fresh execution/session identities, local memory, and assigned rule snapshots. It cannot reset an arbitrary application or undo a real-world action for you.

Register replay

This recipe validates refund behavior using a single task_completion metric. Construct the runtime with matching thresholds:
The following is application integration code: open_refund_sandbox / openRefundSandbox loads your case fixture, and build_refund_agent / buildRefundAgent connects its tools to a fresh traced agent using this Reflect runtime. Each sandbox exposes its independently defined expected refund and the actual ledger result. Close it after every arm.
Call validate_pending() / validatePending() after the original task releases its resources and before closing Reflect. Alternatively, pass cases and replay together when constructing Reflect. It then schedules validation when a proposal is created. Wait separately with settle_validation() / settleValidation(); run() does not hold up the task boundary waiting for replay resources.
Return a real Outcome with boolean success and independently measured scores for every configured threshold metric. With the defaults, that includes both task completion and coherence. Do not fill in an unmeasured metric with a fabricated passing score.

What the validator checks

Every case runs twice: control receives the proposal’s baseline rules, and candidate receives baseline plus the proposal. Both start with fresh local memory and conversation identity. Arm order varies deterministically by case ID. Default acceptance requires:
  • At least three distinct paired cases, including a protected successful control.
  • Complete matching metric sets, finite scores in [0, 1], explicit success, and distinct execution IDs.
  • Unchanged assigned rule snapshots and the expected candidate assignment.
  • No successful control becoming an unsuccessful candidate.
  • No per-case metric decrease greater than 0.02.
  • Mean metric improvement of at least 0.05 across non-protected comparisons.
A strict reduction in guidance bytes can instead be accepted as maintenance if the preceding outcome and regression checks pass. Its reason records non-inferiority under those tolerances, not improved behavior. Pending proposals stay outside ordinary fresh rule views. A stale proposal needs rebasing and revalidation; it cannot overwrite a newer accepted revision. File deletions and rule cleanup follow the same path.

Controlled live trials

When replay is unavailable, explicitly assign matched control/candidate executions with trial_execution() / trialExecution(). Submit their independently verified outcomes through reflect.validator.record_trial() / recordTrial(). Each TrialCase records a case ID, reproducible input hash, base revision, protected flag, and both TrialArm records. Each arm identifies its execution, assigned candidate IDs, scores, success, and evidence. The same assessment policy applies. A later unrelated success is not automatically a control/candidate pair.

Interpret acceptance correctly

Acceptance is evidence under the supplied test conditions, not a statistical proof or a permanent repair guarantee. Use representative held-out cases and fresh business state. Do not reuse the proposing conversation as a clean control. Core acceptance does not require exact rule-text matching or a complete file read. Example-level retrieval audits can help demonstrate the loop; they are not production runtime gates.

More examples

For complete application fixtures and live multi-turn scenarios, see the SDK’s Python examples and TypeScript examples. These optional examples include independent refund ledgers and fresh control/candidate runs for the supported frameworks.