reflect_bash.
Supply three things
- Reproducible cases: failures to improve and at least one known successful case to protect.
- Fresh business state: a sandbox, fixture, or reset environment for every replay arm.
- An independent verifier: check the actual refund ledger, test result, database state, or other outcome; do not accept the agent’s claim as the result.
Register replay
This recipe validates refund behavior using a singletask_completion metric. Construct the runtime with matching thresholds:
open_refund_sandbox / openRefundSandbox loads your case fixture, and build_refund_agent / buildRefundAgent connects its tools to a fresh traced agent using this Reflect runtime. Each sandbox exposes its independently defined expected refund and the actual ledger result. Close it after every arm.
validate_pending() / validatePending() after the original task releases its resources and before closing Reflect.
Alternatively, pass cases and replay together when constructing Reflect. It then schedules validation when a proposal is created. Wait separately with settle_validation() / settleValidation(); run() does not hold up the task boundary waiting for replay resources.
Return a real
Outcome with boolean success and independently measured scores for every configured threshold metric. With the defaults, that includes both task completion and coherence. Do not fill in an unmeasured metric with a fabricated passing score.What the validator checks
Every case runs twice: control receives the proposal’s baseline rules, and candidate receives baseline plus the proposal. Both start with fresh local memory and conversation identity. Arm order varies deterministically by case ID. Default acceptance requires:- At least three distinct paired cases, including a protected successful control.
- Complete matching metric sets, finite scores in
[0, 1], explicit success, and distinct execution IDs. - Unchanged assigned rule snapshots and the expected candidate assignment.
- No successful control becoming an unsuccessful candidate.
- No per-case metric decrease greater than 0.02.
- Mean metric improvement of at least 0.05 across non-protected comparisons.
Pending proposals stay outside ordinary fresh rule views. A stale proposal needs rebasing and revalidation; it cannot overwrite a newer accepted revision. File deletions and rule cleanup follow the same path.
Controlled live trials
When replay is unavailable, explicitly assign matched control/candidate executions withtrial_execution() / trialExecution(). Submit their independently verified outcomes through reflect.validator.record_trial() / recordTrial().
Each TrialCase records a case ID, reproducible input hash, base revision, protected flag, and both TrialArm records. Each arm identifies its execution, assigned candidate IDs, scores, success, and evidence. The same assessment policy applies. A later unrelated success is not automatically a control/candidate pair.

