loop_detection runs only as a signal during session evaluation.
Built-in scores range from 0 to 1, with higher values indicating better results on that metric. Each metric defines a default threshold of 0.5. A score’s SUCCESS status means computation succeeded; quality is described by its value, reason, and metadata.
Trace metrics
Trace metrics analyze the recorded execution and its spans. Judge-based results depend on the available evidence and selected model; inspect the extracted context and reason when diagnosing a score.Task Completion
Registry name:task_completion · Default threshold: 0.5 · Method: LLM judge (2-stage)
Evaluates whether the agent accomplished the user’s stated objective.
How it works
Scoring guide
Example metadata
Example metadata
Tool Correctness
Registry name:tool_correctness · Default threshold: 0.5 · Method: LLM judge (2-stage)
Evaluates whether the agent selected appropriate tools for its task. Catches over-selection (unnecessary tools), under-selection (missing tools), and mis-selection (wrong tools).
How it works
What the judge evaluates
- Correct selection — Were the tools used appropriate and sufficient?
- Over-selection — Were unnecessary or redundant tools called?
- Under-selection — Were useful available tools ignored?
- Mis-selection — Were wrong or irrelevant tools chosen?
Example metadata
Example metadata
Argument Correctness
Registry name:argument_correctness · Default threshold: 0.5 · Method: LLM judge (3-stage)
Evaluates whether the arguments passed to each tool call were correct for the user’s task. While Tool Correctness checks which tools were called, Argument Correctness checks how they were called.
How it works
Score calculation
The score is computed deterministically from the verdicts:
Example metadata
Example metadata
Step Efficiency
Registry name:step_efficiency · Default threshold: 0.5 · Method: LLM judge (2-stage)
Evaluates how efficiently the agent executed its task, penalizing redundant steps, unnecessary tool calls, and speculative work.
How it works
What lowers the score
- Redundant or duplicate tool calls
- Unnecessary intermediate steps
- Speculative work that wasn’t needed for the task
- Overly verbose reasoning chains that don’t add value
Example metadata
Example metadata
Confidence
Registry name:confidence · Default threshold: 0.5 · Method: LLM judge (1-stage)
Evaluates whether the agent’s actions were decisive, appropriate, and well-founded. This metric is also used as a signal for session-level aggregation in agent_reliability and agent_consistency.
How it works
A single LLM call evaluates the entire trace against four criteria:
- Decisiveness — Did the agent act without unnecessary hesitation or contradictory steps?
- Appropriateness — Were the actions relevant to the user’s goal?
- Consistency — Did the agent maintain a coherent strategy throughout?
- Indicators of low confidence — hedging language, contradictions, unnecessary retries, vague outputs, repeated tool calls with identical parameters, or abandoned strategies
Example metadata
Example metadata
Plan Adherence
Registry name:plan_adherence · Default threshold: 0.5 · Method: LLM judge (3-stage)
Evaluates how closely the agent followed its declared or implied plan during execution. Useful for agents that produce a plan before acting.
How it works
If no plan is found in the trace, the metric returns 1.0 with an empty
plan. This means there was no plan to assess, not that planning succeeded.
Metadata includes the extracted task and plan, along with threshold and success.
Scoring guide
What the judge evaluates
- Were all planned steps executed?
- Were steps followed in the intended order?
- Were there extraneous actions not in the plan?
- Were any planned steps skipped?
Plan Quality
Registry name:plan_quality · Default threshold: 0.5 · Method: LLM judge (3-stage)
Evaluates the intrinsic quality of the agent’s plan, independent of whether the plan was followed. While Plan Adherence checks execution vs. plan, Plan Quality checks whether the plan itself was good.
How it works
If no plan is found, the metric returns 1.0 with an empty
plan. Inspect the reason before interpreting this as a high-quality plan.
Metadata includes the extracted task and plan, along with threshold and success.
What the judge evaluates
- Completeness — Does the plan address all aspects of the task?
- Logical coherence — Are steps ordered and structured sensibly?
- Optimality/efficiency — Could the plan be streamlined?
- Level of detail — Sufficiently detailed without being overly verbose?
- Alignment with task — Does the plan match the user’s intent?
Coherence
Registry name:coherence · Default threshold: 0.5 · Method: Embedding distance (no LLM call)
Measures semantic alignment between the trace input and output using embedding-based cosine distance. The calculation is deterministic for fixed embeddings and requires no judge-generation call. Semantic similarity alone does not establish factual correctness.
How it works
- The trace’s
inputandoutputare serialized to text - Both texts are embedded using the configured embedding model
- The cosine distance between the two embeddings is computed
- Score =
1.0 - cosine_distance(clamped to [0, 1])
- If either input or output is empty, the metric returns 1.0 with a note explaining coherence was assumed
- An assumed score for missing input or output is not evidence of coherence.
- The metric also serves as a signal for session-level aggregation
Example metadata
Example metadata
Loop Detection
Registry name:loop_detection · Default threshold: 0.5 · Method: Hybrid semantic + Jaccard similarity
Detects whether the agent is stuck repeating itself across traces in the same session. This metric requires session context — it compares the current trace’s output against previous traces.
How it works
- The current trace’s output and the previous traces’ outputs (up to a window of 3) are collected
- All outputs are embedded using the configured embedding model
- For each previous trace, two similarity scores are computed:
- Cosine similarity (semantic overlap) between embeddings
- Jaccard similarity (lexical overlap) between tokenized word sets (with stop-word removal)
- A hybrid score = cosine × Jaccard is computed for each pair
- Final score =
1.0 - max(hybrid_scores)(clamped to [0, 1])
Multiplying cosine and Jaccard similarity lowers the penalty for related outputs with different wording. The final score is high when repetition is low.
With no prior session context, an empty current output, or no nonempty previous outputs, the metric returns 1.0 with an explanatory note because repetition cannot be assessed.
Example metadata
Example metadata
Session metrics
Session evaluation prepares four trace-level signals before aggregation. The worker reuses available scores and computes missing ones. Aggregation itself makes no LLM or embedding calls.
Override individual weights with
signal_weights in a session run, or customize them in the dashboard. Omitted weights retain their defaults. Higher weights increase the associated risk contribution.
Agent Reliability
Registry name:agent_reliability · Default threshold: 0.5 · Method: Max-compose + top-k tail risk
Emphasizes the highest observed signal risks in a session. Use it to find problematic traces that an average score could obscure.
Algorithm
For each trace in the session, the metric computes a per-trace risk from the precomputed signals:
1
Convert signals to risks
Each signal score is inverted to become a risk value:
2
Weight and max-compose per trace
Each risk is multiplied by its signal weight, and the maximum weighted risk becomes the trace’s risk:Only signals that are present for a trace are included — missing signals are skipped, not treated as zero.
3
Top-k tail risk aggregation
Per-trace risks are sorted in descending order. The top 15% (at least 1) are selected:
4
Ensemble and final score
The raw session risk blends the top-k mean with the single worst trace:
- Max-compose ensures each trace’s risk is driven by its worst signal. An agent that has great tool selection but terrible coherence on one trace still gets flagged.
- Top-k focuses on the tail of the distribution. With 100 evaluable traces, the calculation averages the 15 highest risks, then blends in the single highest risk.
- The 10% max-risk blend gives extra weight to the single worst trace, preventing a handful of high-risk traces from being averaged away.
per_trace_risk > 0.5 are flagged in the metadata. This lets you quickly identify which specific traces are dragging down the session’s reliability score.
Example metadata
Example metadata
Agent Consistency
Registry name:agent_consistency · Default threshold: 0.5 · Method: Weighted RMS aggregation
Aggregates confidence risk across the session, amplified by loop, tool, and coherence risks. Repeated moderate uncertainty can lower the score even without one extreme trace. With a confidence score of 1.0, the trace contributes zero uncertainty even when other signals are poor; read reliability alongside this metric.
Algorithm
1
Filter traces
Only traces with a
confidence signal are included. Traces missing confidence are skipped entirely (unlike reliability, which includes traces with any signal subset).2
Compute weighted uncertainty per trace
For each trace:The penalty terms amplify the confidence risk. A trace with low confidence and poor tool correctness gets a higher uncertainty than one with low confidence alone.
3
RMS aggregation
The root mean square of all weighted uncertainties becomes the raw instability:
Example metadata
Example metadata
Reliability vs. Consistency
These two metrics complement each other:
Read the two scores together. Low reliability points to the worst weighted risks; low consistency points to amplified confidence risk across evaluable traces. Neither score establishes task completion on its own.
Edge cases
Both metrics use these fallback behaviors:A fallback score of 1.0 for missing data is not evidence of successful behavior. Review the reason,
traces_evaluated, and signal coverage alongside the score.Model and threshold settings
The run’s optionalmodel field selects the LLM judge. For session evaluation, it applies when computing missing judge-based signals; the worker can reuse existing signal scores. It does not change the embedding model used by coherence or loop detection, or the deterministic session aggregation.
Trace metrics normally include threshold and success in their metadata. Session metrics return aggregation details instead. Empty-input and other early-return cases may omit threshold metadata. The public eval-run API does not expose a threshold override; apply a custom quality bar when consuming results if needed.
Next steps
Set Up Evaluation
Run these metrics from the dashboard, API, CLI, or a recurring monitor.
Programmatic Scoring
Record your own checks and feedback alongside automated results.

