Skip to main content
PandaProbe includes nine trace metrics and two session metrics. Eight trace metrics can run independently; loop_detection runs only as a signal during session evaluation. Built-in scores range from 0 to 1, with higher values indicating better results on that metric. Each metric defines a default threshold of 0.5. A score’s SUCCESS status means computation succeeded; quality is described by its value, reason, and metadata.

Trace metrics

Trace metrics analyze the recorded execution and its spans. Judge-based results depend on the available evidence and selected model; inspect the extracted context and reason when diagnosing a score.

Task Completion

Registry name: task_completion · Default threshold: 0.5 · Method: LLM judge (2-stage) Evaluates whether the agent accomplished the user’s stated objective. How it works Scoring guide

Tool Correctness

Registry name: tool_correctness · Default threshold: 0.5 · Method: LLM judge (2-stage) Evaluates whether the agent selected appropriate tools for its task. Catches over-selection (unnecessary tools), under-selection (missing tools), and mis-selection (wrong tools). How it works What the judge evaluates
  • Correct selection — Were the tools used appropriate and sufficient?
  • Over-selection — Were unnecessary or redundant tools called?
  • Under-selection — Were useful available tools ignored?
  • Mis-selection — Were wrong or irrelevant tools chosen?

Argument Correctness

Registry name: argument_correctness · Default threshold: 0.5 · Method: LLM judge (3-stage) Evaluates whether the arguments passed to each tool call were correct for the user’s task. While Tool Correctness checks which tools were called, Argument Correctness checks how they were called. How it works Score calculation The score is computed deterministically from the verdicts:
Neutral verdicts remain in the metadata but do not affect the score. If there are no task arguments to evaluate, the metric returns 1.0 (nothing to evaluate); this is not evidence of task completion.

Step Efficiency

Registry name: step_efficiency · Default threshold: 0.5 · Method: LLM judge (2-stage) Evaluates how efficiently the agent executed its task, penalizing redundant steps, unnecessary tool calls, and speculative work. How it works What lowers the score
  • Redundant or duplicate tool calls
  • Unnecessary intermediate steps
  • Speculative work that wasn’t needed for the task
  • Overly verbose reasoning chains that don’t add value

Confidence

Registry name: confidence · Default threshold: 0.5 · Method: LLM judge (1-stage) Evaluates whether the agent’s actions were decisive, appropriate, and well-founded. This metric is also used as a signal for session-level aggregation in agent_reliability and agent_consistency. How it works A single LLM call evaluates the entire trace against four criteria:
  • Decisiveness — Did the agent act without unnecessary hesitation or contradictory steps?
  • Appropriateness — Were the actions relevant to the user’s goal?
  • Consistency — Did the agent maintain a coherent strategy throughout?
  • Indicators of low confidence — hedging language, contradictions, unnecessary retries, vague outputs, repeated tool calls with identical parameters, or abandoned strategies
Scoring guide

Plan Adherence

Registry name: plan_adherence · Default threshold: 0.5 · Method: LLM judge (3-stage) Evaluates how closely the agent followed its declared or implied plan during execution. Useful for agents that produce a plan before acting. How it works If no plan is found in the trace, the metric returns 1.0 with an empty plan. This means there was no plan to assess, not that planning succeeded. Metadata includes the extracted task and plan, along with threshold and success. Scoring guide What the judge evaluates
  • Were all planned steps executed?
  • Were steps followed in the intended order?
  • Were there extraneous actions not in the plan?
  • Were any planned steps skipped?

Plan Quality

Registry name: plan_quality · Default threshold: 0.5 · Method: LLM judge (3-stage) Evaluates the intrinsic quality of the agent’s plan, independent of whether the plan was followed. While Plan Adherence checks execution vs. plan, Plan Quality checks whether the plan itself was good. How it works If no plan is found, the metric returns 1.0 with an empty plan. Inspect the reason before interpreting this as a high-quality plan. Metadata includes the extracted task and plan, along with threshold and success. What the judge evaluates
  • Completeness — Does the plan address all aspects of the task?
  • Logical coherence — Are steps ordered and structured sensibly?
  • Optimality/efficiency — Could the plan be streamlined?
  • Level of detail — Sufficiently detailed without being overly verbose?
  • Alignment with task — Does the plan match the user’s intent?
Scoring guide

Coherence

Registry name: coherence · Default threshold: 0.5 · Method: Embedding distance (no LLM call) Measures semantic alignment between the trace input and output using embedding-based cosine distance. The calculation is deterministic for fixed embeddings and requires no judge-generation call. Semantic similarity alone does not establish factual correctness. How it works
  1. The trace’s input and output are serialized to text
  2. Both texts are embedded using the configured embedding model
  3. The cosine distance between the two embeddings is computed
  4. Score = 1.0 - cosine_distance (clamped to [0, 1])
A small distance (high score) means the output is semantically aligned with the input. A large distance (low score) suggests the output is unrelated or off-topic. Edge cases
  • If either input or output is empty, the metric returns 1.0 with a note explaining coherence was assumed
  • An assumed score for missing input or output is not evidence of coherence.
  • The metric also serves as a signal for session-level aggregation

Loop Detection

Registry name: loop_detection · Default threshold: 0.5 · Method: Hybrid semantic + Jaccard similarity Detects whether the agent is stuck repeating itself across traces in the same session. This metric requires session context — it compares the current trace’s output against previous traces.
Loop Detection is excluded from standalone trace eval runs. It runs automatically as a signal during session-level evaluations.
How it works
  1. The current trace’s output and the previous traces’ outputs (up to a window of 3) are collected
  2. All outputs are embedded using the configured embedding model
  3. For each previous trace, two similarity scores are computed:
    • Cosine similarity (semantic overlap) between embeddings
    • Jaccard similarity (lexical overlap) between tokenized word sets (with stop-word removal)
  4. A hybrid score = cosine × Jaccard is computed for each pair
  5. Final score = 1.0 - max(hybrid_scores) (clamped to [0, 1])
Why the hybrid approach Multiplying cosine and Jaccard similarity lowers the penalty for related outputs with different wording. The final score is high when repetition is low. With no prior session context, an empty current output, or no nonempty previous outputs, the metric returns 1.0 with an explanatory note because repetition cannot be assessed.

Session metrics

Session evaluation prepares four trace-level signals before aggregation. The worker reuses available scores and computes missing ones. Aggregation itself makes no LLM or embedding calls. Override individual weights with signal_weights in a session run, or customize them in the dashboard. Omitted weights retain their defaults. Higher weights increase the associated risk contribution.

Agent Reliability

Registry name: agent_reliability · Default threshold: 0.5 · Method: Max-compose + top-k tail risk Emphasizes the highest observed signal risks in a session. Use it to find problematic traces that an average score could obscure. Algorithm For each trace in the session, the metric computes a per-trace risk from the precomputed signals:
1

Convert signals to risks

Each signal score is inverted to become a risk value:
2

Weight and max-compose per trace

Each risk is multiplied by its signal weight, and the maximum weighted risk becomes the trace’s risk:
Only signals that are present for a trace are included — missing signals are skipped, not treated as zero.
3

Top-k tail risk aggregation

Per-trace risks are sorted in descending order. The top 15% (at least 1) are selected:
4

Ensemble and final score

The raw session risk blends the top-k mean with the single worst trace:
Why max-compose + top-k
  • Max-compose ensures each trace’s risk is driven by its worst signal. An agent that has great tool selection but terrible coherence on one trace still gets flagged.
  • Top-k focuses on the tail of the distribution. With 100 evaluable traces, the calculation averages the 15 highest risks, then blends in the single highest risk.
  • The 10% max-risk blend gives extra weight to the single worst trace, preventing a handful of high-risk traces from being averaged away.
Flagged traces Traces with per_trace_risk > 0.5 are flagged in the metadata. This lets you quickly identify which specific traces are dragging down the session’s reliability score.
Interpreting the score These ranges are reading aids for the default weights, not measured failure probabilities.

Agent Consistency

Registry name: agent_consistency · Default threshold: 0.5 · Method: Weighted RMS aggregation Aggregates confidence risk across the session, amplified by loop, tool, and coherence risks. Repeated moderate uncertainty can lower the score even without one extreme trace. With a confidence score of 1.0, the trace contributes zero uncertainty even when other signals are poor; read reliability alongside this metric. Algorithm
1

Filter traces

Only traces with a confidence signal are included. Traces missing confidence are skipped entirely (unlike reliability, which includes traces with any signal subset).
2

Compute weighted uncertainty per trace

For each trace:
The penalty terms amplify the confidence risk. A trace with low confidence and poor tool correctness gets a higher uncertainty than one with low confidence alone.
3

RMS aggregation

The root mean square of all weighted uncertainties becomes the raw instability:
Why RMS and amplification RMS reflects both magnitude and variation. If every trace has uncertainty 0.3, RMS is 0.3. With uneven uncertainty, squaring gives larger values more influence than a simple average would. The other signals amplify confidence risk: uncertain execution with poor tool selection or coherence contributes more instability than the same confidence risk with strong supporting signals.
Interpreting the score These ranges are reading aids for the default weights, not measured failure probabilities.

Reliability vs. Consistency

These two metrics complement each other: Read the two scores together. Low reliability points to the worst weighted risks; low consistency points to amplified confidence risk across evaluable traces. Neither score establishes task completion on its own.

Edge cases

Both metrics use these fallback behaviors:
A fallback score of 1.0 for missing data is not evidence of successful behavior. Review the reason, traces_evaluated, and signal coverage alongside the score.

Model and threshold settings

The run’s optional model field selects the LLM judge. For session evaluation, it applies when computing missing judge-based signals; the worker can reuse existing signal scores. It does not change the embedding model used by coherence or loop detection, or the deterministic session aggregation. Trace metrics normally include threshold and success in their metadata. Session metrics return aggregation details instead. Empty-input and other early-return cases may omit threshold metadata. The public eval-run API does not expose a threshold override; apply a custom quality bar when consuming results if needed.

Next steps

Set Up Evaluation

Run these metrics from the dashboard, API, CLI, or a recurring monitor.

Programmatic Scoring

Record your own checks and feedback alongside automated results.