Trace evaluation
Trace metrics answer focused questions about an execution: did the agent complete the task, select the right tools, supply correct arguments, and follow its plan? Seven metrics use LLM-as-judge analysis. Most separate extraction from judgment: first identify the task, tool calls, or plan, then evaluate that evidence. For example,task_completion extracts the user’s objective and the agent’s factual outcome before scoring how well they match. confidence judges the trace in a single call.
Coherence uses input and output embeddings to measure semantic alignment without a judge-generation call.
The ninth trace metric, loop detection, compares an output with recent outputs in the same session. It combines semantic and lexical similarity and runs as a session-evaluation signal. It is not available for standalone trace runs.
See trace metrics for the calculations, scoring guides, and metadata for each metric.
Session evaluation
A session groups traces from one agent lifecycle using a stablesession_id. Session evaluation helps identify repeated responses, isolated high-risk steps, and accumulated uncertainty across that lifecycle.
1
Collect the session's traces
PandaProbe loads traces with the same session ID in execution order.
2
Prepare trace-level signals
The worker reuses available scores or computes
confidence, loop_detection, tool_correctness, and coherence. Newly computed signals are stored as trace scores, so you can inspect the evidence behind the session result.3
Aggregate session scores
agent_reliability emphasizes the riskiest traces. agent_consistency combines confidence risk with penalties from the other signals. Aggregation uses deterministic math with no additional LLM or embedding calls.
You can customize these weights in the dashboard or pass
signal_weights through the API or CLI. See session metrics for the formulas and missing-signal behavior.
Read the result
Evaluation runs execute asynchronously. Each selected metric produces a score for each target, with a value, reason, and supporting metadata. Built-in scores range from 0 to 1; higher values indicate better results on the measured dimension. Scorestatus describes whether computation succeeded. A SUCCESS status can accompany a low value. Read the value and reason together, and use metadata to inspect extracted evidence, individual verdicts, or per-trace signals.
Some metrics return 1.0 when there is nothing to evaluate, such as a missing plan or no evaluable signals. Check the reason and metadata before treating a high score as evidence of successful behavior.
Next steps
Metrics
Compare all trace and session metrics and inspect their calculations.
Programmatic Scoring
Attach your own checks or feedback to a trace.
Set Up Evaluation
Run evaluations through the dashboard, API, CLI, or monitors.

