Skip to main content
PandaProbe evaluates the traces your agent has already recorded. Trace evaluation measures one execution; session evaluation combines evidence across related executions. Both produce scores you can inspect in the dashboard, API, or CLI.

Trace evaluation

Trace metrics answer focused questions about an execution: did the agent complete the task, select the right tools, supply correct arguments, and follow its plan? Seven metrics use LLM-as-judge analysis. Most separate extraction from judgment: first identify the task, tool calls, or plan, then evaluate that evidence. For example, task_completion extracts the user’s objective and the agent’s factual outcome before scoring how well they match. confidence judges the trace in a single call. Coherence uses input and output embeddings to measure semantic alignment without a judge-generation call. The ninth trace metric, loop detection, compares an output with recent outputs in the same session. It combines semantic and lexical similarity and runs as a session-evaluation signal. It is not available for standalone trace runs. See trace metrics for the calculations, scoring guides, and metadata for each metric.

Session evaluation

A session groups traces from one agent lifecycle using a stable session_id. Session evaluation helps identify repeated responses, isolated high-risk steps, and accumulated uncertainty across that lifecycle.
1

Collect the session's traces

PandaProbe loads traces with the same session ID in execution order.
2

Prepare trace-level signals

The worker reuses available scores or computes confidence, loop_detection, tool_correctness, and coherence. Newly computed signals are stored as trace scores, so you can inspect the evidence behind the session result.
3

Aggregate session scores

agent_reliability emphasizes the riskiest traces. agent_consistency combines confidence risk with penalties from the other signals. Aggregation uses deterministic math with no additional LLM or embedding calls.
Both session metrics use the same default signal weights: You can customize these weights in the dashboard or pass signal_weights through the API or CLI. See session metrics for the formulas and missing-signal behavior.

Read the result

Evaluation runs execute asynchronously. Each selected metric produces a score for each target, with a value, reason, and supporting metadata. Built-in scores range from 0 to 1; higher values indicate better results on the measured dimension. Score status describes whether computation succeeded. A SUCCESS status can accompany a low value. Read the value and reason together, and use metadata to inspect extracted evidence, individual verdicts, or per-trace signals.
Some metrics return 1.0 when there is nothing to evaluate, such as a missing plan or no evaluable signals. Check the reason and metadata before treating a high score as evidence of successful behavior.

Next steps

Metrics

Compare all trace and session metrics and inspect their calculations.

Programmatic Scoring

Attach your own checks or feedback to a trace.

Set Up Evaluation

Run evaluations through the dashboard, API, CLI, or monitors.