Skip to main content
Evaluation endpoints live under /evaluations. The paths below are relative to https://api.pandaprobe.com for PandaProbe Cloud, or your self-hosted API endpoint.
For server-side API access, send your API key in X-API-Key and project name in X-Project-Name. The dashboard uses bearer authentication with the active project context.
For example, list the available trace metrics using credentials from your environment:
Send Content-Type: application/json when posting the request bodies shown below. See the HTTP API reference for generated endpoint schemas.

Discover available metrics

Before creating eval runs, check which metrics are available.

List trace metrics

Returns metrics available for standalone trace runs. loop_detection is excluded because it requires session context. Example entries:

List session metrics

Returns all registered session-level metrics:

Check LLM provider availability

Returns which LLM providers are configured and available for judge calls.

Trace evaluation runs

Create a filtered trace eval run

Resolves traces matching your filters, samples them, and dispatches background evaluation. Returns 202 Accepted immediately. Request body:
Response (202):
Rate limit: 50/min

Create a batch trace eval run

Evaluate specific traces by ID instead of using filters.
Rate limit: 50/min

Poll eval run status

Check the progress of an eval run. Poll this endpoint until status is COMPLETED or FAILED, then inspect failed_count and the individual scores. COMPLETED can include failed computations; a score’s SUCCESS status describes computation, not whether its value meets your quality bar.

List eval runs

Get scores for a run

Returns all trace scores produced by a specific eval run.

Retry failed metrics

Creates a new eval run targeting only the trace+metric pairs that failed in the original run. Returns 422 if the original run has no failures. Rate limit: 50/min

Delete an eval run

By default, only the run record is deleted — scores are preserved with eval_run_id set to null. Pass ?delete_scores=true to also delete all scores from this run.

Trace scores

Create a manual score

Manually attach a score to a trace (human annotation or programmatic submission).
NUMERIC scores must be in the range [0.0, 1.0]. BOOLEAN scores must be "true" or "false".

List trace scores

Comprehensive filtering:

Get latest scores for a trace

Returns one score per metric name, deduplicated by most recent created_at. Use this to display a score overview panel for a specific trace.

Update a score

Only value, reason, and metadata can be changed. status is automatically set to SUCCESS and source to ANNOTATION.

Delete a score


Session evaluation runs

Session eval runs follow the same pattern as trace eval runs but target sessions instead of traces.

Create a filtered session eval run

Rate limit: 50/min

Create a batch session eval run

Rate limit: 50/min

Other session run endpoints

Session scores


Analytics

PandaProbe provides analytics endpoints for both trace and session scores.

Trace score analytics

Summary — aggregated stats per metric:
Trend — time series of average scores:
Distribution — histogram of score values:

Session score analytics

Session score analytics mirror the trace analytics: Session score history — track how a session’s score evolves over re-evaluations:
Session comparison — rank sessions by a metric (useful for finding worst-performing sessions):
Pass sort_order=asc to surface the worst sessions first.

Get an eval run template

To help build eval run requests, PandaProbe can generate a pre-filled template for a metric:
Returns the metric’s full info (including prompt previews), default filters, sampling rate, and the default model. Use this to populate a form in your own tooling.

Error handling

Next steps

Scheduling

Set up automated recurring evaluations with monitors.

Metrics

Calculations and metadata for trace and session metrics.