Skip to main content
Evaluation monitors save a target type, metrics, filters, sampling rate, and cadence, then create eval runs on that schedule. Use them for daily trace checks, weekly session reviews, or ongoing production monitoring.

Dashboard setup

Create monitors from the Evaluations tab in the PandaProbe dashboard.
1

Open Evaluations

Open the Evaluations tab from the dashboard navigation.
2

Open Monitors

Select the Monitors card from the Evaluations landing page.
3

Click Create monitor

Click Create monitor to open the monitor sidebar.
4

Configure the monitor

Add a name, choose the target type (TRACE or SESSION), select metrics, and add filters that define the traces or sessions the monitor should evaluate.
5

Set the cadence

Choose how often the monitor should create a new eval run. Cadence controls the recurring schedule.
6

Submit

Click Create monitor. The monitor starts in the background and creates eval runs on its configured schedule.

Monitor fields

When creating a monitor from the dashboard, configure:
  • Name: a human-readable label for the monitor.
  • Target type: TRACE for trace evaluation or SESSION for session evaluation.
  • Metrics: the trace-level or session-level metrics to run.
  • Filters: the matching traces or sessions to evaluate.
  • Sampling rate: the portion of matching data to evaluate on each run.
  • Cadence: how often PandaProbe creates a new eval run.
  • Model: optional model selection for LLM-as-judge metrics.
  • Customize signal weights: optional for session monitors.
  • Skip run if no new data since last run: enables only_if_changed (on by default).

Filters

Trace monitors support Started after, Started before, Trace status, Trace name contains, Session ID, User ID, and Tags.

Sampling rate

Sampling rate controls what portion of matching data is evaluated each time the monitor runs. For example:
  • 1.0 evaluates all matching traces or sessions.
  • 0.5 evaluates 50% of matching traces or sessions.
  • 0.1 evaluates 10% of matching traces or sessions.
Use sampling to control evaluation cost and volume for large projects.

API setup

Use the API authentication setup. Paths below are relative to https://api.pandaprobe.com or your self-hosted API endpoint.

Create a monitor

Request fields

Session monitor example

CLI setup

After installing and authenticating the CLI, create a monitor with the same metrics and filters:
For a session monitor, use --target session with session metrics such as agent_reliability. The CLI also supports --signal-weights for session monitors and --only-if-changed=false to disable the new-data check.
See the CLI monitor reference for update and delete commands.

Cadence options

Monitors support predefined intervals and custom cron expressions. Predefined intervals schedule the next run relative to creation or the previous scheduling event. Use a cron expression when you need a fixed time of day. Cron schedules use UTC.

The only_if_changed flag

When only_if_changed is true (the default), PandaProbe checks for traces whose started_at is later than the monitor’s previous run time. For session monitors, those traces must have a session ID. The check also applies the configured date, user, and tag filters. This decides whether to run; it does not restrict evaluation to new data. Once triggered, the run evaluates the monitor’s full filtered and sampled target set. Changes to older traces alone do not satisfy the new-data check. Set it to false when you want the monitor to run on every cadence tick, even if the underlying data has not changed.

Manage monitors

Monitors have two states: Common API operations:
Use trigger to create an immediate eval run from a monitor without waiting for the next scheduled cadence.

Next steps

Evaluation via UI

Create one-off trace and session eval runs from the dashboard.

Evaluation via API

Create and manage eval runs programmatically.