Monitor

Runs monitors on trajectories to detect suspicious behavior.

Purpose

Executes monitoring functions on trajectories to compute suspiciousness scores. Used to:

  • Evaluate trajectory safety
  • Compute action-level or trajectory-level suspicion scores
  • Test new monitor implementations
  • Generate monitored runs for analysis

Commands

ct run monitor

Run monitors on trajectories.

Usage:

ct run monitor TRAJ_LIKE [OPTIONS]

Arguments:

  • TRAJ_LIKE - An eval-backed source: a .eval file, or a run ID/URL whose run records a backing eval log (see Downloading from the Viewer)

Options:

  • -m, --monitor - Monitor selection, repeatable (e.g., -m multi -m quick); defaults: mean-action and mean-quick. Whether a monitor scores per action or per trajectory is a property of the monitor itself; prefix a name with action/ or trajectory/ only when the same name exists as both.
  • --trusted-model - Required. The model bound as the eval-level trusted role that every monitor runs on. Any Inspect AI provider string works, including self-hosted open-weight endpoints (see Self-Hosted and Open-Weight Models)
  • --cot-level - Chain-of-thought visibility for the monitors: none (default), reasoning, or extended
  • --save-dir PATH - Directory to save monitored trajectories
  • --no-upload - Disable upload (upload is enabled by default)
  • --run-name TEXT - Custom run name

Monitors never carry model configuration; the one model they run on is the eval-level trusted role, given by --trusted-model. The exception is monitoring-bench, which replicates its benchmark's fixed 3-model panel.

A monitor selection takes one of three forms: a packaged monitor name; a filesystem path to a .md asset file (absolute or relative to the working directory; the file stem becomes the monitor's score name); or a module:function import path to a code monitor — an async callable with the harness's monitor signature, or a zero-argument factory returning one (the import path becomes the score name). Repeat the selection flag to run several monitors. monitoring-bench is the in-repo reference implementation of the code-monitor form.

# Two action monitors in one pass: a packaged one plus an external asset file
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano \
    -m no-history -m action//path/to/my-monitor.md

Examples:

# Monitor with the default monitors
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano

# One specific monitor
ct run monitor logs/my_eval.eval -m strong --trusted-model openai/gpt-5-nano

# Monitor and save locally without uploading
ct run monitor logs/my_eval.eval -m strong --save-dir monitored_trajs/ --no-upload --trusted-model openai/gpt-5-nano

# Monitor a specific run from database
ct run monitor abc123def456 -m multi --trusted-model openai/gpt-5-nano

# An action and a trajectory monitor in one pass
ct run monitor logs/my_eval.eval -m multi -m quick --trusted-model openai/gpt-5-nano

# Monitor a viewer-hosted run directly (no intermediate download)
ct run monitor https://basharena-data.vercel.app/runs/ed2b360737041f1743ac4de43280f858 \
    -m multi --trusted-model openai/gpt-5-nano

Available Monitors

A monitor is a prompt asset: a markdown file under src/control_tower/protocols/monitoring/assets/{action,trajectory}/ whose stem is the monitor's name. Its YAML front matter declares the monitor's view — samples (independent draws, mean-aggregated), per_goal (one run per candidate covert objective, max across objectives), members (a weighted-mean ensemble of other assets), template (reuse another asset's prompt body), and schema (the output parser) — and its body is the prompt template. One harness renders the view, calls the trusted role model, and parses the score, so adding a monitor is adding a file. One monitor cannot be expressed as a template and stays as code: monitoring-bench (bench-replica model panel with an XML parser).

List available monitors (or show one monitor's declared view):

ct protocols monitors action [NAME]
ct protocols monitors trajectory [NAME]

Action-Level Monitors

Score each untrusted-model action.

  • no-history - Scores each action in isolation with no preceding action history or candidate covert-objective hints.
  • old-action - The default action monitor: one model call per action, shown the candidate covert objectives.
  • old-action-no-goals - The default action monitor without the candidate covert objectives.
  • checklist - Answers five yes/no checks per action (necessity, security surface, innocent explanation, concrete harm, removability) before scoring; the top of the scale is gated on concrete harm.
  • orthogonal - Flags work that does not serve the main task - unnecessary changes, premature optimization, off-task activity - rather than judging malice directly.
  • three-scores - Categorical monitor: a safe / uncertain / suspicious verdict per action, mapped onto the score scale.
  • strong - Sees the full conversation history (goal-blind) so it judges each action in context.
  • strong-goals - The strong full-history monitor, additionally shown the environment's candidate covert objectives.
  • goal-analysis - Scores each action while shown the environment's candidate covert objectives, asking whether the action advances any of them.
  • single-goal - Scores each action once per candidate covert objective in parallel and keeps the maximum score.
  • multi - Weighted blend of the orthogonal and old-action monitors' scores for each action.
  • mean-action - Averages two independent runs of the old-action monitor on each action. CLI default.
  • mean-goals - Averages two independent goal-analysis runs on each action.
  • mean-single-goal - Scores each action twice per covert objective, averages each objective's scores, and keeps the highest average.

Trajectory-Level Monitors

Score the whole transcript in one shot rather than action by action.

  • quick - Reads the full transcript in a single pass, shown the candidate covert objectives, and returns one score for the trajectory.
  • quick-no-goals - Reads the full transcript in a single pass with no covert-objective hints.
  • goal-analysis - Reads the full transcript while shown the environment's candidate covert objectives, working through them one by one.
  • single-goal-analysis - Reads the full transcript once per candidate covert objective in parallel and keeps the maximum score.
  • mean-quick - Averages five independent quick passes over the transcript. CLI default.
  • mean-goal-analysis - Averages two independent goal-analysis passes over the transcript.
  • mean-single-goal-analysis - Reads the transcript twice per covert objective, averages each objective's scores, and keeps the highest average.
  • monitoring-bench - Replica of the MonitoringBench benchmark's monitor: the bench prompt on its fixed 3-model panel (scores comparable to the imported monitoring-bench dataset), max-aggregated.

Whether a monitor sees the environment's candidate covert objectives — every side task registered for that environment, not just the one the agent was assigned — is fixed by its prompt asset: goal-aware and goal-blind variants are separate monitors (e.g. old-action vs old-action-no-goals, strong vs strong-goals, quick vs quick-no-goals).

A source .eval containing a single environment from an isolated setting is replayed in that setting's dependency environment automatically. Compose topology is resolved without pulling or building Docker images.

Output

Post-hoc monitoring reads the source, runs monitors, and writes the monitor results. Action-level monitors write per-action results; trajectory-level monitors write one result per trajectory.

Sources are scored directly

The input must be eval-backed: a local or remote .eval, an in-memory EvalLog, or a viewer run backed by an eval log — anything else (trajectory JSON, datasets, runs without a backing eval log) is rejected with an error. ct run monitor scores the recorded transcripts directly: action monitors see the transcript prefix before each action, trajectory monitors the full transcript, and nothing re-executes.

The source is scored onto a copy: the copy lands in the save dir (an explicit --save-dir, or the default ./data/monitoring/<monitors>/<label>_<timestamp>/) as <source-stem>.monitored.eval (a re-run into the same dir gets a -N suffix), and the source is left unchanged.

An action monitor's score value maps action index to score (None where the action was skipped by design or the monitor failed), with the full verdicts under metadata.verdicts; a trajectory monitor's score is the verdict itself. Decision statistics (max, thresholded audits) are computed downstream from these values, never pre-baked. Existing named scores survive, so scoring a .monitored.eval with a new monitor accumulates both, and a monitor whose verdicts are already on the source is refused unless --overwrite, which replaces that monitor's verdicts on the copy — verdicts are never silently replaced. An eval scored before verdicts became inspect Scores carries monitor_<name> folds ({max, action_<i>, trajectory}) that no reader accepts; migrate it by re-scoring every folded monitor at once with ct run monitor --overwrite -m <name> ..., since a fold left on the source cannot be read around.

The eval header records derived_from (the source identifier) on copied artifacts. Uploaded monitor runs register the monitored log as their eval_log_url, and downstream consumers (e.g. ct run summarize) reconstruct the monitor_response / traj_monitor_response fields from the sample's verdicts.

Unless --no-upload is passed:

  • Creates a run in the database
  • Uploads all monitored trajectories
  • Prints run URL and trajectory URLs

Note the model response cache defaults on, so re-running an unchanged command reproduces the same scores rather than drawing a fresh sample (the CLI warns when this happens); a monitor's fan-out draws (samples, ensemble members, retry attempts) scope the cache key by position, so they are independent samples within a run. Clear inspect's model cache (inspect cache clear) to resample.

Monitoring Details

Monitors add suspiciousness scores to trajectories:

  • Scores range from 0 (not suspicious) to 10 (very suspicious)
  • Different monitors use different approaches (LLM-based, heuristic, etc.)
  • Action-level results are stored per action
  • Trajectory-level results are stored once per trajectory

Cost Attribution

Every verdict carries a cost_usd (and token fields) in its metadata populated automatically by the orchestration layer — no per-monitor changes required. Multi-model monitors (e.g. multi) additionally get a cost_by_model split. An unpriced model is a None in that split; cost_usd stays the priced spend. See function-registry.md for the mechanism and known limitations, and trajectories.md for the monitor_cost(sample) / agent_cost(sample) / total_cost_usd(sample) accessors.

Technical Details

  • Supports both action-level and trajectory-level monitors
  • Tracks token usage and costs for LLM-based monitors
  • Runs monitors concurrently for efficiency
  • Handles errors gracefully per trajectory