CLI

Use this page to look up commands and flags. For a guided first run, start with the Quickstart.

uv run aec-bench --version
uv run aec-bench --help

Global flags --json and --text force machine-readable or human-readable output where commands support both.

Setup

`init`

Scaffold a project directory:

uv run aec-bench init my-bench
uv run aec-bench init --update-skills

--update-skills refreshes the packaged skills in .claude/skills/ and .agents/skills/ while preserving the rest of an existing project.

`config`

Manage user defaults:

uv run aec-bench config view
uv run aec-bench config set tasks-root tasks
uv run aec-bench config set ledger-root artefacts/ledger
uv run aec-bench config reset

Generation

`generate list-templates`

List built-in and user-supplied templates:

uv run aec-bench generate list-templates
uv run aec-bench generate list-templates --discipline mechanical

`generate task`

Generate concrete task instances from a built-in or local template:

uv run aec-bench generate task voltage-drop \
  --instances 5 \
  --difficulty easy,medium \
  --seed 42 \
  --lifecycle proposed \
  --visibility public \
  --output tasks/generated

Useful options:

Flag	Purpose
`--template PATH`	Generate from the template in a local directory
`--instances N`	Number of instances to create
`--difficulty easy,medium`	Comma-separated difficulty filter
`--seed N`	Reproducible sampling seed
`--start-index N`	First stable instance index
`--tool-mode MODE`	Override the template's tool mode
`--lifecycle proposed	active
`--visibility public	private
`--dry-run`	Preview the task generation plan

`generate validate-template`

Validate a template directory:

uv run aec-bench generate validate-template \
  src/aec_bench/templates/builtin/electrical/voltage_drop

`generate suite`

Generate a suite from suite.toml:

uv run aec-bench generate suite --config suite.toml --dry-run
uv run aec-bench generate suite --config suite.toml --validate-only

`generate dockerfiles`

Generate Dockerfiles for tasks that declare container extensions:

uv run aec-bench generate dockerfiles tasks --dry-run
uv run aec-bench generate dockerfiles tasks

Tasks and Datasets

`task validate`

Check a task directory:

uv run aec-bench task validate tasks/electrical/voltage-drop

Validation confirms the task's identity and whether it can run under its declared lifecycle and visibility policy. The result also identifies the verifier entry point and protocol version; missing identity or policy values are errors.

`task explain`

Resolve a canonical task key, alias, or UUID and inspect its current contract:

uv run aec-bench task explain electrical/voltage-drop

Use the result to check which task a reference resolves to and what output its verifier expects. It includes the canonical identity and aliases alongside the task's policy and verifier details.

`task genome` and task structure analysis

These research commands describe the structure and capabilities of tasks. Extract one task description, scan a task or template tree, or derive a consistent template breakdown:

uv run aec-bench task genome <task-path>
uv run aec-bench task genome-batch tasks --output-dir task_genomes
uv run aec-bench task genome-template-batch src/aec_bench/templates/builtin
uv run aec-bench task decomposition-template-batch task_genomes/templates

Single-task and task-tree extraction support heuristic, evidence, and LLM modes. Template extraction is heuristic.

Evidence lifecycles and Interactive Worlds

Inspect finite evidence lifecycles and registered Interactive Worlds:

uv run aec-bench task lifecycle list
uv run aec-bench task lifecycle list-variants <template-id>
uv run aec-bench task lifecycle materialize <template-id> --output <package-dir>
uv run aec-bench task world list
uv run aec-bench task world find seepage
uv run aec-bench task world show dam-seepage-monitoring
uv run aec-bench task world profiles dam-seepage-monitoring

The complete lifecycle command hierarchy is:

aec-bench task lifecycle list
aec-bench task lifecycle list-variants
aec-bench task lifecycle materialize
aec-bench task lifecycle start
aec-bench task lifecycle submit
aec-bench task lifecycle status
aec-bench task lifecycle revisit
aec-bench task lifecycle branch
aec-bench task lifecycle run
aec-bench task lifecycle verify
aec-bench task lifecycle run-smoke
aec-bench task lifecycle study ablation
aec-bench task lifecycle study calibration-freeze

Choose a lifecycle command for the next operation:

Command	Action
`start`	Release the next checkpoint
`submit`	Archive the active checkpoint submission
`status`	Read current state
`revisit`	Read an accepted checkpoint while preserving progression
`branch`	Create a separate run from a submitted checkpoint, inheriting the accepted prefix
`run`	Resume from current state and follow the accepted history

The Interactive World hierarchy is:

aec-bench task world list
aec-bench task world find
aec-bench task world show
aec-bench task world profiles
aec-bench task world run
aec-bench task world pump-station actor-interface
aec-bench task world pump-station control-interface
aec-bench task world pump-station verify
aec-bench task world pump-station branch
aec-bench task world pump-station evaluate
aec-bench task world pump-station export-harbor
aec-bench task world pump-station run-harbor
aec-bench task world pump-station import-harbor-trial

task world run creates one provider-neutral WorldTask, plans its repetitions, validates the world and provider route, and then runs the complete trials. Use --dry-run to validate and inspect the plan:

uv run aec-bench task world run dam-seepage-monitoring \
  --profile synthetic-rising-seepage \
  --instruction "Monitor the dam and respond as conditions evolve." \
  --model <provider/model> \
  --adapter prime-agent \
  --repetitions 3 \
  --dry-run

Use the pump-station subcommands to run an actor or apply host controls, then verify and evaluate the resulting state. The branch commands let you create a child run and compare its outcome before selecting or merging it.

See Finite Lifecycles and Interactive Worlds for their separate execution contracts.

`dataset`

Create a semantic selection, publish one exact reference, and use its immutable label:

uv run aec-bench dataset create electrical-core \
  --domain electrical \
  --description "Cable and power-system baseline"

uv run aec-bench dataset publish electrical-core --label public-2026

uv run aec-bench dataset config electrical-core@public-2026 \
  --model gpt-4.1-mini \
  --output experiment.yaml

uv run aec-bench dataset list
uv run aec-bench dataset info electrical-core@public-2026
uv run aec-bench dataset validate electrical-core@public-2026
uv run aec-bench dataset export electrical-core@public-2026 --output electrical-core.tar.gz
uv run aec-bench dataset import electrical-core.tar.gz
uv run aec-bench dataset results electrical-core@public-2026

`evidence`

Rebuild the disposable metadata index and verify portable evidence:

uv run aec-bench evidence index rebuild \
  --ledger-root artefacts/ledger \
  --database artefacts/ledger/evidence-index.sqlite
uv run aec-bench evidence verify --ledger-root artefacts/ledger
uv run aec-bench evidence verify --run <run-id> --ledger-root artefacts/ledger

The index command reports indexed, unreadable, and conflicting records. Use the Python EvidenceQuery API for bounded metadata filters and cursor pages. The verification command checks structured records and referenced artefact bytes.

Running Experiments

Persisted run plans

Create a resolved specification and ready plan for identity-bearing artefact tasks:

uv run aec-bench run plan \
  --config experiment.yaml \
  --tasks-root tasks \
  --store-root artefacts/runs

The command assigns a new run UUIDv7, resolves exact task releases and agent conditions, expands every task-agent-repetition combination, and persists the complete ordered plan.

Inspect a saved run by readable key or UUID:

uv run aec-bench run inspect <run-key-or-uuid> \
  --store-root artefacts/runs

The inspection output includes the run identity, resolved condition, plan identity, trial count, plan summary, and readiness state.

Compare two persisted runs:

uv run aec-bench run diff <left-run> <right-run> \
  --store-root artefacts/runs

run diff reports semantic field changes across the resolved conditions.

Reconcile typed terminal outcomes with the exact plan:

uv run aec-bench run reconcile <run-key-or-uuid> \
  --observations outcomes.json \
  --store-root artefacts/runs

The result accounts for each planned trial and reports whether the run is complete and valid. Use its accepted trial identities for reporting and inspect quarantined identities separately. See RunAccounting for outcome and membership details.

Control a ready persisted local artefact plan through the scheduler:

uv run aec-bench run start <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks
uv run aec-bench run resume <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks
uv run aec-bench run status <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs
uv run aec-bench run cancel <run-id> \
  --operational-store artefacts/operational.sqlite

run start and run resume execute persisted local artefact plans. run status reads the shared progress projection, and run cancel requests cancellation for queued and active work. Open the same run-progress projection in the TUI with explicit roots:

uv run aec-bench tui \
  --run-id <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

Start the web surface with explicit operational and plan roots:

uv run aec-bench web \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

The endpoint is GET /api/runs/{run_id}/status.

See Runs and Plans for the identity and accounting model.

`run`

Run an experiment manifest:

uv run aec-bench run --config experiment.yaml --tasks-root tasks

The dataset manifest can contain task_kind: "artifact" and task_kind: "world" entries. run loads each task through its declared kind, groups execution by runtime family, and returns results in the planned order. The runner validates the world and provider combination before execution.

For a config-backed run, the manifest selects the model, harness, backend, and repetitions. The common overrides below apply to both manifest and inline runs.

For an inline run, supply a task path and model:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct \
  --backend docker \
  --repetitions 3

Flag	Purpose
`--tasks-root PATH`	Resolve task selections against this root
`--dry-run`	Validate and display the run plan
`--no-verify`	Skip task verification
`--stream`, `--no-stream`	Override the manifest's live sandbox setting
`--reviewer-model MODEL`	Enable post-verifier LLM review with one model
`--reviewer-models-config PATH`	Enable review using configured endpoints

The CLI also accepts --adapter as an alias for --harness.

For public artefact tasks on Docker or Daytona, --stream enables the Harbor viewer's live Stream. It overrides compute.stream in the manifest; --no-stream switches it off for that run.

Move a completed run as one deterministic package:

uv run aec-bench run export run-001 --output run-001.tar.zst
uv run aec-bench run import run-001.tar.zst

The .tar.zst suffix means a tar archive compressed with Zstandard. Export produces deterministic package bytes. Import validates the package before it writes accepted run records.

`run-local`

Run one task through the local runner:

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct

Useful options:

Flag	Purpose
`--output DIR`	Save local run output in a specific directory
`--timeout SECONDS`	Set the whole local run timeout
`--max-tokens N`	Set the maximum output tokens for harnesses that support the limit
`--best-of K`	Run `K` independent candidate attempts inside one trial
`--selector self`	Select the first candidate that completed with a primary output
`--keep-workspace`	Preserve the temporary workspace for debugging
`--no-verify`	Skip verifier execution after the agent run
`--no-import`	Keep the result in local run output only
`--no-normalise`	Skip canonical-reference normalisation before verification
`--reviewer`	Run the optional post-verifier LLM reviewer stage
`--reviewer-model MODEL`	Enable the optional post-verifier LLM reviewer with one model
`--reviewer-models-config PATH`	Enable the reviewer with JSON or YAML endpoint configuration
`--fail-on-reviewer-error`	Fail the run if the reviewer stage cannot complete

Best-of candidates run in isolated workspaces within one planned trial. The built-in self selector chooses the first candidate that completed with a primary output, then the official verifier scores that output once.

Use experiment repetitions to create separate trials. For multiple drafts of a report section within one attempt, configure Lambda-RLM section candidates. The report task example shows how to run either report harness.

`import`, `import-local`, and `import-prime-eval`

Import completed results into the ledger:

uv run aec-bench import jobs/exp-001
uv run aec-bench import-local tasks/electrical/voltage-drop/_local_runs/run-001
uv run aec-bench import-prime-eval <prime-eval-id> --experiment prime-eval-001

import-prime-eval reads Prime hosted eval samples and writes normal aec-bench ledger artefacts, so hosted rollouts can be inspected with the same trace and behavioural report commands as local runs.

Analysis and Review

`evaluation regrade`

Run a revised verifier against a completed Harbor trial:

uv run aec-bench evaluation regrade jobs/<job>/<trial> \
  --task revised-tasks/voltage-drop \
  --output runs/regrades/voltage-drop-check \
  --backend docker

Use --dry-run to validate the plan. The output directory must be new. See Regrading for the retained-input and verifier setup.

`evaluate`

Summarise an experiment's ledger records:

uv run aec-bench evaluate --experiment exp-001
uv run aec-bench evaluate --experiment exp-001 --report report.html
uv run aec-bench evaluate --experiment exp-001 --model gpt-4.1-mini

`report`

Generate focused outputs from ledger data:

uv run aec-bench report summary --experiment-id exp-001
uv run aec-bench report leaderboard
uv run aec-bench report traces --experiment-id exp-001
uv run aec-bench report behavioral --experiment-id exp-001 --classifier claude-sonnet-4-20250514

report summary includes model usage and cost. Use the global --json flag for the per-model rows and cost-completeness fields.

`ledger`

Query and export trial records:

uv run aec-bench ledger list
uv run aec-bench ledger export --output trials.jsonl

`remediate`

Run verifier-driven remediation on a completed run:

uv run aec-bench remediate --help

Use remediation when the verifier can point to a concrete gap and the original task artefacts are still available.

`meta-harness`

Compare harness candidates or diagnose a benchmark problem before proposing a controlled repair:

uv run aec-bench meta-harness logic-evaluate --world world.json --run task-run.json
uv run aec-bench meta-harness review --world world.json --run task-run.json
uv run aec-bench meta-harness operation-orchestrate --brief brief.json --world world.json --emit-request
uv run aec-bench meta-harness harbor-task --brief brief.json --world world.json --plan operation-plan.json --output operation-task
uv run aec-bench meta-harness process "Create a diagnostic task." --brief brief.json --world world.json
uv run aec-bench meta-harness autonomous --task-text "Create a diagnostic task." --brief brief.json --world world.json --max-iterations 3

recipe, compare, and example use the same Python study API described in Meta-Harness Composition.

The diagnostic process can pause when a brief, problem model, task run, operation plan, or approval decision is missing. These pauses use named awaiting_* states. The existing --world flags refer to problem-model JSON files. Registered Interactive Worlds use the task world commands.

Use --emit-run-plan with intake-models, world-models, review-models, or operation-models to validate endpoint configuration before calling a provider.

The provider-free comparison surface includes:

uv run aec-bench meta-harness recipe \
  --task-text "Compare a candidate harness with the baseline." \
  --output comparison-recipe
uv run aec-bench meta-harness compare --help
uv run aec-bench meta-harness example --output comparison-example

Use aec-bench task lifecycle for lifecycle execution and lifecycle-specific studies. The meta-harness remains runtime-neutral and can compare lifecycle candidates through returned TrialRecord evidence.

See Meta-Harness Composition for the Python API, comparison workflow, and problem-model governance boundaries.

Interactive Surfaces

uv sync --extra tui --extra webui --dev
uv run aec-bench tui
uv run aec-bench web

The TUI is for local browsing, triage, comparison, and review. The web UI exposes the same ledger and review concepts through the FastAPI/Svelte surface in the library repo. Use uv sync --extra webui --dev in the source checkout before launching the browser UI.

Discovery

Search templates and seed tasks:

uv run aec-bench search "voltage drop"
uv run aec-bench search "cable" --discipline electrical

Library Catalogue

Export the public task and template catalogue consumed by the website:

uv run aec-bench library export --pretty
uv run aec-bench library export --stdout --pretty

See Library Catalogue for schema and sync details.

Evolution

Single-workspace evolution:

uv run aec-bench evolve init workspaces/my-ws --name voltage-drop-evo --harness rlm
uv run aec-bench evolve run --config evolution.yaml --tasks-root tasks
uv run aec-bench evolve history workspaces/my-ws
uv run aec-bench evolve rollback workspaces/my-ws <candidate-id>

rollback accepts a stable candidate_id or an immutable label. Candidate lineage uses parent_candidate_id, and source_revision is the full Git revision of the candidate source.

Multi-agent QD swarm evolution:

uv run aec-bench swarm run swarm.yaml
uv run aec-bench swarm status <run-id> --state-dir workspaces/my-ws/_swarm_runs
uv run aec-bench swarm history --state-dir workspaces/my-ws/_swarm_runs

status validates the explicit persisted swarm state and exact candidate material. Keep swarm run attached for process supervision.

See Evolution, Agentic Variation, and Swarm.

Prime Lab

Export AEC-Bench tasks as Prime/verifiers environments and run local or hosted evals:

uv run aec-bench prime doctor
uv run aec-bench prime adapters
uv run aec-bench prime export --name aec-smoke --task electrical/voltage-drop
uv run aec-bench prime export-lifecycle --name lifecycle-smoke --package artefacts/lifecycles/example
uv run aec-bench prime push --name aec-smoke --task electrical/voltage-drop
uv run aec-bench prime eval --remote-env owner/aec-smoke --hosted --model "Qwen/Qwen3.5-4B"
uv run aec-bench prime eval --remote-env owner/aec-smoke --hosted --model "Qwen/Qwen3.5-4B" --adapter-from-run <training-run-id>
uv run aec-bench import-prime-eval <prime-eval-id> --experiment prime-smoke-import
uv run aec-bench prime train-config --environment owner/aec-smoke -o train.toml
uv run aec-bench prime train-config \
  --environment owner/aec-suite \
  -o filtered.toml \
  --difficulty-ratio easy=0.45 \
  --difficulty-ratio medium=0.40 \
  --difficulty-ratio hard=0.15 \
  --online-difficulty-filtering
uv run aec-bench prime train train.toml
uv run aec-bench prime smoke --name aec-smoke --task electrical/voltage-drop

Follow the Prime Lab guide to export tasks, run an evaluation, and inspect its results before training or scaling up.

Exit Codes

The CLI uses conventional exit codes:

Code	Meaning
`0`	Success
`1`	User error, missing file, validation failure, or runtime failure depending on command
`2+`	Typer/Click argument parsing and command invocation failures

When scripting, prefer --json where supported. Branch on the returned status, data, and errors fields for detailed control, and use the numeric process code for the overall outcome.

On this page