CLI

Use this page to look up commands and flags. For a guided first run, start with the Quickstart.

uv run aec-bench --version
uv run aec-bench --help

Global flags --json and --text force machine-readable or human-readable output where commands support both.

Setup

`init`

Scaffold a project directory:

uv run aec-bench init my-bench
uv run aec-bench init --update-skills

--update-skills refreshes the packaged skills in .claude/skills/ and .agents/skills/ while preserving the rest of an existing project.

`config`

Manage user defaults:

uv run aec-bench config view
uv run aec-bench config set tasks-root tasks
uv run aec-bench config set ledger-root artefacts/ledger
uv run aec-bench config reset

Generation

`generate list-templates`

List built-in and user-supplied templates:

uv run aec-bench generate list-templates
uv run aec-bench generate list-templates --discipline mechanical

`generate task`

Generate concrete task instances from a built-in or local template:

uv run aec-bench generate task voltage-drop \
  --instances 5 \
  --difficulty easy,medium \
  --seed 42 \
  --lifecycle proposed \
  --visibility public \
  --output tasks/generated

Useful options:

Flag	Purpose
`--template PATH`	Generate from the template in a local directory
`--instances N`	Number of instances to create
`--difficulty easy,medium`	Comma-separated difficulty filter
`--seed N`	Reproducible sampling seed
`--start-index N`	First stable instance index
`--tool-mode MODE`	Override the template's tool mode
`--lifecycle proposed	active
`--visibility public	private
`--dry-run`	Show the plan without writing task instances

`generate validate-template`

Validate a template directory:

uv run aec-bench generate validate-template \
  src/aec_bench/templates/builtin/electrical/voltage_drop

`generate suite`

Generate a suite from suite.toml:

uv run aec-bench generate suite --config suite.toml --dry-run
uv run aec-bench generate suite --config suite.toml --validate-only

`generate dockerfiles`

Generate Dockerfiles for tasks that declare container extensions:

uv run aec-bench generate dockerfiles tasks --dry-run
uv run aec-bench generate dockerfiles tasks

Tasks and Datasets

`task validate`

Check a task directory:

uv run aec-bench task validate tasks/electrical/voltage-drop

Validation reports the task's readable display reference, UUID, version, lifecycle, visibility, runnable state, verifier entry point, and verifier protocol version. Missing identity or policy values are errors.

`task explain`

Resolve a canonical task key, alias, or UUID and inspect its current contract:

uv run aec-bench task explain electrical/voltage-drop

The result includes the canonical key, aliases, UUID, version, lifecycle, visibility, runnable state, output contract, verifier entry point, and verifier protocol version.

`task genome` and task structure analysis

These research commands describe the structure and capabilities of tasks. Extract one task description, scan a task or template tree, or derive a consistent template breakdown:

uv run aec-bench task genome <task-path>
uv run aec-bench task genome-batch tasks --output-dir task_genomes
uv run aec-bench task genome-template-batch src/aec_bench/templates/builtin
uv run aec-bench task decomposition-template-batch task_genomes/templates

Single-task and task-tree extraction support heuristic, evidence, and LLM modes. Template extraction is heuristic.

Evidence lifecycles and Interactive Worlds

Inspect finite evidence lifecycles and registered Interactive Worlds:

uv run aec-bench task lifecycle list
uv run aec-bench task lifecycle list-variants <template-id>
uv run aec-bench task lifecycle materialize <template-id> --output <package-dir>
uv run aec-bench task world list
uv run aec-bench task world find seepage
uv run aec-bench task world show dam-seepage-monitoring
uv run aec-bench task world profiles dam-seepage-monitoring

The complete lifecycle command hierarchy is:

aec-bench task lifecycle list
aec-bench task lifecycle list-variants
aec-bench task lifecycle materialize
aec-bench task lifecycle start
aec-bench task lifecycle submit
aec-bench task lifecycle status
aec-bench task lifecycle revisit
aec-bench task lifecycle branch
aec-bench task lifecycle run
aec-bench task lifecycle verify
aec-bench task lifecycle run-smoke
aec-bench task lifecycle study ablation
aec-bench task lifecycle study calibration-freeze

start releases the next checkpoint. submit archives the active checkpoint submission. status reads state, and revisit reads an accepted checkpoint while preserving progression. branch creates a separate run from a submitted checkpoint and inherits the accepted prefix. run resumes from the current lifecycle state and always follows that accepted history.

The Interactive World hierarchy is:

aec-bench task world list
aec-bench task world find
aec-bench task world show
aec-bench task world profiles
aec-bench task world run
aec-bench task world pump-station actor-interface
aec-bench task world pump-station control-interface
aec-bench task world pump-station verify
aec-bench task world pump-station branch
aec-bench task world pump-station evaluate
aec-bench task world pump-station export-harbor
aec-bench task world pump-station run-harbor
aec-bench task world pump-station import-harbor-trial

task world run creates one provider-neutral WorldTask, plans its repetitions, validates the world and provider route, and then runs the complete trials. Use --dry-run to validate and inspect the plan without calling a provider:

uv run aec-bench task world run dam-seepage-monitoring \
  --profile synthetic-rising-seepage \
  --instruction "Monitor the dam and respond as conditions evolve." \
  --model <provider/model> \
  --adapter prime-agent \
  --repetitions 3 \
  --dry-run

The pump-station subcommands expose task-owned actor, control, verification, branching, evaluation, and Harbor application functions. Branch creation materialises a child and leaves later execution, evaluation, selection, and merging to separate commands.

See Finite Lifecycles and Interactive Worlds for their separate execution contracts.

`dataset`

Create a semantic selection, publish one exact reference, and use its immutable label:

uv run aec-bench dataset create electrical-core \
  --domain electrical \
  --description "Cable and power-system baseline"

uv run aec-bench dataset publish electrical-core --label public-2026

uv run aec-bench dataset config electrical-core@public-2026 \
  --model gpt-4.1-mini \
  --output experiment.yaml

uv run aec-bench dataset list
uv run aec-bench dataset info electrical-core@public-2026
uv run aec-bench dataset validate electrical-core@public-2026
uv run aec-bench dataset export electrical-core@public-2026 --output electrical-core.tar.gz
uv run aec-bench dataset import electrical-core.tar.gz
uv run aec-bench dataset results electrical-core@public-2026

`evidence`

Rebuild the disposable metadata index and verify portable evidence:

uv run aec-bench evidence index rebuild \
  --ledger-root artefacts/ledger \
  --database artefacts/ledger/evidence-index.sqlite
uv run aec-bench evidence verify --ledger-root artefacts/ledger
uv run aec-bench evidence verify --run <run-id> --ledger-root artefacts/ledger

The index command reports indexed, unreadable, and conflicting records. Use the Python EvidenceQuery API for bounded metadata filters and cursor pages. The verification command checks structured records and referenced artefact bytes.

Running Experiments

Persisted run plans

Create a resolved specification and ready plan for identity-bearing artefact tasks:

uv run aec-bench run plan \
  --config experiment.yaml \
  --tasks-root tasks \
  --store-root artefacts/runs

The command assigns a new run UUIDv7, resolves exact task releases and agent conditions, expands every task-agent-repetition combination, and persists the complete ordered plan.

Inspect a saved run by readable key or UUID:

uv run aec-bench run inspect <run-key-or-uuid> \
  --store-root artefacts/runs

The inspection output includes the run identity, resolved condition, plan identity, trial count, plan summary, and readiness state.

Compare two persisted runs:

uv run aec-bench run diff <left-run> <right-run> \
  --store-root artefacts/runs

run diff reports semantic field changes across the resolved conditions.

Reconcile typed terminal outcomes with the exact plan:

uv run aec-bench run reconcile <run-key-or-uuid> \
  --observations outcomes.json \
  --store-root artefacts/runs

The result includes planned, succeeded, failed, cancelled, timed-out, invalid, missing, duplicate, and unexpected counts. It also returns completeness, validity, accepted trial identities, and quarantined trial identities.

Control a ready persisted local artefact plan through the scheduler:

uv run aec-bench run start <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks
uv run aec-bench run resume <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks
uv run aec-bench run status <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs
uv run aec-bench run cancel <run-id> \
  --operational-store artefacts/operational.sqlite

run start and run resume execute persisted local artefact plans. run status reads the shared progress projection, and run cancel requests cancellation for queued and active work. Open the same run-progress projection in the TUI with explicit roots:

uv run aec-bench tui \
  --run-id <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

Start the web surface with explicit operational and plan roots:

uv run aec-bench web \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

The endpoint is GET /api/runs/{run_id}/status.

See Runs and Plans for the identity and accounting model.

`run`

Run an experiment manifest:

uv run aec-bench run --config experiment.yaml --tasks-root tasks

The dataset manifest can contain task_kind: "artifact" and task_kind: "world" entries. run loads each task through its declared kind, groups execution by runtime family, and returns results in the planned order. Unsupported world and provider combinations fail before execution.

For a config-backed run, the manifest owns the model, harness, backend, and repetitions. The accepted CLI overrides are --tasks-root, --dry-run, --no-verify, and the reviewer options.

For an inline run, supply a task path and model:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct \
  --backend docker \
  --repetitions 3

The inline form also accepts --tasks-root, --dry-run, --no-verify, and reviewer options. The CLI accepts --adapter as an alias for --harness. Use --reviewer-model MODEL or --reviewer-models-config PATH to enable the optional post-verifier LLM reviewer.

Move a completed run as one deterministic package:

uv run aec-bench run export run-001 --output run-001.tar.zst
uv run aec-bench run import run-001.tar.zst

The .tar.zst suffix means a tar archive compressed with Zstandard. Export produces deterministic package bytes. Import validates the package before it writes accepted run records.

`run-local`

Run one task through the local runner:

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct

Useful options:

Flag	Purpose
`--output DIR`	Save local run output in a specific directory
`--timeout SECONDS`	Set the whole local run timeout
`--max-tokens N`	Set the maximum output tokens for harnesses that support the limit
`--best-of K`	Run `K` independent candidate attempts inside one trial
`--selector self`	Select the first candidate that completed with a primary output
`--keep-workspace`	Preserve the temporary workspace for debugging
`--no-verify`	Skip verifier execution after the agent run
`--no-import`	Do not import the local result into the ledger
`--no-normalise`	Skip canonical-reference normalisation before verification
`--constitutional-model MODEL`	Override the constitutional inference model for RLM tasks that declare one
`--reviewer`	Run the optional post-verifier LLM reviewer stage
`--reviewer-model MODEL`	Enable the optional post-verifier LLM reviewer with one model
`--reviewer-models-config PATH`	Enable the reviewer with JSON or YAML endpoint configuration
`--fail-on-reviewer-error`	Fail the run if the reviewer stage cannot complete

Best-of candidates use separate isolated workspaces but belong to one planned trial. The built-in self selector is deterministic and reward-blind: it selects the first candidate that completed with a primary output. The official task verifier runs once on the selected output. Experiment repetitions remain separate trials alongside --best-of candidate attempts.

`import`, `import-local`, and `import-prime-eval`

Import completed results into the ledger:

uv run aec-bench import jobs/exp-001
uv run aec-bench import-local tasks/electrical/voltage-drop/_local_runs/run-001
uv run aec-bench import-prime-eval <prime-eval-id> --experiment prime-eval-001

import-prime-eval reads Prime hosted eval samples and writes normal aec-bench ledger artefacts, so hosted rollouts can be inspected with the same trace and behavioural report commands as local runs.

Analysis and Review

`evaluate`

Summarise an experiment's ledger records:

uv run aec-bench evaluate --experiment exp-001
uv run aec-bench evaluate --experiment exp-001 --report report.html
uv run aec-bench evaluate --experiment exp-001 --model gpt-4.1-mini

`report`

Generate focused outputs from ledger data:

uv run aec-bench report summary --experiment-id exp-001
uv run aec-bench report leaderboard
uv run aec-bench report traces --experiment-id exp-001
uv run aec-bench report behavioral --experiment-id exp-001 --classifier claude-sonnet-4-20250514

`ledger`

Query and export trial records:

uv run aec-bench ledger list
uv run aec-bench ledger export --output trials.jsonl

`remediate`

Run verifier-driven remediation on a completed run:

uv run aec-bench remediate --help

Use remediation when the verifier can point to a concrete gap and the original task artefacts are still available.

`meta-harness`

Compare harness candidates or diagnose a benchmark problem before proposing a controlled repair:

uv run aec-bench meta-harness logic-evaluate --world world.json --run task-run.json
uv run aec-bench meta-harness review --world world.json --run task-run.json
uv run aec-bench meta-harness operation-orchestrate --brief brief.json --world world.json --emit-request
uv run aec-bench meta-harness harbor-task --brief brief.json --world world.json --plan operation-plan.json --output operation-task
uv run aec-bench meta-harness process "Create a diagnostic task." --brief brief.json --world world.json
uv run aec-bench meta-harness autonomous --task-text "Create a diagnostic task." --brief brief.json --world world.json --max-iterations 3

recipe, compare, and example use the same Python study API described in Meta-Harness Composition.

The diagnostic process can pause when a brief, problem model, task run, operation plan, or approval decision is missing. These pauses use named awaiting_* states. The existing --world flags refer to problem-model JSON files. Registered Interactive Worlds use the task world commands.

Use --emit-run-plan with intake-models, world-models, review-models, or operation-models to validate endpoint configuration before calling a provider.

The provider-free comparison surface includes:

uv run aec-bench meta-harness recipe \
  --task-text "Compare a candidate harness with the baseline." \
  --output comparison-recipe
uv run aec-bench meta-harness compare --help
uv run aec-bench meta-harness example --output comparison-example

Lifecycle execution and lifecycle-specific studies are under aec-bench task lifecycle, not meta-harness. The meta-harness remains runtime-neutral and can compare lifecycle candidates through returned TrialRecord evidence.

See Meta-Harness Composition for the Python API, comparison workflow, and problem-model governance boundaries.

Interactive Surfaces

uv sync --extra tui --extra webui --dev
uv run aec-bench tui
uv run aec-bench web

The TUI is for local browsing, triage, comparison, and review. The web UI exposes the same ledger and review concepts through the FastAPI/Svelte surface in the library repo. Use uv sync --extra webui --dev in the source checkout before launching the browser UI.

Discovery

Search templates and seed tasks:

uv run aec-bench search "voltage drop"
uv run aec-bench search "cable" --discipline electrical

Library Catalogue

Export the public task and template catalogue consumed by the website:

uv run aec-bench library export --pretty
uv run aec-bench library export --stdout --pretty

See Library Catalogue for schema and sync details.

Evolution

Single-workspace evolution:

uv run aec-bench evolve init workspaces/my-ws --name voltage-drop-evo --harness rlm
uv run aec-bench evolve run --config evolution.yaml --tasks-root tasks
uv run aec-bench evolve history workspaces/my-ws
uv run aec-bench evolve rollback workspaces/my-ws <candidate-id>

rollback accepts a stable candidate_id or an immutable label. Candidate lineage uses parent_candidate_id, and source_revision is the full Git revision of the candidate source.

Multi-agent QD swarm evolution:

uv run aec-bench swarm run swarm.yaml
uv run aec-bench swarm status <run-id> --state-dir workspaces/my-ws/_swarm_runs
uv run aec-bench swarm history --state-dir workspaces/my-ws/_swarm_runs

status validates the explicit persisted swarm state and exact candidate material. The current CLI has no stop or resume command. Keep swarm run attached for process supervision.

See Evolution, Agentic Variation, and Swarm.

Prime Lab

Export AEC-Bench tasks as Prime/verifiers environments and run local or hosted evals:

uv run aec-bench prime doctor
uv run aec-bench prime adapters
uv run aec-bench prime export --name aec-smoke --task electrical/voltage-drop
uv run aec-bench prime export-lifecycle --name lifecycle-smoke --package artefacts/lifecycles/example
uv run aec-bench prime push --name aec-smoke --task electrical/voltage-drop
uv run aec-bench prime eval --remote-env owner/aec-smoke --hosted --model "Qwen/Qwen3.5-4B"
uv run aec-bench prime eval --remote-env owner/aec-smoke --hosted --model "Qwen/Qwen3.5-4B" --adapter-from-run <training-run-id>
uv run aec-bench import-prime-eval <prime-eval-id> --experiment prime-smoke-import
uv run aec-bench prime train-config --environment owner/aec-smoke -o train.toml
uv run aec-bench prime train-config \
  --environment owner/aec-suite \
  -o filtered.toml \
  --difficulty-ratio easy=0.45 \
  --difficulty-ratio medium=0.40 \
  --difficulty-ratio hard=0.15 \
  --online-difficulty-filtering
uv run aec-bench prime train train.toml
uv run aec-bench prime smoke --name aec-smoke --task electrical/voltage-drop

See Prime Lab for task and local lifecycle export lanes, deployed adapter discovery, adapter eval, hosted training, online difficulty filtering, and hosted rollout import.

Exit Codes

The CLI uses conventional exit codes:

Code	Meaning
`0`	Success
`1`	User error, missing file, validation failure, or runtime failure depending on command
`2+`	Typer/Click argument parsing and command invocation failures

When scripting, prefer --json where supported. Branch on the returned status, data, and errors fields for detailed control, and use the numeric process code for the overall outcome.

On this page