Quickstart

This guide takes you from a fresh installation to an evaluated trial: generate and validate an engineering task, run a model-backed agent from your machine, and inspect the recorded result. Use the source checkout when you are changing aec-bench itself.

Prerequisites

• Python 3.13 or later
• An LLM provider key for model-backed runs; generation and validation are provider-free

Install

pip install "aec-bench[local-agents]"
aec-bench --help

The base package is provider-free. The local-agents extra adds the model-backed harnesses used in this walkthrough. Contributors who are changing aec-bench itself should use the source installation and run commands through uv run.

Generate

List the built-in templates and generate one deterministic task instance:

aec-bench generate list-templates --discipline ground

aec-bench generate task terzaghi-bearing-capacity \
  --instances 1 \
  --difficulty easy \
  --seed 42 \
  --start-index 0 \
  --lifecycle proposed \
  --visibility public \
  --output tasks/generated

Generated instances are normal task directories with an instruction, task config, and verifier files. The output root also contains generation-config.json and generation-manifest.json for deterministic replay.

Validate

Before treating a task as benchmark material, validate its structure and verifier:

aec-bench task validate \
  tasks/generated/ground/shallow-foundations/terzaghi-bearing-capacity/perth-coastal-medium-dense-sand-00

In a source checkout, also validate the source template:

uv run aec-bench generate validate-template \
  src/aec_bench/templates/builtin/ground/terzaghi_bearing_capacity

Run

Run one task locally with a real model. The local runner copies the task into a temporary workspace, executes the selected harness, verifies the result, and imports the trial by default.

export OPENAI_API_KEY="..."

aec-bench run-local \
  tasks/generated/ground/shallow-foundations/terzaghi-bearing-capacity/perth-coastal-medium-dense-sand-00 \
  --model openai:gpt-4.1-mini \
  --harness tool_loop

Use --harness rlm for a recursive-language-model harness, or --keep-workspace when debugging the temporary workspace after a run.

Evaluate

By default, the local runner imports the trial under the local experiment id:

aec-bench evaluate --experiment local
aec-bench evaluate --experiment local --report report.html

For a repeatable benchmark slice, publish the selected tasks as a dataset and run the resulting experiment through a configured backend. See Datasets and Backends.