Installation

System Requirements

Python: 3.13 or later
Package installer: pip for the published package; uv for source development
OS: macOS, Linux, or Windows through WSL

Docker, Modal, Morph Cloud, Prime, and Harbor are optional. Install or configure the tool that matches your selected execution or training path.

The base installation supports provider-free task discovery, generation, validation, datasets, recorded trials, deterministic evaluation, reports, and Interactive World state operations. Add an optional agent extra when you want to call a model provider.

pip install aec-bench
aec-bench --help

Optional commands remain visible in --help. When a command needs an optional extra, it reports the required extra.

Optional Extras

Install only the features you use:

Feature	Install command
Harbor execution and import	`pip install "aec-bench[execution]"`
Harbor on Morph Cloud	`pip install "aec-bench[execution,morph]"`
DeepSeek Harness adapter	`pip install "aec-bench[deepseek-harness]"`
Local model-backed agent harnesses and reviewers	`pip install "aec-bench[local-agents]"`
Prime CLI, export, and hosted evaluation	`pip install "aec-bench[prime]"`
Prime Agent interactive worlds over ACP	`pip install "aec-bench[prime-agent]"`
Web UI	`pip install "aec-bench[webui]"`
Terminal UI	`pip install "aec-bench[tui]"`
Evolution with local model execution	`pip install "aec-bench[evolution,local-agents]"`

The prime-agent extra installs ACP support. Install the upstream Prime Agent executable separately and keep prime-agent on PATH.

From Source

Use the source checkout when contributing templates, tasks, documentation, or library code:

git clone https://github.com/TheodoreGalanos/aec-bench.git
cd aec-bench
uv sync --all-extras --dev
uv run aec-bench --help

The console entry point is aec-bench. In a source checkout, run it through uv run so Python, dependencies, and local package code stay aligned.

Provider Credentials

Set credentials for the provider used by your model or endpoint alias:

Provider	Environment variables
Anthropic	`ANTHROPIC_API_KEY`
OpenAI	`OPENAI_API_KEY`
Azure OpenAI or Azure AI Foundry v1	`AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`; optional `AZURE_OPENAI_API_VERSION`
Together AI	`TOGETHER_API_KEY`
DeepSeek API or compatible gateway	`DEEPSEEK_API_KEY`; optional `DEEPSEEK_BASE_URL`
AWS Bedrock	`AWS_BEARER_TOKEN_BEDROCK`, `AWS_REGION` or `AWS_DEFAULT_REGION`
Morph Cloud	`MORPH_API_KEY`
Prime hosted eval/training	authenticated `prime` CLI session

Set credentials in your shell. In a source checkout, aec-bench also loads the repository .env at startup. Existing shell variables take precedence.

For OpenAI, use an explicit openai: model prefix, such as openai:gpt-4.1-mini. For Azure AI Foundry deployments that expose the v1 OpenAI-compatible API, use the /openai/v1/ endpoint and pass the deployment name as --model. For Together AI, use an explicit together: model prefix.

Verify Installation

Use CLI help and a non-provider command first:

aec-bench --help
aec-bench generate list-templates --discipline ground
aec-bench library export --stdout --pretty

These commands are provider-free. Follow the Quickstart when you are ready to run a local model-backed trial.

Project Layout

The source checkout uses a src/ Python layout:

src/aec_bench/          # Library source
tasks/                  # Benchmark task seeds and generated instances
seeds/                  # Expert-created seed files
agents/                 # Ready-to-use agent implementations
artefacts/              # Local generated artefacts and catalogue exports
docs/                   # Architecture and library guides
workspaces/             # Evolution workspaces

Generated artefacts, local runs, Prime packages, and evolution swarm state are intentionally local outputs. Commit curated tasks, templates, source, tests, and docs. Keep transient run artefacts local.

Next Steps

Quickstart — Generate and run a first task
Templates — Understand the built-in template catalogue
CLI Reference — See the current command surface
Prime Lab — Export tasks for Prime eval and training