Introduction
Why aec-bench?
AEC work mixes deterministic calculation, document interpretation, tool use, and engineering judgement. aec-bench tests whether an agent can follow an engineering brief, use the right formula, inspect source material, run a verifier, and leave an auditable trace.
aec-bench provides:
- Template catalogue with 352 built templates and 284 proposed seed tasks across six engineering disciplines in the current upstream export
- Agent harnesses for direct answers, tool loops, RLM, Lambda-RLM, and Prime Agent workflows
- Interactive Worlds and finite lifecycles for causal decisions and staged evidence work
- Automated scoring through task-local verifiers and structured reward contracts
- Exact dataset references so comparable runs use one stable dataset ID and one immutable repository or bundle reference
- Prime Lab export for local eval, hosted eval, adapter eval, and hosted training
- Evolution and swarm workflows for improving agent workspaces against real benchmark failures
Disciplines
| Discipline | Example coverage |
|---|---|
| Civil | Hydrology, hydraulics, drainage, roads, coastal, wind and load derivations |
| Electrical | Cable sizing, PV, grounding, arc flash, thermal rating, short-circuit |
| Ground | Bearing capacity, settlement, CPT/SPT interpretation, slope and retaining-wall checks |
| Maritime | Rule length, freeboard length, block coefficient, and class-society calculations |
| Mechanical | HVAC, pumps, fire services, process calculations, acoustics, wastewater |
| Structural | Concrete, structural fire, load combinations, movement and connection checks |
Mental model
Start with the engineering behaviour that you want to test:
- Artefact task: an agent produces an answer or workspace artefact.
- Finite lifecycle: an agent completes controlled stages and evidence submissions.
- Interactive World: an agent observes state, acts, and receives the next state.
Each task family runs differently, but all of them produce verified trial records that can be evaluated and reported.
The common outer loop is:
- Choose the task family and task.
- Freeze a dataset when you need comparability.
- Run the task through its execution model.
- Verify and evaluate the result.
- Record the evidence, then inspect and report it.
Next Steps
- Quickstart — Generate and run a first task
- Templates — Understand the built-in template catalogue
- Interactive Worlds — Benchmark repeated decisions against task-owned state
- Finite Lifecycles — Control staged evidence and submissions
- Agent Harnesses — Choose an execution strategy
- Prime Lab — Export tasks for hosted eval and training