Templates

How templates work

A template is a reusable engineering problem family. It samples realistic inputs, writes the task instruction, computes the expected answer, and creates complete task directories. You can validate and run those tasks in the same way as hand-authored tasks.

The current upstream library export contains 352 built templates and 284 proposed seed tasks across six disciplines.

See Library Catalogue for the export schema and site sync.

Template anatomy

Each built-in template is a directory under src/aec_bench/templates/builtin/<discipline>/<template>/:

src/aec_bench/templates/builtin/electrical/voltage_drop/
├── params.toml       # metadata, parameters, archetypes, difficulty presets
├── instruction.md    # Jinja2 template for the problem statement
├── engine.py         # pure ground-truth computation
└── __init__.py

Three files define the template:

Templates can also provide verify.py, system_prompt.md, and output_contract.json. These optional files are part of the exact template source in Git.

params.toml

params.toml defines the inputs and metadata for a template:

[meta]
name = "voltage-drop"
description = "Cable voltage drop calculation per AS/NZS 3008.1.1"
discipline = "electrical"
category = "cable-sizing"
standards = ["AS/NZS 3008.1.1"]
tool_mode = "with-tool"

[params.cable_size_mm2]
type = "enum"
unit = "mm²"
description = "Cable conductor cross-sectional area"
values = ["1.5", "2.5", "4", "6", "10", "16", "25", "35", "50", "70", "95", "120", "150", "185", "240"]

[params.length_m]
type = "float"
unit = "m"
description = "Cable route length (one way)"
min = 1
max = 500

[params.load_current_a]
type = "float"
unit = "A"
description = "Design load current"
min = 0.5
max = 500

[params.power_factor]
type = "float"
description = "Load power factor"
min = 0.5
max = 1.0
default = 0.8

[params.conductor_material]
type = "enum"
description = "Conductor material"
values = ["copper", "aluminium"]
derivable_from = "archetype"

[params.circuit_type]
type = "enum"
description = "Circuit type"
values = ["single_phase", "three_phase"]

Supported parameter types include float, int, and enum. Templates can also use archetype-derived values so generated cases follow realistic parameter correlations.

Archetypes

Archetypes bundle values that should move together:

[archetypes.sydney_suburban_lighting]
description = "Suburban lighting circuit with moderate route length"
site_contexts = ["sydney-suburban", "melbourne-suburban"]
length_m = { min = 5, max = 30 }
load_current_a = { min = 1, max = 10 }

This matters in AEC tasks because input independence often creates nonsense. A geotechnical soil, hydraulic duty point, cable route, or structural load case usually has correlated values.

Difficulty presets

Difficulty controls which archetypes can be sampled and how much information is visible:

[difficulty.easy]
description = "All calculation inputs are visible"
visibility = "all_given"
archetypes = ["residential_lighting", "residential_power"]

[difficulty.hard]
description = "Some inputs must be inferred from the scenario"
visibility = "partial"
archetypes = ["commercial_submain", "industrial_feeder"]
hidden_params = ["conductor_material"]
replacement_text = "Use the stated project context to select a suitable conductor material."

The built-in convention is:

The visibility setting can be all_given, partial, or scenario_only. scenario_only presents inputs through rendered scenario context. The template still declares which parameters remain visible, while the engine and verifier retain the exact sampled values.

instruction.md

Instructions are Jinja2 templates:

## Given

| Parameter | Value | Unit |
|-----------|-------|------|
| Cable size | {{ cable_size_mm2 }} | mm² |
| Cable route length | {{ length_m }} | m |
| Design load current | {{ load_current_a }} | A |
| Power factor | {{ power_factor }} | - |
{% if conductor_material is defined %}
| Conductor material | {{ conductor_material }} | - |
{% endif %}
| Circuit type | {{ circuit_type }} | - |

## Required

Calculate the voltage drop percentage and state whether it is within the allowable limit.

Difficulty visibility decides which variables are rendered into the prompt. The hidden values still exist for the engine and verifier.

engine.py

The engine is intentionally small and deterministic:

def compute(
    cable_size_mm2: str,
    length_m: float,
    load_current_a: float,
    power_factor: float,
    conductor_material: str = "copper",
    circuit_type: str = "single_phase",
) -> dict[str, float]:
    vc_mv_per_a_m = ...
    voltage_drop_v = ...
    voltage_drop_pct = ...
    return {
        "vc_mv_per_a_m": vc_mv_per_a_m,
        "voltage_drop_v": voltage_drop_v,
        "voltage_drop_percent": voltage_drop_pct,
        "compliant": 1.0 if voltage_drop_pct <= 5.0 else 0.0,
    }

Good engines are pure functions over sampled parameters. Deterministic constants and lookup values belong in the content-bound template source. Model APIs, rendered prompts, local machine state, unstated intent, and prose-only judgement remain outside the engine.

Generating instances

Generate concrete tasks from a built-in template:

uv run aec-bench generate task voltage-drop \
  --instances 5 \
  --difficulty easy,medium \
  --seed 42 \
  --lifecycle proposed \
  --visibility public \
  --output tasks/generated

List and filter the catalogue:

uv run aec-bench generate list-templates
uv run aec-bench generate list-templates --discipline structural

Validate a custom template before using it:

uv run aec-bench generate validate-template ./my-template

Generate a configured suite:

uv run aec-bench generate suite --config suite.toml --dry-run
uv run aec-bench generate suite --config suite.toml

Generation can write generation-manifest.json at the output root. This file records which template, seed, instance index, difficulty, visibility, and sampled inputs produced the tasks. Keep it when you need to reproduce the same generated set.

Built-in template scope

The built-in templates are strongest when the engineering contract is explicit:

• Deterministic calculations with numeric or categorical inputs
• Stable formulae, embedded lookup tables, or clearly bounded reductions of a design method
• Outputs with concrete tolerances
• Verifiers that score mechanically from explicit rules

Use hand-authored tasks for open-ended document review, protected standards tables, iterative solvers that lack a reduced contract, or broad design judgement that still needs an explicit scoring method.

Writing your own

Start with the smallest useful deterministic contract:

1. Define the expected inputs and outputs in params.toml.
2. Add realistic archetypes so sampled values make sense together.
3. Render a clear instruction.md with difficulty-aware visibility.
4. Implement compute() in engine.py.
5. Run uv run aec-bench generate validate-template ./my-template.
6. Generate easy, medium, and hard instances and validate the generated task directories.

Use templates for reproducible benchmark families. Use hand-authored tasks for bespoke workflows that are not yet reducible to a stable generation contract.