Tasks

This page covers artefact and workspace tasks: work that ends in a submitted file or a bounded workspace. For repeated decisions against task-owned state, use an Interactive World. For a known sequence of evidence stages, use a finite lifecycle.

Within the artefact-task family, aec-bench uses the word "task" at three levels:

Public benchmark reports should name the exact dataset or release that was run. Report proposed ideas, reusable templates, generated tasks, and development-only task families as separate inventory measures.

Anatomy of a task

A runnable task instance is a directory with required files and optional review aids:

tasks/electrical/voltage-drop/
├── task.toml          # metadata and resource limits
├── instruction.md     # the prompt sent to the agent
├── environment/
│   └── Dockerfile     # runtime image for the task
├── solution/
│   └── solve.py       # reference solution (not sent to agent)
└── tests/
    ├── test.sh        # verifier entry point
    ├── verify.py      # scoring logic
    └── fixtures/      # golden pass/fail examples for verifier testing
        ├── golden_pass.md
        └── golden_fail.md

The loader requires task.toml, instruction.md, and a verifier entry point. Add an environment Dockerfile before Harbor execution; without it, validation reports a promotion warning. solution/ and golden fixtures remain private review aids.

The prompt

instruction.md is the prompt sent to the agent. It must stand on its own and explicitly name any staged files, outputs, or constraints the agent is expected to use.

You are a senior electrical engineer specializing in building services.

## Problem
Calculate the voltage drop for a three-phase cable circuit using the
impedance method, and determine whether it complies with the maximum
allowable voltage drop limit.

## Given
| Parameter | Value | Unit |
|-----------|-------|------|
| Load current | 45 | A |
| Cable length (one way) | 80 | m |
| Cable resistance (R) | 0.524 | ohm/km |
| Cable reactance (X) | 0.08 | ohm/km |
| Power factor (cos phi) | 0.85 | - |
| System voltage (line-to-line) | 400 | V |
| Maximum allowable voltage drop | 5 | % |

## Required
Calculate the following:
- Voltage drop (V)
- Voltage drop as a percentage of system voltage (%)
- Compliance with the 5% limit (1 if compliant, 0 if not)

The configuration

task.toml contains metadata, timeouts, and runtime limits. instruction.md contains the problem. When AEC-Bench loads the directory, it validates both files as one TaskDefinition.

version = "1.0"

[identity]
id = "019c2c7a-5a33-7b8d-a702-8f7f3e8c21aa"
key = "electrical/voltage-drop"
version = 3

[metadata]
lifecycle = "active"
visibility = "public"
difficulty = "easy"
category = "reasoning"
tags = ["electrical", "buildings-electrical", "deterministic", "AS-NZS-3008"]

[agent]
timeout_sec = 600.0

[verifier]
timeout_sec = 120.0

[environment]
extensions = []
build_timeout_sec = 600.0
cpus = 1
memory_mb = 2048
storage_mb = 5120
allow_internet = true

The loader requires [identity].id, [identity].key, [identity].version, [metadata].lifecycle, and [metadata].visibility. It does not infer missing policy values. The existing top-level version field is separate from [identity].version; the identity version records a semantic revision of the task.

Task identity

AEC-Bench gives each task three related identity values:

key:      electrical/voltage-drop
UUID:     019c2c7a-5a33-7b8d-a702-8f7f3e8c21aa
version:  3
display:  electrical/voltage-drop · 3e8c21aa

The key is the readable reference used in commands and reviews. The UUIDv7 is the stable identity used to join records. The positive integer version changes when the task's meaning, verifier, lifecycle, visibility, or output contract changes. A renamed key can remain as an alias for the same UUID. The short UUID suffix is for display only.

Checksums have a different purpose: they verify retained bytes. Matching file content does not make two tasks the same task.

The environment

Each runnable task provides a Dockerfile that defines its runtime image. This is usually minimal — just the tools the task requires:

FROM --platform=linux/amd64 ubuntu:24.04

RUN apt-get update && apt-get install -y \
    python3 \
    bc \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /workspace

Some tasks stage input files such as spreadsheets, drawings, code extracts, or reference data into the runtime via fixture directories. The agent reads them as normal files on disk.

The verifier

The verifier computes ground truth independently, reads the agent's output, and writes a reward (0.0–1.0) plus per-field details. For Harbor-backed runs, tests/test.sh is the entry point; it usually calls tests/verify.py.

The voltage-drop verifier uses this core scoring logic:

def compute_ground_truth() -> dict[str, float]:
    """Compute expected answers from three-phase voltage drop formula."""
    current = 45.0
    L = 80.0       # cable length (m)
    R = 0.524      # resistance (ohm/km)
    X = 0.08       # reactance (ohm/km)
    pf = 0.85      # power factor
    V = 400.0      # system voltage (V)

    sin_phi = math.sin(math.acos(pf))
    Vd = math.sqrt(3) * current * L * (R * pf + X * sin_phi) / 1000
    Vd_pct = Vd / V * 100
    compliance = 1.0 if Vd_pct <= 5.0 else 0.0

    return {
        "voltage_drop_v": Vd,
        "voltage_drop_pct": Vd_pct,
        "compliance": compliance,
    }

The verifier writes two files under /logs/verifier/:

• reward.json — the headline score: {"reward": 0.67}
• details.json — per-field breakdown: {"voltage_drop_v": 1.0, "voltage_drop_pct": 1.0, "compliance": 0.0}

Each field is scored independently with configurable tolerances. Numerical fields typically allow 3% relative tolerance; boolean/categorical fields require an exact match.

The reward file is not proof that the verifier completed. AEC-Bench also records the verifier process exit, timeout, cancellation, identity, version, and output parse status. A failed process cannot produce a completed evaluation. See Scoring for the receipt and status rules.

Workspace evidence

Artefact trials capture a base and final WorkspaceManifest for the isolated workspace. WorkspaceDelta classifies paths as added, modified, deleted, or unchanged. The final evidence records the declared primary_output and retains changed actor files and deletion paths with portable path and file facts.

Unchanged task inputs remain represented by the base manifest. The retained trial output contains the primary output and meaningful workspace changes, so reviewers can inspect what the agent produced while the base manifest continues to identify the task inputs.

Golden fixtures

Most verifiers ship with test fixtures — a golden_pass.md that should score 1.0 and a golden_fail.md that should score less. These test the verifier itself.

## Step 2: Voltage Drop
Vd = sqrt(3) x 45 x 80 x (0.524 x 0.85 + 0.08 x 0.5268) / 1000
Vd = 3.0400 V

## Step 3: Percentage
Vd% = 3.0400 / 400 x 100 = 0.7600%

Task lifecycle

Tasks use four catalogue states: proposed → active → deprecated → retired.

• Proposed — draft task or source spec under review; not ready for a published dataset
• Active — reviewed benchmark task eligible for datasets and reports
• Deprecated — superseded by a better version but still runnable for historical comparison
• Retired — archived and excluded from new benchmark selections

Visibility is a separate policy axis:

• Public tasks can enter normal public selections and reports.
• Private tasks require an explicit permitted visibility context and remain outside public catalogue surfaces.
• Holdout tasks also require explicit permission. Their content and individual results remain protected so they can test whether public exposure influenced a model.

New selections normally use active, public tasks. Deprecated tasks require explicit inclusion. Proposed and retired tasks cannot start a new run. The runner checks these rules again before planning. Publication eligibility still requires review and promotion checks beyond successful filesystem loading.

Task categories

Categories describe the type of work a task demands. They are free-form strings. Use the category that best describes the primary demand, and keep more specific descriptors in tags.

Difficulty levels

Each task is rated easy, medium, or hard. Ratings should be calibrated by domain experts against what a professional engineer at each level would be expected to handle.

Source specs, seeds, and generated instances

There are three common ways work enters the benchmark:

Source task specs capture proposed benchmark ideas before they are runnable. They usually live as source_task.json files and preserve the original discipline, category, standards, inputs, outputs, and source file.

Seed runnable instances are hand-authored tasks with a prompt, runtime, and custom verifier. These are useful when the task needs expert judgement, a bespoke source document, or a narrow real-world scenario.

Generated instances come from parameterised templates. A template defines the problem structure, and the generator samples concrete values to create reproducible tasks. It records the inputs needed to repeat generation once in generation-manifest.json at the output root. See Templates for details.

Container extensions

Tasks can reuse common container capabilities:

[environment]
extensions = ["multimodal", "ocr"]

Common extensions add chart-generation dependencies, OCR tools, or a supported agent CLI. Generate or refresh derived Dockerfiles after changing extensions:

uv run aec-bench generate dockerfiles tasks/
uv run aec-bench generate dockerfiles tasks/ --dry-run

Tasks with bespoke runtime requirements can continue to own a handwritten Dockerfile.

Image-returning tools

Tool declarations can mark an output as an image:

[[environment.tools]]
name = "create_chart"
source = "tools/create_chart.py"
description = "Generate a chart from computed data."
returns_image = true

The common tool entry point records tool output as text. An agent can inspect the returned image only when its harness explicitly supports image input. Other harnesses can still retain the image for the verifier or later review.