Datasets

A dataset is the formal "this is the benchmark selection" artefact. It sits between task authoring and experiment execution. It keeps runs comparable while templates and tasks continue to change.

Identity and references

AEC-Bench separates three concepts:

dataset_id defines dataset identity. A publication label such as public-2026 is a permanent discovery name for one exact release. Use a fixed label so later runs select the same material.

Semantic manifest

dataset create writes a manifest under artefacts/datasets/manifests/<dataset_id>/manifest.json. The schema_version value identifies the file format so AEC-Bench can reject an incompatible manifest. The library supplies this value.

{
  "schema_version": 2,
  "dataset_id": "electrical-core",
  "description": "Cable and power-system baseline",
  "tasks": [
    {
      "task_id": "electrical/cable-sizing/voltage-drop/example",
      "path": "tasks/electrical/cable-sizing/voltage-drop/example",
      "task_kind": "artifact"
    },
    {
      "task_id": "civil/dam-monitoring",
      "path": "tasks/civil/dam-monitoring",
      "task_kind": "world"
    }
  ],
  "generation": {
    "seed": 42,
    "config_ref": "suite.toml"
  }
}

The manifest identifies tasks by their semantic identity and repository-relative paths. Set task_kind to "artifact" for a task.toml directory or "world" for an instruction.md and world.toml package. Publishing binds this selection to an exact source or bundle reference.

The optional generation field records replay inputs for the selection. Exact source or byte identity stays in the published dataset reference.

Create and publish

Create a manifest from selected local tasks:

uv run aec-bench dataset create electrical-core \
  --domain electrical \
  --difficulty easy \
  --difficulty medium \
  --description "Cable and power-system baseline"

Create a manifest from the exact selection in a generated-suite sidecar:

uv run aec-bench generate suite --config suite.toml

uv run aec-bench dataset create electrical-core \
  --from-suite-output tasks/generated/my-suite/generation-manifest.json \
  --description "Generated electrical baseline"

Creation is local and write-once for one dataset_id. Publish it under a new label before you use it in a repeatable experiment:

uv run aec-bench dataset publish electrical-core --label public-2026

dataset create discovers artefact tasks through the task registry. To compose a mixed or world-only dataset, use the Python dataset API or write the exact semantic manifest with task_kind: "world", then validate and publish it. aec-bench run executes both task kinds.

The default publication stores a deterministic detached bundle and references its exact bytes with ArtifactRef. Use --repository to bind the manifest to the current full Git commit and repository path:

uv run aec-bench dataset publish electrical-core \
  --label public-2026-repository \
  --repository

Run from a publication

Resolve the label, validate the exact materialisation, and write an experiment that contains the exact reference:

uv run aec-bench dataset validate electrical-core@public-2026

uv run aec-bench dataset config electrical-core@public-2026 \
  --model gpt-4.1-mini \
  --harness tool_loop \
  --backend modal \
  --output experiment.yaml

uv run aec-bench run --config experiment.yaml

dataset config resolves the interactive label before it writes experiment.yaml. Persisted run identity contains the exact repository or bundle reference.

At execution time, aec-bench run loads each entry through its declared task kind, plans all tasks with plan_trials(), groups artefact and world work behind their existing application functions, and restores the declared plan order in the returned records. The runner validates the world and provider combination before execution.

Inspect and move datasets

uv run aec-bench dataset list
uv run aec-bench dataset info electrical-core@public-2026
uv run aec-bench dataset validate electrical-core@public-2026
uv run aec-bench dataset results electrical-core@public-2026
uv run aec-bench dataset export electrical-core@public-2026 --output electrical-core.tar.gz
uv run aec-bench dataset import electrical-core.tar.gz

Dataset export produces a deterministic detached bundle. Import validates the bundle and preserves existing local data when identities conflict.

Change policy

When task membership or meaning changes, create a new semantic dataset ID if it is a different benchmark selection. When the semantic selection is unchanged but you need a new exact retained materialisation, publish it under a new immutable label.

Use one identity mechanism for each purpose:

• Git for exact repository source;
• a stable ID for the dataset;
• ArtifactRef for exact retained bundle bytes; and
• a timestamp only for the real publication event.

Generated project lineages

For related baseline and revision tasks, assign the complete project lineage to one partition before generating sibling rows. All revisions from that project stay together. This prevents a revised copy of a training project from entering an acceptance comparison.

The engineering decision experiments apply this rule to hydraulic source generation with explicit seed-to-partition assignments. Their local experiment.json declares comparison conditions; formal dataset publication continues to use the exact references described above. A sealed holdout requires a frozen, protected acceptance bank.