Datasets

A dataset is the formal "this is the benchmark selection" artefact. It sits between task authoring and experiment execution. It keeps runs comparable while templates and tasks continue to change.

Identity and references

AEC-Bench separates three concepts:

dataset_id defines dataset identity. A publication label such as public-2026 is a permanent discovery name for one exact release. Use a fixed label rather than latest so later runs select the same material.

Semantic manifest

dataset create writes a manifest under artefacts/datasets/manifests/<dataset_id>/manifest.json. The schema_version value identifies the file format so AEC-Bench can reject an incompatible manifest. The library supplies this value.

{
  "schema_version": 2,
  "dataset_id": "electrical-core",
  "description": "Cable and power-system baseline",
  "tasks": [
    {
      "task_id": "electrical/cable-sizing/voltage-drop/example",
      "path": "tasks/electrical/cable-sizing/voltage-drop/example",
      "task_kind": "artifact"
    },
    {
      "task_id": "civil/dam-monitoring",
      "path": "tasks/civil/dam-monitoring",
      "task_kind": "world"
    }
  ],
  "generation": {
    "seed": 42,
    "config_ref": "suite.toml"
  }
}

The manifest contains semantic task identity and repository-relative paths. task_kind: "artifact" selects task.toml loading. task_kind: "world" selects an instruction.md and world.toml package. The published dataset reference carries exact source or bundle identity.

The optional generation field records replay inputs for the selection. Exact source or byte identity stays in the published dataset reference.

Create and publish

Create a manifest from selected local tasks:

uv run aec-bench dataset create electrical-core \
  --domain electrical \
  --difficulty easy \
  --difficulty medium \
  --description "Cable and power-system baseline"

Create a manifest from the exact selection in a generated-suite sidecar:

uv run aec-bench generate suite --config suite.toml

uv run aec-bench dataset create electrical-core \
  --from-suite-output tasks/generated/my-suite/generation-manifest.json \
  --description "Generated electrical baseline"

Creation is local and write-once for one dataset_id. Publish it under a new label before you use it in a repeatable experiment:

uv run aec-bench dataset publish electrical-core --label public-2026

The current dataset create command discovers artefact tasks through the task registry. To compose a mixed or world-only dataset today, use the Python dataset API or write the exact semantic manifest with task_kind: "world", then validate and publish it. This is a creation-surface limit; dataset loading and aec-bench run support world entries.

The default publication stores a deterministic detached bundle and references its exact bytes with ArtifactRef. Use --repository to bind the manifest to the current full Git commit and repository path instead:

uv run aec-bench dataset publish electrical-core \
  --label public-2026-repository \
  --repository

Run from a publication

Resolve the label, validate the exact materialisation, and write an experiment that contains the exact reference:

uv run aec-bench dataset validate electrical-core@public-2026

uv run aec-bench dataset config electrical-core@public-2026 \
  --model gpt-4.1-mini \
  --harness tool_loop \
  --backend modal \
  --output experiment.yaml

uv run aec-bench run --config experiment.yaml

dataset config resolves the interactive label before it writes experiment.yaml. Persisted run identity contains the exact repository or bundle reference.

At execution time, aec-bench run loads each entry through its declared task kind, plans all tasks with plan_trials(), groups artefact and world work behind their existing application functions, and restores the declared plan order in the returned records. Unsupported world and provider combinations fail before execution begins.

Inspect and move datasets

uv run aec-bench dataset list
uv run aec-bench dataset info electrical-core@public-2026
uv run aec-bench dataset validate electrical-core@public-2026
uv run aec-bench dataset results electrical-core@public-2026
uv run aec-bench dataset export electrical-core@public-2026 --output electrical-core.tar.gz
uv run aec-bench dataset import electrical-core.tar.gz

Dataset export produces a deterministic detached bundle. Import validates the bundle and preserves existing local data when identities conflict.

Change policy

When task membership or meaning changes, create a new semantic dataset ID if it is a different benchmark selection. When the semantic selection is unchanged but you need a new exact retained materialisation, publish it under a new immutable label.

Use one identity mechanism for each purpose:

• Git for exact repository source;
• a stable ID for the dataset;
• ArtifactRef for exact retained bundle bytes; and
• a timestamp only for the real publication event.

Generated project lineages

For related baseline and revision tasks, assign the complete project lineage to one partition before generating sibling rows. All revisions from that project stay together. This prevents a revised copy of a training project from entering an acceptance comparison.

The engineering decision experiments apply this rule to hydraulic source generation with explicit seed-to-partition assignments. Their local experiment.json declares comparison conditions; formal dataset publication continues to use the exact references described above. Acceptance labels alone do not create a sealed holdout.