Runs and Plans

An AEC-Bench run is one occurrence of a benchmark condition. Each run receives its own UUIDv7 identity and readable key. The saved condition and plan make two runs comparable when they use the same task, agent, execution, and evaluation versions.

ExperimentManifest
        ↓ resolve
ResolvedRunSpec
        ↓ plan
RunPlan
        ↓ create work
TrialWorkItem
        ↓ execute and finalise
TrialRecord values
        ↓ account
RunAccounting

Resolve the requested condition

ResolvedRunSpec records the complete requested condition before execution. It contains:

• the experiment and run identities;
• exact task snapshot references;
• exact agent conditions;
• compute and repetition settings;
• verification and reviewer settings;
• visibility policy;
• execution policy; and
• optional authority, evaluation, and provider-route references.

The persisted specification records environment-backed credentials as named environment references and stores the resolved benchmark settings as values.

Persist the exact plan

RunPlan expands the resolved condition into an ordered set of PlannedTrial values. Planning combines each selected task release, agent condition, and repetition.

Every planned trial contains:

• a UUIDv7 trial identity and readable key;
• a contiguous ordinal;
• the run identity;
• the exact task release and task metadata;
• the agent and compute condition;
• the repetition number;
• the execution family;
• an attempt recipe; and
• the evaluation profile.

The plan summary records task, agent, repetition, execution-family, backend, visibility, and lifecycle counts.

The evidence store persists the resolved specification first, then the draft and ready plan. Starting the run fixes the ready plan for that occurrence. OperationalStore creates mutable work-item, attempt, submission, and lease rows for the ready plan. The operational rows coordinate execution; the persisted plan remains the authority for planned membership.

Trials, repetitions, and attempts

A repetition creates a separate planned trial. Each repetition receives its own trial UUID and plan ordinal.

An execution attempt belongs to one planned trial. Retries and best-of candidates receive separate attempt UUIDs while the planned trial identity stays the same. The selected and verified result becomes the TrialRecord for that planned trial.

planned trial 019c…
├── attempt 019d…
└── attempt 019e…

This identity model keeps benchmark membership separate from execution recovery and candidate selection.

Control a persisted run

Start or resume a persisted local artefact plan with explicit operational and plan roots:

uv run aec-bench run start <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks

uv run aec-bench run resume <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs \
  --tasks-root tasks

run start executes the saved local artefact plan. run resume expires old leases, reconciles uncertain work, and continues ready work. Request cancellation with run cancel and inspect the shared progress projection with run status:

uv run aec-bench run cancel <run-id> \
  --operational-store artefacts/operational.sqlite
uv run aec-bench run status <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

The TUI and web surface read the same progress projection. The web endpoint is GET /api/runs/{run_id}/status.

Inspect and compare plans

For identity-bearing artefact tasks, create and persist a ready plan from an experiment manifest:

uv run aec-bench run plan \
  --config experiment.yaml \
  --tasks-root tasks \
  --store-root artefacts/runs

Inspect the saved condition, plan summary, and current state:

uv run aec-bench run inspect <run-key-or-uuid> \
  --store-root artefacts/runs

Compare two saved conditions field by field:

uv run aec-bench run diff <left-run> <right-run> \
  --store-root artefacts/runs

run diff reports semantic changes to task releases, agent conditions, compute settings, repetitions, evaluation, and other resolved fields.

Account for the result set

run reconcile matches typed trial outcomes to the exact planned UUID set:

uv run aec-bench run reconcile <run-key-or-uuid> \
  --observations outcomes.json \
  --store-root artefacts/runs

The accounting result partitions the planned trials into succeeded, failed, cancelled, timed out, invalid, and missing outcomes. It also records duplicate and unexpected identities. Accepted records remain in plan order, while quarantined records remain separate from aggregate input.

An identical duplicate is counted once. A conflicting duplicate is quarantined and makes the accounting result invalid.

Execution families

Artefact tasks, Interactive Worlds, and finite lifecycles use the same planned trial identity and result-accounting rules. Each family retains its own execution semantics and binds its result to the exact task, world, profile, or lifecycle release recorded in the plan.

Harbor transport records carry the planned trial UUID alongside Harbor job and trial names. Import reconciliation restores the planned UUID, validates exact membership, and returns accepted records in plan order.

See Contracts for the validated data shapes, Deployment for Harbor execution, and CLI for the complete command reference.