Architecture

AEC-Bench supports three shapes of work: one submitted artefact, a known sequence of stages, or an interactive situation that changes after each action. They share experiment planning, trial records, and reports, but each shape keeps the rules needed to run and score it.

Shared execution control

The execution layer turns each planned trial into a TrialWorkItem. A local scheduler leases work, records attempts and backend submissions, collects an AttemptReceipt, and publishes a TrialFinalization that points to the selected TrialRecord.

OperationalStore holds this mutable coordination state in SQLite. The persisted RunPlan and portable trial evidence remain the authoritative records. RunProgress projects both sources for the CLI, TUI, and web status surface.

Artefact and workspace tasks

An artefact task gives an agent an instruction and an execution environment. The agent changes files or submits an artefact, and the task verifier evaluates the resulting workspace.

The run contracts form one sequence:

ResolvedRunSpec -> RunPlan -> TrialWorkItem -> Attempt -> TrialRecord -> RunAccounting

AttemptRecipe controls whether one trial runs one attempt or several candidates. It starts and tracks attempts, while the outer trial runner retains verifier and reward access. HarnessSpec describes how one agent execution works.

With best-of execution, several TaskAttempt candidates belong to one planned trial. The selector uses only its permitted candidate view. The official verifier evaluates the selected output once. Selection evidence is retained with the trial, and the selected output and trajectory remain resolvable after every temporary attempt workspace is removed.

Templates and suites produce deterministic task instances before this flow starts. The generator can write replay inputs once in generation-manifest.json at the output root. Normal loading and execution use the runnable task directories; generation replay uses the sidecar.

See Tasks and Templates for the authoring path.

Interactive Worlds

An Interactive World repeatedly shows an actor-visible observation and accepts an action that changes task-owned state. Interactive Worlds suit causal decisions over time; artefact tasks cover a final workspace submission.

The world owns its state, rules, and evaluation. The runner connects the agent to the world, records the evidence, and returns a standard TrialRecord.

The current catalogue includes wastewater pump-station stewardship and dam-seepage monitoring. Both use the same world interfaces, while each task keeps its own state, rules, and evaluation.

WorldTask binds an objective to an exact registered world build and profile. The persisted plan binds each world trial to that exact world and profile release. A complete dam, pump, or Harbor world-trial function then returns a TrialRecord. Asynchronous run_world_experiment() applies that operation in plan order and can persist each completed record.

Persistence, recovery, host controls, branching, rollouts, staged evidence, and provider integration are optional capabilities. Add these capabilities when the world requires them.

See Interactive Worlds for the contributor-facing model.

Finite lifecycles

A finite lifecycle is a separate execution family. It declares a bounded sequence of evidence stages before execution. The task owns each stage's objective, visible evidence, required result, and verifier rules. The host owns release, progression, failure, and finalisation.

Use a lifecycle when the work has known stages and controlled evidence release. Use an Interactive World when actions change an ongoing causal state. Use an artefact task when one workspace or submission is sufficient.

Finite lifecycles use one coordinator for all stage changes. The Python functions release_checkpoint(), submit_checkpoint(), read_lifecycle(), revisit_checkpoint(), branch_lifecycle(), and run_lifecycle() use those same rules. A branch can start only from an accepted submission and leaves its parent unchanged.

LifecycleTrial combines one planned trial with its lifecycle package, run directory, context mode, and visibility policy. run_local_lifecycle() keeps fresh and persistent agent sessions separate because they have different memory and recovery rules. Both return the state and evidence needed to construct a trial record.

run_lifecycle_trial() executes and verifies one lifecycle, then returns a normal TrialRecord. It saves the record only when the caller asks. run_lifecycle_experiment() applies the same operation to several planned trials in order. Lifecycle-specific studies keep their own study design, recovery, and selection rules.

A lifecycle evaluator connects lifecycle execution to the separate runtime-neutral meta-harness API through normal TrialRecord values.

See Finite Lifecycles.

Meta-harness composition

aec_bench.experimentation.meta_harness compares candidates across the existing execution families. The caller supplies each candidate, the function that runs it, and the function that compares its TrialRecord results. It supports both one-off comparisons and bounded rounds of proposal and refinement.

Each evaluator defines how its candidate runs, and each assessment function defines what the evidence means. Artefact-task, lifecycle, Interactive World, adaptive, and repair code retain their own execution and evidence rules. Evaluation owns metric meaning; governance owns approval. Failed and invalid trials remain available to the assessment function.

Meta-harness results are normal in-memory Python values. Existing task runtimes and trial records supply persistence and execution. See Meta-Harness Composition.

Agent workspace evolution

run_evolution() coordinates changes to versioned prompt-and-skill workspaces. It receives a CandidateChecks capability and a candidate proposer. Checks own batch planning, execution, and optional observation enrichment. The proposer owns scratch work and returns a CandidateProposal; the outer loop owns scoring, acceptance, and persistence.

The built-in proposer is Agentic Variation. AVO can make and check several bounded scratch revisions before submission. The outer application checks the submitted child against its selected parent, applies the search policy, and only then updates canonical workspace, archive, graveyard, and lineage state.

Swarm uses the same proposal and evidence boundaries concurrently. Each worker has private AVO state. The manager owns shared selection checks, budget, archive effects, and immutable SwarmState.

Where each responsibility lives

Local and remote execution

run-local executes one artefact task through the shared run_experiment() application boundary and a LocalTaskRuntime. Production local execution selects from a fixed set of agent-harness builders.

run uses the same artefact-task application boundary with a HarborExperimentRuntime, which dispatches and imports the Harbor work. Docker, Modal, Morph, e2b, and Daytona are Harbor environment backends.

Harbor also composes with the shared scheduler through its Python boundary. The Harbor adapter maps a planned trial to a backend submission and maps collected results back to the planned UUID before finalization. The synchronous aec-bench run command remains the direct Harbor workflow described in Backends.

Prime is separate from Harbor. Prime Lab exports public artefact tasks into packages for hosted evaluation and training. Its finite-lifecycle export is local-only and keeps references to the source packages and checkout. Prime Agent runs the upstream executable directly for local artefact tasks or through bounded ACP sessions for supported Interactive Worlds.

See Execution Backends, Prime Lab, and Prime Agent.

How evidence is stored

ResolvedRunSpec stores the complete requested condition. RunPlan stores the ordered expected trial set. RunManifest stores shared execution context for trial evidence, and each TrialRecord records one planned result through its run and trial identities. A trial can contain one attempt or select from several candidates.

OperationalStore stores the mutable SQLite coordination rows used while work is queued, leased, retried, cancelled, reconciled, and reported. The EvidenceIndex is a separate rebuildable SQLite metadata projection; portable trial records and their referenced artefacts remain the evidence authority. See Evidence Index for metadata queries and verification.

RunAccounting reconciles the observed records with the exact plan. It supplies run completeness, validity, terminal outcome counts, accepted record identities, and quarantined record identities for reporting.

TrialRecord answers three separate questions: Did the agent run? Was the output evaluated? Is the required evidence complete? It also links to retained outputs, trajectories, and usage information so reports do not depend on temporary workspaces.

Producing authorities own their evidence documents, and trials refer to those documents. Provider evidence has its own single-manifest reference. Keyless protocol proof and live-provider qualification remain separate claims.

trajectory.jsonl contains one validated entry per non-empty line, starting with the first line. Provider transcripts remain separate only when their producing boundary needs the exact representation.

Accepted run manifests and trial records are immutable. Dataset publications bind a stable dataset ID to one exact repository or bundle reference under an immutable label.

Dependency direction

Dependencies follow ownership:

• task definitions and task worlds remain provider-neutral;
• shared world-runtime code remains independent of concrete worlds;
• canonical run planning expands a ResolvedRunSpec into one exact RunPlan;
• artefact-task orchestration can depend on task and runtime boundaries while HarnessSpec remains an execution-capability description;
• lifecycle application composition depends on the lifecycle coordinator, normal trial contracts, and explicit execution effects, not task, world, provider, Harbor, or Prime implementations;
• meta-harness composition depends on TrialRecord and caller-supplied functions, not concrete runtimes, providers, or task families;
• evolution composition depends on check and proposal capabilities; AVO proposes candidates, while Evolution owns selection and persistence;
• evaluation owns policy and consumes verifier evidence, while persistence stores the resulting records; and
• CLI, TUI, web, and reports compose lower-level capabilities.

Pydantic models sit at untrusted, external, persisted, or cross-process boundaries. Normal Python values are enough inside one owner.

Design priorities

When design goals conflict, aec-bench uses this order:

When two designs protect these goals equally, the simpler final system wins.