Architecture

AEC-Bench supports three shapes of work: one submitted artefact, a known sequence of stages, or an interactive situation that changes after each action. They share experiment planning, trial records, and reports, but each shape keeps the rules needed to run and score it.

Shared execution control

The execution layer turns each planned trial into a TrialWorkItem. A local scheduler leases work, records attempts and backend submissions, collects an AttemptReceipt, and publishes a TrialFinalization that points to the selected TrialRecord.

OperationalStore holds this mutable coordination state in SQLite. The persisted RunPlan and portable trial evidence remain the authoritative records. RunProgress projects both sources for the CLI, TUI, and web status surface.

Artefact and workspace tasks

An artefact task gives an agent an instruction and an execution environment. The agent changes files or submits an artefact, and the task verifier evaluates the resulting workspace.

The run contracts form one sequence:

ResolvedRunSpec -> RunPlan -> TrialWorkItem -> Attempt -> TrialRecord -> RunAccounting

AttemptRecipe controls whether one trial runs one attempt or several candidates. It starts and tracks attempts, while the outer trial runner retains verifier and reward access. HarnessSpec describes how one agent execution works.

With best-of execution, a selector chooses among several TaskAttempt candidates using its permitted view of their results. The official verifier evaluates the selected output once, and the trial retains the selection evidence with the output and trajectory so they remain readable after temporary workspace cleanup.

Templates and suites produce deterministic task instances before this flow starts. The generator can write replay inputs once in generation-manifest.json at the output root. Normal loading and execution use the runnable task directories; generation replay uses the sidecar.

RLM and Lambda-RLM generate reports within this artefact-task flow, using shared section state, validation, and assembly from templates/report/. The harnesses direct model work, then pass the submitted report to the normal task verification and evaluation steps.

See Tasks and Templates for the authoring path.

Interactive Worlds

An Interactive World repeatedly shows an actor-visible observation and accepts an action that changes task-owned state. Interactive Worlds suit causal decisions over time; artefact tasks cover a final workspace submission.

The world owns its state, rules, and evaluation. The runner connects the agent to the world, records the evidence, and returns a standard TrialRecord.

The current catalogue includes wastewater pump-station stewardship and dam-seepage monitoring. Both use the same world interfaces, while each task keeps its own state, rules, and evaluation.

WorldTask binds an objective to the exact registered build and profile used by each planned trial. run_world_experiment() runs complete journeys in plan order through the selected world-trial function, returning a TrialRecord for each journey with optional persistence.

Add capabilities such as recovery or branching when the world requires them.

See Interactive Worlds for the contributor-facing model.

Finite lifecycles

A finite lifecycle declares its sequence of evidence stages before execution. The task defines what each stage requires and how to verify it; the host releases stages and controls progression through to the final result.

Use a lifecycle when the work has known stages and controlled evidence release. Use an Interactive World when actions change an ongoing causal state. Use an artefact task when one workspace or submission is sufficient.

The lifecycle Python API uses one coordinator for all stage changes, including submissions and branches. Each branch starts from an accepted submission and preserves its parent's state.

LifecycleTrial combines one planned trial with its lifecycle package, run directory, context mode, and visibility policy. run_local_lifecycle() keeps fresh and persistent agent sessions separate because they have different memory and recovery rules. Both return the state and evidence needed to construct a trial record.

Use run_lifecycle_trial() to execute and verify one lifecycle, or run_lifecycle_experiment() to process several planned trials in order. Both return normal TrialRecord values, with persistence selected by the caller. Lifecycle studies define their own recovery and selection rules.

A lifecycle evaluator connects lifecycle execution to the separate runtime-neutral meta-harness API through normal TrialRecord values.

See Finite Lifecycles.

Meta-harness composition

aec_bench.experimentation.meta_harness compares candidates across the existing execution families. The caller supplies each candidate, the function that runs it, and the function that compares its TrialRecord results. It supports both one-off comparisons and bounded rounds of proposal and refinement.

Each evaluator runs its candidate through the relevant task runtime, then passes the resulting trials to the assessment function, including failed or invalid results. The runtime keeps its execution and evidence rules; evaluation defines metrics and governance controls approval.

Meta-harness results are normal in-memory Python values. Existing task runtimes and trial records supply persistence and execution. See Meta-Harness Composition.

Agent workspace evolution

run_evolution() combines a candidate proposer with CandidateChecks to improve versioned prompt-and-skill workspaces. The proposer develops a CandidateProposal in scratch, while the checks plan and run the comparison. The outer loop uses those results to decide which changes to accept and persist.

The built-in proposer is Agentic Variation. AVO can make and check several bounded scratch revisions before submission. The outer application checks the submitted child against its selected parent, applies the search policy, and only then updates canonical workspace, archive, graveyard, and lineage state.

Swarm uses the same proposal and evidence boundaries concurrently. Each worker has private AVO state. The manager owns shared selection checks, budget, archive effects, and immutable SwarmState.

Where each responsibility lives

Local and remote execution

run-local executes one artefact task through the shared run_experiment() application boundary and a LocalTaskRuntime. Production local execution selects from a fixed set of agent-harness builders.

run uses the same artefact-task application boundary with a HarborExperimentRuntime, which dispatches and imports the Harbor work. Docker, Modal, Morph, e2b, and Daytona are Harbor environment backends.

Harbor also composes with the shared scheduler through its Python boundary. The Harbor adapter maps a planned trial to a backend submission and maps collected results back to the planned UUID before finalization. The synchronous aec-bench run command remains the direct Harbor workflow described in Backends.

Prime Lab exports public artefact tasks for hosted evaluation and training, and finite lifecycles for local evaluation against their source packages. Prime Agent runs the upstream executable directly for local artefact tasks or through bounded ACP sessions for supported Interactive Worlds.

See Execution Backends, Prime Lab, and Prime Agent.

How evidence is stored

Follow the saved records from the requested condition to its results:

A trial can contain one attempt or select from several candidates.

OperationalStore stores the mutable SQLite coordination rows used while work is queued, leased, retried, cancelled, reconciled, and reported. The EvidenceIndex is a separate rebuildable SQLite metadata projection; portable trial records and their referenced artefacts remain the evidence authority. See Evidence Index for metadata queries and verification.

RunAccounting reconciles the observed records with the exact plan. It supplies run completeness, validity, terminal outcome counts, accepted record identities, and quarantined record identities for reporting.

Read the three trial statuses to assess execution, evaluation, and evidence completeness. The same record links to the output and trajectory for review, with usage information for cost reporting.

Producing authorities own their evidence documents, and trials refer to those documents. Provider evidence has its own single-manifest reference. Keyless protocol proof and live-provider qualification remain separate claims.

trajectory.jsonl contains one validated entry per non-empty line, starting with the first line. Provider transcripts remain separate only when their producing boundary needs the exact representation.

Accepted run manifests and trial records are immutable. Dataset publications bind a stable dataset ID to one exact repository or bundle reference under an immutable label.

Dependency direction

Dependencies follow ownership:

• task definitions and task worlds remain provider-neutral;
• shared world-runtime code remains independent of concrete worlds;
• canonical run planning expands a ResolvedRunSpec into one exact RunPlan;
• artefact-task orchestration can depend on task and runtime boundaries while HarnessSpec remains an execution-capability description;
• lifecycle application composition depends on the lifecycle coordinator, normal trial contracts, and explicit execution effects;
• meta-harness composition depends on TrialRecord and caller-supplied functions;
• evolution composition depends on check and proposal capabilities; AVO proposes candidates, while Evolution owns selection and persistence;
• evaluation owns policy and consumes verifier evidence, while persistence stores the resulting records; and
• CLI, TUI, web, and reports compose lower-level capabilities.

Pydantic models sit at untrusted, external, persisted, or cross-process boundaries. Normal Python values are enough inside one owner.

Design priorities

When design goals conflict, aec-bench uses this order:

When two designs protect these goals equally, the simpler final system wins.