Contracts

This page is for developers who need to read or produce AEC-Bench data. A contract validates information when it crosses a boundary, such as imported task material, provider results, another process, or persisted evidence. Ordinary Python values handle work inside one owner.

The run boundary resolves one requested condition, persists its exact trial set, coordinates execution, records each result, and accounts for the complete observed set:

Artefact tasks, Interactive Worlds, and finite lifecycles use the same run and trial identities. Each execution family retains its task-owned evidence and binds the final result to its planned trial.

TaskDefinition — what to solve

A TaskDefinition describes one artefact or workspace task. It binds the instruction to the task's identity and execution requirements, including the verifier that will assess the result.

The task directory supplies the public files:

task/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile
└── tests/
    ├── test.sh
    └── verify.py

Task-specific output meaning remains with the task and its verifier.

Executable Interactive Worlds use WorldTask; static artefact tasks use TaskDefinition. Both enter the same planning, dataset, evidence, and reporting layers through their own task runtimes.

WorldTask — one world objective

WorldTask binds one provider-neutral instruction to an exact WorldBuildRef and InteractiveWorldProfileRef. It carries task identity, a derived semantic revision, normal catalogue metadata, and tags. The world runtime owns task state, provider configuration, and execution state.

A file-backed world task contains instruction.md and world.toml. The TOML document has exact [world] and [profile] references plus registered [metadata]. load_world_task() rejects stale build or profile content and metadata that does not match the registered profile.

The same world task can enter a persisted RunPlan, direct run_world_experiment() composition, or a dataset manifest entry with task_kind: "world".

Lifecycle and visibility

Task catalogue status and evidence progression are separate concepts.

Lifecycle and visibility are explicit. A missing value is invalid. New selections normally use active, public tasks; other permitted states require an explicit selection or visibility context. New work requires an active task or an explicitly selected deprecated task.

AgentOutput — what came back

An agent harness returns a minimal AgentOutput envelope:

class AgentOutput(StrictModel):
    status: AgentOutputStatus   # completed | partial | failed | empty
    output_path: NonEmptyStr
    output_format: NonEmptyStr
    error_message: str | None = None

The harness reports completion and location. Evaluation owns parsing and scoring the task-specific output.

For report harnesses, completion also requires all required section checks to pass and the output bytes to match accepted report state. The task verifier scores the submitted artefact. Private verifier inputs remain outside the actor workspace and model requests.

ResolvedRunSpec and RunPlan — what will run

ResolvedTaskInstance combines the validated TaskDefinition with the exact task directory and resolved environment, verifier, and tool paths. It is the direct task input to artefact-task execution.

ResolvedRunSpec fixes the condition for a run. RunPlan expands it into an ordered set of PlannedTrial values, each bound to that condition:

Each planned trial has a UUIDv7 identity and readable key, plus an ordinal that places it within the run. It binds one task-agent-repetition combination to its compute configuration, execution family, attempt recipe, and evaluation profile. World and lifecycle trials also carry their exact family release.

EvidenceRunStore persists the specification and plans with portable trial evidence. OperationalStore coordinates execution in SQLite through the execution control records.

A ready plan has complete task and agent coverage, with unique trial identities and keys in contiguous ordinal order. Its summary counts must match that trial set.

Execution control contracts

ExecutionPolicy fixes the scheduler limits and recovery rules for a run. The scheduler uses these records to dispatch a planned trial and retain its result:

See Execution Control for scheduler limits and the recovery workflow.

TaskAttempt — one isolated agent execution

Each call to runtime.run_once() returns one TaskAttempt. The attempt records its unique attempt_id, optional parent_attempt_id, workspace, output, timing, and runtime diagnostics.

An AttemptRecipe receives the tracked AttemptRunner. It can request one attempt, retry from a parent attempt, or produce candidates for selection. The outer trial runner retains verifier and reward access. Evolution candidates and execution attempts use separate identities.

TaskAttempt is an internal execution result. TrialRecord is the selected, verified, and reportable trial result.

RunAccounting — the complete result set

TrialAccountingObservation assigns one typed outcome to a planned trial: succeeded, failed, cancelled, timed_out, invalid, or missing.

RunAccounting compares those observations with the exact ResolvedRunSpec and RunPlan. It records accepted, missing, invalid, duplicate, conflicting, and unexpected trial identities. Accepted records remain in plan order, and quarantined records remain separate from aggregate input.

The resulting run status is complete, complete_with_failures, cancelled, incomplete, or invalid. See Runs and Plans for the complete workflow.

Evolution candidates — exact workspace and evidence

WorkspaceSnapshot.candidate_id identifies a specific prompt-and-skill candidate. It is separate from trial and attempt identity.

CandidateChecks plans a candidate-independent batch, runs it, optionally enriches its observations, and returns an EvaluatedCandidate. A parent and child remain separate evaluated values and must use the same ordered evaluation cases before comparison. TrialRecord remains the evidence authority.

CandidateProposalRequest supplies a selected parent, reference candidate snapshots (inspirations), goal, strategy, history, and bounded memory to a proposer. A submitted CandidateProposal carries the child snapshot, mutation summary, and evaluated revision attempt.

The proposer changes scratch only. The application can write canonical workspace state only after selection checks pass and the search gate accepts the child. See Evolution and Agentic Variation.

EvaluationResult — how it scored

EvaluationResult owns the normalised reward and the evidence needed to interpret it:

• mechanical validity;
• per-field or task-specific breakdowns;
• error taxonomy;
• confidence metadata;
• attributable human annotations; and
• registered task-specific evaluation extensions.

A positive reward requires valid, parseable output. Reports use the stored result as the scoring authority.

The verifier remains task-owned. It can retain detailed evidence in a task-specific artefact while the shared result carries the normalised outcome.

Identity vocabulary

Use each identity mechanism for one purpose:

The standard display reference combines a readable key with the final eight hexadecimal characters of its UUID, for example electrical/voltage-drop · 3e8c21aa. A key can change while the UUID stays stable, with the former key retained as an alias. Checksums identify exact bytes.

Schema labels such as v2 or v3 describe document compatibility.

Portable paths

Untrusted or persisted relative paths use PortableRelativePath. It rejects absolute paths, drive-qualified paths, backslashes, empty components, . and .., and platform-reserved components. resolve_below() also checks the final resolved path, including symlinks, against its trusted root. Human-readable keys pass this validation before use as storage paths.

RunManifest — shared trial evidence context

ResolvedRunSpec and RunPlan establish the complete requested condition and planned membership. RunManifest stores execution context shared by the trial records produced for one run. Its schema_version identifies the persisted evidence format. Library code supplies this value.

class RunManifest(FrozenStrictModel):
    schema_version: Literal[2]
    run_id: NonEmptyStr
    experiment_id: NonEmptyStr
    dataset: DatasetRef | None
    source: SourceRef
    agent: AgentConfiguration
    execution_environment: ExecutionEnvironmentRef
    provider_route: ProviderRoute
    expected_authorities: tuple[AuthorityExpectation, ...]
    evaluation_regime: EvaluationRegimeRef | None
    qualification: QualificationRequirement | None

The exact source, agent configuration, environment, provider route, and dataset reference appear once per run. A trial refers to them through run_id.

TrialRecord — one trial result

TrialRecord stores facts that can differ for each planned trial:

class TrialRecord(StrictModel):
    schema_version: Literal[2]
    trial_id: NonEmptyStr
    run_id: NonEmptyStr
    task_id: NonEmptyStr
    attempt: PositiveInt
    execution_status: ExecutionStatus
    evaluation_status: EvaluationStatus
    evidence_status: EvidenceStatus
    started_at: datetime
    completed_at: datetime | None
    input: TrialInput
    output: TrialOutput | None
    evaluation: EvaluationResult | None
    timing: TimingRecord
    cost: CostRecord | None
    authority_evidence: tuple[AuthorityEvidenceRef, ...]
    provider_evidence: ArtifactRef | None
    extension_refs: tuple[TrialExtensionRef, ...]

The retained attempt field is the planned repetition number. TaskAttempt.attempt_id separately identifies one execution attempt.

Three statuses record different parts of completeness:

Publication policy evaluates these facts with the run manifest. A reportable result normally needs completed execution, completed evaluation, a completed verifier, and evidence that is verified or not_required.

Finite lifecycle, Interactive World, adaptation, and proposal-session details use typed extension references. The main trial links to each optional forensic structure. Functional meta-harness results are normal in-memory composition values, and each runtime returns its existing TrialRecord evidence.

A complete Interactive World trial returns one TrialRecord. Each trial groups the world actions and provider sessions for its complete journey. The dam, pump, and Harbor trial functions retain their task-owned authority evidence and provider evidence before temporary session cleanup.

World branch creation materialises child branches through rollout control and the task's branch port. Later application operations execute, evaluate, select, or merge a child. The parent stays unchanged.

Once accepted, a trial record is immutable. Temporary workspaces and internal builders remain replaceable implementation.

One trial can contain several candidate attempts. This is different from experiment repetitions, which create separate planned trials. When selection occurs, the trial retains typed attempt-selection evidence and the official verifier evaluates only the selected output.

Output state

TrialOutput references retained output bytes:

class TrialOutput(StrictModel):
    agent_output: AgentOutput | None
    raw_output: ArtifactRef | None
    conversation: ArtifactRef | None
    trajectory: ArtifactRef | None
    agent_result: dict[str, Any] | None
    artifacts: tuple[TrialArtifactRef, ...]
    terminated: bool
    truncated: bool
    final_reason: NonEmptyStr | None

Termination and truncation are mutually exclusive. Provider completion, truncation, and error facts determine execution status independently of reward.

run_trial() materialises these references before it removes attempt workspaces. A returned record must therefore resolve its retained output and trajectory artefacts after cleanup.

Artefact integrity

Exact retained bytes use one universal reference:

class ArtifactRef(FrozenStrictModel):
    artifact_id: NonEmptyStr
    sha256: Sha256
    size_bytes: PositiveInt
    media_type: NonEmptyStr

The artefact repository verifies size and SHA-256 when it reads the bytes. Domain IDs identify domain objects; ArtifactRef identifies exact retained bytes.

Authority and provider evidence

The authority that produces evidence owns its final evidence document. A trial links to it through an AuthorityEvidenceRef containing the authority kind, protocol, and ArtifactRef.

Provider evidence uses the separate provider_evidence field. One runtime attestation binds to one retained provider evidence manifest. Keyless protocol proof establishes the protocol contract. Live-provider qualification requires live provider evidence.

Provider packages translate provider and tool protocols. Task selection, task intent, scoring, and task-specific behaviour remain with their domain owners.

Cost tracking

CostRecord is the aggregate usage authority. It can record:

• model calls;
• input, output, and cache tokens;
• advisor calls and advisor tokens; and
• estimated cost in USD.

Open-ended provider metadata can retain diagnostics. Reports use CostRecord for aggregate usage.

Trajectory entries

trajectory.jsonl contains one validated TrajectoryEntry per non-empty line, starting with the first line.

Entries can carry the role, content, tool or command details, output, timing, media, and structured metadata for one step. Each object must match the current entry schema.

Provider transcripts remain separate when an external or sealed workflow requires their exact representation.

Finite lifecycle contracts

EvidenceLifecycleSpec declares an ordered set of checkpoints. The task owns instructions, released evidence, submissions, and verification. The host owns the active checkpoint and accepted progression.

Lifecycle run records keep session identity, execution mode, memory visibility, status, usage, and content-bound artefacts. The final trial references that evidence, while lifecycle state remains in its producing record.

LifecycleTrial and LifecycleExecution are in-memory application values. LifecycleTrial carries the planned trial, package and run locations, execution mode, and visibility policy. LifecycleExecution carries canonical lifecycle state plus the agent and tool evidence required for trial construction.

run_lifecycle_trial() converts one execution and its verifier result to the normal protected TrialRecord. Persistence is an explicit optional effect. run_lifecycle_experiment() returns the records that it creates directly to the caller.

One branch can start only from a submitted checkpoint. It inherits the accepted prefix through that checkpoint, reopens the selected checkpoint in a new run, and leaves the parent unchanged. A revisit records read access while preserving lifecycle progression. Every derived run therefore carries its accepted history.

See Finite Lifecycles.

Interactive World contracts

A registered world definition binds an exact executable build and profile to the functions that govern task state and evaluate actions. The episode host runs those decisions within declared limits and records how the episode ends.

A world episode can enter a trial through one verified episode artefact. Shared trial fields reference task-owned snapshots, transitions, and replay facts through that artefact.

See Interactive Worlds.

Design rules

• Validate untrusted, external, persisted, and cross-process data before use.
• Use strict models for repository-owned boundary documents.
• Use lenient models only when an external producer can add fields.
• Keep task-specific payload meaning with the task owner.
• Add a hash only when it protects a named integrity claim.
• Add an ID only for durable identity, correlation, replay, or stale-action detection.