Contracts

This page is for developers who need to read or produce AEC-Bench data. A contract validates information when it crosses a boundary, such as imported task material, provider results, another process, or persisted evidence. Ordinary Python values handle work inside one owner.

The run boundary resolves one requested condition, persists its exact trial set, coordinates execution, records each result, and accounts for the complete observed set:

Artefact tasks, Interactive Worlds, and finite lifecycles use the same run and trial identities. Each execution family retains its task-owned evidence and binds the final result to its planned trial.

TaskDefinition — what to solve

A TaskDefinition is the validated description of one artefact or workspace task. It carries task identity, lifecycle, visibility, instruction, environment, verifier, tools, limits, and metadata.

The task directory supplies the public files:

task/
├── task.toml
├── instruction.md
├── environment/
│   └── Dockerfile
└── tests/
    ├── test.sh
    └── verify.py

Task-specific output meaning remains with the task and its verifier.

Executable Interactive Worlds use WorldTask; static artefact tasks use TaskDefinition. Both enter the same planning, dataset, evidence, and reporting layers through their own task runtimes.

WorldTask — one world objective

WorldTask binds one provider-neutral instruction to an exact WorldBuildRef and InteractiveWorldProfileRef. It carries task identity, a derived semantic revision, normal catalogue metadata, and tags. The world runtime owns task state, provider configuration, and execution state.

A file-backed world task contains instruction.md and world.toml. The TOML document has exact [world] and [profile] references plus registered [metadata]. load_world_task() rejects stale build or profile content and metadata that does not match the registered profile.

The same world task can enter a persisted RunPlan, direct run_world_experiment() composition, or a dataset manifest entry with task_kind: "world".

Lifecycle and visibility

Task catalogue status and evidence progression are separate concepts.

Lifecycle and visibility are explicit. A missing value is invalid. New selections normally use active, public tasks; other permitted states require an explicit selection or visibility context. Proposed and retired tasks cannot start new work.

AgentOutput — what came back

An agent harness returns a minimal AgentOutput envelope:

class AgentOutput(StrictModel):
    status: AgentOutputStatus   # completed | partial | failed | empty
    output_path: NonEmptyStr
    output_format: NonEmptyStr
    error_message: str | None = None

The harness reports completion and location. Evaluation owns parsing and scoring the task-specific output.

ResolvedRunSpec and RunPlan — what will run

ResolvedTaskInstance combines the validated TaskDefinition with the exact task directory and resolved environment, verifier, and tool paths. It is the direct task input to artefact-task execution.

ResolvedRunSpec records the requested run condition: exact task releases, agent conditions, compute settings, repetitions, verification, evaluation, visibility, execution policy, and provider route.

RunPlan expands that condition into an ordered set of PlannedTrial values. Each trial carries its UUIDv7 identity, readable key, ordinal, run identity, task release, agent condition, compute configuration, repetition, execution family, attempt recipe, and evaluation profile. World and lifecycle trials also carry their exact family release.

EvidenceRunStore persists the resolved specification, draft plan, ready plan, and portable trial evidence. OperationalStore owns mutable SQLite coordination for runs, plans, work items, attempts, backend submissions, and leases. A ready plan has contiguous ordinals, unique trial identities and keys, complete task and agent coverage, and summary counts that match its trial set.

Execution control contracts

ExecutionPolicy records the resolved global and scoped scheduler limits, lease timing, priority ageing, and retry policy. A TrialWorkItem binds one planned trial to its execution family, backend, provider and model routes, resource class, priority, and current state.

An Attempt is one execution try. A Lease records worker ownership for a bounded time. BackendSubmission records the external submission and observed backend state. After collection, an AttemptReceipt records process status, cancellation, resource usage, output references, verifier and authority evidence, failure classification, and reconciliation state.

TrialFinalization points to the published TrialRecord selected from the attempt receipts. RunProgress is a read-only projection of the authoritative RunPlan and mutable operational rows. See Execution Control for the scheduler and recovery workflow.

TaskAttempt — one isolated agent execution

Each call to runtime.run_once() returns one TaskAttempt. The attempt records its unique attempt_id, optional parent_attempt_id, workspace, output, timing, and runtime diagnostics.

An AttemptRecipe receives the tracked AttemptRunner. It can request one attempt, retry from a parent attempt, or produce candidates for selection. The outer trial runner retains verifier and reward access. Evolution candidates and execution attempts use separate identities.

TaskAttempt is an internal execution result. TrialRecord is the selected, verified, and reportable trial result.

RunAccounting — the complete result set

TrialAccountingObservation assigns one typed outcome to a planned trial: succeeded, failed, cancelled, timed_out, invalid, or missing.

RunAccounting compares those observations with the exact ResolvedRunSpec and RunPlan. It records accepted, missing, invalid, duplicate, conflicting, and unexpected trial identities. Accepted records remain in plan order, and quarantined records remain separate from aggregate input.

The resulting run status is complete, complete_with_failures, cancelled, incomplete, or invalid. See Runs and Plans for the complete workflow.

Evolution candidates — exact workspace and evidence

WorkspaceSnapshot.candidate_id identifies a specific prompt-and-skill candidate. It is separate from trial and attempt identity.

CandidateChecks plans a candidate-independent batch, runs it, optionally enriches its observations, and returns an EvaluatedCandidate. A parent and child remain separate evaluated values and must use the same ordered evaluation cases before comparison. TrialRecord remains the evidence authority.

CandidateProposalRequest supplies a selected parent, reference candidate snapshots (inspirations), goal, strategy, history, and bounded memory to a proposer. A submitted CandidateProposal carries the child snapshot, mutation summary, and evaluated revision attempt. An abstained, budget-exhausted, or cancelled proposal carries no child material.

The proposer changes scratch only. The application can write canonical workspace state only after selection checks pass and the search gate accepts the child. See Evolution and Agentic Variation.

EvaluationResult — how it scored

EvaluationResult owns the normalised reward and the evidence needed to interpret it:

• mechanical validity;
• per-field or task-specific breakdowns;
• error taxonomy;
• confidence metadata;
• attributable human annotations; and
• registered task-specific evaluation extensions.

A positive reward requires valid, parseable output. Reports use the stored result as the scoring authority.

The verifier remains task-owned. It can retain detailed evidence in a task-specific artefact while the shared result carries the normalised outcome.

Identity vocabulary

Use each identity mechanism for one purpose:

The standard display reference combines a readable key with the final eight hexadecimal characters of its UUID, for example electrical/voltage-drop · 3e8c21aa. The suffix is not a separate identity. A key can change while the UUID stays stable, with the former key retained as an alias. Checksums identify exact bytes, not domain objects.

Schema labels such as v2 or v3 describe compatibility. They do not replace the version of a task, dataset, or run.

Portable paths

Untrusted or persisted relative paths use PortableRelativePath. It rejects absolute paths, drive-qualified paths, backslashes, empty components, . and .., and platform-reserved components. resolve_below() also checks the final resolved path, including symlinks, against its trusted root. Human-readable keys are not used as storage paths without this validation boundary.

RunManifest — shared trial evidence context

ResolvedRunSpec and RunPlan establish the complete requested condition and planned membership. RunManifest stores execution context shared by the trial records produced for one run. Its schema_version identifies the persisted evidence format. Library code supplies this value.

class RunManifest(FrozenStrictModel):
    schema_version: Literal[2]
    run_id: NonEmptyStr
    experiment_id: NonEmptyStr
    dataset: DatasetRef | None
    source: SourceRef
    agent: AgentConfiguration
    execution_environment: ExecutionEnvironmentRef
    provider_route: ProviderRoute
    expected_authorities: tuple[AuthorityExpectation, ...]
    evaluation_regime: EvaluationRegimeRef | None
    qualification: QualificationRequirement | None

The exact source, agent configuration, environment, provider route, and dataset reference appear once per run. A trial refers to them through run_id.

TrialRecord — one trial result

TrialRecord stores facts that can differ for each planned trial:

class TrialRecord(StrictModel):
    schema_version: Literal[2]
    trial_id: NonEmptyStr
    run_id: NonEmptyStr
    task_id: NonEmptyStr
    attempt: PositiveInt
    execution_status: ExecutionStatus
    evaluation_status: EvaluationStatus
    evidence_status: EvidenceStatus
    started_at: datetime
    completed_at: datetime | None
    input: TrialInput
    output: TrialOutput | None
    evaluation: EvaluationResult | None
    timing: TimingRecord
    cost: CostRecord | None
    authority_evidence: tuple[AuthorityEvidenceRef, ...]
    provider_evidence: ArtifactRef | None
    extension_refs: tuple[TrialExtensionRef, ...]

The retained attempt field is the planned repetition number. TaskAttempt.attempt_id separately identifies one execution attempt.

Three statuses record different parts of completeness:

Publication policy evaluates these facts with the run manifest. A reportable result normally needs completed execution, completed evaluation, a completed verifier, and evidence that is verified or not_required.

Finite lifecycle, Interactive World, adaptation, and proposal-session details use typed extension references. The main trial links to each optional forensic structure. Functional meta-harness results are normal in-memory composition values, and each runtime returns its existing TrialRecord evidence.

A complete Interactive World trial returns one TrialRecord. A world action and a provider session are lower-level events, not trials. The dam, pump, and Harbor trial functions retain their task-owned authority evidence and provider evidence before temporary session cleanup.

World branch creation materialises child branches through rollout control and the task's branch port. Later application operations execute, evaluate, select, or merge a child. The parent stays unchanged.

Once accepted, a trial record is immutable. Temporary workspaces and internal builders remain replaceable implementation.

One trial can contain several candidate attempts. This is different from experiment repetitions, which create separate planned trials. When selection occurs, the trial retains typed attempt-selection evidence and the official verifier evaluates only the selected output.

Output state

TrialOutput references retained output bytes:

class TrialOutput(StrictModel):
    agent_output: AgentOutput | None
    raw_output: ArtifactRef | None
    conversation: ArtifactRef | None
    trajectory: ArtifactRef | None
    agent_result: dict[str, Any] | None
    artifacts: tuple[TrialArtifactRef, ...]
    terminated: bool
    truncated: bool
    final_reason: NonEmptyStr | None

Termination and truncation are mutually exclusive. Provider completion, truncation, and error facts determine execution status independently of reward.

run_trial() materialises these references before it removes attempt workspaces. A returned record must therefore resolve its retained output and trajectory artefacts after cleanup.

Artefact integrity

Exact retained bytes use one universal reference:

class ArtifactRef(FrozenStrictModel):
    artifact_id: NonEmptyStr
    sha256: Sha256
    size_bytes: PositiveInt
    media_type: NonEmptyStr

The artefact repository verifies size and SHA-256 when it reads the bytes. Domain IDs identify domain objects; ArtifactRef identifies exact retained bytes.

Authority and provider evidence

The authority that produces evidence owns its final evidence document. A trial links to it through an AuthorityEvidenceRef containing the authority kind, protocol, and ArtifactRef.

Provider evidence uses the separate provider_evidence field. One runtime attestation binds to one retained provider evidence manifest. Keyless protocol proof establishes the protocol contract. Live-provider qualification requires live provider evidence.

Provider packages translate provider and tool protocols. Task selection, task intent, scoring, and task-specific behaviour remain with their domain owners.

Cost tracking

CostRecord is the aggregate usage authority. It can record:

• model calls;
• input, output, and cache tokens;
• advisor calls and advisor tokens; and
• estimated cost in USD.

Open-ended provider metadata can retain diagnostics. Reports use CostRecord for aggregate usage.

Trajectory entries

trajectory.jsonl contains one validated TrajectoryEntry per non-empty line, starting with the first line.

Entries can carry the role, content, tool or command details, output, timing, media, and structured metadata for one step. Each object must match the current entry schema.

Provider transcripts remain separate when an external or sealed workflow requires their exact representation.

Finite lifecycle contracts

EvidenceLifecycleSpec declares an ordered set of checkpoints. The task owns instructions, released evidence, submissions, and verification. The host owns the active checkpoint and accepted progression.

Lifecycle run records keep session identity, execution mode, memory visibility, status, usage, and content-bound artefacts. The final trial references that evidence, while lifecycle state remains in its producing record.

LifecycleTrial and LifecycleExecution are ordinary in-memory application values, not persisted schemas. LifecycleTrial carries the planned trial, package and run locations, execution mode, and visibility policy. LifecycleExecution carries canonical lifecycle state plus the agent and tool evidence required for trial construction.

run_lifecycle_trial() converts one execution and its verifier result to the normal protected TrialRecord. Persistence is an explicit optional effect. run_lifecycle_experiment() returns the records that it creates directly to the caller.

One branch can start only from a submitted checkpoint. It inherits the accepted prefix through that checkpoint, reopens the selected checkpoint in a new run, and leaves the parent unchanged. A revisit records read access while preserving lifecycle progression. Every derived run therefore carries its accepted history.

See Finite Lifecycles.

Interactive World contracts

A registered world definition binds an exact executable build and profile to task-owned state, observation, action, transition, and evaluation functions. The episode host adds decisions, recording, limits, termination, and truncation.

A world episode can enter a trial through one verified episode artefact. Shared trial fields reference task-owned snapshots, transitions, and replay facts through that artefact.

See Interactive Worlds.

Design rules

• Validate untrusted, external, persisted, and cross-process data before use.
• Use strict models for repository-owned boundary documents.
• Use lenient models only when an external producer can add fields.
• Keep task-specific payload meaning with the task owner.
• Add a hash only when it protects a named integrity claim.
• Add an ID only for durable identity, correlation, replay, or stale-action detection.