Traces

The trajectory

During execution, an agent harness can append to /workspace/trajectory.jsonl. The file starts with a TrajectoryEntry. Every non-blank line uses that schema:

{"step": 0, "role": "system", "content": "You are a senior electrical engineer...", "timestamp": "2026-08-15T01:02:03.000Z"}
{"step": 0, "role": "user", "content": "Calculate voltage drop...", "timestamp": "2026-08-15T01:02:03.001Z"}
{"step": 1, "role": "assistant", "content": "I'll compute R and X contributions...", "call_type": "main", "timestamp": "2026-08-15T01:02:04.000Z"}
{"step": 1, "role": "tool_call", "tool_name": "python", "command": "python calc.py", "call_type": "main", "timestamp": "2026-08-15T01:02:05.000Z"}
{"step": 1, "role": "tool_result", "tool_name": "python", "stdout": "V_drop = 5.2V", "stderr": "", "exit_code": 0, "duration_ms": 142, "call_type": "main", "timestamp": "2026-08-15T01:02:05.142Z"}

Each entry carries step and role. Tool entries can also carry tool_name, command, arguments, stdout, stderr, exit_code, duration_ms, media, and structured metadata. A timestamp records the real step event; the retained artefact reference identifies the trace.

Shared run identity

Many trials can share the same dataset, agent, source revision, and execution setup. AEC-Bench stores that shared context once in a RunManifest, and each trial refers to it through run_id.

RunManifest records the facts shared by all trials in one run:

• run_id and experiment_id;
• an exact source reference;
• the agent and effective configuration;
• the execution environment and provider route;
• an exact dataset reference when the run uses one;
• expected evidence authorities; and
• optional evaluation and qualification requirements.

The local record store keeps the manifest once. Each trial refers to it through run_id.

The trial result

TrialRecord stores facts for one planned trial. The trial can contain one agent attempt or select from several best-of candidates:

class TrialRecord(StrictModel):
    schema_version: Literal[2]
    trial_id: NonEmptyStr
    run_id: NonEmptyStr
    task_id: NonEmptyStr
    attempt: PositiveInt

    execution_status: ExecutionStatus
    evaluation_status: EvaluationStatus
    evidence_status: EvidenceStatus
    started_at: datetime
    completed_at: datetime | None

    input: TrialInput
    output: TrialOutput | None
    evaluation: EvaluationResult | None
    timing: TimingRecord
    cost: CostRecord | None
    authority_evidence: tuple[AuthorityEvidenceRef, ...]
    provider_evidence: ArtifactRef | None
    extension_refs: tuple[TrialExtensionRef, ...]

TrialInput carries the instruction, task revision, task kind, visibility, and optional input-file references. TrialOutput references the retained raw output, conversation, trajectory, and other output artefacts with ArtifactRef.

The attempt field is the planned repetition number. Internal TaskAttempt candidates use separate attempt_id values. When a trial uses candidate selection, its extension references include the retained selection evidence and the trajectory points to the selected attempt.

Result status

Read the execution, evaluation, and evidence statuses independently.

Leaderboard publication normally requires:

1. execution_status == "completed";
2. evaluation_status == "completed";
3. evaluation.validity.verifier_completed == true; and
4. evidence_status equal to verified or not_required, subject to the run policy.

A failed execution can still have useful diagnostics. Keep each status visible so readers can see execution, evaluation, and evidence completeness independently.

Cost tracking

Each trial can record model calls, input and output tokens, cache use, advisor use, and an estimated USD cost. Reports use CostRecord as the aggregate usage authority. Provider-specific metadata can remain in provider evidence when it is needed for diagnosis.

Retained records

Accepted run manifests, trial records, and retained evidence cannot be overwritten. New review or diagnostic detail is stored as another artefact that refers to the existing trial. Source code and temporary workspaces can still change normally.

Imported Prime rollouts

Hosted Prime eval samples can enter the same run and trial boundary:

uv run aec-bench import-prime-eval <prime-eval-id> \
  --experiment prime-eval-medium-stateful

The import retains the Prime sample and conversation as artefacts. It maps provider execution, evaluation, and evidence separately. Imported samples are useful for rollout review. Leaderboard claims still need an exact dataset, reconstructive source, repeated trials, verifier evidence, and the selected publication policy.

Inspect a trace

The evaluation layer loads trajectory.jsonl first and can fall back to conversation.jsonl. Trace summaries expose turn counts, tool errors, the first error, and the behavioural classification described in Classification. The retained trace artefact is the step-by-step evidence surface.