Traces
The trajectory
During execution, an agent harness can append to /workspace/trajectory.jsonl, where each non-blank line is a TrajectoryEntry:
{"step": 0, "role": "system", "content": "You are a senior electrical engineer...", "timestamp": "2026-08-15T01:02:03.000Z"}
{"step": 0, "role": "user", "content": "Calculate voltage drop...", "timestamp": "2026-08-15T01:02:03.001Z"}
{"step": 1, "role": "assistant", "content": "I'll compute R and X contributions...", "call_type": "main", "timestamp": "2026-08-15T01:02:04.000Z"}
{"step": 1, "role": "tool_call", "tool_name": "python", "tool_call_id": "calc-1", "command": "python calc.py", "call_type": "main", "timestamp": "2026-08-15T01:02:05.000Z"}
{"step": 1, "role": "tool_result", "tool_name": "python", "tool_call_id": "calc-1", "stdout": "V_drop = 5.2V", "stderr": "", "exit_code": 0, "duration_ms": 142, "call_type": "main", "timestamp": "2026-08-15T01:02:05.142Z"}Use step and role to follow the sequence of events. Tool entries describe the request and its result, so you can inspect a command alongside its output and exit code. Timestamps place events in time, while the retained artefact reference identifies the complete trace.
Model responses and child calls
tool_call_id links a tool request to its result. Additional entry types record the model response and delegated work:
| Role | Evidence |
|---|---|
reasoning | Thinking text or a reasoning summary exposed by the provider |
model_response | Model and provider identity, response ID, and available token counts |
subagent | A child trajectory entry, its child ID, and the parent tool-call ID when recorded |
When the provider exposes reasoning under the model's configured settings, the trace stores it separately from the assistant's answer. Token counts come from reported usage; missing counts remain unspecified.
Child IDs let you follow delegated work even when several children run concurrently. Tool-loop advisors and RLM subcalls use these IDs to group each child's requests, responses, and errors; RLM advisors and Lambda-RLM plan calls use the same structure.
Prime Agent and DeepSeek Harness build their trajectories from retained session evidence after execution.
Shared run identity
Many trials can share the same dataset, agent, source revision, and execution setup. AEC-Bench stores that shared context once in a RunManifest, and each trial refers to it through run_id.
RunManifest records the facts shared by all trials in one run:
run_idandexperiment_id;- an exact source reference;
- the agent and effective configuration;
- the execution environment and provider route;
- an exact dataset reference when the run uses one;
- expected evidence authorities; and
- optional evaluation and qualification requirements.
The trial result
TrialRecord stores facts for one planned trial. The trial can contain one agent attempt or select from several best-of candidates:
class TrialRecord(StrictModel):
schema_version: Literal[2]
trial_id: NonEmptyStr
run_id: NonEmptyStr
task_id: NonEmptyStr
attempt: PositiveInt
execution_status: ExecutionStatus
evaluation_status: EvaluationStatus
evidence_status: EvidenceStatus
started_at: datetime
completed_at: datetime | None
input: TrialInput
output: TrialOutput | None
evaluation: EvaluationResult | None
timing: TimingRecord
cost: CostRecord | None
authority_evidence: tuple[AuthorityEvidenceRef, ...]
provider_evidence: ArtifactRef | None
extension_refs: tuple[TrialExtensionRef, ...]Read TrialInput to establish which task revision and instruction the agent received, including references to staged input files. TrialOutput links to the retained answer and execution evidence through ArtifactRef values.
The attempt field identifies the planned repetition, while attempt_id identifies an internal TaskAttempt candidate. For trials that select among candidates, extension references retain the selection evidence and the trajectory points to the selected attempt.
Result status
Read the execution, evaluation, and evidence statuses independently.
| Field | Meaning |
|---|---|
execution_status | Execution progress or terminal result |
evaluation_status | Whether evaluation was requested and completed |
evidence_status | Whether required authority evidence is ready and valid |
Leaderboard publication normally requires:
execution_status == "completed";evaluation_status == "completed";evaluation.validity.verifier_completed == true; andevidence_statusequal toverifiedornot_required, subject to the run policy.
A failed execution can still have useful diagnostics. Keep each status visible so readers can see execution, evaluation, and evidence completeness independently.
Cost tracking
Use CostRecord to read a trial's total model usage and cost, including cached tokens and advisor work. Provider evidence retains additional metadata for diagnosis.
For a breakdown by model, Harbor records CostRecord.model_usage. Model usage reports show each model's token counts and costs alongside the trial totals.
Report harness evidence
RLM and Lambda-RLM retain the effective report configuration under configuration_record["report_assets"]: the resolved template, public writing rules, public rubric, and source IDs. Retained task artefacts provide source-file integrity.
Use review findings and section candidate traces to follow how an attempt developed its report. A completed report must match the accepted section state; partial runs retain diagnostics that explain why completion stopped.
All main and secondary model calls count towards run usage. Calls already running in parallel can take usage past the stop limit, producing a partial result. Usage remains unknown when a failed provider call omits its usage data.
Retained records
Accepted run manifests, trial records, and retained evidence are immutable. New review or diagnostic detail is stored as another artefact that refers to the existing trial.
Imported Prime rollouts
Hosted Prime eval samples can enter the same run and trial boundary:
uv run aec-bench import-prime-eval <prime-eval-id> \
--experiment prime-eval-medium-statefulThe import keeps the Prime sample and conversation together for review through AEC-Bench's trace and behaviour reports. For benchmark publication, use repeated trials with an exact dataset and source reference, and check the result statuses against the run's publication policy.
Inspect a trace
Start with a trace summary to locate tool errors or understand the run's behavioural classification, then open the retained trace to inspect individual steps. Evaluation reads trajectory.jsonl when available and falls back to conversation.jsonl.
Harbor viewer
Harbor reads an ATIF-v1.8 view at agent/trajectory.json in the trial directory. AEC-Bench derives this file from trajectory.jsonl, including recorded tool links, response metrics, reasoning, and child trajectories. The original JSONL remains available with the trial evidence.
For a live tool-loop view, use a public artefact task on Docker or Daytona. In a source checkout:
uv sync --extra execution --extra local-agents
uv run aec-bench run tasks/electrical/voltage-drop \
--model "<model-id>" --harness tool_loop --backend docker --streamIn another terminal on the same machine:
uv run harbor view jobsOpen the printed viewer URL and select the active trial. The trajectory updates as complete messages and tool calls arrive.
The Stream tab opens the sandbox editor and terminal for use during development. File edits and commands become part of the execution you are observing.
Run the viewer and browser on the same machine. For Daytona, install its Harbor extra and provide credentials in both the run and viewer environments.
For manifest-driven runs, set compute.stream: true. The CLI options --stream and --no-stream override that setting. Completed runs also export available trajectory evidence for later viewing.