Agent Harnesses

An agent harness is the strategy used to drive a model during a task. It answers practical questions: does the model get one call or many, can it use tools, does it keep persistent state, and is the output assembled by a structured workflow?

Choose a harness by the capabilities the task needs. For example, a task that requires source lookup or shell commands needs a tool-capable harness. The table below shows the seven supported local choices.

Choosing a harness

Python protocol

In Python, each harness implements the internal Adapter protocol. This lets the benchmark runner schedule and compare all harnesses through the same trial pipeline:

class Adapter(Protocol):
    def execute(self, request: AdapterRequest) -> AdapterResult: ...
    def adapter_name(self) -> str: ...
    def resolved_model(self) -> str: ...

AdapterRequest contains the task instruction, optional system prompt, allowed tools, harness configuration, and output target:

class AdapterRequest:
    instruction: str
    system_prompt: str | None
    tools: list[ToolSpec]
    configuration: dict[str, Any]
    output_path: str              # defaults to /workspace/output.jsonl
    output_format: str            # "jsonl", "markdown", "json"

AdapterResult records the outcome. This shortened view shows the fields commonly used for debugging and reporting:

class AdapterResult:
    adapter_name: str
    resolved_model: str
    configuration_record: dict[str, Any]
    agent_output: AgentOutput
    transcript: list[TranscriptEntry]
    failure_kind: AdapterFailureKind | None
    raw_output_text: str | None
    provider_error: str | None
    usage_input_tokens: int | None
    usage_output_tokens: int | None
    # + cache and advisor usage fields

Direct

Direct makes one model call and returns one response. It is the right baseline for self-contained tasks.

result = direct_adapter.execute(
    AdapterRequest(
        instruction="Classify this beam as simply-supported or cantilever...",
        configuration={"max_tokens": 8192},
    )
)

Use a tool-capable harness for tasks that require shell execution, source lookup, or iterative correction.

Tool Loop

Tool Loop runs a bounded multi-turn interaction. The model asks for a tool call, the harness executes it, the result goes back to the model, and the loop continues until the model writes the required output or reaches max_turns (default 8).

harness = "tool_loop"
model = "claude-sonnet-4"

[configuration]
max_turns = 12

Tool Loop enforces the task's tool whitelist. A request outside the declared whitelist fails the run with undeclared_tool_request (AdapterFailureKind.UNDECLARED_TOOL_REQUEST in Python). This makes tool access part of the benchmark contract.

The optional advisor is configured as a harness capability, separately from task-declared tools.

Select pydantic_ai when you need the explicit public alias for this PydanticAI-backed path. It uses the same bounded tool-loop strategy and output contract.

A typical turn sequence for the voltage-drop task:

user:      "Calculate voltage drop using bash and the supplied cable data."
assistant: [tool_call: bash, command: "python voltage_calc.py"]
tool:      [stdout: "V_drop = 5.2V (compliant)"]
assistant: [writes /workspace/output.jsonl]

RLM (Recursive Language Model)

RLM is based on Recursive Language Models (Zhang, Kraska, Khattab, 2025). In aec-bench, it gives the model a persistent Python REPL, scratchpad, helper functions, and optional compaction for long runs. Use the execution environment to enforce filesystem isolation.

# rlm.toml
[guardrails]
token_budget = 100_000
max_iterations = 20
max_subcall_depth = 3

[execution]
context_limit = 128_000
compaction_threshold_pct = 0.85

Key features:

• Persistent notes: NOTE("key", value) and RECALL("key") store working data outside the conversation history.
• Context inspection: SHOW_VARS, Python slices, grep, and bounded previews help the agent select relevant parts of large values.
• Model work: declared subcalls and parallel() support bounded delegation and independent operations.
• Guided reports: a loaded template provides START, READ, VALIDATE, FILL, STATUS, and SUBMIT. fill_parallel() shares the same section checks. Free-form runs use FINAL and FINAL_VAR.
• Compaction: when context approaches compaction_threshold_pct, older turns can be summarised while preserving variables, scratchpad entries, and template progress.
• Guardrails: token_budget, max_iterations, max_subcall_depth, and optional budget caps keep long runs bounded.

Use RLM when the model needs to inspect documents in chunks, run calculations, and preserve intermediate state during an attempt. See RLM and Lambda-RLM for context management, report tools, and a worked example.

Lambda-RLM

Lambda-RLM follows a fixed, structured report workflow. It is designed for tasks where the output shape is known before the run starts: scopes, compliance reports, design notes, fee proposals, and other templated technical documents.

The high-level phases are:

1. Plan - build an extraction schedule from the report template
2. Extract - pull required facts from source documents
3. Review (optional) - check extraction against contract requirements
4. Generate - produce prose per section
5. Output - assemble the final document

# lambda-rlm.toml
[template]
tier = "dependency_tree"
definition = "report_template.toml"

[planner]
context_window_chars = 200_000
max_branching_factor = 4

[review]
enabled = true
max_retries_per_source = 1
max_supplements_per_section = 1

[guardrails]
token_budget = 500_000

[execution]
max_parallel_workers = 4

Lambda-RLM runs can use:

• Report templates with guided sections, composed sections, fields, fragments, and required-output metadata
• Planning passes that seed a compose scratchpad before section generation
• Block-task routing so different section types can use different handlers
• Structure enforcement that validates generated blocks against required fields and retries targeted gaps
• Grounding reports that check whether generated content is supported by declared source material
• Candidate synthesis where parallel section drafts are combined using public criteria and permitted references

Lambda-RLM shares accepted section state, public writing rules, rubric guidance, and output assembly with guided RLM. Required checks and section dependencies control completion; the task verifier owns scoring. Optional advice, review, and synthesis count towards aggregate run usage. See the report workflow and configuration reference.

Lambda-RLM trades free-form exploration for repeatable extraction and composition. It is a poor fit for open-ended tasks where the agent must decide the workflow from scratch.

DeepSeek Harness

DeepSeek Harness uses the official Harness SDK and runtime through the deepseek_harness adapter. Install the optional execution support, then select an explicit provider route:

uv sync --extra execution --extra deepseek-harness

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --harness deepseek_harness \
  --model "azure:<deployment-name>" \
  --max-tokens 8192

The current qualified model prefixes are azure: and deepseek:. The adapter retains redacted runtime evidence under logs/deepseek-harness/. Provider credentials remain in the selected environment.

Use the CLI reference for best-of local attempts. DeepSeek Harness runs through ordinary Harbor experiment manifests.

For delegated work, enable subagents in the experiment's agent configuration:

agents:
  - name: deepseek-delegation
    harness: deepseek_harness
    model: "deepseek:<model-id>"
    parameters:
      subagents_enabled: true
      timeout_sec: 1800
      max_tokens: 8192

A child works in a fresh conversation while the parent waits for its result. It shares the parent's model, workspace, and per-request token cap, and uses coding tools or returns text analysis. The parent retains responsibility for final submission and AEC-Bench's world and lifecycle tools.

Delegation is one level deep, with one process timeout covering the parent and children. The completed trace links child sessions to the tool calls that started them, and usage totals include their model calls and reported tokens.

Set subagents_enabled: false, the default, for a single-agent run.

Prime Agent

Prime Agent is an external-process harness. aec-bench stages the task, starts the upstream executable in JSON mode, and then uses the normal output, verifier, evaluation, and ledger-import path.

pip install aec-bench
# Install upstream Prime Agent separately so prime-agent is on PATH.

aec-bench run-local tasks/<task> \
  --harness prime-agent \
  --model anthropic/<model-id>

Each trial uses isolated Prime configuration and session directories. Ambient skills, extensions, prompt templates, themes, and context files are disabled. Provider credentials can still be inherited by the process.

Prime Agent executes model-generated code with the current user's operating-system permissions. Trial directories isolate execution artefacts for reproducibility. Use an externally contained execution path for untrusted task packages.

The ACP integration also supports bounded sessions for selected Interactive Worlds and finite lifecycles. That path needs the prime-agent extra. See Prime Agent for those boundaries.

Tools

Task-declared tools use the ToolSpec contract:

class ToolSpec(StrictModel):
    name: NonEmptyStr
    source: str             # relative path in the task dir
    description: NonEmptyStr
    returns_image: bool = False

Tool Loop resolves each declared name to a ToolExecutor in its registry:

class ToolExecutor(Protocol):
    def execute(self, tool_name: str, arguments: dict[str, Any]) -> ToolExecutionResult: ...

The built-in bash executor runs shell commands in the staged workspace and returns stdout, stderr, and exit code. Custom tools register by name against the same protocol.

RLM exposes a Python REPL with helpers for analysis and report work. Lambda-RLM uses the configured report workflow to extract evidence and assemble the output.

Failure modes

When something goes wrong, failure_kind tells the runner which structural category failed. Python enum names are uppercase; serialized values are lowercase:

The evaluation pipeline treats any failure_kind as a structural failure. The reward is 0.0, regardless of how close the transcript looked.

Transcripts

Runs can emit a structured JSONL trajectory, usually at /workspace/trajectory.jsonl inside the staged workspace. Entries are validated as TrajectoryEntry records:

{"role": "system", "step": 0, "content": "..."}
{"role": "user", "step": 1, "content": "Calculate voltage drop..."}
{"role": "assistant", "step": 2, "content": "I'll run the calculation..."}
{"role": "tool_call", "step": 2, "tool_name": "bash", "command": "python voltage_calc.py"}
{"role": "tool_result", "step": 2, "tool_name": "bash", "stdout": "V_drop = 5.2V", "exit_code": 0}

The first non-empty line is a TrajectoryEntry, and each later non-empty line uses the same schema. Producers that flush entries during execution can retain a readable partial trace after a failure. Prime and DeepSeek convert retained session evidence after execution. See Traces for model-response records, child relationships, and Harbor's ATIF viewer.