Agent Harnesses

An agent harness is the strategy used to drive a model during a task. It answers practical questions: does the model get one call or many, can it use tools, does it keep persistent state, and is the output assembled by a structured workflow?

Choose a harness by the capabilities the task needs. For example, a task that requires source lookup or shell commands needs a tool-capable harness. The table below shows the seven supported local choices.

Choosing a harness

Python protocol

In Python, each harness implements the internal Adapter protocol. This lets the benchmark runner schedule and compare all harnesses through the same trial pipeline:

class Adapter(Protocol):
    def execute(self, request: AdapterRequest) -> AdapterResult: ...
    def adapter_name(self) -> str: ...
    def resolved_model(self) -> str: ...

AdapterRequest contains the task instruction, optional system prompt, allowed tools, harness configuration, and output target:

class AdapterRequest:
    instruction: str
    system_prompt: str | None
    tools: list[ToolSpec]
    configuration: dict[str, Any]
    output_path: str              # defaults to /workspace/output.jsonl
    output_format: str            # "jsonl", "markdown", "json"

AdapterResult records the outcome. This shortened view shows the fields commonly used for debugging and reporting:

class AdapterResult:
    adapter_name: str
    resolved_model: str
    configuration_record: dict[str, Any]
    agent_output: AgentOutput
    transcript: list[TranscriptEntry]
    failure_kind: AdapterFailureKind | None
    raw_output_text: str | None
    provider_error: str | None
    usage_input_tokens: int | None
    usage_output_tokens: int | None
    # + cache and advisor usage fields

Direct

Direct makes one model call and returns one response. It is the right baseline for self-contained tasks.

result = direct_adapter.execute(
    AdapterRequest(
        instruction="Classify this beam as simply-supported or cantilever...",
        configuration={"max_tokens": 8192},
    )
)

Direct ignores task-declared tools. If the task requires shell execution, source lookup, or iterative correction, use a tool-capable harness instead.

Tool Loop

Tool Loop runs a bounded multi-turn interaction. The model asks for a tool call, the harness executes it, the result goes back to the model, and the loop continues until the model writes the required output or reaches max_turns (default 8).

harness = "tool_loop"
model = "claude-sonnet-4"

[configuration]
max_turns = 12

Tool Loop enforces the task's tool whitelist. A request for a tool that was not declared fails the run with undeclared_tool_request (AdapterFailureKind.UNDECLARED_TOOL_REQUEST in Python). This makes tool access part of the benchmark contract.

The optional advisor is configured as a harness capability, separately from task-declared tools.

Select pydantic_ai when you need the explicit public alias for this PydanticAI-backed path. It uses the same bounded tool-loop strategy and output contract.

A typical turn sequence for the voltage-drop task:

user:      "Calculate voltage drop using bash and the supplied cable data."
assistant: [tool_call: bash, command: "python voltage_calc.py"]
tool:      [stdout: "V_drop = 5.2V (compliant)"]
assistant: [writes /workspace/output.jsonl]

RLM (Recursive Language Model)

RLM is based on Recursive Language Models (Zhang, Kraska, Khattab, 2025). In aec-bench, it gives the model a sandboxed Python REPL, persistent scratchpad, helper functions, and optional compaction for long runs.

# rlm.toml
[guardrails]
token_budget = 100_000
max_iterations = 20
max_subcall_depth = 3

[execution]
context_limit = 1_000_000
compaction_threshold_pct = 0.85

Key features:

• Persistent notes: NOTE("key", value) and RECALL("key") store working data outside the conversation history.
• REPL helpers: HELP, SHOW_VARS, grep, parallel, fill_parallel, FINAL, and FINAL_VAR support file inspection, calculation, report filling, and finalisation.
• Compaction: when context approaches compaction_threshold_pct, older turns can be summarised while preserving variables, scratchpad entries, and template progress.
• Guardrails: token_budget, max_iterations, max_subcall_depth, and optional budget caps keep long runs bounded.

Use RLM when the model needs to inspect documents in chunks, run calculations, preserve intermediate state, or resume from partial progress.

Lambda-RLM

Lambda-RLM follows a fixed, structured report workflow. It is designed for tasks where the output shape is known before the run starts: scopes, compliance reports, design notes, fee proposals, and other templated technical documents.

The high-level phases are:

1. Plan - build an extraction schedule from the report template
2. Extract - pull required facts from source documents
3. Review (optional) - check extraction against contract requirements
4. Generate - produce prose per section
5. Output - assemble the final document

# lambda-rlm.toml
[template]
tier = "dependency_tree"
definition = "report_template.toml"

[planner]
context_window_chars = 200_000
max_branching_factor = 4

[review]
enabled = true
max_retries_per_source = 1
max_supplements_per_section = 1

[guardrails]
token_budget = 500_000

[execution]
max_parallel_workers = 4

Current Lambda-RLM runs can use:

• Report templates with guided sections, composed sections, fields, fragments, and required-output metadata
• Planning passes that seed a compose scratchpad before section generation
• Block-task routing so different section types can use different handlers
• Structure enforcement that validates generated blocks against required fields and retries targeted gaps
• Grounding reports that check whether generated content is supported by declared source material
• Best-of-K synthesis where multiple candidate blocks can be generated and synthesised into a final section

Lambda-RLM trades free-form exploration for repeatable extraction and composition. It is a poor fit for open-ended tasks where the agent must decide the workflow from scratch.

DeepSeek Harness

DeepSeek Harness uses the official Harness SDK and runtime through the deepseek_harness adapter. Install the optional execution support, then select an explicit provider route:

uv sync --extra execution --extra deepseek-harness

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --harness deepseek_harness \
  --model "azure:<deployment-name>" \
  --max-tokens 8192

The current qualified model prefixes are azure: and deepseek:. The adapter retains redacted runtime evidence under logs/deepseek-harness/. Provider credentials remain in the selected environment and are not serialised into the execution bundle.

Use the CLI reference for best-of local attempts. DeepSeek Harness runs through ordinary Harbor experiment manifests.

Prime Agent

Prime Agent is an external-process harness. aec-bench stages the task, starts the upstream executable in JSON mode, and then uses the normal output, verifier, evaluation, and ledger-import path.

pip install aec-bench
# Install upstream Prime Agent separately so prime-agent is on PATH.

aec-bench run-local tasks/<task> \
  --harness prime-agent \
  --model anthropic/<model-id>

Each trial uses isolated Prime configuration and session directories. Ambient skills, extensions, prompt templates, themes, and context files are disabled. Provider credentials can still be inherited by the process.

Prime Agent executes model-generated code with the current user's operating-system permissions. Trial isolation improves reproducibility; it is not a security sandbox. Use an externally contained execution path for untrusted task packages.

The ACP integration also supports bounded sessions for selected Interactive Worlds and finite lifecycles. That path needs the prime-agent extra. See Prime Agent for those boundaries.

Tools

Task-declared tools use the ToolSpec contract:

class ToolSpec(StrictModel):
    name: NonEmptyStr
    source: str             # relative path in the task dir
    description: NonEmptyStr
    returns_image: bool = False

Tool Loop resolves each declared name to a ToolExecutor in its registry:

class ToolExecutor(Protocol):
    def execute(self, tool_name: str, arguments: dict[str, Any]) -> ToolExecutionResult: ...

The built-in bash executor runs shell commands in the staged workspace and returns stdout, stderr, and exit code. Custom tools register by name against the same protocol.

RLM and Lambda-RLM expose different capabilities. RLM works through the Python REPL and injected helpers. Lambda-RLM works through its report template, source mapping, sandbox, review, synthesis, and grounding configuration.

Failure modes

When something goes wrong, failure_kind tells the runner which structural category failed. Python enum names are uppercase; serialized values are lowercase:

The evaluation pipeline treats any failure_kind as a structural failure. The reward is 0.0, regardless of how close the transcript looked.

Transcripts

Runs can emit a structured JSONL trajectory, usually at /workspace/trajectory.jsonl inside the staged workspace. Entries are validated as TrajectoryEntry records:

{"role": "system", "step": 0, "content": "..."}
{"role": "user", "step": 1, "content": "Calculate voltage drop..."}
{"role": "assistant", "step": 2, "content": "I'll run the calculation..."}
{"role": "tool_call", "step": 2, "tool_name": "bash", "command": "python voltage_calc.py"}
{"role": "tool_result", "step": 2, "tool_name": "bash", "stdout": "V_drop = 5.2V", "exit_code": 0}

The first non-empty line is a TrajectoryEntry, and each later non-empty line uses the same schema. Because entries are flushed during execution, a crashed run can still leave a readable partial trace. Trajectories feed trace inspection and behavioural classification (see Evaluation).