Configuration

Providers

Provider handling depends on the execution path. Agent harnesses backed by PydanticAI infer provider routing from the model string and available credentials. Script-style agents can also build sandbox environment variables from an explicit provider name.

# .env
ANTHROPIC_API_KEY=sk-ant-...
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://example.services.ai.azure.com/openai/v1/
OPENAI_API_KEY=sk-...
TOGETHER_API_KEY=...
AWS_REGION=us-west-2

Keep credentials out of config files. Agent definitions should reference environment variables or rely on the provider SDK's normal credential chain.

For Azure AI Foundry deployments that expose the v1 OpenAI-compatible API, set AZURE_OPENAI_ENDPOINT to the /openai/v1/ endpoint and pass the deployment name as model. For Together AI, prefix the model with together: so routing stays explicit when multiple provider credentials are present:

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model "together:Qwen/Qwen3.7-Max" \
  --harness direct

Agent definitions

An agent entry in an experiment manifest picks the public harness name, model, and optional parameters:

# experiment.yaml
agents:
  - name: claude-sonnet-tool-loop
    harness: tool_loop
    model: claude-sonnet-4-20250514
    parameters:
      max_turns: 12

  - name: gpt4-direct
    harness: direct
    model: gpt-4.1
    parameters:
      max_tokens: 8192

The manifest parser also accepts the older adapter field, but public docs use harness. Supplying both is an error.

The model field supports $ENV_VAR references so pinned model IDs stay out of config:

agents:
  - name: pinned
    harness: tool_loop
    model: $ANTHROPIC_MODEL     # resolved at run time

Harness parameters

Each harness accepts different settings. In an experiment manifest, parameters are passed to the selected harness.

Direct: simple generation settings such as output token budget:

max_tokens = 16384

Tool Loop: bounded turn count:

max_turns = 8

RLM: workspace-level rlm.toml, grouped into guardrails and execution:

# rlm.toml
[guardrails]
token_budget = 100_000
max_iterations = 20
max_subcall_depth = 3
max_budget_usd = 5.00

[execution]
scaffolding = true
context_limit = 128_000
compaction_threshold_pct = 0.85
max_parallel_workers = 4

Lambda-RLM: workspace-level lambda-rlm.toml, with template, planner, review, guardrails, and execution settings:

# lambda-rlm.toml
[template]
tier = "dependency_tree"
definition = "report_template.toml"

[planner]
context_window_chars = 200_000
max_branching_factor = 4

[review]
enabled = true
max_retries_per_source = 1
max_supplements_per_section = 1

[guardrails]
token_budget = 500_000

[execution]
max_parallel_workers = 4

Report configuration precedence

Place report configuration files in the staged task workspace. RLM reads rlm.toml; Lambda-RLM first looks for lambda-rlm.toml, then falls back to a compatible rlm.toml. You can also supply the complete configuration through condition values.

The experiment manifest selects the harness and model. Nested parameters override the corresponding authored TOML defaults. For example, this agent entry changes the report's total token budget and enables consistency-based extraction review:

agents:
  - name: report-consistency-review
    harness: lambda-rlm
    model: $REPORT_MODEL
    parameters:
      guardrails:
        token_budget: 100000
      extract:
        k_candidates: 3
      review:
        enabled: true
        trigger: consistency
        max_retries_per_source: 1

Host request limits can only tighten the resolved limits. Unknown file keys, unsupported block options, and incompatible modes fail before model requests. The effective configuration retains the resolved settings and public report assets.

guardrails.token_budget counts all model phases, including secondary models and advice. Reaching the limit stops new requests, while parallel calls already running finish and count towards actual usage.

Use max_turns to bound main iterations in RLM or model calls in Lambda-RLM.

The RLM input-context default is 128,000 tokens; choose a value appropriate for the model. See RLM and Lambda-RLM for the report workflow and Harness configs for asset paths, context controls, fixed constitutional parameters, and section synthesis.

Lambda-RLM extraction review

Lambda-RLM can record extraction confidence and deterministic self-consistency before deciding whether a section needs review. Set extract.k_candidates above one to fan out candidate extractions, then select a review.trigger:

[extract]
k_candidates = 3
keep_candidates_artifact = true

[uncertainty]
review_joint_threshold = 1.0

[review]
enabled = true
trigger = "both"
consistency_threshold = 0.7

consistency and both require extract.k_candidates of at least 2. A configuration with only one extraction sample fails validation before model work. extract.k_candidates controls extraction samples; section candidate generation uses fill_section.k_candidates. CLI --best-of runs separate whole attempts.

The trajectory also records candidate payloads, confidence, consistency, uncertainty, and the resulting review decision. The task and verifier remain the authority for task meaning and scoring.

How harness names map to Python

Commands and configuration use the term harness. The Python protocol behind each harness is named Adapter:

To add a production harness, implement the adapter in the library and add it to the fixed builder set.

Every harness must preserve the same core responsibilities:

• respect task-declared capabilities;
• write to the requested output path;
• retain useful execution evidence;
• report model and advisor usage when available; and
• classify provider, tool, timeout, truncation, and missing-output conditions as failures.

The runner compares harnesses through this shared result shape. Task definitions own the rules, and evaluation owns scoring.