Configuration

Providers

Provider handling depends on the execution path. Agent harnesses backed by PydanticAI infer provider routing from the model string and available credentials. Script-style agents can also build sandbox environment variables from an explicit provider name.

# .env
ANTHROPIC_API_KEY=sk-ant-...
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://example.services.ai.azure.com/openai/v1/
OPENAI_API_KEY=sk-...
TOGETHER_API_KEY=...
AWS_REGION=us-west-2

Keep credentials out of config files. Agent definitions should reference environment variables or rely on the provider SDK's normal credential chain.

For Azure AI Foundry deployments that expose the v1 OpenAI-compatible API, set AZURE_OPENAI_ENDPOINT to the /openai/v1/ endpoint and pass the deployment name as model. For Together AI, prefix the model with together: so routing stays explicit when multiple provider credentials are present:

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model "together:Qwen/Qwen3.7-Max" \
  --harness direct

Agent definitions

An agent entry in an experiment manifest picks the public harness name, model, and optional parameters:

# experiment.yaml
agents:
  - name: claude-sonnet-tool-loop
    harness: tool_loop
    model: claude-sonnet-4-20250514
    parameters:
      max_turns: 12

  - name: gpt4-direct
    harness: direct
    model: gpt-4.1
    parameters:
      max_tokens: 8192

The manifest parser also accepts the older adapter field, but public docs use harness. Supplying both is an error.

The model field supports $ENV_VAR references so pinned model IDs stay out of config:

agents:
  - name: pinned
    harness: tool_loop
    model: $ANTHROPIC_MODEL     # resolved at run time

Harness parameters

Each harness accepts different settings. In an experiment manifest, parameters are passed to the selected harness.

Direct: simple generation settings such as output token budget:

max_tokens = 16384

Tool Loop: bounded turn count:

max_turns = 8

RLM: workspace-level rlm.toml, grouped into guardrails and execution:

# rlm.toml
[guardrails]
token_budget = 100_000
max_iterations = 20
max_subcall_depth = 3
max_budget_usd = 5.00

[execution]
scaffolding = true
context_limit = 1_000_000
compaction_threshold_pct = 0.85
max_parallel_workers = 4

Lambda-RLM: workspace-level lambda-rlm.toml, with template, planner, review, guardrails, and execution settings:

# lambda-rlm.toml
[template]
tier = "dependency_tree"
definition = "report_template.toml"

[planner]
context_window_chars = 200_000
max_branching_factor = 4

[review]
enabled = true
max_retries_per_source = 1
max_supplements_per_section = 1

[guardrails]
token_budget = 500_000

[execution]
max_parallel_workers = 4

RLM and Lambda-RLM configuration files live in the staged task workspace. The experiment manifest selects the harness and model; the workspace TOML controls the harness-specific runtime behaviour.

Lambda-RLM extraction review

Lambda-RLM can record extraction confidence and deterministic self-consistency before deciding whether a section needs review. Set extract.k_candidates above one to fan out candidate extractions, then select a review.trigger:

[extract]
k_candidates = 3
keep_candidates_artifact = true

[uncertainty]
review_joint_threshold = 1.0

[review]
enabled = true
trigger = "both"
consistency_threshold = 0.7

Some triggers need evidence from several candidate extractions. For example, consistency requires k_candidates of at least 2. If the required evidence is missing, the harness reviews every section and records why in its trajectory.

The trajectory also records candidate payloads, confidence, consistency, uncertainty, and the resulting review decision. The task and verifier remain the authority for task meaning and scoring.

How harness names map to Python

Commands and configuration use the term harness. The Python protocol behind each harness is named Adapter:

To add a production harness, implement the adapter in the library and add it to the fixed builder set.

Every harness must preserve the same core responsibilities:

• respect task-declared capabilities;
• write to the requested output path;
• retain useful execution evidence;
• report model and advisor usage when available; and
• classify provider, tool, timeout, truncation, and missing-output conditions as failures.

The shared result shape lets the runner compare harnesses without moving task rules or scoring into harness code.