Config

Use this page when you need the exact shape of a configuration file. For explanations and complete workflows, start with Core, Agents, or Advanced.

aec-bench.toml — project root

Defines paths and defaults for a project. Placed at the repo root.

[project]
name = "my-bench"

[paths]
tasks      = "tasks"
templates  = "templates"
seeds      = "seeds"
ledger     = "artefacts/ledger"
feedback   = "artefacts/feedback"
datasets   = "artefacts/datasets"
jobs       = "jobs"

[compute]
backend = "modal"

All paths are relative to the project root. Omitted keys fall back to the defaults above.

The aec-bench config CLI manages per-user JSON path defaults. Project TOML stores path defaults and may contain compute.backend; config-backed runs select the backend from experiment.yaml, and inline runs use --backend.

task.toml — per task

Every task directory has one. A typical file is:

version = "1.0"

[identity]
id = "019c2c7a-5a33-7b8d-a702-8f7f3e8c21aa"
key = "electrical/voltage-drop"
version = 3

[metadata]
lifecycle  = "active"            # proposed | active | deprecated | retired
visibility = "public"            # public | private | holdout
difficulty = "easy"              # easy | medium | hard
category   = "reasoning"
tags       = ["electrical", "AS-NZS-3008"]

[agent]
timeout_sec = 600.0

[verifier]
timeout_sec = 300.0

[environment]
extensions         = []
build_timeout_sec  = 600.0
cpus               = 2
memory_mb          = 4096
storage_mb         = 10240
allow_internet     = true

See Tasks for what lives alongside task.toml in a task directory.

experiment.yaml — per run

A single experiment: which tasks, which agents, how to run them.

experiment_id: exp-20260412-001
name: "Claude vs GPT electrical"
description: "Optional narrative description"

tasks:
  dataset:
    kind: repository
    dataset_id: electrical-core
    source_revision: 0123456789abcdef0123456789abcdef01234567
    manifest_path: artefacts/datasets/manifests/electrical-core/manifest.json
  include_patterns: ["electrical/*"]
  exclude_patterns: []
  domains:      ["electrical"]
  difficulties: ["easy", "medium"]
  lifecycle_filter: ["active"]
  visibility_filter: ["public"]

agents:
  - name: claude-tool-loop
    harness: tool_loop
    model: claude-sonnet-4-20250514
    parameters:
      max_turns: 12
    system_prompt_file: prompts/electrical.md   # optional

  - name: gpt4-direct
    harness: direct
    model: gpt-4.1
    parameters:
      max_tokens: 8192

compute:
  backend: modal
  timeout_override: 1200
  resource_limits:
    memory_mb: 8192

reviewer:                         # optional post-verifier LLM stage
  enabled: true
  required: true
  models:
    - name: primary-reviewer
      model: gpt-4.1-mini
      provider: auto
  fail_on_error: false

repetitions: 3
disable_verification: false

The model field supports $ENV_VAR expansion. The manifest parser accepts harness as the public synonym for the internal adapter field; generated YAML may still contain adapter.

The runner applies dataset task IDs, patterns, domain, and difficulty as intersections. A dataset-backed experiment contains an exact repository or bundle reference. Use dataset config to resolve a human label before you write the experiment. Run dataset validate first and inspect the dry-run plan.

run plan --config resolves the manifest into a persisted ResolvedRunSpec. The saved condition contains exact task releases, agent conditions, compute and repetition settings, verification and reviewer settings, visibility, and execution policy. Environment-backed credentials remain named environment references in the resolved condition.

The resolved ExecutionPolicy includes scheduler limits, retry policy, lease timing, priority ageing, and scoped limits. The CLI planning path resolves max_concurrency to 1; this policy is persisted for inspection and Python composition.

compute.timeout_override sits alongside resource_limits and overrides the task's agent timeout.

For Harbor jobs, these fields control concurrency and live viewing:

Agent limits sit alongside harness and model, outside parameters. Each limit must be at most the overall trial limit. Members of a shared group must set the same limit. See concurrency examples.

Live Stream uses public artefact tasks on Docker or Daytona. run --stream and run --no-stream override the manifest setting. See Harbor viewer setup.

The optional reviewer block runs after deterministic verification:

Reviewer findings are separate from verifier rewards and human annotations.

Harness configs

RLM and Lambda-RLM read authored files from the staged task workspace. Experiment YAML parameters override matching TOML values; host request limits can only tighten the resolved limits. Unknown keys and unsupported modes fail validation. See configuration precedence and the RLM guide.

rlm.toml

This example enables a guided report with bounded context and model work:

[template]
definition = "report_template.toml"
source_mapping = "sources.toml"
validation_rules = "checks.toml"

[guardrails]
token_budget = 100_000
max_iterations = 20
max_subcall_depth = 1

[execution]
scaffolding = true
context_limit = 128_000
compaction_threshold_pct = 0.85
hard_ceiling_pct = 0.95
max_parallel_workers = 4

Optional execution.compaction_model and execution.subcall_model select secondary model routes. Omit unset optional keys; both model routes default to the main model. Input-context limits must be positive, with 0 < compaction_threshold_pct < hard_ceiling_pct <= 1.

lambda-rlm.toml

Lambda-RLM reads lambda-rlm.toml, then falls back to rlm.toml, then condition values alone. Use separate compatible files when a task supports both harnesses.

[template]
definition = "report_template.toml"
source_mapping = "sources.toml"
validation_rules = "checks.toml"

[planner]
context_window_chars = 100_000
max_branching_factor = 20

[review]
enabled = true
trigger = "always"
max_retries_per_source = 1
max_supplements_per_section = 1

[execution]
max_parallel_workers = 4

[guardrails]
token_budget = 100_000

review.confidence_threshold defaults to 0.6 and review.consistency_threshold to 0.7. consistency and both require extract.k_candidates > 1; a single-sample configuration fails before model work.

Report assets

template.definition resolves relative to the harness file, or the workspace for condition-only configuration. source_mapping, validation_rules, and explicit [boilerplate].path values resolve relative to the template. All resolved paths must stay inside the actor workspace, including through symlinks.

A report template declares its sections in order:

# report_template.toml
[[sections]]
id = "findings"
title = "Findings"
input_mapping = ["inspection"]
writing_guidance = ["State the observed count and cite the source ID."]
fields = [{ name = "text", dtype = "str", required = true }]

[[sections]]
id = "summary"
title = "Summary"
depends_on = ["findings"]
fields = [{ name = "text", dtype = "str", required = true }]

Field types are str, int, float, bool, list, dict, and table. Source priorities use input_mapping = { sources = ["inspection"], priority = { inspection = 1 } }; lower numbers mean higher authority. Priorities can only refer to mapped sources. Section IDs must be unique, and dependencies must name existing sections in an acyclic graph.

The mapping gives source IDs an explicit public path:

# sources.toml
[sources.inspection]
path = "documents/inspection.txt"
visibility = "public"

[sections.findings]
sources = ["inspection"]

Public writing rules have an ID, kind, message, and optional section and field scope. Error-level pattern failures block acceptance; warnings are advisory. A prose rule remains unchecked by deterministic validation; a model review can supply separate findings.

# checks.toml
visibility = "public"

[[rules]]
id = "observed-count"
kind = "required_pattern"
pattern = "12"
message = "State the recorded count of 12."
severity = "error"

Pattern kinds are required_pattern and forbidden_pattern; patterns must compile. Required fields and types are checked independently from these rules. The synthetic example also includes a public rubric with weighted dimensions, criterion categories, and reference scope.

Section candidates

This Lambda-RLM example generates three drafts for the summary and synthesises one result:

[fill_section]
k_candidates = 3
temperature = 1.0
tournament_mode = "synthesis"
apply_to_sections = ["summary"]

[fill_section.synthesis]
synthesiser_model = "anthropic:claude-sonnet-4-6"
synthesis_mode = "plain"
max_input_tokens = 16_000
max_output_tokens = 2_000

Choose a synthesiser_model supported by the configured provider route. Secondary model limits are capped by the provider client's per-call maximum. All candidate and synthesis calls count towards the run budget.

At k_candidates = 1, the harness generates one draft per section. To generate alternatives and combine them, increase K and set tournament_mode = "synthesis" with synthesis_mode = "plain".

Use apply_to_sections to select sections; an empty list applies synthesis to all of them. Bound parallel work with execution.max_parallel_workers. See candidate controls for extraction samples and whole-attempt --best-of.

Fixed constitutional parameters

Set constitution.path to a declared constitution file, or supply inline parameters. RLM accepts its parameter tables directly under constitution:

# rlm.toml
[constitution.information_minimality]
default_threshold = 2000
search_threshold = 10000
preview_length = 200
truncation_strategy = "metadata"

[constitution.state_persistence]
preserve_variables = true
preserve_scratchpad = true
compaction_strategy = "llm_summary"

[constitution.source_fidelity]
require_source_tracing = true
tbd_placeholder = "[TBD]"
gap_framing = "exclude"

Lambda-RLM uses constitution.inline and supports information previews and source fidelity:

# lambda-rlm.toml
[constitution.inline.information_minimality]
default_threshold = 2000
search_threshold = 10000
preview_length = 200
truncation_strategy = "head"

[constitution.inline.source_fidelity]
require_source_tracing = true
tbd_placeholder = "[TBD]"
gap_framing = "tbd"

Choose the controls supported by the selected harness:

RLM preserves task instructions and system policy under each state-persistence mode, together with report state and command bindings.

In a separate constitution file, use the parameter tables at the top level. Lambda-RLM uses the information-minimality and source-fidelity parameters shown above.

evolution.yaml

Drives the evolution loop (see Evolution and Agentic Variation):

workspace_path: workspaces/voltage-drop-evo

models:
  classifier: env:AWS_HAIKU_MODEL_ID
  evolver:    env:AWS_SONNET_MODEL_ID

solver:
  name: solver-v1
  harness: rlm
  model: env:AWS_SONNET_MODEL_ID
  client:
    kind: bedrock
    settings: {}

generate:
  template: voltage-drop
  count: 10
  seed: 999
  difficulties: ["easy", "medium"]

tasks:
  domains: [electrical]
  include_patterns: ["electrical/*"]
  exclude_patterns: []

backend: local
batch_size: 5
max_cycles: 10
improvement_threshold: 0.01
stagnation_window: 5
strategy: hill_climb
structural_weight: 0.3
discipline_balanced: false
timeout: 1800
harness_config: experiment.yaml      # optional

strategy: hill_climb keeps moving towards the current best candidate. strategy: qd keeps strong candidates with different behaviours. structural_weight and discipline_balanced tune that quality-diversity comparison. The solver runs tasks, the classifier adds behavioural observations, and the evolver model proposes workspace changes through AVO.

The YAML composition creates separate selection and revision checks from the selected public tasks. Holdout qualification remains a separate operation.

suite.toml — generated-suite config

Declares how aec-bench generate suite should select templates, allocate instances, and write generated task output:

name = "my-suite"
seed = 20260524

[coverage]
difficulties = { easy = 0.3333333333, medium = 0.3333333333, hard = 0.3333333334 }
min_tasks_per_discipline = 3

[templates]
include = ["electrical/*", "civil/*"]
user_dirs = []

[visibility]
mix = { all_given = 0.78, partial = 0.22 }

[tool_mode]
mix = { with_tool = 1.0 }

[instances]
per_task = 3
total_max = 200

[output]
dir = "tasks/generated/my-suite"

The templates.include entries match each template's logical discipline/name path, such as electrical/voltage-drop. generate suite --dry-run returns the planned counts by discipline, difficulty, visibility, and tool mode before writing task instances.

Dataset manifest

Create semantic dataset manifests with aec-bench dataset create:

{
  "schema_version": 2,
  "dataset_id": "electrical-core",
  "description": "Cable and power-system baseline",
  "tasks": [
    {
      "task_id": "electrical/voltage-drop/basic",
      "path": "tasks/electrical/voltage-drop/basic",
      "task_kind": "artifact"
    }
  ],
  "generation": null
}

The manifest defines which tasks belong to the dataset. schema_version identifies the JSON file format and is supplied by the library. Publication binds the selection to one exact Git source or detached bundle. See Datasets.

swarm.yaml

Swarm config adds parallel agents, a shared budget, and an archive that keeps strong candidates with different behaviours:

task:
  workspace: ./workspaces/my-swarm
  task_path: tasks/electrical/voltage-drop

agents:
  count: 4
  default_model: au.anthropic.claude-sonnet-4-6

budget:
  max_cost_usd: 20.0
  eval_budget_usd: 5.0
  wind_down_threshold: 0.8
  final_threshold: 0.95

evaluation:
  timeout: 300
  backend: local

evolution:
  batch_size: 1
  improvement_threshold: 0.01

heartbeat:
  pivot_after: 5

Each worker uses AVO to propose a candidate, while shared selection checks and the manager own scoring and archive effects. See Swarm for the runtime model and Agentic Variation for the proposer boundary.

Prime environment args

Prime export commands take selection flags directly. Hosted eval and training commands can forward environment arguments into load_environment(...):

uv run aec-bench prime eval \
  --remote-env owner/aec-suite \
  --hosted \
  --model "Qwen/Qwen3.5-4B" \
  --split eval \
  --difficulty medium \
  --harness stateful \
  --env-num-examples 10 \
  --seed 20260509 \
  --env-arg max_turns=20

Values passed through --env-arg KEY=VALUE are parsed as JSON where possible, so numbers and booleans keep their types.

Generated Prime environments apply task selection in this order: difficulty filter, deterministic split selection, optional seeded shuffle, then num_examples. Use split=all when you intentionally want the full exported slice.

configs/endpoints.toml

In Prime Lab workspaces, endpoint aliases live in configs/endpoints.toml and are referenced by Prime commands and generated experiment configs. Keep provider credentials in environment variables or the provider's own auth store.