Backends

aec-bench run builds a Harbor job, dispatches it through Harbor, waits for completion, and imports the results into the ledger. --backend selects the Harbor execution environment.

Use aec-bench run-local for fast local development. Use aec-bench run when the task needs a controlled container or remote sandbox. Both commands plan and score trials in the same way; only the execution environment changes.

Execution paths

Both paths use run_experiment() and run_trial() for ordinary artefact tasks. They use the selected agent harness and the same task-owned verifier, which defines correctness in every environment.

Harbor also has a scheduler adapter at the Python composition boundary. It maps TrialWorkItem and Attempt values to Harbor submissions, then maps collected results back to the planned trial before finalization. Interactive World and finite lifecycle adapters use the same scheduler contracts while keeping their task-owned execution rules.

What a backend does

A backend starts the environment and translates the provider's model and tool protocols. AEC-Bench selects the tasks, and the task verifier defines correctness.

Provider-free tests prove the protocol shape. Live qualification requires evidence from a real run through the provider route.

Available Harbor environments

aec-bench run --backend accepts five values:

Install the Harbor execution extra before using these environments:

pip install "aec-bench[execution]"

Harbor owns the native modal, docker, e2b, and daytona environment implementations. Their credentials and service configuration follow the Harbor and provider setup. AEC-Bench validates the selected backend before dispatch; other values are rejected.

Selecting an environment

For a one-task run, pass the backend directly:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model "<model-id>" \
  --backend docker

For a manifest-driven run, use compute.backend:

compute:
  backend: modal
  resource_limits:
    n_concurrent_trials: 4
  timeout_override: 600

n_concurrent_trials configures the local Harbor orchestrator. timeout_override applies an agent timeout override to the generated Harbor job.

Use --dry-run to check task selection, the generated Harbor configuration, agent options, and environment resource policies before starting a job:

uv run aec-bench run --config experiment.yaml --dry-run

Limit concurrent agent work

Use n_concurrent_trials for the overall job limit and n_concurrent for each agent's active execution phases. Agents that share a provider allowance can use the same concurrency_group:

agents:
  - name: tool-loop
    harness: tool_loop
    model: "<model-id>"
    n_concurrent: 2
    concurrency_group: provider-account
  - name: rlm
    harness: rlm
    model: "<model-id>"
    n_concurrent: 2
    concurrency_group: provider-account
compute:
  backend: docker
  resource_limits:
    n_concurrent_trials: 8

This job can have eight active trials, with at most two agent phases across the shared group. Environment setup and verification use the overall limit. Every group member needs the same positive n_concurrent value, at most the overall limit. Omit concurrency_group to give an agent its own allowance.

These settings limit active agent phases within one Harbor job. A harness can make several provider requests during one phase; allow for those calls when choosing a cap.

Follow a job

The CLI reports environment setup, agent execution, verification, and attempt completion as the job runs. Harbor can retry an attempt that ends with an error. Final trial records enter the ledger after the job completes.

For messages, tool calls, and sandbox files, open the Harbor viewer. Live Stream uses Docker or Daytona with a public artefact task. Set compute.stream: true in the manifest, or pass --stream to run.

Morph Cloud through Harbor

Morph uses the same Harbor dispatch and import path as the native environments, with an AEC-Bench environment binding that translates Harbor operations into Morph Cloud calls.

Install both optional extras and set MORPH_API_KEY:

pip install "aec-bench[execution,morph]"
export MORPH_API_KEY="<your-key>"

Then select morph as the backend:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model "<model-id>" \
  --backend morph

Harbor still owns task staging, agent invocation, artefact collection, and verifier invocation. The Morph provider binding owns the environment operations.

Local development

run-local creates an isolated workspace for each agent attempt. By default, it runs one attempt. --best-of K runs several candidate attempts inside one trial and selects one output.

After selection, run_trial() runs the verifier once, retains the selected output and trace, removes temporary workspaces unless --keep-workspace is set, and imports the result. This local path is useful for harness iteration and verifier debugging.

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct

run-local executes on the host. Use a Harbor environment when the task must run inside a controlled container or remote sandbox.

Persisted run control is a separate local artefact path. aec-bench run start and aec-bench run resume execute a ready local artefact plan through the scheduler with explicit operational and plan roots. The synchronous aec-bench run command above continues to dispatch a Harbor job and import its completed results.

The installed library exposes the fixed backend set above. Add a new execution platform at the Harbor environment composition boundary and register it as a supported CLI value.