Backends

aec-bench run builds a Harbor job, dispatches it through Harbor, waits for completion, and imports the results into the ledger. --backend selects the Harbor execution environment.

Use aec-bench run-local for fast local development. Use aec-bench run when the task needs a controlled container or remote sandbox. Both commands plan and score trials in the same way; only the execution environment changes.

Execution paths

Both paths use run_experiment() and run_trial() for ordinary artefact tasks. They use the selected agent harness and the same task-owned verifier, which defines correctness in every environment.

Harbor also has a scheduler adapter at the Python composition boundary. It maps TrialWorkItem and Attempt values to Harbor submissions, then maps collected results back to the planned trial before finalization. Interactive World and finite lifecycle adapters use the same scheduler contracts while keeping their task-owned execution rules.

What a backend does

A backend starts the environment and translates the provider's model and tool protocols. AEC-Bench selects the tasks, and the task verifier defines correctness.

Provider-free tests prove the protocol shape. Live qualification requires evidence from a real run through the provider route.

Available Harbor environments

aec-bench run --backend accepts five values:

Install the Harbor execution extra before using these environments:

pip install "aec-bench[execution]"

Harbor owns the native modal, docker, e2b, and daytona environment implementations. Their credentials and service configuration follow the Harbor and provider setup. AEC-Bench validates the selected backend before dispatch; other values are rejected.

Selecting an environment

For a one-task run, pass the backend directly:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model "<model-id>" \
  --backend docker

For a manifest-driven run, use compute.backend:

compute:
  backend: modal
  resource_limits:
    n_concurrent_trials: 4
  timeout_override: 600

n_concurrent_trials configures the local Harbor orchestrator. timeout_override applies an agent timeout override to the generated Harbor job.

Use --dry-run for planning: validate task selection and inspect the trial plan.

uv run aec-bench run --config experiment.yaml --dry-run

Morph Cloud through Harbor

Morph uses the same Harbor dispatch and import path as the native environments, with an AEC-Bench environment binding that translates Harbor operations into Morph Cloud calls.

Install both optional extras and set MORPH_API_KEY:

pip install "aec-bench[execution,morph]"
export MORPH_API_KEY="<your-key>"

Then select morph as the backend:

uv run aec-bench run tasks/electrical/voltage-drop \
  --model "<model-id>" \
  --backend morph

Harbor still owns task staging, agent invocation, artefact collection, and verifier invocation. The Morph provider binding owns the environment operations.

Local development

run-local creates an isolated workspace for each agent attempt. By default, it runs one attempt. --best-of K runs several candidate attempts inside one trial and selects one output.

After selection, run_trial() runs the verifier once, retains the selected output and trace, removes temporary workspaces unless --keep-workspace is set, and imports the result. This local path is useful for harness iteration and verifier debugging.

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct

run-local executes on the host. Use a Harbor environment when the task must run inside a controlled container or remote sandbox.

Persisted run control is a separate local artefact path. aec-bench run start and aec-bench run resume execute a ready local artefact plan through the scheduler with explicit operational and plan roots. The synchronous aec-bench run command above continues to dispatch a Harbor job and import its completed results.

The installed library exposes the fixed backend set above. Add a new execution platform at the Harbor environment composition boundary and register it as a supported CLI value.