Deployment

Use Harbor when an experiment runs in controlled containers or remote sandboxes. AEC-Bench turns the experiment configuration into a Harbor job, waits for it to finish, and imports validated results into the evidence store.

The public aec-bench run flow is synchronous. The command returns after Harbor finishes and the import step completes.

Persisted local artefact plans use the scheduler control path. run start and run resume read a ready plan, create work items, and execute through the local artefact adapter. The synchronous Harbor command remains the path for a direct Harbor job.

Where Harbor sits

Harbor is the execution boundary around the task, agent harness, and verifier. Each component retains its role inside that boundary.

For environment selection and local execution, see Backends.

Install execution support

Install the Harbor extra before using aec-bench run:

pip install "aec-bench[execution]"

Morph also needs its provider extra:

pip install "aec-bench[execution,morph]"

Keep provider credentials in environment variables or the provider's authentication store. Do not put them in the experiment manifest.

Dispatch and import

A live run follows one sequence:

1. Validate the manifest and selected backend.
2. Resolve the selected tasks and build the trial plan.
3. Write the Harbor job configuration.
4. Invoke uv run harbor run -c <generated-config>.
5. Locate the Harbor job directory.
6. Run the verifier and optional post-verifier reviewer.
7. Validate and retain the completed trial records.

Start the sequence through the supported AEC-Bench command:

uv run aec-bench run --config experiment.yaml

The command reports the job directory, imported trial count, and duplicate count.

Plan-bound Harbor import

HarborExperimentDispatcher writes one-trial Harbor configurations and can write a HarborTrialTransport sidecar for a persisted RunPlan. The transport record carries the planned AEC-Bench trial UUID with the Harbor job and trial names.

HarborImportExperimentRunner.import_harbor_plan() reads that mapping, restores the planned identity, and validates the returned task release, agent condition, compute backend, execution family, and attempt.

HarborImportReconciliation reports expected, observed, missing, duplicate, unexpected, and accepted trial identities. Accepted records retain Harbor job and trial names as execution metadata and use the AEC-Bench UUID as their trial identity.

The scheduler-facing Harbor adapter is a Python composition boundary. It maps one planned trial to one backend submission and returns an AttemptReceipt and TrialFinalization for the shared execution layer. Harbor remains responsible for its environment and job protocol.

Configuration

The experiment manifest selects the Harbor environment and local orchestration concurrency:

compute:
  backend: modal
  resource_limits:
    n_concurrent_trials: 4
  timeout_override: 600

Harbor service configuration stays with Harbor and the selected provider. The public manifest records the benchmark choice: backend, concurrency, and optional timeout override.

Use a dry run for planning: inspect the selected tasks and exact trial plan.

uv run aec-bench run --config experiment.yaml --dry-run

A dry run proves planning only. A live run exercises provider access, environment creation, agent execution, verifier execution, and import.

Import an existing job

Use aec-bench import when a completed Harbor job directory already exists:

uv run aec-bench import jobs/<harbor-job-id>

The importer reads the Harbor results and trial artefacts, validates their task and execution records, and writes accepted TrialRecord values. Re-importing identical records reports duplicates. A conflicting identity returns an import mismatch and preserves the accepted record.

Provider completion, agent failure, verifier evidence, and reviewer evidence remain distinct. Imported trials record execution_status, evaluation_status, and evidence_status separately. The verifier result establishes the score only after execution and evaluation complete and required evidence is valid.

Move a completed run

Export one accepted run and its retained artefacts as a deterministic package:

uv run aec-bench run export run-001 --output run-001.tar.zst

Import the package into another evidence store:

uv run aec-bench run import run-001.tar.zst

.tar.zst is a tar archive compressed with Zstandard. The export has deterministic ordering and metadata, so unchanged run content produces the same package bytes. Import validates package paths, manifests, trial records, artefact size, and SHA-256 before it writes accepted data.

Operational boundaries

• Harbor creates the execution environment and runs the generated job.
• AEC-Bench owns planning, import validation, and retained evidence.
• aec-bench run stays attached until Harbor returns.
• Plan-bound import uses ResolvedRunSpec for the requested condition and RunPlan for exact membership.
• Each accepted plan-bound TrialRecord binds to one planned trial UUID.
• RunAccounting supplies completeness and validity for comparison and reporting workflows.