Prime Lab

Prime Lab support is optional. Use it to run AEC-Bench tasks through Prime for hosted evaluation or training. AEC-Bench still supplies the task and decides the reward; Prime supplies the execution and training platform.

What you can export

The available Prime environment depends on what the task needs:

The original task verifier decides reward consistently across Prime environments.

General task exports accept public tasks only. They record each task's content revision and visibility, and keep verifier files outside the actor workspace.

Readiness check

Install the optional integration and check local readiness:

uv sync --extra prime --dev
uv run aec-bench prime doctor
uv run aec-bench prime doctor --check-inference

doctor checks the optional dependency and Prime CLI availability. --check-inference also checks model-list connectivity.

Export

Export one task or a dataset slice as a generated environment package:

uv run aec-bench prime export \
  --name aec-voltage-drop \
  --task electrical/voltage-drop

uv run aec-bench prime export \
  --name aec-electrical-core \
  --dataset electrical-core@public-2026 \
  --harness-mode auto

Generated packages default to prime-rl/environments/. They are local build artefacts; regenerate them from source tasks when changes are required.

With the default --harness-mode auto, plain tasks use the single-turn environment. If any selected task needs the stateful workspace, every task in that generated package uses the stateful environment.

Generated environments accept split, difficulty, num_examples, seed, and harness. Selection applies difficulty, deterministic split, seeded shuffle, then num_examples. Training and evaluation slices are disjoint when enough tasks exist; very small slices remain intact for smoke tests.

Smoke test

Use smoke before pushing or training:

uv run aec-bench prime smoke \
  --name aec-voltage-drop \
  --task electrical/voltage-drop

Add --model to run a one-example prime eval run; omit it to keep the smoke local and package-load focused.

Lifecycle export

Export one or more existing, materialised public evidence-lifecycle packages into a local persistent Verifiers environment:

uv run aec-bench prime export-lifecycle \
  --name aec-evidence-lifecycle \
  --package /absolute/path/to/materialised/public-package \
  --aec-bench-root /absolute/path/to/aec-bench

Repeat --package to include more lifecycles. The generated package keeps absolute references to the source packages and checks their content hashes before each rollout. One rollout owns one complete lifecycle and one persistent conversation; the task lifecycle verifier remains the sole reward authority.

Lifecycle exports support local evaluation. Continual-learning or transfer claims require a separate study and evidence path. Use the general prime export path for hosted task evaluation and training.

Push

Push a generated environment to Prime Hub:

uv run aec-bench prime push \
  --name aec-electrical-core \
  --dataset electrical-core@public-2026 \
  --visibility PRIVATE \
  --owner your-owner

Private visibility is the safe default while validating reward behaviour and task selection.

Push packages created by the general prime export path. Keep prime export-lifecycle packages local.

Hosted eval

Run a hosted eval against an existing Hub environment:

uv run aec-bench prime eval \
  --remote-env your-owner/aec-electrical-core \
  --hosted \
  --model "Qwen/Qwen3.5-4B" \
  --split eval \
  --difficulty medium \
  --harness stateful \
  --env-num-examples 10 \
  --seed 20260509 \
  --num-examples 5 \
  --rollouts-per-example 3 \
  --max-tokens 4096 \
  --eval-name aec-electrical-base-medium

--split, --difficulty, --harness, --env-num-examples, and --seed are forwarded to the generated environment's load_environment(...) function through Prime env args.

Use repeated --difficulty values for mixed slices:

uv run aec-bench prime eval \
  --remote-env your-owner/aec-electrical-core \
  --hosted \
  --model "Qwen/Qwen3.5-4B" \
  --difficulty easy \
  --difficulty medium

Use --env-arg KEY=VALUE for additional load_environment(...) arguments beyond the current first-class CLI flags.

Evaluate a trained adapter

Prime can evaluate either a base model or a trained adapter in the same environment. Here, an adapter is a deployed set of trained model weights. Inference requires an adapter deployment; a training checkpoint records training progress.

List available adapter deployments:

uv run aec-bench prime adapters
uv run aec-bench --json prime adapters

If you already know the deployed adapter id, pass the base model as --model and the deployed adapter id as --adapter-id:

uv run aec-bench prime eval \
  --remote-env your-owner/aec-electrical-core \
  --hosted \
  --model "Qwen/Qwen3.5-4B" \
  --adapter-id uv124zgh7ttg3in94f7jzmv2 \
  --split eval \
  --difficulty medium \
  --harness stateful \
  --env-num-examples 10 \
  --seed 20260509 \
  --num-examples 5 \
  --rollouts-per-example 3 \
  --max-tokens 4096 \
  --eval-name aec-electrical-adapter-medium

AEC-Bench sends Prime the model string <base-model>:<adapter-id>. If you have a training run rather than a deployed adapter ID, let the command resolve the deployment:

uv run aec-bench prime eval \
  --remote-env your-owner/aec-electrical-core \
  --hosted \
  --model "Qwen/Qwen3.5-4B" \
  --adapter-from-run <training-run-id> \
  --adapter-step latest \
  --split eval \
  --difficulty medium \
  --harness stateful \
  --env-num-examples 10 \
  --seed 20260509 \
  --num-examples 5 \
  --rollouts-per-example 3 \
  --max-tokens 4096 \
  --eval-name aec-electrical-adapter-medium

For base-versus-adapter comparisons, keep the environment slug, split, difficulty, harness, seed, number of examples, rollout count, and token budget fixed. Change only the model adapter.

Hosted training

Write a conservative single-slice training config:

uv run aec-bench prime train-config \
  --environment your-owner/aec-electrical-core \
  --output train.toml \
  --model "Qwen/Qwen3.5-0.8B" \
  --split train \
  --difficulty easy \
  --harness stateful \
  --num-examples 50 \
  --max-steps 20

For broad training suites, generate a multi-environment config with one difficulty-filtered view per ratio. This keeps the environment package fixed while Prime samples from easy, medium, and hard slices according to [buffer].env_ratios:

uv run aec-bench prime train-config \
  --environment your-owner/aec-release-train \
  --output configs/rl/aec-filtered-ratios.toml \
  --model "Qwen/Qwen3.5-9B" \
  --split all \
  --harness stateful \
  --difficulty-ratio easy=0.45 \
  --difficulty-ratio medium=0.40 \
  --difficulty-ratio hard=0.15 \
  --max-steps 50 \
  --batch-size 64 \
  --rollouts-per-example 8 \
  --max-tokens 4096 \
  --online-difficulty-filtering \
  --easy-threshold 0.8 \
  --hard-threshold 0.2 \
  --easy-fraction 0.25 \
  --hard-fraction 0.25

Each --difficulty-ratio DIFFICULTY=RATIO flag emits a separate [[env]] block with that difficulty passed to load_environment(...). The ratio flags are mutually exclusive with repeated --difficulty, which still means one environment view that allows multiple difficulties. --online-difficulty-filtering adds Prime's adaptive buffer thresholds; retaining some easy and hard examples helps keep noisy AEC reward buffers balanced across already-solved examples and zero-signal failures.

Launch training through Prime:

uv run aec-bench prime train train.toml --yes

Before scaling a training run, run a one-example hosted eval and inspect reward, stop conditions, and tool-call metrics. Treat small evals as smoke tests; benchmark claims require the full evaluation controls.

After training finishes, use aec-bench prime adapters to find the deployed inference adapter before running adapter evals. Training checkpoint ids and deployed inference adapter ids are different Prime records.

Bring hosted results back into AEC-Bench

Import a hosted evaluation when you want to inspect it with the same reports used for local and Harbor runs. Each Prime sample includes the prompt, model messages, tool calls, reward, timing, token use, stop reason, and environment metrics:

uv run aec-bench import-prime-eval <prime-eval-id> \
  --experiment prime-eval-medium-stateful

The importer writes one TrialRecord per Prime sample. It also retains the conversation, raw Prime sample, and submitted output when available. The result can then use normal AEC-Bench trace and behaviour reports.

Provider completion, evaluation, and evidence are separate facts. A sample can have a reward and still have failed execution or incomplete evidence. Check execution_status, evaluation_status, and evidence_status with the verifier result.

After import, use the existing report commands:

uv run aec-bench report traces \
  --experiment-id prime-eval-medium-stateful

uv run aec-bench report behavioral \
  --experiment-id prime-eval-medium-stateful \
  --classifier claude-sonnet-4-20250514 \
  --output prime-eval-behaviour.json

Use the same import-and-report path for base and adapter evals. For comparison reports, keep the experiment ids distinct and compare aggregate reward, stop conditions, tool-call counts, submit_answer calls, and behavioural classifications side by side.

What to inspect

For stateful AEC-Bench exports, inspect mean reward together with:

• Whether the rollout calls submit_answer
• Whether file and command tools are actually used when expected
• Runtime errors versus verifier disagreement
• Per-example reward rows alongside the aggregate
• Token budget and max-turn behaviour for RLM-style tasks

The Prime path should remain explicit: select the task slice, export it, smoke it, run a small hosted eval, import the hosted rollouts, inspect behaviour, then scale.