Meta-Harness Composition

Use the meta-harness when you want to compare different ways of running an agent. A candidate can be a harness configuration, a lifecycle strategy, an Interactive World setup, or another value that your application knows how to run.

The meta-harness coordinates the study while each task keeps its normal runtime and scoring rules. Your evaluator runs each candidate and returns TrialRecord results. Your assessment function decides how to compare those results.

The public Python API is aec_bench.experimentation.meta_harness. Artefact tasks, finite lifecycles, Interactive Worlds, and specialist studies keep their own execution and evidence rules.

The aec-bench meta-harness command group remains the public application surface for comparison recipes and the specialist problem-model workflow.

What you provide

The simplest study needs three things:

• a HarnessCandidate that gives your candidate value a unique identity;
• a candidate evaluator that returns a non-empty sequence of TrialRecord values; and
• caller-supplied functions for assessment and, when required, proposal, selection, refinement, and stopping.

The core retains failed and invalid trials as evidence. Each candidate evaluation must return at least one record with unique trial_id values.

One-shot studies

Use run_harness_study() when the baseline and candidate set are already known:

from aec_bench.experimentation.meta_harness import (
    HarnessCandidate,
    run_harness_study,
)

baseline = HarnessCandidate(candidate_id="baseline", value=baseline_config)
candidate = HarnessCandidate(candidate_id="candidate-a", value=candidate_config)

study = await run_harness_study(
    baseline=baseline,
    candidates=[candidate],
    evaluate=evaluate_candidate,
    assess=assess_candidates,
)

evaluate_candidate connects each generic candidate value to its real runtime and returns that runtime's normal TrialRecord values. It can be a synchronous function or an asynchronous function. assess_candidates owns the comparison result. It can return a metric summary, study conclusion, decision object, or another normal Python value.

A one-shot study returns study.assessment for the caller to interpret. Use the bounded refinement API when selection belongs inside the process.

Bounded refinement

Use run_meta_harness() for repeated proposal and refinement:

from aec_bench.experimentation.meta_harness import run_meta_harness

result = await run_meta_harness(
    initial=baseline,
    propose=propose_candidates,
    evaluate=evaluate_candidate,
    assess=assess_candidates,
    select=select_candidate,
    refine=refine_candidate,
    stop=stop_when_sufficient,
    max_rounds=3,
)

max_rounds is required and always bounds execution. Each round can select only the current candidate or an evaluated proposal. A changed refined candidate receives a new candidate_id and becomes the evaluated baseline for the next round. Every final selection therefore has evaluation evidence.

The proposal, assessment, selection, refinement, and stopping policies remain with the caller. Evaluation continues to own metric meaning, and governance continues to own approval.

Connect an existing runtime

An evaluator is the small function that connects a candidate to its real runtime. This keeps the comparison logic independent of artefact tasks, lifecycles, Interactive Worlds, Harbor, Prime, or a specific provider. Both study functions accept synchronous or asynchronous evaluators through the same async API.

For artefact tasks, the evaluator can call the normal run_experiment() boundary directly:

from aec_bench.harness.artifact_tasks import run_experiment

def evaluate_candidate(candidate):
    value = candidate.value
    return run_experiment(
        runtime=value.runtime,
        tasks=value.tasks,
        trials=value.trials,
        recipe=value.attempt_recipe,
    )

Finite lifecycles connect through the implemented lifecycle evaluator:

from aec_bench.experimentation.lifecycle_studies.meta_harness import (
    LifecycleHarnessCandidate,
    evaluate_lifecycle_candidate,
)
from aec_bench.experimentation.meta_harness import HarnessCandidate
from aec_bench.harness.lifecycle_local import run_local_lifecycle
from aec_bench.lifecycles.catalogue import verify_lifecycle

candidate = HarnessCandidate(
    candidate_id="persistent-context",
    value=LifecycleHarnessCandidate(
        trials=tuple(lifecycle_trials),
        execute=run_local_lifecycle,
        verify=verify_lifecycle,
    ),
)

records = evaluate_lifecycle_candidate(candidate)

The evaluator calls the normal lifecycle experiment boundary and returns its TrialRecord values through the shared meta-harness API.

Interactive Worlds use their normal experiment boundary in the same way:

from functools import partial

from aec_bench.harness.world_routing import run_selected_world
from aec_bench.harness.world_trials import run_world_experiment

async def evaluate_world_candidate(candidate):
    value = candidate.value
    return await run_world_experiment(
        tasks=value.tasks,
        trials=value.trials,
        run_trial=partial(run_selected_world, work_root=value.work_root),
    )

The evaluator returns one complete world TrialRecord for each journey. Prime and adaptive-harness studies can use the same comparison functions while keeping their own execution and qualification rules.

CLI comparison

For a scriptable candidate-versus-baseline workflow, start with recipe:

uv run aec-bench meta-harness recipe \
  --task-file task.md \
  --output artefacts/meta-harness/demo

The command writes a workspace for the brief, baseline and candidate problem models, task-run evidence, model endpoint configuration, and comparison output. Existing problem-model, run, or experiment files can be supplied when they already exist.

After both evidence sets are complete, compare them deterministically:

uv run aec-bench meta-harness compare \
  --brief brief.json \
  --baseline-world baseline-world.json \
  --candidate-world candidate-world.json \
  --baseline-run baseline-run.json \
  --candidate-run candidate-run.json \
  --output comparison

The supported file flags retain world in their names. In this workflow, those files contain generated problem representations. Registered Interactive Worlds use a separate runtime. compare delegates candidate evaluation and assessment to the functional study API. Use example --output <directory> for a provider-free worked comparison before connecting model endpoints.

Diagnose and repair a benchmark problem

The pauseable CLI workflow helps diagnose a benchmark failure before changing the task or its generator. It gathers the task description, run evidence, verifier output, reviewer notes, proposed operations, and approval decisions in one process. If required input is missing, the process stops in a named waiting state until that input arrives.

This workflow is one use of the generic API. Model-backed stages are optional. Automation can continue when providers or queued artefacts become available, but explicit limits and approval gates still apply.

Proposed changes and approval

The process can apply declared operations such as selecting fields, comparing values, taking a subset, or forming combinations. Use a model-generated proposal when the evidence requires a change beyond those fixed operations.

Only the approval step can change the shared schema or generator. It can keep a change local to the current run, reject it, or emit a new world_generation_request for review.

The process shape is:

prose
  -> problem_space_brief
  -> problem model
  -> task-run evidence
  -> operation loop
  -> governance loop
  -> world_generation_request, when approved

The problem model is a JSON description of the task unit, allowed operations, expected evidence, and approval policy. Despite the older world file names used by this CLI, it is separate from the registered Interactive World runtime.

Waiting and terminal states

Missing model output keeps the process in a waiting state until real output is available.

Use process when you need to supply or inspect each pauseable artefact directly:

uv run aec-bench meta-harness process "Create a diagnostic task." \
  --brief brief.json \
  --world world.json \
  --task-run task-run.json \
  --output process-output

Use autonomous only when queued artefacts or model endpoints can resolve waiting states. Its limits cover iterations, problem-model regenerations, governance actions, cost, and stagnation. By default, automation can accept run_only directives. world_schema and world_generator changes require human governance.

Lifecycles and Harbor stay separate

All lifecycle commands remain under aec-bench task lifecycle. The lifecycle owns its stages, branching, and study policy. The meta-harness receives only candidate identities and returned TrialRecord evidence. See Finite Lifecycles.

meta-harness harbor-task materialises a Harbor-shaped operation-orchestrator package:

uv run aec-bench meta-harness harbor-task \
  --brief brief.json \
  --world world.json \
  --plan operation-plan.json \
  --output operation-task

The package contains job.yaml, agent/input.json, agent/output.md, agent_result.json, and result.json. External execution produces completed task-run JSON, which process or autonomous accepts through --task-run. Harbor execution remains under the harness boundary.

Scores and approval

The task verifier produces scoring evidence. Your evaluation and assessment functions define what the results mean. The meta-harness keeps the returned trial evidence and coordinates those functions.

Agents can propose repairs, but the relevant approval process decides whether to apply or promote them. Normal evidence and repeatability rules still apply.

Use the functional API when different runtimes need the same candidate-study shape. Use the CLI process when task prose and run evidence should inform a governed problem-model repair.