Review and Reporting

Automated verification supplies the score. Review adds disagreement, uncertainty, and supporting evidence as separate information.

Optional post-verifier LLM review

run and run-local can call an LLM reviewer after the deterministic verifier. The reviewer checks the agent's answer against the task and retained evidence, including the execution trace and cited sources:

uv run aec-bench run --config experiment.yaml \
  --reviewer-model gpt-4.1-mini

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct \
  --reviewer-model gpt-4.1-mini

Use --reviewer-models-config PATH for multiple or custom endpoints. Add --fail-on-reviewer-error to make an incomplete reviewer stage fail the run.

The stage writes request, per-model review, evaluation, and summary artefacts beside the run. On import, its summary is stored under EvaluationResult.breakdown["llm_reviewer"]. reward remains the verifier value, and human review uses Annotation.

Task-genome review evidence

A retained review must stay tied to the task version that was reviewed. TaskSnapshotRef identifies that version, and small SourceSpan records point to the instruction, verifier, or other source behind each claim.

When a review must be retained independently, store the review bytes and use ArtifactRef. Before reuse, verify that the review still matches the selected task snapshot. A changed task needs a new review.

Structured review

Human judgement is recorded separately as data attached to a trial:

class Annotation(StrictModel):
    reviewer_id: str
    reviewer_discipline: str | None
    timestamp: datetime
    judgment: Judgment
    categories: list[str]
    notes: str | None

Record a human review as pass, fail, or defer alongside the automated reward, so readers can inspect any disagreement with the verifier. Optional LLM reviewer findings have their own record type.

Use structured categories for recurring issues such as:

• incorrect engineering method;
• right answer for the wrong reason;
• ambiguous task wording;
• verifier tolerance or parser defect;
• missing or contradictory evidence; and
• unsupported professional claim.

Use the machine-readable category for classification and free-form notes for explanation.

Calibration

Reviewer calibration should use a shared sample with known disagreements. Track:

• agreement before adjudication;
• category-level confusion;
• discipline and experience where relevant;
• changes to the rubric or verifier; and
• the revision under which each annotation was made.

When a rubric changes, retain earlier annotations against their original revision. Re-reviewing creates a new record and preserves history.

Reporting

Every reported metric derives from identified trial, evaluation, and review records and filters.

run reconcile uses RunAccounting to establish the result-set boundary. It matches typed outcomes to the exact persisted plan and returns:

• planned and terminal outcome counts;
• run completeness and validity;
• accepted trial identities in plan order;
• missing and invalid planned identities;
• duplicate and conflicting identities; and
• unexpected identities retained in quarantine.

The run status gives the reporting context:

The reconciliation result provides the accepted record set and accounting counts used by reporting workflows.

For an active persisted run, review the operational projection separately from the final accounting result:

uv run aec-bench run status <run-id> \
  --operational-store artefacts/operational.sqlite \
  --plan-root artefacts/runs

The CLI, TUI run-progress view, and GET /api/runs/{run_id}/status web route use the same RunProgress projection to show completed work and what remains. See Execution Control for the available counts.

Use the Evidence Index to page through trial metadata, then read the portable record path when a review needs the evidence body. Run aec-bench evidence verify before relying on referenced artefact bytes.

Different audiences need different views:

Model usage and cost

Use an experiment summary to compare model usage and the reported reward:

uv run aec-bench report summary --experiment-id exp-001
uv run aec-bench --json report summary --experiment-id exp-001

For Harbor records with model usage, the summary attributes tokens and cost to each model, including child models. It uses reported costs where available and the repository's pricing table to estimate the rest.

Read cost completeness alongside the amount:

JSON output adds these fields for model attribution:

Estimates use the token counts available from Harbor. An explicit reported zero means free usage; an unknown price remains unknown.

Read model rows as a breakdown of the trial total, which already includes cached input. The trial can retain a known aggregate even when only some of its usage is attributed to models.

Holdout protection

Keep holdout-sensitive review evidence in the protected review context. Use allowlisted aggregates for public reports and development data for training, model-visible feedback, and task-generation prompts.

Keep these uses separate:

• full private review evidence;
• allowlisted public aggregates;
• task-author feedback;
• model-visible development feedback; and
• training-visible examples.

Every annotation, summary, or paraphrase derived from a sealed target remains holdout-sensitive. Choose categories and aggregate publication rules before opening the holdout.

Continuous improvement

Review can trigger changes to:

• task instructions and fixtures;
• verifier logic and tolerances;
• task taxonomy;
• agent-harness observability; or
• public documentation.

Those changes produce new source revisions and new evidence. Historical trial records remain immutable, and benchmark improvements produce new evidence linked to the result that exposed the problem.

See Scoring for the verifier contract, Traces for execution evidence, and Classification for automated behavioural analysis. See Runs and Plans for planned membership and reconciliation.