Review and Reporting

Automated verification supplies the score. Review adds disagreement, uncertainty, and supporting evidence as separate information.

Optional post-verifier LLM review

run and run-local can call an LLM reviewer after the deterministic verifier. The reviewer can inspect the task, agent output, retained files, execution trace, cited sources, and evidence of contradictions:

uv run aec-bench run --config experiment.yaml \
  --reviewer-model gpt-4.1-mini

uv run aec-bench run-local tasks/electrical/voltage-drop \
  --model gpt-4.1-mini \
  --harness direct \
  --reviewer-model gpt-4.1-mini

Use --reviewer-models-config PATH for multiple or custom endpoints. Add --fail-on-reviewer-error to make an incomplete reviewer stage fail the run.

The stage writes request, per-model review, evaluation, and summary artefacts beside the run. On import, its summary is stored under EvaluationResult.breakdown["llm_reviewer"]. reward remains the verifier value, and human review uses Annotation.

Task-genome review evidence

A retained review must stay tied to the task version that was reviewed. TaskSnapshotRef identifies that version, and small SourceSpan records point to the instruction, verifier, or other source behind each claim.

When a review must be retained independently, store the review bytes and use ArtifactRef. Before reuse, verify that the review still matches the selected task snapshot. A changed task needs a new review.

Structured review

Human judgement is recorded separately as data attached to a trial:

class Annotation(StrictModel):
    reviewer_id: str
    reviewer_discipline: str | None
    timestamp: datetime
    judgment: Judgment
    categories: list[str]
    notes: str | None

An annotation can pass, fail, or defer. The automated reward remains the verifier result. Keeping both values makes verifier-reviewer disagreement visible and auditable. Human annotations and optional LLM reviewer findings use separate record types.

Use structured categories for recurring issues such as:

• incorrect engineering method;
• right answer for the wrong reason;
• ambiguous task wording;
• verifier tolerance or parser defect;
• missing or contradictory evidence; and
• unsupported professional claim.

Use the machine-readable category for classification and free-form notes for explanation.

Calibration

Reviewer calibration should use a shared sample with known disagreements. Track:

• agreement before adjudication;
• category-level confusion;
• discipline and experience where relevant;
• changes to the rubric or verifier; and
• the revision under which each annotation was made.

When a rubric changes, retain earlier annotations against their original revision. Re-reviewing creates a new record and preserves history.

Reporting

Every reported metric derives from identified trial, evaluation, and review records and filters.

Different audiences need different views:

Holdout protection

Reviewers may see holdout-sensitive evidence that must not enter public reports, training data, model-visible feedback, or task-generation prompts.

Keep these uses separate:

• full private review evidence;
• allowlisted public aggregates;
• task-author feedback;
• model-visible development feedback; and
• training-visible examples.

Every annotation, summary, or paraphrase derived from a sealed target remains holdout-sensitive. Choose categories and aggregate publication rules before opening the holdout.

Continuous improvement

Review can trigger changes to:

• task instructions and fixtures;
• verifier logic and tolerances;
• task taxonomy;
• agent-harness observability; or
• public documentation.

Those changes produce new source revisions and new evidence. Historical trial records remain immutable, and benchmark improvements produce new evidence linked to the result that exposed the problem.

See Scoring for the verifier contract, Traces for execution evidence, and Classification for automated behavioural analysis.