Scoring

When a trial finishes, the execution backend runs the task's verifier against the agent output file. AEC-Bench records the process in a VerifierExecutionReceipt, parses valid verifier output into an EvaluationResult, and attaches both forms of evidence to the trial. The receipt records whether the verifier completed successfully.

To assess a saved answer after changing the verifier, see Regrading.

The verifier contract

Each task exposes a VerifierSpec:

class VerifierSpec(StrictModel):
    script: str
    expected_output_path: str
    reward_path: str
    details_path: str | None = None

For current task directories, the loader resolves tests/test.sh as the verifier entry point. Harbor expects that shell script inside the container. The verifier reads the agent output from expected_output_path, usually a /workspace/... path inferred from the task instruction, and writes:

/logs/verifier/reward.json     # mandatory: {"reward": 0.0-1.0}
/logs/verifier/details.json    # optional: per-field scores and evidence

A minimal reward.json is just the headline number:

{ "reward": 0.93 }

The optional details.json carries the per-dimension breakdown:

{
  "voltage_drop_v": { "score": 0.95, "max_score": 1.0, "evidence": "within 2% of reference" },
  "voltage_drop_pct": { "score": 1.0, "max_score": 1.0, "evidence": "exact match" },
  "compliance": { "score": 1.0, "max_score": 1.0, "evidence": "correctly flagged compliant" }
}

Reward rollup

reward.json supplies the candidate trial reward. The runner validates it and stores details.json as EvaluationResult.breakdown. AEC-Bench retains that reward only when the verifier receipt also completed successfully.

For tasks that need rubric-style scoring, verifier code or judge pipelines can call the rubric scorer before writing artefacts. The scorer normalises each dimension to score / max_score, clamps it to [0, 1], and combines dimensions with a rollup strategy:

Rubric rewards round to 4 decimals and must land in [0.0, 1.0]. If a verifier writes reward.json directly, that value is eligible for use after output and receipt validation.

Public report rubrics

The report harnesses can expose a rubric explicitly marked visibility = "public" to the agent. Dimension weights, criterion categories, and section and reference scopes guide drafting, review, and synthesis. Private verifier criteria remain outside the actor workspace and model requests.

Public writing checks and model reviews help prepare the artefact. Required fields, error-level rules, and unresolved structural checks can block harness completion; warnings, unchecked prose requirements, and grounding findings remain visible as diagnostics. The task verifier checks the submitted artefact and evaluation records the result through the normal scoring contract.

Verifier process receipt

Use the receipt to check which verifier ran and whether its result is usable:

A verifier is complete only when all of these conditions are true:

1. the receipt matches the expected verifier key and version;
2. the process ran to completion within its time limit;
3. the process exited with code 0;
4. reward.json exists and parses as valid verifier output; and
5. the reward artefact is retained with the trial evidence.

A crash or non-zero exit produces a failed verification, even if the process wrote a reward file.

Validity gates

Before scoring, the pipeline checks whether the output is even fit for scoring:

class ValidityCheck(StrictModel):
    output_parseable: bool       # did the file parse as its declared format?
    schema_valid: bool           # did it match the expected shape?
    verifier_completed: bool     # did the authoritative verifier receipt complete?
    errors: list[str] = Field(default_factory=list)

The reward gate is fail-closed:

verifier_completed is a projection of the receipt outcome. It appears in reports with parser and schema flags so readers can distinguish an incorrect engineering answer from an execution or verifier failure.

Evaluation result fields

Putting it together:

class EvaluationResult(StrictModel):
    reward: float                          # 0.0 to 1.0
    validity: ValidityCheck
    breakdown: dict[str, Any] | None       # per-field scores + evidence
    error_taxonomy: list[ErrorTag] | None  # classified failures
    confidence: ConfidenceMetadata | None  # statistical metadata
    annotations: list[Annotation] | None   # human review verdicts
    stewardship: StewardshipEvaluation | None  # optional interactive-world evaluation

stewardship carries the integrity gates, metrics, and evidence identities for supported stewardship worlds.

Error taxonomy

Verifiers (or post-hoc analysis) can tag failures for grouping in reports:

class ErrorTag(StrictModel):
    category: str              # "tool failure", "parsing error", "unit mismatch"
    description: str | None = None
    source: ErrorSource        # mechanical | human | judge

Tasks define their own categories. source tracks whether the classification came from the runner, a reviewer, or an automated judge.

Annotations

annotations is for human review after the automated run:

class Annotation(StrictModel):
    reviewer_id: str
    reviewer_discipline: str | None = None
    timestamp: datetime
    judgment: Judgment             # pass | fail | defer
    categories: list[str] = Field(default_factory=list)
    notes: str | None

Annotation records sit alongside the automated reward, so disagreement between a reviewer and a verifier stays visible on the record.

Descriptive holdout results

Holdout summaries describe results under a frozen condition; the task verifier remains the single score authority. Keep full holdout records private and publish only allowlisted aggregates.

Determinism

Mechanical verifier scoring should be deterministic: the same agent output should produce the same reward. When human review or judge-based classification is used, record the extra uncertainty explicitly in confidence, annotations, and error_taxonomy. See Classification.