Scoring

When a trial finishes, the execution backend runs the task's verifier against the agent output file. AEC-Bench records the process in a VerifierExecutionReceipt, parses valid verifier output into an EvaluationResult, and attaches both forms of evidence to the trial. The receipt decides whether the verifier completed; the presence of a reward file does not.

The verifier contract

Each task exposes a VerifierSpec:

class VerifierSpec(StrictModel):
    script: str
    expected_output_path: str
    reward_path: str
    details_path: str | None = None

For current task directories, the loader resolves tests/test.sh as the verifier entry point. Harbor expects that shell script inside the container. The verifier reads the agent output from expected_output_path, usually a /workspace/... path inferred from the task instruction, and writes:

/logs/verifier/reward.json     # mandatory: {"reward": 0.0-1.0}
/logs/verifier/details.json    # optional: per-field scores and evidence

A minimal reward.json is just the headline number:

{ "reward": 0.93 }

The optional details.json carries the per-dimension breakdown:

{
  "voltage_drop_v": { "score": 0.95, "max_score": 1.0, "evidence": "within 2% of reference" },
  "voltage_drop_pct": { "score": 1.0, "max_score": 1.0, "evidence": "exact match" },
  "compliance": { "score": 1.0, "max_score": 1.0, "evidence": "correctly flagged compliant" }
}

Reward rollup

reward.json supplies the candidate trial reward. The runner validates it and stores details.json as EvaluationResult.breakdown. AEC-Bench retains that reward only when the verifier receipt also completed successfully.

For tasks that need rubric-style scoring, verifier code or judge pipelines can call the rubric scorer before writing artefacts. The scorer normalises each dimension to score / max_score, clamps it to [0, 1], and combines dimensions with a rollup strategy:

Rubric rewards round to 4 decimals and must land in [0.0, 1.0]. If a verifier writes reward.json directly, that value is eligible for use after output and receipt validation.

Verifier process receipt

The receipt records the verifier key and version, process interval, redacted arguments, exit code, timeout or cancellation, bounded stdout and stderr, reward and details artefacts, output parse status, and any failure reason.

A verifier is complete only when all of these conditions are true:

1. the receipt matches the expected verifier key and version;
2. the process was not cancelled and did not time out;
3. the process exited with code 0;
4. reward.json exists and parses as valid verifier output; and
5. the reward artefact is retained with the trial evidence.

A reward written before a crash or non-zero exit cannot turn that process into a successful verification.

Validity gates

Before scoring, the pipeline checks whether the output is even fit for scoring:

class ValidityCheck(StrictModel):
    output_parseable: bool       # did the file parse as its declared format?
    schema_valid: bool           # did it match the expected shape?
    verifier_completed: bool     # did the authoritative verifier receipt complete?
    errors: list[str] = Field(default_factory=list)

The reward gate is fail-closed:

verifier_completed is a projection of the receipt outcome. It appears in reports with parser and schema flags so readers can distinguish an incorrect engineering answer from an execution or verifier failure.

Evaluation result fields

Putting it together:

class EvaluationResult(StrictModel):
    reward: float                          # 0.0 to 1.0
    validity: ValidityCheck
    breakdown: dict[str, Any] | None       # per-field scores + evidence
    error_taxonomy: list[ErrorTag] | None  # classified failures
    confidence: ConfidenceMetadata | None  # statistical metadata
    annotations: list[Annotation] | None   # human review verdicts
    stewardship: StewardshipEvaluation | None  # optional interactive-world evaluation

stewardship carries the integrity gates, metrics, and evidence identities for supported stewardship worlds. The field is reserved for supported stewardship-world results.

Error taxonomy

Verifiers (or post-hoc analysis) can tag failures for grouping in reports:

class ErrorTag(StrictModel):
    category: str              # "tool failure", "parsing error", "unit mismatch"
    description: str | None = None
    source: ErrorSource        # mechanical | human | judge

Tasks define their own categories. source tracks whether the classification came from the runner, a reviewer, or an automated judge.

Annotations

annotations is for human review after the automated run:

class Annotation(StrictModel):
    reviewer_id: str
    reviewer_discipline: str | None = None
    timestamp: datetime
    judgment: Judgment             # pass | fail | defer
    categories: list[str] = Field(default_factory=list)
    notes: str | None

Annotation records sit alongside the automated reward, so disagreement between a reviewer and a verifier stays visible on the record.

Conditional evidence

Conditional evidence is experimental, and no registered lifecycle uses it. Evidence requests can change what an agent sees, but the task verifier still defines correctness and supplies reward evidence.

Descriptive holdout results

Holdout summaries describe results under a frozen condition; the task verifier remains the single score authority. Keep full holdout records private and publish only allowlisted aggregates.

Determinism

Mechanical verifier scoring should be deterministic: the same agent output should produce the same reward. When human review or judge-based classification is used, record the extra uncertainty explicitly in confidence, annotations, and error_taxonomy. See Classification.