LIVEdataset aec-bench@releasetasks 552models 18disciplines 5last submission … · built …

electricalwith-tool

Rail Standards Conflict Operator Review Package

Calculates deterministic SSC-02 operator-review metrics from a task-owned synthetic source pack. The template combines standard selection, review-comment resolution, affected-calculation updates, exception approval, source-value traceability, response completeness, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

authority review comment-response workflow rail operator standards matrix workflow

Tags

comments deterministic electrical exceptions operator-review rail standards traceability

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

01
Template
The reusable contract shown on this page.
02
Scenario
An archetype and site context are sampled.
03
Difficulty tier
Inputs may be hidden at harder tiers.
04
Task prompt
The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

13 inputs

Always given

Included directly in every task prompt.

13

Show 13 inputs

Candidate standards count
candidate_standards_count
STD-02-MATRIX-08 candidate standard count
5
Selected standards count
selected_standards_count
STD-02-MATRIX-08 selected controlling standard count
4
Conflicting comments
conflicting_comments
COMMENT-02-REGISTER-08 comments requiring resolution
3
Resolved comments
resolved_comments
COMMENT-02-REGISTER-08 resolved comments
3
Affected calculations
affected_calculations
CALC-02-EXTRACT-08 calculations affected by review comments
6
Calculations updated
calculations_updated
CALC-02-EXTRACT-08 affected calculations updated
5
Exception requests
exception_requests
EXCEPT-02-APPROVAL-08 exception requests
1
Approved exceptions
approved_exceptions
EXCEPT-02-APPROVAL-08 approved exceptions
1
Source values traced
source_values_traced
ALIGN-02-SIGNAL-08 traced source values
18
Required source values
required_source_values
ALIGN-02-SIGNAL-08 required source values
20
Response sections
response_sections
RESPONSE-02-AUTH-08 completed response sections
9
Required response sections
required_response_sections
RESPONSE-02-AUTH-08 required response sections
10
Critical open comments
critical_open_comments
COMMENT-02-REGISTER-08 unresolved critical comments
0

Scored outputs

10 outputs

Standard selection fraction

standard_selection_fraction

Selected controlling standards divided by candidate standards

Scores if within ±0.3% of the reference value.

Comment resolution fraction

comment_resolution_fraction

Resolved comments divided by conflicting comments

Scores if within ±0.3% of the reference value.

Calculation update fraction

calculation_update_fraction

Updated affected calculations divided by affected calculations

Scores if within ±0.3% of the reference value.

Exception resolution fraction

exception_resolution_fraction

Approved exceptions divided by exception requests

Scores if within ±0.3% of the reference value.

Source trace score

source_trace_score

Traced source values divided by required source values

Scores if within ±0.3% of the reference value.

Response completeness score

response_completeness_score

Completed response sections divided by required response sections

Scores if within ±0.3% of the reference value.

Operator review score

operator_review_score

Average of the six review completeness fractions

Scores if within ±0.3% of the reference value.

Open comments

open_comments

Conflicting comments minus resolved comments

Scores if within ±0.3% of the reference value.

Critical open comments

critical_open_comments

Unresolved critical comments

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when review completeness thresholds pass and no critical comments remain

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:: All source-pack values given for the SSC-02 rail standards conflict and operator-review package
medium:: All source-pack values given for the SSC-02 rail standards conflict and operator-review package
hard:: All source-pack values given for the SSC-02 rail standards conflict and operator-review package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

instruction.md
rail-standards-conflict-operator-review-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a rail operator-review engineer checking `SSC-02-LH-08`, a task-owned synthetic SSC-02 rail standards conflict and operator review package.2 3Use only the task-owned synthetic source pack values below for numeric grading. Rail standards matrices and authority review workflows shape the context only; this instance does not validate a real operator standard, accepted review, or authority approval.4 5## Scene6 7- Design case: `CASE-SSC02-STANDARDS-08`8- Standards matrix: `STD-02-MATRIX-08`9- Comment register: `COMMENT-02-REGISTER-08`10- Alignment and signal source pack: `ALIGN-02-SIGNAL-08`11- Calculation extracts: `CALC-02-EXTRACT-08`12- Exception approval route: `EXCEPT-02-APPROVAL-08`13- Authority-partitioned response: `RESPONSE-02-AUTH-08`14 15## Source Values16 17| Item | Value |18|------|-------|19| Candidate standards | {{ candidate_standards_count }} |20| Selected standards | {{ selected_standards_count }} |21| Conflicting comments | {{ conflicting_comments }} |22| Resolved comments | {{ resolved_comments }} |23| Affected calculations | {{ affected_calculations }} |24| Calculations updated | {{ calculations_updated }} |25| Exception requests | {{ exception_requests }} |26| Approved exceptions | {{ approved_exceptions }} |27| Source values traced | {{ source_values_traced }} |28| Required source values | {{ required_source_values }} |29| Response sections | {{ response_sections }} |30| Required response sections | {{ required_response_sections }} |31| Critical open comments | {{ critical_open_comments }} |32 33Checks:34 35- Standard selection fraction equals selected standards divided by candidate standards.36- Comment resolution fraction equals resolved comments divided by conflicting comments.37- Calculation update fraction equals updated calculations divided by affected calculations.38- Exception resolution fraction equals approved exceptions divided by exception requests.39- Source trace and response completeness are source-owned fractions.40- Overall pass score is `1.0` only when the review completeness thresholds pass and no critical comments remain; otherwise it is `0.0`.41 42## Output Format43 44Write a compact memo to `/workspace/output.md`. Preserve the object IDs above, state the task-owned synthetic source-pack boundary, and explain whether the baseline source pack passes the docs-only checks.45 46Do not claim authority approval, accepted project evidence, operator-standard validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.47 48Include a fenced JSON block with exactly these numeric keys:49 50```json51{52  "standard_selection_fraction": <numeric_value>,53  "comment_resolution_fraction": <numeric_value>,54  "calculation_update_fraction": <numeric_value>,55  "exception_resolution_fraction": <numeric_value>,56  "source_trace_score": <numeric_value>,57  "response_completeness_score": <numeric_value>,58  "operator_review_score": <numeric_value>,59  "open_comments": <numeric_value>,60  "critical_open_comments": <numeric_value>,61  "overall_pass_score": <numeric_value>62}63```64

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc02 rail standards review case

ssc02_rail_standards_review_case

Rail standards review package tied to one standards matrix, comment register, alignment/signal source pack, calculation extracts, exception approval route, and response

rail-standards-review

Parameter ranges

candidate_standards_count: 5
selected_standards_count: 4
conflicting_comments: 3
resolved_comments: 3
affected_calculations: 6
calculations_updated: 5
exception_requests: 1
approved_exceptions: 1
source_values_traced: 18
required_source_values: 20
response_sections: 9
required_response_sections: 10
critical_open_comments: 0

Example task

rail-standards-review-ssc02-rail-standards-review-case-preview — hard difficulty, all inputs given.

Rail standards review package tied to one standards matrix, comment register, alignment/signal source pack, calculation extracts, exception approval route, and response. rail-standards-review. Required outputs: standard_selection_fraction, comment_resolution_fraction, calculation_update_fraction, exception_resolution_fraction, source_trace_score, response_completeness_score

The model sees

Scenario context and visible inputs.

candidate_standards_count: 5 to 5
selected_standards_count: 4 to 4
conflicting_comments: 3 to 3
resolved_comments: 3 to 3
affected_calculations: 6 to 6
calculations_updated: 5 to 5
exception_requests: 1 to 1
approved_exceptions: 1 to 1
source_values_traced: 18 to 18
required_source_values: 20 to 20
response_sections: 9 to 9
required_response_sections: 10 to 10
critical_open_comments: 0 to 0

Executable tool: rail-standards-conflict-operator-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "standard_selection_fraction": <number>,
  "comment_resolution_fraction": <number>,
  "calculation_update_fraction": <number>,
  "exception_resolution_fraction": <number>,
  "source_trace_score": <number>,
  "response_completeness_score": <number>,
  "operator_review_score": <number>,
  "open_comments": <number>,
  "critical_open_comments": <number>,
  "overall_pass_score": <number>
}

standard_selection_fraction · scored within ±0.3%
comment_resolution_fraction · scored within ±0.3%
calculation_update_fraction · scored within ±0.3%
exception_resolution_fraction · scored within ±0.3%
source_trace_score · scored within ±0.3%
response_completeness_score · scored within ±0.3%
operator_review_score · scored within ±0.3%
open_comments · scored within ±0.3%
critical_open_comments · scored within ±0.3%
overall_pass_score · scored within ±0.1%