LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalno-tool

Level Crossing Warning Issue Review Package

Review-native long-horizon task over a generated SSC-02 level-crossing issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering route profile, sighting and warning time, crossing controls, backup power, communications, degraded-mode operations, criteria, and review comments. The task is to inventory the packet, preserve route/crossing identity, recompute source-owned evidence, assign one status per review item, raise findings/information requests/actions, and issue a readiness decision. Packet variants inject missing battery evidence, stale warning-time revisions, sighting chainage mismatches, copied degraded-mode scenarios, open comments, carried actions, or genuine warning-time failures. Scored by a stage-gated custom verifier.

no-tool: The model must reason numerically unaided.

Standards

49 CFR Part 234FHWA MUTCD Part 8operator level-crossing workflow

Tags

backup-powerelectricalissue-readinesslevel-crossinglong-horizonrailreviewwarning-time

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

38 inputs

Always given

Included directly in every task prompt.

32
Show 32 inputs

  • Maximum train speed kmh

    maximum_train_speed_kmh

    Maximum train speed for the controlling crossing approach

    60 – 110 km/h

  • Minimum warning time

    minimum_warning_time_s

    Minimum warning time criterion for the crossing approach

    20 – 30 s

  • Gate lowering time

    gate_lowering_time_s

    Gate lowering duration from the crossing controller sheet

    8 – 13 s

  • Gate start delay

    gate_start_delay_s

    Delay from warning start to gate descent

    2 – 4 s

  • Required gate horizontal before arrival

    required_gate_horizontal_before_arrival_s

    Required time with gates horizontal before train arrival

    4 – 7 s

  • Controller load

    controller_load_w

    Crossing controller load

    160 – 220 W

  • Flashing light load

    flashing_light_load_w

    Load per flashing-light assembly

    30 – 50 W

  • Flashing light count

    flashing_light_count

    Flashing-light assembly count

    4 – 6

  • Gate mechanism load

    gate_mechanism_load_w

    Load per gate mechanism

    80 – 110 W

  • Gate mechanism count

    gate_mechanism_count

    Gate mechanism count

    2 – 4

  • Comms switch load

    comms_switch_load_w

    Communications switch load

    35 – 60 W

  • Track circuit load

    track_circuit_load_w

    Track circuit and detector load

    55 – 85 W

  • Event recorder load

    event_recorder_load_w

    Event recorder load

    20 – 35 W

  • Load future allowance pct

    load_future_allowance_pct

    Spare and future allowance for crossing signal load

    10 – 25 %

  • Required autonomy

    required_autonomy_h

    Required degraded-mode backup autonomy

    6812

  • Dc system voltage

    dc_system_voltage_v

    DC system voltage

    48 V

  • Depth of discharge pct

    depth_of_discharge_pct

    Usable battery depth of discharge

    75 – 85 %

  • Temperature derating factor

    temperature_derating_factor

    Battery temperature derating factor

    0.8 – 0.92

  • Inverter efficiency pct

    inverter_efficiency_pct

    UPS inverter efficiency

    90 – 95 %

  • Load power factor

    load_power_factor

    UPS load power factor

    0.85 – 0.95

  • Feeder resistance milliohm per m

    feeder_resistance_milliohm_per_m

    Conductor resistance

    1.2 – 2.2 milliohm/m

  • Max voltage drop percent

    max_voltage_drop_percent

    Maximum DC feeder voltage drop

    4 – 6 %

  • Fiber length

    fiber_length_km

    Fiber link length

    1 – 4 km

  • Fiber attenuation db per km

    fiber_attenuation_db_per_km

    Fiber attenuation

    0.25 – 0.45 dB/km

  • Fiber connector count

    fiber_connector_count

    Fiber connector count

    2 – 6

  • Connector loss

    connector_loss_db

    Loss per fiber connector

    0.2 – 0.4 dB

  • Fiber splice count

    fiber_splice_count

    Fiber splice count

    2 – 10

  • Splice loss

    splice_loss_db

    Loss per fiber splice

    0.03 – 0.08 dB

  • Patch panel allowance

    patch_panel_allowance_db

    Patch-panel allowance

    0.5 – 1.5 dB

  • Optical tx power

    optical_tx_power_dbm

    Optical transmitter power

    -4 – -1 dBm

  • Receiver sensitivity

    receiver_sensitivity_dbm

    Receiver sensitivity

    -26 – -22 dBm

  • Required fiber margin

    required_fiber_margin_db

    Required excess fiber margin

    3 – 6 dB

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

6

  • Warning time margin

    warning_time_margin_s

    Derivation margin between provided and minimum warning time

    Hidden at easy and medium and hard difficulty.

    4 – 12 s

  • Warning time deficit

    warning_time_deficit_s

    Derivation deficit for the warning-time-deficient variant

    Hidden at easy and medium and hard difficulty.

    1 – 5 s

  • Battery runtime margin

    battery_runtime_margin_h

    Derivation margin between required autonomy and installed battery runtime

    Hidden at easy and medium and hard difficulty.

    0.5 – 2.5 h

  • Selected ups rating margin

    selected_ups_rating_margin_va

    Derivation margin between required and selected UPS rating

    Hidden at easy and medium and hard difficulty.

    80 – 250 VA

  • Voltage drop margin target percent

    voltage_drop_margin_target_percent

    Derivation margin between feeder voltage drop and the source-owned maximum

    Hidden at easy and medium and hard difficulty.

    0.5 – 2 %

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_battery_capacitystale_warning_revisionchainage_sighting_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedwarning_time_deficient

Scored outputs

25 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Warning-time basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Crossing warning and backup adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Degraded-mode scenario status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Feeder and fiber resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Maximum train speed m s

maximum_train_speed_m_s

Approach train speed evidence

Scores if within ±2% of the reference value.

Provided warning time s

provided_warning_time_s

Provided warning time from strike-in distance

Scores if within ±2% of the reference value.

Strike in distance

strike_in_distance_m

Strike-in distance evidence

Scores if within ±2% of the reference value.

Warning time margin s

warning_time_margin_s

Warning-time margin evidence

Scores if within ±2% of the reference value.

Gate horizontal margin s

gate_horizontal_margin_s

Gates-horizontal timing margin evidence

Scores if within ±2% of the reference value.

Design signal load w

design_signal_load_w

Design signal load evidence

Scores if within ±2% of the reference value.

Required battery capacity ah

required_battery_capacity_ah

Required battery capacity evidence

Scores if within ±2% of the reference value.

Installed battery capacity ah

installed_battery_capacity_ah

Installed battery capacity evidence

Scores if within ±2% of the reference value.

Battery runtime h

battery_runtime_h

Battery runtime evidence

Scores if within ±2% of the reference value.

Battery runtime margin h

battery_runtime_margin_h

Battery runtime margin evidence

Scores if within ±2% of the reference value.

Dc voltage drop margin

dc_voltage_drop_margin_percent

DC voltage-drop margin evidence

Scores if within ±2% of the reference value.

Fiber link margin db

fiber_link_margin_db

Fiber link margin evidence

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing backup evidence, or genuine warning-time failure

Hidden inputs

  • Packet variantpacket_variant
  • Warning time margin swarning_time_margin_s
  • Warning time deficit swarning_time_deficit_s
  • Battery runtime margin hbattery_runtime_margin_h
  • Selected ups rating margin vaselected_ups_rating_margin_va
  • Voltage drop margin targetvoltage_drop_margin_target_percent

Packet variant restricted to: clean, missing_battery_capacity, warning_time_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Warning time margin swarning_time_margin_s
  • Warning time deficit swarning_time_deficit_s
  • Battery runtime margin hbattery_runtime_margin_h
  • Selected ups rating margin vaselected_ups_rating_margin_va
  • Voltage drop margin targetvoltage_drop_margin_target_percent

hard

Some inputs hidden

Subtle documentary defects: stale warning basis, sighting identity drift, scenario copy-forward, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Warning time margin swarning_time_margin_s
  • Warning time deficit swarning_time_deficit_s
  • Battery runtime margin hbattery_runtime_margin_h
  • Selected ups rating margin vaselected_ups_rating_margin_va
  • Voltage drop margin targetvoltage_drop_margin_target_percent

Packet variant restricted to: stale_warning_revision, chainage_sighting_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a level-crossing warning, controls, backup-power, and communications issue package.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, route profile, sighting and warning-time worksheet, crossing control layout, backup-power and communications schedule, degraded-mode operations note, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of truth for this review.4 5Your job is not to redesign the crossing. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion.102. Build an identity ledger for the route profile, crossing ID, sighting worksheet, control layout, backup schedule, communications link, degraded-mode scenario, and criteria memo.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, crossing identity drift, and open critical comments before accepting package claims.124. Recompute only the evidence needed for review items, using the assessment bases stated in the criteria memo.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20| Item | Review question |21|---|---|22| RLR-01 | Packet completeness: are the required route, sighting, control, backup, communications, degraded-mode, criteria, and comment sources present with IDs and revisions? |23| RLR-02 | Object identity: do the route profile, sighting worksheet, control layout, backup schedule, communications link, and degraded-mode note refer to the same crossing and approach? |24| RLR-03 | Warning-time basis: are speed, strike-in distance, warning-time, gate timing, backup-power, feeder, and fiber bases traceable, current, and recomputable? |25| RLR-04 | Crossing warning and backup adequacy: do warning time, gate-horizontal timing, and backup runtime clear the source-owned criteria? |26| RLR-05 | Degraded-mode scenario: is the same degraded operating case used across warning, backup, feeder, fiber, and operator sources? |27| RLR-06 | Feeder and communications resilience: are voltage-drop and fiber-link margins source-backed and internally consistent with the selected load? |28| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |29| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |30| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |31 32## Boundary Rules33 34- Use the review matrix definitions to decide the most specific affected item from the source packet.35- If a source value needed for recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source.36- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.37- RLR-08 passes when the readiness decision reconciles with the review matrix, findings, information requests, and action register.38- Every finding, information request, and action must name one exact RLR item.39- Do not rename computed_evidence keys.40 41## Output42 43Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:44 45```json46{47 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],48 "identity_ledger": {49 "route_profile": "...",50 "sighting_warning_time": "...",51 "crossing_layout": "...",52 "backup_power_comms": "...",53 "degraded_mode": "...",54 "criteria_comments": "...",55 "crossing_id": "..."56 },57 "review_matrix": {58 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},59 "RLR-02": {"status": "...", "evidence": "..."},60 "RLR-03": {"status": "...", "evidence": "..."},61 "RLR-04": {"status": "...", "evidence": "..."},62 "RLR-05": {"status": "...", "evidence": "..."},63 "RLR-06": {"status": "...", "evidence": "..."},64 "RLR-07": {"status": "...", "evidence": "..."},65 "RLR-08": {"status": "...", "evidence": "..."},66 "RLR-09": {"status": "...", "evidence": "..."}67 },68 "computed_evidence": {69 "maximum_train_speed_m_s": 0,70 "provided_warning_time_s": 0,71 "strike_in_distance_m": 0,72 "warning_time_margin_s": 0,73 "gate_horizontal_margin_s": 0,74 "design_signal_load_w": 0,75 "required_battery_capacity_ah": 0,76 "installed_battery_capacity_ah": 0,77 "battery_runtime_h": 0,78 "battery_runtime_margin_h": 0,79 "dc_voltage_drop_margin_percent": 0,80 "fiber_link_margin_db": 081 },82 "findings": [83 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}84 ],85 "information_requests": [86 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}87 ],88 "action_register": [89 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}90 ],91 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",92 "claim_boundary_statement": "..."93}94```95 96Rules for the structured block:97 98- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead.99- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.100- Every `insufficient_data` needs an information request naming the exact missing field and its source document.101- Carried actions must appear in `action_register` with an owner.102- `readiness_decision` must reconcile with your matrix.103- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.104

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Mainline highway

mainline_highway

Mainline rail crossing on a highway approach

mainline-highway-crossingregional-arterial-crossing
Parameter ranges
maximum_train_speed_kmh
80 – 110
required_autonomy_h
8, 12

Urban branch

urban_branch

Urban branch-line crossing with tighter equipment cabinet constraints

urban-branch-crossingstation-approach-crossing
Parameter ranges
maximum_train_speed_kmh
60 – 85
required_autonomy_h
6, 8

Example task

mainline-highway-crossing-mainline-highway-preview — hard difficulty, some inputs hidden.

Mainline rail crossing on a highway approach. mainline-highway-crossing. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

maximum_train_speed_kmh
80 to 110 km/h
minimum_warning_time_s
20 to 30 s
gate_lowering_time_s
8 to 13 s
gate_start_delay_s
2 to 4 s
required_gate_horizontal_before_arrival_s
4 to 7 s
controller_load_w
160 to 220 W
flashing_light_load_w
30 to 50 W
flashing_light_count
4 to 6
gate_mechanism_load_w
80 to 110 W
gate_mechanism_count
2 to 4
comms_switch_load_w
35 to 60 W
track_circuit_load_w
55 to 85 W
event_recorder_load_w
20 to 35 W
load_future_allowance_pct
10 to 25 %
required_autonomy_h
8 h
dc_system_voltage_v
48 to 48 V
depth_of_discharge_pct
75 to 85 %
temperature_derating_factor
0.8 to 0.92
inverter_efficiency_pct
90 to 95 %
load_power_factor
0.85 to 0.95
feeder_resistance_milliohm_per_m
1.2 to 2.2 milliohm/m
max_voltage_drop_percent
4 to 6 %
fiber_length_km
1 to 4 km
fiber_attenuation_db_per_km
0.25 to 0.45 dB/km
fiber_connector_count
2 to 6
connector_loss_db
0.2 to 0.4 dB
fiber_splice_count
2 to 10
splice_loss_db
0.03 to 0.08 dB
patch_panel_allowance_db
0.5 to 1.5 dB
optical_tx_power_dbm
-4 to -1 dBm
receiver_sensitivity_dbm
-26 to -22 dBm
required_fiber_margin_db
3 to 6 dB

Executable tool: level-crossing-warning-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Packet variant

    packet_variant

  • Selected ups rating margin va

    selected_ups_rating_margin_va

  • Battery runtime margin h

    battery_runtime_margin_h

  • Voltage drop margin target

    voltage_drop_margin_target_percent

  • Warning time deficit s

    warning_time_deficit_s

  • Warning time margin s

    warning_time_margin_s

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "maximum_train_speed_m_s": <number>,
  "provided_warning_time_s": <number>,
  "strike_in_distance_m": <number>,
  "warning_time_margin_s": <number>,
  "gate_horizontal_margin_s": <number>,
  "design_signal_load_w": <number>,
  "required_battery_capacity_ah": <number>,
  "installed_battery_capacity_ah": <number>,
  "battery_runtime_h": <number>,
  "battery_runtime_margin_h": <number>,
  "dc_voltage_drop_margin_percent": <number>,
  "fiber_link_margin_db": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • maximum_train_speed_m_s · scored within ±2%
  • provided_warning_time_s · scored within ±2%
  • strike_in_distance_m · scored within ±2%
  • warning_time_margin_s · scored within ±2%
  • gate_horizontal_margin_s · scored within ±2%
  • design_signal_load_w · scored within ±2%
  • required_battery_capacity_ah · scored within ±2%
  • installed_battery_capacity_ah · scored within ±2%
  • battery_runtime_h · scored within ±2%
  • battery_runtime_margin_h · scored within ±2%
  • dc_voltage_drop_margin_percent · scored within ±2%
  • fiber_link_margin_db · scored within ±2%