LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalno-tool

Bus Priority Cabinet Issue Review Package

Review-native long-horizon task over a generated SSC-01 bus-priority corridor issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering operations, signal timing, detector/controller membership, cabinet load, feeder and backup power, owner criteria, and comments. The task is to inventory the packet, preserve object identity, recompute only source-owned evidence, assign one status per review item, raise findings/information requests/actions, and make a readiness decision. Packet variants inject missing cabinet capacity evidence, stale signal timing revisions, detector/controller mismatch, scenario copy-forward, open comments, carried comments, or a genuine cabinet load exceedance.

no-tool: The model must reason numerically unaided.

Standards

FHWA MUTCDITS architectureroadside cabinet designtransit signal priority

Tags

bus-prioritycabinetelectricalissue-readinessitsreviewroadsignal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

27 inputs

Always given

Included directly in every task prompt.

21
Show 21 inputs

  • Bus approach speed kmh

    bus_approach_speed_kmh

    SIG-SSC01-005 bus-priority approach speed

    35 – 65 km/h

  • Bus approach grade pct

    bus_approach_grade_pct

    SIG-SSC01-005 signed bus-priority approach grade

    -1 – 3 %

  • Yellow reaction time

    yellow_reaction_time_s

    SIG-SSC01-005 yellow interval reaction term

    0.8 – 1.4 s

  • Yellow deceleration

    yellow_deceleration_m_s2

    SIG-SSC01-005 yellow interval deceleration

    2.8 – 3.6 m/s2

  • Intersection width

    intersection_width_m

    SIG-SSC01-005 intersection width

    18 – 36 m

  • Bus length

    bus_length_m

    BUS-SSC01-005 design bus length

    12 – 18 m

  • All red speed kmh

    all_red_speed_kmh

    SIG-SSC01-005 all-red clearance speed

    35 – 55 km/h

  • Buses per hour

    buses_per_hour

    BUS-SSC01-005 scheduled priority buses

    12 – 36 buses/h

  • Bus occupancy pax

    bus_occupancy_pax

    BUS-SSC01-005 design bus occupancy

    45 – 85 passengers/bus

  • Controller load

    controller_load_w

    CAB-SSC01-005 controller load

    150 – 260 W

  • Detector count

    detector_count

    DET-SSC01-005 detector count

    8 – 18

  • Detector load

    detector_load_w

    DET-SSC01-005 detector load per unit

    5 – 12 W

  • Transit radio load

    transit_radio_load_w

    CAB-SSC01-005 transit radio load

    50 – 110 W

  • Vms load

    vms_load_w

    CAB-SSC01-005 VMS load

    80 – 180 W

  • Signal heads load

    signal_heads_load_w

    CAB-SSC01-005 signal heads load

    300 – 650 W

  • Feeder voltage

    feeder_voltage_v

    FEED-SSC01-005 nominal feeder voltage

    230240

  • Power factor

    power_factor

    FEED-SSC01-005 cabinet power factor

    0.9 – 0.98

  • Feeder length

    feeder_length_km

    FEED-SSC01-005 feeder length

    0.08 – 0.35 km

  • Conductor resistance

    conductor_resistance_ohm_km

    FEED-SSC01-005 conductor resistance

    0.45 – 1.1 ohm/km

  • Battery efficiency

    battery_efficiency

    BATT-SSC01-005 battery efficiency

    0.85 – 0.95

  • Required backup

    required_backup_h

    BATT-SSC01-005 required backup duration

    2 – 6 h

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

6

  • Bus capacity margin target pax h

    bus_capacity_margin_target_pax_h

    Derivation margin between bus handling capacity and peak passenger demand

    Hidden at easy and medium and hard difficulty.

    80 – 450 passengers/h

  • Cabinet capacity margin

    cabinet_capacity_margin_w

    Derivation margin between cabinet load and cabinet capacity for passing packets

    Hidden at easy and medium and hard difficulty.

    120 – 550 W

  • Cabinet load deficit

    cabinet_load_deficit_w

    Derivation deficit between cabinet capacity and cabinet load for failing packets

    Hidden at easy and medium and hard difficulty.

    50 – 250 W

  • Voltage drop margin target percent

    voltage_drop_margin_target_percent

    Derivation margin between voltage drop and allowable voltage drop

    Hidden at easy and medium and hard difficulty.

    2 – 5 %

  • Battery runtime margin

    battery_runtime_margin_h

    Derivation margin between battery runtime and required backup duration

    Hidden at easy and medium and hard difficulty.

    0.8 – 3.5 h

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_cabinet_capacitystale_signal_timing_revisiondetector_controller_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedcabinet_load_exceeded

Scored outputs

24 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Signal and priority basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Cabinet and priority adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Secondary-discipline resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Yellow interval s

yellow_interval_s

Grade-adjusted yellow interval

Scores if within ±2% of the reference value.

All red interval s

all_red_interval_s

All-red clearance interval

Scores if within ±2% of the reference value.

Bus handling capacity pax h

bus_handling_capacity_pax_h

Bus priority passenger handling capacity

Scores if within ±2% of the reference value.

Bus capacity margin pax h

bus_capacity_margin_pax_h

Passenger capacity minus demand

Scores if within ±2% of the reference value.

Cabinet load w

cabinet_load_w

Selected cabinet load

Scores if within ±2% of the reference value.

Cabinet load margin w

cabinet_load_margin_w

Cabinet capacity minus selected load

Scores if within ±2% of the reference value.

Feeder current

feeder_current_a

Cabinet feeder current

Scores if within ±2% of the reference value.

Feeder voltage drop

feeder_voltage_drop_percent

Feeder voltage drop at selected cabinet load

Scores if within ±2% of the reference value.

Voltage drop margin

voltage_drop_margin_percent

Allowable voltage drop minus computed voltage drop

Scores if within ±2% of the reference value.

Battery runtime h

battery_runtime_h

Backup battery runtime

Scores if within ±2% of the reference value.

Battery margin h

battery_margin_h

Backup runtime minus required duration

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing cabinet capacity, or genuine cabinet load exceedance

Hidden inputs

  • Packet variantpacket_variant
  • Bus capacity margin target pax hbus_capacity_margin_target_pax_h
  • Cabinet capacity margin wcabinet_capacity_margin_w
  • Cabinet load deficit wcabinet_load_deficit_w
  • Voltage drop margin targetvoltage_drop_margin_target_percent
  • Battery runtime margin hbattery_runtime_margin_h

Packet variant restricted to: clean, missing_cabinet_capacity, cabinet_load_exceeded

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Bus capacity margin target pax hbus_capacity_margin_target_pax_h
  • Cabinet capacity margin wcabinet_capacity_margin_w
  • Cabinet load deficit wcabinet_load_deficit_w
  • Voltage drop margin targetvoltage_drop_margin_target_percent
  • Battery runtime margin hbattery_runtime_margin_h

hard

Some inputs hidden

Subtle documentary defects: stale revisions, identity mismatch, scenario copy-forward, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Bus capacity margin target pax hbus_capacity_margin_target_pax_h
  • Cabinet capacity margin wcabinet_capacity_margin_w
  • Cabinet load deficit wcabinet_load_deficit_w
  • Voltage drop margin targetvoltage_drop_margin_target_percent
  • Battery runtime margin hbattery_runtime_margin_h

Packet variant restricted to: stale_signal_timing_revision, detector_controller_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a bus-priority corridor and roadside cabinet issue package covering one bus-priority operating scenario, signal timing sheet, detector set, signal controller, roadside cabinet, feeder, backup supply, and owner operations criterion.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, bus-priority operations plan, signal phasing and timing sheet, detector/controller schedule, cabinet load schedule, feeder/backup schedule, owner operations criterion, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the signal or cabinet system. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: bus-priority scenario, signal timing basis, detector set, signal controller, roadside cabinet, cabinet feeder, backup supply, and owner operations criterion.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, detector/controller mismatches, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required bus-priority, signal, detector/controller, cabinet, feeder/backup, owner-criterion, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the bus-priority scenario, signal group, detectors, controller, cabinet, feeder, backup supply, and owner case stay consistent? |26| RLR-03 | Signal and priority basis: are yellow interval, all-red interval, passenger-demand basis, and timing-plan basis traceable, current, and recomputable? |27| RLR-04 | Cabinet and priority adequacy: do the cabinet load and passenger capacity clear the source criteria for the same priority case and equipment set? |28| RLR-05 | Scenario consequence: is the same bus-priority operating scenario used across operations, signal timing, detector/controller, cabinet loads, feeder/backup, and owner criterion? |29| RLR-06 | Secondary-discipline resilience: are feeder voltage drop and backup runtime source-backed and internally consistent with the cabinet load schedule? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.40- Do not rename computed_evidence keys.41 42## Output43 44Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:45 46```json47{48 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],49 "identity_ledger": {50 "bus_priority_scenario": "...",51 "signal_timing_basis": "...",52 "detector_set": "...",53 "signal_controller": "...",54 "roadside_cabinet": "...",55 "cabinet_feeder": "...",56 "backup_supply": "...",57 "owner_operations_criterion": "..."58 },59 "review_matrix": {60 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},61 "RLR-02": {"status": "...", "evidence": "..."},62 "RLR-03": {"status": "...", "evidence": "..."},63 "RLR-04": {"status": "...", "evidence": "..."},64 "RLR-05": {"status": "...", "evidence": "..."},65 "RLR-06": {"status": "...", "evidence": "..."},66 "RLR-07": {"status": "...", "evidence": "..."},67 "RLR-08": {"status": "...", "evidence": "..."},68 "RLR-09": {"status": "...", "evidence": "..."}69 },70 "computed_evidence": {71 "yellow_interval_s": 0.0,72 "all_red_interval_s": 0.0,73 "bus_handling_capacity_pax_h": 0.0,74 "bus_capacity_margin_pax_h": 0.0,75 "cabinet_load_w": 0.0,76 "cabinet_load_margin_w": 0.0,77 "feeder_current_a": 0.0,78 "feeder_voltage_drop_percent": 0.0,79 "voltage_drop_margin_percent": 0.0,80 "battery_runtime_h": 0.0,81 "battery_margin_h": 0.082 },83 "findings": [84 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}85 ],86 "information_requests": [87 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}88 ],89 "action_register": [90 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}91 ],92 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",93 "claim_boundary_statement": "..."94}95```96 97Rules for the structured block:98 99- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.100- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.101- Every `insufficient_data` needs an information request naming the exact missing field and its source document.102- Every `not_applicable` needs a scope reason in its matrix `evidence`.103- Carried actions must appear in `action_register` with an owner.104- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.105- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.106

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban bus priority corridor

urban_bus_priority_corridor

Urban arterial bus-priority corridor with denser roadside equipment

urban-arterial-bus-priority-cabinetcbd-transit-priority-corridor
Parameter ranges
buses_per_hour
20 – 36
bus_occupancy_pax
55 – 85
detector_count
12 – 18
signal_heads_load_w
420 – 650
required_backup_h
3 – 6

Suburban bus priority corridor

suburban_bus_priority_corridor

Suburban bus-priority corridor with moderate equipment and demand

suburban-bus-priority-cabinetstation-access-transit-priority-corridor
Parameter ranges
buses_per_hour
12 – 24
bus_occupancy_pax
45 – 70
detector_count
8 – 14
signal_heads_load_w
300 – 520
required_backup_h
2 – 5

Example task

urban-arterial-bus-priority-cabinet-urban-bus-priority-corridor-preview — hard difficulty, some inputs hidden.

Urban arterial bus-priority corridor with denser roadside equipment. urban-arterial-bus-priority-cabinet. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

bus_approach_speed_kmh
35 to 65 km/h
bus_approach_grade_pct
-1 to 3 %
yellow_reaction_time_s
0.8 to 1.4 s
yellow_deceleration_m_s2
2.8 to 3.6 m/s2
intersection_width_m
18 to 36 m
bus_length_m
12 to 18 m
all_red_speed_kmh
35 to 55 km/h
buses_per_hour
20 to 36 buses/h
bus_occupancy_pax
55 to 85 passengers/bus
controller_load_w
150 to 260 W
detector_count
12 to 18
detector_load_w
5 to 12 W
transit_radio_load_w
50 to 110 W
vms_load_w
80 to 180 W
signal_heads_load_w
420 to 650 W
feeder_voltage_v
230 V
power_factor
0.9 to 0.98
feeder_length_km
0.08 to 0.35 km
conductor_resistance_ohm_km
0.45 to 1.1 ohm/km
battery_efficiency
0.85 to 0.95
required_backup_h
3 to 6 h

Executable tool: bus-priority-cabinet-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Packet variant

    packet_variant

  • Bus capacity margin target pax h

    bus_capacity_margin_target_pax_h

  • Cabinet load deficit w

    cabinet_load_deficit_w

  • Cabinet capacity margin w

    cabinet_capacity_margin_w

  • Voltage drop margin target

    voltage_drop_margin_target_percent

  • Battery runtime margin h

    battery_runtime_margin_h

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "yellow_interval_s": <number>,
  "all_red_interval_s": <number>,
  "bus_handling_capacity_pax_h": <number>,
  "bus_capacity_margin_pax_h": <number>,
  "cabinet_load_w": <number>,
  "cabinet_load_margin_w": <number>,
  "feeder_current_a": <number>,
  "feeder_voltage_drop_percent": <number>,
  "voltage_drop_margin_percent": <number>,
  "battery_runtime_h": <number>,
  "battery_margin_h": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • yellow_interval_s · scored within ±2%
  • all_red_interval_s · scored within ±2%
  • bus_handling_capacity_pax_h · scored within ±2%
  • bus_capacity_margin_pax_h · scored within ±2%
  • cabinet_load_w · scored within ±2%
  • cabinet_load_margin_w · scored within ±2%
  • feeder_current_a · scored within ±2%
  • feeder_voltage_drop_percent · scored within ±2%
  • voltage_drop_margin_percent · scored within ±2%
  • battery_runtime_h · scored within ±2%
  • battery_margin_h · scored within ±2%