LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalno-tool

Emergency Detour Device Issue Review Package

Review-native long-horizon task over a generated SSC-01 emergency detour device issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering detour plan, message library, device inventory, communications topology, power continuity, criteria, and comments. The task is to inventory the packet, preserve object identity, recompute only source-owned evidence, assign one status per review item, raise findings/information requests/actions, and make a readiness decision. Packet variants inject missing closure duration evidence, stale detour revisions, device inventory mismatch, scenario copy-forward, open comments, carried comments, or a genuine battery runtime failure.

no-tool: The model must reason numerically unaided.

Standards

FHWA MUTCDITS architectureNTCIProadside power

Tags

batterydetourelectricalissue-readinessitsreviewrfroadvms

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

30 inputs

Always given

Included directly in every task prompt.

23
Show 23 inputs

  • Vms character height

    vms_character_height_in

    MSG-SSC01-004 VMS character height

    8 – 14 in

  • Detour speed kmh

    detour_speed_kmh

    DETOUR-SSC01-004 detour approach speed

    50 – 90 km/h

  • Reading rate

    reading_rate_chars_s

    MSG-SSC01-004 reading rate

    3.5 – 5 chars/s

  • Cctv count

    cctv_count

    CCTV-SSC01-004 active CCTV count

    2 – 5

  • Cctv load

    cctv_load_mbps

    NET-SSC01-004 load per CCTV stream

    3 – 8 Mbps

  • Vms count

    vms_count

    VMS-SSC01-004 active VMS count

    1 – 3

  • Vms load

    vms_load_mbps

    NET-SSC01-004 load per VMS board

    1.5 – 4 Mbps

  • Radio load

    radio_load_mbps

    RF-SSC01-004 radio telemetry load

    0.5 – 2 Mbps

  • Controller load

    controller_load_mbps

    NET-SSC01-004 controller network load

    0.5 – 1.5 Mbps

  • Network overhead pct

    network_overhead_pct

    NET-SSC01-004 network overhead

    15 – 30 %

  • Rf tx power

    rf_tx_power_dbm

    RF-SSC01-004 transmit power

    18 – 24 dBm

  • Rf tx gain

    rf_tx_gain_db

    RF-SSC01-004 transmit antenna gain

    8 – 14 dB

  • Rf rx gain

    rf_rx_gain_db

    RF-SSC01-004 receive antenna gain

    8 – 14 dB

  • Rf path loss

    rf_path_loss_db

    RF-SSC01-004 path loss

    95 – 115 dB

  • Rf misc loss

    rf_misc_loss_db

    RF-SSC01-004 cable and connector loss

    2 – 5 dB

  • Rf fade margin

    rf_fade_margin_db

    RF-SSC01-004 reserved fade margin

    8 – 12 dB

  • Critical load

    critical_load_w

    PWR-SSC01-004 critical detour device load

    450 – 850 W

  • Battery efficiency

    battery_efficiency

    PWR-SSC01-004 battery efficiency

    0.85 – 0.95

  • Required detour duration

    required_detour_duration_h

    OPS-SSC01-004 required emergency closure duration

    4 – 12 h

  • Feeder length

    feeder_length_km

    PWR-SSC01-004 feeder length

    0.08 – 0.4 km

  • Conductor resistance

    conductor_resistance_ohm_km

    PWR-SSC01-004 conductor resistance

    0.5 – 1.2 ohm/km

  • Feeder voltage

    feeder_voltage_v

    PWR-SSC01-004 nominal feeder voltage

    230240

  • Power factor

    power_factor

    PWR-SSC01-004 load power factor

    0.9 – 0.98

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

7
Show 7 inputs

  • Message margin

    message_margin_chars

    Derivation margin between readable characters and selected message length

    Hidden at easy and medium and hard difficulty.

    2 – 8 chars

  • Network headroom margin

    network_headroom_margin_mbps

    Derivation margin between required network load and uplink capacity

    Hidden at easy and medium and hard difficulty.

    5 – 20 Mbps

  • Rf link margin

    rf_link_margin_db

    Derivation margin between received power and receiver sensitivity

    Hidden at easy and medium and hard difficulty.

    6 – 18 dB

  • Battery runtime margin

    battery_runtime_margin_h

    Derivation margin between battery runtime and required closure duration for passing packets

    Hidden at easy and medium and hard difficulty.

    1 – 5 h

  • Battery runtime deficit

    battery_runtime_deficit_h

    Derivation deficit between battery runtime and required closure duration for failing packets

    Hidden at easy and medium and hard difficulty.

    0.5 – 2 h

  • Voltage drop margin percent

    voltage_drop_margin_percent

    Derivation margin between voltage drop and allowable voltage drop

    Hidden at easy and medium and hard difficulty.

    2 – 5 %

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_closure_durationstale_detour_plan_revisiondevice_inventory_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedbattery_runtime_deficient

Scored outputs

23 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Detour and message basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Continuity adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Secondary-discipline resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Vms reading time s

vms_reading_time_s

Available reading time for the selected VMS message

Scores if within ±2% of the reference value.

Vms message margin chars

vms_message_margin_chars

Readable character capacity minus selected detour message length

Scores if within ±2% of the reference value.

Required network mbps

required_network_mbps

Required detour network load after overhead

Scores if within ±2% of the reference value.

Network headroom mbps

network_headroom_mbps

Uplink capacity minus required detour network load

Scores if within ±2% of the reference value.

Rf received power dbm

rf_received_power_dbm

RF received power after gains and losses

Scores if within ±2% of the reference value.

Rf link margin db

rf_link_margin_db

RF received power minus receiver sensitivity

Scores if within ±2% of the reference value.

Battery runtime h

battery_runtime_h

Battery runtime for the critical detour load

Scores if within ±2% of the reference value.

Battery margin h

battery_margin_h

Battery runtime minus required closure duration

Scores if within ±2% of the reference value.

Feeder voltage drop

feeder_voltage_drop_percent

Feeder voltage drop at the detour critical load

Scores if within ±2% of the reference value.

Voltage drop margin

voltage_drop_margin_percent

Allowable voltage drop minus computed voltage drop

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing closure duration, or genuine battery runtime failure

Hidden inputs

  • Packet variantpacket_variant
  • Message margin charsmessage_margin_chars
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Rf link margin dbrf_link_margin_db
  • Battery runtime margin hbattery_runtime_margin_h
  • Battery runtime deficit hbattery_runtime_deficit_h
  • Voltage drop marginvoltage_drop_margin_percent

Packet variant restricted to: clean, missing_closure_duration, battery_runtime_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Message margin charsmessage_margin_chars
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Rf link margin dbrf_link_margin_db
  • Battery runtime margin hbattery_runtime_margin_h
  • Battery runtime deficit hbattery_runtime_deficit_h
  • Voltage drop marginvoltage_drop_margin_percent

hard

Some inputs hidden

Subtle documentary defects: stale revisions, device mismatch, scenario copy-forward, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Message margin charsmessage_margin_chars
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Rf link margin dbrf_link_margin_db
  • Battery runtime margin hbattery_runtime_margin_h
  • Battery runtime deficit hbattery_runtime_deficit_h
  • Voltage drop marginvoltage_drop_margin_percent

Packet variant restricted to: stale_detour_plan_revision, device_inventory_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for an emergency detour device issue package covering one closure scenario, detour plan, traffic management plan, VMS message library, CCTV/VMS device set, RF link, ITS network uplink, and roadside cabinet power-continuity schedule.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, detour plan, VMS message library, device inventory, communications topology, power-continuity schedule, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the detour system. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: closure scenario, detour plan, traffic management plan, VMS device set, CCTV device set, communications link, network uplink, cabinet power supply, and message library.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, mismatched device IDs, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required detour, message, device, communications, power, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the closure scenario, detour plan, traffic management plan, VMS, CCTV, RF link, network uplink, power supply, and message library stay consistent? |26| RLR-03 | Detour and message basis: are the detour speed, VMS character height, reading rate, and selected message traceable, current, and recomputable? |27| RLR-04 | Continuity adequacy: does the battery runtime clear the required closure duration for the same critical detour device set? |28| RLR-05 | Scenario consequence: is the same emergency closure scenario used across the detour plan, device inventory, communications, power, and comments? |29| RLR-06 | Secondary-discipline resilience: are network headroom, RF link margin, and feeder voltage-drop margin source-backed and internally consistent? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.40- Do not rename computed_evidence keys.41 42## Output43 44Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:45 46```json47{48 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],49 "identity_ledger": {50 "closure_scenario": "...",51 "detour_plan": "...",52 "traffic_management_plan": "...",53 "vms_device_set": "...",54 "cctv_device_set": "...",55 "communications_link": "...",56 "network_uplink": "...",57 "cabinet_power_supply": "...",58 "message_library": "..."59 },60 "review_matrix": {61 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},62 "RLR-02": {"status": "...", "evidence": "..."},63 "RLR-03": {"status": "...", "evidence": "..."},64 "RLR-04": {"status": "...", "evidence": "..."},65 "RLR-05": {"status": "...", "evidence": "..."},66 "RLR-06": {"status": "...", "evidence": "..."},67 "RLR-07": {"status": "...", "evidence": "..."},68 "RLR-08": {"status": "...", "evidence": "..."},69 "RLR-09": {"status": "...", "evidence": "..."}70 },71 "computed_evidence": {72 "vms_reading_time_s": 0.0,73 "vms_message_margin_chars": 0.0,74 "required_network_mbps": 0.0,75 "network_headroom_mbps": 0.0,76 "rf_received_power_dbm": 0.0,77 "rf_link_margin_db": 0.0,78 "battery_runtime_h": 0.0,79 "battery_margin_h": 0.0,80 "feeder_voltage_drop_percent": 0.0,81 "voltage_drop_margin_percent": 0.082 },83 "findings": [84 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}85 ],86 "information_requests": [87 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}88 ],89 "action_register": [90 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}91 ],92 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",93 "claim_boundary_statement": "..."94}95```96 97Rules for the structured block:98 99- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.100- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.101- Every `insufficient_data` needs an information request naming the exact missing field and its source document.102- Every `not_applicable` needs a scope reason in its matrix `evidence`.103- Carried actions must appear in `action_register` with an owner.104- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.105- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.106

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban arterial detour

urban_arterial_detour

Urban arterial closure with higher device and network density

urban-arterial-emergency-detourcbd-fringe-closure-response
Parameter ranges
cctv_count
3 – 5
vms_count
2 – 3
critical_load_w
600 – 850
required_detour_duration_h
6 – 12

Suburban collector detour

suburban_collector_detour

Suburban collector closure with moderate device density

suburban-collector-emergency-detourschool-access-closure-response
Parameter ranges
cctv_count
2 – 3
vms_count
1 – 2
critical_load_w
450 – 700
required_detour_duration_h
4 – 10

Example task

urban-arterial-emergency-detour-urban-arterial-detour-preview — hard difficulty, some inputs hidden.

Urban arterial closure with higher device and network density. urban-arterial-emergency-detour. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

vms_character_height_in
8 to 14 in
detour_speed_kmh
50 to 90 km/h
reading_rate_chars_s
3.5 to 5 chars/s
cctv_count
3 to 5
cctv_load_mbps
3 to 8 Mbps
vms_count
2 to 3
vms_load_mbps
1.5 to 4 Mbps
radio_load_mbps
0.5 to 2 Mbps
controller_load_mbps
0.5 to 1.5 Mbps
network_overhead_pct
15 to 30 %
rf_tx_power_dbm
18 to 24 dBm
rf_tx_gain_db
8 to 14 dB
rf_rx_gain_db
8 to 14 dB
rf_path_loss_db
95 to 115 dB
rf_misc_loss_db
2 to 5 dB
rf_fade_margin_db
8 to 12 dB
critical_load_w
600 to 850 W
battery_efficiency
0.85 to 0.95
required_detour_duration_h
6 to 12 h
feeder_length_km
0.08 to 0.4 km
conductor_resistance_ohm_km
0.5 to 1.2 ohm/km
feeder_voltage_v
230 V
power_factor
0.9 to 0.98

Executable tool: emergency-detour-device-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Message margin chars

    message_margin_chars

  • Battery runtime deficit h

    battery_runtime_deficit_h

  • Rf link margin db

    rf_link_margin_db

  • Packet variant

    packet_variant

  • Voltage drop margin

    voltage_drop_margin_percent

  • Network headroom margin mbps

    network_headroom_margin_mbps

  • Battery runtime margin h

    battery_runtime_margin_h

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "vms_reading_time_s": <number>,
  "vms_message_margin_chars": <number>,
  "required_network_mbps": <number>,
  "network_headroom_mbps": <number>,
  "rf_received_power_dbm": <number>,
  "rf_link_margin_db": <number>,
  "battery_runtime_h": <number>,
  "battery_margin_h": <number>,
  "feeder_voltage_drop_percent": <number>,
  "voltage_drop_margin_percent": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • vms_reading_time_s · scored within ±2%
  • vms_message_margin_chars · scored within ±2%
  • required_network_mbps · scored within ±2%
  • network_headroom_mbps · scored within ±2%
  • rf_received_power_dbm · scored within ±2%
  • rf_link_margin_db · scored within ±2%
  • battery_runtime_h · scored within ±2%
  • battery_margin_h · scored within ±2%
  • feeder_voltage_drop_percent · scored within ±2%
  • voltage_drop_margin_percent · scored within ±2%