LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalno-tool

Road Visual Operations Issue Review Package

Review-native long-horizon task over a generated SSC-01 road visual operations issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering lighting, CCTV, VMS, ITS network, PoE, UPS, storm operations, criteria, and comments. The task is to inventory the packet, preserve object identity, recompute only source-owned evidence, assign one status per review item, raise findings/information requests/actions, and make a readiness decision. Packet variants inject missing PoE budget evidence, stale lighting revisions, device-register mismatch, scenario copy-forward, open comments, carried comments, or a genuine PoE capacity failure.

no-tool: The model must reason numerically unaided.

Standards

FHWA MUTCDIEEE 802.3btIES lighting workflowNTCIP

Tags

cctvelectricalissue-readinessitslightingpoereviewroad

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

30 inputs

Always given

Included directly in every task prompt.

24
Show 24 inputs

  • Lighting lux 01

    lighting_lux_01

    LGT-SSC01-003 grid point 01

    14 – 24 lux

  • Lighting lux 02

    lighting_lux_02

    LGT-SSC01-003 grid point 02

    14 – 24 lux

  • Lighting lux 03

    lighting_lux_03

    LGT-SSC01-003 grid point 03

    14 – 24 lux

  • Lighting lux 04

    lighting_lux_04

    LGT-SSC01-003 grid point 04

    14 – 24 lux

  • Lighting lux 05

    lighting_lux_05

    LGT-SSC01-003 grid point 05

    14 – 24 lux

  • Lighting lux 06

    lighting_lux_06

    LGT-SSC01-003 grid point 06

    14 – 24 lux

  • Cctv count

    cctv_count

    CCTV-SSC01-003 number of active cameras

    2 – 4

  • Cctv network load

    cctv_network_load_mbps

    NET-SSC01-003 network load per CCTV stream

    4 – 8 Mbps

  • Vms network load

    vms_network_load_mbps

    NET-SSC01-003 VMS network load

    2 – 5 Mbps

  • Sensor network load

    sensor_network_load_mbps

    NET-SSC01-003 storm sensor network load

    0.2 – 0.8 Mbps

  • Controller network load

    controller_network_load_mbps

    NET-SSC01-003 controller network load

    0.5 – 1.2 Mbps

  • Network overhead pct

    network_overhead_pct

    NET-SSC01-003 protocol overhead

    15 – 30 %

  • Cctv total bitrate

    cctv_total_bitrate_mbps

    CCTV-SSC01-003 total recording bitrate

    8 – 18 Mbps

  • Retention

    retention_days

    CCTV-SSC01-003 retention period

    71421

  • Storage overhead factor

    storage_overhead_factor

    CCTV-SSC01-003 storage overhead factor

    1.05 – 1.25

  • Cctv poe load

    cctv_poe_load_w

    PWR-SSC01-003 PoE load per CCTV camera

    10 – 25 W

  • Vms poe load

    vms_poe_load_w

    PWR-SSC01-003 VMS PoE load

    8 – 18 W

  • Sensor poe load

    sensor_poe_load_w

    PWR-SSC01-003 storm sensor PoE load

    4 – 10 W

  • Luminaire power

    luminaire_power_w

    PWR-SSC01-003 power per luminaire

    60 – 120 W

  • Luminaire count

    luminaire_count

    PWR-SSC01-003 selected luminaire count

    456

  • Device ups load

    device_ups_load_w

    PWR-SSC01-003 non-lighting UPS device load

    60 – 140 W

  • Ups autonomy

    ups_autonomy_h

    PWR-SSC01-003 required UPS autonomy

    234

  • Ups efficiency

    ups_efficiency

    PWR-SSC01-003 UPS efficiency

    0.85 – 0.95

  • Storm sensor level

    storm_sensor_level_m

    WLS-SSC01-003 storm water-level sensor reading

    40 – 45 m

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

6

  • Minimum uniformity margin

    minimum_uniformity_margin

    Derivation margin between source minimum uniformity criterion and computed uniformity

    Hidden at easy and medium and hard difficulty.

    0.05 – 0.15

  • Network headroom margin

    network_headroom_margin_mbps

    Derivation margin between total network load and uplink capacity

    Hidden at easy and medium and hard difficulty.

    5 – 15 Mbps

  • Poe headroom margin

    poe_headroom_margin_w

    Derivation margin between PoE load and switch budget for passing packets

    Hidden at easy and medium and hard difficulty.

    10 – 40 W

  • Poe headroom deficit

    poe_headroom_deficit_w

    Derivation deficit between PoE load and switch budget for failing packets

    Hidden at easy and medium and hard difficulty.

    5 – 20 W

  • Water level margin

    water_level_margin_m

    Derivation margin between storm sensor level and alarm threshold

    Hidden at easy and medium and hard difficulty.

    0.2 – 0.8 m

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_poe_switch_budgetstale_lighting_grid_revisiondevice_register_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedpoe_budget_exceeded

Scored outputs

24 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Visual operations basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Visual and field-device adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Secondary-discipline resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Average illuminance lux

average_illuminance_lux

Average illuminance across the lighting grid

Scores if within ±2% of the reference value.

Minimum illuminance lux

minimum_illuminance_lux

Minimum illuminance across the lighting grid

Scores if within ±2% of the reference value.

Uniformity ratio

uniformity_ratio

Minimum illuminance divided by average illuminance

Scores if within ±2% of the reference value.

Minimum uniformity ratio

minimum_uniformity_ratio

Source-owned minimum uniformity criterion

Scores if within ±2% of the reference value.

Total network load mbps

total_network_load_mbps

Total communications load after overhead

Scores if within ±2% of the reference value.

Network headroom mbps

network_headroom_mbps

Uplink capacity minus total communications load

Scores if within ±2% of the reference value.

Total cctv storage tb

total_cctv_storage_tb

CCTV storage required for source retention period

Scores if within ±2% of the reference value.

Poe load w

poe_load_w

Total field-device PoE load

Scores if within ±2% of the reference value.

Poe headroom w

poe_headroom_w

PoE switch budget minus field-device load

Scores if within ±2% of the reference value.

Water level margin

water_level_margin_m

Storm alarm threshold minus sensor level

Scores if within ±2% of the reference value.

Ups energy kwh

ups_energy_kwh

UPS energy required for luminaires and field devices

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing PoE budget, or genuine PoE capacity failure

Hidden inputs

  • Packet variantpacket_variant
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Poe headroom margin wpoe_headroom_margin_w
  • Poe headroom deficit wpoe_headroom_deficit_w
  • Water level marginwater_level_margin_m
  • Minimum uniformity marginminimum_uniformity_margin

Packet variant restricted to: clean, missing_poe_switch_budget, poe_budget_exceeded

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Poe headroom margin wpoe_headroom_margin_w
  • Poe headroom deficit wpoe_headroom_deficit_w
  • Water level marginwater_level_margin_m
  • Minimum uniformity marginminimum_uniformity_margin

hard

Some inputs hidden

Subtle documentary defects: stale revisions, device mismatch, scenario copy-forward, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Network headroom margin mbpsnetwork_headroom_margin_mbps
  • Poe headroom margin wpoe_headroom_margin_w
  • Poe headroom deficit wpoe_headroom_deficit_w
  • Water level marginwater_level_margin_m
  • Minimum uniformity marginminimum_uniformity_margin

Packet variant restricted to: stale_lighting_grid_revision, device_register_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a road visual operations issue package covering one road segment, lighting grid, luminaire group, CCTV schedule, VMS device, ITS network, PoE/UPS schedule, storm sensor, and night storm operating case.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, road segment and lighting grid, device register, network and storage schedule, PoE and UPS schedule, storm operations note, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the road systems. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: road segment, lighting grid, luminaire group, CCTV schedule, VMS device, ITS network, cabinet power/PoE, storm sensor, operating case, and datum.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, mismatched device IDs, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required visual, device, network, power, storm, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the road segment, lighting grid, luminaires, CCTV, VMS, storm sensor, cabinet, network, PoE switch, and operating case stay consistent? |26| RLR-03 | Visual operations basis: are lighting grid and method traceable, current, and recomputable? |27| RLR-04 | Visual/field-device adequacy: do controlling lighting uniformity and PoE budget clear the source criterion for the same device set? |28| RLR-05 | Scenario consequence: is the same night/storm operating case used across lighting, CCTV/VMS, network, power, and storm operations? |29| RLR-06 | Secondary-discipline resilience: are network headroom, CCTV storage, storm margin, and UPS energy source-backed and internally consistent? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.40- Do not rename computed_evidence keys.41 42## Output43 44Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:45 46```json47{48 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],49 "identity_ledger": {50 "road_segment": "...",51 "lighting_grid": "...",52 "luminaire_group": "...",53 "cctv_schedule": "...",54 "vms_device": "...",55 "its_network": "...",56 "cabinet_power_poe": "...",57 "storm_sensor": "...",58 "operating_case": "...",59 "datum": "..."60 },61 "review_matrix": {62 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},63 "RLR-02": {"status": "...", "evidence": "..."},64 "RLR-03": {"status": "...", "evidence": "..."},65 "RLR-04": {"status": "...", "evidence": "..."},66 "RLR-05": {"status": "...", "evidence": "..."},67 "RLR-06": {"status": "...", "evidence": "..."},68 "RLR-07": {"status": "...", "evidence": "..."},69 "RLR-08": {"status": "...", "evidence": "..."},70 "RLR-09": {"status": "...", "evidence": "..."}71 },72 "computed_evidence": {73 "average_illuminance_lux": 0.0,74 "minimum_illuminance_lux": 0.0,75 "uniformity_ratio": 0.0,76 "minimum_uniformity_ratio": 0.0,77 "total_network_load_mbps": 0.0,78 "network_headroom_mbps": 0.0,79 "total_cctv_storage_tb": 0.0,80 "poe_load_w": 0.0,81 "poe_headroom_w": 0.0,82 "water_level_margin_m": 0.0,83 "ups_energy_kwh": 0.084 },85 "findings": [86 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}87 ],88 "information_requests": [89 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}90 ],91 "action_register": [92 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}93 ],94 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",95 "claim_boundary_statement": "..."96}97```98 99Rules for the structured block:100 101- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.102- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.103- Every `insufficient_data` needs an information request naming the exact missing field and its source document.104- Every `not_applicable` needs a scope reason in its matrix `evidence`.105- Carried actions must appear in `action_register` with an owner.106- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.107- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.108

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban arterial visual ops

urban_arterial_visual_ops

Urban arterial road segment with higher visual systems density

urban-arterial-night-stormcbd-fringe-incident-response
Parameter ranges
lighting_lux_01
16 – 24
lighting_lux_02
16 – 24
lighting_lux_03
16 – 24
lighting_lux_04
15 – 23
lighting_lux_05
16 – 24
lighting_lux_06
15 – 23
cctv_count
3 – 4
cctv_total_bitrate_mbps
11 – 18

Suburban collector visual ops

suburban_collector_visual_ops

Suburban collector road segment with moderate field-device load

suburban-collector-night-stormschool-access-visual-ops
Parameter ranges
lighting_lux_01
14 – 21
lighting_lux_02
14 – 21
lighting_lux_03
14 – 21
lighting_lux_04
14 – 20
lighting_lux_05
14 – 21
lighting_lux_06
14 – 20
cctv_count
2 – 3
cctv_total_bitrate_mbps
8 – 14

Example task

urban-arterial-night-storm-urban-arterial-visual-ops-preview — hard difficulty, some inputs hidden.

Urban arterial road segment with higher visual systems density. urban-arterial-night-storm. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

lighting_lux_01
16 to 24 lux
lighting_lux_02
16 to 24 lux
lighting_lux_03
16 to 24 lux
lighting_lux_04
15 to 23 lux
lighting_lux_05
16 to 24 lux
lighting_lux_06
15 to 23 lux
cctv_count
3 to 4
cctv_network_load_mbps
4 to 8 Mbps
vms_network_load_mbps
2 to 5 Mbps
sensor_network_load_mbps
0.2 to 0.8 Mbps
controller_network_load_mbps
0.5 to 1.2 Mbps
network_overhead_pct
15 to 30 %
cctv_total_bitrate_mbps
11 to 18 Mbps
retention_days
7 days
storage_overhead_factor
1.05 to 1.25
cctv_poe_load_w
10 to 25 W
vms_poe_load_w
8 to 18 W
sensor_poe_load_w
4 to 10 W
luminaire_power_w
60 to 120 W
luminaire_count
4
device_ups_load_w
60 to 140 W
ups_autonomy_h
2 h
ups_efficiency
0.85 to 0.95
storm_sensor_level_m
40 to 45 m

Executable tool: road-visual-operations-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Packet variant

    packet_variant

  • Poe headroom margin w

    poe_headroom_margin_w

  • Minimum uniformity margin

    minimum_uniformity_margin

  • Water level margin

    water_level_margin_m

  • Network headroom margin mbps

    network_headroom_margin_mbps

  • Poe headroom deficit w

    poe_headroom_deficit_w

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "average_illuminance_lux": <number>,
  "minimum_illuminance_lux": <number>,
  "uniformity_ratio": <number>,
  "minimum_uniformity_ratio": <number>,
  "total_network_load_mbps": <number>,
  "network_headroom_mbps": <number>,
  "total_cctv_storage_tb": <number>,
  "poe_load_w": <number>,
  "poe_headroom_w": <number>,
  "water_level_margin_m": <number>,
  "ups_energy_kwh": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • average_illuminance_lux · scored within ±2%
  • minimum_illuminance_lux · scored within ±2%
  • uniformity_ratio · scored within ±2%
  • minimum_uniformity_ratio · scored within ±2%
  • total_network_load_mbps · scored within ±2%
  • network_headroom_mbps · scored within ±2%
  • total_cctv_storage_tb · scored within ±2%
  • poe_load_w · scored within ±2%
  • poe_headroom_w · scored within ±2%
  • water_level_margin_m · scored within ±2%
  • ups_energy_kwh · scored within ±2%