LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalno-tool

Roadside Cabinet Serviceability Issue Review Package

Review-native long-horizon task over a generated SSC-01 roadside cabinet serviceability issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering cabinet setout/elevation, flood HGL and inundation, enclosure heat derating, critical load and backup energy, feeder/access, owner criteria, and comments. The task is to inventory the packet, preserve object identity, recompute only source-owned evidence, assign one status per review item, raise findings/information requests/actions, and make a readiness decision. Packet variants inject missing heat-derating evidence, stale enclosure revisions, cabinet/event mismatch, scenario copy-forward, open comments, carried comments, or a genuine thermal-capacity deficiency.

no-tool: The model must reason numerically unaided.

Standards

backup energy workflowelectrical feeder voltage droproadside cabinet serviceabilitythermal enclosure derating

Tags

backupcabinetelectricalfeederissue-readinessreviewroadthermal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

23 inputs

Always given

Included directly in every task prompt.

16
Show 16 inputs

  • Hgl level

    hgl_level_m

    HGL-SSC01-007 source HGL level

    41.2 – 42.7 m

  • Inundation level

    inundation_level_m

    HGL-SSC01-007 source inundation level

    41.1 – 42.8 m

  • Minimum freeboard

    minimum_freeboard_m

    OPS-SSC01-007 minimum cabinet freeboard

    0.3 – 0.6 m

  • Enclosure capacity w at reference temp

    enclosure_capacity_w_at_reference_temp

    HEAT-SSC01-007 reference enclosure thermal capacity

    900 – 1650 W

  • Reference temperature

    reference_temperature_c

    HEAT-SSC01-007 reference temperature

    24 – 28 C

  • Event temperature

    event_temperature_c

    HEAT-SSC01-007 event temperature

    39 – 50 C

  • Derate pct

    derate_pct_per_c

    HEAT-SSC01-007 enclosure derating rate

    0.7 – 1.6 % per C

  • Battery efficiency

    battery_efficiency

    BATT-SSC01-007 usable battery efficiency

    0.82 – 0.92

  • Required backup

    required_backup_h

    OPS-SSC01-007 required backup duration

    4 – 12 h

  • Feeder length

    feeder_length_km

    FEED-SSC01-007 feeder route length

    0.08 – 0.55 km

  • Conductor resistance

    conductor_resistance_ohm_km

    FEED-SSC01-007 conductor resistance

    0.25 – 0.9 ohm/km

  • Feeder voltage

    feeder_voltage_v

    FEED-SSC01-007 feeder voltage

    230 – 240 V

  • Power factor

    power_factor

    FEED-SSC01-007 critical load power factor

    0.82 – 0.96

  • Road lighting power

    road_lighting_power_w

    FEED-SSC01-007 road lighting power

    700 – 2400 W

  • Annual operating hours

    annual_operating_hours

    FEED-SSC01-007 annual lighting operating hours

    3800 – 4300 h/y

  • Lit area

    lit_area_m2

    FEED-SSC01-007 lit road area

    900 – 2600 m2

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

7
Show 7 inputs

  • Flood freeboard margin target

    flood_freeboard_margin_target_m

    Derivation margin between cabinet freeboard and minimum freeboard for passing packets

    Hidden at easy and medium and hard difficulty.

    0.08 – 0.35 m

  • Thermal margin target

    thermal_margin_target_w

    Derivation margin between derated enclosure capacity and critical load for passing packets

    Hidden at easy and medium and hard difficulty.

    70 – 260 W

  • Thermal deficit

    thermal_deficit_w

    Derivation deficit between critical load and derated enclosure capacity for failing packets

    Hidden at easy and medium and hard difficulty.

    35 – 180 W

  • Battery runtime margin

    battery_runtime_margin_h

    Derivation margin between battery runtime and required backup duration

    Hidden at easy and medium and hard difficulty.

    1 – 5 h

  • Bess power margin

    bess_power_margin_kw

    Derivation margin between BESS inverter capacity and critical load

    Hidden at easy and medium and hard difficulty.

    0.2 – 1.2 kW

  • Voltage drop margin percent

    voltage_drop_margin_percent

    Derivation margin between allowable and computed feeder voltage drop

    Hidden at easy and medium and hard difficulty.

    1 – 3 %

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_derating_ratestale_enclosure_derating_revisioncabinet_event_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedthermal_capacity_deficient

Scored outputs

25 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Cabinet serviceability basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Cabinet serviceability adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Secondary power/access resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Cabinet freeboard

cabinet_freeboard_m

Cabinet pad level minus controlling water level

Scores if within ±2% of the reference value.

Flood freeboard margin

flood_freeboard_margin_m

Cabinet freeboard minus minimum freeboard

Scores if within ±2% of the reference value.

Thermal derated capacity w

thermal_derated_capacity_w

Derated enclosure thermal capacity

Scores if within ±200% of the reference value.

Thermal margin w

thermal_margin_w

Derated enclosure capacity minus critical load

Scores if within ±200% of the reference value.

Thermal utilization

thermal_utilization

Critical load divided by derated enclosure capacity

Scores if within ±1% of the reference value.

Battery runtime h

battery_runtime_h

Backup battery runtime

Scores if within ±5% of the reference value.

Battery margin h

battery_margin_h

Battery runtime minus required backup duration

Scores if within ±5% of the reference value.

Bess power margin

bess_power_margin_kw

BESS inverter capacity minus load kW

Scores if within ±5% of the reference value.

Bess energy margin kwh

bess_energy_margin_kwh

Usable battery energy minus required load energy

Scores if within ±5% of the reference value.

Feeder voltage drop

feeder_voltage_drop_percent

Computed feeder voltage drop

Scores if within ±2% of the reference value.

Voltage drop margin

voltage_drop_margin_percent

Allowable voltage drop minus computed voltage drop

Scores if within ±2% of the reference value.

Road lighting aeci kwh m2 y

road_lighting_aeci_kwh_m2_y

Annual road-lighting energy divided by lit area

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing derating rate, or genuine thermal-capacity deficiency

Hidden inputs

  • Packet variantpacket_variant
  • Flood freeboard margin targetflood_freeboard_margin_target_m
  • Thermal margin target wthermal_margin_target_w
  • Thermal deficit wthermal_deficit_w
  • Battery runtime margin hbattery_runtime_margin_h
  • Bess power marginbess_power_margin_kw
  • Voltage drop marginvoltage_drop_margin_percent

Packet variant restricted to: clean, missing_derating_rate, thermal_capacity_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Flood freeboard margin targetflood_freeboard_margin_target_m
  • Thermal margin target wthermal_margin_target_w
  • Thermal deficit wthermal_deficit_w
  • Battery runtime margin hbattery_runtime_margin_h
  • Bess power marginbess_power_margin_kw
  • Voltage drop marginvoltage_drop_margin_percent

hard

Some inputs hidden

Subtle documentary defects: stale revisions, cabinet/event mismatch, copied cases, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Flood freeboard margin targetflood_freeboard_margin_target_m
  • Thermal margin target wthermal_margin_target_w
  • Thermal deficit wthermal_deficit_w
  • Battery runtime margin hbattery_runtime_margin_h
  • Bess power marginbess_power_margin_kw
  • Voltage drop marginvoltage_drop_margin_percent

Packet variant restricted to: stale_enclosure_derating_revision, cabinet_event_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a roadside cabinet serviceability issue package covering one cabinet setout/elevation, flood HGL and inundation table, enclosure heat derating note, critical load and backup schedule, feeder/access note, owner serviceability criterion, and criteria/comments memo.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, cabinet setout/elevation note, flood HGL and inundation table, enclosure heat derating note, critical load and backup schedule, feeder/access note, owner serviceability criterion, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the cabinet, enclosure, battery, feeder, or lighting system. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: cabinet, flood/HGL table, heat derating note, critical load schedule, backup energy schedule, feeder/access note, serviceability scenario, and criteria memo.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, cabinet/event mismatches, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required cabinet setout, flood/HGL, heat derating, load/backup, feeder/access, owner-criterion, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the cabinet ID, chainage, flood/HGL event, heat event, critical load, backup schedule, feeder, and access note stay consistent? |26| RLR-03 | Cabinet serviceability basis: are flood freeboard, heat derating, backup runtime, and feeder voltage-drop basis traceable, current, and recomputable? |27| RLR-04 | Cabinet serviceability adequacy: do flood freeboard and thermal derated capacity clear the source criteria for the same cabinet and event case? |28| RLR-05 | Scenario consequence: is the same flood, heat, and outage event case used across cabinet setout, HGL, heat derating, load, backup, feeder, access, and owner criterion? |29| RLR-06 | Secondary power/access resilience: are backup energy, BESS power, feeder voltage drop, maintenance access, and road-lighting AECI source-backed and internally consistent with the selected cabinet load? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- RLR-08 passes when the readiness decision reconciles with the review matrix, findings, information requests, and action register.40- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.41- Do not rename computed_evidence keys.42 43## Output44 45Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:46 47```json48{49 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],50 "identity_ledger": {51 "cabinet": "...",52 "flood_hgl_table": "...",53 "heat_derating_note": "...",54 "critical_load_schedule": "...",55 "backup_energy_schedule": "...",56 "feeder_access_note": "...",57 "serviceability_scenario": "...",58 "criteria_memo": "..."59 },60 "review_matrix": {61 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},62 "RLR-02": {"status": "...", "evidence": "..."},63 "RLR-03": {"status": "...", "evidence": "..."},64 "RLR-04": {"status": "...", "evidence": "..."},65 "RLR-05": {"status": "...", "evidence": "..."},66 "RLR-06": {"status": "...", "evidence": "..."},67 "RLR-07": {"status": "...", "evidence": "..."},68 "RLR-08": {"status": "...", "evidence": "..."},69 "RLR-09": {"status": "...", "evidence": "..."}70 },71 "computed_evidence": {72 "cabinet_freeboard_m": 0.0,73 "flood_freeboard_margin_m": 0.0,74 "thermal_derated_capacity_w": 0.0,75 "thermal_margin_w": 0.0,76 "thermal_utilization": 0.0,77 "battery_runtime_h": 0.0,78 "battery_margin_h": 0.0,79 "bess_power_margin_kw": 0.0,80 "bess_energy_margin_kwh": 0.0,81 "feeder_voltage_drop_percent": 0.0,82 "voltage_drop_margin_percent": 0.0,83 "road_lighting_aeci_kwh_m2_y": 0.084 },85 "findings": [86 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}87 ],88 "information_requests": [89 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}90 ],91 "action_register": [92 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}93 ],94 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",95 "claim_boundary_statement": "..."96}97```98 99Rules for the structured block:100 101- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.102- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.103- Every `insufficient_data` needs an information request naming the exact missing field and its source document.104- Every `not_applicable` needs a scope reason in its matrix `evidence`.105- Carried actions must appear in `action_register` with an owner.106- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.107- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.108

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban roadside cabinet

urban_roadside_cabinet

Urban roadside cabinet with tighter feeder route and higher lighting load

urban-roadside-cabinetarterial-its-cabinet
Parameter ranges
feeder_length_km
0.08 – 0.32
road_lighting_power_w
1200 – 2400
lit_area_m2
900 – 1800

Suburban field cabinet

suburban_field_cabinet

Suburban field cabinet with longer feeder route and moderate lighting load

suburban-field-cabinetperi-urban-lighting-cabinet
Parameter ranges
feeder_length_km
0.2 – 0.55
road_lighting_power_w
700 – 1600
lit_area_m2
1400 – 2600

Example task

urban-roadside-cabinet-urban-roadside-cabinet-preview — hard difficulty, some inputs hidden.

Urban roadside cabinet with tighter feeder route and higher lighting load. urban-roadside-cabinet. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

hgl_level_m
41.2 to 42.7 m
inundation_level_m
41.1 to 42.8 m
minimum_freeboard_m
0.3 to 0.6 m
enclosure_capacity_w_at_reference_temp
900 to 1650 W
reference_temperature_c
24 to 28 C
event_temperature_c
39 to 50 C
derate_pct_per_c
0.7 to 1.6 % per C
battery_efficiency
0.82 to 0.92
required_backup_h
4 to 12 h
feeder_length_km
0.08 to 0.32 km
conductor_resistance_ohm_km
0.25 to 0.9 ohm/km
feeder_voltage_v
230 to 240 V
power_factor
0.82 to 0.96
road_lighting_power_w
1200 to 2400 W
annual_operating_hours
3800 to 4300 h/y
lit_area_m2
900 to 1800 m2

Executable tool: roadside-cabinet-serviceability-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Thermal deficit w

    thermal_deficit_w

  • Thermal margin target w

    thermal_margin_target_w

  • Packet variant

    packet_variant

  • Flood freeboard margin target

    flood_freeboard_margin_target_m

  • Bess power margin

    bess_power_margin_kw

  • Voltage drop margin

    voltage_drop_margin_percent

  • Battery runtime margin h

    battery_runtime_margin_h

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "cabinet_freeboard_m": <number>,
  "flood_freeboard_margin_m": <number>,
  "thermal_derated_capacity_w": <number>,
  "thermal_margin_w": <number>,
  "thermal_utilization": <number>,
  "battery_runtime_h": <number>,
  "battery_margin_h": <number>,
  "bess_power_margin_kw": <number>,
  "bess_energy_margin_kwh": <number>,
  "feeder_voltage_drop_percent": <number>,
  "voltage_drop_margin_percent": <number>,
  "road_lighting_aeci_kwh_m2_y": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • cabinet_freeboard_m · scored within ±2%
  • flood_freeboard_margin_m · scored within ±2%
  • thermal_derated_capacity_w · scored within ±200%
  • thermal_margin_w · scored within ±200%
  • thermal_utilization · scored within ±1%
  • battery_runtime_h · scored within ±5%
  • battery_margin_h · scored within ±5%
  • bess_power_margin_kw · scored within ±5%
  • bess_energy_margin_kwh · scored within ±5%
  • feeder_voltage_drop_percent · scored within ±2%
  • voltage_drop_margin_percent · scored within ±2%
  • road_lighting_aeci_kwh_m2_y · scored within ±2%