LIVEdataset aec-bench@releasetasks 552models 18disciplines 5last submission … · built …

civilno-tool

Road Low Point Issue Review Package

Review-native long-horizon task over a generated SSC-01 road low-point issue packet. The agent receives a multi-file source packet in /workspace/sources/ (document register, road geometry, drainage package, field equipment, power/comms, traffic operations, criteria and comments), inventories it, preserves corridor identity, checks the package's own calculations as review evidence, assigns one status per review item, raises findings/information requests/actions, and issues a readiness decision. Packet variants inject missing evidence, stale revisions, datum mismatches, scenario copy-forward, open comments, or genuine criterion failures. Scored by a stage-gated custom verifier, not per-key answer matching.

no-tool: The model must reason numerically unaided.

Standards

FHWA HEC-22 FHWA MUTCD

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

01
Template
The reusable contract shown on this page.
02
Scenario
An archetype and site context are sampled.
03
Difficulty tier
Inputs may be hidden at harder tiers.
04
Task prompt
The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

36 inputs

Always given

Included directly in every task prompt.

Show 28 inputs

Runoff coefficient
runoff_coefficient
DRN-CAT-01 runoff coefficient for the contributing low-point catchment
0.75 – 0.88
Rainfall intensity
rainfall_intensity_mm_h
STORM-01 rainfall intensity for the low-point design burst
80 – 120 mm/h
Catchment area
catchment_area_ha
DRN-CAT-01 contributing catchment area draining to LP-01
1 – 1.8 ha
Upstream bypass flow
upstream_bypass_flow_m3_s
DRN-BYP-01 upstream inlet bypass flow reaching LP-01
0.03 – 0.09 m3/s
Cross slope pct
cross_slope_pct
RD-SSC01-001 pavement crossfall at LP-01
2 – 3 %
Longitudinal slope pct
longitudinal_slope_pct
RD-SSC01-001 approach longitudinal slope into LP-01
0.5 – 1.2 %
Gutter mannings n
gutter_mannings_n
DRN-GUT-01 Manning roughness for the pavement gutter
0.013 – 0.017
Road low point level
road_low_point_level_m
LP-01 pavement level at the sag point (AHD)
40 – 46 m
Inlet efficiency
inlet_efficiency
DRN-PIT-01 low-point inlet interception efficiency
0.75 – 0.9
Inlet capture capacity
inlet_capture_capacity_m3_s
DRN-PIT-01 maximum inlet capture capacity
0.25 – 0.4 m3/s
Pipe diameter
pipe_diameter_mm
DRN-PIPE-01 storm-drain diameter leaving DRN-PIT-01
450525600
Pipe length
pipe_length_m
DRN-PIPE-01 reach length to the downstream tailwater node
30 – 60 m
Pipe mannings n
pipe_mannings_n
DRN-PIPE-01 full-pipe Manning roughness
0.011 – 0.014
Pit loss coefficient
pit_loss_coefficient
HGL-01 pit and junction loss coefficient at DRN-PIT-01
0.5 – 1.5
Minimum cabinet freeboard
minimum_cabinet_freeboard_m
CRIT-SSC01-001 minimum freeboard from the controlling water level to the CAB-01 pad
0.2 – 0.4 m
Road design speed kmh
road_design_speed_kmh
RD-SSC01-001 design speed at the low point
50607080
Vms character height
vms_character_height_in
VMS-01 character height
1218
Reading rate
reading_rate_chars_s
TOPS-SSC01-CASE-01 driver reading rate for VMS legibility
3.5 – 5 chars/s
Camera count
camera_count
ITS-NET-01 CCTV cameras served from CAB-01
2 – 4
Camera data rate
camera_data_rate_mbps
ITS-NET-01 data rate per CCTV camera
4 – 8 Mbps
Vms data rate
vms_data_rate_mbps
ITS-NET-01 VMS-01 data rate
0.5 – 1.5 Mbps
Controller data rate
controller_data_rate_mbps
ITS-NET-01 controller data rate
0.2 – 0.6 Mbps
Sensor data rate
sensor_data_rate_mbps
ITS-NET-01 water-level sensor data rate
0.1 – 0.3 Mbps
Network overhead pct
network_overhead_pct
ITS-NET-01 protocol overhead allowance
15 – 25 %
Future capacity buffer pct
future_capacity_buffer_pct
ITS-NET-01 future capacity buffer
20 – 30 %
Required autonomy
required_autonomy_h
CRIT-SSC01-001 required battery autonomy for CAB-01 critical load
468
Battery efficiency
battery_efficiency
BATT-01 usable-energy efficiency
0.85 – 0.95
Critical load
critical_load_w
CAB-01 critical load during the event
150 – 400 W

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

Show 8 inputs

Allowable spread margin
allowable_spread_margin_m
Derivation margin between computed spread and the CRIT-SSC01-001 allowable spread
Hidden at easy and medium and hard difficulty.
0.3 – 0.9 m
Hgl clearance target
hgl_clearance_target_m
Derivation clearance between the upstream HGL and the LP-01 pavement level
Hidden at easy and medium and hard difficulty.
0.3 – 0.8 m
Cabinet freeboard margin
cabinet_freeboard_margin_m
Derivation margin above minimum freeboard used to set the CAB-01 pad level
Hidden at easy and medium and hard difficulty.
0.1 – 0.35 m
Cabinet freeboard deficit
cabinet_freeboard_deficit_m
Derivation deficit below minimum freeboard for the freeboard-deficient variant
Hidden at easy and medium and hard difficulty.
0.05 – 0.2 m
Vms message margin chars target
vms_message_margin_chars_target
Derivation margin between readable characters and the selected message length
Hidden at easy and medium and hard difficulty.
2 – 6 chars
Network headroom margin
network_headroom_margin_mbps
Derivation margin between required network load and the provisioned uplink
Hidden at easy and medium and hard difficulty.
5 – 15 Mbps
Battery runtime margin
battery_runtime_margin_h
Derivation margin between required autonomy and provisioned battery runtime
Hidden at easy and medium and hard difficulty.
0.5 – 2 h
Packet variant
packet_variant
Hidden packet-defect variant controlling gold review statuses
Hidden at easy and medium and hard difficulty.
cleanmissing_cabinet_levelstale_hgl_revisionchainage_datum_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedfreeboard_deficient

Scored outputs

22 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Drainage basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Equipment exposure status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Traffic operation consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Power/comms resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Peak runoff

peak_runoff_m3_s

Rational-method peak runoff evidence

Scores if within ±2% of the reference value.

Gutter approach flow

gutter_approach_flow_m3_s

Gutter approach flow evidence

Scores if within ±2% of the reference value.

Spread width

spread_width_m

Triangular gutter spread evidence

Scores if within ±2% of the reference value.

Allowable spread

allowable_spread_m

Criteria allowable spread (source-owned)

Scores if within ±0% of the reference value.

Controlling water level

controlling_water_level_m

Controlling water level evidence

Scores if within ±2% of the reference value.

Cabinet freeboard

cabinet_freeboard_m

Cabinet freeboard evidence (absent when the pad level is missing)

Scores if within ±2% of the reference value.

Vms message margin chars

vms_message_margin_chars

VMS message margin evidence

Scores if within ±2% of the reference value.

Battery runtime h

battery_runtime_h

Battery runtime evidence

Scores if within ±2% of the reference value.

Network headroom mbps

network_headroom_mbps

Network headroom evidence

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing evidence, or a genuine criterion failure

Hidden inputs

Packet variantpacket_variant
Allowable spread marginallowable_spread_margin_m
Cabinet freeboard margincabinet_freeboard_margin_m
Cabinet freeboard deficitcabinet_freeboard_deficit_m
Hgl clearance targethgl_clearance_target_m
Vms message margin chars targetvms_message_margin_chars_target
Network headroom margin mbpsnetwork_headroom_margin_mbps
Battery runtime margin hbattery_runtime_margin_h

Packet variant restricted to: clean, missing_cabinet_level, freeboard_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

Packet variantpacket_variant
Allowable spread marginallowable_spread_margin_m
Cabinet freeboard margincabinet_freeboard_margin_m
Cabinet freeboard deficitcabinet_freeboard_deficit_m
Hgl clearance targethgl_clearance_target_m
Vms message margin chars targetvms_message_margin_chars_target
Network headroom margin mbpsnetwork_headroom_margin_mbps
Battery runtime margin hbattery_runtime_margin_h

hard

Some inputs hidden

Subtle documentary defects: stale revisions, datum drift, scenario copy-forward, carried comments

Hidden inputs

Packet variantpacket_variant
Allowable spread marginallowable_spread_margin_m
Cabinet freeboard margincabinet_freeboard_margin_m
Cabinet freeboard deficitcabinet_freeboard_deficit_m
Hgl clearance targethgl_clearance_target_m
Vms message margin chars targetvms_message_margin_chars_target
Network headroom margin mbpsnetwork_headroom_margin_mbps
Battery runtime margin hbattery_runtime_margin_h

Packet variant restricted to: stale_hgl_revision, chainage_datum_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a road-corridor issue package covering a sag low point near roadside field equipment.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, road geometry, a drainage design package, a field equipment layout, a power and network schedule, a traffic operations case, and a criteria memo with review comments. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the package. Your job is to decide whether it is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build a corridor identity ledger: road segment, chainage frame, datum, low point, cabinet, VMS, storm case, network case, and battery.113. Check for source conflicts, stale revisions, contradictory datums, and missing evidence first.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required source documents present with IDs and revisions? |25| RLR-02 | Object identity: do chainage, datum, low point, cabinet, storm case, and scenario stay consistent across documents? |26| RLR-03 | Drainage basis: are the package's runoff, spread, and HGL results traceable to the stated storm case and current tailwater basis, and do they reconcile when recomputed? |27| RLR-04 | Equipment exposure: is the CAB-01 pad level adequate against the controlling water level and the required freeboard? |28| RLR-05 | Traffic operation consequence: does the VMS legibility case use this corridor's design speed and scenario? |29| RLR-06 | Power/comms resilience: do battery runtime and network headroom clear the criteria using source-backed values? |30| RLR-07 | Comment and action closure: is every review comment closed, or carried with an owner and agreed action, or blocked by named missing data? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, or acceptance claims? |33 34## Output35 36Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:37 38```json39{40  "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],41  "identity_ledger": {42    "road_segment": "...",43    "low_point": "...",44    "low_point_chainage": "...",45    "datum": "...",46    "cabinet": "...",47    "vms": "...",48    "storm_case": "...",49    "network_case": "...",50    "battery": "..."51  },52  "review_matrix": {53    "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},54    "RLR-02": {"status": "...", "evidence": "..."},55    "RLR-03": {"status": "...", "evidence": "..."},56    "RLR-04": {"status": "...", "evidence": "..."},57    "RLR-05": {"status": "...", "evidence": "..."},58    "RLR-06": {"status": "...", "evidence": "..."},59    "RLR-07": {"status": "...", "evidence": "..."},60    "RLR-08": {"status": "...", "evidence": "..."},61    "RLR-09": {"status": "...", "evidence": "..."}62  },63  "computed_evidence": {64    "peak_runoff_m3_s": 0.0,65    "gutter_approach_flow_m3_s": 0.0,66    "spread_width_m": 0.0,67    "allowable_spread_m": 0.0,68    "controlling_water_level_m": 0.0,69    "cabinet_freeboard_m": 0.0,70    "vms_message_margin_chars": 0.0,71    "battery_runtime_h": 0.0,72    "network_headroom_mbps": 0.073  },74  "findings": [75    {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}76  ],77  "information_requests": [78    {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}79  ],80  "action_register": [81    {"action": "...", "owner": "...", "linked_item": "RLR-0X"}82  ],83  "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",84  "claim_boundary_statement": "..."85}86```87 88Rules for the structured block:89 90- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet (then raise the matching information request instead).91- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.92- Every `insufficient_data` needs an information request naming the exact missing field and its source document.93- Every `not_applicable` needs a scope reason in its matrix `evidence`.94- Carried actions must appear in `action_register` with an owner.95- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.96- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.97

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban arterial

urban_arterial

Urban arterial sag with kerbed gutter and dense ITS assets

urban-arterial-sagcbd-underpass-approach

Parameter ranges

rainfall_intensity_mm_h: 95 – 120
catchment_area_ha: 1.2 – 1.8

Suburban collector

suburban_collector

Suburban collector sag with roadside field cabinet

suburban-collector-sagindustrial-access-low-point

Parameter ranges

rainfall_intensity_mm_h: 80 – 105
catchment_area_ha: 1 – 1.5

Example task

urban-arterial-sag-urban-arterial-preview — hard difficulty, some inputs hidden.

Urban arterial sag with kerbed gutter and dense ITS assets. urban-arterial-sag. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

runoff_coefficient: 0.75 to 0.88
rainfall_intensity_mm_h: 95 to 120 mm/h
catchment_area_ha: 1.2 to 1.8 ha
upstream_bypass_flow_m3_s: 0.03 to 0.09 m3/s
cross_slope_pct: 2 to 3 %
longitudinal_slope_pct: 0.5 to 1.2 %
gutter_mannings_n: 0.013 to 0.017
road_low_point_level_m: 40 to 46 m
inlet_efficiency: 0.75 to 0.9
inlet_capture_capacity_m3_s: 0.25 to 0.4 m3/s
pipe_diameter_mm: 450
pipe_length_m: 30 to 60 m
pipe_mannings_n: 0.011 to 0.014
pit_loss_coefficient: 0.5 to 1.5
minimum_cabinet_freeboard_m: 0.2 to 0.4 m
road_design_speed_kmh: 50 km/h
vms_character_height_in: 12 in
reading_rate_chars_s: 3.5 to 5 chars/s
camera_count: 2 to 4
camera_data_rate_mbps: 4 to 8 Mbps
vms_data_rate_mbps: 0.5 to 1.5 Mbps
controller_data_rate_mbps: 0.2 to 0.6 Mbps
sensor_data_rate_mbps: 0.1 to 0.3 Mbps
network_overhead_pct: 15 to 25 %
future_capacity_buffer_pct: 20 to 30 %
required_autonomy_h: 4 h
battery_efficiency: 0.85 to 0.95
critical_load_w: 150 to 400 W

Executable tool: road-low-point-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Packet variant
packet_variant
Cabinet freeboard deficit
cabinet_freeboard_deficit_m
Cabinet freeboard margin
cabinet_freeboard_margin_m
Vms message margin chars target
vms_message_margin_chars_target
Network headroom margin mbps
network_headroom_margin_mbps
Hgl clearance target
hgl_clearance_target_m
Battery runtime margin h
battery_runtime_margin_h
Allowable spread margin
allowable_spread_margin_m

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "peak_runoff_m3_s": <number>,
  "gutter_approach_flow_m3_s": <number>,
  "spread_width_m": <number>,
  "allowable_spread_m": <number>,
  "controlling_water_level_m": <number>,
  "cabinet_freeboard_m": <number>,
  "vms_message_margin_chars": <number>,
  "battery_runtime_h": <number>,
  "network_headroom_mbps": <number>
}

rlr_01_status · scored within ±0%
rlr_02_status · scored within ±0%
rlr_03_status · scored within ±0%
rlr_04_status · scored within ±0%
rlr_05_status · scored within ±0%
rlr_06_status · scored within ±0%
rlr_07_status · scored within ±0%
rlr_08_status · scored within ±0%
rlr_09_status · scored within ±0%
readiness_code · scored within ±0%
required_findings_count · scored within ±0%
required_information_requests_count · scored within ±0%
required_carried_actions_count · scored within ±0%
peak_runoff_m3_s · scored within ±2%
gutter_approach_flow_m3_s · scored within ±2%
spread_width_m · scored within ±2%
allowable_spread_m · scored within ±0%
controlling_water_level_m · scored within ±2%
cabinet_freeboard_m · scored within ±2%
vms_message_margin_chars · scored within ±2%
battery_runtime_h · scored within ±2%
network_headroom_mbps · scored within ±2%

Road Low Point Issue Review Package

How this task is generated

Template

Scenario

Difficulty tier

Task prompt

Parameters

Inputs

Scored outputs

Rlr 01 status

Rlr 02 status

Rlr 03 status

Rlr 04 status

Rlr 05 status

Rlr 06 status

Rlr 07 status

Rlr 08 status

Rlr 09 status

Readiness code

Required findings count

Required information requests count

Required carried actions count

Peak runoff

Gutter approach flow

Spread width

Allowable spread

Controlling water level

Cabinet freeboard

Vms message margin chars

Battery runtime h

Network headroom mbps

Difficulty

easy

medium

hard

Task bundle

Scenario archetypes

Urban arterial

Suburban collector

Example task

The model sees

The model must infer

The model must produce