LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
civilno-tool

Driveway Access Safety Issue Review Package

Review-native long-horizon task over a generated SSC-01 driveway access safety issue packet. The agent receives a multi-file source packet in /workspace/sources/ covering access profile, road edge, culvert, tailwater, roadway spread, sight distance, owner criteria, and comments. The task is to inventory the packet, preserve object identity, recompute only source-owned evidence, assign one status per review item, raise findings/information requests/actions, and make a readiness decision. Packet variants inject missing road-edge evidence, stale access-profile revisions, culvert chainage mismatch, scenario copy-forward, open comments, carried comments, or a genuine freeboard deficiency.

no-tool: The model must reason numerically unaided.

Standards

AASHTO Green BookFHWA HEC-22driveway access designroad drainage workflow

Tags

accesscivilculvertdrivewayissue-readinessreviewroadsafety

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

25 inputs

Always given

Included directly in every task prompt.

20
Show 20 inputs

  • Driveway low level

    driveway_low_level_m

    ACCESS-SSC01-006 driveway low point level

    41.8 – 43.2 m

  • Driveway grade pct source

    driveway_grade_pct_source

    ACCESS-SSC01-006 source driveway grade used to derive the high point

    2 – 6.5 %

  • Driveway length

    driveway_length_m

    ACCESS-SSC01-006 driveway grade length

    14 – 32 m

  • Allowable driveway grade pct

    allowable_driveway_grade_pct

    OPS-SSC01-006 maximum driveway grade

    7 – 9 %

  • Culvert diameter

    culvert_diameter_m

    CULV-SSC01-006 culvert diameter

    0.75 – 1.2 m

  • Culvert mannings n

    culvert_mannings_n

    CULV-SSC01-006 culvert Manning roughness

    0.012 – 0.016

  • Culvert slope pct

    culvert_slope_pct

    CULV-SSC01-006 culvert slope

    0.8 – 2 %

  • Design flow

    design_flow_m3_s

    CULV-SSC01-006 design flow

    0.25 – 0.7 m3/s

  • Tailwater level

    tailwater_level_m

    TAIL-SSC01-006 tailwater level

    41.2 – 42.4 m

  • Headwater base depth

    headwater_base_depth_m

    TAIL-SSC01-006 base headwater depth allowance

    0.32 – 0.58 m

  • Headwater loss factor

    headwater_loss_factor_m

    TAIL-SSC01-006 ratio-squared headwater loss factor

    0.12 – 0.32 m

  • Minimum freeboard

    minimum_freeboard_m

    OPS-SSC01-006 minimum road-edge freeboard

    0.1 – 0.2 m

  • Gutter flow

    gutter_flow_m3_s

    ROAD-SSC01-006 gutter flow across the access path

    0.04 – 0.14 m3/s

  • Cross slope pct

    cross_slope_pct

    ROAD-SSC01-006 cross slope

    2 – 4 %

  • Longitudinal slope pct

    longitudinal_slope_pct

    ROAD-SSC01-006 longitudinal slope

    0.6 – 2 %

  • Gutter mannings n

    gutter_mannings_n

    ROAD-SSC01-006 gutter Manning roughness

    0.014 – 0.018

  • Access speed kmh

    access_speed_kmh

    SIGHT-SSC01-006 access sight-distance speed

    25 – 50 km/h

  • Sight reaction time

    sight_reaction_time_s

    SIGHT-SSC01-006 sight-distance reaction time

    2 – 2.7 s

  • Braking friction coefficient

    braking_friction_coefficient

    SIGHT-SSC01-006 braking friction coefficient

    0.32 – 0.42

  • Access grade pct

    access_grade_pct

    SIGHT-SSC01-006 signed access grade

    1 – 6 %

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

5

  • Freeboard margin target

    freeboard_margin_target_m

    Derivation margin between road-edge freeboard and minimum freeboard for passing packets

    Hidden at easy and medium and hard difficulty.

    0.08 – 0.35 m

  • Freeboard deficit

    freeboard_deficit_m

    Derivation deficit between road-edge freeboard and minimum freeboard for failing packets

    Hidden at easy and medium and hard difficulty.

    0.03 – 0.18 m

  • Spread margin target

    spread_margin_target_m

    Derivation margin between allowable and computed roadway spread

    Hidden at easy and medium and hard difficulty.

    0.4 – 2.5 m

  • Available sight margin

    available_sight_margin_m

    Derivation margin between available and required sight distance

    Hidden at easy and medium and hard difficulty.

    12 – 55 m

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_road_edge_levelstale_access_profile_revisionculvert_chainage_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedaccess_freeboard_deficient

Scored outputs

24 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Access profile and hydraulic basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Access, culvert, and freeboard adequacy status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Secondary road-safety status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Driveway grade

driveway_grade_percent

Computed driveway grade

Scores if within ±2% of the reference value.

Driveway grade margin

driveway_grade_margin_percent

Allowable grade minus computed grade

Scores if within ±2% of the reference value.

Culvert capacity

culvert_capacity_m3_s

Full-pipe culvert capacity

Scores if within ±2% of the reference value.

Culvert capacity margin

culvert_capacity_margin_m3_s

Culvert capacity minus design flow

Scores if within ±2% of the reference value.

Headwater level

headwater_level_m

Synthetic headwater level

Scores if within ±2% of the reference value.

Freeboard

freeboard_m

Road edge level minus headwater level

Scores if within ±2% of the reference value.

Freeboard margin

freeboard_margin_m

Freeboard minus minimum freeboard

Scores if within ±2% of the reference value.

Roadway spread

roadway_spread_m

Triangular-gutter spread across the access path

Scores if within ±2% of the reference value.

Spread margin

spread_margin_m

Allowable spread minus computed spread

Scores if within ±2% of the reference value.

Sight distance required

sight_distance_required_m

Grade-adjusted sight distance required

Scores if within ±2% of the reference value.

Sight distance margin

sight_distance_margin_m

Available sight distance minus required sight distance

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing road edge, or genuine freeboard deficiency

Hidden inputs

  • Packet variantpacket_variant
  • Freeboard margin targetfreeboard_margin_target_m
  • Freeboard deficitfreeboard_deficit_m
  • Spread margin targetspread_margin_target_m
  • Available sight marginavailable_sight_margin_m

Packet variant restricted to: clean, missing_road_edge_level, access_freeboard_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Freeboard margin targetfreeboard_margin_target_m
  • Freeboard deficitfreeboard_deficit_m
  • Spread margin targetspread_margin_target_m
  • Available sight marginavailable_sight_margin_m

hard

Some inputs hidden

Subtle documentary defects: stale revisions, chainage mismatch, copied cases, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Freeboard margin targetfreeboard_margin_target_m
  • Freeboard deficitfreeboard_deficit_m
  • Spread margin targetspread_margin_target_m
  • Available sight marginavailable_sight_margin_m

Packet variant restricted to: stale_access_profile_revision, culvert_chainage_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a driveway access safety issue package covering one access profile, road edge basis, culvert drainage schedule, tailwater table, roadway spread note, sight-distance note, owner access criterion, and criteria/comments memo.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, access profile, culvert drainage schedule, surface/tailwater table, roadway spread note, sight-distance note, owner access criterion, and criteria/comments memo. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the driveway, road edge, or culvert. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: driveway access, road edge basis, culvert, tailwater basis, sight-distance basis, owner access criterion, and criteria memo.113. Check for source conflicts, stale revisions, copied scenarios, missing evidence, culvert chainage mismatches, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required access profile, road edge, culvert, tailwater, spread, sight-distance, owner-criterion, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the driveway access, road edge, culvert, tailwater, spread note, sight-distance note, and owner case stay consistent? |26| RLR-03 | Access and hydraulic basis: are driveway grade, culvert capacity, headwater basis, and access profile revision traceable, current, and recomputable? |27| RLR-04 | Access usability adequacy: do the driveway grade, culvert capacity, and road-edge freeboard clear the source criteria for the same access case? |28| RLR-05 | Scenario consequence: is the same access storm and vehicle case used across access profile, road edge, culvert, tailwater, spread, sight-distance, and owner criterion? |29| RLR-06 | Secondary road-safety checks: are roadway spread and sight distance source-backed and internally consistent with the same access case? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.40- Do not rename computed_evidence keys.41 42## Output43 44Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:45 46```json47{48 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],49 "identity_ledger": {50 "driveway_access": "...",51 "road_edge_basis": "...",52 "culvert": "...",53 "tailwater_basis": "...",54 "sight_distance_basis": "...",55 "owner_access_criterion": "...",56 "criteria_memo": "..."57 },58 "review_matrix": {59 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},60 "RLR-02": {"status": "...", "evidence": "..."},61 "RLR-03": {"status": "...", "evidence": "..."},62 "RLR-04": {"status": "...", "evidence": "..."},63 "RLR-05": {"status": "...", "evidence": "..."},64 "RLR-06": {"status": "...", "evidence": "..."},65 "RLR-07": {"status": "...", "evidence": "..."},66 "RLR-08": {"status": "...", "evidence": "..."},67 "RLR-09": {"status": "...", "evidence": "..."}68 },69 "computed_evidence": {70 "driveway_grade_percent": 0.0,71 "driveway_grade_margin_percent": 0.0,72 "culvert_capacity_m3_s": 0.0,73 "culvert_capacity_margin_m3_s": 0.0,74 "headwater_level_m": 0.0,75 "freeboard_m": 0.0,76 "freeboard_margin_m": 0.0,77 "roadway_spread_m": 0.0,78 "spread_margin_m": 0.0,79 "sight_distance_required_m": 0.0,80 "sight_distance_margin_m": 0.081 },82 "findings": [83 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}84 ],85 "information_requests": [86 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}87 ],88 "action_register": [89 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}90 ],91 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",92 "claim_boundary_statement": "..."93}94```95 96Rules for the structured block:97 98- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.99- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.100- Every `insufficient_data` needs an information request naming the exact missing field and its source document.101- Every `not_applicable` needs a scope reason in its matrix `evidence`.102- Carried actions must appear in `action_register` with an owner.103- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.104- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.105

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban service access

urban_service_access

Urban arterial service driveway with tighter spread and sight-distance controls

urban-service-driveway-accessarterial-maintenance-access
Parameter ranges
driveway_length_m
14 – 24
driveway_grade_pct_source
3 – 6.5
gutter_flow_m3_s
0.08 – 0.14
access_speed_kmh
35 – 50

Suburban maintenance access

suburban_maintenance_access

Suburban maintenance access with moderate culvert and access constraints

suburban-maintenance-drivewayperi-urban-culvert-access
Parameter ranges
driveway_length_m
18 – 32
driveway_grade_pct_source
2 – 5
gutter_flow_m3_s
0.04 – 0.1
access_speed_kmh
25 – 40

Example task

urban-service-driveway-access-urban-service-access-preview — hard difficulty, some inputs hidden.

Urban arterial service driveway with tighter spread and sight-distance controls. urban-service-driveway-access. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

driveway_low_level_m
41.8 to 43.2 m
driveway_grade_pct_source
3 to 6.5 %
driveway_length_m
14 to 24 m
allowable_driveway_grade_pct
7 to 9 %
culvert_diameter_m
0.75 to 1.2 m
culvert_mannings_n
0.012 to 0.016
culvert_slope_pct
0.8 to 2 %
design_flow_m3_s
0.25 to 0.7 m3/s
tailwater_level_m
41.2 to 42.4 m
headwater_base_depth_m
0.32 to 0.58 m
headwater_loss_factor_m
0.12 to 0.32 m
minimum_freeboard_m
0.1 to 0.2 m
gutter_flow_m3_s
0.08 to 0.14 m3/s
cross_slope_pct
2 to 4 %
longitudinal_slope_pct
0.6 to 2 %
gutter_mannings_n
0.014 to 0.018
access_speed_kmh
35 to 50 km/h
sight_reaction_time_s
2 to 2.7 s
braking_friction_coefficient
0.32 to 0.42
access_grade_pct
1 to 6 %

Executable tool: driveway-access-safety-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Packet variant

    packet_variant

  • Freeboard margin target

    freeboard_margin_target_m

  • Freeboard deficit

    freeboard_deficit_m

  • Spread margin target

    spread_margin_target_m

  • Available sight margin

    available_sight_margin_m

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "driveway_grade_percent": <number>,
  "driveway_grade_margin_percent": <number>,
  "culvert_capacity_m3_s": <number>,
  "culvert_capacity_margin_m3_s": <number>,
  "headwater_level_m": <number>,
  "freeboard_m": <number>,
  "freeboard_margin_m": <number>,
  "roadway_spread_m": <number>,
  "spread_margin_m": <number>,
  "sight_distance_required_m": <number>,
  "sight_distance_margin_m": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • driveway_grade_percent · scored within ±2%
  • driveway_grade_margin_percent · scored within ±2%
  • culvert_capacity_m3_s · scored within ±2%
  • culvert_capacity_margin_m3_s · scored within ±2%
  • headwater_level_m · scored within ±2%
  • freeboard_m · scored within ±2%
  • freeboard_margin_m · scored within ±2%
  • roadway_spread_m · scored within ±2%
  • spread_margin_m · scored within ±2%
  • sight_distance_required_m · scored within ±2%
  • sight_distance_margin_m · scored within ±2%