LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
civilno-tool

Intersection Signal Safety Issue Review Package

Review-native long-horizon task over a generated SSC-01 intersection signal safety issue packet. The agent receives a multi-file source packet in /workspace/sources/ (document register, intersection layout, approach profile, signal timing sheet, pedestrian crossing sheet, sight-distance note, controller handoff note, criteria and comments), inventories it, preserves object identity, recomputes the package's own stopping-distance and signal-clearance evidence, assigns one status per review item, raises findings/information requests/actions, and issues a readiness decision. Packet variants inject missing pedestrian clearance evidence, stale timing revisions, datum mismatch, scenario copy-forward, open comments, carried comments, or a genuine pedestrian-clearance failure. Scored by a stage-gated custom verifier, not per-key answer matching.

no-tool: The model must reason numerically unaided.

Standards

AASHTO Green BookFHWA MUTCD

Tags

civilintersectionissue-readinessreviewroadsight-distancesignal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

16 inputs

Always given

Included directly in every task prompt.

12
Show 12 inputs

  • Approach speed kmh

    approach_speed_kmh

    APP-NB-02 approach design speed

    50 – 75 km/h

  • Approach grade pct

    approach_grade_pct

    APP-NB-02 signed approach grade; downhill is negative

    -3.5 – 2 %

  • Reaction time

    reaction_time_s

    CRIT-SSC01-002 stopping sight-distance reaction time

    2.3 – 2.7 s

  • Braking friction coefficient

    braking_friction_coefficient

    CRIT-SSC01-002 braking friction coefficient

    0.34 – 0.4

  • Yellow reaction time

    yellow_reaction_time_s

    SIG-SSC01-002 yellow interval perception-reaction term

    0.9 – 1.2 s

  • Yellow deceleration

    yellow_deceleration_m_s2

    SIG-SSC01-002 comfortable deceleration rate for yellow timing

    2.9 – 3.4 m/s2

  • Intersection width

    intersection_width_m

    INT-SSC01-002 conflict area width for SG-02 all-red clearance

    24 – 36 m

  • Design vehicle length

    design_vehicle_length_m

    CRIT-SSC01-002 design vehicle length for all-red clearance

    6.012.519.0

  • All red speed kmh

    all_red_speed_kmh

    SIG-SSC01-002 all-red clearance speed

    40 – 55 km/h

  • Pedestrian startup time

    pedestrian_startup_time_s

    PED-X-02 pedestrian startup allowance

    3 – 4 s

  • Crossing width

    crossing_width_m

    PED-X-02 crossing width

    12 – 22 m

  • Pedestrian walk speed

    pedestrian_walk_speed_m_s

    PED-X-02 pedestrian walk speed

    1 – 1.3 m/s

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

4

  • Sight distance margin

    sight_distance_margin_m

    Derivation margin between computed stopping distance and available sight distance

    Hidden at easy and medium and hard difficulty.

    12 – 45 m

  • Pedestrian clearance margin

    pedestrian_clearance_margin_s

    Derivation margin between required and available pedestrian clearance

    Hidden at easy and medium and hard difficulty.

    1 – 4 s

  • Pedestrian clearance deficit

    pedestrian_clearance_deficit_s

    Derivation deficit for the pedestrian-clearance-deficient variant

    Hidden at easy and medium and hard difficulty.

    0.4 – 2 s

  • Packet variant

    packet_variant

    Hidden packet-defect variant controlling gold review statuses

    Hidden at easy and medium and hard difficulty.

    cleanmissing_pedestrian_clearancestale_timing_revisionapproach_datum_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedpedestrian_clearance_deficient

Scored outputs

20 outputs

Rlr 01 status

rlr_01_status

Packet completeness status code (0 pass, 1 fail, 2 not applicable, 3 insufficient data)

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Object identity status code

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Traffic safety basis status code

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Approach user safety status code

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Timing scenario consequence status code

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Signal handoff resilience status code

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Comment and action closure status code

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Readiness decision consistency status code

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Claim boundary status code

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Gold readiness decision (0 ready, 1 ready with carried actions, 2 not ready)

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Findings the review must raise

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Information requests the review must raise

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Carried actions the review must record

Scores if within ±0% of the reference value.

Stopping distance

stopping_distance_m

Grade-adjusted stopping distance evidence

Scores if within ±2% of the reference value.

Sight distance margin

sight_distance_margin_m

Available sight distance minus stopping distance evidence

Scores if within ±2% of the reference value.

Yellow interval s

yellow_interval_s

Grade-adjusted yellow interval evidence

Scores if within ±2% of the reference value.

All red interval s

all_red_interval_s

All-red clearance interval evidence

Scores if within ±2% of the reference value.

Ped clearance required s

ped_clearance_required_s

Required pedestrian clearance evidence

Scores if within ±2% of the reference value.

Ped clearance margin s

ped_clearance_margin_s

Available pedestrian clearance minus required clearance evidence

Scores if within ±2% of the reference value.

Grade adjusted braking distance

grade_adjusted_braking_distance_m

Braking distance component of stopping distance evidence

Scores if within ±2% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing evidence, or a genuine pedestrian-clearance failure

Hidden inputs

  • Packet variantpacket_variant
  • Sight distance marginsight_distance_margin_m
  • Pedestrian clearance margin spedestrian_clearance_margin_s
  • Pedestrian clearance deficit spedestrian_clearance_deficit_s

Packet variant restricted to: clean, missing_pedestrian_clearance, pedestrian_clearance_deficient

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

  • Packet variantpacket_variant
  • Sight distance marginsight_distance_margin_m
  • Pedestrian clearance margin spedestrian_clearance_margin_s
  • Pedestrian clearance deficit spedestrian_clearance_deficit_s

hard

Some inputs hidden

Subtle documentary defects: stale revisions, datum drift, scenario copy-forward, carried comments

Hidden inputs

  • Packet variantpacket_variant
  • Sight distance marginsight_distance_margin_m
  • Pedestrian clearance margin spedestrian_clearance_margin_s
  • Pedestrian clearance deficit spedestrian_clearance_deficit_s

Packet variant restricted to: stale_timing_revision, approach_datum_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for an intersection signal safety issue package covering one road approach, one signal group, one pedestrian crossing, and one controller timing handoff.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, intersection layout, approach profile, signal timing sheet, pedestrian crossing sheet, sight-distance note, controller handoff note, and a criteria memo with review comments. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the intersection. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: intersection, approach, approach chainage, datum, signal group, pedestrian crossing, controller, timing case, and sight-distance case.113. Check for source conflicts, stale revisions, contradictory datums, copied scenarios, missing evidence, and open critical comments before accepting any package claim.124. Recompute the package's own calculations only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required source documents present with IDs and revisions? |25| RLR-02 | Object identity: do intersection, approach, chainage, datum, signal group, crossing, controller, and case stay consistent across documents? |26| RLR-03 | Traffic safety basis: are stopping distance, yellow interval, all-red interval, and sight-distance calculations traceable to the current basis and recomputable? |27| RLR-04 | Approach user safety: do available sight distance and pedestrian clearance both clear the source-owned criteria? |28| RLR-05 | Timing scenario consequence: does the timing case use this approach's design speed and crossing case rather than a copied scenario? |29| RLR-06 | Signal handoff resilience: are controller timing, clearance intervals, and exported signal group values source-backed and internally consistent? |30| RLR-07 | Comment and action closure: is every review comment closed, or carried with an owner and agreed action, or blocked by named missing data? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, or acceptance claims? |33 34## Output35 36Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:37 38```json39{40 "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],41 "identity_ledger": {42 "intersection": "...",43 "approach": "...",44 "approach_chainage": "...",45 "datum": "...",46 "signal_group": "...",47 "pedestrian_crossing": "...",48 "controller": "...",49 "timing_case": "...",50 "sight_distance_case": "..."51 },52 "review_matrix": {53 "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},54 "RLR-02": {"status": "...", "evidence": "..."},55 "RLR-03": {"status": "...", "evidence": "..."},56 "RLR-04": {"status": "...", "evidence": "..."},57 "RLR-05": {"status": "...", "evidence": "..."},58 "RLR-06": {"status": "...", "evidence": "..."},59 "RLR-07": {"status": "...", "evidence": "..."},60 "RLR-08": {"status": "...", "evidence": "..."},61 "RLR-09": {"status": "...", "evidence": "..."}62 },63 "computed_evidence": {64 "stopping_distance_m": 0.0,65 "sight_distance_margin_m": 0.0,66 "yellow_interval_s": 0.0,67 "all_red_interval_s": 0.0,68 "ped_clearance_required_s": 0.0,69 "ped_clearance_margin_s": 0.0,70 "grade_adjusted_braking_distance_m": 0.071 },72 "findings": [73 {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}74 ],75 "information_requests": [76 {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}77 ],78 "action_register": [79 {"action": "...", "owner": "...", "linked_item": "RLR-0X"}80 ],81 "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",82 "claim_boundary_statement": "..."83}84```85 86Rules for the structured block:87 88- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead.89- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.90- Every `insufficient_data` needs an information request naming the exact missing field and its source document.91- Every `not_applicable` needs a scope reason in its matrix `evidence`.92- Carried actions must appear in `action_register` with an owner.93- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.94- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.95

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Urban arterial

urban_arterial

Urban arterial signalized intersection with wide crossing and stronger approach speed

urban-arterial-signalcbd-fringe-intersection
Parameter ranges
approach_speed_kmh
55 – 75
intersection_width_m
28 – 36
crossing_width_m
16 – 22

Suburban collector

suburban_collector

Suburban collector signalized intersection with shorter crossing and moderate approach speed

suburban-collector-signalschool-access-intersection
Parameter ranges
approach_speed_kmh
50 – 65
intersection_width_m
24 – 31
crossing_width_m
12 – 18

Example task

urban-arterial-signal-urban-arterial-preview — hard difficulty, some inputs hidden.

Urban arterial signalized intersection with wide crossing and stronger approach speed. urban-arterial-signal. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

approach_speed_kmh
55 to 75 km/h
approach_grade_pct
-3.5 to 2 %
reaction_time_s
2.3 to 2.7 s
braking_friction_coefficient
0.34 to 0.4
yellow_reaction_time_s
0.9 to 1.2 s
yellow_deceleration_m_s2
2.9 to 3.4 m/s2
intersection_width_m
28 to 36 m
design_vehicle_length_m
6.0 m
all_red_speed_kmh
40 to 55 km/h
pedestrian_startup_time_s
3 to 4 s
crossing_width_m
16 to 22 m
pedestrian_walk_speed_m_s
1 to 1.3 m/s

Executable tool: intersection-signal-safety-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

  • Packet variant

    packet_variant

  • Pedestrian clearance deficit s

    pedestrian_clearance_deficit_s

  • Pedestrian clearance margin s

    pedestrian_clearance_margin_s

  • Sight distance margin

    sight_distance_margin_m

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "stopping_distance_m": <number>,
  "sight_distance_margin_m": <number>,
  "yellow_interval_s": <number>,
  "all_red_interval_s": <number>,
  "ped_clearance_required_s": <number>,
  "ped_clearance_margin_s": <number>,
  "grade_adjusted_braking_distance_m": <number>
}
  • rlr_01_status · scored within ±0%
  • rlr_02_status · scored within ±0%
  • rlr_03_status · scored within ±0%
  • rlr_04_status · scored within ±0%
  • rlr_05_status · scored within ±0%
  • rlr_06_status · scored within ±0%
  • rlr_07_status · scored within ±0%
  • rlr_08_status · scored within ±0%
  • rlr_09_status · scored within ±0%
  • readiness_code · scored within ±0%
  • required_findings_count · scored within ±0%
  • required_information_requests_count · scored within ±0%
  • required_carried_actions_count · scored within ±0%
  • stopping_distance_m · scored within ±2%
  • sight_distance_margin_m · scored within ±2%
  • yellow_interval_s · scored within ±2%
  • all_red_interval_s · scored within ±2%
  • ped_clearance_required_s · scored within ±2%
  • ped_clearance_margin_s · scored within ±2%
  • grade_adjusted_braking_distance_m · scored within ±2%