LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
structuralwith-tool

Structural Review Authority Overlay Package

Calculates source-bound structural review metrics from a deterministic SSC-14 task-owned source pack. The template combines ULS and SLS load combinations, capacity and deflection margins, material carbon-equivalent evidence, evidence completeness, review comment closeout, authority overlay count, and overall pass scoring.

with-tool: The model is given an executable Python calculator script.

Standards

authority overlaycomment closeoutload-combination reviewmaterial certificate check

Tags

authoritycommentsdeterministicload-combinationsmaterial-certificatereviewstructural

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

26 inputs

Always given

Included directly in every task prompt.

26
Show 26 inputs

  • Dead load

    dead_load_kn

    LOAD-SSC14-008 dead load

    118 kN

  • Live load

    live_load_kn

    LOAD-SSC14-008 live load

    36 kN

  • Wind load

    wind_load_kn

    LOAD-SSC14-008 wind load

    58 kN

  • Uls dead factor

    uls_dead_factor

    LOAD-SSC14-008 ULS dead load factor

    1.2

  • Uls live factor

    uls_live_factor

    LOAD-SSC14-008 ULS live load factor

    1.5

  • Uls wind factor

    uls_wind_factor

    LOAD-SSC14-008 ULS wind load factor

    1

  • Sls wind factor

    sls_wind_factor

    LOAD-SSC14-008 SLS wind load factor

    0.7

  • Member capacity

    member_capacity_kn

    INDEX-SSC14-008 member capacity from calculation appendix

    260 kN

  • Sls deflection limit

    sls_deflection_limit_mm

    INDEX-SSC14-008 SLS deflection limit

    18 mm

  • Calculated sls deflection

    calculated_sls_deflection_mm

    INDEX-SSC14-008 calculated SLS deflection

    14.2 mm

  • Carbon percent

    carbon_percent

    MAT-SSC14-008 material certificate carbon percentage

    0.18 %

  • Manganese percent

    manganese_percent

    MAT-SSC14-008 material certificate manganese percentage

    1.3 %

  • Chromium percent

    chromium_percent

    MAT-SSC14-008 material certificate chromium percentage

    0.35 %

  • Molybdenum percent

    molybdenum_percent

    MAT-SSC14-008 material certificate molybdenum percentage

    0.05 %

  • Vanadium percent

    vanadium_percent

    MAT-SSC14-008 material certificate vanadium percentage

    0.02 %

  • Nickel percent

    nickel_percent

    MAT-SSC14-008 material certificate nickel percentage

    0.2 %

  • Copper percent

    copper_percent

    MAT-SSC14-008 material certificate copper percentage

    0.25 %

  • Carbon equivalent limit

    carbon_equivalent_limit

    MAT-SSC14-008 carbon equivalent limit for review acceptance

    0.55

  • Required evidence items

    required_evidence_items

    INDEX-SSC14-008 required evidence item count

    9 count

  • Present evidence items

    present_evidence_items

    INDEX-SSC14-008 present evidence item count

    8 count

  • Review comment

    review_comment_count

    COMMENTS-SSC14-008 total review comments

    12 count

  • Closed review comment

    closed_review_comment_count

    COMMENTS-SSC14-008 closed review comments

    11 count

  • Unresolved critical comments

    unresolved_critical_comments

    COMMENTS-SSC14-008 unresolved critical comments

    0 count

  • Authority override

    authority_override_count

    COMMENTS-SSC14-008 authority override count recorded in the memo

    1 count

  • Minimum evidence percent

    minimum_evidence_percent

    MEMO-SSC14-008 minimum evidence completeness threshold

    85 %

  • Minimum comment closeout percent

    minimum_comment_closeout_percent

    MEMO-SSC14-008 minimum review comment closeout threshold

    90 %

Scored outputs

11 outputs

Governing uls load kn

governing_uls_load_kn

ULS load combination

Scores if within ±3% of the reference value.

Governing sls load kn

governing_sls_load_kn

SLS load combination

Scores if within ±3% of the reference value.

Uls capacity margin kn

uls_capacity_margin_kn

Member capacity minus ULS demand

Scores if within ±3% of the reference value.

Sls deflection margin

sls_deflection_margin_mm

SLS deflection limit minus calculated deflection

Scores if within ±3% of the reference value.

Material carbon equivalent

material_carbon_equivalent

IIW carbon equivalent from material certificate

Scores if within ±3% of the reference value.

Carbon equivalent margin

carbon_equivalent_margin

Carbon equivalent limit minus calculated value

Scores if within ±3% of the reference value.

Evidence complete

evidence_complete_percent

Present evidence items divided by required evidence items

Scores if within ±3% of the reference value.

Comment closeout

comment_closeout_percent

Closed review comments divided by total review comments

Scores if within ±3% of the reference value.

Unresolved critical comments

unresolved_critical_comments

Unresolved critical review comments

Scores if within ±3% of the reference value.

Authority override count

authority_override_count

Authority override count named in the acceptance response

Scores if within ±3% of the reference value.

Overall pass score

overall_pass_score

1.0 when load, material, evidence, and comment checks pass

Scores if within ±3% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-14 structural review package
medium:
All source-pack values given for the SSC-14 structural review package
hard:
All source-pack values given for the SSC-14 structural review package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • structural-review-authority-overlay-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a structural engineer checking a task-owned synthetic SSC-14 structural review packet and authority overlay.2 3Use only the task-owned synthetic source pack values below for numeric grading. Structural source indices, load schedules, material certificates, review comments, and authority criteria shape the workflow context only; they are not extra data sources for this instance.4 5## Scene6 7- Product: `SSC-14-LH-08`8- Source index and calculation appendix: `INDEX-SSC14-008`9- Load schedule: `LOAD-SSC14-008`10- Material certificate: `MAT-SSC14-008`11- Review comment register: `COMMENTS-SSC14-008`12- Acceptance/gap response memo: `MEMO-SSC14-008`13 14## Source Values15 16- Dead, live, and wind loads: {{ dead_load_kn }} kN, {{ live_load_kn }} kN, {{ wind_load_kn }} kN17- ULS load factors D, L, W and SLS wind factor: {{ uls_dead_factor }}, {{ uls_live_factor }}, {{ uls_wind_factor }}, {{ sls_wind_factor }}18- Member capacity, SLS deflection limit, and calculated SLS deflection: {{ member_capacity_kn }} kN, {{ sls_deflection_limit_mm }} mm, {{ calculated_sls_deflection_mm }} mm19- Material certificate chemistry: C {{ carbon_percent }} %, Mn {{ manganese_percent }} %, Cr {{ chromium_percent }} %, Mo {{ molybdenum_percent }} %, V {{ vanadium_percent }} %, Ni {{ nickel_percent }} %, Cu {{ copper_percent }} %20- Carbon equivalent limit: {{ carbon_equivalent_limit }}21- Evidence items present/required: {{ present_evidence_items }} / {{ required_evidence_items }}22- Review comments closed/total and unresolved critical comments: {{ closed_review_comment_count }} / {{ review_comment_count }}, {{ unresolved_critical_comments }}23- Authority override count and thresholds: {{ authority_override_count }}, {{ minimum_evidence_percent }} %, {{ minimum_comment_closeout_percent }} %24 25## Required Calculations26 27- ULS load is `D_factor x D + L_factor x L + W_factor x W`.28- SLS load is `D + L + SLS_wind_factor x W`.29- ULS capacity margin is member capacity minus ULS load.30- SLS deflection margin is limit minus calculated deflection.31- Material carbon equivalent is `C + Mn / 6 + (Cr + Mo + V) / 5 + (Ni + Cu) / 15`.32- Evidence completeness is present evidence divided by required evidence.33- Comment closeout is closed comments divided by total comments.34- Overall pass score is `1.0` only when load margins, material margin, evidence completeness, comment closeout, and unresolved critical comment checks pass.35 36Write a compact acceptance/gap response memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the source IDs above and state whether the baseline source pack passes the current synthetic checks.37 38Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.39 40Include a fenced JSON block with exactly these numeric keys:41 42```json43{44 "governing_uls_load_kn": <numeric_value>,45 "governing_sls_load_kn": <numeric_value>,46 "uls_capacity_margin_kn": <numeric_value>,47 "sls_deflection_margin_mm": <numeric_value>,48 "material_carbon_equivalent": <numeric_value>,49 "carbon_equivalent_margin": <numeric_value>,50 "evidence_complete_percent": <numeric_value>,51 "comment_closeout_percent": <numeric_value>,52 "unresolved_critical_comments": <numeric_value>,53 "authority_override_count": <numeric_value>,54 "overall_pass_score": <numeric_value>55}56```57

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc14 structural review case

ssc14_structural_review_case

Structural source index, load schedule, material certificate, and review comment source pack

structural-reviewauthority-overlaycomment-closeout
Parameter ranges
dead_load_kn
118
live_load_kn
36
wind_load_kn
58
uls_dead_factor
1.2
uls_live_factor
1.5
uls_wind_factor
1
sls_wind_factor
0.7
member_capacity_kn
260
sls_deflection_limit_mm
18
calculated_sls_deflection_mm
14.2
carbon_percent
0.18
manganese_percent
1.3
chromium_percent
0.35
molybdenum_percent
0.05
vanadium_percent
0.02
nickel_percent
0.2
copper_percent
0.25
carbon_equivalent_limit
0.55
required_evidence_items
9
present_evidence_items
8
review_comment_count
12
closed_review_comment_count
11
unresolved_critical_comments
0
authority_override_count
1
minimum_evidence_percent
85
minimum_comment_closeout_percent
90

Example task

structural-review-ssc14-structural-review-case-preview — hard difficulty, all inputs given.

Structural source index, load schedule, material certificate, and review comment source pack. structural-review. Required outputs: governing_uls_load_kn, governing_sls_load_kn, uls_capacity_margin_kn, sls_deflection_margin_mm, material_carbon_equivalent, carbon_equivalent_margin

The model sees

Scenario context and visible inputs.

dead_load_kn
118 to 118 kN
live_load_kn
36 to 36 kN
wind_load_kn
58 to 58 kN
uls_dead_factor
1.2 to 1.2
uls_live_factor
1.5 to 1.5
uls_wind_factor
1 to 1
sls_wind_factor
0.7 to 0.7
member_capacity_kn
260 to 260 kN
sls_deflection_limit_mm
18 to 18 mm
calculated_sls_deflection_mm
14.2 to 14.2 mm
carbon_percent
0.18 to 0.18 %
manganese_percent
1.3 to 1.3 %
chromium_percent
0.35 to 0.35 %
molybdenum_percent
0.05 to 0.05 %
vanadium_percent
0.02 to 0.02 %
nickel_percent
0.2 to 0.2 %
copper_percent
0.25 to 0.25 %
carbon_equivalent_limit
0.55 to 0.55
required_evidence_items
9 to 9 count
present_evidence_items
8 to 8 count
review_comment_count
12 to 12 count
closed_review_comment_count
11 to 11 count
unresolved_critical_comments
0 to 0 count
authority_override_count
1 to 1 count
minimum_evidence_percent
85 to 85 %
minimum_comment_closeout_percent
90 to 90 %

Executable tool: structural-review-authority-overlay-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "governing_uls_load_kn": <number>,
  "governing_sls_load_kn": <number>,
  "uls_capacity_margin_kn": <number>,
  "sls_deflection_margin_mm": <number>,
  "material_carbon_equivalent": <number>,
  "carbon_equivalent_margin": <number>,
  "evidence_complete_percent": <number>,
  "comment_closeout_percent": <number>,
  "unresolved_critical_comments": <number>,
  "authority_override_count": <number>,
  "overall_pass_score": <number>
}
  • governing_uls_load_kn · scored within ±3%
  • governing_sls_load_kn · scored within ±3%
  • uls_capacity_margin_kn · scored within ±3%
  • sls_deflection_margin_mm · scored within ±3%
  • material_carbon_equivalent · scored within ±3%
  • carbon_equivalent_margin · scored within ±3%
  • evidence_complete_percent · scored within ±3%
  • comment_closeout_percent · scored within ±3%
  • unresolved_critical_comments · scored within ±3%
  • authority_override_count · scored within ±3%
  • overall_pass_score · scored within ±3%