LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
structuralwith-tool

Facade Submittal Source Policy Package

Calculates source-bound facade submittal review metrics from a deterministic SSC-09 task-owned source pack. The template combines source traceability, calculator row checks, material schedule matching, utilization row pass checks, review comment closure, boundary-exception resolution, substitution control, response completeness, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

facade submittal workflowsresponse-to-comments workflowsource traceability review

Tags

calculatorcommentsdeterministicfacadematerialssource-policystructuralsubmittal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

15 inputs

Always given

Included directly in every task prompt.

15
Show 15 inputs

  • Source items traced

    source_items_traced

    SOURCE-09-INDEX-08 traced source items

    16

  • Required source items

    required_source_items

    SOURCE-09-INDEX-08 required source items

    18

  • Calculator rows checked

    calculator_rows_checked

    CALC-09-REPORT-08 checked calculator rows

    12

  • Required calculator rows

    required_calculator_rows

    CALC-09-REPORT-08 required calculator rows

    12

  • Matching material items

    matching_material_items

    MAT-09-SCHEDULE-08 matching material items

    9

  • Material schedule items

    material_schedule_items

    MAT-09-SCHEDULE-08 material schedule items

    10

  • Passing utilization rows

    passing_utilization_rows

    CALC-09-REPORT-08 passing utilization rows

    8

  • Utilization rows

    utilization_rows

    CALC-09-REPORT-08 utilization rows

    8

  • Resolved comments

    resolved_comments

    COMMENT-09-REVIEW-08 resolved comments

    5

  • Review comments

    review_comments

    COMMENT-09-REVIEW-08 review comments

    6

  • Approved boundary exceptions

    approved_boundary_exceptions

    SOURCE-09-INDEX-08 approved boundary exceptions

    2

  • Boundary exceptions

    boundary_exceptions

    SOURCE-09-INDEX-08 boundary exceptions

    2

  • Unapproved substitution count

    unapproved_substitution_count

    MAT-09-SCHEDULE-08 unapproved substitution count

    0

  • Response sections

    response_sections

    RESPONSE-09-SUBMITTAL-08 completed response sections

    9

  • Required response sections

    required_response_sections

    RESPONSE-09-SUBMITTAL-08 required response sections

    10

Scored outputs

10 outputs

Source trace score

source_trace_score

Traced source items divided by required source items

Scores if within ±0.3% of the reference value.

Calculator check fraction

calculator_check_fraction

Checked calculator rows divided by required calculator rows

Scores if within ±0.3% of the reference value.

Material match fraction

material_match_fraction

Matching material items divided by material schedule items

Scores if within ±0.3% of the reference value.

Utilization pass fraction

utilization_pass_fraction

Passing utilization rows divided by utilization rows

Scores if within ±0.3% of the reference value.

Comment resolution fraction

comment_resolution_fraction

Resolved comments divided by review comments

Scores if within ±0.3% of the reference value.

Boundary exception resolution fraction

boundary_exception_resolution_fraction

Approved boundary exceptions divided by boundary exceptions

Scores if within ±0.3% of the reference value.

Unapproved substitution count

unapproved_substitution_count

Unapproved substitution count

Scores if within ±0.1% of the reference value.

Response completeness score

response_completeness_score

Completed response sections divided by required response sections

Scores if within ±0.3% of the reference value.

Evidence boundary score

evidence_boundary_score

Average of source, calculator, material, utilization, comment, exception, and response fractions

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when facade submittal source-policy checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-09 facade submittal review package
medium:
All source-pack values given for the SSC-09 facade submittal review package
hard:
All source-pack values given for the SSC-09 facade submittal review package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • facade-submittal-source-policy-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a structural/facade engineer checking a task-owned synthetic SSC-09 facade submittal review and source-policy package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. Facade submittal review, source traceability, calculator checking, material schedule matching, and response-to-comments workflows shape the practice context only; they are not extra data sources for this instance.4 5## Scene6 7- Product family: `SSC-09-LH-08`8- Source index: `SOURCE-09-INDEX-08`9- Redrawn facade elevation: `ELEV-09-REDRAWN-08`10- Calculation report: `CALC-09-REPORT-08`11- Material schedule: `MAT-09-SCHEDULE-08`12- Review comment register: `COMMENT-09-REVIEW-08`13- Submittal response: `RESPONSE-09-SUBMITTAL-08`14 15## Source Values16 17| Item | Value |18|------|-------|19| Source items traced | {{ source_items_traced }} |20| Required source items | {{ required_source_items }} |21| Calculator rows checked | {{ calculator_rows_checked }} |22| Required calculator rows | {{ required_calculator_rows }} |23| Matching material items | {{ matching_material_items }} |24| Material schedule items | {{ material_schedule_items }} |25| Passing utilization rows | {{ passing_utilization_rows }} |26| Utilization rows | {{ utilization_rows }} |27| Resolved comments | {{ resolved_comments }} |28| Review comments | {{ review_comments }} |29| Approved boundary exceptions | {{ approved_boundary_exceptions }} |30| Boundary exceptions | {{ boundary_exceptions }} |31| Unapproved substitution count | {{ unapproved_substitution_count }} |32| Response sections | {{ response_sections }} |33| Required response sections | {{ required_response_sections }} |34 35## Checks36 37- Source trace score equals source items traced divided by required source items.38- Calculator check fraction equals calculator rows checked divided by required calculator rows.39- Material match fraction equals matching material items divided by material schedule items.40- Utilization pass fraction equals passing utilization rows divided by utilization rows.41- Comment resolution fraction equals resolved comments divided by review comments.42- Boundary exception resolution fraction equals approved boundary exceptions divided by boundary exceptions.43- Response completeness score equals response sections divided by required response sections.44- Evidence boundary score is the average of source trace, calculator, material, utilization, comment, boundary exception, and response completeness fractions.45- Overall pass score is `1.0` only when calculator checks, utilization rows, and boundary exceptions are fully complete, with zero unapproved substitutions; otherwise it is `0.0`.46 47## Output Format48 49Write a compact facade submittal review memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the object IDs above and state whether the baseline source pack passes the current docs-only checks.50 51Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.52 53Include a fenced JSON block with exactly these numeric keys:54 55```json56{57 "source_trace_score": <numeric_value>,58 "calculator_check_fraction": <numeric_value>,59 "material_match_fraction": <numeric_value>,60 "utilization_pass_fraction": <numeric_value>,61 "comment_resolution_fraction": <numeric_value>,62 "boundary_exception_resolution_fraction": <numeric_value>,63 "unapproved_substitution_count": <numeric_value>,64 "response_completeness_score": <numeric_value>,65 "evidence_boundary_score": <numeric_value>,66 "overall_pass_score": <numeric_value>67}68```69

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc09 facade submittal policy case

ssc09_facade_submittal_policy_case

Facade submittal review and source-policy case

Parameter ranges
source_items_traced
16
required_source_items
18
calculator_rows_checked
12
required_calculator_rows
12
matching_material_items
9
material_schedule_items
10
passing_utilization_rows
8
utilization_rows
8
resolved_comments
5
review_comments
6
approved_boundary_exceptions
2
boundary_exceptions
2
unapproved_substitution_count
0
response_sections
9
required_response_sections
10

Example task

ssc09-facade-submittal-policy-case-preview — hard difficulty, all inputs given.

Facade submittal review and source-policy case. Required outputs: source_trace_score, calculator_check_fraction, material_match_fraction, utilization_pass_fraction, comment_resolution_fraction, boundary_exception_resolution_fraction

The model sees

Scenario context and visible inputs.

source_items_traced
16 to 16
required_source_items
18 to 18
calculator_rows_checked
12 to 12
required_calculator_rows
12 to 12
matching_material_items
9 to 9
material_schedule_items
10 to 10
passing_utilization_rows
8 to 8
utilization_rows
8 to 8
resolved_comments
5 to 5
review_comments
6 to 6
approved_boundary_exceptions
2 to 2
boundary_exceptions
2 to 2
unapproved_substitution_count
0 to 0
response_sections
9 to 9
required_response_sections
10 to 10

Executable tool: facade-submittal-source-policy-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "source_trace_score": <number>,
  "calculator_check_fraction": <number>,
  "material_match_fraction": <number>,
  "utilization_pass_fraction": <number>,
  "comment_resolution_fraction": <number>,
  "boundary_exception_resolution_fraction": <number>,
  "unapproved_substitution_count": <number>,
  "response_completeness_score": <number>,
  "evidence_boundary_score": <number>,
  "overall_pass_score": <number>
}
  • source_trace_score · scored within ±0.3%
  • calculator_check_fraction · scored within ±0.3%
  • material_match_fraction · scored within ±0.3%
  • utilization_pass_fraction · scored within ±0.3%
  • comment_resolution_fraction · scored within ±0.3%
  • boundary_exception_resolution_fraction · scored within ±0.3%
  • unapproved_substitution_count · scored within ±0.1%
  • response_completeness_score · scored within ±0.3%
  • evidence_boundary_score · scored within ±0.3%
  • overall_pass_score · scored within ±0.1%