LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
mechanicalwith-tool

Product Submittal Compliance Package

Calculates source-bound product-submittal review metrics from a deterministic SSC-15 task-owned source pack. The template combines one pump submittal register, datasheet, certificate, calculation excerpt, review-comment log, deviation register, and disposition memo for one compact generated task instance.

with-tool: The model is given an executable Python calculator script.

Standards

Autodesk Construction Cloud SubmittalsBBA certificatesFM ApprovalsICC-ES evaluation reportsProcore SubmittalsUL Product iQ

Tags

certificatedatasheetdeterministicmechanicalproduct-compliancereview-commentssubmittal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

23 inputs

Always given

Included directly in every task prompt.

23
Show 23 inputs

  • Required flow

    required_flow_l_s

    CALC-15-DUTY-001 required design flow consumed by the product submittal review

    42 L/s

  • Submitted flow

    submitted_flow_l_s

    DAT-15-PUMP-001 submitted pump datasheet flow at the selected duty point

    45 L/s

  • Required head

    required_head_m

    CALC-15-DUTY-001 required design head consumed by the product submittal review

    38 m

  • Submitted head

    submitted_head_m

    DAT-15-PUMP-001 submitted pump datasheet head at the selected duty point

    40 m

  • Bep flow

    bep_flow_l_s

    DAT-15-PUMP-001 best efficiency point flow used to check the POR band

    40 L/s

  • Por low percent

    por_low_percent

    SPEC-15-15100 preferred operating range lower bound as percent of BEP flow

    70 %

  • Por high percent

    por_high_percent

    SPEC-15-15100 preferred operating range upper bound as percent of BEP flow

    120 %

  • Motor nameplate

    motor_nameplate_kw

    DAT-15-PUMP-001 submitted motor nameplate power

    30 kW

  • Motor service factor

    motor_service_factor

    DAT-15-PUMP-001 submitted motor service factor

    1.15

  • Required motor

    required_motor_kw

    CALC-15-DUTY-001 required motor input power for the design duty

    23 kW

  • Max system pressure

    max_system_pressure_kpa

    CALC-15-DUTY-001 maximum system pressure applied to the submitted product certificate

    860 kPa

  • Certificate pressure rating

    certificate_pressure_rating_kpa

    CERT-15-PRESS-001 pressure rating declared by the submitted certificate

    1000 kPa

  • Required evidence items

    required_evidence_items

    REG-15-SUB-001 required evidence item count for this submittal package

    8 count

  • Submitted evidence items

    submitted_evidence_items

    REG-15-SUB-001 submitted evidence item count

    8 count

  • Required certificate items

    required_certificate_items

    CERT-15-PRESS-001 required certificate field count for the pressure and identity check

    4 count

  • Matching certificate items

    matching_certificate_items

    CERT-15-PRESS-001 certificate fields that match the submitted product and use case

    4 count

  • Total review comments

    total_review_comments

    CMT-15-REVIEW-001 total reviewer comments on the submittal

    5 count

  • Closed review comments

    closed_review_comments

    CMT-15-REVIEW-001 reviewer comments closed with evidence

    5 count

  • Open critical comments

    open_critical_comments

    CMT-15-REVIEW-001 open critical comments at disposition

    0 count

  • Review period

    review_period_days

    REG-15-SUB-001 allowed review period

    14 days

  • Elapsed review

    elapsed_review_days

    REG-15-SUB-001 elapsed review period at disposition

    10 days

  • Approved deviation

    approved_deviation_count

    DEV-15-LOG-001 deviations accepted with reviewer evidence

    1 count

  • Unresolved deviation

    unresolved_deviation_count

    DEV-15-LOG-001 deviations still unresolved at disposition

    0 count

Scored outputs

20 outputs

Flow capacity margin

flow_capacity_margin_l_s

Submitted pump flow minus required design flow

Scores if within ±3% of the reference value.

Flow capacity ratio

flow_capacity_ratio

Submitted pump flow divided by required design flow

Scores if within ±3% of the reference value.

Head capacity margin

head_capacity_margin_m

Submitted pump head minus required design head

Scores if within ±3% of the reference value.

Head capacity ratio

head_capacity_ratio

Submitted pump head divided by required design head

Scores if within ±3% of the reference value.

Bep flow

bep_flow_percent

Required flow as a percentage of BEP flow

Scores if within ±3% of the reference value.

Por min flow

por_min_flow_l_s

Preferred operating range lower flow bound

Scores if within ±3% of the reference value.

Por max flow

por_max_flow_l_s

Preferred operating range upper flow bound

Scores if within ±3% of the reference value.

Por low margin

por_low_margin_l_s

Required flow minus POR lower flow bound

Scores if within ±3% of the reference value.

Por high margin

por_high_margin_l_s

POR upper flow bound minus required flow

Scores if within ±3% of the reference value.

Motor available

motor_available_kw

Motor nameplate power multiplied by service factor

Scores if within ±3% of the reference value.

Motor margin

motor_margin_kw

Motor available power minus required motor power

Scores if within ±3% of the reference value.

Pressure certificate margin

pressure_certificate_margin_kpa

Certificate pressure rating minus maximum system pressure

Scores if within ±3% of the reference value.

Evidence completeness

evidence_completeness_percent

Submitted evidence items as a percentage of required evidence items

Scores if within ±3% of the reference value.

Certificate match

certificate_match_percent

Matching certificate fields as a percentage of required certificate fields

Scores if within ±3% of the reference value.

Review closeout

review_closeout_percent

Closed reviewer comments as a percentage of total reviewer comments

Scores if within ±3% of the reference value.

Review days remaining

review_days_remaining

Allowed review period minus elapsed review days

Scores if within ±3% of the reference value.

Approved deviation count

approved_deviation_count

Deviation count accepted with reviewer evidence

Scores if within ±3% of the reference value.

Unresolved deviation count

unresolved_deviation_count

Deviation count still unresolved at disposition

Scores if within ±3% of the reference value.

Open critical comments

open_critical_comments

Critical reviewer comments still open at disposition

Scores if within ±3% of the reference value.

Overall pass score

overall_pass_score

One when duty, evidence, certificate, comment, schedule, and deviation checks pass

Scores if within ±3% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-15 product submittal compliance package
medium:
All source-pack values given for the SSC-15 product submittal compliance package
hard:
All source-pack values given for the SSC-15 product submittal compliance package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • product-submittal-compliance-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a mechanical product reviewer checking a task-owned synthetic SSC-15 product submittal review packet for one pump package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. Procore Submittals, Autodesk Construction Cloud Submittals, ICC-ES evaluation reports, UL Product iQ, FM Approvals, and BBA certificate routes shape the workflow context only; they are not extra data sources for this instance.4 5## Scene6 7- Submittal package: `SUB-15-PKG-001`8- Submittal register: `REG-15-SUB-001`9- Pump datasheet: `DAT-15-PUMP-001`10- Pressure certificate: `CERT-15-PRESS-001`11- Duty calculation excerpt: `CALC-15-DUTY-001`12- Review comment log: `CMT-15-REVIEW-001`13- Deviation register: `DEV-15-LOG-001`14- Disposition memo: `MEMO-15-DISP-001`15- Specification section: `SPEC-15-15100`16 17All checks use the same `SUB-15-PKG-001` product package and `REG-15-SUB-001` register. Do not change the product identity, certificate identity, duty point, review comments, deviation status, or specification basis unless you explicitly flag a source conflict.18 19Unit convention for this source pack:20 21- Flow is in L/s.22- Head is in m.23- Motor power is in kW.24- Pressure is in kPa.25- Review periods are calendar days.26- Evidence, certificates, comments, and deviations are counts.27 28## Duty And Certificate Basis29 30| Item | Value |31|------|-------|32| Required design flow | {{ required_flow_l_s }} L/s |33| Submitted datasheet flow | {{ submitted_flow_l_s }} L/s |34| Required design head | {{ required_head_m }} m |35| Submitted datasheet head | {{ submitted_head_m }} m |36| Pump BEP flow | {{ bep_flow_l_s }} L/s |37| POR lower bound | {{ por_low_percent }} percent of BEP flow |38| POR upper bound | {{ por_high_percent }} percent of BEP flow |39| Motor nameplate power | {{ motor_nameplate_kw }} kW |40| Motor service factor | {{ motor_service_factor }} |41| Required motor power | {{ required_motor_kw }} kW |42| Maximum system pressure | {{ max_system_pressure_kpa }} kPa |43| Certificate pressure rating | {{ certificate_pressure_rating_kpa }} kPa |44 45Duty and certificate checks:46 47- Flow capacity margin equals `submitted_flow_l_s - required_flow_l_s`.48- Flow capacity ratio equals `submitted_flow_l_s / required_flow_l_s`.49- Head capacity margin equals `submitted_head_m - required_head_m`.50- Head capacity ratio equals `submitted_head_m / required_head_m`.51- BEP flow percent equals `required_flow_l_s / bep_flow_l_s x 100`.52- POR minimum flow equals `bep_flow_l_s x por_low_percent / 100`.53- POR maximum flow equals `bep_flow_l_s x por_high_percent / 100`.54- POR low margin equals `required_flow_l_s - por_min_flow_l_s`.55- POR high margin equals `por_max_flow_l_s - required_flow_l_s`.56- Motor available power equals `motor_nameplate_kw x motor_service_factor`.57- Motor margin equals `motor_available_kw - required_motor_kw`.58- Pressure certificate margin equals `certificate_pressure_rating_kpa - max_system_pressure_kpa`.59 60## Evidence And Review Basis61 62| Item | Value |63|------|-------|64| Required evidence items | {{ required_evidence_items }} |65| Submitted evidence items | {{ submitted_evidence_items }} |66| Required certificate fields | {{ required_certificate_items }} |67| Matching certificate fields | {{ matching_certificate_items }} |68| Total review comments | {{ total_review_comments }} |69| Closed review comments | {{ closed_review_comments }} |70| Open critical comments | {{ open_critical_comments }} |71| Allowed review period | {{ review_period_days }} days |72| Elapsed review period | {{ elapsed_review_days }} days |73| Approved deviations | {{ approved_deviation_count }} |74| Unresolved deviations | {{ unresolved_deviation_count }} |75 76Evidence and review checks:77 78- Evidence completeness percent equals `submitted_evidence_items / required_evidence_items x 100`.79- Certificate match percent equals `matching_certificate_items / required_certificate_items x 100`.80- Review closeout percent equals `closed_review_comments / total_review_comments x 100`.81- Review days remaining equals `review_period_days - elapsed_review_days`.82- Approved deviation count is the source value from `DEV-15-LOG-001`.83- Unresolved deviation count is the source value from `DEV-15-LOG-001`.84- Open critical comments is the source value from `CMT-15-REVIEW-001`.85- Overall pass score is `1.0` only when flow, head, POR low/high margins, motor margin, pressure certificate margin, evidence completeness, certificate match, review closeout, review days remaining, open critical comments, and unresolved deviations all pass; otherwise it is `0.0`.86 87## Output Format88 89Write a compact submittal disposition memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Explain the calculations briefly, preserve the object IDs above, and state whether the baseline source pack passes the current docs-only checks.90 91Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.92 93Include a fenced JSON block with exactly these numeric keys:94 95```json96{97 "flow_capacity_margin_l_s": <numeric_value>,98 "flow_capacity_ratio": <numeric_value>,99 "head_capacity_margin_m": <numeric_value>,100 "head_capacity_ratio": <numeric_value>,101 "bep_flow_percent": <numeric_value>,102 "por_min_flow_l_s": <numeric_value>,103 "por_max_flow_l_s": <numeric_value>,104 "por_low_margin_l_s": <numeric_value>,105 "por_high_margin_l_s": <numeric_value>,106 "motor_available_kw": <numeric_value>,107 "motor_margin_kw": <numeric_value>,108 "pressure_certificate_margin_kpa": <numeric_value>,109 "evidence_completeness_percent": <numeric_value>,110 "certificate_match_percent": <numeric_value>,111 "review_closeout_percent": <numeric_value>,112 "review_days_remaining": <numeric_value>,113 "approved_deviation_count": <numeric_value>,114 "unresolved_deviation_count": <numeric_value>,115 "open_critical_comments": <numeric_value>,116 "overall_pass_score": <numeric_value>117}118```119

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc15 product submittal case

ssc15_product_submittal_case

Pump product submittal review packet with datasheet, certificate, duty, comment, and deviation checks

pump-product-submittal-reviewequipment-compliance-overlay
Parameter ranges
required_flow_l_s
42
submitted_flow_l_s
45
required_head_m
38
submitted_head_m
40
bep_flow_l_s
40
por_low_percent
70
por_high_percent
120
motor_nameplate_kw
30
motor_service_factor
1.15
required_motor_kw
23
max_system_pressure_kpa
860
certificate_pressure_rating_kpa
1000
required_evidence_items
8
submitted_evidence_items
8
required_certificate_items
4
matching_certificate_items
4
total_review_comments
5
closed_review_comments
5
open_critical_comments
0
review_period_days
14
elapsed_review_days
10
approved_deviation_count
1
unresolved_deviation_count
0

Example task

pump-product-submittal-review-ssc15-product-submittal-case-preview — hard difficulty, all inputs given.

Pump product submittal review packet with datasheet, certificate, duty, comment, and deviation checks. pump-product-submittal-review. Required outputs: flow_capacity_margin_l_s, flow_capacity_ratio, head_capacity_margin_m, head_capacity_ratio, bep_flow_percent, por_min_flow_l_s

The model sees

Scenario context and visible inputs.

required_flow_l_s
42 to 42 L/s
submitted_flow_l_s
45 to 45 L/s
required_head_m
38 to 38 m
submitted_head_m
40 to 40 m
bep_flow_l_s
40 to 40 L/s
por_low_percent
70 to 70 %
por_high_percent
120 to 120 %
motor_nameplate_kw
30 to 30 kW
motor_service_factor
1.15 to 1.15
required_motor_kw
23 to 23 kW
max_system_pressure_kpa
860 to 860 kPa
certificate_pressure_rating_kpa
1000 to 1000 kPa
required_evidence_items
8 to 8 count
submitted_evidence_items
8 to 8 count
required_certificate_items
4 to 4 count
matching_certificate_items
4 to 4 count
total_review_comments
5 to 5 count
closed_review_comments
5 to 5 count
open_critical_comments
0 to 0 count
review_period_days
14 to 14 days
elapsed_review_days
10 to 10 days
approved_deviation_count
1 to 1 count
unresolved_deviation_count
0 to 0 count

Executable tool: product-submittal-compliance-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "flow_capacity_margin_l_s": <number>,
  "flow_capacity_ratio": <number>,
  "head_capacity_margin_m": <number>,
  "head_capacity_ratio": <number>,
  "bep_flow_percent": <number>,
  "por_min_flow_l_s": <number>,
  "por_max_flow_l_s": <number>,
  "por_low_margin_l_s": <number>,
  "por_high_margin_l_s": <number>,
  "motor_available_kw": <number>,
  "motor_margin_kw": <number>,
  "pressure_certificate_margin_kpa": <number>,
  "evidence_completeness_percent": <number>,
  "certificate_match_percent": <number>,
  "review_closeout_percent": <number>,
  "review_days_remaining": <number>,
  "approved_deviation_count": <number>,
  "unresolved_deviation_count": <number>,
  "open_critical_comments": <number>,
  "overall_pass_score": <number>
}
  • flow_capacity_margin_l_s · scored within ±3%
  • flow_capacity_ratio · scored within ±3%
  • head_capacity_margin_m · scored within ±3%
  • head_capacity_ratio · scored within ±3%
  • bep_flow_percent · scored within ±3%
  • por_min_flow_l_s · scored within ±3%
  • por_max_flow_l_s · scored within ±3%
  • por_low_margin_l_s · scored within ±3%
  • por_high_margin_l_s · scored within ±3%
  • motor_available_kw · scored within ±3%
  • motor_margin_kw · scored within ±3%
  • pressure_certificate_margin_kpa · scored within ±3%
  • evidence_completeness_percent · scored within ±3%
  • certificate_match_percent · scored within ±3%
  • review_closeout_percent · scored within ±3%
  • review_days_remaining · scored within ±3%
  • approved_deviation_count · scored within ±3%
  • unresolved_deviation_count · scored within ±3%
  • open_critical_comments · scored within ±3%
  • overall_pass_score · scored within ±3%