LIVEdataset aec-bench@releasetasks 552models 18disciplines 5last submission … · built …

mechanicalno-tool

Product Submittal Compliance Issue Review Package

Review-native long-horizon task over a generated SSC-15 product submittal issue packet. The agent receives a multi-file source packet in /workspace/sources/ (document register, submittal manifest, mill certificates, heat traceability table, product application schedule, deviation register, and criteria/comments), inventories it, preserves product, heat, certificate, and application identity, checks source-owned carbon-equivalent and property calculations as review evidence, assigns one status per review item, raises findings/information requests/actions, and issues a readiness decision. Packet variants inject missing heat traceability, stale certificate revisions, heat identity mismatches, copied standard applicability, open comments, or genuine carbon-equivalent failures. Scored by a stage-gated custom verifier, not per-key answer matching.

no-tool: The model must reason numerically unaided.

Standards

AS/NZS 3678 AWS D1.1 EN 10204 ISO 15614 Project submittal review workflow

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

01
Template
The reusable contract shown on this page.
02
Scenario
An archetype and site context are sampled.
03
Difficulty tier
Inputs may be hidden at harder tiers.
04
Task prompt
The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

8 inputs

Always given

Included directly in every task prompt.

Cev limit
cev_limit
Source-owned carbon-equivalent limit for the product use case
0.42 – 0.48
Required yield
required_yield_mpa
Source-owned minimum yield strength for the application grade
345 – 420 MPa
Required tensile
required_tensile_mpa
Source-owned minimum tensile strength for the application grade
470 – 560 MPa

Hidden at higher difficulty

Visible in easier tasks and withheld in one or more harder tiers.

Carbon equivalent margin
carbon_equivalent_margin
Derivation margin below the CEV limit for the controlling heat
Hidden at easy and medium and hard difficulty.
0.025 – 0.075
Carbon equivalent deficit
carbon_equivalent_deficit
Derivation exceedance above the CEV limit for the carbon-equivalent failure variant
Hidden at easy and medium and hard difficulty.
0.012 – 0.04
Yield strength margin
yield_strength_margin_mpa
Derivation margin between certificate yield and required yield
Hidden at easy and medium and hard difficulty.
30 – 85 MPa
Tensile strength margin
tensile_strength_margin_mpa
Derivation margin between certificate tensile strength and required tensile strength
Hidden at easy and medium and hard difficulty.
35 – 100 MPa
Packet variant
packet_variant
Hidden source-packet variant
Hidden at easy and medium and hard difficulty.
cleanmissing_heat_numberstale_certificate_revisionheat_number_mismatchscenario_copy_forwardopen_critical_commentminor_open_comment_carriedcarbon_equivalent_exceeds

Scored outputs

19 outputs

Rlr 01 status

rlr_01_status

Review matrix status code for RLR-01

Scores if within ±0% of the reference value.

Rlr 02 status

rlr_02_status

Review matrix status code for RLR-02

Scores if within ±0% of the reference value.

Rlr 03 status

rlr_03_status

Review matrix status code for RLR-03

Scores if within ±0% of the reference value.

Rlr 04 status

rlr_04_status

Review matrix status code for RLR-04

Scores if within ±0% of the reference value.

Rlr 05 status

rlr_05_status

Review matrix status code for RLR-05

Scores if within ±0% of the reference value.

Rlr 06 status

rlr_06_status

Review matrix status code for RLR-06

Scores if within ±0% of the reference value.

Rlr 07 status

rlr_07_status

Review matrix status code for RLR-07

Scores if within ±0% of the reference value.

Rlr 08 status

rlr_08_status

Review matrix status code for RLR-08

Scores if within ±0% of the reference value.

Rlr 09 status

rlr_09_status

Review matrix status code for RLR-09

Scores if within ±0% of the reference value.

Readiness code

readiness_code

Readiness decision code

Scores if within ±0% of the reference value.

Required findings count

required_findings_count

Required finding count

Scores if within ±0% of the reference value.

Required information requests count

required_information_requests_count

Required information request count

Scores if within ±0% of the reference value.

Required carried actions count

required_carried_actions_count

Required carried action count

Scores if within ±0% of the reference value.

Carbon equivalent max

carbon_equivalent_max

Maximum recomputed carbon equivalent for applied heats

Scores if within ±2% of the reference value.

Carbon equivalent margin

carbon_equivalent_margin

CEV limit minus maximum recomputed carbon equivalent

Scores if within ±2% of the reference value.

Yield strength margin mpa

yield_strength_margin_mpa

Minimum certified yield strength minus required yield strength

Scores if within ±2% of the reference value.

Tensile strength margin mpa

tensile_strength_margin_mpa

Minimum certified tensile strength minus required tensile strength

Scores if within ±2% of the reference value.

Certificate coverage count

certificate_coverage_count

Applied heats covered by mill certificates

Scores if within ±0% of the reference value.

Traceability match count

traceability_match_count

Application rows with traceable heat numbers

Scores if within ±0% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

easy

Some inputs hidden

Obvious packet defects: clean, missing heat traceability, or a carbon-equivalent failure

Hidden inputs

Packet variantpacket_variant
Carbon equivalent margincarbon_equivalent_margin
Carbon equivalent deficitcarbon_equivalent_deficit
Yield strength margin mpayield_strength_margin_mpa
Tensile strength margin mpatensile_strength_margin_mpa

Packet variant restricted to: clean, missing_heat_number, carbon_equivalent_exceeds

medium

Some inputs hidden

Full packet-variant distribution

Hidden inputs

Packet variantpacket_variant
Carbon equivalent margincarbon_equivalent_margin
Carbon equivalent deficitcarbon_equivalent_deficit
Yield strength margin mpayield_strength_margin_mpa
Tensile strength margin mpatensile_strength_margin_mpa

hard

Some inputs hidden

Subtle documentary defects: stale certificate revisions, heat identity mismatch, copied standard basis, carried comments

Hidden inputs

Packet variantpacket_variant
Carbon equivalent margincarbon_equivalent_margin
Carbon equivalent deficitcarbon_equivalent_deficit
Yield strength margin mpayield_strength_margin_mpa
Tensile strength margin mpatensile_strength_margin_mpa

Packet variant restricted to: stale_certificate_revision, heat_number_mismatch, scenario_copy_forward, minor_open_comment_carried

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

instruction.md

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are the independent reviewing engineer for a product submittal issue package covering steel product identity, mill certificates, heat traceability, application schedule, deviations, review comments, and criteria.2 3A source packet has been placed in `/workspace/sources/`. It contains a document register, submittal manifest, mill certificates, heat traceability table, product application schedule, deviation register, and a criteria memo with review comments. The packet is a task-owned synthetic source pack; treat it as the only source of numeric truth for this review.4 5Your job is not to redesign the product or accept the material for construction. Your job is to decide whether the package is ready to issue, and to produce an auditable review record.6 7## Review Workflow8 91. Inventory the source packet before drawing any conclusion. Record every document ID, revision, and status.102. Build an identity ledger: submittal, product, grade, heats, certificates, application schedule, deviation register, and criteria memo.113. Check for source conflicts, stale revisions, copied standards or use cases, missing evidence, heat/certificate mismatches, unresolved deviations, and open critical comments before accepting any package claim.124. Recompute the package's own checks only where they answer review items, using the assessment bases stated in the criteria memo. Do not import methods or values from outside the packet.135. Assign exactly one status to every review item: `pass`, `fail`, `not_applicable`, or `insufficient_data`.146. Do not invent missing values. Mark missing evidence as `insufficient_data` and request the exact missing field and source. A value that a source explicitly marks as pending or awaiting confirmation is missing evidence of this kind: it does not make otherwise-reconciling identifiers inconsistent and does not make evidence that is present untraceable; the check that cannot be completed without it takes `insufficient_data`.157. Convert every failure into a finding with a source pointer, affected object, consequence, and corrective action.168. Issue a readiness decision that reconciles with your matrix, findings, information requests, and action register.17 18## Review Matrix19 20Assess each item and give it exactly one status:21 22| Item | Review question |23|---|---|24| RLR-01 | Packet completeness: are all required submittal, certificate, traceability, application, deviation, and criteria files present with IDs and revisions? |25| RLR-02 | Object identity: do the submittal, product, grade, heat numbers, certificates, application rows, deviations, and criteria memo stay consistent? |26| RLR-03 | Certificate and property basis: are chemistry, mechanical properties, certificate scope, and source revisions traceable, current, and recomputable? |27| RLR-04 | Product compliance adequacy: do carbon equivalent, yield strength, tensile strength, certificate coverage, and traceability clear the source criteria for the same product use case? |28| RLR-05 | Application consequence: is the same product grade, standard, and application use case used across the schedule, certificates, deviations, and criteria? |29| RLR-06 | Source-status resilience: are certificate coverage and heat traceability source-backed for the issue package? |30| RLR-07 | Comment and action closure: is every critical comment closed, and are carried minor comments owner/action controlled? |31| RLR-08 | Readiness consistency: does your final decision match your own matrix, findings, information requests, and action register? |32| RLR-09 | Claim boundary: does your review avoid unsupported approval, compliance, source-hardening, executable-verifier, or benchmark-readiness claims? |33 34## Boundary Rules35 36- Use the review matrix definitions to decide the most specific affected item from the source packet. Do not infer a status from this instruction alone.37- If a source value needed for a recomputation is absent, omit the dependent `computed_evidence` key and raise an information request for the missing field and source. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.38- Assign additional failures only when the source packet gives independent evidence for them; do not double-count one source issue across unrelated matrix rows.39- RLR-08 passes when the readiness decision reconciles with the review matrix, findings, information requests, and action register.40- Every finding, information request, and action must name one exact RLR item. Use a single RLR item per register row; do not write combined items such as `RLR-04/RLR-06`.41- Do not rename computed_evidence keys.42 43## Output44 45Write your complete review to `/workspace/output.md`. Explain your reasoning briefly in prose, then end with exactly one fenced JSON block:46 47```json48{49  "source_inventory": [{"doc_id": "...", "revision": "...", "status": "..."}],50  "identity_ledger": {51    "submittal": "...",52    "product": "...",53    "grade": "...",54    "heat_a": "...",55    "heat_b": "...",56    "certificates": "...",57    "application_schedule": "...",58    "deviation_register": "...",59    "criteria_memo": "..."60  },61  "review_matrix": {62    "RLR-01": {"status": "pass|fail|not_applicable|insufficient_data", "evidence": "..."},63    "RLR-02": {"status": "...", "evidence": "..."},64    "RLR-03": {"status": "...", "evidence": "..."},65    "RLR-04": {"status": "...", "evidence": "..."},66    "RLR-05": {"status": "...", "evidence": "..."},67    "RLR-06": {"status": "...", "evidence": "..."},68    "RLR-07": {"status": "...", "evidence": "..."},69    "RLR-08": {"status": "...", "evidence": "..."},70    "RLR-09": {"status": "...", "evidence": "..."}71  },72  "computed_evidence": {73    "carbon_equivalent_max": 0.0,74    "carbon_equivalent_margin": 0.0,75    "yield_strength_margin_mpa": 0.0,76    "tensile_strength_margin_mpa": 0.0,77    "certificate_coverage_count": 0.0,78    "traceability_match_count": 0.079  },80  "findings": [81    {"item": "RLR-0X", "severity": "critical|minor", "source_id": "...", "object_id": "...", "consequence": "...", "action": "..."}82  ],83  "information_requests": [84    {"item": "RLR-0X", "missing_field": "...", "source_id": "..."}85  ],86  "action_register": [87    {"action": "...", "owner": "...", "linked_item": "RLR-0X"}88  ],89  "readiness_decision": "ready_to_issue|ready_with_carried_actions|not_ready_to_issue",90  "claim_boundary_statement": "..."91}92```93 94Rules for the structured block:95 96- `computed_evidence` values must come from your own recomputation from packet source values. Omit a key only when its inputs are missing from the packet, then raise the matching information request instead. Do not include missing or unrecomputable keys with `null`, `0`, or placeholder values.97- Every `fail` needs at least one finding with a non-empty `source_id`, `object_id`, `consequence`, and `action`.98- Every `insufficient_data` needs an information request naming the exact missing field and its source document.99- Every `not_applicable` needs a scope reason in its matrix `evidence`.100- Carried actions must appear in `action_register` with an owner.101- `readiness_decision` must reconcile with your matrix: unresolved failures or missing critical evidence mean the package is not ready.102- `claim_boundary_statement` must state that this review covers a task-owned synthetic source packet and does not claim authority approval, accepted project evidence, full standards compliance, source-pack hardening, executable-verifier readiness, or benchmark readiness.103

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Bridge plate submittal

bridge_plate_submittal

Bridge-plate steel product submittal with mill certificates and weldability criteria

bridge-plate-product-reviewwelded-plate-submittal

Parameter ranges

cev_limit: 0.42 – 0.46
required_yield_mpa: 345 – 380
required_tensile_mpa: 470 – 520

Plant skid frame submittal

plant_skid_frame_submittal

Plant skid-frame steel product submittal with heat traceability and deviation closeout

skid-frame-product-reviewfabricated-frame-submittal

Parameter ranges

cev_limit: 0.44 – 0.48
required_yield_mpa: 380 – 420
required_tensile_mpa: 500 – 560

Example task

bridge-plate-product-review-bridge-plate-submittal-preview — hard difficulty, some inputs hidden.

Bridge-plate steel product submittal with mill certificates and weldability criteria. bridge-plate-product-review. Required outputs: rlr_01_status, rlr_02_status, rlr_03_status, rlr_04_status, rlr_05_status, rlr_06_status

The model sees

Scenario context and visible inputs.

cev_limit: 0.42 to 0.46
required_yield_mpa: 345 to 380 MPa
required_tensile_mpa: 470 to 520 MPa

Executable tool: product-submittal-compliance-issue-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Packet variant
packet_variant
Carbon equivalent deficit
carbon_equivalent_deficit
Yield strength margin mpa
yield_strength_margin_mpa
Carbon equivalent margin
carbon_equivalent_margin
Tensile strength margin mpa
tensile_strength_margin_mpa

The model must produce

The scored JSON answer schema.

{
  "rlr_01_status": <number>,
  "rlr_02_status": <number>,
  "rlr_03_status": <number>,
  "rlr_04_status": <number>,
  "rlr_05_status": <number>,
  "rlr_06_status": <number>,
  "rlr_07_status": <number>,
  "rlr_08_status": <number>,
  "rlr_09_status": <number>,
  "readiness_code": <number>,
  "required_findings_count": <number>,
  "required_information_requests_count": <number>,
  "required_carried_actions_count": <number>,
  "carbon_equivalent_max": <number>,
  "carbon_equivalent_margin": <number>,
  "yield_strength_margin_mpa": <number>,
  "tensile_strength_margin_mpa": <number>,
  "certificate_coverage_count": <number>,
  "traceability_match_count": <number>
}

rlr_01_status · scored within ±0%
rlr_02_status · scored within ±0%
rlr_03_status · scored within ±0%
rlr_04_status · scored within ±0%
rlr_05_status · scored within ±0%
rlr_06_status · scored within ±0%
rlr_07_status · scored within ±0%
rlr_08_status · scored within ±0%
rlr_09_status · scored within ±0%
readiness_code · scored within ±0%
required_findings_count · scored within ±0%
required_information_requests_count · scored within ±0%
required_carried_actions_count · scored within ±0%
carbon_equivalent_max · scored within ±2%
carbon_equivalent_margin · scored within ±2%
yield_strength_margin_mpa · scored within ±2%
tensile_strength_margin_mpa · scored within ±2%
certificate_coverage_count · scored within ±0%
traceability_match_count · scored within ±0%

Product Submittal Compliance Issue Review Package

How this task is generated

Template

Scenario

Difficulty tier

Task prompt

Parameters

Inputs

Scored outputs

Rlr 01 status

Rlr 02 status

Rlr 03 status

Rlr 04 status

Rlr 05 status

Rlr 06 status

Rlr 07 status

Rlr 08 status

Rlr 09 status

Readiness code

Required findings count

Required information requests count

Required carried actions count

Carbon equivalent max

Carbon equivalent margin

Yield strength margin mpa

Tensile strength margin mpa

Certificate coverage count

Traceability match count

Difficulty

easy

medium

hard

Task bundle

Scenario archetypes

Bridge plate submittal

Plant skid frame submittal

Example task

The model sees

The model must infer

The model must produce