LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
mechanicalwith-tool

Certificate Conflict Repair Portfolio Package

Calculates source-bound certificate conflict repair metrics from a deterministic SSC-15 task-owned source pack. The template combines two conflicting product datasheets, a certificate record, source index, calculation trace, and repair response.

with-tool: The model is given an executable Python calculator script.

Standards

product certificate conflict reviewsource authority reviewsubstitution review

Tags

certificatedatasheetdeterministicmechanicalproduct-compliancerepairsource-conflict

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

14 inputs

Always given

Included directly in every task prompt.

14
Show 14 inputs

  • Selected source is current

    selected_source_is_current

    INDEX-15-SRC-08 selected source is current

    1

  • Selected source has authority

    selected_source_has_authority

    INDEX-15-SRC-08 selected source has governing authority

    1

  • Affected calculation

    affected_calculation_count

    TRACE-15-CALC-08 affected calculation count

    5 count

  • Updated calculation

    updated_calculation_count

    TRACE-15-CALC-08 updated affected calculation count

    5 count

  • Governing certificate capacity

    governing_certificate_capacity_kn

    CERT-15-REC-08 governing certificate capacity

    125 kN

  • Conflicting datasheet capacity

    conflicting_datasheet_capacity_kn

    DAT-15-A-08 superseded datasheet capacity

    110 kN

  • Replacement capacity

    replacement_capacity_kn

    DAT-15-B-08 replacement product capacity

    142 kN

  • Required capacity

    required_capacity_kn

    TRACE-15-CALC-08 required capacity after repair

    130 kN

  • Total conflict items

    total_conflict_items

    RESPONSE-15-REPAIR-08 total conflict items

    4 count

  • Closed conflict items

    closed_conflict_items

    RESPONSE-15-REPAIR-08 closed conflict items

    4 count

  • Unresolved conflict

    unresolved_conflict_count

    RESPONSE-15-REPAIR-08 unresolved conflict count

    0 count

  • Expired source

    expired_source_count

    INDEX-15-SRC-08 expired source count still used by the repair

    0 count

  • Completed repair memo sections

    completed_repair_memo_sections

    RESPONSE-15-REPAIR-08 completed repair memo sections

    9 count

  • Required repair memo sections

    required_repair_memo_sections

    RESPONSE-15-REPAIR-08 required repair memo sections

    10 count

Scored outputs

9 outputs

Source authority score

source_authority_score

Minimum of current-source and authority-source indicators

Scores if within ±0.1% of the reference value.

Affected calculation update fraction

affected_calculation_update_fraction

Updated calculations divided by affected calculations

Scores if within ±0.3% of the reference value.

Certificate capacity delta kn

certificate_capacity_delta_kn

Governing certificate capacity minus conflicting datasheet capacity

Scores if within ±3% of the reference value.

Replacement capacity margin kn

replacement_capacity_margin_kn

Replacement product capacity minus required capacity

Scores if within ±3% of the reference value.

Source conflict closure fraction

source_conflict_closure_fraction

Closed conflict items divided by total conflict items

Scores if within ±0.3% of the reference value.

Unresolved conflict count

unresolved_conflict_count

Unresolved conflict count left in the repair response

Scores if within ±0.1% of the reference value.

Expired source count

expired_source_count

Expired source count still used by the repair

Scores if within ±0.1% of the reference value.

Repair memo completeness fraction

repair_memo_completeness_fraction

Completed repair memo sections divided by required repair memo sections

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when source authority, affected calculations, substitution, conflict, and memo checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-15 certificate conflict repair package
medium:
All source-pack values given for the SSC-15 certificate conflict repair package
hard:
All source-pack values given for the SSC-15 certificate conflict repair package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • certificate-conflict-repair-portfolio-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a product compliance reviewer checking a task-owned synthetic SSC-15 certificate conflict and repair portfolio.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. Substitution, certificate, and review workflows shape the context only; they are not extra data sources for this instance.4 5## Scene6 7- Product: `SSC-15-LH-08`8- Conflicting datasheet A: `DAT-15-A-08`9- Replacement datasheet B: `DAT-15-B-08`10- Certificate record: `CERT-15-REC-08`11- Source index: `INDEX-15-SRC-08`12- Calculation trace: `TRACE-15-CALC-08`13- Repair response memo: `RESPONSE-15-REPAIR-08`14 15All checks use the same source index, certificate record, affected calculations, and repair memo. Do not change the selected authority source, replacement product, capacity value, or conflict status unless you explicitly flag a source conflict.16 17## Source Values18 19| Item | Value |20|------|-------|21| Selected source is current | {{ selected_source_is_current }} |22| Selected source has authority | {{ selected_source_has_authority }} |23| Affected calculations | {{ affected_calculation_count }} |24| Updated calculations | {{ updated_calculation_count }} |25| Governing certificate capacity | {{ governing_certificate_capacity_kn }} kN |26| Conflicting datasheet capacity | {{ conflicting_datasheet_capacity_kn }} kN |27| Replacement capacity | {{ replacement_capacity_kn }} kN |28| Required capacity | {{ required_capacity_kn }} kN |29| Total conflict items | {{ total_conflict_items }} |30| Closed conflict items | {{ closed_conflict_items }} |31| Unresolved conflicts | {{ unresolved_conflict_count }} |32| Expired sources still used | {{ expired_source_count }} |33| Completed repair memo sections | {{ completed_repair_memo_sections }} |34| Required repair memo sections | {{ required_repair_memo_sections }} |35 36Source authority score is the lower of the current-source and authority-source indicators.37 38## Output Format39 40Write a compact repair memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.41 42Include a fenced JSON block with exactly these numeric keys:43 44```json45{46 "source_authority_score": <numeric_value>,47 "affected_calculation_update_fraction": <numeric_value>,48 "certificate_capacity_delta_kn": <numeric_value>,49 "replacement_capacity_margin_kn": <numeric_value>,50 "source_conflict_closure_fraction": <numeric_value>,51 "unresolved_conflict_count": <numeric_value>,52 "expired_source_count": <numeric_value>,53 "repair_memo_completeness_fraction": <numeric_value>,54 "overall_pass_score": <numeric_value>55}56```57

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc15 certificate conflict case

ssc15_certificate_conflict_case

Certificate conflict and repair portfolio compliance case

ssc15-certificate-conflict
Parameter ranges

Example task

ssc15-certificate-conflict-ssc15-certificate-conflict-case-preview — hard difficulty, all inputs given.

Certificate conflict and repair portfolio compliance case. ssc15-certificate-conflict. Required outputs: source_authority_score, affected_calculation_update_fraction, certificate_capacity_delta_kn, replacement_capacity_margin_kn, source_conflict_closure_fraction, unresolved_conflict_count

The model sees

Scenario context and visible inputs.

selected_source_is_current
1 to 1
selected_source_has_authority
1 to 1
affected_calculation_count
5 to 5 count
updated_calculation_count
5 to 5 count
governing_certificate_capacity_kn
125 to 125 kN
conflicting_datasheet_capacity_kn
110 to 110 kN
replacement_capacity_kn
142 to 142 kN
required_capacity_kn
130 to 130 kN
total_conflict_items
4 to 4 count
closed_conflict_items
4 to 4 count
unresolved_conflict_count
0 to 0 count
expired_source_count
0 to 0 count
completed_repair_memo_sections
9 to 9 count
required_repair_memo_sections
10 to 10 count

Executable tool: certificate-conflict-repair-portfolio-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "source_authority_score": <number>,
  "affected_calculation_update_fraction": <number>,
  "certificate_capacity_delta_kn": <number>,
  "replacement_capacity_margin_kn": <number>,
  "source_conflict_closure_fraction": <number>,
  "unresolved_conflict_count": <number>,
  "expired_source_count": <number>,
  "repair_memo_completeness_fraction": <number>,
  "overall_pass_score": <number>
}
  • source_authority_score · scored within ±0.1%
  • affected_calculation_update_fraction · scored within ±0.3%
  • certificate_capacity_delta_kn · scored within ±3%
  • replacement_capacity_margin_kn · scored within ±3%
  • source_conflict_closure_fraction · scored within ±0.3%
  • unresolved_conflict_count · scored within ±0.1%
  • expired_source_count · scored within ±0.1%
  • repair_memo_completeness_fraction · scored within ±0.3%
  • overall_pass_score · scored within ±0.1%