LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
electricalwith-tool

Electrical Source Policy Product Datasheet Package

Calculates deterministic SSC-05 datasheet and source-policy metrics from a task-owned synthetic source pack. The template combines datasheet completeness, source traceability, derated product current, cable and breaker margins, voltage-drop margin, protection-setting margin, review comments, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

electrical source-policy gate workflowproduct datasheet review workflowsubmittal register workflow

Tags

datasheetdeterministicelectricalprotectionratingssource-policysubmittal

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

15 inputs

Always given

Included directly in every task prompt.

15
Show 15 inputs

  • Provided datasheet fields

    provided_datasheet_fields

    DATA-05-PRODUCT-08 provided datasheet fields

    18

  • Required datasheet fields

    required_datasheet_fields

    DATA-05-PRODUCT-08 required datasheet fields

    20

  • Traced source values

    traced_source_values

    CALC-05-WORKSHEET-08 source values traced into the calculation

    6

  • Required source values

    required_source_values

    CALC-05-WORKSHEET-08 required source values

    7

  • Product nameplate current

    product_nameplate_current_a

    RATING-05-TABLE-08 product nameplate current

    240 A

  • Product derating factor

    product_derating_factor

    RATING-05-TABLE-08 product derating factor

    0.884

  • Design current

    design_current_a

    CALC-05-WORKSHEET-08 design current consumed by the calculation

    160 A

  • Cable allowable current

    cable_allowable_current_a

    RATING-05-TABLE-08 cable allowable current

    195 A

  • Max voltage drop percent

    max_voltage_drop_percent

    CALC-05-WORKSHEET-08 maximum allowed voltage drop

    4 %

  • Calculated voltage drop percent

    calculated_voltage_drop_percent

    CALC-05-WORKSHEET-08 calculated voltage drop

    3.1 %

  • Protection setting margin percent

    protection_setting_margin_percent

    CALC-05-WORKSHEET-08 protection setting margin

    8 %

  • Review comment count

    review_comment_count

    SUBMIT-05-REGISTER-08 review comment count

    9

  • Resolved comment count

    resolved_comment_count

    SUBMIT-05-REGISTER-08 resolved review comments

    7

  • Critical open comments

    critical_open_comments

    SUBMIT-05-REGISTER-08 critical open comments

    0

  • Response completeness score

    response_completeness_score

    RESPONSE-05-SUBMITTAL-08 response completeness score

    0.92

Scored outputs

11 outputs

Datasheet completeness fraction

datasheet_completeness_fraction

Provided datasheet fields divided by required fields

Scores if within ±0.3% of the reference value.

Source trace score

source_trace_score

Traced source values divided by required source values

Scores if within ±0.3% of the reference value.

Derated product current

derated_product_current_a

Product nameplate current after derating

Scores if within ±3% of the reference value.

Breaker rating margin

breaker_rating_margin_a

Derated product current minus design current

Scores if within ±3% of the reference value.

Cable rating margin

cable_rating_margin_a

Cable allowable current minus design current

Scores if within ±3% of the reference value.

Voltage drop margin

voltage_drop_margin_percent

Maximum voltage drop minus calculated voltage drop

Scores if within ±3% of the reference value.

Protection setting margin

protection_setting_margin_percent

Protection setting margin carried from the worksheet

Scores if within ±3% of the reference value.

Open comments

open_comments

Open review comments

Scores if within ±3% of the reference value.

Critical open comments

critical_open_comments

Critical open comments

Scores if within ±3% of the reference value.

Response completeness score

response_completeness_score

Response completeness score

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when source, rating, setting, response, and critical-comment checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-05 datasheet/source-policy package
medium:
All source-pack values given for the SSC-05 datasheet/source-policy package
hard:
All source-pack values given for the SSC-05 datasheet/source-policy package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • electrical-source-policy-product-datasheet-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are an electrical design engineer checking `SSC-05-LH-08`, a task-owned synthetic SSC-05 electrical source-policy and product datasheet package.2 3Use only the task-owned synthetic source pack values below for numeric grading. Datasheet review, certificate/listing, rating-table, worksheet, and submittal-register workflows shape the context only; this instance does not parse a real datasheet, certificate, product listing, or calculation workbook.4 5## Scene6 7- Design case: `CASE-SSC05-DATASHEET-08`8- Product datasheet: `DATA-05-PRODUCT-08`9- Certificate/listing record: `CERT-05-LISTING-08`10- Rating table: `RATING-05-TABLE-08`11- Calculation worksheet: `CALC-05-WORKSHEET-08`12- Submittal register: `SUBMIT-05-REGISTER-08`13- Submittal response: `RESPONSE-05-SUBMITTAL-08`14 15## Source Values16 17| Item | Value |18|------|-------|19| Provided datasheet fields | {{ provided_datasheet_fields }} |20| Required datasheet fields | {{ required_datasheet_fields }} |21| Traced source values | {{ traced_source_values }} |22| Required source values | {{ required_source_values }} |23| Product nameplate current | {{ product_nameplate_current_a }} A |24| Product derating factor | {{ product_derating_factor }} |25| Design current | {{ design_current_a }} A |26| Cable allowable current | {{ cable_allowable_current_a }} A |27| Maximum voltage drop | {{ max_voltage_drop_percent }} % |28| Calculated voltage drop | {{ calculated_voltage_drop_percent }} % |29| Protection-setting margin | {{ protection_setting_margin_percent }} % |30| Review comments | {{ review_comment_count }} |31| Resolved comments | {{ resolved_comment_count }} |32| Critical open comments | {{ critical_open_comments }} |33| Response completeness score | {{ response_completeness_score }} |34 35Checks:36 37- Datasheet completeness equals provided fields divided by required fields.38- Source trace score equals traced source values divided by required source values.39- Derated product current equals nameplate current times the derating factor.40- Breaker and cable margins compare derated or allowable current against design current.41- Open comments equal review comments minus resolved comments.42- Overall pass score is `1.0` only when source, rating, setting, response, and critical-comment checks pass; otherwise it is `0.0`.43 44## Output Format45 46Write a compact memo to `/workspace/output.md`. Preserve the object IDs above, state the task-owned synthetic source-pack boundary, and explain whether the baseline source pack passes the docs-only checks.47 48Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated datasheet extraction evidence, generated benchmark readiness, or benchmark readiness.49 50Include a fenced JSON block with exactly these numeric keys:51 52```json53{54 "datasheet_completeness_fraction": <numeric_value>,55 "source_trace_score": <numeric_value>,56 "derated_product_current_a": <numeric_value>,57 "breaker_rating_margin_a": <numeric_value>,58 "cable_rating_margin_a": <numeric_value>,59 "voltage_drop_margin_percent": <numeric_value>,60 "protection_setting_margin_percent": <numeric_value>,61 "open_comments": <numeric_value>,62 "critical_open_comments": <numeric_value>,63 "response_completeness_score": <numeric_value>,64 "overall_pass_score": <numeric_value>65}66```67

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc05 datasheet source policy case

ssc05_datasheet_source_policy_case

Electrical datasheet/source-policy package tied to one product datasheet, certificate, rating table, worksheet, register, and response

electrical-source-policy-datasheet
Parameter ranges
provided_datasheet_fields
18
required_datasheet_fields
20
traced_source_values
6
required_source_values
7
product_nameplate_current_a
240
product_derating_factor
0.884
design_current_a
160
cable_allowable_current_a
195
max_voltage_drop_percent
4
calculated_voltage_drop_percent
3.1
protection_setting_margin_percent
8
review_comment_count
9
resolved_comment_count
7
critical_open_comments
0
response_completeness_score
0.92

Example task

electrical-source-policy-datasheet-ssc05-datasheet-source-policy-case-preview — hard difficulty, all inputs given.

Electrical datasheet/source-policy package tied to one product datasheet, certificate, rating table, worksheet, register, and response. electrical-source-policy-datasheet. Required outputs: datasheet_completeness_fraction, source_trace_score, derated_product_current_a, breaker_rating_margin_a, cable_rating_margin_a, voltage_drop_margin_percent

The model sees

Scenario context and visible inputs.

provided_datasheet_fields
18 to 18
required_datasheet_fields
20 to 20
traced_source_values
6 to 6
required_source_values
7 to 7
product_nameplate_current_a
240 to 240 A
product_derating_factor
0.884 to 0.884
design_current_a
160 to 160 A
cable_allowable_current_a
195 to 195 A
max_voltage_drop_percent
4 to 4 %
calculated_voltage_drop_percent
3.1 to 3.1 %
protection_setting_margin_percent
8 to 8 %
review_comment_count
9 to 9
resolved_comment_count
7 to 7
critical_open_comments
0 to 0
response_completeness_score
0.92 to 0.92

Executable tool: electrical-source-policy-product-datasheet-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "datasheet_completeness_fraction": <number>,
  "source_trace_score": <number>,
  "derated_product_current_a": <number>,
  "breaker_rating_margin_a": <number>,
  "cable_rating_margin_a": <number>,
  "voltage_drop_margin_percent": <number>,
  "protection_setting_margin_percent": <number>,
  "open_comments": <number>,
  "critical_open_comments": <number>,
  "response_completeness_score": <number>,
  "overall_pass_score": <number>
}
  • datasheet_completeness_fraction · scored within ±0.3%
  • source_trace_score · scored within ±0.3%
  • derated_product_current_a · scored within ±3%
  • breaker_rating_margin_a · scored within ±3%
  • cable_rating_margin_a · scored within ±3%
  • voltage_drop_margin_percent · scored within ±3%
  • protection_setting_margin_percent · scored within ±3%
  • open_comments · scored within ±3%
  • critical_open_comments · scored within ±3%
  • response_completeness_score · scored within ±0.3%
  • overall_pass_score · scored within ±0.1%