LIVEdataset aec-bench@releasetasks 552models 18disciplines 5last submission … · built …

electricalwith-tool

Instrumentation Source Policy Extension Package

Calculates source-bound instrumentation source-policy metrics from a deterministic SSC-18 task-owned source pack. The template combines source traceability, linked process/electrical table updates, gap documentation, verification-case closure, authority partitioning, memo completeness, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

authority partition workflow instrumentation verification workflow source policy workflow

Tags

authority deterministic electrical gap-register instrumentation source-policy verification

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

01
Template
The reusable contract shown on this page.
02
Scenario
An archetype and site context are sampled.
03
Difficulty tier
Inputs may be hidden at harder tiers.
04
Task prompt
The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

14 inputs

Always given

Included directly in every task prompt.

14

Show 14 inputs

Source items traced
source_items_traced
INDEX-18-SOURCE-08 traced source item count
7
Source items total
source_items_total
INDEX-18-SOURCE-08 total source item count
7
Linked tables updated
linked_tables_updated
LINK-18-TABLE-08 updated linked table count
5
Linked tables total
linked_tables_total
LINK-18-TABLE-08 total linked table count
5
Documented gap count
documented_gap_count
GAP-18-REGISTER-08 documented gap count
3
Expected gap count
expected_gap_count
GAP-18-REGISTER-08 expected gap count
3
Verification cases passed
verification_cases_passed
CASE-18-VERIFY-08 passing verification case count
8
Verification cases total
verification_cases_total
CASE-18-VERIFY-08 total verification case count
8
Process margin
process_margin
LINK-18-TABLE-08 process-side margin
1.4
Electrical margin
electrical_margin
LINK-18-TABLE-08 electrical-side margin
0.8
Authority partitions signed
authority_partitions_signed
MEMO-18-EXT-08 signed authority partition count
4
Authority partitions total
authority_partitions_total
MEMO-18-EXT-08 total authority partition count
4
Unresolved conflict count
unresolved_conflict_count
MEMO-18-EXT-08 unresolved conflict count
0
Extension memo completeness fraction
extension_memo_completeness_fraction
MEMO-18-EXT-08 memo completeness fraction
0.95

Scored outputs

9 outputs

Source traceability fraction

source_traceability_fraction

Traced source items divided by total source items

Scores if within ±3% of the reference value.

Linked table update fraction

linked_table_update_fraction

Updated linked tables divided by total linked tables

Scores if within ±3% of the reference value.

Gap documentation fraction

gap_documentation_fraction

Documented gaps divided by expected gaps

Scores if within ±3% of the reference value.

Verification case pass fraction

verification_case_pass_fraction

Passing verification cases divided by total cases

Scores if within ±3% of the reference value.

Min cross domain margin

min_cross_domain_margin

Minimum process/electrical linked-table margin

Scores if within ±3% of the reference value.

Authority partition fraction

authority_partition_fraction

Signed authority partitions divided by total partitions

Scores if within ±3% of the reference value.

Unresolved conflict count

unresolved_conflict_count

Unresolved source conflict count

Scores if within ±3% of the reference value.

Extension memo completeness fraction

extension_memo_completeness_fraction

Extension memo completeness fraction

Scores if within ±3% of the reference value.

Overall pass score

overall_pass_score

Pass score for instrumentation source-policy extension checks

Scores if within ±3% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:: All source-pack values given for the SSC-18 source-policy extension package
medium:: All source-pack values given for the SSC-18 source-policy extension package
hard:: All source-pack values given for the SSC-18 source-policy extension package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

instruction.md
instrumentation-source-policy-extension-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are checking `SSC-18-LH-08`, a task-owned synthetic instrumentation source-policy and thin-substrate extension package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. Source-index, P&ID, loop schedule, linked process/electrical table, verification-case, and gap-register workflows shape the context only; this instance does not parse real source packs, PLC exports, vendor files, or authority-approved review records.4 5Source pack:6 7- Source index: `INDEX-18-SOURCE-08`8- P&ID/loop schedule: `PID-18-LOOP-08`9- Linked process/electrical tables: `LINK-18-TABLE-08`10- Verification cases: `CASE-18-VERIFY-08`11- Gap register: `GAP-18-REGISTER-08`12- Extension memo: `MEMO-18-EXT-08`13 14Given values:15 16| Field | Value |17| --- | ---: |18| Source items traced/total | {{ source_items_traced }} / {{ source_items_total }} |19| Linked tables updated/total | {{ linked_tables_updated }} / {{ linked_tables_total }} |20| Documented/expected gaps | {{ documented_gap_count }} / {{ expected_gap_count }} |21| Verification cases passed/total | {{ verification_cases_passed }} / {{ verification_cases_total }} |22| Process/electrical margins | {{ process_margin }} / {{ electrical_margin }} |23| Authority partitions signed/total | {{ authority_partitions_signed }} / {{ authority_partitions_total }} |24| Unresolved conflicts | {{ unresolved_conflict_count }} |25| Extension memo completeness | {{ extension_memo_completeness_fraction }} |26 27Required calculations:28 29- Fractions equal completed counts divided by total counts.30- Minimum cross-domain margin equals the lesser of process and electrical margins.31- Overall pass requires complete traceability, updates, gap documentation, verification, authority partitioning, nonnegative cross-domain margin, and zero unresolved conflicts.32 33Return one JSON object with keys:34 35```json36{37  "source_traceability_fraction": <numeric_value>,38  "linked_table_update_fraction": <numeric_value>,39  "gap_documentation_fraction": <numeric_value>,40  "verification_case_pass_fraction": <numeric_value>,41  "min_cross_domain_margin": <numeric_value>,42  "authority_partition_fraction": <numeric_value>,43  "unresolved_conflict_count": <numeric_value>,44  "extension_memo_completeness_fraction": <numeric_value>,45  "overall_pass_score": <numeric_value>46}47```48 49Do not claim authority approval, accepted project evidence, executable real source-pack parsing, PLC export validation, full standards compliance, generated benchmark readiness, or benchmark readiness.50

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc18 instrumentation source policy case

ssc18_instrumentation_source_policy_case

Instrumentation source-policy and thin-substrate extension source pack

instrumentation-source-policythin-substrate-extension

Parameter ranges

source_items_traced: 7
source_items_total: 7
linked_tables_updated: 5
linked_tables_total: 5
documented_gap_count: 3
expected_gap_count: 3
verification_cases_passed: 8
verification_cases_total: 8
process_margin: 1.4
electrical_margin: 0.8
authority_partitions_signed: 4
authority_partitions_total: 4
unresolved_conflict_count: 0
extension_memo_completeness_fraction: 0.95

Example task

instrumentation-source-policy-ssc18-instrumentation-source-policy-case-preview — hard difficulty, all inputs given.

Instrumentation source-policy and thin-substrate extension source pack. instrumentation-source-policy. Required outputs: source_traceability_fraction, linked_table_update_fraction, gap_documentation_fraction, verification_case_pass_fraction, min_cross_domain_margin, authority_partition_fraction

The model sees

Scenario context and visible inputs.

source_items_traced: 7 to 7
source_items_total: 7 to 7
linked_tables_updated: 5 to 5
linked_tables_total: 5 to 5
documented_gap_count: 3 to 3
expected_gap_count: 3 to 3
verification_cases_passed: 8 to 8
verification_cases_total: 8 to 8
process_margin: 1.4 to 1.4
electrical_margin: 0.8 to 0.8
authority_partitions_signed: 4 to 4
authority_partitions_total: 4 to 4
unresolved_conflict_count: 0 to 0
extension_memo_completeness_fraction: 0.95 to 0.95

Executable tool: instrumentation-source-policy-extension-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "source_traceability_fraction": <number>,
  "linked_table_update_fraction": <number>,
  "gap_documentation_fraction": <number>,
  "verification_case_pass_fraction": <number>,
  "min_cross_domain_margin": <number>,
  "authority_partition_fraction": <number>,
  "unresolved_conflict_count": <number>,
  "extension_memo_completeness_fraction": <number>,
  "overall_pass_score": <number>
}

source_traceability_fraction · scored within ±3%
linked_table_update_fraction · scored within ±3%
gap_documentation_fraction · scored within ±3%
verification_case_pass_fraction · scored within ±3%
min_cross_domain_margin · scored within ±3%
authority_partition_fraction · scored within ±3%
unresolved_conflict_count · scored within ±3%
extension_memo_completeness_fraction · scored within ±3%
overall_pass_score · scored within ±3%