LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
civilwith-tool

Marine Asset Source Policy Review Package

Calculates source-bound marine asset source-policy review metrics from a deterministic SSC-04 task-owned source pack. The template combines datum trace, criteria resolution, asset schedule matching, calculation trace, comment resolution, authority partition, unsupported-source count, response completeness, evidence boundary score, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

criteria matrix reviewdatum trace reviewmarine source-policy workflowreview comment response workflow

Tags

civilcriteriadatumdeterministicmarinereviewsource-policytraceability

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

15 inputs

Always given

Included directly in every task prompt.

15
Show 15 inputs

  • Datum items traced

    datum_items_traced

    DATUM-04-STATEMENT-08 traced datum items

    5

  • Required datum items

    required_datum_items

    DATUM-04-STATEMENT-08 required datum items

    5

  • Resolved criteria items

    resolved_criteria_items

    CRITERIA-04-MATRIX-08 resolved criteria items

    6

  • Criteria items

    criteria_items

    CRITERIA-04-MATRIX-08 criteria items

    6

  • Matching asset rows

    matching_asset_rows

    ASSET-04-MARINE-08 matching asset rows

    14

  • Asset schedule rows

    asset_schedule_rows

    ASSET-04-MARINE-08 asset schedule rows

    15

  • Traced calculation rows

    traced_calculation_rows

    CALC-04-APPENDIX-08 traced calculation rows

    18

  • Calculation rows

    calculation_rows

    CALC-04-APPENDIX-08 calculation rows

    20

  • Resolved comments

    resolved_comments

    COMMENT-04-REGISTER-08 resolved comments

    7

  • Review comments

    review_comments

    COMMENT-04-REGISTER-08 review comments

    8

  • Separated authority roles

    separated_authority_roles

    MEMO-04-SOURCE-08 separated authority roles

    4

  • Required authority roles

    required_authority_roles

    MEMO-04-SOURCE-08 required authority roles

    4

  • Unsupported source value count

    unsupported_source_value_count

    MEMO-04-SOURCE-08 unsupported source value count

    0

  • Response sections

    response_sections

    MEMO-04-SOURCE-08 response sections

    9

  • Required response sections

    required_response_sections

    MEMO-04-SOURCE-08 required response sections

    10

Scored outputs

10 outputs

Datum trace score

datum_trace_score

Datum items traced divided by required datum items

Scores if within ±0.3% of the reference value.

Criteria resolution fraction

criteria_resolution_fraction

Resolved criteria items divided by criteria items

Scores if within ±0.3% of the reference value.

Asset schedule match fraction

asset_schedule_match_fraction

Matching asset rows divided by asset schedule rows

Scores if within ±0.3% of the reference value.

Calculation trace fraction

calculation_trace_fraction

Traced calculation rows divided by calculation rows

Scores if within ±0.3% of the reference value.

Comment resolution fraction

comment_resolution_fraction

Resolved comments divided by review comments

Scores if within ±0.3% of the reference value.

Authority partition score

authority_partition_score

Separated authority roles divided by required authority roles

Scores if within ±0.3% of the reference value.

Unsupported source value count

unsupported_source_value_count

Unsupported source value count

Scores if within ±0.1% of the reference value.

Response completeness score

response_completeness_score

Response sections divided by required response sections

Scores if within ±0.3% of the reference value.

Evidence boundary score

evidence_boundary_score

Average evidence-boundary score

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when source-policy review checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-04 marine source-policy review package
medium:
All source-pack values given for the SSC-04 marine source-policy review package
hard:
All source-pack values given for the SSC-04 marine source-policy review package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • marine-asset-source-policy-review-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a civil/marine reviewer checking a task-owned synthetic SSC-04 marine asset source-policy and review packet.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. External datum, criteria, asset-register, calculation-appendix, comment-register, and authority-boundary workflows shape the practice context only; they are not extra data sources for this instance.4 5## Scene6 7- Product family: `SSC-04-LH-08`8- Datum statement: `DATUM-04-STATEMENT-08`9- Criteria matrix: `CRITERIA-04-MATRIX-08`10- Marine asset schedule: `ASSET-04-MARINE-08`11- Calculation appendix: `CALC-04-APPENDIX-08`12- Comment register: `COMMENT-04-REGISTER-08`13- Source-policy memo: `MEMO-04-SOURCE-08`14 15## Source Values16 17| Item | Value |18|------|-------|19| Datum items traced | {{ datum_items_traced }} |20| Required datum items | {{ required_datum_items }} |21| Resolved criteria items | {{ resolved_criteria_items }} |22| Criteria items | {{ criteria_items }} |23| Matching asset rows | {{ matching_asset_rows }} |24| Asset schedule rows | {{ asset_schedule_rows }} |25| Traced calculation rows | {{ traced_calculation_rows }} |26| Calculation rows | {{ calculation_rows }} |27| Resolved comments | {{ resolved_comments }} |28| Review comments | {{ review_comments }} |29| Separated authority roles | {{ separated_authority_roles }} |30| Required authority roles | {{ required_authority_roles }} |31| Unsupported source value count | {{ unsupported_source_value_count }} |32| Response sections | {{ response_sections }} |33| Required response sections | {{ required_response_sections }} |34 35## Checks36 37- Datum trace score equals datum items traced divided by required datum items.38- Criteria resolution fraction equals resolved criteria items divided by criteria items.39- Asset schedule match fraction equals matching asset rows divided by asset schedule rows.40- Calculation trace fraction equals traced calculation rows divided by calculation rows.41- Comment resolution fraction equals resolved comments divided by review comments.42- Authority partition score equals separated authority roles divided by required authority roles.43- Response completeness score equals response sections divided by required response sections.44- Evidence boundary score is the average of datum trace, criteria resolution, asset match, calculation trace, comment resolution, authority partition, and response completeness scores.45- Overall pass score is `1.0` only when datum, criteria, and authority partition checks are complete, unsupported source values are zero, and response completeness is at least 0.9; otherwise it is `0.0`.46 47## Output Format48 49Write a compact source-policy review memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the object IDs above and state whether the baseline source pack passes the current docs-only checks.50 51Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.52 53Include a fenced JSON block with exactly these numeric keys:54 55```json56{57 "datum_trace_score": <numeric_value>,58 "criteria_resolution_fraction": <numeric_value>,59 "asset_schedule_match_fraction": <numeric_value>,60 "calculation_trace_fraction": <numeric_value>,61 "comment_resolution_fraction": <numeric_value>,62 "authority_partition_score": <numeric_value>,63 "unsupported_source_value_count": <numeric_value>,64 "response_completeness_score": <numeric_value>,65 "evidence_boundary_score": <numeric_value>,66 "overall_pass_score": <numeric_value>67}68```69

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc04 marine source policy case

ssc04_marine_source_policy_case

Marine asset source-policy and review packet case

Parameter ranges
datum_items_traced
5
required_datum_items
5
resolved_criteria_items
6
criteria_items
6
matching_asset_rows
14
asset_schedule_rows
15
traced_calculation_rows
18
calculation_rows
20
resolved_comments
7
review_comments
8
separated_authority_roles
4
required_authority_roles
4
unsupported_source_value_count
0
response_sections
9
required_response_sections
10

Example task

ssc04-marine-source-policy-case-preview — hard difficulty, all inputs given.

Marine asset source-policy and review packet case. Required outputs: datum_trace_score, criteria_resolution_fraction, asset_schedule_match_fraction, calculation_trace_fraction, comment_resolution_fraction, authority_partition_score

The model sees

Scenario context and visible inputs.

datum_items_traced
5 to 5
required_datum_items
5 to 5
resolved_criteria_items
6 to 6
criteria_items
6 to 6
matching_asset_rows
14 to 14
asset_schedule_rows
15 to 15
traced_calculation_rows
18 to 18
calculation_rows
20 to 20
resolved_comments
7 to 7
review_comments
8 to 8
separated_authority_roles
4 to 4
required_authority_roles
4 to 4
unsupported_source_value_count
0 to 0
response_sections
9 to 9
required_response_sections
10 to 10

Executable tool: marine-asset-source-policy-review-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "datum_trace_score": <number>,
  "criteria_resolution_fraction": <number>,
  "asset_schedule_match_fraction": <number>,
  "calculation_trace_fraction": <number>,
  "comment_resolution_fraction": <number>,
  "authority_partition_score": <number>,
  "unsupported_source_value_count": <number>,
  "response_completeness_score": <number>,
  "evidence_boundary_score": <number>,
  "overall_pass_score": <number>
}
  • datum_trace_score · scored within ±0.3%
  • criteria_resolution_fraction · scored within ±0.3%
  • asset_schedule_match_fraction · scored within ±0.3%
  • calculation_trace_fraction · scored within ±0.3%
  • comment_resolution_fraction · scored within ±0.3%
  • authority_partition_score · scored within ±0.3%
  • unsupported_source_value_count · scored within ±0.1%
  • response_completeness_score · scored within ±0.3%
  • evidence_boundary_score · scored within ±0.3%
  • overall_pass_score · scored within ±0.1%