LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
civilwith-tool

Swmm Hec Report Source Policy Package

Calculates source-bound model/report source-policy metrics from a deterministic stormwater task-owned source pack. The template combines model/report object matching, hash completeness, manual/model peak delta, continuity error margin, outlet row counts, negative-case capture, unresolved source conflicts, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

EPA SWMM report workflowsHEC-style model report reviewnegative-case verificationsource manifest hashing

Tags

civildeterministichecreport-outputsource-policystormwaterswmm

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

18 inputs

Always given

Included directly in every task prompt.

18
Show 18 inputs

  • Model subcatchment

    model_subcatchment_count

    MODEL-SSC03-008 model subcatchment count

    14 count

  • Report subcatchment

    report_subcatchment_count

    REPORT-SSC03-008 report subcatchment count

    14 count

  • Model node

    model_node_count

    MODEL-SSC03-008 model node count

    28 count

  • Report node

    report_node_count

    REPORT-SSC03-008 report node count

    28 count

  • Model link

    model_link_count

    MODEL-SSC03-008 model link count

    31 count

  • Report link

    report_link_count

    REPORT-SSC03-008 report link count

    31 count

  • Storage unit

    storage_unit_count

    MODEL-SSC03-008 storage unit count

    2 count

  • Outlet row

    outlet_row_count

    RESULT-SSC03-008 outlet result row count

    5 count

  • Required hash

    required_hash_count

    HASH-SSC03-008 required hash count

    6 count

  • Present hash

    present_hash_count

    HASH-SSC03-008 present hash count

    6 count

  • Manual peak flow

    manual_peak_flow_m3_s

    REPORT-SSC03-008 manual peak flow

    1.84 m3/s

  • Model peak flow

    model_peak_flow_m3_s

    RESULT-SSC03-008 model peak flow

    1.812 m3/s

  • Allowed peak delta

    allowed_peak_delta_m3_s

    MEMO-SSC03-008 allowed manual/model peak delta

    0.05 m3/s

  • Continuity error percent

    continuity_error_percent

    REPORT-SSC03-008 continuity error

    0.37 %

  • Maximum continuity error percent

    maximum_continuity_error_percent

    MEMO-SSC03-008 maximum continuity error

    1 %

  • Expected negative cases

    expected_negative_cases

    VERIFY-SSC03-008 expected negative cases

    7 count

  • Captured negative cases

    captured_negative_cases

    VERIFY-SSC03-008 captured negative cases

    7 count

  • Unresolved source conflicts

    unresolved_source_conflicts

    MEMO-SSC03-008 unresolved source conflicts

    0 count

Scored outputs

11 outputs

Object match

object_match_percent

Model/report object match percentage

Scores if within ±3% of the reference value.

Hash completeness

hash_completeness_percent

Present hashes divided by required hashes

Scores if within ±3% of the reference value.

Peak delta

peak_delta_m3_s

Absolute manual/model peak flow delta

Scores if within ±3% of the reference value.

Peak delta margin

peak_delta_margin_m3_s

Allowed peak delta minus observed peak delta

Scores if within ±3% of the reference value.

Continuity error

continuity_error_percent

Reported continuity error

Scores if within ±3% of the reference value.

Continuity margin

continuity_margin_percent

Maximum continuity error minus reported continuity error

Scores if within ±3% of the reference value.

Storage unit count

storage_unit_count

Model storage unit count

Scores if within ±3% of the reference value.

Outlet row count

outlet_row_count

Outlet result row count

Scores if within ±3% of the reference value.

Negative case capture

negative_case_capture_percent

Captured negative cases divided by expected negative cases

Scores if within ±3% of the reference value.

Unresolved source conflicts

unresolved_source_conflicts

Unresolved source conflict count

Scores if within ±3% of the reference value.

Overall pass score

overall_pass_score

Pass score for model/report source-policy checks

Scores if within ±3% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the stormwater source-policy package
medium:
All source-pack values given for the stormwater source-policy package
hard:
All source-pack values given for the stormwater source-policy package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • swmm-hec-report-source-policy-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a civil stormwater modeller checking a task-owned synthetic stormwater SWMM/HEC-style report output and source-policy package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. EPA SWMM report workflows, HEC-style model report review, source manifest hashing, and negative-case verification shape the workflow context only; they are not extra data sources for this instance.4 5## Scene6 7- Product family: `SSC-03-LH-08`8- Model input file: `MODEL-SSC03-008`9- Manual/report PDF: `REPORT-SSC03-008`10- Result table: `RESULT-SSC03-008`11- Hash/source manifest: `HASH-SSC03-008`12- Verification case matrix: `VERIFY-SSC03-008`13- Source-policy memo: `MEMO-SSC03-008`14 15## Source Values16 17| Item | Value |18|------|-------|19| Model subcatchments | {{ model_subcatchment_count }} |20| Report subcatchments | {{ report_subcatchment_count }} |21| Model nodes | {{ model_node_count }} |22| Report nodes | {{ report_node_count }} |23| Model links | {{ model_link_count }} |24| Report links | {{ report_link_count }} |25| Storage units | {{ storage_unit_count }} |26| Outlet result rows | {{ outlet_row_count }} |27| Required hashes | {{ required_hash_count }} |28| Present hashes | {{ present_hash_count }} |29| Manual peak flow | {{ manual_peak_flow_m3_s }} m3/s |30| Model peak flow | {{ model_peak_flow_m3_s }} m3/s |31| Allowed peak delta | {{ allowed_peak_delta_m3_s }} m3/s |32| Continuity error | {{ continuity_error_percent }} percent |33| Maximum continuity error | {{ maximum_continuity_error_percent }} percent |34| Expected negative cases | {{ expected_negative_cases }} |35| Captured negative cases | {{ captured_negative_cases }} |36| Unresolved source conflicts | {{ unresolved_source_conflicts }} |37 38## Checks39 40- Object match percent compares model and report subcatchment, node, and link counts.41- Hash completeness percent equals present hashes divided by required hashes.42- Peak delta equals absolute manual/model peak-flow difference.43- Overall pass score is `1.0` only when object, hash, peak-delta, continuity, negative-case, and source-conflict checks pass; otherwise it is `0.0`.44 45## Output Format46 47Write a compact memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the object IDs above and state whether the baseline source pack passes the current docs-only checks.48 49Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated SWMM report-output evidence, generated benchmark readiness, or benchmark readiness.50 51Include a fenced JSON block with exactly these numeric keys:52 53```json54{55 "object_match_percent": <numeric_value>,56 "hash_completeness_percent": <numeric_value>,57 "peak_delta_m3_s": <numeric_value>,58 "peak_delta_margin_m3_s": <numeric_value>,59 "continuity_error_percent": <numeric_value>,60 "continuity_margin_percent": <numeric_value>,61 "storage_unit_count": <numeric_value>,62 "outlet_row_count": <numeric_value>,63 "negative_case_capture_percent": <numeric_value>,64 "unresolved_source_conflicts": <numeric_value>,65 "overall_pass_score": <numeric_value>66}67```68

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Stormwater model source policy case

stormwater_model_source_policy_case

SWMM/HEC-style report output and source-policy case

swmm-hec-reportsource-policy-check
Parameter ranges
model_subcatchment_count
14
report_subcatchment_count
14
model_node_count
28
report_node_count
28
model_link_count
31
report_link_count
31
storage_unit_count
2
outlet_row_count
5
required_hash_count
6
present_hash_count
6
manual_peak_flow_m3_s
1.84
model_peak_flow_m3_s
1.812
allowed_peak_delta_m3_s
0.05
continuity_error_percent
0.37
maximum_continuity_error_percent
1
expected_negative_cases
7
captured_negative_cases
7
unresolved_source_conflicts
0

Example task

swmm-hec-report-stormwater-model-source-policy-case-preview — hard difficulty, all inputs given.

SWMM/HEC-style report output and source-policy case. swmm-hec-report. Required outputs: object_match_percent, hash_completeness_percent, peak_delta_m3_s, peak_delta_margin_m3_s, continuity_error_percent, continuity_margin_percent

The model sees

Scenario context and visible inputs.

model_subcatchment_count
14 to 14 count
report_subcatchment_count
14 to 14 count
model_node_count
28 to 28 count
report_node_count
28 to 28 count
model_link_count
31 to 31 count
report_link_count
31 to 31 count
storage_unit_count
2 to 2 count
outlet_row_count
5 to 5 count
required_hash_count
6 to 6 count
present_hash_count
6 to 6 count
manual_peak_flow_m3_s
1.84 to 1.84 m3/s
model_peak_flow_m3_s
1.812 to 1.812 m3/s
allowed_peak_delta_m3_s
0.05 to 0.05 m3/s
continuity_error_percent
0.37 to 0.37 %
maximum_continuity_error_percent
1 to 1 %
expected_negative_cases
7 to 7 count
captured_negative_cases
7 to 7 count
unresolved_source_conflicts
0 to 0 count

Executable tool: swmm-hec-report-source-policy-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "object_match_percent": <number>,
  "hash_completeness_percent": <number>,
  "peak_delta_m3_s": <number>,
  "peak_delta_margin_m3_s": <number>,
  "continuity_error_percent": <number>,
  "continuity_margin_percent": <number>,
  "storage_unit_count": <number>,
  "outlet_row_count": <number>,
  "negative_case_capture_percent": <number>,
  "unresolved_source_conflicts": <number>,
  "overall_pass_score": <number>
}
  • object_match_percent · scored within ±3%
  • hash_completeness_percent · scored within ±3%
  • peak_delta_m3_s · scored within ±3%
  • peak_delta_margin_m3_s · scored within ±3%
  • continuity_error_percent · scored within ±3%
  • continuity_margin_percent · scored within ±3%
  • storage_unit_count · scored within ±3%
  • outlet_row_count · scored within ±3%
  • negative_case_capture_percent · scored within ±3%
  • unresolved_source_conflicts · scored within ±3%
  • overall_pass_score · scored within ±3%