LIVEdataset aec-bench@releasetasks 552models 18last submission … · built …
mechanicalwith-tool

Fire Review Response Evidence Boundary Package

Calculates source-bound fire review and evidence-boundary metrics from a deterministic SSC-19 task-owned source pack. The template combines source traceability, comment resolution, affected-check updates, unresolved gaps, authority role separation, conflict resolution, response completeness, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

authority response workflowsfire review workflowssource evidence workflows

Tags

authoritydeterministicevidencefiremechanicalreviewsource-boundary

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

  1. 01

    Template

    The reusable contract shown on this page.

  2. 02

    Scenario

    An archetype and site context are sampled.

  3. 03

    Difficulty tier

    Inputs may be hidden at harder tiers.

  4. 04

    Task prompt

    The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

15 inputs

Always given

Included directly in every task prompt.

15
Show 15 inputs

  • Source items traced

    source_items_traced

    SOURCE-19-INDEX-08 traced source items

    18

  • Required source items

    required_source_items

    SOURCE-19-INDEX-08 required source items

    20

  • Review comments

    review_comments

    COMMENT-19-REVIEW-08 total review comments

    7

  • Resolved comments

    resolved_comments

    COMMENT-19-REVIEW-08 resolved comments

    6

  • Affected checks

    affected_checks

    CALC-19-EXTRACT-08 affected checks

    5

  • Updated checks

    updated_checks

    CALC-19-EXTRACT-08 updated checks

    5

  • Unresolved gaps

    unresolved_gaps

    HAZ-19-TABLE-08 unresolved gaps

    2

  • Allowed gaps

    allowed_gaps

    AUTH-19-SOURCE-08 allowed gaps

    3

  • Authority roles

    authority_roles

    AUTH-19-SOURCE-08 authority role count

    4

  • Separated authority roles

    separated_authority_roles

    AUTH-19-SOURCE-08 separated authority roles

    4

  • Evidence conflicts

    evidence_conflicts

    SOURCE-19-INDEX-08 evidence conflicts

    3

  • Resolved conflicts

    resolved_conflicts

    RESPONSE-19-MEMO-08 resolved conflicts

    2

  • Critical open comments

    critical_open_comments

    COMMENT-19-REVIEW-08 critical open comments

    0

  • Response sections

    response_sections

    RESPONSE-19-MEMO-08 completed response sections

    11

  • Required response sections

    required_response_sections

    RESPONSE-19-MEMO-08 required response sections

    12

Scored outputs

11 outputs

Source trace score

source_trace_score

Traced source items divided by required source items

Scores if within ±3% of the reference value.

Comment resolution fraction

comment_resolution_fraction

Resolved comments divided by review comments

Scores if within ±3% of the reference value.

Affected check update fraction

affected_check_update_fraction

Updated checks divided by affected checks

Scores if within ±3% of the reference value.

Unresolved gap count

unresolved_gap_count

Unresolved evidence gaps

Scores if within ±0.1% of the reference value.

Allowed gap margin

allowed_gap_margin

Allowed gaps minus unresolved gaps

Scores if within ±3% of the reference value.

Authority role separation score

authority_role_separation_score

Separated authority roles divided by authority roles

Scores if within ±3% of the reference value.

Conflict resolution fraction

conflict_resolution_fraction

Resolved conflicts divided by evidence conflicts

Scores if within ±3% of the reference value.

Response completeness score

response_completeness_score

Completed response sections divided by required response sections

Scores if within ±3% of the reference value.

Review boundary score

review_boundary_score

Mean of source, comment, check, role, conflict, and response scores

Scores if within ±3% of the reference value.

Critical open comments

critical_open_comments

Critical open comments

Scores if within ±0.1% of the reference value.

Overall pass score

overall_pass_score

1.0 when gap, check-update, role-separation, and critical-comment checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:
All source-pack values given for the SSC-19 fire review response package
medium:
All source-pack values given for the SSC-19 fire review response package
hard:
All source-pack values given for the SSC-19 fire review response package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

  • instruction.md
  • fire-review-response-evidence-boundary-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a fire protection reviewer checking a task-owned synthetic SSC-19 fire review response and evidence-boundary package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. Fire review response, source-index checking, authority-role separation, and evidence-conflict workflows shape the context only; this instance does not validate a real authority decision, accepted project evidence, or source-pack parser.4 5## Scene6 7- Product: `SSC-19-LH-08`8- Source index: `SOURCE-19-INDEX-08`9- Review comment register: `COMMENT-19-REVIEW-08`10- Hazard table: `HAZ-19-TABLE-08`11- Calculation extract: `CALC-19-EXTRACT-08`12- Authority source matrix: `AUTH-19-SOURCE-08`13- Response memo: `RESPONSE-19-MEMO-08`14 15## Source Values16 17| Item | Value |18| --- | --- |19| Source items traced | {{ source_items_traced }} |20| Required source items | {{ required_source_items }} |21| Review comments | {{ review_comments }} |22| Resolved comments | {{ resolved_comments }} |23| Affected checks | {{ affected_checks }} |24| Updated checks | {{ updated_checks }} |25| Unresolved gaps | {{ unresolved_gaps }} |26| Allowed gaps | {{ allowed_gaps }} |27| Authority roles | {{ authority_roles }} |28| Separated authority roles | {{ separated_authority_roles }} |29| Evidence conflicts | {{ evidence_conflicts }} |30| Resolved conflicts | {{ resolved_conflicts }} |31| Critical open comments | {{ critical_open_comments }} |32| Response sections | {{ response_sections }} |33| Required response sections | {{ required_response_sections }} |34 35## Checks36 37- Source trace score equals traced source items divided by required source items.38- Comment resolution fraction equals resolved comments divided by review comments.39- Affected check update fraction equals updated checks divided by affected checks.40- Allowed gap margin equals allowed gaps minus unresolved gaps.41- Authority role separation score equals separated roles divided by authority roles.42- Conflict resolution fraction equals resolved conflicts divided by evidence conflicts.43- Response completeness score equals response sections divided by required response sections.44- Review boundary score averages source, comment, check, authority-role, conflict, and response scores.45- Overall pass score is `1.0` only when gap margin, check updates, role separation, and critical-comment checks pass; otherwise it is `0.0`.46 47## Output Format48 49Write a compact memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the source IDs above and state whether the baseline source pack passes the current docs-only checks.50 51Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, source-pack hardening, generated benchmark readiness, or benchmark readiness.52 53Include a fenced JSON block with exactly these numeric keys:54 55```json56{57 "source_trace_score": <numeric_value>,58 "comment_resolution_fraction": <numeric_value>,59 "affected_check_update_fraction": <numeric_value>,60 "unresolved_gap_count": <numeric_value>,61 "allowed_gap_margin": <numeric_value>,62 "authority_role_separation_score": <numeric_value>,63 "conflict_resolution_fraction": <numeric_value>,64 "response_completeness_score": <numeric_value>,65 "review_boundary_score": <numeric_value>,66 "critical_open_comments": <numeric_value>,67 "overall_pass_score": <numeric_value>68}69```70

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc19 fire review boundary case

ssc19_fire_review_boundary_case

Fire review response case with source traceability, gaps, conflicts, and authority boundaries

Parameter ranges
source_items_traced
18
required_source_items
20
review_comments
7
resolved_comments
6
affected_checks
5
updated_checks
5
unresolved_gaps
2
allowed_gaps
3
authority_roles
4
separated_authority_roles
4
evidence_conflicts
3
resolved_conflicts
2
critical_open_comments
0
response_sections
11
required_response_sections
12

Example task

ssc19-fire-review-boundary-case-preview — hard difficulty, all inputs given.

Fire review response case with source traceability, gaps, conflicts, and authority boundaries. Required outputs: source_trace_score, comment_resolution_fraction, affected_check_update_fraction, unresolved_gap_count, allowed_gap_margin, authority_role_separation_score

The model sees

Scenario context and visible inputs.

source_items_traced
18 to 18
required_source_items
20 to 20
review_comments
7 to 7
resolved_comments
6 to 6
affected_checks
5 to 5
updated_checks
5 to 5
unresolved_gaps
2 to 2
allowed_gaps
3 to 3
authority_roles
4 to 4
separated_authority_roles
4 to 4
evidence_conflicts
3 to 3
resolved_conflicts
2 to 2
critical_open_comments
0 to 0
response_sections
11 to 11
required_response_sections
12 to 12

Executable tool: fire-review-response-evidence-boundary-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "source_trace_score": <number>,
  "comment_resolution_fraction": <number>,
  "affected_check_update_fraction": <number>,
  "unresolved_gap_count": <number>,
  "allowed_gap_margin": <number>,
  "authority_role_separation_score": <number>,
  "conflict_resolution_fraction": <number>,
  "response_completeness_score": <number>,
  "review_boundary_score": <number>,
  "critical_open_comments": <number>,
  "overall_pass_score": <number>
}
  • source_trace_score · scored within ±3%
  • comment_resolution_fraction · scored within ±3%
  • affected_check_update_fraction · scored within ±3%
  • unresolved_gap_count · scored within ±0.1%
  • allowed_gap_margin · scored within ±3%
  • authority_role_separation_score · scored within ±3%
  • conflict_resolution_fraction · scored within ±3%
  • response_completeness_score · scored within ±3%
  • review_boundary_score · scored within ±3%
  • critical_open_comments · scored within ±0.1%
  • overall_pass_score · scored within ±0.1%