LIVEdataset aec-bench@releasetasks 552models 18disciplines 5last submission … · built …

mechanicalwith-tool

Building Operations Scenario Repair Package

Calculates source-bound building operations review metrics from a deterministic SSC-08 task-owned source pack. The template combines source trace, occupancy update, affected system checks, comment closure, authority partition, repair action closure, unsupported-value count, evidence-boundary score, and an overall synthetic pass score.

with-tool: The model is given an executable Python calculator script.

Standards

authority partition review building operations review workflow comment response workflow occupancy scenario repair

How this task is generated

One template produces many comparable benchmark tasks while keeping the scoring contract fixed.

01
Template
The reusable contract shown on this page.
02
Scenario
An archetype and site context are sampled.
03
Difficulty tier
Inputs may be hidden at harder tiers.
04
Task prompt
The model responds with the declared outputs.

Parameters

Inputs the model receives, and the outputs it is scored on.

Inputs

14 inputs

Always given

Included directly in every task prompt.

Show 14 inputs

Source items traced
source_items_traced
PLAN-08-FLOOR-08 source items traced
10
Required source items
required_source_items
PLAN-08-FLOOR-08 required source items
10
Occupancy rows updated
occupancy_rows_updated
OCC-08-SOURCE-08 occupancy rows updated
8
Required occupancy rows
required_occupancy_rows
OCC-08-SOURCE-08 required occupancy rows
8
Affected system checks complete
affected_system_checks_complete
SYSTEM-08-SCHED-08 affected system checks complete
11
Required system checks
required_system_checks
SYSTEM-08-SCHED-08 required system checks
12
Resolved comments
resolved_comments
COMMENT-08-REG-08 resolved comments
9
Review comments
review_comments
COMMENT-08-REG-08 review comments
10
Open critical comment count
open_critical_comment_count
COMMENT-08-REG-08 open critical comment count
0
Partitioned authority roles
partitioned_authority_roles
CRIT-08-MATRIX-08 partitioned authority roles
5
Required authority roles
required_authority_roles
CRIT-08-MATRIX-08 required authority roles
5
Repair actions closed
repair_actions_closed
MEMO-08-REPAIR-08 repair actions closed
7
Required repair actions
required_repair_actions
MEMO-08-REPAIR-08 required repair actions
8
Unsupported value count
unsupported_value_count
MEMO-08-REPAIR-08 unsupported value count
0

Scored outputs

10 outputs

Source trace score

source_trace_score

Source items traced divided by required source items

Scores if within ±0.3% of the reference value.

Occupancy update fraction

occupancy_update_fraction

Updated occupancy rows divided by required occupancy rows

Scores if within ±0.3% of the reference value.

Affected system check fraction

affected_system_check_fraction

Affected system checks complete divided by required system checks

Scores if within ±0.3% of the reference value.

Comment resolution fraction

comment_resolution_fraction

Resolved comments divided by review comments

Scores if within ±0.3% of the reference value.

Open critical comment count

open_critical_comment_count

Open critical comment count

Scores if within ±0.1% of the reference value.

Authority partition score

authority_partition_score

Partitioned authority roles divided by required authority roles

Scores if within ±0.3% of the reference value.

Repair action closure fraction

repair_action_closure_fraction

Closed repair actions divided by required repair actions

Scores if within ±0.3% of the reference value.

Unsupported value count

unsupported_value_count

Unsupported value count

Scores if within ±0.1% of the reference value.

Evidence boundary score

evidence_boundary_score

Average evidence-boundary score

Scores if within ±0.3% of the reference value.

Overall pass score

overall_pass_score

1.0 when operations repair checks pass

Scores if within ±0.1% of the reference value.

Difficulty

Each template is sampled at three tiers. Harder tiers may hide inputs, forcing the model to infer them from the scenario description.

All inputs remain visible at every tier

For this template, difficulty scales through parameter and scenario ranges rather than hidden information.

easy:: All source-pack values given for the SSC-08 operations repair package
medium:: All source-pack values given for the SSC-08 operations repair package
hard:: All source-pack values given for the SSC-08 operations repair package

Task bundle

The exact instruction and parameter contract used to generate this task, pinned to the published library source.

/workspace

instruction.md
building-operations-scenario-repair-package_calc.py

Teal lines show Jinja input conditions, not task visibility policy. A line renders only when that input or tool is visible.

1You are a building services reviewer checking a task-owned synthetic SSC-08 building operations review and scenario repair package.2 3Use only the task-owned synthetic source pack values shown below for numeric grading. External room/floor plan, occupancy change, system-schedule, criteria-matrix, review-comment, and repair-ledger workflows shape the practice context only; they are not extra data sources for this instance.4 5## Scene6 7- Product family: `SSC-08-LH-08`8- Room/floor plan: `PLAN-08-FLOOR-08`9- Occupancy source table: `OCC-08-SOURCE-08`10- System schedules: `SYSTEM-08-SCHED-08`11- Criteria matrix: `CRIT-08-MATRIX-08`12- Comment register: `COMMENT-08-REG-08`13- Operations repair memo: `MEMO-08-REPAIR-08`14 15## Source Values16 17| Item | Value |18|------|-------|19| Source items traced | {{ source_items_traced }} |20| Required source items | {{ required_source_items }} |21| Occupancy rows updated | {{ occupancy_rows_updated }} |22| Required occupancy rows | {{ required_occupancy_rows }} |23| Affected system checks complete | {{ affected_system_checks_complete }} |24| Required system checks | {{ required_system_checks }} |25| Resolved comments | {{ resolved_comments }} |26| Review comments | {{ review_comments }} |27| Open critical comment count | {{ open_critical_comment_count }} |28| Partitioned authority roles | {{ partitioned_authority_roles }} |29| Required authority roles | {{ required_authority_roles }} |30| Repair actions closed | {{ repair_actions_closed }} |31| Required repair actions | {{ required_repair_actions }} |32| Unsupported value count | {{ unsupported_value_count }} |33 34## Checks35 36- Source trace score equals source items traced divided by required source items.37- Occupancy update fraction equals updated occupancy rows divided by required occupancy rows.38- Affected system check fraction equals completed affected system checks divided by required system checks.39- Comment resolution fraction equals resolved comments divided by review comments.40- Authority partition score equals partitioned authority roles divided by required authority roles.41- Repair action closure fraction equals repair actions closed divided by required repair actions.42- Evidence boundary score is the average of source trace, occupancy update, affected-system, comment, authority, and repair fractions.43- Overall pass score is `1.0` only when source trace, occupancy update, authority, comment, affected-system, critical-comment, and unsupported-value checks pass; otherwise it is `0.0`.44 45## Output Format46 47Write a compact operations response memo to `/workspace/output.md`. Include a source-boundary statement that this is a task-owned synthetic source pack. Preserve the object IDs above and state whether the baseline source pack passes the current docs-only checks.48 49Do not claim authority approval, accepted project evidence, software export validity, full standards compliance, executable real source-pack parsing, generated benchmark readiness, or benchmark readiness.50 51Include a fenced JSON block with exactly these numeric keys:52 53```json54{55  "source_trace_score": <numeric_value>,56  "occupancy_update_fraction": <numeric_value>,57  "affected_system_check_fraction": <numeric_value>,58  "comment_resolution_fraction": <numeric_value>,59  "open_critical_comment_count": <numeric_value>,60  "authority_partition_score": <numeric_value>,61  "repair_action_closure_fraction": <numeric_value>,62  "unsupported_value_count": <numeric_value>,63  "evidence_boundary_score": <numeric_value>,64  "overall_pass_score": <numeric_value>65}66```67

Scenario archetypes

Each generated task is drawn from one of these realistic scenario bands.

Site contexts ground each scenario in a real locale the model can use to infer hidden values.

Ssc08 operations repair case

ssc08_operations_repair_case

Building operations review and scenario repair case

ssc08-operations-repair

Parameter ranges

Example task

ssc08-operations-repair-ssc08-operations-repair-case-preview — hard difficulty, all inputs given.

Building operations review and scenario repair case. ssc08-operations-repair. Required outputs: source_trace_score, occupancy_update_fraction, affected_system_check_fraction, comment_resolution_fraction, open_critical_comment_count, authority_partition_score

The model sees

Scenario context and visible inputs.

source_items_traced: 10 to 10
required_source_items: 10 to 10
occupancy_rows_updated: 8 to 8
required_occupancy_rows: 8 to 8
affected_system_checks_complete: 11 to 11
required_system_checks: 12 to 12
resolved_comments: 9 to 9
review_comments: 10 to 10
open_critical_comment_count: 0 to 0
partitioned_authority_roles: 5 to 5
required_authority_roles: 5 to 5
repair_actions_closed: 7 to 7
required_repair_actions: 8 to 8
unsupported_value_count: 0 to 0

Executable tool: building-operations-scenario-repair-package_calc.py

The model must infer

Inputs withheld at this difficulty.

Nothing. All inputs are supplied.

The model must produce

The scored JSON answer schema.

{
  "source_trace_score": <number>,
  "occupancy_update_fraction": <number>,
  "affected_system_check_fraction": <number>,
  "comment_resolution_fraction": <number>,
  "open_critical_comment_count": <number>,
  "authority_partition_score": <number>,
  "repair_action_closure_fraction": <number>,
  "unsupported_value_count": <number>,
  "evidence_boundary_score": <number>,
  "overall_pass_score": <number>
}

source_trace_score · scored within ±0.3%
occupancy_update_fraction · scored within ±0.3%
affected_system_check_fraction · scored within ±0.3%
comment_resolution_fraction · scored within ±0.3%
open_critical_comment_count · scored within ±0.1%
authority_partition_score · scored within ±0.3%
repair_action_closure_fraction · scored within ±0.3%
unsupported_value_count · scored within ±0.1%
evidence_boundary_score · scored within ±0.3%
overall_pass_score · scored within ±0.1%

Building Operations Scenario Repair Package

How this task is generated

Template

Scenario

Difficulty tier

Task prompt

Parameters

Inputs

Scored outputs

Source trace score

Occupancy update fraction

Affected system check fraction

Comment resolution fraction

Open critical comment count

Authority partition score

Repair action closure fraction

Unsupported value count

Evidence boundary score

Overall pass score

Difficulty

All inputs remain visible at every tier

Task bundle

Scenario archetypes

Ssc08 operations repair case

Example task

The model sees

The model must infer

The model must produce