Classification

The bond taxonomy

Classification describes how an agent worked, not whether its answer was correct. Use it to compare patterns such as planning, tool use, checking, and exploration across traces with different rewards.

Each assistant turn in a trajectory can be tagged with one of four bond types. The chemistry language is an analogy; the stored values are lowercase strings:

High-reward traces are used to build reference transition patterns and, where useful, an ideal bond sequence. The common shape is deliberation -> execution -> verification, but the selected traces can define the reference directly.

Turn classification

The LLMTurnClassifier reads assistant turns in batches and emits a TurnClassification for each one:

@dataclass(frozen=True)
class TurnClassification:
    turn_index: int
    bond_type: BondType           # execution | verification | deliberation | exploration
    confidence: float             # 0.0–1.0
    rationale: str = ""           # why this bond was assigned

The loader prefers structured trajectory.jsonl and falls back to conversation.jsonl when needed. Each classification carries its own confidence so low-certainty tags can be inspected before they influence aggregates.

Structural scoring

A run is characterised by its individual turn tags and their sequence. Two metrics compare a trajectory against a reference built from high-reward trials:

• Transition matrix similarity: cosine similarity between the observed transition matrix and the reference matrix. Captures which bonds tend to follow which.
• Sequence edit distance: how many insertions, deletions, and substitutions separate the observed bond sequence from the reference sequence.

A run can have plausible bond counts but the wrong order, such as verifying before executing. The structural score catches that case better than a tag histogram.

Confidence metadata

Before a classification is trusted for aggregation, its statistical footing gets recorded:

class ConfidenceMetadata(StrictModel):
    annotator_count: int | None = None
    inter_rater_agreement: float | None = None
    confidence_interval: tuple[float, float] | None = None
    confidence_method: str | None = None

Two standard statistical tools are useful when classifications are being calibrated against human labels:

• Cohen's kappa measures inter-rater agreement above chance. Values above 0.8 indicate strong agreement; values below 0.6 require recalibration before use.
• Wilson confidence interval gives a binomial interval for a success rate. It behaves better than the normal approximation on small samples.

Judgment readiness

Automated behavioural judgment should only affect aggregate analysis when the calibration evidence is good enough:

If any gate fails, keep the classification as analysis evidence. Reserve aggregate claims for classifications that pass every gate.