Trust is a measured rate
Review time may fall only while escapes stay flat. Each metric has a guardrail, so the factory cannot optimize for speed alone.
What we measure
Escape rateDo more defects reach main as people read less?Reverts and incidents per 100 merges, per area, over 30 daysGuardrail · Must not go up while review time goes down.
Finding acceptanceAre the AI's findings real?Findings that the owner accepted divided by findings raisedGuardrail · Below 70% means the reviewer is noisy, so cut rules.
Spot-check miss rateWhat does the AI pass that it should not?Spot-checked verified claims that a person rejectedGuardrail · Must be above 0. A reviewer that never misses is not being checked.
Share routed to humansHow much human attention does a change cost?Share of changed lines marked 'read'Guardrail · Should go down over time, never to 0.
Time to mergeIs it faster?Median time from the PR opening to a signed receiptGuardrail · Track it, but do not optimize it alone.
Refusal outcomesDoes the AI refuse the right things?Refusals per change, and how the owner decidedGuardrail · A refusal that people always accept unchanged is a candidate for a rule.
Trust per area
An area's track record sets the default attention for each hunk. One escape moves it back to full reads. Sample values, not real history.
| Area | Default | Record | Spot checks |
|---|---|---|---|
| SourceGenerator / diagnostics | Human reads | 1 escape in 30 days | 100% |
| SourceGenerator / code emit | Check evidence | 42 clean merges | 10% |
| Tests and docs | Skim | 120 clean merges | 5% |
| Runtime / formatters | Human reads | Security-critical | 100% |
How it scales
- Human time grows with the hunks marked Human reads, not with the size of the diff.
- Probes run only on hunks marked Human reads or Check evidence, so CI cost follows risk.
- A 237-file change (REQ-2297) splits into areas. Each area gets its own claims and can have its own named reviewer.
- A refusal that owners always decide the same way becomes a written rule.