Failure modes and decisions

What would make us distrust our own output. These are the answers we found on a real agent-written change.

Failure modes

The agent bundles a product decision with the bug fix
SeenThe 2283 change adds a filter for all diagnostics that the crash fix does not need. The filter carries out a choice to stay silent about problems in referenced types.GuardMutation probe: we removed the filter and only one test cared. A reviewer that only reads would call the filter 'part of the fix'.
Tests that lock in an unowned choice
SeenThe agent wrote a test that requires no diagnostic. It passes, so it looks like evidence, but it only proves that the agent's choice is consistent.GuardRefuse claims that need knowledge outside the repo, and treat a test that requires silence as a decision, not as evidence.
Proving less than the issue asks
SeenThe issue's real harm is AOT apps that fail at runtime. The tests stop at 'the resolver file exists'. A green suite looks like it proves more than it does.GuardEach claim lists which acceptance criterion it covers, and the receipt shows criteria that no evidence covers.
All tests pass, so it must be safe
Seen170 of 170 tests pass. The probe shows that no test covers a diagnostic without a location on a type in the project.GuardThe verdict needs a probe as well as a pass, and a full green suite is listed as 'executed', not as 'safe'.
The reviewer agrees with the author
SeenA risk we expect. Two models from the same family tend to accept the same plausible story.GuardThe reviewer runs on another model family, and the 'read' kind of evidence never verifies a claim alone.
A risk score nobody can explain
SeenA risk we designed out. Each hunk shows a reason in plain words.GuardRisk factors are named rules with a reason, never one opaque score.
A rule hit taken as a bug
SeenWith the 2283 commit applied to master, the crash rule still hits TypeCollector.cs:735. Our probe could not make that line crash.GuardA rule hit is evidence of a pattern, not of a bug. We ran a probe test, it passed, and the receipt says refused, not finding.
A bug that cannot be traced to its cause
SeenNone of the four sample commits says which session or model wrote it. We could fix the lines, but only guess why the agent wrote them.GuardProvenance gate: every agent commit carries Agent-Session, Agent-Model, Agent-Tools, Agent-Context-Digest and Agent-Verified.
A rule that holds the shape, not the bug
SeenA fresh agent on the 2234 task wrote the same silent drop as if/else. The syntax rule passed it. The generator test caught it.GuardFence a bug class with a behaviour test or a type when its shape can vary. Use a syntax rule only for a fixed shape.

Decision log

  1. Fri 22:35Use ju26-issue-2283 as the reference change.It is small (4 files), the bug is real and has an upstream issue, and it has a subtle design choice. 2297 (237 files) is a better test of scale, and we plan it next.
  2. Fri 22:38Evidence comes from running .NET tests, not from an LLM reading the diff.An LLM saying that a test covers the bug is advice. A test that fails with the fix reverted is proof.
  3. Fri 22:41Add mutation probes after the red-green run.Red-green proves that the fix works. It does not show which parts of the diff are needed. The probes showed that the filter is not.
  4. Fri 22:44Classify C5 as refused, not as a finding.Staying silent may be the right product choice. That is not for the AI to decide. It names the choice and asks.
  5. Fri 22:46No numeric confidence score.The judges' brief rejects scores that nobody can explain. A kind of evidence can be checked.
  6. Fri 22:48We use our own requirement IDs (REQ-2283-*) until Tieto's requirement set arrives.The flow must trace each claim to a requirement ID. We wrote the criteria from the upstream issue.
  7. Sat 15:10Each finding ends in an artifact that a machine checks: an ast-grep rule, a TLA+ invariant or a gate.A fixed line fixes one bug. A rule that blocks the pattern stops the whole class. The 2283 rule hits exactly the 2 crash lines on the base.
  8. Sat 15:15Rules block only new hits. Old hits are debt that a person owns.The silent-drop rule also hits 2 older guards that a diagnostic covers. Blocking them would block every change to the file.
  9. Sat 15:20No TLA+ spec for the MessagePack diffs. Model-check the merge gate instead.The detector found no implicit state machine in the 2283 and 2234 diffs. A spec of nothing proves nothing. The gate is a real state machine, and its stale-evidence bug is easy to miss.
  10. Sat 16:00A model from another family (GPT-6.1 Sol) writes the rule. ast-grep decides if it holds.It keeps the model's part small and checked. The check also showed its limit: it accepted a rule narrower than the bug class.
  11. Sat 16:20Move the 2234 drop from a syntax rule to a generator behaviour test.The replay agent wrote the same bug in a shape the rule did not match. The test fails on both shapes.