What we measure when we say a review is good

Independent benchmarks put various AI reviewers around an 80% bug catch rate, and those numbers are argued about constantly — for good reason. Catch rate alone is trivially gamed: report everything and you catch everything.

1. Did it find the real bug

The honest way to measure this is against merged pull requests that fixed a known bug, where the linked issue gives you ground truth. Review the pre-fix commit and see whether the tool identifies the thing that was actually wrong.

2. How much else did it say

This is the number that keeps catch rate honest. A reviewer that finds the bug and posts forty other comments has not helped; it has moved the work from finding the problem to filtering the output.

We publish the discard count on every review for this reason. On one 80-file pull request: 226 raw findings, 65 posted inline, 79 discarded outright. If a tool never discards anything, it is not filtering — it is forwarding.

3. Did a human agree

Every finding has helpful, not helpful and false positive buttons. This is the only feedback that reflects whether a comment was worth a developer's attention, and it is the signal we most want and currently have least of.

4. What it cost

A review that finds one more bug for ten times the money is not obviously better. Measured AI spend per review is recorded against every run, which is also what unit billing is derived from.

What we have not done

We have not published a benchmark against CodeRabbit or Greptile. Until we do, every claim on our site is our own assertion, and you are right to weigh it accordingly. It is the most useful thing we could publish next, and the fact that it might not flatter us is not a good reason to delay it.