kodebeat / papers / code

Reviewing code that no human wrote, and still owning it

A review sheet that replaces reading with mechanical checks.

Machine-written bugs arrive neatly formatted and commented, so instincts trained on human mistakes never fire and the diff is approved. A browser review sheet built on one rule: every check replaces reading with something mechanical. Seven checks run on every pull request in about five minutes, seven more are escalations, and four preconditions gate the review entirely.

The problem

A pull request arrives. It is four hundred lines, formatted correctly, commented, consistent with the conventions of the codebase around it, with tests that pass. A model wrote it. The reviewer reads it, finds nothing to object to, and approves.

This is the failure. The reviewer's instincts are calibrated on human error, and human error has a texture: the variable named tmp2, the copy-pasted block with one line not updated, the comment that contradicts the code beneath it, the shortcut taken at 17:45 on a Friday. Those signals are what "this needs a closer look" is made of, and generated code emits none of them. It is uniformly plausible. The parts that are correct and the parts that are wrong look alike.

Worse, the failure mode inverts. Human code tends to be wrong in what it does; generated code tends to be wrong in what it omits. A model writes what the prompt described and nothing else, so the migration is missing, the rollback path is missing, the feature flag is missing, the error case that nobody mentioned is missing, and the resource that was opened is never closed. Absence is invisible when reading. You cannot notice a line that was never written. You cannot notice a convention the model did not know to follow.

And a second, quieter problem sits underneath. Quality drifts across a long generated file: error handling that was careful in the first function is half-hearted by the eighth, validation quietly stops appearing, and later sections carry stubs marked "similar to above". A reviewer reads the beginning attentively, establishes that the code is good, and skims the rest, which is backwards.

The team knows something has changed and responds by trying harder: read more carefully, spend longer. That does not work, because the problem is not insufficient attention. It is that reading is the wrong instrument for this class of defect, and more of a wrong instrument yields nothing.

The method

Every check replaces reading with something mechanical. This is the sheet's single organising rule and the reason it exists. A check that says "consider whether the error handling is adequate" is reading with extra words. A check that says "flip a conditional and rerun the suite" is a procedure with a verifiable outcome. Where a check cannot be made mechanical it is stated as a precondition or a standing rule instead, so the distinction stays visible. The cost is time: every check is run by hand, which makes the sheet slower than a bot, and a missing requirement has to be written before the review can start.

Two tiers, one labelled by trigger. Seven checks run on every pull request and take about five minutes. Seven more are escalations, each carrying the condition that turns it on — when: diff over ~400 lines, or a new module; when: money, auth, permissions, or possible data loss; when: schema, config or infra is touched. Each trigger is a property of the change that can be read off the diff, and putting it on the item rather than in a policy document makes the escalation decidable in the moment, by the reviewer, without a meeting.

Four preconditions, checked before reading starts. Each one, failed, means the review cannot reach a real verdict, so the fix belongs upstream rather than in review comments:

Name the two stances, because the same finding means different things. If you generated the code yourself, every upstream fix is yours to make before opening the pull request: split the diff, rerun the session for what it skipped, tighten the prompt. If somebody handed it to you, the same findings become requests, and quietly fixing them yourself is how the author never learns what their tooling drops. The sheet states this explicitly because the self-review case is now extremely common and is usually treated as though it were the other one.

Push repeated work into gates, once. Anything run twice by hand should become automation: types that make invalid states unrepresentable, property and integration tests in CI, a mutation-score gate on changed paths, and a shared prompt configuration that caps what arrives: minimal diff, no abstraction it was not asked for, uncertain APIs flagged, and a closing list of assumptions and stubs. That last item is the most valuable line on the page, and asking for it at generation time costs nothing.

Attack the happy path from a menu, not a list. Stage 03 is a menu: pick the one or two lines that describe this code and ignore the rest. A pure function has no concurrency; a CRUD endpoint has no float precision. Each entry names an assumption the happy path depends on and never states, and applying all of them to everything would produce the exhaustive checklist that gets abandoned in week two.

Standing rules sit at the end of each stage, where they generalise the stage's checks rather than adding to them: read the end of a long generated file harder than the start; manual testing never substitutes for a test, because it does not run tomorrow when somebody else edits this; when reading cannot settle a question, run that one path rather than debating it in comments.

No account, no backend, no telemetry. It is a single page. Progress lives in the browser, the summary is copied to the clipboard, and nothing about the code under review leaves the machine, which makes it usable on a diff that belongs to a client.

The headline results

  • Each check names an action with an observable outcome, and that is what can be verified.
  • Stage 03's entries are grouped by the assumption each one attacks
  • Property-style assertions are recommended where exact outputs are unknowable.

The full results, with the tables and the reasoning behind each figure, are in the PDF.

What is in the PDF

  • The problem — what goes unanswered without it, and who notices first.
  • Why the obvious alternative falls short — stated plainly, including where it is the better choice.
  • How it works — the method, not a feature list.
  • Concrete use cases — with console output quoted from the repository, never reconstructed.
  • The methodology behind any number it emits — every term shown, so the figure survives a question.
  • What it deliberately does not do — the section most papers leave out.

Part of the Code & repo intelligence theme. The tool it describes runs in your browser at https://kodebeat.com/ai-review.html.

Get the PDF

This paper already selected; the download starts when you send the form. No follow-up sequence.

Get the PDF