AI-generated code rarely fails in obvious ways. It usually looks fine, and that is the problem. These are the five failures to look for when you read the diff.
The five failures#
| Failure | What it looks like | How you catch it |
|---|---|---|
| Plausible but wrong logic | Compiles, maybe passes some tests, but has an off-by-one, a wrong condition or a missing edge case | Tests for edge cases; read conditions and loop bounds carefully |
| Invented or outdated APIs | A function or library that does not exist, or was removed or renamed in a newer version | Run the code; check the docs for the version you use |
| Ignoring your design | A new helper instead of the one you have, or a new pattern instead of yours | Give context up front; compare the diff with existing code |
| Security mistakes | String-built SQL, missing authorization checks, hardcoded secrets, logged sensitive data | Review security-relevant lines on purpose, not in passing |
| Editing tests to pass | Asked to make the tests pass, the agent changes the tests instead of the code | Check every change to test files separately |
Why these are hard to spot#
Plausible but wrong logic is the hardest, because it does not announce itself. The code reads well, so a quick review passes it. It is the same root cause as a hallucination: the model writes what looks likely, not what it has checked.
Security mistakes are hard for a related reason. The code does exactly what you asked, just unsafely, so it passes a review that only checks behavior. The same rules about secrets apply to generated code as to prompts.
Editing tests is specific to agents. An agent told to "make the tests pass" can meet that goal in the wrong way. When an agent's diff touches test files, read those changes first.
Short line#
AI code fails quietly: plausible logic with a subtle bug, APIs that do not exist, new helpers that ignore your design, unsafe SQL or missing auth checks, and agents that change the tests instead of the code. Read test-file changes and security-relevant lines on purpose.
Key takeaways#
- Plausible but wrong logic is the hardest failure to catch: off-by-one, wrong condition, missing edge case.
- Tools invent APIs or use outdated ones. Run the code and check the docs.
- Tools ignore existing helpers and patterns unless you give them context.
- Watch for string-built SQL, missing authorization, hardcoded secrets and logged sensitive data.
- Agents can make tests pass by editing the tests. Check test-file changes separately.