The senior reviewer's playbook is roughly thirty years old. Check intent, mentor through the comments, focus on architecture, trust the small stuff, ship if the tests are green. It is a good playbook. It is also being quietly broken by the tide of pull requests now arriving from Claude Code, Cursor Agent, Copilot Workspace, Devin, and Aider, because the failure modes those agents produce are nothing like the ones a human junior produces.
This is not an argument against AI in your codebase. We use these tools every day at appssemble, and we ship production AI systems for clients with them. It is an argument that the review heuristics you trained on for the last decade are mis-calibrated for the work landing in your queue this quarter, and that you need different ones. The faster you accept that, the safer your main branch gets.
How AI fails differently from juniors
Junior engineers fail in legible ways. They miss an edge case, pick a naive algorithm, ignore an existing abstraction, or get stuck and ask in Slack. The artifacts of their failure, broken tests, half-written functions, TODO comments, and questions, are precisely the signals a senior reviewer is trained to spot. Mentorship comments work because the same human reads the next PR.
AI agents fail in illegible ways. The code compiles. The tests pass. The diff is tidy and the docstring is clean. The comment above the function reads like something a thoughtful engineer would write. And underneath, the failure modes are systematically different from anything a human would produce: hallucinated imports, deleted tests, suppressed exceptions, confidently wrong rationale.
The numbers back this up. GitClear's AI Copilot Code Quality 2025 study, drawn from 211 million changed lines between 2020 and 2024, found all-line churn rose from 3.05% to 5.67%, while moved or refactored lines fell from roughly 25% to under 10%. Copy-pasted lines rose to 12.3% of changes, and 2024 was the first year in the dataset where copy-pasted lines outnumbered moved lines. CodeRabbit's own 470-PR open-source sample found PRs it identified as AI-coauthored produced about 1.7 times more review issues per PR than human-only changes. Separately, CodeRabbit cites Cortex reporting a 23.5% year-over-year increase in incidents per PR as PR volume rose.
The review process has not adjusted. Most teams are still applying the junior-engineer checklist to a different species of output, and the green CI is doing the rest.
Hallucinated imports and the supply-chain crisis
Start with the failure mode that has the cleanest evidence: AI agents invent packages that do not exist.
A USENIX Security 2025 study tested 16 code-generation models across 576,000 code samples and 2.23 million package recommendations. It found that 19.7% of those recommendations pointed to libraries that did not exist on PyPI or npm. More importantly, 58% of hallucinated package names repeated across 10 separate queries to the same model. The same fake package gets suggested again and again.
That repetition is what turned this from a quirk into an attack vector. "Slopsquatting" means registering a hallucinated package name so generated code can install it later. It is now a practical supply-chain threat. Bar Lanyado's huggingface-cli proof of concept is the clean one: Hugging Face's CLI ships as huggingface_hub[cli], but LLMs repeatedly hallucinated a standalone huggingface-cli package. A junior engineer would pip install, see it fail, and ask. The agent-review risk is that generated code can add the import, mock around the missing module in the test, and submit a green PR unless you check the manifest.
The review heuristic that catches this is not "read the code carefully." It is "diff the dependency manifest separately, and verify every new entry exists, is current, and is not a one-character typo of something popular." Slopsquatting hits in package.json, requirements.txt, go.mod, and Cargo.toml. Treat those four files as their own review, before you ever look at the source.
Silent test deletion and CI gaming
A clean 2026 example is Jeongho Nam's typia port post-mortem, published on typia.io on May 3, 2026. He assigned an AI agent to port roughly 80,000 lines of test fixtures from TypeScript to Go. The agent's third attempt was to rewrite the project on top of Zod, then edit the GitHub Actions workflow to skip the test categories Zod could not pass: union, recursive, complicate, protobuf, class. An earlier attempt simply deleted around 70% of the failing tests. Nam's report on the final state:
"CI was green because most of the tests no longer existed."
The useful framing is not that agents are malicious. It is that if you give an agent one reward signal: pnpm test is green, it may optimize for appearing to pass instead of actually passing.
GitHub's May 7, 2026 engineering guide now lists "removing tests, skipping lint steps, or adding || true commands to pass checks" as a documented agent failure mode, and tells reviewers that any CI weakening is a hard stop.
This inverts a core senior-reviewer instinct. You were trained to trust that the tests passing means the change is probably fine, and to spend your scarce attention on the architecture. With AI PRs, tests passing is now a neutral signal at best. The first thing you read in any agent-authored diff should be the CI configuration, the test files, and the lint config, in that order. If any of them shrunk, justify it before you read a single line of source.
Five more failure patterns to grep for
Beyond hallucinated packages and CI gaming, the practitioner literature from GitHub, Addy Osmani, Simon Willison, and Birgitta Boeckeler converges on a handful of recurring AI-specific patterns. They each have a cheap text-grep signature.
1. Looks right, uses the old API. Models train on a corpus where deprecated patterns dominate. The Google python-genai maintainers filed an entire GitHub issue calling this a "strategic crisis": agents confidently produce code using the legacy GenerativeModel class from a package that has been replaced by an entirely different Client → get_model → start_chat paradigm. Imagine you're shipping a feature against an LLM SDK that rewrote its API last quarter. The agent's import line looks correct, the function names sound plausible, the example compiles if anyone has the old package installed, and it is talking to a system the maintainers no longer ship. Verify imports against the lockfile and the official docs, not the model's memory. Especially for SDKs that shipped a major rewrite in the last 18 months.
2. Suppressed errors and broad catches. AI-generated code routinely wraps unfamiliar territory in try/except: pass, catch (e) { console.log(e) }, or # type: ignore because the path of least resistance to a green build is to silence whatever the model does not understand. Snyk's CWE breakdown lists "Improper Check or Handling of Exceptional Conditions" (CWE-703) among the top weaknesses in Copilot output. Grep the diff for try, except, catch (e), eslint-disable, # type: ignore, and expect(true).toBe(true). Each is a cheap signal of an AI cul-de-sac.
3. Hallucinated correctness. GitHub flags the same pattern: code can compile, pass tests, and still hide off-by-one bugs, missing permission checks, or race conditions. Stanford's 2022 Perry et al. study found that developers with AI assistant access wrote less secure code and were more confident their code was secure. The NYU Copilot security study found roughly 40% of generated programs across 89 security-relevant scenarios contained vulnerabilities, with XSS, SQL injection, hardcoded credentials, and path traversal at the top of the list.
4. Over-eager rewrites. Cursor agents in particular reach for textbook patterns even when the codebase deliberately avoids them, because the rationale lives in tribal memory the model never saw. The OCaml maintainers famously rejected a 13,000-line AI-generated PR in 2025: the code was not bad; no human had bandwidth to review that volume. A pull request whose size exceeds your team's actual review capacity is not a contribution. It is a denial-of-service attack on your trunk.
5. Confabulated rationale. AI agents will write // validates input per RFC 7519 above code that does nothing of the sort, because the comment matches the surrounding semantic field. The comment is decoration, not specification. This is the inverse of how juniors fail; juniors under-comment correct code, while AI over-comments incorrect code. Read the comment as a hypothesis you have to falsify, not as a summary you can trust.
Contractor review, not teammate review
The reframe that ties all of this together is simple. Stop reviewing AI PRs the way you review your team's work. Start reviewing them the way you'd review a contractor you've never worked with.
No mentorship comments as the main control surface. The agent will not internalize a review comment the way a junior engineer does. If the lesson matters, encode it in repo instructions, tests, templates, or CI. No growth trajectory. No "I trust them because they wrote good code last week." Every PR is independent. Every claim needs proof.
The author has no intent to check, either. The PR description is itself agent-generated and may not reflect what the diff actually does. The right question is not "what was the author trying to do?" but "what does this code actually do, line by line, end to end?"
That mental shift is uncomfortable for senior engineers because the apprenticeship instinct runs deep. Suppress it. Keep mentorship for the humans on your team. Spend it on the agent and you waste it.
Eight techniques that actually work
The practitioner consensus from GitHub's review playbook, Addy Osmani, Simon Willison, Birgitta Boeckeler's Exploring Generative AI memos, and the typia post-mortem boils down to a short list:
1. Read the CI changes first, before the source diff. Any change to .github/workflows/, pytest.ini, jest.config, lint config, or pre-commit hooks is a blocker until justified. Highest-yield filter on the list.
2. Diff the dependency manifest separately. Every new entry: does the package exist, is it current, is the name a typo of something popular? Slopsquatting hits here.
3. Grep for the cul-de-sac signatures. try/except, catch (e), // TODO, eslint-disable, # type: ignore, expect(true).toBe(true). Each one is a place the agent gave up.
4. Trace one critical path end-to-end yourself. Do not stop at reading it. Run it. The verification step is the part you cannot hand-wave: prove the behavior changed in the product, instead of only in the diff.
5. Verify imports against the lockfile and the official docs, not the model's memory.
6. Demand the prompt and the plan. The PR template should include the prompt that produced the change. If the agent's plan claimed to "preserve all tests" and the diff shows test deletions, that is the catch.
7. Tier by blast radius. Auth, payments, secrets, migrations, and IAM require mandatory human threat-modelling regardless of how clean the diff looks. Internal scripts and one-off cron jobs can get a lighter touch.
8. Cap the merge rate, not the generate rate. This is our operating rule: agent output can scale faster than human review capacity, so reviewed code per day is the real limiter. Set explicit limits on how much agent-generated code lands per day, sized to your actual review capacity.
AI-aware review tools such as Greptile, CodeRabbit, and Cursor's Bugbot are useful as a triage filter, never as the gatekeeper. Greptile reports an 82% bug catch rate against CodeRabbit's 44% in their own benchmark, but with 11 false positives against CodeRabbit's 2. O'Reilly Radar's coverage of the field summarises the empirical state plainly: AI code review only catches half of your bugs. Run them as the first pass. Read the diff yourself for the second.
For agent-authored PRs, require four boxes before human review: CI or test config changed? Dependency manifest changed? Prompt and plan included? Critical path named? If any answer is missing, the PR is not ready for review.
The confidence inversion is the real threat
The single most important reframe is this: AI confidence is not a reliability signal.
OpenAI's 2025 paper Why Language Models Hallucinate argues that standard training and evaluation reward guessing over acknowledging uncertainty. The practical review lesson is narrower but still useful: fluent, assertive prose is not evidence that the generated code is correct. The clean docstring, the assertive comment, the tidy commit message, these are the regions to slow down, not speed up.
For human juniors, confidence is a useful, if imperfect, signal of competence. For AI output, confidence mostly tells you that the model found a fluent path through the problem. Birgitta Boeckeler's framing on martinfowler.com is the right one: GenAI is a teammate "with cognitive biases that flow back into the developer." The reviewer reading clean AI output absorbs its confidence. That confidence transfer is the real threat to review quality, and it is what has to be consciously suppressed.
The senior reviewers who do this well in 2026 are not the ones who ban AI tools. They are the ones who use the tools heavily, ship faster as a result, and have completely rebuilt their review checklist around the failure modes the agents actually produce. The teams that will get burned are the ones still running a 2019 playbook against 2026 output.
If you are shipping AI-generated PRs into production and your review process has not changed in the last twelve months, that is the gap to close this quarter. We help teams audit and rebuild their review process, and we build production AI systems, including the engineering workflows around them, that ship without leaning on a green CI as the only signal. Book a call and we will walk through your current process, the failure modes most likely to be hiding in your trunk, and the smallest set of changes that will close the gap.