In July 2025, METR ran the most-quoted AI productivity study of the last two years. They took 16 experienced open-source developers, paid them $150/hr, gave them Cursor Pro with frontier Claude models, and watched them work on 246 real issues in mature codebases, repos averaging 22,000+ stars and a million-plus lines of code.
The developers expected AI would make them 24% faster. After the experiment, they still believed it had made them about 20% faster. The stopwatch said they were 19% slower.
That gap, between what AI feels like and what it measures, is the single most important thing a CTO needs to understand before deciding whether to outsource an AI-assisted build, hire vibe-coders, or set up an internal team. Because the same Claude Code that makes a solo founder feel godlike for a weekend is the same Claude Code we use to ship production features in days. The tool isn't doing the work. The delivery system around the tool is.
The slowdown nobody on Twitter talks about
The METR result is uncomfortable for everyone selling AI productivity. It is also extremely well-controlled: real issues, real codebases, real senior developers with around 5 years and 1,500 commits in the repos they were working on. METR's own factor analysis pointed at five likely causes, high quality bars in mature projects, limited Cursor experience, and large-codebase complexity among them.
The contrasting study, GitHub's Copilot experiment, later published with Microsoft Research, found a 55% speedup: 1h 11m with Copilot versus 2h 41m without, p=0.0017. The difference between 19% slower and 55% faster isn't a contradiction. It's the gap between greenfield and mature. GitHub's task was "implement an HTTP server in JavaScript from scratch." METR's tasks were "fix this bug in a million-line codebase someone else maintains."
McKinsey saw the same gradient in their 40-developer study: up to 2x faster on simple boilerplate and refactors, less than 10% time saved on complex tasks where the developer wasn't familiar with the framework. Junior developers, in some scenarios, were 7-10% slower with AI tools.
The pattern is consistent across the evidence that matters. AI gives a large speedup on greenfield work that has no installed base of constraints, a smaller speedup on well-scoped work in code you know, and a slowdown on mature code with high quality bars and unfamiliar territory. The 70% wall, our shorthand for vibe-coding's familiar failure mode, not a formal benchmark, sits exactly where greenfield meets the installed base.
The 70% wall is real and predictable
In vibe-code rescue work, 70% is a useful warning label: the demo exists, the product shape is visible, and the next change feels dangerous. Public rescue shops such as Outsourcify and Ulam describe multi-week engagements, often in the 2-6 week range, to turn a Lovable or Bolt MVP into something that won't fall over at 200 users. The pattern of failure is so consistent it reads like a checklist:
- No error handling, so the app crashes instead of degrading.
- Missing or token-flavoured authentication.
- No input validation: the SQL injection, the XSS, the unbounded uploads.
- Hardcoded secrets in the repo or the bundle.
- No tests, so any change is a roll of the dice.
- A schema that won't survive past a few hundred users because the agent never thought about indexes.
GitClear has been measuring this shift in aggregate across 211 million changed lines. From 2021 to 2024, copy/pasted lines rose from 8.3% to 12.3% of all changes. Code clones grew roughly 4x in volume by their 2025 update. Refactoring's share of all changes collapsed from 25% to under 10%. Code churn, lines reverted or rewritten within two weeks, is on track to double versus the pre-AI baseline.
GitClear is vendor research, and it measures correlation, not a clean causal path from Copilot to every clone. But the direction matches what we see in rescue work: teams add and clone more code while refactoring less. The bill arrives in v2.
The repo scaffold is the real multiplier
Vibe-coders and we use the same tools. Same Claude Code. Same Cursor. Same Vercel preview deploys. Same GitHub Actions. The difference is what we wire up around them before we let the agent touch a file. Here is the five-item scaffold, in the order we set it up on every new repo:
1. Types and tests as the contract. TypeScript, Pydantic, sqlc-generated types from the schema. A failing test written before the agent runs. Stack Overflow's 2025 survey found 66% of developers cite "AI solutions that are almost right, but not quite" as their top frustration. Types catch a meaningful share of "almost right" before review. A failing test turns the agentic loop into a measurable target, red, green, refactor, with PostToolUse hooks running the suite after every edit.
2. Linting, formatting, and pre-commit enforcement. Biome or Ruff or golangci-lint. Lefthook (Go binary, parallel by default, no npm post-install) over Husky. The hook stays fast, format only, and CI enforces the heavier rules. A CLAUDE.md file is guidance. A hook is enforcement.
3. CI and preview deploys per pull request. In our repos, GitHub Actions with Turborepo caching has cut small build checks from 2m 34s to roughly 15s. Every PR gets a Vercel or Render URL. There is no "but it worked locally."
4. Observability from day one. Honeycomb, Sentry, OpenTelemetry, structured wide events. Charity Majors's rule, which we've adopted verbatim: never accept a pull request unless you can answer the question "how will I know when this isn't working?" Dashboards are owned per feature, not per team.
5. Scoping with explicit non-goals. A one-page brief: goal, acceptance criteria, I/O contracts, what's not included, rollback plan, observability plan. Augment Code's spec guidance makes the same point we see in practice: sloppy specs create rework that looks like model failure. The "Not Included" section cuts scope creep before the agent starts guessing.
Five hours is our normal first-pass setup on a small repo. It is not the entire project. It is the part that makes every later agent run bounded.
How we move: roughly 3 days for bounded CRUD, 5-10 for production AI
Here is the concrete pattern, not the aspirational one. Imagine you're a CTO who needs a new dashboard endpoint with auth, persistence, and a UI screen. When the repo scaffold already exists and the slice is that bounded, our target is about three working days.
Day 1, morning. One-page design doc. Schema sketch, acceptance criteria, "Not Included" section, rollback plan, the dashboard we'll watch after shipping. Thirty minutes.
Day 1, afternoon. Failing tests written against the contract. The tests are the spec the agent gets handed. They are also the feedback loop the agent iterates against. We run them locally and they fail in the way we expect.
Day 2, morning. Hand the spec, the tests, and the relevant context window to Claude Code. PostToolUse hook re-runs the test suite after every file edit. The senior on the work reviews diffs as they land, typing has been replaced by reading. Simon Willison's call here is right: if you're fast and productive at code review, AI accelerates you; if you'd rather write code than read it, you'll be slower.
Day 2, afternoon. Wire dashboards, structured logs, alert thresholds. Deploy to a preview URL behind a feature flag.
Day 3. Manual QA on preview. Edge cases. Promote behind the flag in production. Watch the dashboard. Roll the flag forward.
For a bounded production AI feature inside a product like the AI Communication Platform, where voice translation runs at sub-500ms latency across web, desktop, iOS, Android, Slack, and Teams, our planning window is usually 5-10 days. The shape is the same; what gets added is an evaluation suite and golden test set in days 1-2, prompt versioning and cost dashboards in days 7-8, and a gradual rollout with rollback ready in days 9-10. That timeline assumes the production shape already exists: real environments, feature flags, evals in CI, and someone accountable for rollout. AI features need eval suites the way regular features need unit tests.
These are not record times. They are normal weeks for bounded slices. The Claude Code public changelog is a useful sanity check on cadence: early 2026 had a dense run of releases. We are not Anthropic, and release notes are not the same thing as customer features, but the pattern shows what disciplined release machinery makes possible.
Why this matters beyond the MVP
DORA's 2025 State of AI-Assisted Software Development is the report to read if you're skeptical of any single number above. It surveyed nearly 5,000 technology professionals and surfaced two findings that look like a contradiction until you sit with them:
- AI adoption is now positively associated with software delivery throughput. More than 80% of respondents report increased productivity.
- AI adoption is still negatively correlated with delivery stability. Change failure rates go up.
Faros's 2026 telemetry discussion makes the same tradeoff more concrete: epics completed per developer were up 66.2%, while review time, bugs, and incidents also rose. That number belongs to Faros, not DORA.
DORA's own framing is the line we'd use to anchor any conversation with a CTO: AI amplifies the system already there. Strong teams get more efficient. Struggling teams expose more failure modes. Stability, in their data, requires strong automated testing, mature version control practices, and fast feedback loops. The repo scaffold, in other words.
DORA's performance-cluster numbers show what small-batch operating models make possible: elite performers deploy 182x more frequently than low performers, with 127x faster lead times and 8x lower change failure rates. Trunk-based development is one capability behind that pattern, not the whole explanation. AI accelerates the throughput gap without closing the stability gap. The team that ships small batches multiple times a day pulls further ahead. The team that ships big batches with broken tests breaks bigger and more often.
This is the answer to the founder who looks at a vibe-coded MVP and asks why it took us five days to ship what they shipped in a weekend. The weekend MVP and our five-day version are the same speed at v1. By v3, the weekend MVP is in rescue and our version is on its tenth deploy.
Objections we hear and what we tell people
"Aren't you just slow because you over-engineer?" No: the scaffold exists because the alternative is more expensive, not because we like ceremony. A one-page spec takes 30 minutes. A Lefthook config takes 20. A failing test takes the time it would have taken regardless. The setup is a first-pass cost per repo. The payoff is every subsequent agent run being bounded.
"Couldn't a strong solo founder do this too?" Yes, and the ones who do are exactly who we're describing. Willison calls this "vibe engineering", accountable, test-backed, observable AI-assisted work, and reserves "vibe coding" for the prompt-driven, no-attention-paid version. Both ship code. Only one ships software.
"What about the 19% slowdown? Doesn't that apply to you too?" It applies to everyone working in mature, high-quality codebases. Our answer is the scaffold: TDD with Claude Code so the agent has a target, scoped specs so it knows where to stop, observability so we can see what it shipped. METR's February 2026 update suggests the 2025 slowdown may already be shrinking, but the authors are careful: selection effects made the new experiment hard to interpret. For the original-developer subset, the raw estimate was an 18% speedup with a confidence interval from 38% faster to 9% slower. The useful takeaway is narrower: experience with the tool matters, and measurement is still messy.
The difference starts with five hours
We use the same Claude Code, the same Cursor, the same Vercel preview deploys as the founder who shipped a broken MVP last weekend. The difference isn't the tools. It's a one-page spec, a failing test, a Lefthook config, and a Honeycomb dashboard. Five hours of first-pass setup that lets us ship bounded work in days instead of dragging v2 into a month-long rescue.
If you have an AI-assisted build going sideways, or a v1 that needs a v2 the vibe-coder can't reach, our engineering team does this every week. For an example of what the discipline produces in a real product, see how we built the AI Communication Platform, sub-500ms voice translation across six platforms, shipped behind eval suites and gradual rollouts.