When AI writes half your code, what's really worth investing in?
Andrej Karpathy—former Tesla AI Director, OpenAI founding member—recently wrote something that stopped me cold:
"Code is suddenly free, ephemeral, malleable, discardable after single use."
At the same time, GitHub reports that Copilot is now writing 46% of the average developer's code. Karpathy himself admits he "vibe coded entire ephemeral apps just to find a single bug because why not."
Code is indeed getting cheaper. Fast.
But here's the question that haunts me as someone who builds testing infrastructure: if code becomes disposable, what happens to testing? What happens to quality?
I've been chewing on this for months. And I've landed somewhere that might sound counterintuitive: when code becomes free, testing becomes more important, not less.
Here's why.
Yes, Code is Becoming Disposable
Karpathy isn't exaggerating. The numbers back him up:
- 15 million developers now use GitHub Copilot—a 4x increase in one year
- 80% of new developers on GitHub use Copilot in their first week
- 25% of Y Combinator's Winter 2025 batch have codebases that are 95% AI-generated
- 67% of developers use AI coding assistants at least five days per week
This isn't a trend. It's already the new normal.
Karpathy calls this "vibe coding"—a mode where you "fully give in to the vibes, embrace exponentials, and forget that the code even exists." He describes his own practice:
"I 'Accept All' always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension."
Coming from Karpathy, this matters. He's not being reckless—he's describing how software actually gets made now.
Think about it: when code can be generated faster than it can be read, why spend hours crafting perfect abstractions? Just regenerate the whole thing.
So yeah—code is becoming disposable. And honestly? That's fine.
The Real Problem: Did AI Build What You Actually Asked For?
But here's what Karpathy glossed over: How do you know the AI gave you what you wanted?
Many people focus on security vulnerabilities in AI-generated code—and yes, those exist. But that's not the core issue.
The core issue is requirement conformance.
You ask AI to build a "user login" feature. It produces code that runs. But how do you know:
- Did it build what you asked for? You wanted email login. It added phone login too. Did you want that?
- Did it do extra things you didn't ask for? You didn't request a "remember me" checkbox, but it added one—enabled by default. Does that align with your privacy policy?
- Did it miss something? You needed account lockout after failed attempts. It didn't implement that. Did you catch it?
- Does what it built fully match your requirements? You specified passwords must be at least 8 characters. It implemented 6. Do you know?
This is the real problem: AI-generated code runs, but "runs" doesn't mean "correct."
Before, developers wrote their own code—they knew what they wrote. Now AI writes the code. Developers know what they wanted—but not necessarily what they got.
Karpathy said it himself:
"The code grows beyond my usual comprehension."
If you don't understand what the code does, how can you be sure it does the right thing?
Here's the paradox: The easier it becomes to generate code, the harder it becomes to verify it's correct.
Put another way:
AI didn't reduce the need for testing. It increased the need to verify that requirements were correctly implemented.
This isn't about testing for bugs. It's about testing: Did AI give you what you asked for?
What's Disposable vs. What's Durable
This led me to a framework I've found useful: distinguishing between what's disposable and what's durable in the software development lifecycle.
Disposable (Low investment value):
- The test script code itself
- Concrete implementation details
- Boilerplate code
- Syntax and formatting
Durable (High investment value):
- Intent — What are you actually trying to verify? "Ensure users can complete checkout" is durable. The specific Selenium script that checks the button click is not.
- Orchestration — How do tests relate to each other? What runs when? What happens when something fails? The logic of your test pipeline outlives any individual test.
- Persistent Data — Test execution history, trend analysis, flaky test detection. You can't regenerate historical data with a prompt.
- Integration Points — How does testing connect to CI/CD? To monitoring? To alerting? These connections are infrastructure, not code.
Here's the insight: Pipeline YAML isn't "code" in the disposable sense. It's an expression of business logic—the orchestration of what you want to verify and when. That doesn't become obsolete just because AI can regenerate individual test scripts.
Similarly, when your monitoring system detects an anomaly and automatically triggers a regression test suite—that capability isn't disposable. The webhook integration, the conditional logic, the escalation rules—those represent architectural decisions that transcend any particular implementation.
What This Means for Testing Platforms
At Testany, we've been wrestling with this question: where does our value actually lie in this new world?
Here's what we've figured out:
We've embraced generative testing. Our OpenAPI Test Generator creates test cases from API specs automatically. Code generation isn't the moat—it's table stakes now. AI can write test scripts. Fighting that is pointless.
But our real value isn't in the generated code. It's in:
-
Pipeline Definition — Our YAML-based pipeline language expresses test intent and orchestration. It describes what you want to verify, not how to implement it. This is durable.
-
Gatekeeper Integration — Webhooks that connect testing to CI/CD pipelines, monitoring systems, chat platforms. When PagerDuty fires, tests run automatically. When tests fail, Slack notifies the right people. These integration points represent infrastructure decisions that outlast any individual test.
-
Execution History — Every test run creates data: pass rates, duration trends, flaky patterns, failure correlations. LLMs can't generate your historical execution data. This is irreplaceable institutional knowledge.
-
Credential Management — API keys, authentication tokens, environment-specific configurations. These need to be managed securely and persistently—not regenerated per session.
The shift is subtle but real: we're not a platform for managing test code. We're a platform for managing test intent, orchestration, and history.
The code is disposable. The system isn't.
What Should You Do?
If you're rethinking your testing strategy for an AI-first world, here's what I'd suggest:
1. Reframe what "testing" means. Testing used to be about finding bugs. Now it's about confirming AI gave you what you asked for.
This means your test cases should start from requirements, not from code. First clarify "what should this feature do," then verify "does it actually do that."
2. Invest in orchestration. What runs when? What depends on what? What happens on failure? These decisions outlive any individual test. Get the orchestration right—it'll outlive many generations of test code.
3. Build for continuous verification. The old model: write tests, run them, ship if green. The new model: run tests continuously, treat them like monitoring. When AI generates code faster than you can review it, your safety net needs to be automated and always on.
4. Treat test infrastructure like production infrastructure. Same observability. Same reliability requirements. Same investment priority. If your test environment is a black box you only look at when something breaks, you're not ready for the vibe coding era.
5. Protect your institutional knowledge. Execution history, trend data, failure patterns—these compound over time. They inform future decisions. They provide context no LLM has access to. Don't throw them away.
The Real Transformation
Karpathy was right: code is becoming free, ephemeral, disposable.
But he said something else that stuck with me:
"Despite rapid progress, the industry has realized less than 10% of LLM potential, with substantial work remaining."
If we've only scratched the surface—if code generation is about to get even easier—then verification becomes proportionally more valuable.
Think about it: when anyone can spin up an app with a few prompts, what separates the ones that actually work correctly in production?
It's not whether the code runs. It's whether the code does what you wanted.
Code can be disposable.
Confidence that "this is what I asked for"? That's not something you can throw away.
The author is a co-founder at Testany, an enterprise end-to-end testing platform. The views expressed here are informed by work on testing infrastructure but apply broadly to the industry.
References:
