Testing in the Age of AI
🇩🇪 Auf Deutsch lesen~7 min read
testingaillmsoftware-qualityci-cdtest-generationdevops
Back openDesk Edu for a sovereign, open-source education — every vote counts.
Vote nowSave products you love by clicking the heart icon.
LLMs can generate Terraform code—but only 3 out of 7 models produce security-compliant IaC. A recent study benchmarks models for secure infrastructure generation and reveals how to use AI safely.
For three decades, software testing was the tax you paid for shipping. You wrote the feature, then — if the deadline allowed, if the coverage gate didn't block the release at 11 p.m. — you wrote the tests. The research corpus tells a different story now. Of 4,358 software-engineering papers ingested over the past year, "automated testing" is a steady keyword burst, and a new cell has appeared from nowhere: agentic testing. The bottleneck hasn't gone away. It has changed hands.
The dominant pattern of 2025–2026 is no longer "a human writes the test". It is "an LLM writes the test, a human reviews it". The work clusters into three moves.
Unit test generation from real code. Benchmarks such as ULT (UnLeakedTestbench) exist precisely because most earlier benchmarks were contaminated and built from structurally trivial functions, so their success numbers never generalised. ULT is built from real-world Python functions to strip out that bias.
Test-case carving from production traffic. NL2Test, an experience paper from enterprise microservice teams, turns a natural-language scenario plus a recorded traffic capture into replayable API regression tests — binding dynamic values from responses instead of hard-coding them, and generating assertions that track business intent rather than brittle field equality.
Assertion generation. The hard part was never the request; it was the oracle — what "correct" actually means. NL2Test generates assertions aligned with business meaning, not with the byte-for-byte response.
None of this removes the human. It moves them upstream: from authoring boilerplate to judging intent.
Search-based tools gave high coverage and unreadable tests. LLM tools gave the reverse — human-readable tests that often failed to compile or covered little. AdverTest frames this directly: it pits a test-generation agent against a mutant-generation agent that rewrites the code to inject the bugs the tests should catch. If the tests still pass on the mutant, they were never testing anything.
Two empirical findings should cool the enthusiasm.
A generated test suite is a hypothesis, not a certificate.
The cleanest idea in the corpus is ConVerTest: synthesise reliable tests without holding the correct implementation to check against. It combines self-consistency (majority-vote convergent tests), chain-of-verification (reasoning-guided refinement) and dual-execution agreement (cross-validate code and tests by consensus). On BIGCODEBENCH and LBPP it raises test validity, line coverage and mutation score by up to 39%, 28% and 18% over baselines. The practical reading: you can grow a suite from a specification alone, then harden it by agreement.
# An LLM-generated test — and why mutation score, not coverage, is the real signal
def parse_version(s: str) -> tuple[int, int, int]:
major, minor, patch = s.split(".")
return int(major), int(minor), int(patch)
def test_parse_version():
assert parse_version("1.2.3") == (1, 2, 3)
assert parse_version("10.0.1") == (10, 0, 1)
assert parse_version("0.0.0") == (0, 0, 0) # kills the `>` -> `>=` mutant
A coverage tool reports 100% on that function. A mutation tool injects > → >= somewhere and asks: does any test fail? If not, your "100% coverage" is a story the model told you.
| Signal | What it tells you | Limit |
|---|---|---|
| Line coverage | The code was executed | Says nothing about assertions |
| Branch coverage | The paths were taken | Still no oracle check |
| Mutation score | Tests kill injected bugs | The quality signal that matters |
The DORA and SonarQube data agree: above roughly 80% coverage is necessary, not sufficient for lower defect density in hyperscale systems.
Testing AI systems brings back a problem classical testing thought it had solved: the oracle. When the system under test is an LLM or an agent, "correct output" is fuzzy. The corpus answers in two directions.
And the meta-pattern: The Rise of Agentic Testing describes multi-agent systems where a generator, a critic and a runner argue about whether a test is any good. FeedbackLLM does the same with specialised feedback agents per concern. SAINT pushes it to the service level, generating endpoint and scenario tests for enterprise Java from static analysis plus LLM agents.
Two papers name the danger precisely. (Over)Reliance on Test Agents in AI-Assisted Software Testing shows teams trust AI-generated tests more than they should. On the risk of coding before testing is an empirical study that finds the exact anti-pattern its title warns against: generating code and tests together yields confident, wrong suites.
The failure mode is subtle. An AI writes a function and a test that agree with each other. They are consistent, not correct. The test passes; the bug ships. TestLoop, a process model, puts the human back in the loop on purpose: the machine proposes, the person disposes.
The second half of the title is the newer discipline. You do not only test with AI; you test the AI.
Secure code generation has its own red team: adversarial prompts that probe LLM defences. The lesson repeats — evaluation must be adversarial, not ceremonial.
If you take one thing from this: the test is now a first-class artifact you review like code. Concretely.
# Quality gate: fail the build when mutation score drops
- name: Mutation score
run: |
mutmut run --paths-to-mutate=src/
score=$(mutmut results | compute_score)
test "$score" -ge 80 # surface, don't hide, weak tests
Testing did not get easier. It moved up a level. You no longer test only the code; you test the tests, and you test the tester. The teams that win are not the ones with the most AI-generated tests. They are the ones who review them.