Property-Based Testing Belongs in Your Agentic Security Harness
My case that coverage, mutation scores, and LLM-written tests share the same blind spots. And how property-based testing (PBT), which searches for counterexamples to declared invariants, is emerging as a low-cost complement for agentic development harnesses.
Full coverage does not cover everything
Branch coverage and mutation score are the usual yardsticks for judging a test suite, including one generated by a language model. Xiaoyu Shi, Wenqiang Luo, Boyang Yang and Haoye Tian argue in a 2026 paper (cs.SE) that these aggregate metrics do not reveal the behavioral obligations a suite systematically leaves unverified. That limit in existing evaluation protocols motivates their BlindSpotEval framework.
The authors evaluated 14,320 tests generated by four LLMs under four prompting strategies. Their conclusion is that LLM-written tests reproduce the happy-path bias seen in human-written ones: normal inputs and expected flows are exercised, while unusual values and exception paths are neglected. The gap appears on the boundary-value and control-flow dimensions.
Two figures from the study illustrate the problem. Among executable LLM-generated tests, negative arguments were only covered at a rate of 23.8%, according to the authors. Incorrect test assertions accounted for 15.6% of test failures, which the authors present as evidence that coverage alone does not establish the correctness of a test oracle, the mechanism that decides whether an output is right.
Complexity does not explain the blind spots. Using the part of their ObligBench benchmark made up of 700 low-complexity functions, the authors observed the same gaps.
Declaring invariants instead of listing cases
Property-based testing removes the need to enumerate edge cases by hand. A developer describes invariants, such as a round trip that returns the original value, commutativity or positivity, and the framework searches for counterexamples. Anthropic's Frontier Red Team describes the method in a recent post, using the Hypothesis Python library.
Instead of writing individual examples, a developer states an invariant, such as a round trip that returns the original value, commutativity or positivity. A framework such as Hypothesis for Python then generates many inputs and searches for a counterexample. When one is found, it is reduced to a minimal failing case and reported.
This approach also addresses the "self-deception cycle": When the same model writes both code and tests, the tests tend to share the flaws of the code they are meant to validate. I would argue that properties break this loop and are simpler to define than exhaustive test oracles.
Anchoring expected behavior outside the code
In his blog, Mathias Wolff argues that an AI agent does not correct the bias of hand-written examples, because the ones it proposes are statistically those humans already think of. A declared property, from which the framework generates hundreds of inputs, escapes that bias. He cites NumPy's random.wald, which silently returned negative values, and slice_dictionary in AWS Lambda Powertools.
For code-generating agents, feedback built on verified properties, such as "a sort produces a non-decreasing sequence," paired with the smallest failing counterexample, is a more useful signal than self-generated tests. In a paper, the PGS framework reports improvement over TDD methods and up to 64% correction on well-disposed problems, that initially failed . On difficult problems from CodeContests, this rate falls between 9% and 25%.
When agents use PBT as a bug-hunting tool
MATS, Anthropic and Northeastern University presented a large-scale evaluation at the NeurIPS 2025 Deep Learning for Code workshop. Their agent is built on Claude Code and writes Hypothesis tests.
Run on 100 popular Python packages comprising 933 modules, the agent produced 984 bug reports, according to the authors. It flagged problems in 786 modules, or 84.2%, which works out to slightly more than one report per module on average. Total cost amounted to $5.56 per bug report that they produced, yes that cost includes definition of the properties on the tooled code.
You should still review each bug carefully to avoid burdening maintainers with unreachable bugs. For teams budgeting a harness, that review time is a cost the API figures do not capture.
The study was led on popular open-source projects including 15 standard-library packages, 15 hand-picked PyPI packages such as numpy and pandas, and 70 drawn at random from the 5,000 most downloaded. Chosing a mature piece of code is the best case scenario for this: interfaces being mostly defined and stable means your properties will be as well so the cost of coding them will be amortized over the long run.
Get my analysis of testing and security practices for AI-assisted development in your inbox.
A (personal) case for self-testing contracts
In the case of contractme, a design-by-contract (DbC) library built on Hypothesis that I maintain, the self-test function derives hundreds of generated inputs from executable contracts. This sidesteps the oracle issue, since expected behavior comes from the contracts and not from the code under test.
Design-by-Contract (DbC) is an approach to coding where you write input and output properties of your functions, alongside their code. These properties should be executable and are known as precondition (respectively "require") and postcondition (resp. "ensures").
DbC has a long flourishing history in academia, not so much in industry, due to the cost of training engineers to its use, but recent evolutions of LLMs have seen this cost plummet. My personal opinion is that industry as a whole is going in that direction, and PBT is the next step in that journey.
This also has the added benefit of reducing the weight of code reviews, as they only need to be performed on the contractual part, which is arguably shorter, easier to read, and more stable.
Properties and examples catch different defects
Complementarity is the central finding of a study by Tanaka and colleagues. The researchers used tests generated with Claude-4-sonnet on 16 HumanEval problems whose standard solutions failed extended tests.
Property-based tests and example-based tests each detected 68.75% of the bugs, according to the authors. Combined, they detected 81.25%.
The authors conclude that a hybrid approach is preferable. Property-based tests were better at surfacing performance problems and edge cases through broad exploration, while example-based tests were better at pinning down precise boundary conditions.
It is to be noted that most property-based fuzzers allow for "example-lists" that is extending the generated examples with known counterexamples, typically the ones found during previous testing sessions. I would argue that this makes in fact property-based testing a strict superset of unit testing, a tool able to run the usual tests your developpers are comfortable with, and more.
Where everything fits, and how I learnt to love PBT
Anthropic's Frontier Red Team points out that logic bugs suit property-based testing and that many security vulnerabilities originate in them, blockchain smart contracts (not to be confused with DbC) being one example. The team suggests the approach could help catch such flaws before deployment. The evidence cited here comes from general Python packages, not from security-specific benchmarks.
The studies point to a layered arrangement. Unit tests keep the precise boundary checks that Tanaka and colleagues found examples handle better. Property checks add exploration at scale and, as the Property-Generated Solver authors argue, avoid the shared flaws between generated code and generated tests. Mutation testing continues to measure suite strength, with the caveat from Shi and colleagues that its score does not expose unverified obligations.
To end this whole journey accross the testing landscape in the age of LLM, here's my personal take: review remains necessary, but with DbC it can be focused on a few lines of contract rather than the large diffs your tooling might produce. Properly contracted and exercised code should guarantee that your code cannot act out of the bounds you've set for it, which is, in my opinion, a necessary condition to truly unleash your coding assistants' productivity.