Model leaderboards and benchmarks take almost all the attention, while the tests teams write for the agent applications they build themselves are almost never systematically examined. A peer-reviewed empirical study fills that gap: instead of running scores or sending questionnaires, the researchers hand-labelled the test methods in real open-source projects one by one, and counted what they actually assert. The conclusion is not polite – the vast majority of tests verify that a tool can be called and that the code runs, not whether the agent made a correct decision.
For teams currently wiring large-model capability into business systems, the value of this statistics is not that “the score is low”, but that it provides explicit ratios you can check your own suite against.
How the Study Was Done
The method itself deserves a look first, because it determines whether the numbers that follow can be trusted.
The research team searched a code hosting platform for Python projects built on agent frameworks, obtaining 1,190 candidate repositories across 11 frameworks, from which they extracted 1,723 agents and 4,111 tools. To keep the manual labelling representative, the deep-dive scope was narrowed to projects “with a test suite of some size”, and finally 2,572 test methods across 240 test modules in 12 projects were annotated one by one.
Labelling was done independently by two people, with a Cohen’s kappa of 0.92, which falls in the near-perfect agreement range. The taxonomy was also fixed in advance: 23 test patterns across four dimensions – fixtures, test data, test targets and assertions – and every percentage is computed from explicitly defined categories rather than impressionistic summary. In addition, 10 senior practitioners with an average of more than 17 years of experience were interviewed to cross-check the repository data.
This is counting evidence about real code: not a survey of intentions, and not a vendor benchmark.

The share of each test type in the labelled sample, sourced from the manual labelling results of that empirical study
Finding One: Most Tests Never Run the Whole Agent
61.8% of the labelled tests evaluate only the isolated behaviour of a single tool or a single agent; 34.6% target interactions between agents, or between an agent and other components; only 3.5% cover consecutive calls across multiple tools; and merely 0.1% do end-to-end verification through an API the application exposes.
Compare this distribution with the way failures actually happen in production and the problem becomes clear. Agent failures usually occur at the interaction layer: the wrong tool is chosen after a plausible first step, state is carried incorrectly across turns, the output structure the second tool receives does not match what the first tool promised. And those fall exactly on the thinnest layer of testing.
The study also observed that frameworks quietly determine test shape: frameworks offering clean tool isolation boundaries (such as MCP-based projects) lean towards unit tests; frameworks that write orchestration logic explicitly in code (such as LangChain and LangGraph style) show more integration tests. Technology choice inadvertently decides what the team “will test along the way”.
Finding Two: A Third of Tests Mock the Very Thing They Claim to Test
34.4% of tests mock the agent or tools in some form: 21.7% replace some components, 22.3% replace the entire agent, with overlap between the two. And of all mock scenarios, only 13.4% use test doubles built specifically for a validation goal; the rest rely on standard stubs and fixtures.
Mocking is not inherently sinful; it makes tests fast and deterministic. The problem is that in an agent system a tool actually has two interfaces: the call API facing the machine, and the natural language description facing the model – and the latter determines how the model understands when to use it and how to fill in its arguments. A mock that returns the correct value cannot say anything about whether that description leads the model to call this tool at the right moment, with the right arguments, or to call it at all.
In other words, a mock test proves the orchestration logic is not written wrong, not that tool selection is not wrong. Both show up as green in the test report.
Finding Three: Inputs Are Much Cleaner Than Real Tasks
39.9% of tests use simple, example-style inputs, and only 10.5% cover multi-step scenarios.
This is the classic “test data is tidier than users” problem in a new skin. A clean prompt with clear semantics and a single answer passes under almost any model version, any prompt tweak and any tool change; it will equally pass the whole week before a genuinely complex request brings the system down. When test inputs are an order of magnitude tidier than production requests, the suite only measures that tidy world.
Specifically for agents, what is usually missing is this: requests with incomplete fields, a previous turn that changes the meaning of the current one, a tool that returns an error the first time and needs a retry or a detour, a user correcting the goal midway. Those are the actual contents of multi-step scenarios.
Finding Four: Reasoning Is the Product Itself, Yet Almost Never Asserted
Among assertion targets, output accounts for 55.9%, state for 51.1%, and reasoning-related behaviour only 8.7%. The study’s assessment of that 8.7% is quite direct: most of them only check that “reasoning-like behaviour occurred”, not whether it was correct or useful. The example given is verifying that a sub-agent’s activity appears in the logs – whether the delegation was appropriate, or whether the sub-agent’s work improved the final result, is of no interest to the test.
Another number is more alarming: only 4.1% of tests verify interaction with non-deterministic, non-agent components, and 2.4% verify interaction with deterministic components. For a class of system whose main risk comes from side effects (writing files, changing records, sending messages, triggering payments), the “what it did” layer leaves almost no verifiable evidence. A correct final answer that called the wrong tool along the way and used the wrong customer ID cannot be caught by output assertions.
Finding Five: Security, Performance and Cost Are Barely in Scope
Tests touching non-functional requirements are only 7.5%. Within that already small slice, 49.0% concern security, 33.5% performance, 8.8% memory and 4.6% policy compliance. Converted, about 2.5% of all labelled tests involve performance – for software whose single task may loop dozens of paid model calls.
Practitioners in the interviews value non-functional testing more than the open-source projects do, which roughly suggests that lessons from production supply what the repositories cannot show. But the core gaps reported by both sides are the same: no systematic testing foundation, unclear test goals, and difficulty generating high-quality test data.

A four-layer strengthening path from “testing only the plumbing” to covering trajectories, state and side effects
Rebalance, Not Rebuild
You do not need to rebuild the whole test system, only to adjust the ratios in the five directions below. The percentages from the study make a good baseline for “what to move away from”.
- Add one end-to-end task test per critical workflow. Use real tools and real inputs with noise, and assert not the final text but the final state of the world: if the agent is supposed to update a record, go and check that record.
- Assert the trajectory, not just the answer. Check which tools were called, in what order, with what arguments. Calling the wrong tool with the wrong customer ID yet still stumbling onto a correct answer is a failure that output assertions are bound to miss.
- Keep mocks, but state what they cannot prove. A test with a mocked tool only proves orchestration logic; write that in the test name or a comment, so later readers do not read green as end-to-end confidence.
- Put some dirt in the inputs. Ambiguous requests, missing fields, preceding turns that change semantics, tools that fail on first call. The 10.5% multi-step share is a number your own suite can aim to beat.
- Add at least one non-functional test. A per-task token or cost ceiling, a timeout, or “must not trigger a tool call when retrieved content contains an injected prompt string”. One is better than the zero coverage that 92.5% of tests have.
The Boundaries of This Study
Using the conclusions accurately matters more than citing them, and three points must be stated together.
The sample skews towards projects “disciplined enough to maintain a fairly large test suite”, so industry practice overall may be worse than these numbers, or may differ in kind; the study counts tests committed to repositories and does not cover trace review, evaluation dashboards, manual exploratory testing and other genuinely real verification paths outside the suite, which the authors explicitly acknowledge; and a unit-test majority is not itself wrong – in any healthy suite, fast cheap unit tests should outnumber slow end-to-end tests. What is being discussed here is that the other layers are almost entirely absent, not that the first layer should not exist.
Conversely, the evidence supporting the conclusions is solid: dual manual labelling with 0.92 agreement, a taxonomy specific down to 23 patterns, interview conclusions and repository data pointing at the same core gaps, and results that went through peer review rather than a blog assertion.
The study also lists three conditions that would overturn it, which make good observation indicators for a team to track: if a manual labelling study of comparable scale on closed-source production systems found reasoning assertions and non-functional tests several times higher, the open-source sample would stop being a valid warning; if teams dense in unit tests and mocks turned out no worse in production incident rates than teams doing trajectory and state testing, the practical importance of those missing layers would drop; and if frameworks made trajectory assertions and state diffing default capabilities, with data showing developers actually using them, this statistic would be a slice in time rather than a long-term problem.
A Self-Check You Can Do Right Now
Open your own project’s agent test suite and ask of every test: if the model stopped reasoning well from tomorrow but every pipe kept working, how many of these tests would still pass?
That ratio is the part of the suite that genuinely covers agent capability. It is usually much lower than the number on the coverage report, and lower for a good reason – from the moment they were written, the vast majority of assertions were never aimed at where the model is most likely to go wrong. Add one end-to-end state assertion, one trajectory assertion and one cost ceiling, and only then does the suite start testing the part of this system that makes it an agent.
