<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Empirical Study on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/empirical-study/</link><description>Recent content in Empirical Study on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 28 Oct 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/empirical-study/index.xml" rel="self" type="application/rss+xml"/><item><title>Keeping the Acceptance Criteria Away From the People Who Write the Code - Conclusions After 967 Runs</title><link>https://chengxiaqiucao.github.io/en/post/143-spec-driven-verification/</link><pubDate>Wed, 28 Oct 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/143-spec-driven-verification/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/143-spec-driven-verification/cover.webp" alt="Featured image of post Keeping the Acceptance Criteria Away From the People Who Write the Code - Conclusions After 967 Runs" />&lt;p>There is a rule repeatedly validated in safety-critical fields: the people who build a system must never be the people who verify it. It is not bureaucratic ceremony but a precondition of verification validity &amp;ndash; if a person implements according to their own understanding and then verifies according to their own understanding, the two understandings are naturally identical, and verification degenerates into self-confirmation.&lt;/p>
&lt;p>When the coding work is done by an AI agent, this rule runs into a new situation: the same model can write the code, write the tests, and also deliver the conclusion that &amp;ldquo;all tests pass&amp;rdquo;. So a question worth taking seriously appears &amp;ndash; is there a way to enforce this rule inside a machine pipeline, rather than only writing it in a standard.&lt;/p>
&lt;h2 id="splitting-a-spec-into-two-halves">Splitting a Spec Into Two Halves
&lt;/h2>&lt;p>One approach splits the spec file into two parts: requirements and acceptance criteria. The coding agent only receives the requirements part, and the acceptance criteria are cut off before the file is handed over; a separate, independent testing agent writes tests based only on the acceptance criteria and does not read the implementation code.&lt;/p>
&lt;p>The two agents run in the same pipeline, but the information they see does not overlap:&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig1_hu_7d4a9e891d06bd6e.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig1_hu_12455b82a278d43.webp 480w, https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig1_hu_7d4a9e891d06bd6e.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>The two information channels do not overlap and only meet at the test failure message&lt;/p>
&lt;ul>
&lt;li>Coding agent: requirements + test data, no acceptance criteria;&lt;/li>
&lt;li>Testing agent: requirements + acceptance criteria + test data, no implementation;&lt;/li>
&lt;li>The only place the two meet is the failure message returned after the test run.&lt;/li>
&lt;/ul>
&lt;p>The key is that the &amp;ldquo;cut off&amp;rdquo; step happens in code, not by asking the model in a prompt to be self-disciplined. It is a pipeline property that can be demonstrated and audited, not a promise.&lt;/p>
&lt;h2 id="one-complete-run-the-failure-message-is-just-one-line">One Complete Run: The Failure Message Is Just One Line
&lt;/h2>&lt;p>Looking at a fragment of a real run is more useful than describing it.&lt;/p>
&lt;p>The task is: read recorded radar samples, reject invalid samples, compute the time headway to the vehicle ahead, and warn when the headway is below two seconds. Based only on the requirements, the coding agent wrote this range check &amp;ndash; it kept the upper bound but let the zero-gap case through:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-python" data-lang="python">&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="n">gap_m&lt;/span> &lt;span class="o">&amp;lt;&lt;/span> &lt;span class="mi">0&lt;/span> &lt;span class="ow">or&lt;/span> &lt;span class="n">gap_m&lt;/span> &lt;span class="o">&amp;gt;&lt;/span> &lt;span class="mi">250&lt;/span>&lt;span class="p">:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="n">rejected&lt;/span>&lt;span class="o">.&lt;/span>&lt;span class="n">append&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="o">...&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="c1"># the row, with a reason attached&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>And the test fixture CSV contains a row S005, a radar sample whose gap is exactly 0 metres. The test generated from the acceptance criteria requires that row to be rejected, and it was not:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">E AssertionError: &amp;#39;S005&amp;#39; was not rejected
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>That is the entire feedback the coding agent gets: which test failed and what the test asserted, with no hint about the underlying rule. The model inferred from it that an exactly zero gap is not a valid measurement.&lt;/p>
&lt;p>Then it produced a PATCH changing one character:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-diff" data-lang="diff">&lt;span class="line">&lt;span class="cl">&lt;span class="gd">- if gap_m &amp;lt; 0 or gap_m &amp;gt; 250:
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="gd">&lt;/span>&lt;span class="gi">+ if gap_m &amp;lt;= 0 or gap_m &amp;gt; 250:
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>From writing the test to all stage suites going green, the whole round took under a minute, with fewer than ten model calls and a cost of a fraction of a cent. But what really deserves attention is not the speed but the causality: the agent tightened the lower bound because a test it had never seen put the 0-metre gap into the &amp;ldquo;must reject&amp;rdquo; pile, not because some written standard told it that zero is invalid.&lt;/p>
&lt;p>Had it seen that standard directly, it would have written the correct check on the first attempt and the suite would have passed the first time &amp;ndash; and that green result would prove nothing, like a student scoring full marks on an exam they set themselves.&lt;/p>
&lt;h2 id="the-easiest-mistake-hiding-a-decision-in-the-acceptance-criteria">The Easiest Mistake: Hiding a Decision in the Acceptance Criteria
&lt;/h2>&lt;p>The run above actually comes from the second version of the task. The problem exposed by the first version is more worth recording, because from the outside it looks a lot like &amp;ldquo;keep editing the spec until it passes&amp;rdquo;.&lt;/p>
&lt;p>The first version&amp;rsquo;s requirements only said: warn when the headway violates the &amp;ldquo;two-second rule&amp;rdquo;. The question &amp;ldquo;should a headway of exactly 2.00 seconds raise a warning&amp;rdquo; was placed in the hidden acceptance criteria:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># Version 1. The boundary is settled in the half the coding agent never sees.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">requirements&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;Flag a sample as following too closely when its time headway breaks the two-second rule&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">acceptance_criteria&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="c"># cut before the coding agent is handed the file&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;A headway of exactly 2.00 s does not raise a warning&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Back to that criterion: given only the requirements, would two competent developers disagree about 2.00 seconds? Very easily. &amp;ldquo;Breaks the two-second rule&amp;rdquo; can be read as &amp;ldquo;below&amp;rdquo; or as &amp;ldquo;not above&amp;rdquo;. That is a decision, and a decision hidden inside the test criteria is exactly the mistake this split is meant to avoid.&lt;/p>
&lt;p>The first batch of runs showed the cost: only three of ten random seeds converged. For the other seven, the agent writing the tests resolved the ambiguity the wrong way and wrote a test that warns at exactly 2.00 seconds &amp;ndash; which contradicts the acceptance criterion it was itself given. That is worth pausing over: the testing agent held both the vague requirement and the clear acceptance criterion, and still went the wrong way. A requirement loose enough to support two readings does not just make the coder guess; it also drags the test writer away from the criterion it was given. And no implementation can pass a test that contradicts its own criterion.&lt;/p>
&lt;p>The correct fix only moves the decision itself, and only the decision. The acceptance criteria kept every consequence derived from that decision, and the other speed and gap constraints were not touched by a single character, including &amp;ldquo;zero gap must be rejected&amp;rdquo;. Nothing was weakened and nothing was deleted.&lt;/p>
&lt;p>After moving that one sentence, the same ten seeds converged in all ten cases on the first sweep; the zero-gap fix above still occurred in seven of them, again caught by a test the coding agent had never seen.&lt;/p>
&lt;h2 id="the-line-between-fixing-the-spec-and-lowering-the-bar">The Line Between Fixing the Spec and Lowering the Bar
&lt;/h2>&lt;p>There is a very fine but very important line here.&lt;/p>
&lt;p>Moving a decision that the coder needs is a correction the spec owed all along, and it can justify itself with the single question &amp;ldquo;would two competent developers disagree&amp;rdquo;, without looking at any code. Whereas deleting that criterion, loosening it, or copying its boundary value into the requirements would also make the run pass &amp;ndash; but then nothing is proven.&lt;/p>
&lt;p>This line is especially dangerous in the AI era because the cost of &amp;ldquo;make it green&amp;rdquo; has been pushed down to a single conversation. When tests do not pass, the least effortful option for an agent is often to change the test or the criterion rather than change the implementation. The pipeline must carry a hard rule: when tests do not pass, you may change the implementation, you may change the wording of the requirements, but you may not loosen the acceptance criteria. What the tool does at this point is &amp;ndash; exit non-zero when a run does not converge, name the tests blocking it, and deliver nothing.&lt;/p>
&lt;h2 id="what-changes-when-the-code-already-exists">What Changes When the Code Already Exists
&lt;/h2>&lt;p>Everything above assumes the implementation is written during this run, which is the only situation where isolation can truly be enforced: one process drives both agents and the acceptance criteria are stripped out of one side&amp;rsquo;s prompt by code.&lt;/p>
&lt;p>But most code in the real world is not like that. It already exists, written months ago by a person or by an assistant, and the team wants &amp;ldquo;this piece of code checked once&amp;rdquo;.&lt;/p>
&lt;p>The same two-agent workflow can point at existing code, with the suite still generated from the criteria by an agent that never read the implementation. What changes is the weight of each outcome:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Outcome&lt;/th>
&lt;th>Code written during the run&lt;/th>
&lt;th>Code that already exists&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Test fails&lt;/td>
&lt;td>A real finding&lt;/td>
&lt;td>Still a real finding&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Test passes&lt;/td>
&lt;td>The code satisfied a criterion it never saw&lt;/td>
&lt;td>The code agrees with a criterion its author may have seen&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Failure is reliable in both cases, because the test was written from the criterion without reading the implementation, so any mismatch is a real divergence between code and spec. Only &amp;ldquo;pass&amp;rdquo; weakens, and it weakens for a reason unrelated to the tool: the person who wrote that code may have had the acceptance criteria open in a window next to it at the time.&lt;/p>
&lt;p>This is a claim about timelines, so it is worth drawing clearly. For existing code there is no way to enforce order after the fact; the only thing you can ask is version control, which has been taking notes all along:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">criteria file, last commit ───────────┐
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> │ in this order, the criteria were
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">implementation, first commit ─────────┘ not written against code that was already in the repository
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>A run against existing code now asks exactly this question, and prints the answer together with its limits &amp;ndash; because the limits are most of the truth. A commit date is not a writing date: code can sit in a working tree for weeks without being committed, so this ordering narrows the risk that &amp;ldquo;the criterion was reverse-engineered from the code&amp;rdquo; but does not eliminate it; and the criterion being committed first says nothing at all about whether the developer was allowed to read it before implementing.&lt;/p>
&lt;p>So this evidence is most valuable when the result is bad: a criterion revised after the code landed is a question someone should answer. That asymmetry matches the table above &amp;ndash; better to deliver a check that states its own limitations than to print a green tick that lets the reader believe it proves more.&lt;/p>
&lt;p>Version control does know, and can state clearly, one thing: who committed each side. Two different people, which is exactly the separation of duties the whole method borrows.&lt;/p>
&lt;h2 id="the-numbers-from-967-runs">The Numbers From 967 Runs
&lt;/h2>&lt;p>The evidence so far is this: on the tool&amp;rsquo;s built-in sample task, three batches totalling 967 runs (using a small, cheap model so the sweeps could be repeated affordably) produced code that passed all integration and system tests in about eighty percent of runs, and passed every stage including unit tests in about sixty percent. After hardening the process and making the task harder in two places, a fourth batch of 390 runs reproduced the same two numbers.&lt;/p>
&lt;p>That number is listed separately rather than merged with the earlier batches &amp;ndash; describing two different systems with one number is exactly how results get retracted later. Also, these runs used a cheap model, so both percentages should be read as a lower bound on performance.&lt;/p>
&lt;p>The gap between the two stages comes mainly from unit tests, which run last and are the strictest. Most of the remaining problems can be covered by two questions:&lt;/p>
&lt;ul>
&lt;li>If the FIX and PATCH produced on each iteration are nearly identical, the agent is guessing at a decision nobody told it &amp;ndash; the problem is in the spec;&lt;/li>
&lt;li>If a rule cannot be triggered at all by your sample data, you get a test that passes no matter how the code is written &amp;ndash; the problem is in the data, not the code.&lt;/li>
&lt;/ul>
&lt;h2 id="edge-cases-and-experimental-design-why-passes-on-average-is-an-illusion">Edge Cases and Experimental Design: Why &amp;ldquo;Passes on Average&amp;rdquo; Is an Illusion
&lt;/h2>&lt;p>The concrete form this separation-of-duties principle takes in a production driver-assistance project is the edge case. One class of problem recurs: the perception module met the requirement &amp;ndash; as the developers themselves understood it &amp;ndash; but did not measure what really needed measuring.&lt;/p>
&lt;p>A detection accuracy requirement will state that the system must stay within some bound. The implementation team read it as &amp;ldquo;the average accuracy across one recorded drive&amp;rdquo;, a very natural reading when you are building as you go, and their own tests matched that reading. An independent team comparing the same data against ground truth based on the requirement itself (rather than the developers&amp;rsquo; reading) exposed failures that average accuracy over a whole drive can never reveal.&lt;/p>
&lt;p>For example, the acceptance criteria would ask: does it hold on every single frame? Including those frames where a vehicle cuts in from the edge of the sensor field of view, or some occlusion occurs. Averaged over the whole drive, such modules pass easily; frame by frame they do not, because thousands of easy frames drown out the few dozen that determine whether the system is safe. The requirement never said whether the bound was a mean or a worst case, the implementers picked the reading their code naturally produced, and they honestly verified themselves against that reading. Developers usually do not have the time or resources to check every edge case personally, and tend to verify a few important scenarios before moving to the next task.&lt;/p>
&lt;p>This is the operational design domain (ODD) problem. Defining edge cases accurately and completely in a complex system is an enormous amount of work. The parameter combinations are far too many for brute-force coverage to be practical, and only an ideal oracle could generate perfect coverage over all scenarios. So you need to follow design of experiments (DOE) principles to cover the ODD over the whole test space in an optimal way.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig2_hu_ffdcc34aea1a1732.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig2_hu_d1b52afc884d30be.webp 480w, https://chengxiaqiucao.github.io/post/143-spec-driven-verification/03-fig2_hu_ffdcc34aea1a1732.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Sample data is evenly distributed in the space, but rare edge regions have almost no coverage points&lt;/p>
&lt;p>The same problem reproduced at a smaller scale inside this tool: the faults planted to measure &amp;ldquo;whether clearer criteria catch more faults&amp;rdquo; were for the most part invisible to any test suite &amp;ndash; because the inputs the acceptance criteria describe never exist in the fixture data. &lt;strong>A criterion that nothing can trigger is not a criterion.&lt;/strong> If the driving data under test never includes heavy rain, then no matter how good the test team is, they learn nothing about heavy rain.&lt;/p>
&lt;h2 id="validation-still-needs-a-human-in-the-loop">Validation Still Needs a Human in the Loop
&lt;/h2>&lt;p>One point must be made clear: everything above belongs to verification &amp;ndash; given a written criterion, does the code satisfy it, checked by a party that could not see the criterion when it was built. It does not touch validation at all: whether the criterion itself describes what should happen.&lt;/p>
&lt;p>A tool can force a coding agent to never see the acceptance criteria. No tool can tell you whether those criteria are the right ones to write. For now that still requires a person who understands the domain, understands the users, and understands why the system exists at all. It also requires writing &amp;ldquo;correct behaviour&amp;rdquo; as concrete, checkable descriptions, including the edge cases that careless readings miss. AI can of course assist with all of this, but it still needs human experts to supervise, approve the spec and stay in the loop as the spec evolves.&lt;/p>
&lt;h2 id="three-things-you-can-borrow-right-away">Three Things You Can Borrow Right Away
&lt;/h2>&lt;p>First, write &amp;ldquo;decisions&amp;rdquo; and &amp;ldquo;consequences&amp;rdquo; separately in the spec. Any statement two competent developers would disagree about is a decision that must be settled on the requirements side; the acceptance criteria keep only the checkable consequences derived from that decision. The test is simple: read the requirement and ask yourself whether two reasonable readings exist.&lt;/p>
&lt;p>Second, add a hard constraint to the pipeline &amp;ndash; do not allow loosening the criteria when a run does not converge. Changing the implementation is allowed, clarifying the requirements is allowed, but &amp;ldquo;delete or loosen an acceptance criterion&amp;rdquo; becomes an action requiring human approval. Without that constraint, an agent will always take the least effortful path.&lt;/p>
&lt;p>Third, check that your test data can trigger every rule. When you write a criterion, ask at the same time: which row of sample data would make this fail? A criterion with no answer to that is not a criterion.&lt;/p>
&lt;p>More fundamentally, verification and validation are not two steps in a process but a professional mindset about &amp;ldquo;who has the right to answer which question, and why&amp;rdquo;. What deserves vigilance is not trust itself but methodological omission: a check performed by the builder is not really a check. That holds whether the builder is a person rushing a deadline or an LLM with a context window and a budget limit, and it will still hold in the next generation of software development.&lt;/p></description></item></channel></rss>