<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Defect Analysis on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/defect-analysis/</link><description>Recent content in Defect Analysis on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 04 Nov 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/defect-analysis/index.xml" rel="self" type="application/rss+xml"/><item><title>Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue</title><link>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</link><pubDate>Wed, 04 Nov 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/cover.webp" alt="Featured image of post Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue" />&lt;p>One failure, one pass, final green. This is what most CI pipelines with automatic retries produce every day, and it is also the result where information is most easily lost: the failed attempt gets overwritten in the logs, deleted during cleanup of the successful job, and the team only sees that &amp;ldquo;this build is green&amp;rdquo;. The real problem is that the reason the second attempt passed is usually not &amp;ldquo;random jitter&amp;rdquo; but some specific state, timing or data condition &amp;ndash; and users will run into it in production too.&lt;/p>
&lt;p>One approach deserves a full breakdown: write a structured record for every retry at the end of the suite and keep it separately, aggregate it weekly into a list sorted by impact, and then let a coding agent follow the records into the code to locate root causes. It does not aim to &amp;ldquo;make CI greener&amp;rdquo; but the opposite &amp;ndash; to re-expose the failures that retries pressed down.&lt;/p>
&lt;h2 id="step-one-let-the-failed-attempt-survive-success-cleanup">Step One: Let the Failed Attempt Survive Success Cleanup
&lt;/h2>&lt;p>Most CI configurations delete application logs and diagnostic artifacts after a job succeeds. That is reasonable cost control, but it happens to delete exactly the evidence most needed to investigate intermittent failures. The solution is not to keep all logs, but to write a very small JSON file for every test node that retried, at the end of the suite:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;span class="lnt">8
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;id&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb[1:2]&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;location&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb:20&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;description&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;renders a synthetic report&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;failed_attempts&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;final_result&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;passed&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;first_error&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;expected the default window size&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Two fields deserve separate mention. &lt;code>id&lt;/code> is the unique example identifier the test framework provides, and &lt;code>location&lt;/code> is the source line number. Shared examples cause several different test cases to point at the same line, so aggregating by line number alone merges unrelated failures into one lump; keeping both is what allows correct grouping.&lt;/p>
&lt;p>This JSON must be written to a directory &lt;strong>outside the scope of success-job cleanup&lt;/strong> and uploaded as a build artifact. With a common CI service, that means adding one line to the config declaring the directory as an artifact; any CI platform has an equivalent mechanism. There is only one hard constraint: even if this test ultimately passes, the retry history must be kept.&lt;/p>
&lt;p>The retry policy itself should also be written down clearly: one retry for ordinary cases, two for cases involving a JavaScript browser. The count is not the point; the point is that &lt;strong>every retry must leave a clue about why the result changed&lt;/strong>. Merely increasing the retry count only makes the report quieter, it does not answer any question.&lt;/p>
&lt;h2 id="step-two-aggregate-the-records-into-a-prioritized-weekly-queue">Step Two: Aggregate the Records into a Prioritized Weekly Queue
&lt;/h2>&lt;p>With records in hand, the next step is aggregation. A scheduled job reads the past week&amp;rsquo;s retry artifacts through the CI platform API, groups them by example identifier, and sorts descending by &lt;strong>the number of distinct pipelines in which a &amp;ldquo;failed then passed&amp;rdquo; occurred&lt;/strong>.&lt;/p>
&lt;p>There is a detail that is easy to get wrong here: if a case failed twice within the same pipeline, it counts as only one pipeline. The ranking wants to measure &amp;ldquo;how many independent builds this flakiness affected&amp;rdquo;, not &amp;ldquo;how many times it failed in total&amp;rdquo;. Keep the first error message, branch name and number of failed attempts alongside for later reading.&lt;/p>
&lt;p>One class should be counted separately: cases where &lt;strong>all attempts failed&lt;/strong>. This distinction only describes the observed run outcome and does not mean &amp;ldquo;the root cause is deterministic&amp;rdquo; &amp;ndash; an intermittent problem can equally exhaust the retry budget, so you cannot use it to attach different labels to the two classes.&lt;/p>
&lt;p>When a given example exceeds a configured weekly budget threshold, make the summary job itself fail, borrowing the team&amp;rsquo;s existing CI failure notifications to push the list in front of people. At that point, hand the report to a coding agent that can read the repository and build artifacts, and the localization phase begins.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp"
width="1024"
height="820"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_ed4aa59c7fcf933b.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="124"
data-flex-basis="299px"
>
&lt;/p>
&lt;p>The full chain from retry record generation and retention to the weekly priority queue&lt;/p>
&lt;h2 id="step-three-seeing-clearly-why-the-second-attempt-passed---four-real-cases">Step Three: Seeing Clearly &amp;ldquo;Why the Second Attempt Passed&amp;rdquo; - Four Real Cases
&lt;/h2>&lt;p>The following classes of problem share one trait: the retry did not make the test recover by itself, the &lt;strong>retry mechanism happened to reset the state that was causing the error&lt;/strong>. Without looking at the first failure, you would never know there was a defect in the product.&lt;/p>
&lt;h3 id="a-browser-restart-wiped-the-window-size-left-by-the-previous-case">A Browser Restart Wiped the Window Size Left by the Previous Case
&lt;/h3>&lt;p>A group of screenshot comparisons failed, reporting a change in screenshot dimensions, and a retry usually passed. The root cause lay in an earlier case: a billing-related test changed the browser window to a non-default size without restoring it, and the later screenshot case inherited that viewport. The retry configuration restarts the browser before rerunning, the window returns to its default size, and the second attempt passes.&lt;/p>
&lt;p>The fix is to explicitly set the default window size before every browser case, with cases needing another size changing it themselves. In this case, &amp;ldquo;the retry passed&amp;rdquo; actually provided the key clue: since a fresh browser fixed it, the state must have been brought in from outside rather than restored by the logic under test.&lt;/p>
&lt;h3 id="an-apostrophe-broke-an-auto-generated-curl-command">An Apostrophe Broke an Auto-Generated cURL Command
&lt;/h3>&lt;p>A dashboard generates example cURL commands from a template schema, and the tests execute those commands using generated data. The generated fake names occasionally contain an apostrophe, and the inline command wraps the JSON request body in single quotes &amp;ndash; the apostrophe ends the shell string early and the command breaks. The next attempt generates a different name, and the command works again.&lt;/p>
&lt;p>The same feature also has a downloadable Bash script that puts the JSON in a quoted heredoc, but incorrectly applies the escaping intended for single-quoted arguments, corrupting the request body. The two places need completely different handling: the inline command must escape apostrophes for the shell context, while the JSON inside the heredoc must be preserved verbatim.&lt;/p>
&lt;p>The corresponding regression test is well worth borrowing: replace &lt;code>curl&lt;/code> with a function, actually execute the generated script in Bash, capture the request body that would have been sent with that function, then parse and validate the original value &amp;ndash; without making any network request at all. Replay the fixed and unfixed versions with a synthetic value like &lt;code>Editor's proof&lt;/code> that contains an apostrophe: before the fix the inline command fails outright in Bash and the downloadable script runs but the request body cannot be parsed as JSON; after the fix both preserve the text as-is.&lt;/p>
&lt;p>The &amp;ldquo;shortcut&amp;rdquo; that is easiest to choose here is to change the test data so names contain no apostrophes. That would make the test more stable and keep the product broken for every legitimate input.&lt;/p>
&lt;h3 id="a-temporary-file-was-collected-before-the-render-process-read-it">A Temporary File Was Collected Before the Render Process Read It
&lt;/h3>&lt;p>An inline HTML-to-PDF renderer writes header and footer HTML to temporary files and then hands the paths to a render process. The code keeps the temp file objects in local variables while building the command; after the method returns, the command string still holds the paths, but nothing references those objects any more.&lt;/p>
&lt;p>Ruby&amp;rsquo;s &lt;code>Tempfile&lt;/code> hooks cleanup onto garbage collection. If GC happens to run before the render process opens the file, the file is already gone. Whether the test passes depends on when garbage collection occurs &amp;ndash; the most typical &amp;ldquo;seemingly random&amp;rdquo; failure.&lt;/p>
&lt;p>The fix is to keep the header and footer objects on the service instance until rendering completes, then close and delete them in an &lt;code>ensure&lt;/code> block. The regression test &lt;strong>forces GC&lt;/strong> between &amp;ldquo;build the command&amp;rdquo; and &amp;ldquo;simulate the render process reading the file&amp;rdquo;, asserting that the file content is still readable and gets cleaned up afterwards. Deliberately manufacturing a deterministic failure condition is far more reliable than waiting to get lucky in some PDF case.&lt;/p>
&lt;h3 id="rubys-clock-was-frozen-rediss-clock-was-still-running">Ruby&amp;rsquo;s Clock Was Frozen, Redis&amp;rsquo;s Clock Was Still Running
&lt;/h3>&lt;p>A rate-limit test uses counters that count within a fixed time window. Once a case crosses the window boundary, requests get counted into the new window and the expected threshold is never reached. The first fix frozen the process clock, but the counters live in Redis with an expiry time, and Redis keeps running on real time. If you freeze near the end of a window, the counter&amp;rsquo;s remaining time to live is very short and the problem persists.&lt;/p>
&lt;p>The final approach freezes at the start of the hour, aligned with the time window these cases use, so the counter gets a full expiry cycle; the cases still have to finish within that cycle. The value of this case lies in the fact that review found a second clock after the first fix &amp;ndash; the first explanation covered only half of the timing mechanisms in the system.&lt;/p>
&lt;p>The same investigation round also surfaced several other classes of problem: a background worker still running when database cleanup started; test helper functions overwriting each other&amp;rsquo;s frozen state on a shared clock library; and a sample data generator using truthiness to decide whether to skip lower-bound adjustment when the bound is zero (in JavaScript 0 is falsy, yet in this schema it is a legitimate lower bound). Each falls into the category of &amp;ldquo;a retry makes it disappear, but a user may still hit it&amp;rdquo;.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_2bea94c3ea199544.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Four real defects hidden by retries: state leakage, escaping context, object lifetime, dual clocks&lt;/p>
&lt;h2 id="step-four-draw-clear-task-boundaries-for-the-coding-agent">Step Four: Draw Clear Task Boundaries for the Coding Agent
&lt;/h2>&lt;p>When handing the weekly report to a coding agent, the task description should be specific enough to be verifiable, not &amp;ldquo;please take a look at these failures&amp;rdquo;. A reusable set of requirements is:&lt;/p>
&lt;ul>
&lt;li>Find the corresponding CI run and read the failure artifacts, then follow the test code into the application code;&lt;/li>
&lt;li>Reproduce locally where possible first, then add a regression test that is required to fail before the fix and pass after it;&lt;/li>
&lt;li>Split unrelated root causes into separate commits, so each can be reviewed and rolled back individually;&lt;/li>
&lt;li>Run the affected checks and self-review the diff, explicitly listing items that could not be reproduced, without vague hand-waving;&lt;/li>
&lt;li>Stop and wait for human approval before pushing.&lt;/li>
&lt;/ul>
&lt;p>The reason for insisting that &amp;ldquo;artifacts and code must be read together&amp;rdquo; is that neither side alone yields a conclusion: a mismatched screenshot size does not by itself explain who changed the viewport, and a missing file does not by itself explain why that Ruby object became collectable. The agent&amp;rsquo;s real value is joining those two kinds of information, not replacing the human final judgement.&lt;/p>
&lt;p>Any rule in the investigation conclusions that &amp;ldquo;future development must follow&amp;rdquo; goes into the &lt;code>AGENTS.md&lt;/code> closest to the code (or an equivalent team convention file). For example, &amp;ldquo;browser cases may change their own viewport, but the next case must start from a known size&amp;rdquo; &amp;ndash; setup code enforces it, comments explain why. Better still, anything that can be turned into a static check should not rely on memory: add a custom lint rule for the common flaky patterns and let CI block them, which works far better than hoping someone remembers an investigation from six months ago.&lt;/p>
&lt;h2 id="an-action-checklist">An Action Checklist
&lt;/h2>&lt;p>If your suite already retries, you do not need to refactor the whole system first. Just follow this order:&lt;/p>
&lt;ol>
&lt;li>At the end of the suite, write one structured record per retry, with fields covering at least the unique example identifier, source location, number of failed attempts, final result and first error;&lt;/li>
&lt;li>Put the records in a directory that success-job cleanup cannot reach, and configure it as a build artifact upload;&lt;/li>
&lt;li>Aggregate and sort weekly by &amp;ldquo;number of distinct pipelines with a failed-then-passed&amp;rdquo;, set a budget threshold, and when it is exceeded make the summary job fail and notify;&lt;/li>
&lt;li>Pick one recurring problem and demand a clear explanation of &amp;ldquo;why the second attempt passed&amp;rdquo;, rather than simply adding more retries;&lt;/li>
&lt;li>The fix must come with a regression test that reproduces deterministically, and only merges into the regular CI, QA and release flow after review passes.&lt;/li>
&lt;/ol>
&lt;p>There is no published data yet on hours saved by this process, and the fix rounds are too recent to judge how far it can push the recurrence rate down over the long run &amp;ndash; the weekly reports over the coming weeks will answer that. But its immediate effect is already clear: problems that used to require someone to set aside half a day to chase, often losing the trail halfway through, can now be investigated in parallel with other work, and coming back only requires assessing whether the explanation holds and whether the change is reasonable.&lt;/p>
&lt;p>That is exactly why the quality of code written by the agent matters more: a reproducible failure and a test that asserts the real intent are concrete targets it can anchor on. And keeping the retry history is equivalent to putting more of those targets on the table.&lt;/p></description></item></channel></rss>