<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Flaky on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/flaky/</link><description>Recent content in Flaky on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 04 Nov 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/flaky/index.xml" rel="self" type="application/rss+xml"/><item><title>Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue</title><link>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</link><pubDate>Wed, 04 Nov 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/cover.webp" alt="Featured image of post Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue" />&lt;p>One failure, one pass, final green. This is what most CI pipelines with automatic retries produce every day, and it is also the result where information is most easily lost: the failed attempt gets overwritten in the logs, deleted during cleanup of the successful job, and the team only sees that &amp;ldquo;this build is green&amp;rdquo;. The real problem is that the reason the second attempt passed is usually not &amp;ldquo;random jitter&amp;rdquo; but some specific state, timing or data condition &amp;ndash; and users will run into it in production too.&lt;/p>
&lt;p>One approach deserves a full breakdown: write a structured record for every retry at the end of the suite and keep it separately, aggregate it weekly into a list sorted by impact, and then let a coding agent follow the records into the code to locate root causes. It does not aim to &amp;ldquo;make CI greener&amp;rdquo; but the opposite &amp;ndash; to re-expose the failures that retries pressed down.&lt;/p>
&lt;h2 id="step-one-let-the-failed-attempt-survive-success-cleanup">Step One: Let the Failed Attempt Survive Success Cleanup
&lt;/h2>&lt;p>Most CI configurations delete application logs and diagnostic artifacts after a job succeeds. That is reasonable cost control, but it happens to delete exactly the evidence most needed to investigate intermittent failures. The solution is not to keep all logs, but to write a very small JSON file for every test node that retried, at the end of the suite:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;span class="lnt">8
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;id&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb[1:2]&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;location&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb:20&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;description&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;renders a synthetic report&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;failed_attempts&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;final_result&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;passed&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;first_error&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;expected the default window size&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Two fields deserve separate mention. &lt;code>id&lt;/code> is the unique example identifier the test framework provides, and &lt;code>location&lt;/code> is the source line number. Shared examples cause several different test cases to point at the same line, so aggregating by line number alone merges unrelated failures into one lump; keeping both is what allows correct grouping.&lt;/p>
&lt;p>This JSON must be written to a directory &lt;strong>outside the scope of success-job cleanup&lt;/strong> and uploaded as a build artifact. With a common CI service, that means adding one line to the config declaring the directory as an artifact; any CI platform has an equivalent mechanism. There is only one hard constraint: even if this test ultimately passes, the retry history must be kept.&lt;/p>
&lt;p>The retry policy itself should also be written down clearly: one retry for ordinary cases, two for cases involving a JavaScript browser. The count is not the point; the point is that &lt;strong>every retry must leave a clue about why the result changed&lt;/strong>. Merely increasing the retry count only makes the report quieter, it does not answer any question.&lt;/p>
&lt;h2 id="step-two-aggregate-the-records-into-a-prioritized-weekly-queue">Step Two: Aggregate the Records into a Prioritized Weekly Queue
&lt;/h2>&lt;p>With records in hand, the next step is aggregation. A scheduled job reads the past week&amp;rsquo;s retry artifacts through the CI platform API, groups them by example identifier, and sorts descending by &lt;strong>the number of distinct pipelines in which a &amp;ldquo;failed then passed&amp;rdquo; occurred&lt;/strong>.&lt;/p>
&lt;p>There is a detail that is easy to get wrong here: if a case failed twice within the same pipeline, it counts as only one pipeline. The ranking wants to measure &amp;ldquo;how many independent builds this flakiness affected&amp;rdquo;, not &amp;ldquo;how many times it failed in total&amp;rdquo;. Keep the first error message, branch name and number of failed attempts alongside for later reading.&lt;/p>
&lt;p>One class should be counted separately: cases where &lt;strong>all attempts failed&lt;/strong>. This distinction only describes the observed run outcome and does not mean &amp;ldquo;the root cause is deterministic&amp;rdquo; &amp;ndash; an intermittent problem can equally exhaust the retry budget, so you cannot use it to attach different labels to the two classes.&lt;/p>
&lt;p>When a given example exceeds a configured weekly budget threshold, make the summary job itself fail, borrowing the team&amp;rsquo;s existing CI failure notifications to push the list in front of people. At that point, hand the report to a coding agent that can read the repository and build artifacts, and the localization phase begins.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp"
width="1024"
height="820"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_ed4aa59c7fcf933b.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="124"
data-flex-basis="299px"
>
&lt;/p>
&lt;p>The full chain from retry record generation and retention to the weekly priority queue&lt;/p>
&lt;h2 id="step-three-seeing-clearly-why-the-second-attempt-passed---four-real-cases">Step Three: Seeing Clearly &amp;ldquo;Why the Second Attempt Passed&amp;rdquo; - Four Real Cases
&lt;/h2>&lt;p>The following classes of problem share one trait: the retry did not make the test recover by itself, the &lt;strong>retry mechanism happened to reset the state that was causing the error&lt;/strong>. Without looking at the first failure, you would never know there was a defect in the product.&lt;/p>
&lt;h3 id="a-browser-restart-wiped-the-window-size-left-by-the-previous-case">A Browser Restart Wiped the Window Size Left by the Previous Case
&lt;/h3>&lt;p>A group of screenshot comparisons failed, reporting a change in screenshot dimensions, and a retry usually passed. The root cause lay in an earlier case: a billing-related test changed the browser window to a non-default size without restoring it, and the later screenshot case inherited that viewport. The retry configuration restarts the browser before rerunning, the window returns to its default size, and the second attempt passes.&lt;/p>
&lt;p>The fix is to explicitly set the default window size before every browser case, with cases needing another size changing it themselves. In this case, &amp;ldquo;the retry passed&amp;rdquo; actually provided the key clue: since a fresh browser fixed it, the state must have been brought in from outside rather than restored by the logic under test.&lt;/p>
&lt;h3 id="an-apostrophe-broke-an-auto-generated-curl-command">An Apostrophe Broke an Auto-Generated cURL Command
&lt;/h3>&lt;p>A dashboard generates example cURL commands from a template schema, and the tests execute those commands using generated data. The generated fake names occasionally contain an apostrophe, and the inline command wraps the JSON request body in single quotes &amp;ndash; the apostrophe ends the shell string early and the command breaks. The next attempt generates a different name, and the command works again.&lt;/p>
&lt;p>The same feature also has a downloadable Bash script that puts the JSON in a quoted heredoc, but incorrectly applies the escaping intended for single-quoted arguments, corrupting the request body. The two places need completely different handling: the inline command must escape apostrophes for the shell context, while the JSON inside the heredoc must be preserved verbatim.&lt;/p>
&lt;p>The corresponding regression test is well worth borrowing: replace &lt;code>curl&lt;/code> with a function, actually execute the generated script in Bash, capture the request body that would have been sent with that function, then parse and validate the original value &amp;ndash; without making any network request at all. Replay the fixed and unfixed versions with a synthetic value like &lt;code>Editor's proof&lt;/code> that contains an apostrophe: before the fix the inline command fails outright in Bash and the downloadable script runs but the request body cannot be parsed as JSON; after the fix both preserve the text as-is.&lt;/p>
&lt;p>The &amp;ldquo;shortcut&amp;rdquo; that is easiest to choose here is to change the test data so names contain no apostrophes. That would make the test more stable and keep the product broken for every legitimate input.&lt;/p>
&lt;h3 id="a-temporary-file-was-collected-before-the-render-process-read-it">A Temporary File Was Collected Before the Render Process Read It
&lt;/h3>&lt;p>An inline HTML-to-PDF renderer writes header and footer HTML to temporary files and then hands the paths to a render process. The code keeps the temp file objects in local variables while building the command; after the method returns, the command string still holds the paths, but nothing references those objects any more.&lt;/p>
&lt;p>Ruby&amp;rsquo;s &lt;code>Tempfile&lt;/code> hooks cleanup onto garbage collection. If GC happens to run before the render process opens the file, the file is already gone. Whether the test passes depends on when garbage collection occurs &amp;ndash; the most typical &amp;ldquo;seemingly random&amp;rdquo; failure.&lt;/p>
&lt;p>The fix is to keep the header and footer objects on the service instance until rendering completes, then close and delete them in an &lt;code>ensure&lt;/code> block. The regression test &lt;strong>forces GC&lt;/strong> between &amp;ldquo;build the command&amp;rdquo; and &amp;ldquo;simulate the render process reading the file&amp;rdquo;, asserting that the file content is still readable and gets cleaned up afterwards. Deliberately manufacturing a deterministic failure condition is far more reliable than waiting to get lucky in some PDF case.&lt;/p>
&lt;h3 id="rubys-clock-was-frozen-rediss-clock-was-still-running">Ruby&amp;rsquo;s Clock Was Frozen, Redis&amp;rsquo;s Clock Was Still Running
&lt;/h3>&lt;p>A rate-limit test uses counters that count within a fixed time window. Once a case crosses the window boundary, requests get counted into the new window and the expected threshold is never reached. The first fix frozen the process clock, but the counters live in Redis with an expiry time, and Redis keeps running on real time. If you freeze near the end of a window, the counter&amp;rsquo;s remaining time to live is very short and the problem persists.&lt;/p>
&lt;p>The final approach freezes at the start of the hour, aligned with the time window these cases use, so the counter gets a full expiry cycle; the cases still have to finish within that cycle. The value of this case lies in the fact that review found a second clock after the first fix &amp;ndash; the first explanation covered only half of the timing mechanisms in the system.&lt;/p>
&lt;p>The same investigation round also surfaced several other classes of problem: a background worker still running when database cleanup started; test helper functions overwriting each other&amp;rsquo;s frozen state on a shared clock library; and a sample data generator using truthiness to decide whether to skip lower-bound adjustment when the bound is zero (in JavaScript 0 is falsy, yet in this schema it is a legitimate lower bound). Each falls into the category of &amp;ldquo;a retry makes it disappear, but a user may still hit it&amp;rdquo;.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_2bea94c3ea199544.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Four real defects hidden by retries: state leakage, escaping context, object lifetime, dual clocks&lt;/p>
&lt;h2 id="step-four-draw-clear-task-boundaries-for-the-coding-agent">Step Four: Draw Clear Task Boundaries for the Coding Agent
&lt;/h2>&lt;p>When handing the weekly report to a coding agent, the task description should be specific enough to be verifiable, not &amp;ldquo;please take a look at these failures&amp;rdquo;. A reusable set of requirements is:&lt;/p>
&lt;ul>
&lt;li>Find the corresponding CI run and read the failure artifacts, then follow the test code into the application code;&lt;/li>
&lt;li>Reproduce locally where possible first, then add a regression test that is required to fail before the fix and pass after it;&lt;/li>
&lt;li>Split unrelated root causes into separate commits, so each can be reviewed and rolled back individually;&lt;/li>
&lt;li>Run the affected checks and self-review the diff, explicitly listing items that could not be reproduced, without vague hand-waving;&lt;/li>
&lt;li>Stop and wait for human approval before pushing.&lt;/li>
&lt;/ul>
&lt;p>The reason for insisting that &amp;ldquo;artifacts and code must be read together&amp;rdquo; is that neither side alone yields a conclusion: a mismatched screenshot size does not by itself explain who changed the viewport, and a missing file does not by itself explain why that Ruby object became collectable. The agent&amp;rsquo;s real value is joining those two kinds of information, not replacing the human final judgement.&lt;/p>
&lt;p>Any rule in the investigation conclusions that &amp;ldquo;future development must follow&amp;rdquo; goes into the &lt;code>AGENTS.md&lt;/code> closest to the code (or an equivalent team convention file). For example, &amp;ldquo;browser cases may change their own viewport, but the next case must start from a known size&amp;rdquo; &amp;ndash; setup code enforces it, comments explain why. Better still, anything that can be turned into a static check should not rely on memory: add a custom lint rule for the common flaky patterns and let CI block them, which works far better than hoping someone remembers an investigation from six months ago.&lt;/p>
&lt;h2 id="an-action-checklist">An Action Checklist
&lt;/h2>&lt;p>If your suite already retries, you do not need to refactor the whole system first. Just follow this order:&lt;/p>
&lt;ol>
&lt;li>At the end of the suite, write one structured record per retry, with fields covering at least the unique example identifier, source location, number of failed attempts, final result and first error;&lt;/li>
&lt;li>Put the records in a directory that success-job cleanup cannot reach, and configure it as a build artifact upload;&lt;/li>
&lt;li>Aggregate and sort weekly by &amp;ldquo;number of distinct pipelines with a failed-then-passed&amp;rdquo;, set a budget threshold, and when it is exceeded make the summary job fail and notify;&lt;/li>
&lt;li>Pick one recurring problem and demand a clear explanation of &amp;ldquo;why the second attempt passed&amp;rdquo;, rather than simply adding more retries;&lt;/li>
&lt;li>The fix must come with a regression test that reproduces deterministically, and only merges into the regular CI, QA and release flow after review passes.&lt;/li>
&lt;/ol>
&lt;p>There is no published data yet on hours saved by this process, and the fix rounds are too recent to judge how far it can push the recurrence rate down over the long run &amp;ndash; the weekly reports over the coming weeks will answer that. But its immediate effect is already clear: problems that used to require someone to set aside half a day to chase, often losing the trail halfway through, can now be investigated in parallel with other work, and coming back only requires assessing whether the explanation holds and whether the change is reasonable.&lt;/p>
&lt;p>That is exactly why the quality of code written by the agent matters more: a reproducible failure and a test that asserts the real intent are concrete targets it can anchor on. And keeping the retry history is equivalent to putting more of those targets on the table.&lt;/p></description></item><item><title>Why Mobile E2E Tests Go Flaky - Four Failure Modes Web Testing Never Meets</title><link>https://chengxiaqiucao.github.io/en/post/145-mobile-e2e-flakiness/</link><pubDate>Wed, 28 Oct 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/145-mobile-e2e-flakiness/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/cover.webp" alt="Featured image of post Why Mobile E2E Tests Go Flaky - Four Failure Modes Web Testing Never Meets" />&lt;p>Retries are not a fix. Longer timeouts are not a fix. A quarantine list is not a fix. And that shared Slack channel the team set up specifically for &amp;ldquo;known flaky, safe to ignore&amp;rdquo;, that is even less of a fix.&lt;/p>
&lt;p>Most testing teams agree with these sentences, but the real issue is this: mobile E2E tests are unstable for reasons that are simply not of the same kind as those behind web tests; and every solution borrowed from the web playbook makes the problem structurally worse.&lt;/p>
&lt;p>A recurring sequence looks like this: a team takes over an end-to-end suite, adds automatic retries when tests start failing, lengthens timeouts when retries are not enough, and finally stops trusting the cases altogether. Nobody looks when CI goes red, and nobody knows whether it is a real defect or &amp;ldquo;just another ordinary Tuesday&amp;rdquo;.&lt;/p>
&lt;p>This is not a discipline problem. The problem is that mobile E2E has four failure modes that simply do not exist on the web, and web-first fixes drag the wrong mental model along with them.&lt;/p>
&lt;h2 id="mobile-instability-is-a-different-problem">Mobile Instability Is a Different Problem
&lt;/h2>&lt;p>Web automation runs inside a browser. The browser is a sandboxed, largely deterministic environment: the DOM is queryable, animations can be disabled, network requests can be intercepted, and the rendering engine behaves predictably across runs. When web tests go flaky, the causes are long since well studied &amp;ndash; async waits, test order dependencies, shared state pollution; every testing textbook covers them.&lt;/p>
&lt;p>Mobile automation runs on top of a whole operating system, and that difference invalidates every assumption inherited from web testing.&lt;/p>
&lt;p>A mobile app test has to cope with system-level interruptions the test script never triggered; has to cope with a gesture model where &amp;ldquo;a 10 millisecond difference in click timing changes the result&amp;rdquo;; has to survive CI emulators that behave differently on the 47th run than on the first; and has to produce consistent results across a device ecosystem that, on Android alone, spans thousands of vendors and dozens of still-active OS versions.&lt;/p>
&lt;p>Fixing mobile flakiness with web solutions is like diagnosing an aircraft hydraulic fault with an automobile brake repair manual: the vocabulary overlaps, the physics do not.&lt;/p>
&lt;p>Those web-oriented fixes (unified async waiting patterns, isolated test state, enforced test order) are correct for web problems. Applied to mobile, they probably solve half the problem; the other half has different causes and needs different solutions.&lt;/p>
&lt;p>The data shows this has not improved as tooling matured. An industry report on mobile development covering three years and more than ten million builds showed that the share of mobile teams reporting flaky tests rose from 10% in 2022 to 26% in 2025. The more complex the pipeline, the worse the instability.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_a35256f31869fbc3.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_53f7eb880eafdad6.webp 480w, https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_a35256f31869fbc3.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Top: emulator responsiveness keeps degrading as a run progresses; bottom: the decision boundaries of gesture timing are not evenly spaced&lt;/p>
&lt;h2 id="four-failure-modes-web-testing-has-never-seen">Four Failure Modes Web Testing Has Never Seen
&lt;/h2>&lt;h3 id="one-emulator-behaviour-drift">One: Emulator Behaviour Drift
&lt;/h3>&lt;p>CI pipelines run on emulators, not real devices. An emulator is a software simulation of hardware, and like any software running in a resource-constrained CI environment, its behaviour degrades continuously &lt;strong>over the course of a single test run&lt;/strong>.&lt;/p>
&lt;p>At the start of a suite the emulator responds well: GPU acceleration is warm, memory is clean, the virtual CPU has headroom. Forty cases later the same emulator may be under memory pressure, the virtual CPU may be throttling, or it may start dropping gesture events because the event queue is backed up. The test code is identical; the environment has drifted.&lt;/p>
&lt;p>Web CI environments do not have a problem of the same scale, because browser instances are lightweight and stateless. An emulator runs an entire mobile operating system: camera drivers, Bluetooth stack, network simulation, GPS simulation &amp;ndash; all of it runs even if your test only touches one login page.&lt;/p>
&lt;p>The typical symptom: cases at positions 1-15 pass reliably, and by positions 30-50 they start failing intermittently. The developer&amp;rsquo;s reaction is to add retries; retries occasionally work because the second attempt happens to catch a moment when the emulator state is slightly better; and the root cause &amp;ndash; the emulator changed behaviour mid-run &amp;ndash; is never addressed.&lt;/p>
&lt;h3 id="two-gesture-timing-sensitivity">Two: Gesture Timing Sensitivity
&lt;/h3>&lt;p>The essence of mobile interaction is the gesture. A tap is not a click and a swipe is not a scroll, and that difference matters far more than it sounds.&lt;/p>
&lt;p>A browser click triggers a discrete set of events (mousedown, mouseup, click) that the DOM handles predictably. A mobile gesture is a stream of touch events with time attributes, interpreted by the operating system in context: holding for 100 milliseconds is a tap, holding for 500 milliseconds triggers a long-press context menu; a swipe that starts 5 pixels further left may be recognised as a scroll rather than a swipe on a particular device or OS version.&lt;/p>
&lt;p>Automation frameworks do expose gesture APIs, but the underlying timing behaviour differs between iOS and Android, between OS versions, and between emulators and real devices. A gesture command that is stable on a local emulator may fail on a CI host with different virtualisation characteristics. Long lists usually expose this difference first, because a fast swipe crosses a different number of rows each time &amp;ndash; which is why infinite-scroll tests usually swipe one step at a time and then wait for the target cell to appear.&lt;/p>
&lt;p>The web does not have this problem; click events are abstracted away at the browser layer. Mobile developers have no equivalent abstraction layer and must work directly against an OS gesture model that varies by implementation.&lt;/p>
&lt;h3 id="three-system-permission-dialogs">Three: System Permission Dialogs
&lt;/h3>&lt;p>iOS and Android pop system-level permission dialogs during app flows: camera, microphone, location, notifications, contacts, Bluetooth. These dialogs are not triggered by your test; the operating system decides when to show them based on internal state.&lt;/p>
&lt;p>Once a dialog appears mid-test it covers the interface. The script tries to click an element that is now obscured, the click fails, the element is not found, and the test reports a failure that has nothing to do with the code under test.&lt;/p>
&lt;p>When the dialog appears is not fully deterministic: a clean emulator state may not show the notification permission on the first run but may show it on the third, because by then the app has been installed long enough for the system to decide it is time to ask. Teams respond by adding &amp;ldquo;dismiss the dialog&amp;rdquo; logic to test setup; but system behaviour changes with iOS and Android versions, so a setup script that works on one version silently stops working on another.&lt;/p>
&lt;p>This class of flakiness is especially expensive, because the failure report points at the wrong element &amp;ndash; the one the script wanted to click, not the dialog that actually blocked it.&lt;/p>
&lt;h3 id="four-network-conditions-and-device-fragmentation">Four: Network Conditions and Device Fragmentation
&lt;/h3>&lt;p>Mobile apps run under network conditions web apps largely avoid: switching between WiFi and cellular, the app being backgrounded mid-request, carrier-specific DNS behaviour, and latency that varies with signal quality.&lt;/p>
&lt;p>A test that passes on a WiFi-only emulator may fail on a real device when it switches to cellular mid-flow; a test that handles a 200 millisecond API response may fail in a CI environment whose network simulation is configured differently. These are not async waiting problems but &lt;strong>environment fidelity problems&lt;/strong>: the test environment and real operating conditions do not match.&lt;/p>
&lt;p>Device fragmentation amplifies it further. The Android ecosystem stretches from entry-level phones with 1GB of memory running Android 9 all the way to current flagship hardware. A test designed for &amp;ldquo;a high-spec virtual machine plus an Android 14 emulator&amp;rdquo; will meet differences in rendering timing, memory management and garbage collection behaviour on real entry-level hardware. Screen density differences alone are enough to make a click coordinate that works on one device miss on another.&lt;/p>
&lt;p>Selector-based suites assume the view hierarchy is stable across devices &amp;ndash; and it frequently is not.&lt;/p>
&lt;h2 id="why-script-level-fixes-make-things-worse">Why Script-Level Fixes Make Things Worse
&lt;/h2>&lt;p>Facing mobile flakiness, the standard move is a predictable escalation path: retry first, because it is the least effortful lever; then add timeouts, because more time usually lets emulator drift and gesture timing eventually slip through; then quarantine, because some cases are reliably unreliable and the team simply stops expecting them to pass.&lt;/p>
&lt;p>Each has a cost, and none of it shows up on the CI dashboard.&lt;/p>
&lt;p>&lt;strong>Retries mask real failures.&lt;/strong> A case that passes on the third retry is a flaky case that got lucky; it may equally be a real regression buried by the retry logic. One widely cited study found that flakiness caused 84% of &amp;ldquo;pass to fail&amp;rdquo; state transitions in its test suites &amp;ndash; meaning teams that rely on retries are habitually attributing real regressions to &amp;ldquo;just flakiness&amp;rdquo;. The confidence signal the suite is supposed to provide is no longer trustworthy.&lt;/p>
&lt;p>&lt;strong>Longer timeouts slow the suite without touching the root cause.&lt;/strong> Extending a gesture timeout from 3 seconds to 8 seconds does cover &amp;ldquo;the emulator needs more time to process gesture events&amp;rdquo;, but it also adds 5 seconds to every run of that case. A suite of 200 cases with 3 extra seconds of timeout buffer each runs 10 minutes longer overall. A longer feedback loop means developers have already switched to the next task by the time they learn about a failure.&lt;/p>
&lt;p>&lt;strong>Quarantine postpones the problem to a future that never arrives.&lt;/strong> &amp;ldquo;We will fix the quarantined cases next iteration&amp;rdquo; almost never happens, because fixing them requires understanding the root cause, and understanding the root cause requires environmental investigation that is slower than writing new cases. So the quarantine list grows longer; and the quarantined cases represent real coverage holes &amp;ndash; regressions hide in those holes.&lt;/p>
&lt;p>The common problem with all three is the same: treating a structural problem as a maintenance problem. Flakiness is not because the team is not careful enough, but because &lt;strong>selector-based tests couple cases to UI implementation details&lt;/strong>, and those details keep changing as the app evolves, the OS updates and the CI environment drifts.&lt;/p>
&lt;p>Public data from one engineering team shows that even with heavy investment in detection and monitoring tooling, 15-21% of builds were still affected by flakiness. Detection and monitoring can tell you which cases are broken, but they will not tell you why mobile-specific failure modes keep manufacturing new broken cases.&lt;/p>
&lt;p>At some tipping point the ratio flips: a quality team starts out spending 90% of its time writing cases and 10% maintaining them; after the debt accumulates it becomes 10% writing and 90% maintaining. That compression is what the script-level fix treadmill looks like at team scale.&lt;/p>
&lt;h2 id="five-practical-actions-that-need-no-tooling">Five Practical Actions That Need No Tooling
&lt;/h2>&lt;p>Before discussing architecture-level solutions, there are things any team can do immediately that significantly reduce the four failure classes above:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Run each case on a clean emulator snapshot&lt;/strong>, or at least rebuild the emulator per batch, directly eliminating the root cause of &amp;ldquo;drift during a run&amp;rdquo;;&lt;/li>
&lt;li>&lt;strong>Preset permission state during test preparation&lt;/strong> (through launch arguments or system image configuration), rather than dismissing dialogs with scripts inside the flow;&lt;/li>
&lt;li>&lt;strong>Wrap gestures in an abstraction layer&lt;/strong> that always performs the interaction after real rendering state, instead of hard-coded wait constants scattered everywhere;&lt;/li>
&lt;li>&lt;strong>Adopt &amp;ldquo;single-step swipe plus wait for target to appear&amp;rdquo;&lt;/strong> for long lists and infinite scroll, instead of swipes that depend on inertia distance;&lt;/li>
&lt;li>&lt;strong>Tier the device matrix&lt;/strong>: run 2-3 representative configurations in the daily pipeline, and put fragmentation verification into separate, less frequent but broader batches.&lt;/li>
&lt;/ul>
&lt;p>These actions bring the total volume of flakiness down, but they cannot change one fact: as long as cases depend on selectors, UI changes, OS updates and device differences will keep manufacturing new failures.&lt;/p>
&lt;h2 id="the-architecture-level-step-removing-the-selector-dependency">The Architecture-Level Step: Removing the Selector Dependency
&lt;/h2>&lt;p>The selector-based model &amp;ndash; locating elements by CSS selector, XPath, accessibility ID or resource ID &amp;ndash; is precisely where the fragility of mobile cases comes from. Change an attribute and the case breaks; the system changes how the view hierarchy is structured and the case breaks; a layout adjustment moves an element 10 pixels and changes its bounds, and a case based on click coordinates breaks.&lt;/p>
&lt;p>The alternative is to &lt;strong>identify an element by what it looks like and what it does&lt;/strong>, rather than by implementation attributes. A &amp;ldquo;Submit&amp;rdquo; button is recognisable because it says &amp;ldquo;Submit&amp;rdquo;, appears in a particular position in the form, and responds to clicks. When that button is redesigned, moved, or given a different accessibility ID, visual identification can still recognise it the way a human tester would.&lt;/p>
&lt;p>Looking back at the four failure modes from this angle, their roots all sit in the selector model:&lt;/p>
&lt;ul>
&lt;li>Emulator drift causes gesture events to land on the wrong element, because element positions shift in a degraded emulator state &amp;ndash; identifying by appearance does not depend on coordinates;&lt;/li>
&lt;li>Gesture timing problems are amplified by hard-coded time constants &amp;ndash; observe the actual rendering state before interacting and the timing assumption disappears;&lt;/li>
&lt;li>System permission dialogs interrupt selector-based flows because the script is following an element path that is now obscured &amp;ndash; identify by visual object and dismiss the dialog without changing the script for every OS version&amp;rsquo;s dialog format;&lt;/li>
&lt;li>Device fragmentation breaks selector-based cases because layouts render differently across devices &amp;ndash; identifying by &amp;ldquo;what the user sees&amp;rdquo; adapts to layout differences automatically.&lt;/li>
&lt;/ul>
&lt;p>It should be clear that this is not a feature you can install on top of an existing Appium or Detox suite, but a different test execution architecture: the user flows the cases verify stay the same, but they no longer depend on the implementation details that made those flows hard to maintain.&lt;/p>
&lt;p>When evaluating this shift, the right question is not &amp;ldquo;can it eliminate all flakiness&amp;rdquo; &amp;ndash; the answer is no, network faults and genuine environmental problems will still produce failures. The real question is whether flakiness caused by UI changes, device fragmentation and selector fragility (which is the majority of mobile-specific flakiness) becomes something you fix once at the architecture level, or remains a maintenance tax you pay every iteration.&lt;/p>
&lt;h2 id="closing-thoughts">Closing Thoughts
&lt;/h2>&lt;p>The loop is predictable: flakiness appears → add automatic retries → timeouts grow → the suite slows down → trust erodes → the team stops treating red CI as a signal worth investigating. This is not a discipline failure but the inevitable result of applying web testing assumptions to mobile testing problems.&lt;/p>
&lt;p>The reasons mobile E2E fails simply do not exist in web automation. Emulator drift, gesture timing, system dialogs and device fragmentation are all mobile-specific problems that need mobile-specific structural solutions, not more layers of script-level patches.&lt;/p>
&lt;p>And the maintenance treadmill has no sustainable pace. It has only one exit.&lt;/p></description></item></channel></rss>