<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Mobile Testing on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/mobile-testing/</link><description>Recent content in Mobile Testing on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 28 Oct 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/mobile-testing/index.xml" rel="self" type="application/rss+xml"/><item><title>Why Mobile E2E Tests Go Flaky - Four Failure Modes Web Testing Never Meets</title><link>https://chengxiaqiucao.github.io/en/post/145-mobile-e2e-flakiness/</link><pubDate>Wed, 28 Oct 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/145-mobile-e2e-flakiness/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/cover.webp" alt="Featured image of post Why Mobile E2E Tests Go Flaky - Four Failure Modes Web Testing Never Meets" />&lt;p>Retries are not a fix. Longer timeouts are not a fix. A quarantine list is not a fix. And that shared Slack channel the team set up specifically for &amp;ldquo;known flaky, safe to ignore&amp;rdquo;, that is even less of a fix.&lt;/p>
&lt;p>Most testing teams agree with these sentences, but the real issue is this: mobile E2E tests are unstable for reasons that are simply not of the same kind as those behind web tests; and every solution borrowed from the web playbook makes the problem structurally worse.&lt;/p>
&lt;p>A recurring sequence looks like this: a team takes over an end-to-end suite, adds automatic retries when tests start failing, lengthens timeouts when retries are not enough, and finally stops trusting the cases altogether. Nobody looks when CI goes red, and nobody knows whether it is a real defect or &amp;ldquo;just another ordinary Tuesday&amp;rdquo;.&lt;/p>
&lt;p>This is not a discipline problem. The problem is that mobile E2E has four failure modes that simply do not exist on the web, and web-first fixes drag the wrong mental model along with them.&lt;/p>
&lt;h2 id="mobile-instability-is-a-different-problem">Mobile Instability Is a Different Problem
&lt;/h2>&lt;p>Web automation runs inside a browser. The browser is a sandboxed, largely deterministic environment: the DOM is queryable, animations can be disabled, network requests can be intercepted, and the rendering engine behaves predictably across runs. When web tests go flaky, the causes are long since well studied &amp;ndash; async waits, test order dependencies, shared state pollution; every testing textbook covers them.&lt;/p>
&lt;p>Mobile automation runs on top of a whole operating system, and that difference invalidates every assumption inherited from web testing.&lt;/p>
&lt;p>A mobile app test has to cope with system-level interruptions the test script never triggered; has to cope with a gesture model where &amp;ldquo;a 10 millisecond difference in click timing changes the result&amp;rdquo;; has to survive CI emulators that behave differently on the 47th run than on the first; and has to produce consistent results across a device ecosystem that, on Android alone, spans thousands of vendors and dozens of still-active OS versions.&lt;/p>
&lt;p>Fixing mobile flakiness with web solutions is like diagnosing an aircraft hydraulic fault with an automobile brake repair manual: the vocabulary overlaps, the physics do not.&lt;/p>
&lt;p>Those web-oriented fixes (unified async waiting patterns, isolated test state, enforced test order) are correct for web problems. Applied to mobile, they probably solve half the problem; the other half has different causes and needs different solutions.&lt;/p>
&lt;p>The data shows this has not improved as tooling matured. An industry report on mobile development covering three years and more than ten million builds showed that the share of mobile teams reporting flaky tests rose from 10% in 2022 to 26% in 2025. The more complex the pipeline, the worse the instability.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_a35256f31869fbc3.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_53f7eb880eafdad6.webp 480w, https://chengxiaqiucao.github.io/post/145-mobile-e2e-flakiness/05-fig1_hu_a35256f31869fbc3.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Top: emulator responsiveness keeps degrading as a run progresses; bottom: the decision boundaries of gesture timing are not evenly spaced&lt;/p>
&lt;h2 id="four-failure-modes-web-testing-has-never-seen">Four Failure Modes Web Testing Has Never Seen
&lt;/h2>&lt;h3 id="one-emulator-behaviour-drift">One: Emulator Behaviour Drift
&lt;/h3>&lt;p>CI pipelines run on emulators, not real devices. An emulator is a software simulation of hardware, and like any software running in a resource-constrained CI environment, its behaviour degrades continuously &lt;strong>over the course of a single test run&lt;/strong>.&lt;/p>
&lt;p>At the start of a suite the emulator responds well: GPU acceleration is warm, memory is clean, the virtual CPU has headroom. Forty cases later the same emulator may be under memory pressure, the virtual CPU may be throttling, or it may start dropping gesture events because the event queue is backed up. The test code is identical; the environment has drifted.&lt;/p>
&lt;p>Web CI environments do not have a problem of the same scale, because browser instances are lightweight and stateless. An emulator runs an entire mobile operating system: camera drivers, Bluetooth stack, network simulation, GPS simulation &amp;ndash; all of it runs even if your test only touches one login page.&lt;/p>
&lt;p>The typical symptom: cases at positions 1-15 pass reliably, and by positions 30-50 they start failing intermittently. The developer&amp;rsquo;s reaction is to add retries; retries occasionally work because the second attempt happens to catch a moment when the emulator state is slightly better; and the root cause &amp;ndash; the emulator changed behaviour mid-run &amp;ndash; is never addressed.&lt;/p>
&lt;h3 id="two-gesture-timing-sensitivity">Two: Gesture Timing Sensitivity
&lt;/h3>&lt;p>The essence of mobile interaction is the gesture. A tap is not a click and a swipe is not a scroll, and that difference matters far more than it sounds.&lt;/p>
&lt;p>A browser click triggers a discrete set of events (mousedown, mouseup, click) that the DOM handles predictably. A mobile gesture is a stream of touch events with time attributes, interpreted by the operating system in context: holding for 100 milliseconds is a tap, holding for 500 milliseconds triggers a long-press context menu; a swipe that starts 5 pixels further left may be recognised as a scroll rather than a swipe on a particular device or OS version.&lt;/p>
&lt;p>Automation frameworks do expose gesture APIs, but the underlying timing behaviour differs between iOS and Android, between OS versions, and between emulators and real devices. A gesture command that is stable on a local emulator may fail on a CI host with different virtualisation characteristics. Long lists usually expose this difference first, because a fast swipe crosses a different number of rows each time &amp;ndash; which is why infinite-scroll tests usually swipe one step at a time and then wait for the target cell to appear.&lt;/p>
&lt;p>The web does not have this problem; click events are abstracted away at the browser layer. Mobile developers have no equivalent abstraction layer and must work directly against an OS gesture model that varies by implementation.&lt;/p>
&lt;h3 id="three-system-permission-dialogs">Three: System Permission Dialogs
&lt;/h3>&lt;p>iOS and Android pop system-level permission dialogs during app flows: camera, microphone, location, notifications, contacts, Bluetooth. These dialogs are not triggered by your test; the operating system decides when to show them based on internal state.&lt;/p>
&lt;p>Once a dialog appears mid-test it covers the interface. The script tries to click an element that is now obscured, the click fails, the element is not found, and the test reports a failure that has nothing to do with the code under test.&lt;/p>
&lt;p>When the dialog appears is not fully deterministic: a clean emulator state may not show the notification permission on the first run but may show it on the third, because by then the app has been installed long enough for the system to decide it is time to ask. Teams respond by adding &amp;ldquo;dismiss the dialog&amp;rdquo; logic to test setup; but system behaviour changes with iOS and Android versions, so a setup script that works on one version silently stops working on another.&lt;/p>
&lt;p>This class of flakiness is especially expensive, because the failure report points at the wrong element &amp;ndash; the one the script wanted to click, not the dialog that actually blocked it.&lt;/p>
&lt;h3 id="four-network-conditions-and-device-fragmentation">Four: Network Conditions and Device Fragmentation
&lt;/h3>&lt;p>Mobile apps run under network conditions web apps largely avoid: switching between WiFi and cellular, the app being backgrounded mid-request, carrier-specific DNS behaviour, and latency that varies with signal quality.&lt;/p>
&lt;p>A test that passes on a WiFi-only emulator may fail on a real device when it switches to cellular mid-flow; a test that handles a 200 millisecond API response may fail in a CI environment whose network simulation is configured differently. These are not async waiting problems but &lt;strong>environment fidelity problems&lt;/strong>: the test environment and real operating conditions do not match.&lt;/p>
&lt;p>Device fragmentation amplifies it further. The Android ecosystem stretches from entry-level phones with 1GB of memory running Android 9 all the way to current flagship hardware. A test designed for &amp;ldquo;a high-spec virtual machine plus an Android 14 emulator&amp;rdquo; will meet differences in rendering timing, memory management and garbage collection behaviour on real entry-level hardware. Screen density differences alone are enough to make a click coordinate that works on one device miss on another.&lt;/p>
&lt;p>Selector-based suites assume the view hierarchy is stable across devices &amp;ndash; and it frequently is not.&lt;/p>
&lt;h2 id="why-script-level-fixes-make-things-worse">Why Script-Level Fixes Make Things Worse
&lt;/h2>&lt;p>Facing mobile flakiness, the standard move is a predictable escalation path: retry first, because it is the least effortful lever; then add timeouts, because more time usually lets emulator drift and gesture timing eventually slip through; then quarantine, because some cases are reliably unreliable and the team simply stops expecting them to pass.&lt;/p>
&lt;p>Each has a cost, and none of it shows up on the CI dashboard.&lt;/p>
&lt;p>&lt;strong>Retries mask real failures.&lt;/strong> A case that passes on the third retry is a flaky case that got lucky; it may equally be a real regression buried by the retry logic. One widely cited study found that flakiness caused 84% of &amp;ldquo;pass to fail&amp;rdquo; state transitions in its test suites &amp;ndash; meaning teams that rely on retries are habitually attributing real regressions to &amp;ldquo;just flakiness&amp;rdquo;. The confidence signal the suite is supposed to provide is no longer trustworthy.&lt;/p>
&lt;p>&lt;strong>Longer timeouts slow the suite without touching the root cause.&lt;/strong> Extending a gesture timeout from 3 seconds to 8 seconds does cover &amp;ldquo;the emulator needs more time to process gesture events&amp;rdquo;, but it also adds 5 seconds to every run of that case. A suite of 200 cases with 3 extra seconds of timeout buffer each runs 10 minutes longer overall. A longer feedback loop means developers have already switched to the next task by the time they learn about a failure.&lt;/p>
&lt;p>&lt;strong>Quarantine postpones the problem to a future that never arrives.&lt;/strong> &amp;ldquo;We will fix the quarantined cases next iteration&amp;rdquo; almost never happens, because fixing them requires understanding the root cause, and understanding the root cause requires environmental investigation that is slower than writing new cases. So the quarantine list grows longer; and the quarantined cases represent real coverage holes &amp;ndash; regressions hide in those holes.&lt;/p>
&lt;p>The common problem with all three is the same: treating a structural problem as a maintenance problem. Flakiness is not because the team is not careful enough, but because &lt;strong>selector-based tests couple cases to UI implementation details&lt;/strong>, and those details keep changing as the app evolves, the OS updates and the CI environment drifts.&lt;/p>
&lt;p>Public data from one engineering team shows that even with heavy investment in detection and monitoring tooling, 15-21% of builds were still affected by flakiness. Detection and monitoring can tell you which cases are broken, but they will not tell you why mobile-specific failure modes keep manufacturing new broken cases.&lt;/p>
&lt;p>At some tipping point the ratio flips: a quality team starts out spending 90% of its time writing cases and 10% maintaining them; after the debt accumulates it becomes 10% writing and 90% maintaining. That compression is what the script-level fix treadmill looks like at team scale.&lt;/p>
&lt;h2 id="five-practical-actions-that-need-no-tooling">Five Practical Actions That Need No Tooling
&lt;/h2>&lt;p>Before discussing architecture-level solutions, there are things any team can do immediately that significantly reduce the four failure classes above:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Run each case on a clean emulator snapshot&lt;/strong>, or at least rebuild the emulator per batch, directly eliminating the root cause of &amp;ldquo;drift during a run&amp;rdquo;;&lt;/li>
&lt;li>&lt;strong>Preset permission state during test preparation&lt;/strong> (through launch arguments or system image configuration), rather than dismissing dialogs with scripts inside the flow;&lt;/li>
&lt;li>&lt;strong>Wrap gestures in an abstraction layer&lt;/strong> that always performs the interaction after real rendering state, instead of hard-coded wait constants scattered everywhere;&lt;/li>
&lt;li>&lt;strong>Adopt &amp;ldquo;single-step swipe plus wait for target to appear&amp;rdquo;&lt;/strong> for long lists and infinite scroll, instead of swipes that depend on inertia distance;&lt;/li>
&lt;li>&lt;strong>Tier the device matrix&lt;/strong>: run 2-3 representative configurations in the daily pipeline, and put fragmentation verification into separate, less frequent but broader batches.&lt;/li>
&lt;/ul>
&lt;p>These actions bring the total volume of flakiness down, but they cannot change one fact: as long as cases depend on selectors, UI changes, OS updates and device differences will keep manufacturing new failures.&lt;/p>
&lt;h2 id="the-architecture-level-step-removing-the-selector-dependency">The Architecture-Level Step: Removing the Selector Dependency
&lt;/h2>&lt;p>The selector-based model &amp;ndash; locating elements by CSS selector, XPath, accessibility ID or resource ID &amp;ndash; is precisely where the fragility of mobile cases comes from. Change an attribute and the case breaks; the system changes how the view hierarchy is structured and the case breaks; a layout adjustment moves an element 10 pixels and changes its bounds, and a case based on click coordinates breaks.&lt;/p>
&lt;p>The alternative is to &lt;strong>identify an element by what it looks like and what it does&lt;/strong>, rather than by implementation attributes. A &amp;ldquo;Submit&amp;rdquo; button is recognisable because it says &amp;ldquo;Submit&amp;rdquo;, appears in a particular position in the form, and responds to clicks. When that button is redesigned, moved, or given a different accessibility ID, visual identification can still recognise it the way a human tester would.&lt;/p>
&lt;p>Looking back at the four failure modes from this angle, their roots all sit in the selector model:&lt;/p>
&lt;ul>
&lt;li>Emulator drift causes gesture events to land on the wrong element, because element positions shift in a degraded emulator state &amp;ndash; identifying by appearance does not depend on coordinates;&lt;/li>
&lt;li>Gesture timing problems are amplified by hard-coded time constants &amp;ndash; observe the actual rendering state before interacting and the timing assumption disappears;&lt;/li>
&lt;li>System permission dialogs interrupt selector-based flows because the script is following an element path that is now obscured &amp;ndash; identify by visual object and dismiss the dialog without changing the script for every OS version&amp;rsquo;s dialog format;&lt;/li>
&lt;li>Device fragmentation breaks selector-based cases because layouts render differently across devices &amp;ndash; identifying by &amp;ldquo;what the user sees&amp;rdquo; adapts to layout differences automatically.&lt;/li>
&lt;/ul>
&lt;p>It should be clear that this is not a feature you can install on top of an existing Appium or Detox suite, but a different test execution architecture: the user flows the cases verify stay the same, but they no longer depend on the implementation details that made those flows hard to maintain.&lt;/p>
&lt;p>When evaluating this shift, the right question is not &amp;ldquo;can it eliminate all flakiness&amp;rdquo; &amp;ndash; the answer is no, network faults and genuine environmental problems will still produce failures. The real question is whether flakiness caused by UI changes, device fragmentation and selector fragility (which is the majority of mobile-specific flakiness) becomes something you fix once at the architecture level, or remains a maintenance tax you pay every iteration.&lt;/p>
&lt;h2 id="closing-thoughts">Closing Thoughts
&lt;/h2>&lt;p>The loop is predictable: flakiness appears → add automatic retries → timeouts grow → the suite slows down → trust erodes → the team stops treating red CI as a signal worth investigating. This is not a discipline failure but the inevitable result of applying web testing assumptions to mobile testing problems.&lt;/p>
&lt;p>The reasons mobile E2E fails simply do not exist in web automation. Emulator drift, gesture timing, system dialogs and device fragmentation are all mobile-specific problems that need mobile-specific structural solutions, not more layers of script-level patches.&lt;/p>
&lt;p>And the maintenance treadmill has no sustainable pace. It has only one exit.&lt;/p></description></item></channel></rss>