<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>CI on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/ci/</link><description>Recent content in CI on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 04 Nov 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/ci/index.xml" rel="self" type="application/rss+xml"/><item><title>Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue</title><link>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</link><pubDate>Wed, 04 Nov 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/147-flaky-retry-ci-digest/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/cover.webp" alt="Featured image of post Failures Hidden by Retries - Turning the 'Passed on the Second Try' in CI into a Triageable Defect Queue" />&lt;p>One failure, one pass, final green. This is what most CI pipelines with automatic retries produce every day, and it is also the result where information is most easily lost: the failed attempt gets overwritten in the logs, deleted during cleanup of the successful job, and the team only sees that &amp;ldquo;this build is green&amp;rdquo;. The real problem is that the reason the second attempt passed is usually not &amp;ldquo;random jitter&amp;rdquo; but some specific state, timing or data condition &amp;ndash; and users will run into it in production too.&lt;/p>
&lt;p>One approach deserves a full breakdown: write a structured record for every retry at the end of the suite and keep it separately, aggregate it weekly into a list sorted by impact, and then let a coding agent follow the records into the code to locate root causes. It does not aim to &amp;ldquo;make CI greener&amp;rdquo; but the opposite &amp;ndash; to re-expose the failures that retries pressed down.&lt;/p>
&lt;h2 id="step-one-let-the-failed-attempt-survive-success-cleanup">Step One: Let the Failed Attempt Survive Success Cleanup
&lt;/h2>&lt;p>Most CI configurations delete application logs and diagnostic artifacts after a job succeeds. That is reasonable cost control, but it happens to delete exactly the evidence most needed to investigate intermittent failures. The solution is not to keep all logs, but to write a very small JSON file for every test node that retried, at the end of the suite:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt">1
&lt;/span>&lt;span class="lnt">2
&lt;/span>&lt;span class="lnt">3
&lt;/span>&lt;span class="lnt">4
&lt;/span>&lt;span class="lnt">5
&lt;/span>&lt;span class="lnt">6
&lt;/span>&lt;span class="lnt">7
&lt;/span>&lt;span class="lnt">8
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;id&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb[1:2]&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;location&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;./spec/features/report_spec.rb:20&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;description&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;renders a synthetic report&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;failed_attempts&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;final_result&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;passed&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;first_error&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;expected the default window size&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>Two fields deserve separate mention. &lt;code>id&lt;/code> is the unique example identifier the test framework provides, and &lt;code>location&lt;/code> is the source line number. Shared examples cause several different test cases to point at the same line, so aggregating by line number alone merges unrelated failures into one lump; keeping both is what allows correct grouping.&lt;/p>
&lt;p>This JSON must be written to a directory &lt;strong>outside the scope of success-job cleanup&lt;/strong> and uploaded as a build artifact. With a common CI service, that means adding one line to the config declaring the directory as an artifact; any CI platform has an equivalent mechanism. There is only one hard constraint: even if this test ultimately passes, the retry history must be kept.&lt;/p>
&lt;p>The retry policy itself should also be written down clearly: one retry for ordinary cases, two for cases involving a JavaScript browser. The count is not the point; the point is that &lt;strong>every retry must leave a clue about why the result changed&lt;/strong>. Merely increasing the retry count only makes the report quieter, it does not answer any question.&lt;/p>
&lt;h2 id="step-two-aggregate-the-records-into-a-prioritized-weekly-queue">Step Two: Aggregate the Records into a Prioritized Weekly Queue
&lt;/h2>&lt;p>With records in hand, the next step is aggregation. A scheduled job reads the past week&amp;rsquo;s retry artifacts through the CI platform API, groups them by example identifier, and sorts descending by &lt;strong>the number of distinct pipelines in which a &amp;ldquo;failed then passed&amp;rdquo; occurred&lt;/strong>.&lt;/p>
&lt;p>There is a detail that is easy to get wrong here: if a case failed twice within the same pipeline, it counts as only one pipeline. The ranking wants to measure &amp;ldquo;how many independent builds this flakiness affected&amp;rdquo;, not &amp;ldquo;how many times it failed in total&amp;rdquo;. Keep the first error message, branch name and number of failed attempts alongside for later reading.&lt;/p>
&lt;p>One class should be counted separately: cases where &lt;strong>all attempts failed&lt;/strong>. This distinction only describes the observed run outcome and does not mean &amp;ldquo;the root cause is deterministic&amp;rdquo; &amp;ndash; an intermittent problem can equally exhaust the retry budget, so you cannot use it to attach different labels to the two classes.&lt;/p>
&lt;p>When a given example exceeds a configured weekly budget threshold, make the summary job itself fail, borrowing the team&amp;rsquo;s existing CI failure notifications to push the list in front of people. At that point, hand the report to a coding agent that can read the repository and build artifacts, and the localization phase begins.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp"
width="1024"
height="820"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_ed4aa59c7fcf933b.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig1_hu_83cb9ef2bd28c4d1.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="124"
data-flex-basis="299px"
>
&lt;/p>
&lt;p>The full chain from retry record generation and retention to the weekly priority queue&lt;/p>
&lt;h2 id="step-three-seeing-clearly-why-the-second-attempt-passed---four-real-cases">Step Three: Seeing Clearly &amp;ldquo;Why the Second Attempt Passed&amp;rdquo; - Four Real Cases
&lt;/h2>&lt;p>The following classes of problem share one trait: the retry did not make the test recover by itself, the &lt;strong>retry mechanism happened to reset the state that was causing the error&lt;/strong>. Without looking at the first failure, you would never know there was a defect in the product.&lt;/p>
&lt;h3 id="a-browser-restart-wiped-the-window-size-left-by-the-previous-case">A Browser Restart Wiped the Window Size Left by the Previous Case
&lt;/h3>&lt;p>A group of screenshot comparisons failed, reporting a change in screenshot dimensions, and a retry usually passed. The root cause lay in an earlier case: a billing-related test changed the browser window to a non-default size without restoring it, and the later screenshot case inherited that viewport. The retry configuration restarts the browser before rerunning, the window returns to its default size, and the second attempt passes.&lt;/p>
&lt;p>The fix is to explicitly set the default window size before every browser case, with cases needing another size changing it themselves. In this case, &amp;ldquo;the retry passed&amp;rdquo; actually provided the key clue: since a fresh browser fixed it, the state must have been brought in from outside rather than restored by the logic under test.&lt;/p>
&lt;h3 id="an-apostrophe-broke-an-auto-generated-curl-command">An Apostrophe Broke an Auto-Generated cURL Command
&lt;/h3>&lt;p>A dashboard generates example cURL commands from a template schema, and the tests execute those commands using generated data. The generated fake names occasionally contain an apostrophe, and the inline command wraps the JSON request body in single quotes &amp;ndash; the apostrophe ends the shell string early and the command breaks. The next attempt generates a different name, and the command works again.&lt;/p>
&lt;p>The same feature also has a downloadable Bash script that puts the JSON in a quoted heredoc, but incorrectly applies the escaping intended for single-quoted arguments, corrupting the request body. The two places need completely different handling: the inline command must escape apostrophes for the shell context, while the JSON inside the heredoc must be preserved verbatim.&lt;/p>
&lt;p>The corresponding regression test is well worth borrowing: replace &lt;code>curl&lt;/code> with a function, actually execute the generated script in Bash, capture the request body that would have been sent with that function, then parse and validate the original value &amp;ndash; without making any network request at all. Replay the fixed and unfixed versions with a synthetic value like &lt;code>Editor's proof&lt;/code> that contains an apostrophe: before the fix the inline command fails outright in Bash and the downloadable script runs but the request body cannot be parsed as JSON; after the fix both preserve the text as-is.&lt;/p>
&lt;p>The &amp;ldquo;shortcut&amp;rdquo; that is easiest to choose here is to change the test data so names contain no apostrophes. That would make the test more stable and keep the product broken for every legitimate input.&lt;/p>
&lt;h3 id="a-temporary-file-was-collected-before-the-render-process-read-it">A Temporary File Was Collected Before the Render Process Read It
&lt;/h3>&lt;p>An inline HTML-to-PDF renderer writes header and footer HTML to temporary files and then hands the paths to a render process. The code keeps the temp file objects in local variables while building the command; after the method returns, the command string still holds the paths, but nothing references those objects any more.&lt;/p>
&lt;p>Ruby&amp;rsquo;s &lt;code>Tempfile&lt;/code> hooks cleanup onto garbage collection. If GC happens to run before the render process opens the file, the file is already gone. Whether the test passes depends on when garbage collection occurs &amp;ndash; the most typical &amp;ldquo;seemingly random&amp;rdquo; failure.&lt;/p>
&lt;p>The fix is to keep the header and footer objects on the service instance until rendering completes, then close and delete them in an &lt;code>ensure&lt;/code> block. The regression test &lt;strong>forces GC&lt;/strong> between &amp;ldquo;build the command&amp;rdquo; and &amp;ldquo;simulate the render process reading the file&amp;rdquo;, asserting that the file content is still readable and gets cleaned up afterwards. Deliberately manufacturing a deterministic failure condition is far more reliable than waiting to get lucky in some PDF case.&lt;/p>
&lt;h3 id="rubys-clock-was-frozen-rediss-clock-was-still-running">Ruby&amp;rsquo;s Clock Was Frozen, Redis&amp;rsquo;s Clock Was Still Running
&lt;/h3>&lt;p>A rate-limit test uses counters that count within a fixed time window. Once a case crosses the window boundary, requests get counted into the new window and the expected threshold is never reached. The first fix frozen the process clock, but the counters live in Redis with an expiry time, and Redis keeps running on real time. If you freeze near the end of a window, the counter&amp;rsquo;s remaining time to live is very short and the problem persists.&lt;/p>
&lt;p>The final approach freezes at the start of the hour, aligned with the time window these cases use, so the counter gets a full expiry cycle; the cases still have to finish within that cycle. The value of this case lies in the fact that review found a second clock after the first fix &amp;ndash; the first explanation covered only half of the timing mechanisms in the system.&lt;/p>
&lt;p>The same investigation round also surfaced several other classes of problem: a background worker still running when database cleanup started; test helper functions overwriting each other&amp;rsquo;s frozen state on a shared clock library; and a sample data generator using truthiness to decide whether to skip lower-bound adjustment when the bound is zero (in JavaScript 0 is falsy, yet in this schema it is a legitimate lower bound). Each falls into the category of &amp;ldquo;a retry makes it disappear, but a user may still hit it&amp;rdquo;.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_2bea94c3ea199544.webp 480w, https://chengxiaqiucao.github.io/post/147-flaky-retry-ci-digest/01-fig2_hu_81b443d809eab84c.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Four real defects hidden by retries: state leakage, escaping context, object lifetime, dual clocks&lt;/p>
&lt;h2 id="step-four-draw-clear-task-boundaries-for-the-coding-agent">Step Four: Draw Clear Task Boundaries for the Coding Agent
&lt;/h2>&lt;p>When handing the weekly report to a coding agent, the task description should be specific enough to be verifiable, not &amp;ldquo;please take a look at these failures&amp;rdquo;. A reusable set of requirements is:&lt;/p>
&lt;ul>
&lt;li>Find the corresponding CI run and read the failure artifacts, then follow the test code into the application code;&lt;/li>
&lt;li>Reproduce locally where possible first, then add a regression test that is required to fail before the fix and pass after it;&lt;/li>
&lt;li>Split unrelated root causes into separate commits, so each can be reviewed and rolled back individually;&lt;/li>
&lt;li>Run the affected checks and self-review the diff, explicitly listing items that could not be reproduced, without vague hand-waving;&lt;/li>
&lt;li>Stop and wait for human approval before pushing.&lt;/li>
&lt;/ul>
&lt;p>The reason for insisting that &amp;ldquo;artifacts and code must be read together&amp;rdquo; is that neither side alone yields a conclusion: a mismatched screenshot size does not by itself explain who changed the viewport, and a missing file does not by itself explain why that Ruby object became collectable. The agent&amp;rsquo;s real value is joining those two kinds of information, not replacing the human final judgement.&lt;/p>
&lt;p>Any rule in the investigation conclusions that &amp;ldquo;future development must follow&amp;rdquo; goes into the &lt;code>AGENTS.md&lt;/code> closest to the code (or an equivalent team convention file). For example, &amp;ldquo;browser cases may change their own viewport, but the next case must start from a known size&amp;rdquo; &amp;ndash; setup code enforces it, comments explain why. Better still, anything that can be turned into a static check should not rely on memory: add a custom lint rule for the common flaky patterns and let CI block them, which works far better than hoping someone remembers an investigation from six months ago.&lt;/p>
&lt;h2 id="an-action-checklist">An Action Checklist
&lt;/h2>&lt;p>If your suite already retries, you do not need to refactor the whole system first. Just follow this order:&lt;/p>
&lt;ol>
&lt;li>At the end of the suite, write one structured record per retry, with fields covering at least the unique example identifier, source location, number of failed attempts, final result and first error;&lt;/li>
&lt;li>Put the records in a directory that success-job cleanup cannot reach, and configure it as a build artifact upload;&lt;/li>
&lt;li>Aggregate and sort weekly by &amp;ldquo;number of distinct pipelines with a failed-then-passed&amp;rdquo;, set a budget threshold, and when it is exceeded make the summary job fail and notify;&lt;/li>
&lt;li>Pick one recurring problem and demand a clear explanation of &amp;ldquo;why the second attempt passed&amp;rdquo;, rather than simply adding more retries;&lt;/li>
&lt;li>The fix must come with a regression test that reproduces deterministically, and only merges into the regular CI, QA and release flow after review passes.&lt;/li>
&lt;/ol>
&lt;p>There is no published data yet on hours saved by this process, and the fix rounds are too recent to judge how far it can push the recurrence rate down over the long run &amp;ndash; the weekly reports over the coming weeks will answer that. But its immediate effect is already clear: problems that used to require someone to set aside half a day to chase, often losing the trail halfway through, can now be investigated in parallel with other work, and coming back only requires assessing whether the explanation holds and whether the change is reasonable.&lt;/p>
&lt;p>That is exactly why the quality of code written by the agent matters more: a reproducible failure and a test that asserts the real intent are concrete targets it can anchor on. And keeping the retry history is equivalent to putting more of those targets on the table.&lt;/p></description></item><item><title>250,000 Test Runs a Day - Architecture and Trade-offs of a Centralized Test Execution Platform</title><link>https://chengxiaqiucao.github.io/en/post/144-centralized-test-platform/</link><pubDate>Wed, 28 Oct 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/144-centralized-test-platform/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/cover.webp" alt="Featured image of post 250,000 Test Runs a Day - Architecture and Trade-offs of a Centralized Test Execution Platform" />&lt;p>Test automation is usually discussed as a framework problem. But when thousands of repositories all need to run their own suites, the nature of the problem changes &amp;ndash; it becomes an infrastructure problem. When every team pushes at nine in the morning, who gets execution capacity? When the nightly full regression collides with the pre-merge check blocking some developer, which one runs first? And when a good portion of failures are not real defects, on what basis do you declare a build &amp;ldquo;certified&amp;rdquo;?&lt;/p>
&lt;p>One class of platform grows out of exactly this pressure: it does not help you write tests, it provides a shared execution and certification infrastructure. This article breaks down the architecture, key policies and real trade-offs of such a platform, with data from an instance that runs about 250,000 automated test executions a day across four Amazon EKS clusters and serves all engineering repositories of a large financial institution.&lt;/p>
&lt;h2 id="start-by-seeing-the-five-characteristics-of-the-workload">Start by Seeing the Five Characteristics of the Workload
&lt;/h2>&lt;p>Before designing, you must acknowledge that a test execution workload has several properties that break naive designs:&lt;/p>
&lt;p>&lt;strong>Extreme bursts.&lt;/strong> Demand spikes during working hours and release windows, while capacity sits idle the rest of the time &amp;ndash; always swinging between &amp;ldquo;not enough&amp;rdquo; and &amp;ldquo;wasted&amp;rdquo;.&lt;/p>
&lt;p>&lt;strong>Mixed urgency.&lt;/strong> A pre-merge check holds up one engineer&amp;rsquo;s time, the nightly full regression holds up nobody&amp;rsquo;s, yet both compete for the same pool of resources with completely different latency requirements.&lt;/p>
&lt;p>&lt;strong>Heterogeneous tenants.&lt;/strong> Different teams bring different runtimes, browser stacks and data dependencies; a suite with no resource constraints can starve every other suite.&lt;/p>
&lt;p>&lt;strong>Untrustworthy signals.&lt;/strong> Empirical research shows that async waits, concurrency and test order dependencies keep producing flaky results. Public engineering experience reports that about 1.5% of test runs produced flaky results and nearly 16% of tests showed some degree of instability. Apply that ratio to 250,000 executions a day and 1.5% is about 3,750 misleading results &amp;ndash; if retries, classification and quarantine are not platform-level policies but per-team improvisation, then &amp;ldquo;red&amp;rdquo; loses its meaning entirely.&lt;/p>
&lt;p>&lt;strong>Audit requirements.&lt;/strong> In a regulated environment, &amp;ldquo;this build passed certification&amp;rdquo; must be reproducible from records, not from memory.&lt;/p>
&lt;h2 id="three-kubernetes-primitives-carry-most-of-the-design">Three Kubernetes Primitives Carry Most of the Design
&lt;/h2>&lt;p>Such a platform usually relies on only a few Kubernetes capabilities:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Job&amp;rsquo;s Indexed completion mode&lt;/strong>: each Pod gets a stable index, which maps naturally onto a test shard;&lt;/li>
&lt;li>&lt;strong>PriorityClass&lt;/strong>: a high-priority Pod that cannot be scheduled can preempt lower-priority Pods;&lt;/li>
&lt;li>&lt;strong>ResourceQuota and LimitRange&lt;/strong>: limit how much a namespace can consume and set default requests, avoiding unbounded containers on a shared cluster.&lt;/li>
&lt;/ul>
&lt;p>On top of that, queue managers such as Kueue provide queue-level quota, deciding whether a job waits, is admitted or is preempted; autoscalers such as Karpenter provision nodes directly for Pods that cannot be scheduled.&lt;/p>
&lt;h2 id="architecture-control-plane-execution-plane-results-plane">Architecture: Control Plane, Execution Plane, Results Plane
&lt;/h2>&lt;p>The platform separates three things: &amp;ldquo;deciding what to run&amp;rdquo; (control plane), &amp;ldquo;actually running it&amp;rdquo; (execution plane) and &amp;ldquo;turning execution into decisions&amp;rdquo; (results plane).&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_93a37452f58422c9.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_33329837ce4e233a.webp 480w, https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_93a37452f58422c9.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Control plane, execution plane and results plane are separated, yet the three must be built as one system&lt;/p>
&lt;p>&lt;strong>Submission.&lt;/strong> Each team&amp;rsquo;s CI system triggers a run, declaring the suite, the target environment and an urgency class. Teams never touch Kubernetes directly &amp;ndash; the submission API is the contract, so execution details can be replaced without affecting callers.&lt;/p>
&lt;p>&lt;strong>Admission and queuing.&lt;/strong> Each tenant gets a namespace with a quota and a local queue. The queue manager suspends workloads until quota is available, and performs preemption and fair sharing within a cohort (capacity pool): idle capacity borrowed by one team can be reclaimed when another team needs its own share. A borrowing cap prevents a single tenant from swallowing the whole pool during a release sprint.&lt;/p>
&lt;p>&lt;strong>Shard planning.&lt;/strong> Large suites are cut into shards and executed as an Indexed Job, with the shard index exposed to the runner through an environment variable &amp;ndash; the same contract as Bazel&amp;rsquo;s &lt;code>TEST_TOTAL_SHARDS&lt;/code> / &lt;code>TEST_SHARD_INDEX&lt;/code>.&lt;/p>
&lt;p>&lt;strong>Execution plane.&lt;/strong> Four EKS clusters host the platform&amp;rsquo;s 40 applications plus a large number of ephemeral test Pods. Multiple clusters trade higher control-plane and tooling costs for isolation and a smaller blast radius &amp;ndash; a trade-off explicitly documented in multi-tenant designs. Placement policy needs deliberate design: give each tenant a &amp;ldquo;home cluster&amp;rdquo; where image caches and test data stay warm; when the home cluster&amp;rsquo;s queue time exceeds a threshold, the control plane spills the &lt;strong>entire run&lt;/strong> to another cluster rather than splitting one run across clusters &amp;ndash; splitting makes result collection and infrastructure failure attribution extremely hard.&lt;/p>
&lt;p>&lt;strong>Results and certification.&lt;/strong> Each shard sends structured result messages to a reporting and insights service, which decides pass/fail per suite, identifies flakes, and registers a certification verdict on the release. The most important contract in the whole platform is this event schema:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;runId&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;checkout-regression-7f3a&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;shardIndex&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">17&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;attempt&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">2&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;test&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;CheckoutFlowTest.appliesPromoCode&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;outcome&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;passed&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;failureCause&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="kc">null&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;previousAttemptCause&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;environment&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;node&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;ip-10-0-42-7&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;durationMs&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">48210&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>&lt;code>failureCause&lt;/code> (test / environment / platform) and &lt;code>attempt&lt;/code> are what let the results layer distinguish &amp;ldquo;a flaky test&amp;rdquo; from &amp;ldquo;a node was lost&amp;rdquo;, and they are also the basis for charging infrastructure failures to the platform&amp;rsquo;s account rather than the team&amp;rsquo;s.&lt;/p>
&lt;h2 id="what-the-configuration-looks-like">What the Configuration Looks Like
&lt;/h2>&lt;p>A tenant&amp;rsquo;s queue and quota are declared once by the platform team:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">apiVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">kueue.x-k8s.io/v1beta1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">kind&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ClusterQueue&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">metadata&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">team-payments }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">cohort&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">shared-test-capacity&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">preemption&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">reclaimWithinCohort&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Any&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">withinClusterQueue&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">LowerPriority&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resourceGroups&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">coveredResources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;cpu&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;memory&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">flavors&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">default&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: cpu, nominalQuota: 200, borrowingLimit&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">100&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: memory, nominalQuota&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">800Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>The cohort lets teams borrow idle capacity, &lt;code>reclaimWithinCohort&lt;/code> lets the owner take capacity back, and &lt;code>borrowingLimit&lt;/code> stops any single tenant from draining the whole pool during a release sprint.&lt;/p>
&lt;p>A sharded suite is then an ordinary Indexed Job with a priority class:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">apiVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">batch/v1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">kind&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Job&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">metadata&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">checkout-regression-7f3a&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">kueue.x-k8s.io/queue-name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">team-payments }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">completionMode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Indexed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">completions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">40&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">parallelism&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">40&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">backoffLimitPerIndex&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">priorityClassName&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">premerge-high&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restartPolicy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Never&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">containers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">runner&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">registry.internal/test-runner:2.14&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">env&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: TEST_TOTAL_SHARDS, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;40&amp;#34;&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: TEST_SHARD_INDEX, valueFrom&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">fieldRef&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">fieldPath&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">metadata.annotations[&amp;#39;batch.kubernetes.io/job-completion-index&amp;#39;] } } }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">requests&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">cpu&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;2&amp;#34;&lt;/span>&lt;span class="nt">, memory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">4Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">limits&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">memory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">4Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>A few details are worth copying directly: &lt;code>backoffLimitPerIndex&lt;/code> retries only the shard that failed and does not rerun the other 39, which matters a lot when a shard hits an infrastructure failure; memory must have a limit, because one OOM runner on a shared node drags its neighbours down; CPU has no ceiling above the request, to absorb short bursts.&lt;/p>
&lt;p>One more thing that is easily overlooked: &lt;strong>shards should be cut by historical execution time, not by test count&lt;/strong>. A suite&amp;rsquo;s completion time depends on its slowest shard, not on how many cases a shard contains.&lt;/p>
&lt;h2 id="latency-comes-from-three-waits-each-with-its-own-lever">Latency Comes From Three Waits, Each With Its Own Lever
&lt;/h2>&lt;p>Run latency is basically made up of three segments of waiting, and each has a different optimisation lever and cost:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Wait&lt;/th>
&lt;th>Lever&lt;/th>
&lt;th>Cost&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Queueing&lt;/td>
&lt;td>Raise nominal quota, cohort borrowing&lt;/td>
&lt;td>Reserved idle capacity&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Node provisioning&lt;/td>
&lt;td>Warm buffer nodes, pre-pull runner images&lt;/td>
&lt;td>Paying for idle nodes&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Slowest shard&lt;/td>
&lt;td>Time-based sharding, more shards&lt;/td>
&lt;td>Per-Pod startup overhead&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Autoscalers do well on this kind of spiky and varied demand, because they provision nodes directly against pending Pod requirements; traditional node-group approaches still work when designed properly. Whichever you choose, keep the runner image small and cacheable &amp;ndash; at this scale, image pull time is paid tens of thousands of times a day.&lt;/p>
&lt;h2 id="security-test-code-is-semi-trusted-code">Security: Test Code Is &amp;ldquo;Semi-Trusted Code&amp;rdquo;
&lt;/h2>&lt;p>Test code executes whatever a repository commits, so it cannot be treated as a trusted workload. Practical points include: have Pods follow the Restricted security standard as far as possible; use namespaces plus RBAC to ensure one team&amp;rsquo;s service account cannot read another team&amp;rsquo;s jobs; access cloud resources through IAM Roles for Service Accounts or EKS Pod Identity rather than long-lived keys; keep test credentials in a secrets management system with least-privilege access; scan runner images and pin their versions. In regulated settings, test environments must also be isolated from production data through network policies rather than by convention.&lt;/p>
&lt;h2 id="a-typical-failure-scenario-release-day-morning">A Typical Failure Scenario: Release-Day Morning
&lt;/h2>&lt;p>Three teams submit full regression suites just before the release cutoff, while dozens of engineers are running pre-merge checks.&lt;/p>
&lt;p>Without quotas, the first regression suite to arrive would fill every available node, pre-merge checks would queue, and development across the whole organisation would stall. With the design above: each regression suite runs within its team&amp;rsquo;s nominal quota plus a limited borrowing allowance; pre-merge Pods carry a higher priority class and preempt overnight-grade work when nodes run short; the autoscaler adds capacity for the remaining backlog. The result is that regression suites finish later than they would on an empty cluster, but no single team can slam the brakes on everyone. That is the trade-off a shared platform offers.&lt;/p>
&lt;h2 id="trade-offs-and-limits-which-must-be-stated-up-front">Trade-offs and Limits, Which Must Be Stated Up Front
&lt;/h2>&lt;p>Centralization creates a team everyone depends on: its roadmap becomes the bottleneck and one of its failures stops every release at once. Multiple clusters reduce the blast radius but multiply upgrades, plugin versions and configuration drift. How much each team should get is fundamentally an organisational question &amp;ndash; the scheduler does not solve it.&lt;/p>
&lt;p>Automatic retries also mask real intermittent problems: a race condition in production code looks exactly like a flaky test. If you retry everything, the two can never be separated.&lt;/p>
&lt;p>Why not use the parallelism built into the CI system? Because runners allocated per pipeline serve one repository at a time; they cannot achieve organisation-level fair scheduling, cannot attribute infrastructure failures uniformly, and cannot produce a single certification report. A centralized platform solves these things at the cost of team autonomy. For a small team with few repositories, or for jobs that need special hardware and cannot stay warm in a shared pool, it is the wrong choice.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_de42ba8d20771171.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_3d2757c65d2f0434.webp 480w, https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_de42ba8d20771171.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>The lifecycle of one run: admission, sharded execution, failure classification, single-shard retry, certification after quarantine&lt;/p>
&lt;h2 id="five-lessons-from-production">Five Lessons From Production
&lt;/h2>&lt;p>&lt;strong>Distinguish infrastructure failure from test failure at the source.&lt;/strong> If a node eviction and a failed assertion produce the same shade of red, teams stop trusting the platform fast. The runner should report the failure cause as a first-class field: test, environment or platform.&lt;/p>
&lt;p>&lt;strong>Retry once, on a different Pod, and record it.&lt;/strong> Retrying once on different infrastructure separates environmental noise from deterministic failure. Silent, repeated retries only raise cost and bury genuine concurrency defects &amp;ndash; and concurrency is the number one root cause of flakiness.&lt;/p>
&lt;p>&lt;strong>Quarantine is a workflow, not a switch.&lt;/strong> A quarantined test must have an owner, a ticket and an expiry date, otherwise the quarantine list keeps growing until the word &amp;ldquo;certification&amp;rdquo; means nothing.&lt;/p>
&lt;p>&lt;strong>The reporting layer is the product itself.&lt;/strong> Teams judge the platform by the results view, not by the scheduler. The reporting and insights service deserves a dedicated engineering team rather than being treated as an accessory.&lt;/p>
&lt;p>&lt;strong>Onboarding is part of the platform too.&lt;/strong> When every repository depends on the same execution platform, misuse &amp;ndash; unbounded suites, missing resource declarations, retry loops baked into test code &amp;ndash; is paid for by everyone. Running continuous delivery onboarding for newcomers is, in effect, as much a platform control as any quota.&lt;/p>
&lt;h2 id="best-practices-you-can-copy-directly">Best Practices You Can Copy Directly
&lt;/h2>&lt;ul>
&lt;li>Expose a stable submission API and never let tenants depend on cluster internals;&lt;/li>
&lt;li>Give every tenant a quota, a queue and default resource limits from day one;&lt;/li>
&lt;li>Shard by historical duration; retry by shard, never by whole suite;&lt;/li>
&lt;li>Publish the flake rate of every suite and trace every run end to end.&lt;/li>
&lt;/ul>
&lt;p>Observability must be built at the platform level, not just the test level: instrument the whole path from submission through admission and scheduling to result ingest, so that &amp;ldquo;why is my run slow&amp;rdquo; can be answered stage by stage. The core service-level metrics are four: admission time, time to first result, infrastructure failure rate (counted separately from test failure rate), and per-suite flake rate. Delivery impact can be tracked with DORA metrics to see whether the platform really shortened lead time and reduced change failure rate.&lt;/p>
&lt;h2 id="the-two-most-valuable-directions-next">The Two Most Valuable Directions Next
&lt;/h2>&lt;p>One is change-based test selection. Public data shows that the vast majority of tests rarely fail, so &amp;ldquo;which ones to run&amp;rdquo; remains the biggest cost lever. The other is AI-assisted triage: it can cluster failures by likely cause, but it should assist the owner rather than auto-quarantine tests &amp;ndash; misclassification directly undermines the credibility of certification.&lt;/p>
&lt;p>The conclusion can be compressed into one sentence: a centralized test orchestration platform does more than scheduling. It also isolates tenants safely and provides an authoritative record of quality. Kubernetes provides usable primitives &amp;ndash; Indexed Jobs, priorities, quotas, queue managers &amp;ndash; but what really determines value is policy: which teams get capacity, how failures are classified, and what counts as &amp;ldquo;certified&amp;rdquo;. Queues, the failure taxonomy and the reporting layer must be built as one system; if any one of the three is untrustworthy, so are the other two.&lt;/p></description></item></channel></rss>