<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Test Platform on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/test-platform/</link><description>Recent content in Test Platform on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 28 Oct 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/test-platform/index.xml" rel="self" type="application/rss+xml"/><item><title>250,000 Test Runs a Day - Architecture and Trade-offs of a Centralized Test Execution Platform</title><link>https://chengxiaqiucao.github.io/en/post/144-centralized-test-platform/</link><pubDate>Wed, 28 Oct 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/144-centralized-test-platform/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/cover.webp" alt="Featured image of post 250,000 Test Runs a Day - Architecture and Trade-offs of a Centralized Test Execution Platform" />&lt;p>Test automation is usually discussed as a framework problem. But when thousands of repositories all need to run their own suites, the nature of the problem changes &amp;ndash; it becomes an infrastructure problem. When every team pushes at nine in the morning, who gets execution capacity? When the nightly full regression collides with the pre-merge check blocking some developer, which one runs first? And when a good portion of failures are not real defects, on what basis do you declare a build &amp;ldquo;certified&amp;rdquo;?&lt;/p>
&lt;p>One class of platform grows out of exactly this pressure: it does not help you write tests, it provides a shared execution and certification infrastructure. This article breaks down the architecture, key policies and real trade-offs of such a platform, with data from an instance that runs about 250,000 automated test executions a day across four Amazon EKS clusters and serves all engineering repositories of a large financial institution.&lt;/p>
&lt;h2 id="start-by-seeing-the-five-characteristics-of-the-workload">Start by Seeing the Five Characteristics of the Workload
&lt;/h2>&lt;p>Before designing, you must acknowledge that a test execution workload has several properties that break naive designs:&lt;/p>
&lt;p>&lt;strong>Extreme bursts.&lt;/strong> Demand spikes during working hours and release windows, while capacity sits idle the rest of the time &amp;ndash; always swinging between &amp;ldquo;not enough&amp;rdquo; and &amp;ldquo;wasted&amp;rdquo;.&lt;/p>
&lt;p>&lt;strong>Mixed urgency.&lt;/strong> A pre-merge check holds up one engineer&amp;rsquo;s time, the nightly full regression holds up nobody&amp;rsquo;s, yet both compete for the same pool of resources with completely different latency requirements.&lt;/p>
&lt;p>&lt;strong>Heterogeneous tenants.&lt;/strong> Different teams bring different runtimes, browser stacks and data dependencies; a suite with no resource constraints can starve every other suite.&lt;/p>
&lt;p>&lt;strong>Untrustworthy signals.&lt;/strong> Empirical research shows that async waits, concurrency and test order dependencies keep producing flaky results. Public engineering experience reports that about 1.5% of test runs produced flaky results and nearly 16% of tests showed some degree of instability. Apply that ratio to 250,000 executions a day and 1.5% is about 3,750 misleading results &amp;ndash; if retries, classification and quarantine are not platform-level policies but per-team improvisation, then &amp;ldquo;red&amp;rdquo; loses its meaning entirely.&lt;/p>
&lt;p>&lt;strong>Audit requirements.&lt;/strong> In a regulated environment, &amp;ldquo;this build passed certification&amp;rdquo; must be reproducible from records, not from memory.&lt;/p>
&lt;h2 id="three-kubernetes-primitives-carry-most-of-the-design">Three Kubernetes Primitives Carry Most of the Design
&lt;/h2>&lt;p>Such a platform usually relies on only a few Kubernetes capabilities:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Job&amp;rsquo;s Indexed completion mode&lt;/strong>: each Pod gets a stable index, which maps naturally onto a test shard;&lt;/li>
&lt;li>&lt;strong>PriorityClass&lt;/strong>: a high-priority Pod that cannot be scheduled can preempt lower-priority Pods;&lt;/li>
&lt;li>&lt;strong>ResourceQuota and LimitRange&lt;/strong>: limit how much a namespace can consume and set default requests, avoiding unbounded containers on a shared cluster.&lt;/li>
&lt;/ul>
&lt;p>On top of that, queue managers such as Kueue provide queue-level quota, deciding whether a job waits, is admitted or is preempted; autoscalers such as Karpenter provision nodes directly for Pods that cannot be scheduled.&lt;/p>
&lt;h2 id="architecture-control-plane-execution-plane-results-plane">Architecture: Control Plane, Execution Plane, Results Plane
&lt;/h2>&lt;p>The platform separates three things: &amp;ldquo;deciding what to run&amp;rdquo; (control plane), &amp;ldquo;actually running it&amp;rdquo; (execution plane) and &amp;ldquo;turning execution into decisions&amp;rdquo; (results plane).&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_93a37452f58422c9.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_33329837ce4e233a.webp 480w, https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig1_hu_93a37452f58422c9.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Control plane, execution plane and results plane are separated, yet the three must be built as one system&lt;/p>
&lt;p>&lt;strong>Submission.&lt;/strong> Each team&amp;rsquo;s CI system triggers a run, declaring the suite, the target environment and an urgency class. Teams never touch Kubernetes directly &amp;ndash; the submission API is the contract, so execution details can be replaced without affecting callers.&lt;/p>
&lt;p>&lt;strong>Admission and queuing.&lt;/strong> Each tenant gets a namespace with a quota and a local queue. The queue manager suspends workloads until quota is available, and performs preemption and fair sharing within a cohort (capacity pool): idle capacity borrowed by one team can be reclaimed when another team needs its own share. A borrowing cap prevents a single tenant from swallowing the whole pool during a release sprint.&lt;/p>
&lt;p>&lt;strong>Shard planning.&lt;/strong> Large suites are cut into shards and executed as an Indexed Job, with the shard index exposed to the runner through an environment variable &amp;ndash; the same contract as Bazel&amp;rsquo;s &lt;code>TEST_TOTAL_SHARDS&lt;/code> / &lt;code>TEST_SHARD_INDEX&lt;/code>.&lt;/p>
&lt;p>&lt;strong>Execution plane.&lt;/strong> Four EKS clusters host the platform&amp;rsquo;s 40 applications plus a large number of ephemeral test Pods. Multiple clusters trade higher control-plane and tooling costs for isolation and a smaller blast radius &amp;ndash; a trade-off explicitly documented in multi-tenant designs. Placement policy needs deliberate design: give each tenant a &amp;ldquo;home cluster&amp;rdquo; where image caches and test data stay warm; when the home cluster&amp;rsquo;s queue time exceeds a threshold, the control plane spills the &lt;strong>entire run&lt;/strong> to another cluster rather than splitting one run across clusters &amp;ndash; splitting makes result collection and infrastructure failure attribution extremely hard.&lt;/p>
&lt;p>&lt;strong>Results and certification.&lt;/strong> Each shard sends structured result messages to a reporting and insights service, which decides pass/fail per suite, identifies flakes, and registers a certification verdict on the release. The most important contract in the whole platform is this event schema:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;runId&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;checkout-regression-7f3a&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;shardIndex&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">17&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;attempt&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">2&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;test&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;CheckoutFlowTest.appliesPromoCode&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;outcome&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;passed&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;failureCause&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="kc">null&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;previousAttemptCause&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;environment&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;node&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;ip-10-0-42-7&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;durationMs&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">48210&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>&lt;code>failureCause&lt;/code> (test / environment / platform) and &lt;code>attempt&lt;/code> are what let the results layer distinguish &amp;ldquo;a flaky test&amp;rdquo; from &amp;ldquo;a node was lost&amp;rdquo;, and they are also the basis for charging infrastructure failures to the platform&amp;rsquo;s account rather than the team&amp;rsquo;s.&lt;/p>
&lt;h2 id="what-the-configuration-looks-like">What the Configuration Looks Like
&lt;/h2>&lt;p>A tenant&amp;rsquo;s queue and quota are declared once by the platform team:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">apiVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">kueue.x-k8s.io/v1beta1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">kind&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ClusterQueue&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">metadata&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">team-payments }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">cohort&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">shared-test-capacity&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">preemption&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">reclaimWithinCohort&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Any&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">withinClusterQueue&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">LowerPriority&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resourceGroups&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">coveredResources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;cpu&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;memory&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">flavors&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">default&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: cpu, nominalQuota: 200, borrowingLimit&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">100&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: memory, nominalQuota&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">800Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>The cohort lets teams borrow idle capacity, &lt;code>reclaimWithinCohort&lt;/code> lets the owner take capacity back, and &lt;code>borrowingLimit&lt;/code> stops any single tenant from draining the whole pool during a release sprint.&lt;/p>
&lt;p>A sharded suite is then an ordinary Indexed Job with a priority class:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;span class="lnt">11
&lt;/span>&lt;span class="lnt">12
&lt;/span>&lt;span class="lnt">13
&lt;/span>&lt;span class="lnt">14
&lt;/span>&lt;span class="lnt">15
&lt;/span>&lt;span class="lnt">16
&lt;/span>&lt;span class="lnt">17
&lt;/span>&lt;span class="lnt">18
&lt;/span>&lt;span class="lnt">19
&lt;/span>&lt;span class="lnt">20
&lt;/span>&lt;span class="lnt">21
&lt;/span>&lt;span class="lnt">22
&lt;/span>&lt;span class="lnt">23
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">apiVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">batch/v1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">kind&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Job&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">metadata&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">checkout-regression-7f3a&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">kueue.x-k8s.io/queue-name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">team-payments }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">completionMode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Indexed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">completions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">40&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">parallelism&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">40&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">backoffLimitPerIndex&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">spec&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">priorityClassName&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">premerge-high&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restartPolicy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Never&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">containers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">runner&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">registry.internal/test-runner:2.14&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">env&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: TEST_TOTAL_SHARDS, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;40&amp;#34;&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="w"> &lt;/span>&lt;span class="nt">name: TEST_SHARD_INDEX, valueFrom&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">fieldRef&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">fieldPath&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">metadata.annotations[&amp;#39;batch.kubernetes.io/job-completion-index&amp;#39;] } } }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resources&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">requests&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">cpu&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;2&amp;#34;&lt;/span>&lt;span class="nt">, memory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">4Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">limits&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="w"> &lt;/span>&lt;span class="nt">memory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">4Gi }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>A few details are worth copying directly: &lt;code>backoffLimitPerIndex&lt;/code> retries only the shard that failed and does not rerun the other 39, which matters a lot when a shard hits an infrastructure failure; memory must have a limit, because one OOM runner on a shared node drags its neighbours down; CPU has no ceiling above the request, to absorb short bursts.&lt;/p>
&lt;p>One more thing that is easily overlooked: &lt;strong>shards should be cut by historical execution time, not by test count&lt;/strong>. A suite&amp;rsquo;s completion time depends on its slowest shard, not on how many cases a shard contains.&lt;/p>
&lt;h2 id="latency-comes-from-three-waits-each-with-its-own-lever">Latency Comes From Three Waits, Each With Its Own Lever
&lt;/h2>&lt;p>Run latency is basically made up of three segments of waiting, and each has a different optimisation lever and cost:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Wait&lt;/th>
&lt;th>Lever&lt;/th>
&lt;th>Cost&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Queueing&lt;/td>
&lt;td>Raise nominal quota, cohort borrowing&lt;/td>
&lt;td>Reserved idle capacity&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Node provisioning&lt;/td>
&lt;td>Warm buffer nodes, pre-pull runner images&lt;/td>
&lt;td>Paying for idle nodes&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Slowest shard&lt;/td>
&lt;td>Time-based sharding, more shards&lt;/td>
&lt;td>Per-Pod startup overhead&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Autoscalers do well on this kind of spiky and varied demand, because they provision nodes directly against pending Pod requirements; traditional node-group approaches still work when designed properly. Whichever you choose, keep the runner image small and cacheable &amp;ndash; at this scale, image pull time is paid tens of thousands of times a day.&lt;/p>
&lt;h2 id="security-test-code-is-semi-trusted-code">Security: Test Code Is &amp;ldquo;Semi-Trusted Code&amp;rdquo;
&lt;/h2>&lt;p>Test code executes whatever a repository commits, so it cannot be treated as a trusted workload. Practical points include: have Pods follow the Restricted security standard as far as possible; use namespaces plus RBAC to ensure one team&amp;rsquo;s service account cannot read another team&amp;rsquo;s jobs; access cloud resources through IAM Roles for Service Accounts or EKS Pod Identity rather than long-lived keys; keep test credentials in a secrets management system with least-privilege access; scan runner images and pin their versions. In regulated settings, test environments must also be isolated from production data through network policies rather than by convention.&lt;/p>
&lt;h2 id="a-typical-failure-scenario-release-day-morning">A Typical Failure Scenario: Release-Day Morning
&lt;/h2>&lt;p>Three teams submit full regression suites just before the release cutoff, while dozens of engineers are running pre-merge checks.&lt;/p>
&lt;p>Without quotas, the first regression suite to arrive would fill every available node, pre-merge checks would queue, and development across the whole organisation would stall. With the design above: each regression suite runs within its team&amp;rsquo;s nominal quota plus a limited borrowing allowance; pre-merge Pods carry a higher priority class and preempt overnight-grade work when nodes run short; the autoscaler adds capacity for the remaining backlog. The result is that regression suites finish later than they would on an empty cluster, but no single team can slam the brakes on everyone. That is the trade-off a shared platform offers.&lt;/p>
&lt;h2 id="trade-offs-and-limits-which-must-be-stated-up-front">Trade-offs and Limits, Which Must Be Stated Up Front
&lt;/h2>&lt;p>Centralization creates a team everyone depends on: its roadmap becomes the bottleneck and one of its failures stops every release at once. Multiple clusters reduce the blast radius but multiply upgrades, plugin versions and configuration drift. How much each team should get is fundamentally an organisational question &amp;ndash; the scheduler does not solve it.&lt;/p>
&lt;p>Automatic retries also mask real intermittent problems: a race condition in production code looks exactly like a flaky test. If you retry everything, the two can never be separated.&lt;/p>
&lt;p>Why not use the parallelism built into the CI system? Because runners allocated per pipeline serve one repository at a time; they cannot achieve organisation-level fair scheduling, cannot attribute infrastructure failures uniformly, and cannot produce a single certification report. A centralized platform solves these things at the cost of team autonomy. For a small team with few repositories, or for jobs that need special hardware and cannot stay warm in a shared pool, it is the wrong choice.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_de42ba8d20771171.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_3d2757c65d2f0434.webp 480w, https://chengxiaqiucao.github.io/post/144-centralized-test-platform/04-fig2_hu_de42ba8d20771171.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>The lifecycle of one run: admission, sharded execution, failure classification, single-shard retry, certification after quarantine&lt;/p>
&lt;h2 id="five-lessons-from-production">Five Lessons From Production
&lt;/h2>&lt;p>&lt;strong>Distinguish infrastructure failure from test failure at the source.&lt;/strong> If a node eviction and a failed assertion produce the same shade of red, teams stop trusting the platform fast. The runner should report the failure cause as a first-class field: test, environment or platform.&lt;/p>
&lt;p>&lt;strong>Retry once, on a different Pod, and record it.&lt;/strong> Retrying once on different infrastructure separates environmental noise from deterministic failure. Silent, repeated retries only raise cost and bury genuine concurrency defects &amp;ndash; and concurrency is the number one root cause of flakiness.&lt;/p>
&lt;p>&lt;strong>Quarantine is a workflow, not a switch.&lt;/strong> A quarantined test must have an owner, a ticket and an expiry date, otherwise the quarantine list keeps growing until the word &amp;ldquo;certification&amp;rdquo; means nothing.&lt;/p>
&lt;p>&lt;strong>The reporting layer is the product itself.&lt;/strong> Teams judge the platform by the results view, not by the scheduler. The reporting and insights service deserves a dedicated engineering team rather than being treated as an accessory.&lt;/p>
&lt;p>&lt;strong>Onboarding is part of the platform too.&lt;/strong> When every repository depends on the same execution platform, misuse &amp;ndash; unbounded suites, missing resource declarations, retry loops baked into test code &amp;ndash; is paid for by everyone. Running continuous delivery onboarding for newcomers is, in effect, as much a platform control as any quota.&lt;/p>
&lt;h2 id="best-practices-you-can-copy-directly">Best Practices You Can Copy Directly
&lt;/h2>&lt;ul>
&lt;li>Expose a stable submission API and never let tenants depend on cluster internals;&lt;/li>
&lt;li>Give every tenant a quota, a queue and default resource limits from day one;&lt;/li>
&lt;li>Shard by historical duration; retry by shard, never by whole suite;&lt;/li>
&lt;li>Publish the flake rate of every suite and trace every run end to end.&lt;/li>
&lt;/ul>
&lt;p>Observability must be built at the platform level, not just the test level: instrument the whole path from submission through admission and scheduling to result ingest, so that &amp;ldquo;why is my run slow&amp;rdquo; can be answered stage by stage. The core service-level metrics are four: admission time, time to first result, infrastructure failure rate (counted separately from test failure rate), and per-suite flake rate. Delivery impact can be tracked with DORA metrics to see whether the platform really shortened lead time and reduced change failure rate.&lt;/p>
&lt;h2 id="the-two-most-valuable-directions-next">The Two Most Valuable Directions Next
&lt;/h2>&lt;p>One is change-based test selection. Public data shows that the vast majority of tests rarely fail, so &amp;ldquo;which ones to run&amp;rdquo; remains the biggest cost lever. The other is AI-assisted triage: it can cluster failures by likely cause, but it should assist the owner rather than auto-quarantine tests &amp;ndash; misclassification directly undermines the credibility of certification.&lt;/p>
&lt;p>The conclusion can be compressed into one sentence: a centralized test orchestration platform does more than scheduling. It also isolates tenants safely and provides an authoritative record of quality. Kubernetes provides usable primitives &amp;ndash; Indexed Jobs, priorities, quotas, queue managers &amp;ndash; but what really determines value is policy: which teams get capacity, how failures are classified, and what counts as &amp;ldquo;certified&amp;rdquo;. Queues, the failure taxonomy and the reporting layer must be built as one system; if any one of the three is untrustworthy, so are the other two.&lt;/p></description></item></channel></rss>