<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Self-Healing Locator on Testing Observation</title><link>https://chengxiaqiucao.github.io/en/tags/self-healing-locator/</link><description>Recent content in Self-Healing Locator on Testing Observation</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>|by Daniel Wei</copyright><lastBuildDate>Wed, 04 Nov 2026 09:00:00 +0800</lastBuildDate><atom:link href="https://chengxiaqiucao.github.io/en/tags/self-healing-locator/index.xml" rel="self" type="application/rss+xml"/><item><title>The Locator Broke but the Test Stayed Green - Runtime Self-Healing and Its Three Audit Gates</title><link>https://chengxiaqiucao.github.io/en/post/152-self-healing-locator-architecture/</link><pubDate>Wed, 04 Nov 2026 09:00:00 +0800</pubDate><guid>https://chengxiaqiucao.github.io/en/post/152-self-healing-locator-architecture/</guid><description>&lt;img src="https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/cover.webp" alt="Featured image of post The Locator Broke but the Test Stayed Green - Runtime Self-Healing and Its Three Audit Gates" />&lt;p>A front-end refactor renames a class and moves a button, the application works as before, but half the selectors no longer match. The traditional fix is to have someone update the locators one by one and rerun. Runtime self-healing offers another path: when a selector fails, let a model decide from the elements &lt;strong>actually present&lt;/strong> on the page which one you meant, continue execution after verification passes, and cache the fix so the next run does not call the model again.&lt;/p>
&lt;p>The idea sounds like &amp;ldquo;making tests less brittle&amp;rdquo;, but the real engineering difficulty is not fixing it &amp;ndash; it is that &lt;strong>fixing it wrongly also goes green&lt;/strong>. If the checkout button is removed and the model confidently clicks a different button, the suite is all green and nobody notices the product is broken. So what deserves a breakdown is not the model prompt but the whole mechanism of verification, auditing and cost control &amp;ndash; those decide whether self-healing is a productivity tool or a new source of silent failure.&lt;/p>
&lt;h2 id="the-six-steps-of-one-self-heal">The Six Steps of One Self-Heal
&lt;/h2>&lt;p>Taking a working implementation from the Playwright ecosystem as an example, one self-heal follows a fixed, observable flow:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Run the original locator first.&lt;/strong> Give it a very short timeout (1 second by default); if the element appears normally, perform the action directly and the self-healing logic never engages &amp;ndash; when matching works, the suite behaves exactly as before integration.&lt;/li>
&lt;li>&lt;strong>Check the cache.&lt;/strong> A locator previously fixed for the same page path, same action and same description is tried first. On a hit, no model call happens.&lt;/li>
&lt;li>&lt;strong>Narrow the candidates.&lt;/strong> Collect up to 1,000 interactive elements from the page, score and sort them by keyword match, element type, ARIA role and test id, and send only the top 40 to the model. This step prevents the target element from being cut off for being late in DOM order.&lt;/li>
&lt;li>&lt;strong>The model proposes candidate strategies.&lt;/strong> It returns at most three Playwright locator strategies &amp;ndash; test id, role with name, label, placeholder, text, alt text, title or CSS &amp;ndash; each with a confidence score; the output must conform to a fixed JSON schema, not free text.&lt;/li>
&lt;li>&lt;strong>Verify one by one.&lt;/strong> To be accepted, a suggestion must satisfy all of: it matches exactly one element, that element is visible, and confidence is not below &lt;code>minConfidence&lt;/code>.&lt;/li>
&lt;li>&lt;strong>Execute and persist.&lt;/strong> The effective locator is written to the cache file, and this self-heal is recorded as one line of JSONL in the event log, including the model&amp;rsquo;s reasoning, confidence and token consumption.&lt;/li>
&lt;/ol>
&lt;p>&amp;ldquo;Call the model once per element, not once per run&amp;rdquo; is the heart of cost control. The cache is organised by action, page path and description, and once committed to the repository the team and CI share the same fixes.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig1_hu_5d0602477b153178.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig1_hu_4e93e1d0f0437916.webp 480w, https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig1_hu_5d0602477b153178.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>The full chain from short-timeout detection through cache hit, candidate narrowing, uniqueness verification and persistence&lt;/p>
&lt;h2 id="verification-matters-more-than-generation-but-verification-alone-is-not-enough">Verification Matters More Than Generation, but Verification Alone Is Not Enough
&lt;/h2>&lt;p>Steps 4 and 5 are the most valuable part of this design: the model does not invent a selector out of thin air but chooses from candidate elements really present on the page, and the answer must resolve to one concrete, visible, unique element. By comparison, &amp;ldquo;let the model write a new CSS selector&amp;rdquo; is almost uncontrollable.&lt;/p>
&lt;p>But uniqueness and visibility only guarantee that &amp;ldquo;this element exists and is unique&amp;rdquo;, not that &amp;ldquo;it is the one you wanted&amp;rdquo;. The original article states the risk bluntly: delete the checkout button and if the model confidently picks another button, the test still passes and the defect is silently swallowed. For quality assurance, a self-healing mechanism that never reports a false positive is more dangerous than one that occasionally fails.&lt;/p>
&lt;p>Hence three gates are needed to force &amp;ldquo;self-healing happened&amp;rdquo; into a visible signal.&lt;/p>
&lt;h3 id="gate-one-confidence-threshold">Gate One: Confidence Threshold
&lt;/h3>&lt;p>&lt;code>minConfidence&lt;/code> defaults to 0, meaning any score is accepted. In production it should be set explicitly to something like 0.8: when the model is not sure enough, better to let the action fail than to guess. This switch turns &amp;ldquo;uncertain&amp;rdquo; back into a red light.&lt;/p>
&lt;h3 id="gate-two-report-only-mode">Gate Two: Report-Only Mode
&lt;/h3>&lt;p>&lt;code>warn&lt;/code> mode records &amp;ldquo;how it would have been fixed&amp;rdquo; but keeps the original failure; &lt;strong>cached fixes are also only reported, not applied&lt;/strong>. The latter point is especially critical: otherwise a cache file committed to the repository can quietly turn an audit run all green, and the team never sees which selectors should have been updated long ago.&lt;/p>
&lt;h3 id="gate-three-make-the-build-answer-for-self-healing">Gate Three: Make the Build Answer for Self-Healing
&lt;/h3>&lt;p>Provide a run-level summary interface that counts how many self-heals happened in this run, lists the selectors to fix in global teardown, and fails the build:&lt;/p>
&lt;div class="highlight">&lt;div class="chroma">
&lt;table class="lntable">&lt;tr>&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code>&lt;span class="lnt"> 1
&lt;/span>&lt;span class="lnt"> 2
&lt;/span>&lt;span class="lnt"> 3
&lt;/span>&lt;span class="lnt"> 4
&lt;/span>&lt;span class="lnt"> 5
&lt;/span>&lt;span class="lnt"> 6
&lt;/span>&lt;span class="lnt"> 7
&lt;/span>&lt;span class="lnt"> 8
&lt;/span>&lt;span class="lnt"> 9
&lt;/span>&lt;span class="lnt">10
&lt;/span>&lt;/code>&lt;/pre>&lt;/td>
&lt;td class="lntd">
&lt;pre tabindex="0" class="chroma">&lt;code class="language-ts" data-lang="ts">&lt;span class="line">&lt;span class="cl">&lt;span class="c1">// global-teardown.ts
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="kr">export&lt;/span> &lt;span class="k">default&lt;/span> &lt;span class="kd">function&lt;/span> &lt;span class="p">()&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="kr">const&lt;/span> &lt;span class="p">{&lt;/span> &lt;span class="nx">healed&lt;/span>&lt;span class="p">,&lt;/span> &lt;span class="nx">entries&lt;/span> &lt;span class="p">}&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="nx">getHealSummary&lt;/span>&lt;span class="p">();&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nx">healed&lt;/span> &lt;span class="o">===&lt;/span> &lt;span class="mi">0&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="k">return&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">console&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">error&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="sb">`&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nx">healed&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="sb"> locators need fixing, please update the following selectors:`&lt;/span>&lt;span class="p">);&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="kr">const&lt;/span> &lt;span class="nx">e&lt;/span> &lt;span class="k">of&lt;/span> &lt;span class="nx">entries&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">filter&lt;/span>&lt;span class="p">((&lt;/span>&lt;span class="nx">e&lt;/span>&lt;span class="p">)&lt;/span> &lt;span class="o">=&amp;gt;&lt;/span> &lt;span class="nx">e&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">outcome&lt;/span> &lt;span class="o">===&lt;/span> &lt;span class="s1">&amp;#39;healed&amp;#39;&lt;/span>&lt;span class="p">))&lt;/span> &lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">console&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">error&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="sb">` &lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nx">e&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">action&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="sb"> &amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nx">e&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">contextName&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="sb">&amp;#34; -&amp;gt; &lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nx">JSON&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">stringify&lt;/span>&lt;span class="p">(&lt;/span>&lt;span class="nx">e&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">strategy&lt;/span>&lt;span class="p">)&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="sb">`&lt;/span>&lt;span class="p">);&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nx">process&lt;/span>&lt;span class="p">.&lt;/span>&lt;span class="nx">exitCode&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="mi">1&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/td>&lt;/tr>&lt;/table>
&lt;/div>
&lt;/div>&lt;p>There is a process-model pitfall here: tests run in worker processes and teardown runs in the runner process, so in-memory data is not shared &amp;ndash; each result must additionally be written to a JSONL summary file, from which the summary interface reads; and counting must cover only the current run, so if the runner does not exit between runs (UI mode, watch mode), reset the counters explicitly in globalSetup.&lt;/p>
&lt;p>The combined semantics of the three gates are clear: &lt;strong>self-healing is a temporary means of keeping the build unblocked, not a licence to stop maintaining selectors.&lt;/strong>&lt;/p>
&lt;h2 id="controlling-cost-speed-and-privacy">Controlling Cost, Speed and Privacy
&lt;/h2>&lt;p>Every self-heal calls the model once, and left unchecked the cost and latency rise linearly with the number of failing selectors. Four reusable optimizations:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Compress candidate attributes&lt;/strong>: send only attributes that have values, and use short keys (such as &lt;code>tid&lt;/code> for test id and &lt;code>txt&lt;/code> for text);&lt;/li>
&lt;li>&lt;strong>Flag hidden elements instead of dropping them&lt;/strong>: real applications often deliberately hide the target element (custom-styled checkboxes, buttons that only appear on hover), so such elements are sent along with &lt;code>hid: true&lt;/code>;&lt;/li>
&lt;li>&lt;strong>Sort before truncating&lt;/strong>: after scoring by keyword, element type, ARIA role and test id, take only the first &lt;code>maxCandidates&lt;/code>, default 40; simple pages can drop to 15, dense pages can rise to 80;&lt;/li>
&lt;li>&lt;strong>Limit output length&lt;/strong>: the model returns at most three strategies, and token consumption is printed on every self-heal and recorded in the event log, so it can be accounted per team.&lt;/li>
&lt;/ul>
&lt;p>The scope of what is sent must also be clear: the model receives a structured list of candidate elements (tag name, role, ARIA labels, text, test id and so on) plus the page URL, and &lt;strong>no screenshot and no full page HTML&lt;/strong>. If the DOM may contain personal information, financial data or internal addresses, the options include using a local model so data never leaves the machine, pointing at a self-hosted gateway, enabling it only in non-production environments, and periodically reviewing what was actually sent as recorded in the event log.&lt;/p>
&lt;p>&lt;img src="https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig2_hu_600ab10795af6ac9.webp"
width="1024"
height="683"
srcset="https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig2_hu_9d1d4d0015ba507d.webp 480w, https://chengxiaqiucao.github.io/post/152-self-healing-locator-architecture/06-fig2_hu_600ab10795af6ac9.webp 1024w"
loading="lazy"
class="gallery-image"
data-flex-grow="150"
data-flex-basis="360px"
>
&lt;/p>
&lt;p>Where the three audit gates and the cost control points sit in the flow&lt;/p>
&lt;h3 id="empirical-trade-offs-for-local-models">Empirical Trade-offs for Local Models
&lt;/h3>&lt;p>For a fully offline local inference setup, one set of comparison data has been published and is well suited as a selection reference:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Model&lt;/th>
&lt;th>Size&lt;/th>
&lt;th>Test result&lt;/th>
&lt;th>Characteristics&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>llama3.2:3b&lt;/code>&lt;/td>
&lt;td>2.0 GB&lt;/td>
&lt;td>5/5 passed&lt;/td>
&lt;td>Best overall: fast, accurate, small&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>mistral&lt;/code>&lt;/td>
&lt;td>4.1 GB&lt;/td>
&lt;td>5/5 passed&lt;/td>
&lt;td>Most accurate but slowest, about 1.4 minutes per run&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>gemma3:4b&lt;/code>&lt;/td>
&lt;td>3.3 GB&lt;/td>
&lt;td>4/5 passed&lt;/td>
&lt;td>Faster than mistral (about 53 seconds), slightly lower accuracy&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Small models from 1B to 4B struggle in scenarios like &amp;ldquo;a row of nearly identical checkboxes&amp;rdquo;; when accuracy matters, choose a larger model. The cost of the local option is that model latency per self-heal is usually higher than cloud (about 1-2 seconds in the cloud), in exchange for zero API keys, zero data egress and zero call fees.&lt;/p>
&lt;p>There is another class of model that does not &amp;ldquo;generate text&amp;rdquo; and deserves separate mention: it does not write selectors but answers a multiple-choice question with calibrated probability &amp;ndash; among the collected candidate elements, which one is the element you described, and is that element even on the page. The answer can only be an element that really exists on the page, its probability is used directly as the confidence checked against &lt;code>minConfidence&lt;/code>, and the locator is then constructed by code from that element&amp;rsquo;s own attributes. On an evaluation set containing 76 deliberately broken locators (covering buttons, links, inputs, checkboxes, ARIA widgets and hidden controls), this approach fixed all of them; the cost per call is about $0.0001, and a round trip with 40 candidates takes under a quarter of a second. When the description matches no element on the page, it answers plainly &amp;ldquo;none&amp;rdquo; and puts the best guess with its score into the error message, so a badly written description is easy to spot too.&lt;/p>
&lt;h2 id="the-four-config-options-most-worth-tuning">The Four Config Options Most Worth Tuning
&lt;/h2>&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Option&lt;/th>
&lt;th>Default&lt;/th>
&lt;th>Tuning advice&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>quickTimeout&lt;/code>&lt;/td>
&lt;td>1000 ms&lt;/td>
&lt;td>Deliberately short so bad selectors fail fast; raise it if the application really has slow-rendering regions, otherwise &amp;ldquo;slow&amp;rdquo; is misjudged as &amp;ldquo;broken&amp;rdquo; and a model call is wasted&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>minConfidence&lt;/code>&lt;/td>
&lt;td>0&lt;/td>
&lt;td>Set to around 0.8 so uncertainty becomes failure&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>maxCandidates&lt;/code>&lt;/td>
&lt;td>40&lt;/td>
&lt;td>Drop to 15 on simple pages, raise to 80 on dense pages; this directly determines tokens and accuracy&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>maxAiTries&lt;/code>&lt;/td>
&lt;td>4&lt;/td>
&lt;td>How many suggestions to verify before giving up, used together with the three-strategy output cap&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>Artifact files need clear responsibilities: the cache file (fixed locators, organised by action / page path / description) should be committed to the repository so the team and CI share it; the event log (reason, confidence, token count of each attempt) and the summary log go into the ignore list; lock files and temporary files likewise. You can also choose not to commit the whole directory and let each machine maintain its own fixes, at the cost of the model being called again on every CI run.&lt;/p>
&lt;h2 id="known-limitations">Known Limitations
&lt;/h2>&lt;ul>
&lt;li>Cloud models need a key and a network; local models need neither but have higher latency;&lt;/li>
&lt;li>Only the main frame is searched; elements inside iframes are not candidates yet, while open shadow roots can be found;&lt;/li>
&lt;li>A click with &lt;code>force: true&lt;/code> dispatches the event directly without pointer coordinates, so it cannot cover interactions that depend on real pointer position;&lt;/li>
&lt;li>Parallel workers share the cache through a lock file, which is best effort, not a database;&lt;/li>
&lt;li>Self-healing is off by default and only takes effect when the switch is explicitly turned on, so suite behaviour is unchanged during early integration and the risk is contained.&lt;/li>
&lt;/ul>
&lt;h2 id="a-suggested-integration-order">A Suggested Integration Order
&lt;/h2>&lt;ol>
&lt;li>Keep self-healing off and run in &lt;code>warn&lt;/code> mode for one or two weeks first, collecting a list of &amp;ldquo;which selectors break most often&amp;rdquo; &amp;ndash; that list is itself the debt register of your automation assets;&lt;/li>
&lt;li>Fix the frequently failing selectors by hand first and leave the rest to self-healing, so architectural problems do not get hidden in the cache;&lt;/li>
&lt;li>When turning self-healing on, set &lt;code>minConfidence&lt;/code> at the same time and enable the &amp;ldquo;any self-heal fails the build&amp;rdquo; rule in global teardown;&lt;/li>
&lt;li>Every change to the cache file goes through code review; treat it as test code, not a build artifact;&lt;/li>
&lt;li>Enable it only in non-production environments, and switch to a local model or self-hosted endpoint where sensitive DOM is involved;&lt;/li>
&lt;li>Regularly review token consumption and the distribution of reasons in the event log, to judge whether &amp;ldquo;there are more refactors&amp;rdquo; or &amp;ldquo;the descriptions are vague&amp;rdquo;.&lt;/li>
&lt;/ol>
&lt;p>What the self-healing mechanism really solves is the queueing problem during a refactor, when a large number of cases go red at once and people cannot fix them fast enough. What it does not solve, and should not try to solve, is &amp;ldquo;what the tests are actually verifying&amp;rdquo;. Make every self-heal a record that must be seen, reviewed and eventually eliminated, and this capability will not degrade into a green light that also flattens real defects.&lt;/p></description></item></channel></rss>