Featured image of post Handing Change Points to an Agent - Precision and Latency in Production Monitoring Need Not Be a Trade-off

Handing Change Points to an Agent - Precision and Latency in Production Monitoring Need Not Be a Trade-off

A rule of thumb long circulated in operations is that about 70% of production incidents are caused by changes to a running system - binary releases, configuration pushes, feature flag switches. Given that, concentrating monitoring resources…

A rule of thumb long circulated in operations is that about 70% of production incidents are caused by changes to a running system – binary releases, configuration pushes, feature flag switches. Given that, concentrating monitoring resources near those change points should in theory contain the impact of most regressions at minimal cost. Yet in engineering this is widely acknowledged as hard: not that nobody can write an alert, but that it is very hard to be both “accurate” and “early”, and harder still to make it run by itself inside an organisation.

Large-model agents are changing that equation. What deserves attention is not the concept of “AI operations” but a specific engineering shape: let the model read code and write monitoring specs, let deterministic programs execute the specs, and let the model investigate the monitoring items that fire. This structure can already find regressions in systems it knows nothing about at prototype stage, with better precision and latency than a year ago.

Why Change-Point Monitoring Has Always Been Hard to Land

First, see where the old approach got stuck. Change-point monitoring is not a technical problem but the “takeoff” problem of platform engineering, and it has three typical dilemmas.

The first is picking the wrong first users. The natural move is to pilot with the teams that have the best operations foundation and the strongest motivation. But those teams’ releases are already relatively safe, so the platform does not actually raise the overall operations water level; the value it offers is mainly toil reduction, a nice-to-have rather than a first-order safety improvement. Worse, precision is easy to achieve with such teams, so the new system ends up competing with existing alerts on recall and latency, with little chance of winning.

The second is too wide a spread gets drowned in noise. Switching to all services with only a small set of generic health metrics does give coverage, but those metrics carry irreducible noise, so precision is poor, alert fatigue sets in, and the platform gets switched off quickly.

The third is the structural trade-off between precision and latency. Before large models entered, available signals lined up along a spectrum: structured logs were precise but slow, metrics were coarse but fast, alerts were coarser still with unsatisfactory recall and precision but “at least there was something”. Getting both speed and accuracy inside a change window required a human to pre-label the metrics and pre-define the signals – which depends on long-term organisational investment and the experience of senior engineers, experience that only years on the front line can build.

There is one more piece of background worth counting: as code output and release volume grow fast, the share of the system’s “code that any single operations person has read” keeps falling. The system increasingly looks like something that was raised rather than fully designed, and people’s mental model of its internal state keeps thinning. That actually amplifies the value of automated investigation.

The Effective Shape: an Agent That Writes Specs Plus a Deterministic System That Executes Them

Why the agent works in this scenario is quite plain: it is happy to fire a large number of queries, can write queries on the spot for one specific change, and never tires of scanning logs. The trade-offs once forced by saving money and headcount – how much to query, how finely, for how long – are greatly relaxed, so precision and recall can be obtained at the same time.

A working minimal prototype has three parts:

  • A plan generator: reads the service code (especially the configuration consumption logic), proposes a monitoring plan for this change, and specifies which signals to watch, how long the window is, and what conditions count as anomalous;
  • An execution scaffold: compiles the plan into deterministic monitoring tasks and runs them on a cadence, handling collection, comparison and triggering, without depending on the model staying online;
  • An investigation agent: engages only when a monitoring item fires, reads code, checks metrics and logs, gives an explanation, and keeps a channel for escalating to a human.

The most important engineering judgement here is that a pure agent loop is not the optimal shape. Having the model write a spec for “how to monitor this” and then handing it to a deterministic system to execute is cleaner and uses far fewer tokens than having the model loop over queries itself. The real “secret sauce” to polish in the prototype is not model capability but three very traditional pieces of engineering: noise-reduction heuristics, prompts that get the model to write high-quality specs, and a labelled evaluation set large enough to verify whether each subsequent adjustment is actually better.

In other words, a monitoring agent needs scaffolding just like a coding agent. The round of adaptation the software development lifecycle went through for coding agents has not yet happened in monitoring.

The three-stage structure of plan generation, deterministic execution and post-trigger investigation, plus the human escalation channel

Three Structural Advantages Humans Cannot Match, or Cannot Afford

Once you accept the two premises – that an agent can be better at monitoring than a person, and that a monitoring agent needs scaffolding – several advantages become very concrete.

All-hours, full-volume tracking. If inference cost is low enough, then any regression in any metric you care about, or any suspicious log line, can be investigated without first crossing the tangent line of “worth a human’s attention”. Traditional on-call needs shifts across time zones; an agent does not.

Parallel fan-out investigation. During incident triage you can dispatch several investigation threads at once, each tracing a possible causal chain, and then quickly consolidate. This is already how many teams do incident triage – a person manually starts a batch of investigation tasks; the starting action itself could well be handed to an agent, letting the “incident responder” role scale elastically on demand. This is especially true for security incidents: attackers already use agent clusters, so defenders naturally need the corresponding capability.

Faster time to understanding. Models read code, query and digest metrics and logs faster than people. Speed does not always translate directly into shorter detection time – someone with extremely deep system knowledge may still find the key signal faster by intuition – but as long as tacit knowledge is written down and retrievable, an agent will be consistently ahead at going from zero to understanding. For systems with thin operations experience, or that nobody really knows, the gap is even wider.

From Read-Only to Action: A Path of Tiered Delegation

What has been discussed so far is mostly read-only monitoring. The real incremental value is on the write path, but delegation must be tiered, with corresponding guardrails at each tier.

Tier one: more diligent observation. Nobody used to pay a full-time engineer to walk through every commit of every service, pick out what should be monitored, and hang a batch of fast-firing alerts on those changes specifically. At current inference costs this becomes feasible. Likewise, a model can watch performance data, read heap dumps, stack traces and CPU profiles, and propose optimizations from the patterns; the suggestions go into A/B experiments as usual and are monitored like any other change. One documented example: after several rounds of model investigation on a public hot library that had already been carefully optimized by hand, it found roughly another 50% heap memory saving and a 30% latency reduction. Give a strong enough model a virtual machine and a benchmark, and its ability to climb the local optimum is considerable.

Tier two: deterministic remediation actions. Scaling instances, rotating compute pools, rolling back a configuration or a binary – these actions have a clear scope, are predictable and easy to roll back, and are suitable to delegate to an agent first.

Tier three: open-ended fixes. Locate the defect, propose a patch, redeploy. Risk rises with the scope of authorization, but as capability expands, more and more write-path work can be responsibly delegated. This is exactly what makes self-healing systems attractive, provided every tier has audit and rollback.

Three tiers of delegation from read-only observation to open-ended fixing, each with different guardrails and rollback requirements

Cost Is No Longer the Constraint, but the Evaluation Set Is

With current lightweight models, work such as watching charts and comparing trends is already within reach. For a mid-sized technology company, continuous monitoring of the entire service fleet lands at roughly the low hundreds of dollars a day, and a startup-scale fleet is only a fraction of that. Compared with human on-call, or even with the cost of observability infrastructure itself, this number is not a decision obstacle – one team saved millions of dollars a year in observability spend after a single optimization round, and that was only a part of one team’s spend on it.

With cost out of the way, the bottleneck shifts to “how do we know monitoring is getting better”. That is why the labelled evaluation set cannot be skipped: without it, changing prompts, changing noise-reduction rules or switching models is all guesswork, and the failure mode of a monitoring system is precisely silence – nobody discovers the problems that were never reported.

From this a virtuous cycle can also form: coding agents let teams deliver more systems, monitoring agents let teams support more systems, and the human capacity both release gets invested in building more things. Engineers spend more of their time “building factories” and “tending gardens” rather than staring at dashboards.

The Order to Roll This Out

If you intend to try it, this order is suggested so you do not hand over write permission at step one.

  1. Pick one class of high-frequency change point (release or configuration push) as the single entry point, and first connect the three data streams: change events, service code, telemetry and logs;
  2. Have the model do only two things: generate a monitoring spec for each change, and explain the phenomenon after something fires – read-only throughout, with all conclusions subject to human review and escalation;
  3. Hand spec execution to a deterministic program so the model is not in the always-on loop, and record the tokens and duration of every call along the way;
  4. Accumulate a labelled evaluation set: label confirmed real regressions, false positives and problem-free changes separately, and use it to test every change to the noise-reduction heuristics and prompts;
  5. Once evaluation is stable, delegate tier by tier, starting with deterministic rollback actions, with audit logs and one-click reclamation at every tier;
  6. Review with two metrics, “time to detection” and “false positive rate”, rather than process quantities such as “how many logs were investigated”.

Two premises need to be stated up front: that an agent really can do this better than a person, and that it must be constrained by scaffolding. Skip the first and you build an expensive chat window; skip the second and you get a loop that keeps burning money while the signal gets dirtier and dirtier.

Before the industry turns this into standard practice, teams that start accumulating evaluation sets and noise-reduction experience now will gain a real head start. On current trends, it will not be long before “no agent is watching your system” is itself seen as an operations risk.

Built with Hugo
Theme Stack designed by Jimmy