Featured image of post The 11-18 Internet Sabbath - A Cloudflare Outage

The 11-18 Internet Sabbath - A Cloudflare Outage

On November 18, the global network infrastructure provider `Cloudflare` -- jokingly called the cyber Buddha -- suffered its worst outage since 2019, with an extremely wide blast radius.

The 11-18 Internet Sabbath - A Look Back at the Cloudflare Outage

Impact

On November 18, the global network infrastructure provider Cloudflare – jokingly called the cyber Buddha – suffered its worst outage since 2019, with an extremely wide blast radius.

Internet sites relying on Cloudflare services, including famous names such as X, ChatGPT and Claude, plus almost every site of the sort we all know about, all became unreachable.

The outage lasted more than six hours. Rumor has it that Cloudflare engineers could not reach the ChatGPT instance that the outage had taken down, which hurt the speed of the fix. πŸ˜‚πŸ˜‚πŸ˜‚ That, from another angle, shows just how large the impact was!

How It Unfolded

Back to the starting point. Before 11:05 UTC on the 18th, the sites served by Cloudflare were still enjoying peace and quiet. But from around 11:20, large numbers of sites started throwing “5XX - internal server error” en masse. You could call it a cliff-edge outage.

Even the core CDN, identity verification and the internal monitoring dashboard were unusable.

This bizarre failure pattern made Cloudflare engineers initially assume a large-scale cyber attack, a formidable adversary, perhaps even an attack from the inside. In their internal chat room, people suspected another Aisuru DDOS attack.

So in the early phase a great deal of effort went into hunting this “ghost attack”.

But as the investigation went on, the real cause turned out to be jaw-dropping.

Bot Management

In an era where AI technology is applied at scale, Cloudflare is no exception. It runs a Bot Management module that, using AI machine-learning models, scores security and configuration policies and in turn controls network access.

That model takes a feature file as one of its inputs, and it was precisely this file, used as AI input, that caused this epic bug.

  1. First came routine maintenance work. To improve security, database permissions were adjusted, with the hope of replacing a shared “system account” with “personal accounts” that carry clearer ownership.
  2. That change triggered a latent problem – specifically in the code that generates the “feature file”. That code used to look up the feature list only from the default database, but after the permission change it could suddenly reach another standby database too. And the code never constrained the database source to read from, so every feature got read twice, and the resulting feature file ballooned.
  3. To keep processing efficient, Cloudflare’s bot management module also limits the feature file: the list must not exceed 200 entries.
  4. The bloated feature file blew past that limit and therefore triggered the out-of-memory protection mechanism.
  5. The program crashed, and for safety all service connections were automatically cut.

The timeline of the whole incident is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
timeline
    title Cloudflare Outage Timeline (UTC)
    section Deployment and Impact
    11:05 : Database access control change
    11:28 : Impact appears, mass customer HTTP errors
    section Investigation and Mitigation
    11:32-13:05 : Workers KV traffic surge, attempt traffic control / account limits
    13:05 : Mitigation applied, impact reduced
    13:37 : Cause confirmed, rollback begins
    section Fix and Recovery
    14:24 : Stop creating / propagating bad configuration files
    14:24 : Old configuration file tests pass, global fix
    14:30 : Main issue resolved, correct configuration file deployed globally
    17:06 : All services restored, impact ends

In short, the root cause was a tiny database permission change!

Lessons

After resolving the issue, the official side immediately reflected and summarized four measures:

  • Harden the ingestion of configuration files generated by Cloudflare so they are as safe as user-generated input files

  • Enable more global kill switches for individual features

  • Prevent core dumps and other error reports from consuming excessive system resources

  • Audit failure modes across all core proxy modules to find the causes of errors

And from a quality perspective, we might ask ourselves:

  1. Why was this kind of permission change applied straight to production? Was it tested at all?
  2. Why did a single point of failure exist, and why did it spread globally?
  3. Why did the exception protection itself crash the program?

πŸ₯² The Suspected Culprit

The guy below posted on the 18th that he was joining Cloudflare

Update on the 19th: he has started looking for a new job

What a magnificent amateur-hour outfit….. πŸ˜‚πŸ˜‚πŸ˜‚πŸ˜‚

Built with Hugo
Theme Stack designed by Jimmy