The 11-18 Internet Sabbath - A Look Back at the Cloudflare Outage

Impact
On November 18, the global network infrastructure provider Cloudflare – jokingly called the cyber Buddha – suffered its worst outage since 2019, with an extremely wide blast radius.
Internet sites relying on Cloudflare services, including famous names such as X, ChatGPT and Claude, plus almost every site of the sort we all know about, all became unreachable.

The outage lasted more than six hours. Rumor has it that Cloudflare engineers could not reach the ChatGPT instance that the outage had taken down, which hurt the speed of the fix. πππ
That, from another angle, shows just how large the impact was!

How It Unfolded
Back to the starting point. Before 11:05 UTC on the 18th, the sites served by Cloudflare were still enjoying peace and quiet. But from around 11:20, large numbers of sites started throwing “5XX - internal server error” en masse. You could call it a cliff-edge outage.

Even the core CDN, identity verification and the internal monitoring dashboard were unusable.
This bizarre failure pattern made Cloudflare engineers initially assume a large-scale cyber attack, a formidable adversary, perhaps even an attack from the inside. In their internal chat room, people suspected another Aisuru DDOS attack.

So in the early phase a great deal of effort went into hunting this “ghost attack”.
But as the investigation went on, the real cause turned out to be jaw-dropping.
Bot Management
In an era where AI technology is applied at scale, Cloudflare is no exception. It runs a Bot Management module that, using AI machine-learning models, scores security and configuration policies and in turn controls network access.
That model takes a feature file as one of its inputs, and it was precisely this file, used as AI input, that caused this epic bug.
- First came routine maintenance work. To improve security, database permissions were adjusted, with the hope of replacing a shared “system account” with “personal accounts” that carry clearer ownership.
- That change triggered a latent problem – specifically in the code that generates the “feature file”. That code used to look up the feature list only from the default database, but after the permission change it could suddenly reach another standby database too. And the code never constrained the database source to read from, so every feature got read twice, and the resulting feature file ballooned.
- To keep processing efficient, Cloudflare’s bot management module also limits the feature file: the list must not exceed 200 entries.
- The bloated feature file blew past that limit and therefore triggered the out-of-memory protection mechanism.
- The program crashed, and for safety all service connections were automatically cut.
The timeline of the whole incident is shown below:
| |
In short, the root cause was a tiny database permission change!
Lessons
After resolving the issue, the official side immediately reflected and summarized four measures:
Harden the ingestion of configuration files generated by Cloudflare so they are as safe as user-generated input files
Enable more global kill switches for individual features
Prevent core dumps and other error reports from consuming excessive system resources
Audit failure modes across all core proxy modules to find the causes of errors
And from a quality perspective, we might ask ourselves:
- Why was this kind of permission change applied straight to production? Was it tested at all?
- Why did a single point of failure exist, and why did it spread globally?
- Why did the exception protection itself crash the program?
π₯² The Suspected Culprit
The guy below posted on the 18th that he was joining Cloudflare

Update on the 19th: he has started looking for a new job

What a magnificent amateur-hour outfit….. ππππ
