SEO Tool Cost

From The SEO Playbook for 2027

Is Cloudflare blocking Googlebot?

By SEO Tool Cost · Published · Facts as of the book's research cut-off, 8 October 2026; the date beside each source is the source's own. Methodology

Yes, if your site uses Cloudflare's "Block" option for AI crawlers. Since 15 September 2026 that setting also blocks crawlers that serve both search and AI. In Cloudflare's words, it will stop "Applebot, Bingbot, and Googlebot from reaching your site — search included". The option that keeps search crawling while refusing AI training is the new "Disallow AI Training" (Cloudflare blog, 2026-09-15) [Official].

This is the newest way for a site to disappear from search, and nobody has to touch your code for it to happen. Below: why the change reaches further than Google, the other Cloudflare settings that catch sites out, how to tell from Search Console whether it has happened to you, and what to check.

Why one switch reaches every answer engine

The three crawlers the setting now stops feed most of search, AI answers included:

So a site that switched on "Block" to keep its content out of AI training can also have taken itself out of Google, Bing, Copilot and Apple's search features at the same time.

It has already happened to a real site

In August 2026, Jonathan Bird and Brodie Clark described a site whose IT provider switched on Cloudflare's crawler controls to stop all bots. Organic traffic, Google Ads and Merchant Center listings broke for two weeks. In Bird's words, "This completely blocked Google from crawling the site." (Search Engine Roundtable, 2026-08-13) [Practitioner]. That case came a month before the September change made "Block" reach search crawlers too.

Three more Cloudflare settings that surprise people

  • Managed robots.txt can add Cloudflare's own lines to your robots.txt, so the file crawlers read isn't quite the one you wrote (Cloudflare docs, updated 2026-08-03). Cloudflare says it is being replaced by a newer preference-sync feature (2026-09-15).
  • Bot Fight Mode cannot be bypassed with custom WAF rules (Cloudflare docs, updated 2026-08-03). An allow rule for Googlebot won't help while it is on.
  • Challenge pages. Google's John Mueller described bot protection that serves an "Are you a bot?" page with a 200 status. Google may index that page in place of your content, or fold your pages into other sites' identical challenge pages as duplicates (Search Off the Record, 2026-07-16) [Googler].

Other CDNs and security plugins have their own versions of these controls, so the checks below apply to them too.

How to tell whether it's happening to you

A block at the CDN happens before a crawler ever reads your robots.txt, so the file can look perfect while Googlebot is turned away. Signs to look for:

  • A URL Inspection live test in Search Console fails, or shows a challenge page instead of your content.
  • The Crawl Stats report shows a jump in errors after a CDN, firewall or security-plugin change. Server errors and timeouts that last more than about a day are when Google's systems start reacting (Mueller and Splitt, 2026-07-16).
  • Your robots.txt, fetched from outside your network, contains lines nobody on your team wrote.

Once the cause is fixed, marking the issue as fixed in the Page indexing report makes Google test a sample of the affected URLs and, if they are fine, recrawl the rest faster (same episode, 2026-07-16).

What Google says

The disagreement: should AI crawlers get a different version of your site? Mike King of iPullRank has proposed hiding proprietary content from AI crawlers behind scripts and serving them Markdown versions through an edge worker, while explicitly not doing this to Googlebot (iPullRank, 2026-05-14) [Practitioner]. Most practitioners point the other way: it is complex to maintain and easy to get wrong.

Google's position is one version for everyone. Danny Sullivan said sites should not produce two versions of their content, one for language models and one for people. And if the aim is to keep content out of Gemini's training, Google's own switch is the Google-Extended token in robots.txt, which Google says does not affect inclusion in Google Search or AI Overviews. Blocking Googlebot is not needed for that.

Sources: Danny Sullivan on Search Off the Record, 2026-01-08; Google's common crawlers, updated 2026-07-14.

Do this

  1. Open Cloudflare's AI crawler setting and read what it says. If it is "Block" and what you wanted was only to refuse AI training, "Disallow AI Training" is the option that keeps search crawling.Done when: the setting is the one you chose on purpose, and you know which crawlers it stops.
  2. Fetch your robots.txt from outside your network and look for a stray Disallow: / or lines your CDN added. Then paste it into our robots.txt tester for search and AI crawlers to see which crawlers it lets in.Done when: every line in the live file is one you meant.
  3. Run a URL Inspection live test on a key page in Search Console, and the equivalent in Bing Webmaster Tools.Done when: the live test succeeds in both.
  4. Check Bot Fight Mode, the firewall and any security plugin for challenge pages served to verified search crawlers.Done when: no challenge page reaches Googlebot or Bingbot.
  5. Write the settings down with the date you checked them, and repeat the live test after every CDN, firewall or plugin change.Done when: the settings and the date are in your change log.

Where the book goes further

This is the first of the five quick checks the book asks you to run in your first week. Chapter 8 also has a quickest-first list for overnight traffic drops (robots.txt, CDN, noindex, status codes, canonicals) and Chapter 9 a worked robots.txt that separates search crawlers from training crawlers.

The SEO Playbook for 2027 covers this in Chapter 8 (technical essentials) and Chapter 2 (crawling and indexing), with the evidence label, date and link behind every claim and numbered steps that each end in a check. It costs $39.

See what's in the book · Read Chapter 9 free: how to optimize your website for AI search