CDN and SEO: configure caching without blocking Googlebot
Connecting a CDN requires checking what kind of responses visitors and search robots receive. The home page may open faster while some requests continue to load the origin server and security rules prevent other URLs from being crawled.
On November 11, 2026, Google published recommendations for switching to CDN. The company explained cold caches and the risk of blocking Googlebot. This is operational guidance; the post does not announce a new ranking factor.
This guide is for website owners, developers and SEO specialists before connecting a CDN or changing web application firewall (WAF) rules. The proposed acceptance process checks page delivery, load and crawler access. It does not report tests of a particular project or promise better rankings.
Why a CDN may initially increase server load
A CDN stores copies of matching resources on distributed nodes. If there is no copy of the requested URL yet, the node contacts the origin server. At startup, the cache may be cold and the first request to the resource still requires origin to work.
Google explains that fast, stable responses can allow its infrastructure to gradually increase the crawl rate when there is crawl demand. Crawlers explore a broad set of URLs, including rarely visited pages. A warmed-up homepage therefore does not prove that other requests are served from cache. Load may increase, but this is not inevitable for every website.
For acceptance, this leads to a practical question: which requests actually stopped reaching the application? Measure CDN node responses and origin requests separately. A single browser speed metric does not show this difference.
Who needs a detailed launch plan?
Pay particular attention to large catalogs, expensive page generation and frequently updated data. A small website may need a narrower review of its main templates, forms, supporting files and security rules.
B Google documentation about crawl budget distinguishes crawl capacity from crawl demand. Most small websites do not need complex crawl-budget management. A fast response and a successful crawl do not guarantee that a page will be indexed.
Before configuring a CDN, define the goal: faster static-resource delivery, reduced public-page generation load or additional protection. Give each goal its own acceptance criterion. If the CMS, domain or page URLs are changing too, use a site relocation plan.
Define what can be cached first
Having a CDN doesn't mean it caches the entire site. For example, Cloudflare by default does not cache HTML and JSON; behavior depends on file extensions, headers, and specified rules. This is a supplier specific description and does not automatically transfer to any delivery network.
List the response types. A public article, a signed-in account and a form-submission result have different requirements. Exclude personal data from shared caching. Extend HTML caching only after checking the response content and existing rules.
For each type, record whether a shared copy is allowed, what distinguishes response variants, when the data becomes stale and who purges the cache. Language versions and catalog query parameters may require distinct cache keys. If the configuration fails to distinguish different variants, visitors may receive the wrong response.
Test content updates separately. Modify an approved test page and verify that purging or expiration produces the expected result. Set the acceptable delay from project requirements rather than choosing it solely to maximize the cache-hit rate.
How to check Googlebot and maintain protection
The User-Agent value can be spoofed. The “pass anyone who says they're Googlebot” rule doesn't confirm that the request belongs to Google.
Official verification of Google requests matches an IP against published ranges or uses a reverse DNS lookup followed by a forward lookup. For the latter, the hostname must match the relevant crawler group, and the forward lookup must return the original IP. Common crawlers and other Google fetchers belong to different groups.
If you're using an out-of-the-box verified bot engine from your CDN provider, check with your CDN provider to see what categories it recognizes and what rules it applies to them. The exception for a verified crawler should be for a desired restriction, such as mandatory browser verification. It does not require opening administrative pages or disabling security for the entire site.
Check more than just the start page. You need services, articles, catalogue, images and service files available for indexing. A request with a self-supplied User-Agent is useful for diagnosing individual conditions, but does not prove access from Googlebot's real IP. Compare the result with actual crawl logs and available search engine tools.
The general architecture for protecting forms and processing suspicious requests is discussed in material about anti-bot protection. Search-crawler rules must be consistent with that setup.
CDN acceptance checklist
| What we check | Sign of a problem | What to keep as proof |
|---|---|---|
| Public page | Placeholder page, browser challenge or incorrect content | URL, time, response code and HTML snippet |
| Cold and repeat request | Origin is contacted again where a cache hit was expected | Cache status and correlated logs |
| Signed-in area | A personalized response is stored in shared cache | Checking isolated test sessions |
| Page update | An old version remains after the agreed delay | Time of the change and delivery of the updated version |
| robots.txt and sitemap | Error, redirect to a browser challenge or stale content | Response and contents of each file |
| Non-existent URL | Error page with success code | Response code and page displayed |
| Application form | Submission result is lost or sent twice | The agreed test record at the receiving system |
| Verified Googlebot | Blocked by a WAF rule or rate limit | IP verification and the triggered rule in the logs |
This is a working checklist template. Add an owner, an expected outcome and a status for your project. A 200 response requires checking the content: according to Google documentation about HTTP, a successful status alone does not guarantee indexing, and an error inside such a response may be treated as a soft 404.
How to conduct a test run
- Preserve the current state. Record current settings, rollback methods and comparable baseline metrics: errors, response times, application load and crawler requests.
- Select a set of URLs. Include popular and rarely visited pages of different templates, static resources and addresses with expected errors. Do not use personal data to warm up the shared cache.
- Agree on stopping criteria. Agree on acceptable errors, response times and load, and name the person who decides when to roll back. Generic thresholds without project data are not useful here.
- Connect in a controlled manner. If the architecture allows it, start with a limited set of resources or rules. Warm the cache at an agreed rate: requesting the entire catalog at once can itself create excessive load.
- Repeat acceptance checks. Compare visitor behavior, CDN responses, and origin state. Check cache clearing, forms, and triggered protection rules.
- Continue observing. Check different load periods and pending updates. The absence of an error in the first minutes does not confirm the correctness of all routes.
To check application limits, use a separate, agreed load testing plan. Do not run these tests unannounced on the production website during the CDN switch.
What to do if origin is overloaded or the robot is blocked
Identify the cause in the logs first. More requests may result from a cold cache, unnecessary filter combinations, rule changes or unrelated traffic. Correlate the incident with a specific change before modifying the entire setup.
If a WAF rule blocks a verified robot from public pages, adjust the rule and repeat the check. If generation is overloaded, follow a prepared recovery or load limitation plan. Don't disguise an error with a 200 code page for the sake of appearance of accessibility.
B Google emergency instructions Temporary 500, 503 or 429 responses can reduce crawling for a few hours or 1–2 days. A 503/429 response may include Retry-After. This is a short-term emergency measure: reduced crawling affects the whole hostname, and prolonged errors can cause URLs to drop from the index. It is not a routine way to improve SEO.
Once the cause is resolved, restore the correct responses, check the changed URLs, and observe the actual crawl. The recovery time for intensity and positions cannot be guaranteed by CDN settings alone.
In a task for technical support of the site , record the acceptance outcome: public pages and supporting files are accessible, personalized responses stay out of shared cache, updates arrive within the agreed time and the origin handles the actual load. This gives the team a concrete checkpoint after launch.