Checking Whether W2's WAF Is Blocking Crawlers

On a site run on W2, the indexed count suddenly dropped one week. Nothing changed on the site side, but around that time the WAF settings were changed as part of a security hardening. Shops operating on their own platform raise this.

The first suspect is the WAF or bot protection. A crawler is a program that reads through pages. The WAF may be judging search engine and AI crawlers as hostile bots. A crawler judged that way is blocked outright. Crawlers send many requests in a short time, so they're prone to tripping rate limits and behavior detection. When blocked, Googlebot receives errors (403, 429, 5xx). The indexed count falls and rankings drop. The site's HTML hasn't changed at all, so the cause stays hard to notice.

This article lays out the steps for confirming a block and how to tell legitimate crawlers apart. It also covers the policy for allowing or refusing AI crawlers and the operating rules for changing settings.

Conclusion: Check Search Console crawl stats and server log response codes, and put legitimate crawlers on the allow list

Key idea: A WAF settings change is a site change. Even if you didn't touch the HTML, the site as search engines see it has changed. Record the date of each settings change so you can compare it against changes in indexing. Start by writing down the most recent date the WAF was changed.

Step 1: Check in Search Console

1. Open Settings → Crawl stats

2. Check whether "Host status" shows a problem

3. In the "By response" breakdown, identify when 403, 429, and 5xx rose

4. Run a live test on core pages in URL Inspection and confirm they can be fetched

If the live test can't fetch, Googlebot is blocked at that point. Run your core pages through the live test.

Step 2: Check server logs

Filter the WAF or web server access log on the following conditions.

• User-Agent contains Googlebot, bingbot, GPTBot, ClaudeBot, PerplexityBot, or similar

• Response code is 403, 429, or 5xx

If there are hits, use the WAF log to pin down which rule blocked them. First, check the response codes sent to Googlebot in the log.

Step 3: Tell legitimate crawlers apart

User-Agent can be spoofed. Combine the allow list with IP verification.

Each company's published IPs get updated. Review the allow list regularly too. Compare Googlebot's published IP ranges against your allow list.

Allow or refuse AI crawlers

This is a matter of policy, not technology.

If you refuse, the basic rule is to do it in robots.txt, not the WAF. Block with the WAF and the crawler records it as an error. That can affect how the site's health is assessed. A refusal in robots.txt is treated as intentional. Confirm that the bots you want to refuse are listed in robots.txt.

Operating rules for settings changes

• Record the date and content of every change to WAF, bot protection, and rate limiting settings

• For a week after a change, check response codes in Search Console crawl stats

• Before updating the allow list, fetch each company's latest IP ranges

• Set the rate limit threshold no lower than Googlebot's normal crawl frequency (check it in crawl stats)

Operation after the check

The security team changes WAF settings and the SEO owner doesn't know. It's a common situation. Even if one check finds no problem, the next settings change causes it again. New AI crawlers keep appearing too, so the assumptions behind the allow list keep shifting.

Search OS is a layer that supports self-operated platforms such as W2. From bot logs, it continuously monitors response codes for search engine and AI crawlers. It finds blocks as they occur, new crawlers as they appear, and gaps between the allow list and reality, and organizes them into items to fix. It isn't the layer that configures the WAF. It's a layer that keeps watching what happens to crawlers as a result of the configuration. Start by making it a rule that whoever changes the WAF tells the SEO owner.

Frequently asked questions

Does even a temporary block affect indexing?

A block of a few days recovers once lifted. If it lasts weeks, pages start dropping out of the index. Recovery then takes time.

Is there bot protection on the CDN side too?

CDNs have bot protection as well. Sometimes it's blocking in tandem with the WAF. Look at the CDN logs too.

Isn't tripping the rate limit a sign that Googlebot crawls too much?

Search Console used to have a setting to lower Googlebot's crawl rate. Now there's only automatic adjustment. If the site is slow or returns many errors, crawling decreases on its own. Stabilizing the site's responses is more reliable than refusing through rate limits.

Related reading

• How to split domains in a W2 multi-brand, multi-store setup

• SEO and AI search on an SI-built site: how to proceed without a development cycle

• Crawl budget and index coverage for large catalogs

• W2 Not in Search: Five Checks

References

• Google Search Central: Verifying Googlebot and other Google crawlers

• Google Search Central: Crawl Stats report

• Google Search Central: Overview of Google crawlers and fetchers

• OpenAI: Overview of OpenAI Crawlers