W2 Not in Search: Five Checks

A site running on W2 loses indexed pages suddenly one week. Nothing changed on the site, but the WAF settings were changed around then as part of a security upgrade. A company runs three brands on W2, each on its own domain, and none of them does well in search, so nobody knows whether they should have been combined. Questions from shops running their own infrastructure cluster around these two situations.

What they share is that the site changes in the eyes of a search engine without anyone touching the HTML. A WAF settings change is a site change, and a domain is what a search engine treats as the unit of a company. This article gathers two separate articles into five checks, in order. Numbers and steps come from those articles unchanged; the detailed work lives in the links at the end.

1. If indexed pages drop suddenly, write down the date the WAF changed and compare it with Crawl Stats in Search Console

The first suspect is the WAF or bot protection. It can classify search engine and AI crawlers as hostile bots, and once classified they are simply blocked. Crawlers send many requests in a short time, which makes them easy targets for rate limits and behavior detection. When blocked, Googlebot receives errors (403, 429, 5xx), indexed pages fall, and rankings drop. Because the HTML hasn't changed, the cause stays hard to see.

In Search Console, open Crawl Stats from Settings, check whether "Host status" reports a problem, find when 403, 429, and 5xx grew in the "By response" breakdown, and run a live test on a main page to see whether it can be fetched. If the live test fails, Googlebot is blocked at that moment. A block of a few days recovers once lifted; one that lasts weeks drops pages from the index and takes time to recover. CDNs have bot protection too and sometimes block on top of the WAF, so check CDN logs as well.

2. Search server logs for 403 and 429 responses to legitimate crawlers, verify them by IP, and allowlist them

Filter the WAF or web server access log for User-Agents containing Googlebot, bingbot, GPTBot, ClaudeBot, PerplexityBot, and similar, with response codes 403, 429, or 5xx. If there are hits, find which rule blocked them in the WAF log. User-Agents can be spoofed, so combine the allowlist with IP verification.

Published IPs change, so review the allowlist regularly. Set rate-limit thresholds no lower than Googlebot's normal crawl rate, which Crawl Stats shows. Search Console used to offer a setting to reduce Googlebot's crawl rate; now there is only automatic adjustment, which lowers the rate on its own when a site is slow or returns many errors. Stabilizing the site's responses is more reliable than rejecting crawlers with a rate limit.

3. Decide AI crawler access as a policy, and refuse with robots.txt rather than the WAF

Whether to allow or refuse AI crawlers is a policy question, not a technical one. Pick one of three before touching settings.

If you refuse, do it in robots.txt, not the WAF. A WAF block is recorded by the crawler as an error and can affect how the site's health is judged; a robots.txt refusal is treated as intentional. When changing settings, record the date and content of every WAF, bot-protection, and rate-limit change, watch response codes in Crawl Stats for a week afterward, and fetch each vendor's latest IP ranges before updating the allowlist. The usual situation is that security changes the WAF and SEO doesn't hear about it. One clean check doesn't prevent the next change from causing it again, so set a rule first: whoever changes the WAF tells SEO.

4. Decide multi-brand domain structure as a trade-off between brand independence and consolidated search performance

Separate domains make each brand independent but split search performance three ways. Subdirectories under one domain consolidate it but make the brands look like one company's product lines. The right answer depends on whether the brands need to be known as independent entities. To a search engine a domain is the unit of a company: separate means separate companies, combined means one. Don't let the domain contradict what brand strategy decided.

Five questions decide it: are brands searched by name, do customer bases differ between brands, does each brand have a dedicated operator, are many products shared, and do you want to raise recognition of the company name. More "yes" answers point to separating, more "no" answers to combining, and subdomains when undecided. Google has said in recent years that it doesn't strongly distinguish subdomains from subdirectories, but that is not a firm guarantee, so choose subdirectories if consolidation must be certain. In an Ahrefs study of 75,000 brands, the signal most strongly correlated with AI Overviews exposure was brand web mentions (0.664), not backlink count (0.218). Merging separate domains into subdirectories is possible, but it is a site move in scale: redirects from every old URL, a change of address in Search Console, and rebuilt structured data.

5. Design shared-product canonicals and Organization to match the structure

When several stores carry the same product, the same product page exists once per store. How duplicates are handled, and where Organization markup describing the brand-company relationship goes, both depend on the structure.

For duplicates, either pick a primary store and point other stores' shared-product pages to it with canonical, or write different product descriptions per store so they aren't duplicates (high operating load); if most products are shared, reconsider whether separate stores make sense at all. Check W2's multi-store specification for how canonical is output on shared products. Stating the brand-company relationship in structured data lets AI search understand that this brand belongs to this company, so each is more likely to be referenced in questions about the other. When adding a store, match the existing structure, add a Search Console property (unnecessary for subdomains and subdirectories under a Domain property), register a sitemap, check robots.txt (independent for separate domains and subdomains, shared for subdirectories), set shared-product canonicals and Organization and Brand structured data, and link the company top page and each brand to each other.

Search OS keeps watching what happens to crawlers as a result of the settings

The WAF will break things again at the next change, and new AI crawlers keep appearing, so the allowlist's assumptions keep shifting. Across several stores, a change in one store routinely diverges from the others, whether a template update, an added product, or a canonical setting, and the more stores there are, the more work it takes to confirm that every store follows the same policy. Search OS is a layer supporting self-operated infrastructure and multi-store setups such as W2. From bot logs it continuously monitors response codes for search engine and AI crawlers, and finds blocks as they occur, new crawlers as they appear, and gaps between the allowlist and reality. Across stores it checks that canonical tags, structured data, sitemaps, and metadata agree, and turns shared-product duplicates, Organization contradictions, and per-store differences in bot fetch status into a fix list. It doesn't configure the WAF and it doesn't decide the structure.

Questions to check first

• Were WAF, bot-protection, or rate-limit settings changed around the week indexed pages fell, and is the date recorded

• Does Crawl Stats in Search Console show a period when the share of 403, 429, and 5xx grew

• Does a URL Inspection live test on a main page fetch successfully

• Is the allowlist verified against each vendor's published IP ranges, not just User-Agent

• If AI crawlers are refused, is it written in robots.txt rather than the WAF

• Was the choice between separate domains, subdomains, and subdirectories made on branded search, customer base, and shared products

• Do shared-product canonicals point to the primary store, and is each brand attached to the company's top-page Organization

Conclusion

When a W2 shop is missing from search, first compare the date of WAF and bot-protection changes with Crawl Stats in Search Console, and search server logs for 403 and 429 responses to legitimate crawlers. Verify legitimate crawlers by IP and allowlist them, decide AI crawler access as a policy, and refuse in robots.txt. For multiple brands, choose the domain structure by whether brand independence or consolidated performance comes first, and design shared-product canonicals and Organization to match. Everything up to here is a settings question; what follows is keeping the same state through every WAF change and every added store.

Related reading

• How to check whether W2's WAF and bot protection are blocking crawlers

• How to split domains in a W2 multi-brand, multi-store setup: consolidating or separating search performance

References

• Google Search Central: Verifying Googlebot and other Google crawlers

• Google Search Central: Crawl Stats report

• Google Search Central: Overview of Google crawlers and fetchers

• Google Search Central: Consolidate duplicate URLs

• Google Search Central: Organization structured data

• Google Search Central: Site moves with URL changes