Crawl Budget and Index Gaps at Tens of Thousands of Products

On a marketplace with tens of thousands of products, new products are slow to be indexed. Search Console shows "Discovered - currently not indexed" in the tens of thousands. Shops with catalogs this size ask about managing crawl budget.

A crawler has a limit on how many URLs it can process in a given period. At this scale, that limit becomes the direct cause of index gaps. Factors you could ignore up to a few thousand products start to decide the outcome past tens of thousands.

The answer is to run four tasks as a set. A sitemap is the list of page URLs handed to search engines. Sitemap splitting and lastmod accuracy, blocking low-value URLs, conveying priority through internal links, and observation in bot logs. Any one of them alone doesn't work. This article explains how to run the cycle.

Conclusion: Run the four as a set

Key idea: Crawl budget is the time and number of requests a crawler spends on a site. It's determined by the site's size and response speed, and by how many URLs the crawler finds. The shortest route to fewer index gaps is to reduce the URLs that don't need to be seen and make clear the URLs that should be. Start by counting how many URLs don't need to be seen.

The path to an index gap

Here is the typical path to index gaps on a marketplace with tens of thousands of products, in order.

1. Filters, sorting, and pagination generate URLs numbering dozens of times the product count.

2. The crawler discovers and fetches them (canonical and noindex are read only after fetching).

3. Most of the budget goes to fetching low-value URLs.

4. Discovery of new and updated products lags, and "Discovered - currently not indexed" grows.

5. URLs of discontinued products remain as-is and consume still more budget as errors or thin pages.

Look at the "Crawled - currently not indexed" URLs in Search Console. Check the "Discovered - currently not indexed" URLs by type too. That shows where on this path things are stuck. If they are mostly filter URLs, blocking comes first.

Sitemap operations: splitting and lastmod

lastmod accuracy matters most. If lastmod updates every time a product's stock count changes, the crawler reads it as always updated and stops using it as an update signal. Design it to update only when price, description, or specifications change. Check what triggers your lastmod updates.

Blocking low-value URLs: handling by type

robots.txt is the file that tells crawlers where they may go. URLs disallowed there never have their canonical or noindex read. If you want to remove URLs that are already indexed, order matters. First remove them from the index with noindex, then block them in robots.txt once they're gone. Check first whether the URLs you want to block are already indexed.

Convey priority with internal links

Crawlers favor pages that gather many links and pages close to the top page. At tens of thousands of products, check these three points.

• See whether main products and main categories are within 3 clicks of the top page.

• See whether category pagination has grown too deep (if one category runs to hundreds of pages, split it into subcategories).

• Check that links from product pages to related products and the parent category are output consistently by the template.

Verify the effect in bot logs

The effect of the three tasks above can only be verified in bot logs. At minimum, look at the following weekly.

The last item bears directly on product recommendations in AI search. It's not unusual for AI-side bots to use up their budget on low-value URLs and never reach product pages. Check in the logs whether GPTBot is reaching product pages.

At tens of thousands of products, manual operation doesn't last

The work above isn't done once and finished. Products are added and discontinued every day. Filter conditions multiply and templates change. Sitemap accuracy, the validity of blocking rules, the state of internal links, and bot fetch patterns all need continual review.

Search OS takes on this continual part as an operation. It doesn't change the existing marketplace's structure. It provides sitemaps, canonical, metadata, and structured data for search engines and AI crawlers as a separate layer. Which URLs Googlebot, GPTBot, and PerplexityBot fetch, and how much: it observes this continuously in bot logs. It compiles the URL groups causing index gaps and the places to fix first. The goal is to keep crawlers reaching the pages that should be seen, even at millions of pages. Whether you can keep reviewing without relying on people is what separates the outcomes.

Frequently asked questions

Can I apply to have crawl budget increased?

No. You respond by raising the site's response speed and reducing wasted URLs. The idea is to increase the number of important URLs processed within the same budget.

Will putting noindex on every filter URL solve it?

noindex removes them from the index, but fetching continues. Reducing budget use requires blocking in robots.txt. Combinations with search demand, though, should be promoted to proper pages.

Are AI search bots affected by crawl budget too?

Yes. GPTBot and others decide how much to fetch based on the site's structure and response speed. On a site with many low-value URLs, they may stop fetching before reaching product pages.

Related reading

• Sitemap Operations for Large Sites: lastmod, Splitting, and Submission

• Do Duplicate URLs from Shopify Collections and Tags Affect Crawling

• SEO and GEO Explained: AI Search Optimization for Large Platforms

References

• Google Search Central: Managing crawl budget for large sites

• Google Search Central: Managing large sitemaps

• Google Search Central: Faceted navigation best practices

• Google Search Central: Page Indexing report