A website with 10,000 pages is not automatically too large for Google. Many websites have far more pages and perform well in search.
The problem is usually found in the extra URLs.
A single page can create several versions of itself through filters, tracking codes, sorting choices, and old URL formats. A large online store may have 15,000 products but hundreds of thousands of URLs that Googlebot can request.
Most of those extra URLs do not help customers. They may show the same products, an empty result, or a slightly different version of an existing page. Google still has to request and assess them.
This is where the crawl budget becomes a practical concern.
Key takeaways
- Crawl budget is based on the URLs Google can and wants to crawl.
- The number of published pages does not show the full crawl situation.
- Filters, parameters, and duplicate paths can create unnecessary URLs.
- XML sitemaps should contain important canonical pages.
- Server problems can limit Googlebot activity.
- Log files are useful when Search Console does not explain the problem.
Highlights
- Audit the complete URL list before blocking anything.
- Find out which URLs Googlebot requests most often.
- Remove redirects and error pages from XML sitemaps.
- Link to important pages from relevant sections of the website.
- Review crawl budget after redesigns, migrations, and platform changes.
What crawl budget means
Crawl budget is the amount of time and resources Google uses to crawl a website. Google considers how much crawling a server can handle and how much crawling the site appears to need.
A site with frequent content changes may need more regular crawling than a site that rarely changes. Google also considers the size, quality, and usefulness of a website’s pages.
Crawling and indexing are not the same thing. Google can crawl a page and decide not to index it. This is an important point when reviewing crawl budget. A high crawl count does not mean that the website has good index coverage.
Google’s own guidance focuses mainly on very large or frequently updated sites. A 10,000-page website may still benefit from a crawl budget review when it has many discovered but unindexed URLs, slow updates, or a large number of low-value pages.
Count all crawlable URLs
Begin with an inventory of every URL Google can reach.
Use data from:
- Your CMS.
- XML sitemap files.
- Google Search Console.
- Server logs.
- A technical SEO crawler.
- Internal search and filtering tools.
Put the URLs into groups. For example, keep product pages, category pages, service pages, blog posts, location pages, redirects, errors, and parameter URLs separate.
Then ask a simple question about each group: should this URL be crawled and indexed?
Some pages are useful for visitors but not useful in search. Account pages and cart pages are common examples. Some pages should be removed because they are empty or outdated. Others need to stay online but should point to a stronger canonical URL.
Without this step, it is easy to block the wrong URL pattern and create a new technical problem.
Watch out for filter URLs
Filter URLs cause trouble on many large sites.
A visitor may filter a clothing store by size, brand, color, price, and material. The website may create a new URL every time someone selects an option. Most combinations have little original content and no reason to appear in search.
Keep a filter page when it targets a genuine search topic and provides useful information. Do not keep every possible combination open to crawling.
Check for URLs that contain:
- Price ranges.
- Sort orders.
- Tracking codes.
- Session numbers.
- Empty search terms.
- Temporary campaign values.
- Multiple filter combinations.
A canonical tag can tell Google which page is preferred. It does not stop Google from visiting other versions. If you need to control crawling, assess whether a robots.txt rule is suitable. Test the rule before applying it across a large group of URLs.
Clean the XML sitemap
An XML sitemap should be a short list of pages you want Google to find and process.
It should not be a dump of every URL in the database.
Remove URLs that:
- Redirect to another page.
- Return a 404 or 410 response.
- Have a noindex instruction.
- Are duplicate versions.
- Are blocked from crawling.
- Contain little useful content.
- Are not intended for organic search.
Check the sitemap after every major website change. A migration can leave old URLs in the file. A catalog import can add pages that are not ready for publication. A plugin can create URL versions that were not part of the original plan.
Google supports sitemap indexes for large sites. You can separate pages by type, such as products, services, locations, and articles. Follow Google’s sitemap limits when creating the files.
Use server logs for real evidence
Search Console shows reports about Google’s activity. Server logs show the individual requests received by your website.
This difference matters.
A server log review can show that Googlebot keeps requesting an old filter URL, a redirect, or a page that returns an error. It can also show whether Googlebot reaches the pages your business depends on.
Look for:
- Repeated requests for the same parameter URLs.
- Large numbers of 3xx responses.
- 404 and 410 pages.
- 5xx server errors.
- 429 rate-limit responses.
- Long response times.
- Important pages that receive no crawl requests.
Do not remove a page only because Google visits it often. First confirm what the page does, whether it receives traffic, and whether another page depends on it.
Fix server problems
Crawl budget is affected by the way a server responds.
If a website takes too long to respond, times out, or returns repeated server errors, Google may reduce crawling to avoid putting more pressure on the site.
Ask your developer or host to review:
- Database queries.
- Hosting resources.
- Cache settings.
- CDN performance.
- Large files.
- Bot rate limits.
- Server error reports.
Google recommends monitoring server availability and improving response handling for large websites. Its documentation also discusses caching and 304 responses for content that has not changed.
This work should be tested on a staging environment first. A change that improves speed but serves outdated information can create problems for visitors and search engines.
Reduce redirect and soft 404 waste
A redirect is helpful when a page has moved to a relevant replacement. It is not helpful when several redirects are chained together.
For example, an old page may redirect to a category, which redirects to a new category, which redirects to the final product. Update internal links so they go directly to the final URL.
Review soft 404 pages too. These pages return a normal response but do not provide useful content. Expired job listings, empty search results, and discontinued products often fall into this group.
Improve the page if it still has a purpose. Redirect it when a close replacement exists. Return a proper 404 or 410 response when the page is permanently gone.
Help Google find priority pages
Internal links help search engines understand which pages matter.
A service page should be linked from the relevant service section. A location page should be connected to the services offered in that area. A product page should be reachable from its category and related products.
Check your website for:
- Orphan pages.
- Broken internal links.
- Links that pass through redirects.
- Important pages that are difficult to reach.
- Navigation pointing to outdated URLs.
Use normal, descriptive anchor text. Do not add a large block of links just to influence crawling. The link should make sense to a person reading the page.
Crawl budget and AI search
Crawl budget does not guarantee a place in AI Overviews. It helps with a more basic requirement: Google must be able to access and process the page.
Google says AI Overviews and AI Mode use existing Search systems. There is no separate AI markup that guarantees inclusion. Pages still need to meet Google’s technical requirements and be eligible to appear in Search.
That makes technical accuracy important. Keep useful content accessible. Link to important pages. Avoid duplicate URL paths. Keep server errors under control. Use structured data only when it accurately describes the page.
A practical review schedule
Check crawl budget data after a migration, redesign, or major catalog change. For a large site, a monthly review is reasonable.
Look at:
- Crawl Stats.
- Page Indexing.
- XML sitemap coverage.
- Server log requests.
- Redirects.
- Response codes.
- New parameter URLs.
- Traffic to important pages.
The goal is not to make Google crawl more URLs. The goal is to make the available URLs more useful.
For help with technical SEO, sitemap cleanup, internal linking, and crawl budget analysis, visit Adams Internet Marketing’s SEO services. A useful audit should be based on the website’s actual URL data, server behavior, and business priorities.