Crawlability

Whether and how easily search engine bots can discover and access the pages on your website.

Crawlability refers to how accessible your web pages are to search engine crawlers (bots) like Googlebot. A page must be crawlable before it can be indexed or ranked. Common crawlability barriers include: robots.txt rules that block crawlers, noindex directives, pages that require login, JavaScript-heavy rendering that bots can't process, and server errors (5xx status codes).

Google allocates each site a 'crawl budget' — an estimated limit on how many pages it will crawl in a given timeframe. Large sites with thousands of pages need to ensure their crawl budget is not wasted on low-value URLs (thin pages, session ID URLs, parameter duplicates). Reducing crawl waste helps important pages get discovered and indexed faster.

Internal linking is one of the most effective ways to improve crawlability. Pages that are deeply buried in the site architecture — requiring many clicks to reach from the homepage — may be crawled less frequently or not at all. Shallow site architecture and strong internal linking help crawlers reach important pages.

You can monitor crawlability using Google Search Console's Coverage report, which shows which pages have been indexed and which have been excluded — and why.