Indexing.

Indexing is the process by which a search engine stores and organises web page content in its database (the index) after discovering and crawling that content. Only indexed pages can appear in search results. The index is effectively a massive structured database mapping queries to relevant pages based on their content, authority, and quality signals.

Crawling vs. Indexing

Crawling and indexing are distinct steps. Crawling: Googlebot visits a URL and downloads its content. Indexing: Google processes the downloaded content and decides whether to add it to the index based on quality signals, canonicalisation, robots directives, and resource constraints. A page can be crawled without being indexed ('Crawled - currently not indexed' in Search Console), and a URL can appear in the index without being recently crawled (cached from a previous visit).

Why Pages Fail to Index

Common reasons for non-indexation: noindex meta tag or X-Robots-Tag header, blocked by robots.txt (preventing crawl, and therefore reading of any index directives), thin or duplicate content (Google chooses not to index low-value pages), crawl budget exhaustion (large sites where Googlebot can't crawl every URL), soft 404 detection (pages that return 200 but appear empty or error-like), and canonicalisation (pages consolidated under another canonical URL).

Requesting Indexation

New or recently updated pages can be submitted to Google's indexing queue via: Google Search Console's URL Inspection tool (request indexing for individual URLs), XML sitemap submission (flagging new URLs for Googlebot discovery), Indexing API (for eligible content types including job postings and live stream pages; Google has limited expansion of Indexing API eligibility). Indexing typically occurs within days for high-priority pages on authoritative domains; weeks or months for low-authority or rarely-crawled sites.