Robots Meta Tag.
Learn what Robots Meta Tag means in modern search and SEO.
An HTML meta tag placed in a page's head section to control search engine crawling and indexing behaviour for that specific page.
The robots meta tag is an HTML `<meta>` element placed in a page's `<head>` section that communicates indexing directives to search engine crawlers on a page-by-page basis. Unlike robots.txt (which operates at the server level, controlling which URLs can be crawled), the robots meta tag is read by the crawler after the page is accessed and controls how the page's content is indexed and followed.
Common Robots Meta Directives
`<meta name='robots' content='index, follow'>` — default behaviour; index the page and follow its links. `<meta name='robots' content='noindex'>` — do not add this page to the search index. `<meta name='robots' content='nofollow'>` — do not follow or pass PageRank through any links on this page. `<meta name='robots' content='noindex, nofollow'>` — neither index nor follow. `<meta name='robots' content='nosnippet'>` — do not show a text snippet in search results. `<meta name='robots' content='max-image-preview:large'>` — allow full-size image preview in results (important for Google Discover).
Robots Meta Tag vs. X-Robots-Tag
The robots meta tag works for HTML pages — it is embedded in the page HTML. The `X-Robots-Tag` HTTP header performs the same function for non-HTML content (PDF files, images, XML files) that don't have an HTML head section. Both are page-level directives, unlike robots.txt which is domain-level.
Common Implementation Errors
Blocking robots.txt while also using noindex: if robots.txt disallows crawling, Googlebot cannot read the noindex meta tag — the page may remain in the index without the noindex being respected. Applying noindex to pages that should be indexed (common CMS configuration errors). Using user-agent-specific robots tags incorrectly (a `name='Googlebot'` robots tag applies only to Googlebot; `name='robots'` applies to all well-behaved crawlers).
Related Terms
Ready to close the loop?
See every term in action
Aergos tracks your AI and organic visibility across every channel, in one platform.
Not ready to talk? Audit your site free →