Robots.txt.

Robots.txt is a plain text file placed at the root of a domain (e.g., `yourdomain.com/robots.txt`) that specifies crawling instructions for search engine bots using the Robots Exclusion Standard. It can allow or disallow specific bots from crawling specific URLs or directories, and can reference the location of the XML sitemap.

What Robots.txt Can and Cannot Do

Robots.txt controls crawling; it does not control indexing. A page blocked in robots.txt cannot be crawled, but Google can still index it if it discovers the URL through links elsewhere. To prevent indexing, use a noindex tag on the page itself. Common robots.txt uses: blocking staging environments, preventing crawler access to admin directories, and managing crawl budget by excluding low-value pages like filtered e-commerce URLs.

Common Robots.txt Mistakes

Critical mistakes include: accidentally blocking CSS and JavaScript files that Googlebot needs to render pages; blocking the entire site with `Disallow: /`; using robots.txt to try to hide pages (it's publicly readable: blocking sensitive URLs in robots.txt actually advertises their existence); and forgetting to update robots.txt when site structure changes, leaving outdated rules that block or expose unintended pages.