Robots.txt: What It Is and How to Configure It Correctly
Robots.txt is a text file in a site's root directory that contains directives telling search bots which sections of the site they can crawl and which they can't. The file lives at domain.com/robots.txt and controls crawler access to crawling specifically — not removal of pages from a search engine's index.
How does it work?
Robots.txt is advisory, not enforced: well-behaved bots (Googlebot, Bingbot) read the file before crawling a site and follow its rules, but malicious crawlers can ignore it entirely.
Core directives
The file consists of directive blocks for specific user agents:
- User-agent — specifies which bot the rule block applies to (
*means all bots) - Disallow — blocks crawling of the given path
- Allow — permits crawling of a specific path inside an otherwise disallowed directory
- Sitemap — points to the full URL of the XML sitemap
Rule priority
Rules are read top to bottom, and when Disallow and Allow conflict, search engines apply whichever path is more specific (longer).
Why do you need it?
Robots.txt saves crawl budget by directing bots toward priority content instead of utility sections (admin panels, carts, parameter-based filters, pagination duplicates).
It also prevents servers from being overloaded with unnecessary bot requests and lets you point search engines straight to your Sitemap, speeding up discovery of new URLs.
Example
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?filter=
Allow: /wp-admin/admin-ajax.php
User-agent: Googlebot-Image
Disallow: /private-images/
Sitemap: https://example.com/sitemap.xmlCommon mistakes
Disallow: / in the root block
Accidentally blocks crawling of the entire site — the most critical mistake to check for after every release.
Confusing it with meta robots noindex
Robots.txt blocks crawling but doesn't guarantee a page is removed from the index: if external links point to it, it can stay indexed without a description.
Blocking CSS/JS files
Prevents Googlebot from rendering the page correctly and evaluating its mobile usability.
Using Crawl-delay for Google
Googlebot doesn't support this directive — crawl rate for Google is controlled through Search Console settings instead.
Incorrect Sitemap address
A relative path instead of a full URL, or a protocol mismatch (http instead of https).
How do you check it?
Google Search Console has a robots.txt report under Settings that shows the last version Google read and any syntax errors. You can also open the file directly at domain.com/robots.txt and check a specific URL with the URL Inspection tool in the same Search Console.
FAQ
Is robots.txt mandatory for every site?
No, the file isn't required. Without it, search bots crawl the site without restrictions, relying on other signals (canonical, noindex, link structure).
Can you list multiple Sitemap entries in one robots.txt?
Yes, the Sitemap directive can be repeated multiple times in the file — this is the standard way to point to several sitemaps or a sitemap index file.
Where exactly does robots.txt need to live?
Only in the domain's root directory (example.com/robots.txt). A file in a subfolder (example.com/blog/robots.txt) is ignored by search engines.
Can robots.txt remove an already-indexed page?
Blocking crawling doesn't automatically remove a page from the index — guaranteed removal requires a noindex tag or deleting the page and returning a 404/410 status.
Related terms
- Meta robots (noindex)Controls indexing of a specific page, unlike robots.txt, which only controls crawling.
- Sitemap.xmlThe sitemap whose address is often declared right inside robots.txt.
- Crawl budgetThe crawl limit that robots.txt manages by pointing bots to priority pages.
- Duplicate contentOne of the reasons utility and parameterized URLs get blocked via robots.txt.
- IndexationA process robots.txt affects only indirectly, by blocking crawling.
- Canonical URLAnother duplicate-content tool that works at the indexing level, not the crawling level.
Prepared by the LuchanLabs team