Robots.txt File Explained: Examples and Common Mistakes
Understand robots.txt rules with copy-ready examples, and avoid the small mistakes that can hide a whole website from search.
Robots.txt is a small text file at the root of your domain that tells crawlers which parts of your site they may visit. It is one of the oldest web standards still in use, and also one of the easiest to get wrong: a single misplaced slash can block an entire website from Google.
Below you will find how the file is read, examples for common situations, and the mistakes worth double-checking before you publish a change.
How crawlers read robots.txt
Before crawling a site, a well-behaved bot requests https://yourdomain.com/robots.txt. The file is split into groups. Each group starts with one or more User-agent lines, followed by Disallow and Allow rules that apply to those bots.
- A crawler follows the most specific group that matches its name, and ignores the others.
- Within a group, Google applies the longest matching rule. A longer
Allowcan override a shorterDisallow. - Paths are case-sensitive.
/Admin/and/admin/are different rules.
Examples you can adapt
Allow everything (a perfectly valid choice for most small sites): one group for all crawlers (user-agent set to an asterisk) with an empty disallow rule, plus a line giving the full address of your sitemap.
Keep crawlers out of admin and internal search pages: in the group for all crawlers, add one disallow rule for your admin folder, one for your search page, and one wildcard rule for search result addresses that contain a query parameter.
Block one folder but allow a file inside it: disallow the folder (for example your downloads folder), then add an allow rule for the single file inside it that you still want crawled, such as a public brochure. Because the allow rule is longer and more specific, it wins for that file.
Opt out of a specific AI training crawler while staying open to search: create a separate group naming that crawler (for example GPTBot) with a rule that disallows the whole site, and keep the general group for all other crawlers open.
Whether to block AI crawlers is a business decision. Blocking them can keep your content out of some AI answers as well as out of training data. The AI crawler checker shows which AI bots your current file allows.
Robots.txt is not a way to remove pages from Google
This is the single most misunderstood point. Disallow stops crawling, not indexing. If other sites link to a blocked page, Google can still list the bare URL in results โ it just cannot read the content. To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead. Blocking a page in robots.txt and adding noindex is counterproductive, because Google never gets to see the noindex.
Mistakes that cause real damage
Disallow: /left over from a staging site. Developers often block a test site, then copy the file to production. Always check robots.txt after a launch or migration.- Blocking CSS and JavaScript. Google renders pages like a browser. If it cannot load your stylesheets and scripts, it may judge the page as broken or not mobile-friendly.
- Using robots.txt to hide private content. The file is public; anyone can read it and see exactly which folders you consider sensitive. Protect private areas with passwords instead.
- Wrong file location or name. It must be at the root and named exactly
robots.txt, in lowercase. A file at/blog/robots.txtis ignored. - Huge files. Google reads only the first 500 KiB. If your file is that long, the rules can almost certainly be simplified with wildcards.
Building and testing a robots.txt file
The quickest safe route is to generate the file rather than type it. The free robots.txt generator builds the groups and syntax for you, including your sitemap line. Our guide on creating a robots.txt file with a free generator walks through it step by step. After uploading, check your most important URLs with the bulk indexability checker to confirm nothing valuable is blocked.
Related guides
More free tools: browse all our free SEO tools, no sign-up needed.
Frequently asked questions
Do I need a robots.txt file at all?
No. If the file is missing, crawlers assume they may visit everything. Having one is still useful for pointing to your sitemap and keeping bots away from low-value URLs.
Do all bots obey robots.txt?
Reputable search engines and most AI crawlers from major companies do. Scrapers and malicious bots often ignore it, which is why it is not a security tool.
How quickly does Google notice changes?
Google usually caches robots.txt for up to 24 hours, so changes typically apply within a day.
Can I use wildcards?
Yes. Google and Bing support * to match any sequence of characters and $ to match the end of a URL, for example Disallow: /*.pdf$.