Free SEO extension 🔥
SeoSpidie
Home Free tools Site audit Blog About Contact
Run free audit

AI crawler & llms.txt checker

Check whether your robots.txt allows or blocks the crawlers that power ChatGPT, Claude, Perplexity, Google's AI features, and more — then generate a starter llms.txt file, a proposed standard for pointing AI systems at your most important pages.

Works with
  • OpenAI GPTBot
  • Anthropic ClaudeBot
  • Perplexity
  • Google

Enter a domain and press Enter. We read only your public /robots.txt.

Which crawlers this checks
GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI) · ClaudeBot, Claude-User, Claude-SearchBot (Anthropic) · Google-Extended (Google's AI features, separate from regular Googlebot) · PerplexityBot (Perplexity) · CCBot (Common Crawl — used as training data by many LLMs, not just one) · Bytespider (ByteDance) · Amazonbot (Amazon) · Applebot-Extended (Apple Intelligence) · meta-externalagent (Meta AI). Blocking a bot here doesn't remove you from that engine entirely — some products (like ChatGPT's live browsing) also fetch pages directly regardless of the training crawler's status — but it's the main lever you actually control.

What are AI crawlers?

AI crawlers are bots run by AI companies to fetch web pages. Some collect content that may be used to train models; others fetch pages to answer a user's question or to build an AI search index. Examples include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which controls use of content for Google's AI models.

Because each company publishes separate user-agent names, site owners can allow some uses and block others in robots.txt. A publisher might block training crawlers but allow search crawlers so its pages can still be cited.

llms.txt is a proposed standard — a simple Markdown file at the root of a site — that points AI systems to a site's most important pages with short descriptions. It is new and not universally supported, but it is easy to add.

Why use the AI Crawler & llms.txt Checker?

AI assistants learn from and quote the web through their own crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others. Your robots.txt decides which of them may read your pages, and many sites block or allow them by accident because a broad rule written years ago now matches bots that did not exist then.

How to use it

  1. Enter your domain and run the check.
  2. Review each AI crawler: allowed, blocked, or partially blocked, plus the robots.txt rule responsible.
  3. Copy the suggested rules or the starter llms.txt file and upload them to your site root.

Tips for better results

  • Decide per bot: blocking training crawlers does not have to block search-style assistants that send you visitors.
  • Re-check after every CMS or security-plugin update — they sometimes rewrite robots.txt.
  • Keep llms.txt short: a one-line summary of the site and links to your most useful pages.

Step by step, with screenshots

  1. 1Enter your website address.
    Step 1: Enter your website address.
  2. 2See which AI crawlers your robots.txt allows or blocks, and the rule responsible.
    Step 2: See which AI crawlers your robots.txt allows or blocks, and the rule responsible.

Features of the AI Crawler & llms.txt Checker

  • Reads your robots.txt. Checks it from SeoSpidie's server.
  • Status for each AI crawler. Allowed, blocked or partially blocked.
  • Shows the responsible rule. See which robots.txt line causes each result.
  • Suggested rules. Copy rules to allow or block specific bots.
  • Starter llms.txt. Generate a first version to upload to your site root.

Who is it for?

  • Publishers: control how AI companies use your content.
  • Businesses: make sure AI search tools can see your pages.
  • SEO specialists: audit AI access alongside search engine access.
  • Developers: catch rules added by plugins or security tools.

Understanding your results

Review each bot and decide whether its result matches your intention. If a search-related crawler is blocked by accident, find the rule responsible and change it. If you want to block training use only, block those agents specifically rather than every bot.

Remember that robots.txt is a request respected by well-behaved crawlers; it is not access control.

Example: separating AI search from AI training

A news site in Ireland wanted to stop its articles being used to train AI models but still appear as a cited source in AI search answers.

The AI Crawler Checker showed that its robots.txt blocked every listed bot with a single wildcard rule added years earlier. Using the suggested rules, the site blocked training-focused crawlers by name and allowed search-related ones. It also published a starter llms.txt pointing to its main sections. The checker then showed exactly the split the editors intended.

Common mistakes to avoid

  • Blocking every bot with a wildcard and hiding from AI search.
  • Confusing training crawlers with search crawlers.
  • Forgetting rules added by security plugins or hosts.
  • Expecting llms.txt to replace robots.txt.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT answers?

It stops OpenAI's training crawler from reading your pages going forward. Content already collected and answers built from other sources are not affected, and ChatGPT's live browsing uses a separate user agent.

What is llms.txt?

A proposed plain-text file at your site root that gives AI tools a concise overview of your site and its key pages. It is optional and not an official standard, but it is cheap to add.

Is robots.txt legally binding for AI crawlers?

No — it is a voluntary convention. The major AI companies say their crawlers respect it, which is why it is the main control available to site owners.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI uses GPTBot to collect content that may be used for training, and OAI-SearchBot to find pages for ChatGPT search. You can allow or block each separately.

Does blocking AI crawlers affect Google rankings?

Blocking AI-only crawlers does not affect Googlebot. Google-Extended controls AI use of content and is separate from search crawling.

Is my data private?

To read other websites, SeoSpidie's server fetches the public URLs you enter and sends the result back to your browser. We don't keep your reports or require an account. Text you paste stays in your browser. Fair-use limits apply to protect the service.

Is it free?

Yes. The AI Crawler & llms.txt Checker is free to use with no sign-up. SeoSpidie is supported by advertising.

View all 7 →
AI Overview OptimizerOptimize content for visibility in AI-powered search results.GEO CheckerAnalyze geographic and location-related SEO signals.Featured Snippet OptimizerStructure your content for better featured snippet opportunities.Blog Topic GeneratorGenerate useful, search-focused blog topic ideas.