Are you letting AI in?
If your robots.txt blocks AI search crawlers, you can lose eligibility for citations in some AI answers, often without realizing it. Enter your domain and we read your live robots.txt and, if present, your llms.txt, then show which AI crawlers your robots.txt appears to allow or block, and hand you a recommended file. No API key needed. Note: robots.txt is honored voluntarily (well-behaved bots respect it, some ignore it), and llms.txt is a content-guidance convention, not an access-control standard.
Who can read your site, and who you shut out.
We check your domain’s /robots.txt rules (and whether you publish an /llms.txt) against the documented crawler user-agents behind ChatGPT, Claude, Perplexity, Copilot (via Bing) and the open training datasets, plus Google-Extended, which controls training and grounding data for Gemini and AI Overviews (Gemini itself has no separate search-retrieval crawler; it draws on the standard Googlebot index). robots.txt governs only compliant bots, content already in a model’s training data persists regardless.
We fetch only your site’s public robots.txt and llms.txt over HTTPS. Nothing is stored. Want this watched for you? Talk to us.
Questions, answered.
We run proactive optimization with clear reporting and documented actions. You always know what's being worked on, why it matters, and what comes next.
Full-service digital marketing and AI consulting, SEO, paid advertising, content strategy, web design and development, local marketing, AI agent development, AI strategy, automation, analytics, and business strategy.
Yes. We set the strategy and then own the execution, with clear priorities and timelines so nothing drifts or stalls.
We measure against the outcomes that matter to your business, qualified leads, conversions, and revenue, and report on what changed and what it impacted.
Both. Our local marketing and maps optimization services are built around your service areas, whether that's one location or many.
Unclear ownership, reporting that hides impact, and no documented plan for what's being worked on or why.
What is AI Crawler & llms.txt Auditor?
The AI Crawler & llms.txt Auditor is a free tool that reads your site's live robots.txt and llms.txt and shows which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more) your rules appear to allow or block. It separates search-retrieval crawlers, the ones that fetch pages to answer live questions, from training opt-out controls, and hands you a recommended robots.txt and clean llms.txt to copy in.
How it works
Enter a domain and the tool fetches only your public /robots.txt and /llms.txt over HTTPS, then checks the documented user-agents behind ChatGPT, Claude, Gemini, Perplexity, Copilot (via Bing) and the open training datasets against your rules. It flags what is blocked, explains the difference between blocking retrieval versus training, and generates files you can paste in. Nothing is stored, and no API key is needed.
Who it’s for
For site owners, marketers, and developers who want to make sure AI search engines can actually reach their pages instead of being blocked by accident. The outcome is eligibility: if the retrieval crawlers can read you, you stay in the running to be cited in AI answers, and you get a clear file to fix it rather than guessing. It is also useful for anyone who wants to allow AI search but opt out of model training, which are two separate settings people often confuse.
In practice
A clinic pastes in its domain and finds its robots.txt is silently blocking OAI-SearchBot and PerplexityBot, so it never appears when people ask ChatGPT or Perplexity for a local provider. The tool shows the exact rules causing it and gives a corrected robots.txt that opens the site to search-retrieval crawlers while keeping the training opt-out for GPTBot and Google-Extended, plus a starter llms.txt to publish.