CLICK CODED · free tool
robots.txt is the first thing every crawler checks, including the AI ones. Decide who gets in, generate the file, done. Free, no signup, runs in your browser.
Every listed AI crawler can read your site. Traditional search engines are always allowed. Best choice if you want to be found and cited by ChatGPT, Perplexity, and Claude.
robots.txt decides who can crawl. It doesn't tell them what you offer, that's llms.txt. It doesn't state your business facts, that's structured data. It doesn't make your answers quotable, that's FAQ schema. Do all four, then verify with the free AI visibility checker. Want it all done and confirmed? See the agent-ready retrofit ($60-175).
Policies drift too. A plugin update or a migration can quietly reset them. Pulse's Agency tier re-checks robots.txt and the rest of your AI-visibility score every month and flags it in your next report if it changes, $99/mo, built for agencies managing multiple client sites, not a single-site plan.
Want the full 12-point checklist this file is one line of? Take the free AI Visibility Checklist. Pay what you want, including nothing.
AI crawlers. GPTBot (OpenAI), ClaudeBot and Claude-User (Anthropic), PerplexityBot, OAI-SearchBot, Google-Extended, anthropic-ai. Check robots.txt before reading anything. Not deciding means someone else decided for you, usually by leaving you off a default deny-list you never wrote. Built by Click Coded, AI-operated and human-reviewed. Contact: [email protected]
We add new free tools and benchmarks periodically. Get an email when we do, no spam, unsubscribe anytime.
Part of Click Coded: trust between humans and AI, checkable. The Checkable Standard