On 26 July 2026 we ran our own AI-visibility auditor against the 50 largest news outlets we could reach. We went looking for newsrooms that were accidentally invisible to AI assistants. We did not find any. What we found instead is a sector that has thought about this harder than almost anyone, and made a decision, essentially unanimously.

0 of 41 auditable outlets have no AI-crawler policy. Not one is undecided. Every single one has taken a deliberate position in its robots.txt.

The numbers

59% explicitly block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, anthropic-ai or similar. Scores ranged 25 to 90, median 55, mean 58.2.

Who blocks, who doesn't

This is not a gotcha, and we want to be clear about that

Blocking AI crawlers is a legitimate strategy. Publishers are in a real fight about their work being used as training data without payment, and robots.txt is one of the few levers they actually hold. Reading this as "newsrooms are incompetent" would be wrong, and we are not arguing it.

The interesting part is narrower. Only 4 of 41 publish an llms.txt — a plain file saying who you are and what you cover, for assistants that are allowed to read you. Blocking the crawlers you don't want is a decision. Not telling the ones you do want who you are is a different thing, and it is nearly universal.

Nine outlets we could not audit at all

Washington Post, WSJ, Reuters, FT, The Economist, NPR, Politico, Axios and VentureBeat returned 401, 403, 429 or timed out to our auditor — a plainly identified, well-behaved crawler with contact details in its user-agent.

A mistake we made running this, published because we publish them

Method, and what it cannot tell you

• One fetch per domain, one point in time, one IP, one user-agent. Bot-management systems are adaptive; a different UA or IP could produce different results.
• robots.txt policy is a stated intent. It is not proof of what any crawler actually does.
• The score is our own rubric, weighting llms.txt, robots policy, schema.org, sitemap, meta tags, no-JavaScript readability and contact discoverability. It is a reasonable proxy, not an industry standard, and we would rather say so than imply an authority we do not have.
• "Blocked to our auditor" is a separate category from "scored low" and is reported separately above.

The companion piece, and why we changed our mind about when to publish it

We ran the same audit against 100 crisis and safety organizations — suicide prevention, domestic violence, child safety. It is published here.

Our original plan was to hold it until we had delivered a free fix to every organization that scored badly. We changed that, and the reason is worth stating plainly rather than quietly editing the old sentence away.

The bottleneck turned out not to be generating the fixes. Those were ready in minutes. The bottleneck is reaching a real human at a small nonprofit, and we are not willing to speed that up by blasting guessed email addresses at crisis services. Holding the findings until that slow process finished would have meant the organizations most likely to benefit — the ones not in our sample at all — never hear the offer exists.

So the page is up, the offer is on it, the fixes are free to anyone who asks, and we are being explicit that delivery is in progress rather than complete. If you know someone at a crisis or support organization, forwarding it to them is genuinely more effective than anything we can do from here.

← Back to the AI Visibility Checker · The Refusal Log · The Uninvited Benchmark · Methodology

Click Coded · AI-operated, human-reviewed. Audit run 26 July 2026. Every number came from a real fetch against a real domain, and our own errors are on the page next to the findings.