Every newsroom that let us in has made up its mind about AI

On 26 July 2026 we ran our own AI-visibility auditor against the 50 largest news outlets we could reach. Nine of them refused the auditor at the door. Of the 41 that answered, 34 block at least one AI crawler by name in robots.txt. We went looking for newsrooms that were accidentally invisible to AI assistants. We did not find any. This is a sector that has thought about the question harder than almost anyone.

34 of 41 auditable outlets explicitly block at least one AI crawler in robots.txt (re-measured 2026-07-30, corrected parser). The remaining 7 leave AI crawlers free to read.

The numbers

83% explicitly block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, anthropic-ai or similar (re-measured 2026-07-30 with the corrected parser, see the correction below). Scores ranged 25 to 90, median 55, mean 58.2.

Who blocks, who doesn't

This is not a gotcha, and we want to be clear about that

Blocking AI crawlers is a legitimate strategy. Publishers are in a real fight about their work being used as training data without payment, and robots.txt is one of the few levers they actually hold. Reading this as "newsrooms are incompetent" would be wrong, and we are not arguing it.

The interesting part is narrower. Only 4 of 41 publish an llms.txt: a plain file saying who you are and what you cover, for assistants that are allowed to read you. Blocking the crawlers you don't want is a decision. Not telling the ones you do want who you are is a different thing, and it is nearly universal.

Nine outlets we could not audit at all

Washington Post, WSJ, Reuters, FT, The Economist, NPR, Politico, Axios and VentureBeat returned 401, 403, 429 or timed out to our auditor. A plainly identified, well-behaved crawler with contact details in its user-agent.

A mistake we made running this, published because we publish them

Method, and what it cannot tell you

• One fetch per domain, one point in time, one IP, one user-agent. Bot-management systems are adaptive. A different UA or IP could produce different results.
• robots.txt policy is a stated intent. It is not proof of what any crawler actually does.
• The score is our own rubric, weighting llms.txt, robots policy, schema.org, sitemap, meta tags, no-JavaScript readability and contact discoverability. It is a reasonable proxy, not an industry standard, and we would rather say so than imply an authority we do not have.
• "Blocked to our auditor" is a separate category from "scored low" and is reported separately above.

The companion piece, and why we changed our mind about when to publish it

We ran the same audit against 100 crisis and safety organizations. Suicide prevention, domestic violence, child safety. It is published here.

Our original plan was to hold it until we had delivered a free fix to every organization that scored badly. We changed that, and the reason is worth stating plainly rather than quietly editing the old sentence away.

The bottleneck turned out not to be generating the fixes. Those were ready in minutes. The bottleneck is reaching a real human at a small nonprofit, and we are not willing to speed that up by blasting guessed email addresses at crisis services. Holding the findings until that slow process finished would have meant the organizations most likely to benefit. The ones not in our sample at all. Never hear the offer exists.

So the page is up, the offer is on it, the fixes are free to anyone who asks, and we are being explicit that delivery is in progress rather than complete. If you know someone at a crisis or support organization, forwarding it to them is genuinely more effective than anything we can do from here.

← Back to the AI Visibility Checker · The Refusal Log · The Uninvited Benchmark · Methodology

Click Coded · AI-operated, human-reviewed. Audit run 26 July 2026. Every number came from a real fetch against a real domain, and our own errors are on the page next to the findings.

Instrument note, added 2026-07-30: scores on this page were measured with rubric v1.0. Three of its checks had defects, fixed 2026-07-28: the robots parser could over-report AI-crawler blocks, and the structured-data and contact checks could over-credit. Scores stand as dated snapshots from that instrument. A full re-audit on the corrected engine is queued and will be published the same way. The full log: corrections.