{"corpus": "sept15-saas", "domain": "stripe.com", "fetchedAt": "2026-07-30T17:18:36.651300Z", "result": {"base": "https://stripe.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (64986 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Stripe | Financial Infrastructure to Grow Your Revenue\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://stripe.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12855 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://stripe.com/contact/sales).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 85, "title": "Stripe | Financial Infrastructure to Grow Your Revenue", "textOnly": "Stripe | Financial Infrastructure to Grow Your Revenue Products Solutions Developers Resources Pricing Sign in Start now Contact sales Back Sign in Sign in Contact sales Global GDP running on Stripe: Financial infrastructure to grow your revenue. Accept payments, offer financial services, and implement custom revenue models\u2014from your first transaction to your billionth. Financial infrastructure to grow your revenue. Accept payments, offer financial services, and implement custom revenue models\u2014from your first transaction to your billionth. Get started Sign up with Google Flexible solutions for every business model. Grow your business with a comprehensive set of payments and financial tools\u2060\u2014\u2060designed to work individually or together. Grow your business with the most comprehensive set of payments and financial tools. Accept and optimize payments globally\u2014online and in person Enable any billing model Pro Plan Billed monthly Tokens $0.01 per 1,000 units Usage meter Tokens used in the last 30 days Monetize through agentic commerce Create a card issuing program Access borderless money movement with stablecoins and crypto Embed payments in your platform Building the economic infrastructure for AI Watch now The backbone of global commerce 135+ currencies and payment methods supported $1.9T in payments volume processed in 2025 99.999% historical uptime for Stripe services 200M+ active subscriptions managed on Stripe Billing 135+ currencies and payment methods supported $1.9T in payments volume processed in 2025 99.999% historical uptime for Stripe services 200M+ active subscriptions managed on Stripe Billing Powering businesses of all sizes. Run your business on a reliable platform that adapts to your needs. Transform your enterprise with agile financial infrastructure Stripe for enterprises 50% of Fortune 100 companies have used Stripe to grow their businesses\u2014from expanding internationally to reimagining the customer experience. Hertz unifies commerce with Stripe. Read the story 160 countries 11K+ locations globally Products used Payments, Terminal, Connect, Radar, and Stripe Sigma Read the story URBN consolidates $5 billion in online and in-store revenue onto Stripe. Read the story 5+ consumer brands in retailer portfolio 700+ store locations Products used Payments, Terminal, Connect, Stripe Sigma, Radar, and Link Read the story Instacart powers online grocery delivery with Stripe. Read the story 600K+ shoppers 1.8K retail partners across nearly 100K stores Products used Payments, Connect, Data Pipeline, and Issuing Read the story Le Monde improves local and international payments with Stripe. Read the story 100% of digital and print payments powered by Stripe Less than 3 months to implement and go live Products used Payments, Stripe Sigma, and Radar Read the story Realize value faster with dedicated experts Professional services. Get tailored guidance from Stripe on implementation, complex integrations, or major migrations. View services Stripe-certified experts. Work with a Stripe consulting partner that can integrate and deploy Stripe solutions on your behalf. View partners Support plans. Receive ongoing assistance and day-to-day support for technical questions with tiered plans based on your needs. View plans Build a foundation for your startup that enables faster growth Stripe for startups From stablecoin pioneers to 78% of the Forbes AI 50, Stripe helps startups build what\u2019s next on easy-to-integrate financial infrastructure. Lovable grows into a vibe-coding juggernaut with Stripe. Read Lovable\u2019s story Gamma expands to $100M ARR and 70 million users with Stripe. Read Gamma\u2019s story Runway protects developer time with no-code solutions from Stripe. Read Runway\u2019s story Supabase delivers its backend-as-a-service to 150 countries with Stripe. Read Supabase\u2019s story Linear partners with Stripe to handle billing and payments. Read Linear\u2019s story ElevenLabs grows into a $3B AI audio leader with Stripe. Watch the video Browserbase offe", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://stripe.com/contact/sales", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "notion.so", "fetchedAt": "2026-07-30T17:18:42.910724Z", "result": {"base": "https://notion.so", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (6776 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The AI workspace that works for you. | Notion\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.notion.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3904 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://notion.so/contact-sales).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "The AI workspace that works for you. | Notion", "textOnly": "The AI workspace that works for you. | Notion Product Notion AI AI tools for work Agents Automate busywork AI Meeting Notes Perfectly written by AI Enterprise Search Find answers instantly Knowledge Base Centralize your knowledge Docs Simple and powerful Projects Manage any project Connections Connect your apps Security Safe and scalable Notion Calendar It&#x27;s time . See what\u2019s new \u2192 Download the Notion App \u2192 AI AI features Notion AI AI tools for work Agents Automate busywork AI Meeting Notes Perfectly written by AI Enterprise Search Find answers instantly Explore use cases For work For life Solutions Teams Eng &amp; Product Design Marketing IT Company size Startups Small businesses Enterprise Use Cases Education Personal AI use cases Resources Browse Templates Consultants Connections Discover What&#x27;s New Customer stories Blog Webinars Learn Developers Academy Product tours Help Developers Enterprise Pricing Request a demo Log in Get Notion free Where teams and agents Think together. Capture context, find answers, and automate tasks with AI built for your team. Get Notion free Request a demo Play Trusted by 98% of the Forbes Cloud 100 Keep work moving 24/7. Custom Agents Automate repetitive work for your team. \u2192 Q&amp;A agents Answers questions instantly using knowledge you already have. Task routing agents Assigns, prioritizes, and routes tasks on its own. Reporting agents Summarizes, writes, and sends reports for you. Create your own Make a Custom Agent for any work that repeats. Q&amp;A agents Answers questions instantly using knowledge you already have. See what Custom Agents can do Triage product feedback \u2192 Resolve support tickets in Slack \u2192 Respond to security alerts faster \u2192 Automate weekly reporting \u2192 Create your own Custom Agent \u2192 Ask your on-demand assistants. Notion Agent You assign the tasks. Notion Agent does the work. \u2192 Enterprise Search One search for everything. \u2192 AI Meeting Notes Perfect notes, every time. \u2192 Bring all your work together. Docs Simple and powerful. \u2192 Knowledge Base One source of truth for teams and agents. \u2192 Projects Less tracking. More progress. \u2192 Trusted by teams that ship. There\u2019s power in a single platform where you can do all your work. Notion is that single place. Read the full story \u2192 View the video Streamlined workflows to reduce timelines by 3x. \u2192 \u201cAgents get created in three minutes between meetings, then hours of manual operational work disappear.\u201d \u2192 \u201cNotion understands that you can solve a lot of problems with one tool.\u201d \u2192 \u201cNotion has been the most powerful and impactful way to streamline our workflow.\u201d \u2192 \u201cUsing the most AI-native tools like Notion is an important competitive advantage for us.\u201d \u2192 One hub for work and AI keeps everyone informed and work flowing. \u2192 Over 100M users worldwide #1 knowledge base 3 years running (G2) #1 AI enterprise search (G2) #1 rated AI writing (G2) 62% of Fortune 100 Over 50% of YC companies 1.4M+ community members Over 100M users worldwide #1 knowledge base 3 years running (G2) #1 AI enterprise search (G2) #1 rated AI writing (G2) 62% of Fortune 100 Over 50% of YC companies 1.4M+ community members Over 100M users worldwide #1 knowledge base 3 years running (G2) #1 AI enterprise search (G2) #1 rated AI writing (G2) 62% of Fortune 100 Over 50% of YC companies 1.4M+ community members English (US) English (US) Bahasa Indonesia Dansk Deutsch English (UK) Espa\u00f1ol (Espa\u00f1a) Espa\u00f1ol (Latinoam\u00e9rica) Fran\u00e7ais (France) Italiano Nederlands Norsk Portugu\u00eas (Brasil) Suomi Svenska Ti\u1ebfng Vi\u1ec7t \u05e2\u05d1\u05e8\u05d9\u05ea \u0627\u0644\u0639\u0631\u0628\u064a\u0629 \u0e20\u0e32\u0e29\u0e32\u0e44\u0e17\u0e22 \ud55c\uad6d\uc5b4 \u65e5\u672c\u8a9e \u7b80\u4f53\u4e2d\u6587 \u7e41\u9ad4\u4e2d\u6587 Cookie settings \u00a9 2026 Notion Labs, Inc. Company About us Careers Security Status Terms &amp; privacy Your privacy rights Download iOS &amp; Android Mac &amp; Windows Calendar Web Clipper Resources Help center Pricing Blog Community Connections Templates Partner programs Notion for Enterprise Startups Small business Personal Explore more \u2192", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://notion.so/contact-sales", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "figma.com", "fetchedAt": "2026-07-30T17:18:52.399259Z", "result": {"base": "https://figma.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Figma: The collaborative canvas for design, code, and AI\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.figma.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3983 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.figma.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Figma: The collaborative canvas for design, code, and AI", "textOnly": "Figma: The collaborative canvas for design, code, and AI Skip to main content Products What you can do in figma Design and explore Build and ship products Explore all products Figma Design Design and prototype in one place Figma Make Prompt to code anything you can imagine Figma Weave AI workflows for imagery, video, audio, and more Dev Mode Translate designs into code Figma Motion New Animate your designs FigJam Collaborate with a digital whiteboard Figma Slides Co-create presentations Figma Draw Illustrate with advanced vector tools Figma Sites Beta Publish fully responsive websites Figma Buzz Beta Produce on-brand assets at scale Additional product resources AI workflows Release notes Downloads Solutions Use cases Design systems Prototyping UX design Web design Wireframing Online whiteboard Agile Strategic planning Brainstorming Diagramming Product development Web development Design handoff See all solutions Roles Design Engineering Product managers Organizations Enterprise Education Government Community Config 2026 Join virtually June 24\u201325 for free Events Learn best practices at virtual events User groups Join a local Friends of Figma group Customer stories Read about leading product teams Figma Gallery Explore Figma Make and Figma Sites projects made by our users Resources Shortcut: The Figma blog Stories about how products take shape\u2014and shape our world Get started Plugins UI kits Icons Widgets Templates Tools Partners See all Learn Best practices Reports &amp; insights Resource library Demos Webinars Help center Figma Support Pricing Log in Contact sales Get started Get started for free Log in Contact sales Get started Get started for free The intelligent canvas for infinite creativity Get started One workspace for your entire product development process. Made so your whole team can go from WIP to ship, together. Design anything you can imagine Turn your ideas into apps, websites, and products. Explore design tools Build with intention Dev tools that take you all the way to production. Explore build tools Move fast in the right direction on an AI-native canvas. Teammates and AI agents work in the same space with shared context. Generate new design directions, refine, and align\u2014together. Powerfully expressive. Incredibly precise. Every technical tool you need to dial in the details. Explore Figma Design Start with design context. Build with consistency. Your context, codebase, and agents stay connected so what you design is what gets shipped. Explore Figma Make The products you love are designed in Figma Everyone is able to influence, inspire, and give input without ever leaving the design file. That creates a really transparent, open, and honest process throughout the whole project. Francisco Seiz Senior Design Director, Code and Theory 95% 95 % 95% of the Fortune 500 uses Figma Based on data from March 2025. Explore what you can do in Figma. Browse resources made by the community. Browse all templates Slide 1 of 10 1 of 10 UI kits UI Kits Websites Websites Social media Social media Mobile apps Mobile apps Presentations Presentations Wireframes Wireframes Illustrations Illustrations Portfolios Portfolios Web ads Web ads Icons Icons Figma Socials Product Figma Design Figma Make Figma Weave New Dev Mode Figma Motion New FigJam Figma Slides Figma Draw Figma Sites Beta Figma Buzz Beta Design and explore Build and ship products AI workflows Release notes Downloads Plans Pricing Enterprise Organization Professional Use cases UI design UX design Wireframing Diagramming Prototyping Brainstorming Presentation Maker Online whiteboard Agile Strategic planning Mind mapping Resources Blog Best practices Gradient Library Gradient Generator GIF maker Color wheel Colors Color picker Font Library Templates Developers Integrations Affiliate program Resource library Reports and insights Support Status Legal and privacy Modern slavery statement Climate disclosure statement Company Events Customers Careers Newsroom English", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: CHATGPT-User, OAI-SearchBot, PerplexityBot, Claude-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: CHATGPT-User, OAI-SearchBot, PerplexityBot, Claude-SearchBot\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: CHATGPT-User, OAI-SearchBot, PerplexityBot, Claude-SearchBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.figma.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "vercel.com", "fetchedAt": "2026-07-30T17:18:58.582415Z", "result": {"base": "https://vercel.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (202600 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: SoftwareApplication.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Agentic Infrastructure - Vercel\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://vercel.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3646 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://vercel.com/contact/sales).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 100, "title": "Agentic Infrastructure - Vercel", "textOnly": "Agentic Infrastructure - Vercel Skip to content Copy Logo as SVG Copy Wordmark as SVG Brand Guidelines Products Agent Stack AI SDK AI Gateway Sandbox Workflows Passport eve Core Platform Security Content Delivery Fluid Compute Observability CI/CD Tools Next.js Vercel Agent Vercel Plugin Domains \u2197 v0 \u2197 Resources Learn Docs About Blog Changelog Knowledge Base Build AI Apps Web Apps Marketing Sites Platforms Commerce Explore Customers Marketplace Partner Finder AWS Community \u2197 Enterprise Pricing Get a Demo Log In Sign Up Ask AI Dashboard Products Agent Stack AI SDK AI Gateway Sandbox Workflows Passport eve Core Platform Security Content Delivery Fluid Compute Observability CI/CD Tools Next.js Vercel Agent Vercel Plugin Domains \u2197 v0 \u2197 Resources Learn Docs About Blog Changelog Knowledge Base Build AI Apps Web Apps Marketing Sites Platforms Commerce Explore Customers Marketplace Partner Finder AWS Community \u2197 Enterprise Pricing Get a Demo Log In Sign Up Drop to deploy Loading Agentic Infrastructure To ship apps and agents For coding agents For coding agents to ship apps and agents automated by agents. Deploy now Talk to sales For coding agents to deploy in their native language, with Vercel&#x27;s API, CLI, MCP, and Skills. To ship apps and agents in Sandboxed VMs, with durable backends, powered by hundreds of models. Automated by agents who autonomously investigate errors, plan fixes, and open PRs. Build agents on infrastructure that thinks like them Notion powers millions of agent conversations daily on Vercel. Features Durable Orchestration Sandboxed Environments AI Model Gateway Fluid Compute Ship apps that scale from zero to millions instantly Zapier serves over 100 million monthly website visits on Vercel. Features Global Delivery Deployment Environments Serverless Functions Web Application Firewall Host platforms that serve every customer Mintlify powers documentation for over 20,000 companies on Vercel. Features Tenant Isolation Domain Management Custom SSL Certificates Preview URLs Recently shipped A framework for building durable agents. Vercel Passport Pasaporte Passaporto \u30d1\u30b9\u30dd\u30fc\u30c8 Iss 2026 Passport Secure every internal agent, app, and deployment with your identity provider. \u25b2 vercel deploy Vercel CLI \u2713 Building image from Dockerfile.vercel \u2713 Stored image in your project&#x27;s registry \u2713 Deployed to Fluid compute Production: https://my-server.vercel.app Containers Run production workloads in isolated containers on Vercel. Built by you, or your agents Deploy now Plugin Plugin $ npx plugins add vercel/vercel-plugin npx plugins add vercel/vercel-plugin Agent Stack AI SDK AI Gateway Sandbox Workflows Connect New Passport New eve New Core Platform CI/CD Content Delivery Fluid Compute Observability Security Platform Security WAF Bot Management Bot ID Tools Vercel Drop New Vercel Agent Vercel Plugin New Agent Skills Next.js Domains v0 Frameworks eve New Nuxt SvelteKit Nitro Turborepo Tanstack Start FastAPI xmcp All frameworks SDKs Vercel SDK Workflow SDK New Flags SDK Chat SDK New Queues SDK New Streamdown Build AI Apps Web Apps Marketing Sites Platforms Commerce Platform Engineers Design Engineers Learn Docs Blog Changelog Knowledge Base Academy Articles Community Explore Customers Marketplace Templates Partner Finder Vercel + AWS Company About Careers Press Events Startups Shipped on Vercel Open Source Program Enterprise Pricing Help Legal &amp; Trust Privacy Policy Terms of Service Cookie Policy DPA Acceptable Use Policy Legal (all documents) Trust Center Status Cookie Preferences Social GitHub X LinkedIn YouTube Instagram Loading status\u2026 Select a display theme: system light dark", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://vercel.com/contact/sales", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "linear.app", "fetchedAt": "2026-07-30T17:19:05.580610Z", "result": {"base": "https://linear.app", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (9733 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Linear \u2013 The system for product development\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://linear.app).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "735127 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://linear.app/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 82, "title": "Linear \u2013 The system for product development", "textOnly": "Linear \u2013 The system for product development Skip to content \u2192 Product Resources Customers Pricing Now Contact Docs Open app Log in Sign up The product development system for teams and agents The product development system for teams and agents The product development system for teams and agents Purpose-built for planning and building products. Designed for the AI era. New Coding Sessions \u2192 New Coding Sessions \u2192 Linear Inbox My issues Reviews Pulse Workspace Initiatives Projects More Favorites Faster app launch Agent tasks UI Refresh Agents Insights Faster app launch 02 / 145 Faster app launch Render UI before vehicle_state sync when minimum required state is present, instead of blocking on full refresh during iOS startup. Activity Linear created the issue via Slack on behalf of karri \u00b7 2min ago Triage Intelligence added the label Performance and iOS \u00b7 2min ago karri \u00b7 4 min ago Right now we show a spinner forever, which makes it look like the car disappeared... jori \u00b7 just now @Linear can you take a stab at this? Linear connected by jori \u00b7 2 min ago Changed 2 files Draft PR awaiting your review \u00b7 2 min ago Linear moved from Todo to In Progress \u00b7 just now Linear Opus 5 jori connected Linear to ENG-2703 Examining the startup path... Thinking... ENG-2703 In Progress High jori Linear Labels Performance iOS Cycle Cycle 144 Project Core Performance A new species of product tool. Purpose-built for modern teams with AI workflows at its core, Linear sets a new standard for planning and building products. Purpose-built Linear is shaped by the practices and principles of world-class product teams. Powered by agents Designed for workflows shared by humans and agents. From drafting PRDs to pushing PRs. Designed for speed Reduces noise and restores momentum to help teams ship with high velocity and focus. FIG 0.2 Purpose-built Linear is shaped by the practices and principles of world-class product teams. FIG 0.3 Powered by agents Designed for workflows shared by humans and agents. From drafting PRDs to pushing PRs. Designed for workflows shared by humans and agents. From drafting PRDs to pushing PRs. FIG 0.4 Designed for speed Reduces noise and restores momentum to help teams ship with high velocity and focus. Make product operations self-driving Turn conversations and customer feedback into actionable issues that are routed, labeled, and prioritized for the right team. 1.0 Intake \u2192 Backlog 8 ENG-2085 Reduce UI flicker during autonomy... ENG-2094 Add buffering for autonomy event streams ENG-2092 Reduce startup delay caused by vehicle sync ENG-2200 Fix delayed route updates during rerouting Todo 71 ENG-926 Remove UI inconsistencies Bug Design ENG-2088 TypeError: Cannot read properties Bug ENG-924 Upgrade to Claude Opus 4.8 AI ENG-1882 Optimize load times Performance In Progress 3 ENG-1487 Remove contentData from GraphQL API 61039 MKT-1028 Launch page assets Design ENG-2187 Prevent duplicate ride requests on poor... Bug 62048 Done 53 ENG-2074 Clean up deprecated APIs... API 61002 ENG-1912 Reduce latency in autonomy st... 61005 ENG-1951 Reduce ETA fluctuations durin... 61202 ENG-1960 Improve fallback messaging UI 61149 ENG-1991 Improve rider visibility into veh... Thread in #feedback didier Has anyone been looking into the iOS startup performance issues? lena Anyone else noticing the iOS app feels slow to open if you haven&#x27;t used it in a bit? didier Yea, we&#x27;re still blocking initial render on a full vehicle_state sync every time... andreas Feels like we could render sooner and load the rest in the background. Probably also worth tracking startup timing so we know how often this happens! @Linear create issues urgent issues and assign to me 1 . 1 Linear Agent + 1 . 2 Triage + 1 . 3 Customer Requests + 1 . 4 Linear Asks + Define the product direction Plan and navigate from idea to launch. Align your team with product initiatives, strategic roadmaps, and clear, up-to-date PRDs. 2.0 Plan \u2192 FEB MAR APR MAY JUN JUL AUG SEP 2 9 16 23 2 9 16 2", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://linear.app/contact", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "airtable.com", "fetchedAt": "2026-07-30T17:19:11.130934Z", "result": {"base": "https://airtable.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Airtable: Build Enterprise-ready AI Workflows, Apps &amp; Ag\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.airtable.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "8304 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://support.airtable.com/docs/contacting-airtable-support).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 54, "title": "Airtable: Build Enterprise-ready AI Workflows, Apps &amp; Agents | Airtable", "textOnly": "Airtable: Build Enterprise-ready AI Workflows, Apps &amp; Agents | Airtable Bring your Airtable data into Claude. Try it now Airtable home or view your bases Platform Platform Platform &amp; Products Airtable Platform Unlock the power of a next-gen app-building platform AI App Building NEW Enable teams to build custom business apps\u2014no code needed AI Agents NEW Deploy thousands of agents inside your apps Portals Collaborate seamlessly with guest users outside your organization Scale Build apps on top of massive data sets with 100M records Platform Features Learn about automations, interfaces, and more Back Features Automations Achieve more in less time via automations Databases Use relational databases to connect data from apps, workflows, and tools Interfaces Create custom interfaces from your data Reporting Measure, analyze, and share insights Views Tailor data visualizations to your teams\u2019 needs Governance and Security Centrally manage security, permissions and data protection Airtable AI Embed AI throughout end-to-end workflows What&#x27;s New See Airtable&#x27;s latest product updates Tools Integrations Integrate with tools like Slack, Google Drive, and Salesforce Back Integrations Slack Add real-time messaging to your workflows Google Drive Sync information and build workflows Salesforce Leverage CRM data across teams and tools Jira Generate tickets and sync work from Airtable Zendesk Simplify and streamline workflows See all integrations Download Get the Airtable app for desktop and mobile The Airtable Platform Propel your business with a next-gen app-building platform. See data move in real-time and securely transform your work with AI. Explore the platform Solutions Solutions Teams Product See what&#x27;s happening across your portfolio, and ship better products. Marketing Plan smarter campaigns and scale impact with unifed marketing workflows. Project Management Strategic project management built for outcomes, not just to-do lists. Operations Run operations in one platform that connects workflows, data, and tools. Sales Surface sales opportunities, spot churn risk early, and close more deals faster. Design &amp; Creative Unify design workflows, stay on-brand, and innovate creative quickly. Human Resources Manage and streamline processes from hire to retire Finance Streamline requests, manage budgets, and automate reporting Industries Media &amp; Entertainment Run content production, digital assets, and crew operations in one place Retail Power critical retail workflows from product planning to final purchase CPG Manufacturing Keep track of your products, supply chain, and quality assurance Education Manage student services, planning, and grants more efficiently Technology Unite your entire SaaS org to drive faster GTM execution and greater impact. Agency &amp; Professional Services Streamline coordination, protect margins, and deliver better client outcomes. Customer Story More money, fewer problems How Code and Theory consolidated tooling and saved more than 10,000 hours with Airtable. Learn more Resources Resources Explore Resources Hub Back Resources Hub Reports eBooks Webinars Quick Reads Demos See all resources Templates Back Templates Project Management New Content Calendar Event Planning Product Roadmap Organizational Chart Resource Allocation New Social Media Calendar Marketing Campaign Tracker Product Launch Content Strategy Product Strategy Inventory Tracker Product Catalog See all templates AI Plays Blog Customer Stories Videos Events Learn &amp; Support Airtable Academy Airtable Community Developers Docs Back Developers Dev Community Share ideas and find solutions from peers API Docs Build with the Airtable API Custom Extensions Create real-time, interactive extensions Scripting Build something custom with scripts How-to guides Support Back Support Help center Contact Support Ecosystem Marketplace Partners Services Resource Center Stay one step ahead of the trends This content is packed with demos, best pr", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://support.airtable.com/docs/contacting-airtable-support", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "hubspot.com", "fetchedAt": "2026-07-30T17:19:17.029936Z", "result": {"base": "https://hubspot.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (30688 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, WebPage, Product.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"HubSpot | Software &amp; Tools for your Business - Homepage\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.hubspot.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "15055 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://offers.hubspot.com/contact-sales).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "HubSpot | Software &amp; Tools for your Business - Homepage", "textOnly": "HubSpot | Software &amp; Tools for your Business - Homepage Logo - Full (Color) Skip to content English Select a language \u65e5\u672c\u8a9e Deutsch English Espa\u00f1ol Portugu\u00eas Fran\u00e7ais High Contrast Customer Support Contact Sales Close Search Log in About About About Us Careers Contact Us Investor Relations Management Team Back Menu Close Search Products Products The HubSpot Customer Platform All of HubSpot's marketing, sales, and customer service software on one agentic platform. Free HubSpot CRM Overview of all products Marketing Hub Marketing automation software Free and premium plans Sales Hub Sales software Free and premium plans Service Hub Customer service software Free and premium plans Content Hub Content marketing software Free and premium plans Data Hub Data management software Free and premium plans Revenue Hub CPQ, billing, and payments software Free and premium plans Smart CRM AI-powered, flexible CRM software Learn more Agent Hub Your central home for building and managing AI agents across the platform Learn more Small Business Bundle The Starter edition of each product, built for startups and small businesses Learn more AEO (Beta) Answer engine optimization tools that track and improve your brand's visibility in AI results Learn more HubSpot Marketplace Connect your favorite apps to HubSpot See all integrations Solutions Solutions By Use Case By Team Size Why HubSpot? Marketing Generate leads Convert visitors into contacts for your database. Automate marketing Create campaigns with automation and AI. Sales Build pipeline Generate a pipeline of high-quality prospects. Close deals Streamline your process and close more deals faster. Customer Service Scale support Support customers at scale with AI and a help desk. Drive retention Identify opportunities to improve customer health. Content Create content Easily craft content for your audience, with help from AI. Manage content Organize, update, and distribute content in one place. Startups &amp; Small Businesses Find and reach customers Generate leads through content, AI, and automation. Grow sales and get paid Prospect smarter and automate payments. Organize customer data Unify your team and data on one platform. Artificial Intelligence Resolve customer queries 24/7 Scale customer service with an AI customer agent. Automate sales prospecting Identify and engage high-value leads with an AI prospecting agent. Research customers faster Get instant answers about your customers with an AI data agent. By Team Size For Small Businesses &amp; Startups HubSpot\u2019s all-in-one Starter Customer Platform helps your growing startup or small business find and win customers from day one. Learn more about HubSpot\u2019s Starter Customer Platform For Enterprises With HubSpot\u2019s integrated Enterprise Customer Platform, you don\u2019t have to sacrifice power for ease of use. Learn more about HubSpot\u2019s Enterprise Customer Platform Why HubSpot? Why Choose HubSpot? After just one year, HubSpot customers acquire 129% more leads, close 36% more deals, and see a 37% improvement in ticket closure rates. Learn more about why how HubSpot\u2019s solution is different Case Studies Explore examples of companies like yours from all over the globe that use HubSpot to unite their teams, empower their businesses, and grow better. See all case studies Spotlight: Product Updates Learn about HubSpot\u2019s featured product releases and announcements in this semi-annual product showcase. Explore product updates Pricing Resources Resources Featured Links Spotlight: Product Updates What's New in HubSpot Why Choose HubSpot? Sustainability Community &amp; Events UNBOUND Event Webinars HubSpot Community HubSpot User Groups Partners Solutions Partner Program Technology Partner Program Affiliate Partner Program Education Partner Program Startup Partner Program Education The Loop Marketing Playbook What Is Inbound Marketing? HubSpot Blogs Free Courses &amp; Certifications Ebooks, Guides &amp; More HubSpot Knowledge Base Tools Website Templates Devel", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://offers.hubspot.com/contact-sales", "sept15": {"behindCloudflare": true, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "posture-choice", "detail": "Served through Cloudflare with no ad code detected, currently readable by AI. The September 15 default targets ad-bearing pages, so no automatic flip is expected here. But this site\u2019s AI access is now a Cloudflare dashboard setting \u2014 block, allow, or charge under Pay Per Use, where publishers get paid when content is actually used in an AI answer. Confirm the current posture is a decision, not a default."}}}
{"corpus": "sept15-saas", "domain": "shopify.com", "fetchedAt": "2026-07-30T17:19:22.463494Z", "result": {"base": "https://shopify.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (861 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Shopify: The All-in-One Commerce Platform for Businesses - S\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.shopify.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7828 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 85, "title": "Shopify: The All-in-One Commerce Platform for Businesses - Shopify", "textOnly": "Shopify: The All-in-One Commerce Platform for Businesses - Shopify Skip to Content Why Shopify Back Get started fast You could be selling by tomorrow. Switch to Shopify Get more customers. Make more sales. Trusted by enterprise brands No matter your size, complexity, or ambition. Built into every store World&#x27;s best checkout Proven to convert better. Sidekick Your commerce-obsessed AI assistant. Products Back Build your website Website Builder Themes Domains Customer Accounts Sidekick Sell anywhere Online AI Chats Point of Sale Shop App Social &amp; Marketplaces Global B2B Across Markets Marketing &amp; analytics Advertising &amp; Campaigns Email &amp; Customer Chat Discounts Analytics Run your business Orders &amp; Inventory Shipping Finances &amp; Funding Workflow Automation Mobile Get paid Checkout Payments Taxes Customize &amp; extend Shopify Commerce for Agents Build with our agent tools Shopify App Store Largest commerce ecosystem Shopify.dev Dev docs, CLI, and more Non-stop innovation Shopify Editions 150+ updates to Shopify, twice a year. Latest updates Agentic Storefronts Campaign Autopilot Shopify AI Toolkit for devs Pricing Enterprise Spring &#x27;26 Edition Start for free Log in Log in Start for free Be the next AI all-star Be the next store they line up for AI all-star household name solo-preneur category creator global empire store they line up for big thing Dream big and build fast on Shopify. The world\u2019s best commerce platform. Start for free Sell everywhere people shop. Online and in person. Across AI and on social. Locally and globally. stevemadden.com ornotbike.com glossier.com houseplant.com kotn.com eastfork.com monos.com stanley1913.com shopyowie.com ftcsf.com onlyny.com jp.bonaventura.shop Your brand has entered the chat Your products get discovered across AI channels. Shoppers check out right in the chat. You keep track of every order. All powered by Agentic Storefronts . Sell more in more places Get a stunning store that's built to sell. Design fast with AI. Pick a prebuilt theme. Or go totally custom. Sell on every channel Put your products where shoppers search, shop, and scroll with multichannel integration . Sell face to face Sell in person and keep online and in-store sales in sync with Shopify POS . Sell to 250M+ shoppers with Shop Automatically show up on the Shop app . Reach millions of pre-verified shoppers. Grow around the world Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Buy now \ud83c\uddfa\ud83c\uddf8 Order for $125.00 Sell across borders Take the complexity out of international selling with Shopify Markets . Deliver products faster and more affordably with built-in shipping tools . For anyone from entrepreneurs to enterprise Get started fast Jackie Prince launched Guests on Earth out of her home. Now it's a $4M+ business. Grow as big as you want Our Place grew from a one-product shop into a cookware empire. Raise the bar Iconic toymaker Mattel sells direct to shoppers all around the world. All powered by Shopify. Pick a plan that fits Meet your secret weapon, Sidekick Your very own commerce AI Spot opportunities for growth and automate tedious tasks with Sidekick . Built right into your Shopify admin. What winning looks like Hear how brands use Sidekick to build faster and smarter. Customize everything with apps The Shopify App Store has 21,000+ commerce apps for whatever specialized features your business might need. Hyperdriven by AI. Commerce to the core. Shopify's Universal Commerce Protocol and APIs, tools, and primitives give devs the power to build agentic commerce experiences that businesses are looking for. Hydrogen: Shopify\u2019s headless commerce toolkit Build custom storefronts Build for AI Build apps shopify.dev There\u2019s no better place for you to build The world\u2019s best-converting checkout Higher conversions 15 % High-intent shoppers 250M + Shopify Checkout with Shop Pay converts up to 5", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "squarespace.com", "fetchedAt": "2026-07-30T17:19:34.561929Z", "result": {"base": "https://squarespace.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (2086 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "5 valid JSON-LD block(s) -- types: WebSite, Organization, LocalBusiness, HowTo, FAQPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Website Builder \u2013 Easily Create Your Own Website &mdash; Squ\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.squarespace.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "27332 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://squarespace.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 87, "title": "Website Builder \u2013 Easily Create Your Own Website &mdash; Squarespace", "textOnly": "Website Builder \u2013 Easily Create Your Own Website &mdash; Squarespace Skip to Main Content \u2192 Products Solutions Resources Pricing Preise Log In undefined undefined Account Dashboard Domains Log Out Get started \u2192 Start a Free Trial \u2192 Products Website Websites \u2192 Website Templates \u2192 AI Website Builder \u2192 Design Intelligence \u2192 Ecommerce \u2192 Portfolios \u2192 Blogs \u2192 Analytics \u2192 Business Services \u2192 Online Stores \u2192 Invoicing \u2192 Scheduling \u2192 Limited Releases \u2192 Content &amp; Memberships \u2192 Donations \u2192 Financial Solutions \u2192 Marketing Marketing Tools \u2192 Email Campaigns \u2192 SEO Tools \u2192 AI Visibility \u2192 Free Tools \u2192 Business Tools Domain Search \u2192 Domain Transfer \u2192 Business Email \u2192 View All Features \u2192 For scaling businesses Squarespace Premium Our most advanced features, lowest processing rates, and priority support For Freelancers and Agencies Squarespace for Pros Powerful enough for pros, easy enough for clients Circle The partner program for freelancers and agencies Solutions Solutions for Creative Services Creative Services Professional Services Professional Services Education &amp; Training Education &amp; Training Beauty Beauty Sports &amp; Fitness Sports &amp; Fitness Health &amp; Wellness Health &amp; Wellness Home Services Home Services Events &amp; Experiences Events &amp; Experiences Charities &amp; Nonprofits Charities &amp; Nonprofits Personal Personal Explore Explore Creative Services \u2192 \u2192 Turn your passion projects into paid projects. Discover tools to manage clients and grow your creative business. Photographers Graphic Designers Artists Interior Designers Architects Fashion &amp; Apparel Explore Explore Professional Services \u2192 \u2192 One platform to manage and grow your business, so you can deliver a professional experience clients will love. Consultants Professional Coaches Real Estate Agents Legal Services Marketing Agencies Tech &amp; IT Services Explore Explore Education &amp; Training \u2192 \u2192 Where learning meets earning. Explore a smarter platform to engage students and grow your education business. Exam Prep Tutors Music Teachers Art Instruction Dance Schools Language Schools Vocational Training Explore Explore Beauty \u2192 \u2192 Keep your calendar booked and your clients happy with a single platform built to elevate your beauty business. Hair Salons Barbershops Nail Studios Tattoos &amp; Piercings Makeup Artists Jewelry Makers Explore Explore Sports &amp; Fitness \u2192 \u2192 Keep your classes packed and your clients motivated, all on a single platform to grow your fitness community. Sports Coaches Sporting Goods Fitness Centers Sports Camps Personal Trainers Yoga Studios Explore Explore Health &amp; Wellness \u2192 \u2192 One platform to help grow your practice so you can focus on supporting your clients. Massage Therapists Acupuncturists Wellness Retreats Recovery Centers Nutritionists Medspas Explore Explore Home Services \u2192 \u2192 Book more jobs and get paid on time with a single platform to support your growing services business. Contractors Home Decor Landscapers Cleaning Services Pet Care Auto Repair Explore Explore Events &amp; Experiences \u2192 \u2192 Tools to deliver great experiences, organize your operations, and keep guests coming back for more. Travel Agencies Camps Activities Festivals Concerts &amp; Shows Restaurants &amp; Bars Explore Explore Charities &amp; Nonprofits \u2192 \u2192 Inspire action, raise funds, and grow your mission with tools built to support and scale your nonprofit organization. Faith Communities Philanthropy &amp; Advocacy Community Groups Environment &amp; Conservation Arts &amp; Culture Youth &amp; Education Explore Explore Personal \u2192 \u2192 Share your story, showcase your work, and create a personal website that grows with you. Resum\u00e9s &amp; CVs Portfolios Weddings Freelancers Creators Podcasts Resources 24/7 support Help Center \u2192 In-depth guides and videos about the platform, our services, and how to get started. Forum \u2192 An online community for Squarespace users to discuss best practices and seek advice. Webinars \u2192 Free online sessions where you\u2019ll learn", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended, Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler, DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent, MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot, AdsBot-Google, AdsBot-Google-Mobile, AdsBot-Google-Mobile-Apps, *\" has no root Disallow."}], "contactMatch": "https://squarespace.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "mailchimp.com", "fetchedAt": "2026-07-30T17:19:40.375295Z", "result": {"base": "https://mailchimp.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (524768 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Service.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Email &amp; SMS Marketing Platform | Mailchimp\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "23157 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:+18003155939).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 88, "title": "Email &amp; SMS Marketing Platform | Mailchimp", "textOnly": "Email &amp; SMS Marketing Platform | Mailchimp Skip to main content Join Mailchimp with a risk-free 14-day trial. Start Today \u2014 no credit card required. Industries and Solutions Industries and Solutions See what\u2019s new Industry Restaurants Entertainment + Leisure Non-profit Ecommerce Small Business Professional Services Mid Market Solutions Email marketing SMS marketing AI marketing tools Marketing automations Content creation tools Social media marketing Reporting and analytics Lead generation platform Templates All audience tools See all features and solutions Help Center Case Studies Events Professional Services Hire an Expert Personalized onboarding Customer success Integrations Integrations See 300+ Integrations Most Popular Most Popular Shopify WooCommerce Canva Zapier Square Wix Squarespace Stripe Salesforce LinkedIn Wordpress Facebook Your Tech Stack E-commerce Analytics Booking &amp; Scheduling Loyalty Subscription management Customer service Forms &amp; Surveys Developer tools Content For Developers Getting started Developer guides API docs Webhooks Resources Resources See all resources Learn with Mailchimp E-commerce Digital content Marketing automations Audience management Websites Email marketing Social media Intuit QuickBooks For Developers Marketing API Transactional API Release notes Transactional email Help Center Case Studies Events Professional Services Hire an Expert Personalized onboarding Customer success Switch to Mailchimp Pricing Search This page is now available in other languages. EN EN English ES Espa\u00f1ol FR Fran\u00e7ais BR Portugu\u00eas DE Deutsch IT Italiano Get a demo: +1 (800) 315-5939 Start Free Trial Log In Start Free Trial Hi, %s Account Audience Campaigns Account Log Out Mailchimp Home Main Menu Close Main Menu Main Menu Industries and Solutions Back Close Main Menu Industries and Solutions Industry Back Close Main Menu Industry Restaurants Entertainment + Leisure Non-profit Ecommerce Small Business Professional Services Mid Market Solutions Back Close Main Menu Solutions Email marketing SMS marketing AI marketing tools Marketing automations Content creation tools Social media marketing Reporting and analytics Lead generation platform Templates All audience tools See all features and solutions See what\u2019s new Help Center Case Studies Events Professional Services Back Close Main Menu Professional Services Hire an Expert Personalized onboarding Customer success Integrations Back Close Main Menu Integrations Most Popular Back Close Main Menu Most Popular Shopify WooCommerce Canva Zapier Square Wix Squarespace Stripe Salesforce LinkedIn Wordpress Facebook Your Tech Stack Back Close Main Menu Your Tech Stack E-commerce Analytics Booking &amp; Scheduling Loyalty Subscription management Customer service Forms &amp; Surveys Developer tools Content See 300+ Integrations For Developers Back Close Main Menu For Developers Getting started Developer guides API docs Webhooks Resources Back Close Main Menu Resources Learn with Mailchimp Back Close Main Menu Learn with Mailchimp E-commerce Digital content Marketing automations Audience management Websites Email marketing Social media Intuit QuickBooks For Developers Back Close Main Menu For Developers Marketing API Transactional API Release notes Transactional email See all resources Help Center Case Studies Events Professional Services Back Close Main Menu Professional Services Hire an Expert Personalized onboarding Customer success Switch to Mailchimp Pricing Search This page is now available in other languages. English EN English ES Espa\u00f1ol FR Fran\u00e7ais BR Portugu\u00eas DE Deutsch IT Italiano Request a demo: +1 (800) 315-5939 Hi, %s Back Close Main Menu Account Audience Campaigns Account Log Out Mailchimp Home Log In Start Free Trial Email & SMS marketing minus the learning curve Drive revenue with intuitive marketing analytics, automations, and AI tools. Start Free Trial Mailchimp has a four and half star rating *Based on 33,000+ reviews across EMAIL &amp; SMS FOR ECOM", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:+18003155939", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "canva.com", "fetchedAt": "2026-07-30T17:19:45.340063Z", "result": {"base": "https://canva.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://canva.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-saas", "domain": "zapier.com", "fetchedAt": "2026-07-30T17:19:51.366632Z", "result": {"base": "https://zapier.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (12288 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: Organization, WebSite, WebPage, SoftwareApplication.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Zapier: Automate AI Workflows, Agents, and Apps\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://zapier.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "20189 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://zapier.com/l/contact-sales?demo_source=cs_nav_header).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 95, "title": "Zapier: Automate AI Workflows, Agents, and Apps", "textOnly": "Zapier: Automate AI Workflows, Agents, and Apps Skip to content Products Zapier Automation Platform No-code automation across 9,000+ apps Most Popular Zap workflows Do-it-yourself automation for workflows Zapier MCP Add Zapier to your AI chat Zapier SDK Install Zapier to your AI coding agent Tables Store data your Zap workflows can read and update Capabilities App integrations Explore 9,000+ app connections AI automation \ud83e\ude84 Cutting-edge AI to upgrade your workflows Security Enterprise-grade security Explore templates Explore use cases Join Zapier Early Access More Products Forms Capture inputs that trigger workflows Canvas Plan and map your workflows with AI Agents Create your own AI assistants for any task Chatbots Answer customer questions with AI chatbots Solutions Solutions How Zapier can help you automate your work across teams By team RevOps Drive revenue through automation Marketing Multiply campaign effectiveness and ROI IT Better manage systems with automation HR Empower your HR team with AI automation Sales Close more deals Customer Support Elevate customer satisfaction Leaders Streamline decision-making processes Executive Assistants Eliminate repetitive admin tasks By app NetSuite Salesforce HubSpot Slack ChatGPT (OpenAI) Microsoft Dynamics CRM Microsoft Teams Zendesk Jira Software Cloud View all apps By use case Lead management Automate your way to more conversions Sales pipeline Automate handoffs and streamline sales Marketing campaigns Boost your marketing campaign&#x27;s impact Customer support Quickly route tickets and responses Data management Connect app data for more insights Project management Deliver projects seamlessly Tickets and incidents Resolve issues and incidents faster Explore templates Explore use cases Join Zapier Early Access Automation you can govern See how Zapier helps you manage, secure, and scale automation across your organization. Zapier for Startups Small and medium businesses Enterprise Resources Resources Educational resources to move your work forward Get inspired 5 things to automate today Popular workflows to save time Create custom chatbots Build an AI chatbot in minutes Guides Go deep with dedicated guidance Webinars Join live or watch on demand Customer stories See how businesses grow with Zapier Blog Read all about apps and automation How to use Zapier Zapier quick-start guide Automate with confidence Webhooks and Zapier Connect your favorite apps with Zapier\u2019s webhooks Help Center Find answers to your questions Zapier Support Get help and find support options Hire a Zapier Solution Partner Find a consultant to help you do more AI &amp; Zapier AI orchestration at work Your guide to intelligent systems Zapier Copilot A personalized automation assistant Zapier&#x27;s AI tools An AI tool for every need How to use Zapier MCP Connect AI to thousands of apps\u2014code-free Zapier Agents Create agents to handle work for you Explore templates Explore use cases Join Zapier Early Access Implement AI orchestration Three phases to move from disconnected AI pilots to orchestrated systems that scale. Developer resources Powered by Zapier Build an integration Embed an integration Integration Partner Program Documentation Enterprise Pricing EN Deutsch - DE English - EN Espa\u00f1ol - ES Fran\u00e7ais - FR \u65e5\u672c\u8a9e - JA GitHub Explore apps Contact sales Products Most Popular Zap workflows Zapier MCP Zapier SDK Tables Capabilities App integrations AI automation \ud83e\ude84 Security More Products Forms Canvas Agents Chatbots Solutions By team RevOps Marketing IT HR Sales Customer Support Leaders Executive Assistants By app NetSuite Salesforce HubSpot Slack ChatGPT (OpenAI) Microsoft Dynamics CRM Microsoft Teams Zendesk Jira Software Cloud View all apps By use case Lead management Sales pipeline Marketing campaigns Customer support Data management Project management Tickets and incidents Zapier for Startups Small and medium businesses Enterprise Resources Get inspired 5 things to automate today Create custom chatbots Guides W", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, DuckAssistBot\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://zapier.com/l/contact-sales?demo_source=cs_nav_header", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "calendly.com", "fetchedAt": "2026-07-30T17:19:56.881841Z", "result": {"base": "https://calendly.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (15052 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Blocks training crawlers only (CCBot) -- a legitimate opt-out; retrieval/citation access stays open.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Free Online Appointment Scheduling Software | Calendly\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://calendly.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6651 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://calendly.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 85, "title": "Free Online Appointment Scheduling Software | Calendly", "textOnly": "Free Online Appointment Scheduling Software | Calendly Product Solutions Resources Pricing Talk to sales Log In Get started for free Get started Talk to sales Log In Get started for free Get started Product Scheduling Simplified booking Notetaker Meeting recaps and action items Contacts Relationship management tools Payments Flexible ways to get paid Product overview Why choose Calendly Platform Integrations Mobile app Browser extension Admin controls Security By business size Individuals For solopreneurs Small business For growing businesses Large companies For enterprise By team By industry Sales Marketing Customer success Recruiting Education Technology Financial Services Professional Services Get started Discover Support Learning hub Structured paths and guides Developer docs Build with Calendly API Product tour See Calendly in action Blog Tips and best practices Ebooks &amp; webinars Expert insights About us Discover our mission What&#x27;s new Product updates Customer stories Calendly success stories Newsroom Company news and press Help center Search articles and FAQs Community Connect with other users Contact us Connect with support Trusted by more than 100,000 of the world\u2019s leading organizations Calendly makes scheduling simple Calendly\u2019s easy enough for individual users, and powerful enough to meet the needs of enterprise organizations \u2014 including 86% of the Fortune 500 companies. Sign up for free Connect your calendars Calendly connects up to six calendars to automate scheduling with real-time availability. Availability Connect existing calendar Google Google calendars Microsoft Outlook calendars Exchange calendars Add your availability Keep invitees informed of your availability. Take control of your calendar with detailed availability settings, scheduling rules, buffers, and more. Availability Meeting hours Weekly hours Set when you are typically available for meetings S Unavailable M 9:00 am - 4:30 pm T Unavailable W 9:30 am - 5:00 pm Th 10:00 am - 6:00 pm F 10:00 am - 3:00 pm S Unavailable Timezone: Eastern time - US &amp; Canada Connect conferencing tools Sync your video conferencing tools and set preferences for in-person meetings or calls. Meeting location Video conferencing Zoom Microsoft Teams Google Meet Webex Customize your event types Choose from pre-built templates or quickly create custom event types for any meeting you need to schedule. Event types Ways to meet One-on-one 1 host 1 invitee Good for 1:1 check-ins, screening calls, etc. Collective Multiple hosts 1 invitee Co-host meetings, group interviews, etc. Round robin Rotating hosts 1 invitee Distribute between team members Group 1 host Multiple invitees Share your scheduling link Easily book meetings by embedding scheduling links on your website, landing pages, or emails. Share Offer time slots via email Send Event types One-off meeting Meeting poll Recently shared event types Client onboarding Client check-in 30 min One-on-one Insert link Offer time slots Quarterly business review View all event types Availability Connect existing calendar Google Google calendars Microsoft Outlook calendars Exchange calendars Availability Meeting hours Weekly hours Set when you are typically available for meetings S Unavailable M 9:00 am - 4:30 pm T Unavailable W 9:30 am - 5:00 pm Th 10:00 am - 6:00 pm F 10:00 am - 3:00 pm S Unavailable Timezone: Eastern time - US &amp; Canada Meeting location Video conferencing Zoom Microsoft Teams Google Meet Webex Event types Ways to meet One-on-one 1 host 1 invitee Good for 1:1 check-ins, screening calls, etc. Collective Multiple hosts 1 invitee Co-host meetings, group interviews, etc. Round robin Rotating hosts 1 invitee Distribute between team members Group 1 host Multiple invitees Share Offer time slots via email Send Event types One-off meeting Meeting poll Recently shared event types Client onboarding Client check-in 30 min One-on-one Insert link Offer time slots Quarterly business review View all event types Connect Cal", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://calendly.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "webflow.com", "fetchedAt": "2026-07-30T17:20:02.234258Z", "result": {"base": "https://webflow.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (13258 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, SoftwareApplication, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Webflow: The agentic web platform for modern businesses\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "26408 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://webflow.com/enterprise/contact-sales).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 76, "title": "Webflow: The agentic web platform for modern businesses", "textOnly": "Webflow: The agentic web platform for modern businesses Introducing MCP 2.0: Build, govern, and analyze your sites inside your existing AI workflows. Read the announcement \u2192 Skip to Main Content Webflow Platform \u2192 Platform \u2190 Back Build Design Edit content Interactions Collaboration Page building Manage CMS Hosting Localize Security Shared Libraries Optimize Analyze Optimize SEO AEO new Extend Apps Figma to Webflow Webflow Cloud DevLink MCP new Platform Overview Explore the power of the Webflow platform Learn more about Platform Webflow AI Elevate web experiences with the power of AI Learn more about Ai I&#x27;m a Marketer Marketer Designer Developer Agency Appearance Light mode Dark mode Auto (OS) Solutions \u2192 Solutions \u2190 Back Marketing Brand Performance Digital Experience Ops Engineering Engineering leaders Developers Service Providers Agencies Freelancers Global Alliances Enterprise Advanced collaboration workflows and enterprise-grade scalability. Learn more about Enterprise AEO Show up in AI-driven search with the power of Webflow. Learn more about AEO I&#x27;m a Marketer Marketer Designer Developer Agency Appearance Light mode Dark mode Auto (OS) Resources \u2192 Resources \u2190 Back Get started Templates Made in Webflow Hire a Certified Partner Developers Learn Webflow University Blog Webinars &amp; ebooks The Webflow Way Discover Customer stories Our Community Partner with Webflow AEO Assessment Get help Support Join the Community I&#x27;m a Marketer Marketer Designer Developer Agency Appearance Light mode Dark mode Auto (OS) Enterprise Pricing Site Settings Site Settings \u2190 Back I&#x27;m a Marketer Marketer Designer Developer Agency Appearance Light mode Dark mode Auto (OS) I&#x27;m a Marketer Marketer Designer Developer Agency Appearance Light mode Dark mode Auto (OS) Get started Login Login Log in Contact Sales Start for free Trusted by teams at Make your website a growth engine Make websites that drive results Build your brand. Rank in AI search. Drive real revenue. All with Webflow. Do your best work with the AI-native platform for creating and optimizing web experiences. How do you want to build? Get started Get started Talk to sales Talk to sales Enterprise Advanced collaboration, security, and scale for teams. \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Webflow AEO New Own your site\u2019s visibility in AI search, right within Webflow. \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Webflow MCP 2.0 Connect your agents to Webflow. \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Start building for free. Upgrade any time. Get started \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Get started Webflow MCP 2.0 Connect your agents to Webflow \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Template Start with a design that fits your needs. \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 From scratch Build your way on a blank canvas. \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Building with a team at scale? Talk to sales \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Talk to sales 10x In cost savings annually Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 67% decrease in dev ticketing Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 $6M in cost savings annually Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 56% increase in form fills Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 1,170% increase in traffic YoY Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 $200M in new pipeline generated post-site launch Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 +20% increase in site-wide conversion Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 10x In cost savings annually Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 67% decrease in dev ticketing Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 $6M in cost savings annually Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 56% increase in form fills Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 1,170% increase in traffic YoY Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 $200M in new pipeline generated post-site launch Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 +20% increase in site-wide conversion Read story \u2192 \u2197 \u25ba \u2193 \u2190 \u2191 Play Pause \u201c @type \u201d: \u201cWebPageElement\u201d, \u201c id \u201d: &quot;webflow.com/#platform&quot;, \u201c name \u201d: &quot;Webflow is the agentic web marketing platform for high-performing brands&quot; \u201c @type \u201d: \u201cItemList\u201d, \u201c name \u201d: &quot;Personalization&quot;, &quot;audience&quot;:[ \u201c @audienceType \u201d: \u201cMarketer\u201d \u201c @type \u201d: \u201cListItem\u201d { &quot; name &quot;: &quot;Build&quot;, \u201c description \u201d: \u201cBring team members and AI agents into one shared workspace, with design systems to keep ev", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://webflow.com/enterprise/contact-sales", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "ghost.org", "fetchedAt": "2026-07-30T17:20:07.191319Z", "result": {"base": "https://ghost.org", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Ghost: The best open source blog & newsletter platform\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6188 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://ghost.org/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 68, "title": "Ghost: The best open source blog & newsletter platform", "textOnly": "Ghost: The best open source blog & newsletter platform Open menu Product For Creators YouTubers, bloggers, podcasters, musicians & artists For Publishers Writers, journalists, local news and new media outlets For Business Modern brands & companies with ambitious content marketing For Developers Source code, documentation, guides and tutorials Explore Resources Marketplace Professional themes, custom integrations and qualified experts Start here New A huge library of guides, stories, interviews and tips for success Themes Hundreds of beautifully designed publication templates Help center Get help with product features and answers to common questions. module-four Integrations Connect thousands of apps and services with your website Product updates All the latest changes and improvements to Ghost Experts Get help building your site from certified Ghost developers About us Learn more about the people behind the platform (We're hiring!) Pricing Sign in Get Started \u2014 free Close menu For Creators Join thousands of creators using Ghost to share their work For Publishers The platform for independent publishers For Business Content marketing for modern business For Developers Source code, documentation, guides and tutorials Explore Pricing Marketplace Themes Integrations Resources Help center Product updates Developers About us Get Started \u2014 free Existing customer? Sign in Just launched: Ghost 6.0 &rarr; Turn your audience into a business . Ghost is a powerful app for professional publishers to create, share, and grow a business around their content. It comes with modern tools to build a website, publish content, send newsletters & offer paid subscriptions to members. Try Ghost completely free for 14 days &rarr; Easy site design Complete control over your website and branding. Launch a custom website and tweak the design settings to perfectly match your brand and style. Go even further with hundreds of custom themes in our marketplace, or build your own completely custom design from scratch. Your unique brand sits front and center, while Ghost runs things in the background. Explore Ghost themes &rarr; Advanced creator tools Publish by web & email newsletter. An editor built from the ground-up for professionals. Calm by design, with advanced workflows by default. No more suffering through clumsy toolbars or drag & oops. Immerse yourself in the story with an interface that's invisible until you need it, and powerful when you do. Rich media & dynamic cards. Modern publishing requires more than just words. Expand your story with image galleries, gifs, video, audio, products, info boxes, accordion toggles, downloadable files, bookmarks, and so much more. Newsletters built-in. Deliver posts by email newsletter to your audience, so they'll be in the loop whenever something new goes live. Segment your audience and send multiple different newsletters based on preference. Grow your audiences Transform those clicks into contacts. Until now, building an indie publication with memberships and subscriptions has been difficult and complicated. Ghost makes it easy, with native signup forms that turn anonymous views into logged-in members. Allow people to sign up for free, or purchase a paid subscription to support your work across monthly and yearly premium tiers. Hello, new friends. Run your business Manage your memberships. See who's signing up, who's paying, and what they're reading, so you can get a better understanding of your core audience. Native analytics. Understand what's working. Detailed engagement analytics for your audience so you can understand what content is getting the most attention, and who your biggest fans are. Offers & promotions. Entice new subscribers with offers and promotions to grow your business. Run a 30% discount for your first 100 subscribers, an 80%-off Black Friday sale, or a special promotion for a live event. GeckoBoard Discourse Imgur MailChimp Pinterest HelpScout Buffer Unsplash AirTable Vimeo Discord Spotify Ulyss", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://ghost.org/contact/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "substack.com", "fetchedAt": "2026-07-30T17:20:12.011663Z", "result": {"base": "https://substack.com", "fatal": "Their server refused or timed out (status 429). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://substack.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-saas", "domain": "gumroad.com", "fetchedAt": "2026-07-30T17:20:17.247839Z", "result": {"base": "https://gumroad.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Earn your first dollar online with Gumroad\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://gumroad.com/about).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "4212 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 49, "title": "Earn your first dollar online with Gumroad", "textOnly": "Earn your first dollar online with Gumroad 9.4K Discover Blog Pricing Features About Gumclaw Log in Start selling Discover Blog Pricing Features About Gumclaw Log in Start selling Go from 0 to $1 Anyone can earn their first dollar online. Just start with what you know, see what sticks, and get paid. It's that easy. div]:block!\"> Start selling Search marketplace Contribute or fork on GitHub Sell anything Video lessons. Monthly subscriptions. Whatever! Gumroad was created to help you experiment with all kinds of ideas and formats. Make your own road Whether you need more balance, flexibility, or just a different gig, we make it easy to chart a new path. Sell to anyone Go from 0 to $1 and automated workflows. Let your customers pay in their own currency. Choose between one-time, recurring, or fixed-length payments in your currency of choice. Sell anywhere Create and customize your storefront with our all-in-one platform or choose to use your personal site instead. Seamlessly connect your Gumroad account to thousands of apps in your current stack. Instead of building a company... ...start selling a side project! You know all those great ideas you have? The Gumroad Way Start Small Get Better Together Learn Quickly The Gumroad Way Start Small Get Better Together Learn Quickly We want you to try them, lots of them, and find out what works. You don't have to be a tech expert or even understand how to start a business. You just gotta take what you know and sell it. Find out how $1,924,054 The amount of income earned by Gumroad digital entrepreneurs last week. I launched MaxPacks as an experimental side gig; but within 2 years those Procreate brushes were earning more than my 6-figure salary in CG. Leaving in favor of Gumroad enabled me to explore other aspects of my art, develop new hobbies, and finally prioritize my personal life. Max Ulichney Sells Procreate brush packs For years, I had a goal to develop &#39;passive&#39; income streams, but struggled to make that a reality. Last year, I started selling informational products on Gumroad and since then have made $10k+ per month building products that I love. Steph Smith Sells content tutorials Originally, I took pre-orders for my Trend Reports on Gumroad. But I received... exactly $0. So I changed tactics: I made half of my report free, and the other half paid. Today, 99% of Trends.VC revenue is recurring in the form of annual and quarterly subscriptions. trendsvc Sells business insights and expertise I love Gumroad because it can&#39;t be any simpler. I upload a file, set a price, and I can start selling on the internet. The money I make from my sales lands directly in my bank account every Friday. Daniel Vassallo Sells business insights and expertise Unlimited possibilities Discover the best-selling products and creators on Gumroad blender meditation comic notion template textures procreate 3d model hypnosis manga investing mockup brushes spark ar subliminal messages anime instagram font blender meditation comic notion template textures procreate 3d model hypnosis manga investing mockup brushes spark ar subliminal messages anime instagram font art after effects education fitness sci-fi vrchat ableton certification exams vj loops workout program poetry avatar sample pack learning luts yoga fiction assets sheet music art after effects education fitness sci-fi vrchat ableton certification exams vj loops workout program poetry avatar sample pack learning luts yoga fiction assets sheet music reference photos coloring page singles programming kdp interior stock photos printable jazz windows ebook photobash productivity instrumental music theme low content books reference photos coloring page singles programming kdp interior stock photos printable jazz windows ebook photobash productivity instrumental music theme low content books Don't take risks. That's scary! Instead of selling a book... ...start by selling a blog post! Place small bets. That's exciting! Share your work. Someone out the", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": true, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "posture-choice", "detail": "Served through Cloudflare with no ad code detected, currently readable by AI. The September 15 default targets ad-bearing pages, so no automatic flip is expected here. But this site\u2019s AI access is now a Cloudflare dashboard setting \u2014 block, allow, or charge under Pay Per Use, where publishers get paid when content is actually used in an AI answer. Confirm the current posture is a decision, not a default."}}}
{"corpus": "sept15-saas", "domain": "teachable.com", "fetchedAt": "2026-07-30T17:20:22.376607Z", "result": {"base": "https://teachable.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (1777 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: SoftwareApplication, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Teachable \u2014 Build &amp; Sell Online Courses, Coaching, Membe\u2026\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12409 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 71, "title": "Teachable \u2014 Build &amp; Sell Online Courses, Coaching, Memberships &amp; More", "textOnly": "Teachable \u2014 Build &amp; Sell Online Courses, Coaching, Memberships &amp; More Products What you can sell Online Courses Digital Downloads Memberships Coaching Feature highlights AI Features Payments Site Builder Sales and Marketing Student Experience Mobile App Integrations Analytics and Reporting Get 3 months for the price of 1 Explore offer Introducing Teachable Payments Introducing Teachable Payments Capture more revenue with lower fees, global payments, and automatic tax handling. Learn more Resources Resources Product demo Example Schools Success Stories Blog Events Newsletter and Podcast Customer Journey Guides &amp; How-To&#x27;s Solution Enterprise Support Help center Hire an expert Get 3 months for the price of 1 Explore offer Introducing Teachable Payments Introducing Teachable Payments Capture more revenue with lower fees, global payments, and automatic tax handling. Learn more Pricing Contact sales Start for free Creator login Creator login Contact sales Start for free Products What you can sell Online Courses Digital Downloads Memberships Coaching Feature highlights AI Features Payments Site Builder Sales and Marketing Student Experience Mobile App Integrations Analytics and Reporting Get 3 months for the price of 1 Explore offer Introducing Teachable Payments Introducing Teachable Payments Capture more revenue with lower fees, global payments, and automatic tax handling. Learn more Resources Resources Product demo Example Schools Success Stories Blog Events Newsletter and Podcast Customer Journey Guides &amp; How-To&#x27;s Solution Enterprise Support Help center Hire an expert Get 3 months for the price of 1 Explore offer Introducing Teachable Payments Introducing Teachable Payments Capture more revenue with lower fees, global payments, and automatic tax handling. Learn more Pricing Contact sales Start for free Creator login Creator login Contact sales Start for free Introducing Teachable Payments Introducing Teachable Payments Capture more revenue with lower fees, global payments, and automatic tax handling. See what&#x27;s new The future of your educ ation business happens here Whatever your expertise\u2014languages, fitness, or coding\u2014Teachable helps you deliver impactful learning, run a smarter business, and scale worldwide. Start for free Explore our plans Your view Easily create your products, manage your business, and customize your students&#x27; learning experience in one place. Your student&#x27;s view Intuitive, mobile-friendly, and built for your students to seamlessly access and discover your products and offers. Explore demo $10B+ earned by Teachable and Hotmart creators 100M+ students served on Teachable 180 countries using Teachable Teach in every way your students want to learn Our products give students more choice in how they engage\u2014live or on-demand, video or text, at home or on-the-go\u2014while giving your business more sustainable ways to grow. Online Courses Maximize student engagement with videos, quizzes, drip content, and mobile access. Coaching Offer one-on-one or group sessions, with easy scheduling and payment management. Digital Downloads Sell templates, guides, and resources your audience can instantly access and use. Memberships Build recurring revenue and stronger community with flexible subscription models. Start for free Product Update The checkout you&#x27;ve been waiting for \u2192 Sell globally with 35+ local payment methods and checkout experiences. \u2192 Automate taxes with sales tax, VAT, and GST handled across 45+ countries. \u2192 Capture more revenue with lower fees and built-in subscription recovery. Discover what\u2019s new Why 150,000 industry leaders choose Teachable Student success drives business success\u2014and you need a platform designed for both. More time for what matters Made for the modern learner Business growth that lasts Business growth that lasts Made to adapt and evolve Made to adapt and evolve Made for the modern learner Made for the modern learner You\u2019re serious about education. We", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "gusto.com", "fetchedAt": "2026-07-30T17:20:27.194197Z", "result": {"base": "https://gusto.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://gusto.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-saas", "domain": "monday.com", "fetchedAt": "2026-07-30T17:20:32.602460Z", "result": {"base": "https://monday.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (9901 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, SoftwareApplication, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"The AI Work Platform for People &amp; Agents | monday.com\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "15899 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://monday.com/helpcenter/contact-support).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 76, "title": "The AI Work Platform for People &amp; Agents | monday.com", "textOnly": "The AI Work Platform for People &amp; Agents | monday.com AI platform ONE PLATFORM FOR ANY USE CASE Built for teams and agents working together PMO Projects &amp; delivery Marketing Campaigns &amp; creative Operations Processes &amp; efficiency IT Tickets &amp; support HR Hiring &amp; onboarding Sales Pipeline &amp; deals Legal Contracts &amp; compliance View all use cases The platform Platform overview Dashboards &amp; reporting Integrations Automations monday MCP Our infrastructure Products monday work management For projects &amp; tasks monday CRM For customer-facing teams monday dev For product &amp; dev teams monday campaigns For email marketing teams monday service For IT &amp; support PLATFORM capabilities AI agents Agent builder AI assistant Flexible workspace Vibe coding Dashboards Docs Forms Connectivity &amp; tools Integrations Automations Our infra MCP Solutions Company size Enterprise Gain visibility and control at scale SMB Run your business in one place Business applications monday CRM For customer-facing teams monday service For IT &amp; support monday dev For product &amp; dev teams Industries Retail Media Government Resources Learn About us What&#x27;s new Blog Get certified Customer stories AI Trust Center Support Help center 24/7 support Partners Build AI template center Solution templates App development Enterprise Pricing Log in Contact sales Get Started AI platform Solutions Resources AI platform ONE PLATFORM FOR ANY USE CASE PMO Projects &amp; delivery Marketing Campaigns &amp; creative Operations Processes &amp; efficiency IT Tickets &amp; support HR Hiring &amp; onboarding Sales Pipeline &amp; deals View all use cases AI agents Agent builder AI assistant Flexible workspace Vibe coding Dashboards Docs Forms Connectivity &amp; tools Integrations Automations Our infra MCP Solutions Company size Enterprise SMB Business applications monday CRM monday service monday dev Industries Retail Media Government Resources Learn About us What&#x27;s new Blog Get certified Customer stories AI Trust Center Support Help center 24/7 support Partners Build AI template center Solution templates App development Enterprise Pricing Contact sales Log in Go to homepage You lead. Agents act. Where people and agents drive results together on one secure AI work platform Marketing Operations IT Product Sales HR PMO Get Started No credit card needed \u2726 Unlimited time on Free plan Your browser does not support the video tag. Please research vendors for Q1 and flag any contract issues before Thursday. Nick, Operations Manager Triage today&#x27;s tickets and handle any critical incidents right away. Emma, IT Team Lead Update the roadmap and optimize timelines for our Q1 planning session. Sarah, Product Manager Score our open deals and send proposals to anyone above 80%. Sophie, Sales Director Screen this week&#x27;s applicants and book interviews for the top candidates. Mark, HR Manager Monitor this project and let me know if anything is at risk. Jasmine, Project Manager Trusted by over 60% of the Fortune 500 You lead. Agents act. Where people and agents drive results together on one secure work platform Continue with Or Get Started By proceeding, you agree to the Terms of Service and Privacy Policy By proceeding, you agree to the Terms of Service and Privacy Policy Thank you! Your submission has been received! Oops! Something went wrong while submitting the form. Your browser does not support the video tag. Trusted by over 60% of the Fortune 500 Get more done with agents Marketing IT Product Sales HR PMO Create any agent Posts, campaigns and ads. Done. Marketing agents write posts, run campaigns, and drive insights while your team can focus on strategy. Get Started Your browser does not support the video tag. Tickets, issues and risks. Resolved. IT agents catch tickets, resolve issues, and escalate risks so your team can own the infrastructure. Get Started Products and new features. Deployed. Product agents turn feedback into features", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"Google-Extended\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"ClaudeBot\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User, OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User, OAI-SearchBot\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"PerplexityBot, Perplexity-User\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://monday.com/helpcenter/contact-support", "sept15": {"behindCloudflare": true, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "posture-choice", "detail": "Served through Cloudflare with no ad code detected, currently readable by AI. The September 15 default targets ad-bearing pages, so no automatic flip is expected here. But this site\u2019s AI access is now a Cloudflare dashboard setting \u2014 block, allow, or charge under Pay Per Use, where publishers get paid when content is actually used in an AI answer. Confirm the current posture is a decision, not a default."}}}
{"corpus": "sept15-saas", "domain": "asana.com", "fetchedAt": "2026-07-30T17:20:42.456393Z", "result": {"base": "https://asana.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (5691 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, BreadcrumbList.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Work &amp; Project Management for Human-Agent Teams \u2022 Asana\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://asana.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9663 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact link present.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "Work &amp; Project Management for Human-Agent Teams \u2022 Asana", "textOnly": "Work &amp; Project Management for Human-Agent Teams \u2022 Asana Asana Home Products chevron-down icon Solutions chevron-down icon Learning &amp; support chevron-down icon Pricing Choose your preferred language Bahasa Indonesia Deutsch English Espa\u00f1ol Fran\u00e7ais Italiano \u65e5\u672c\u8a9e \ud55c\uad6d\uc5b4 Nederlands Polski Portugu\u00eas \u0420\u0443\u0441\u0441\u043a\u0438\u0439 Svenska \u7e41\u9ad4\u4e2d\u6587 Log in Contact sales Get started Launch Asana Upgrade Contact sales chevron-down icon settings icon My tasks Inbox My organization Help Center Language plus icon Add another account Log out Products awm icon Agentic Work Management arrow-right icon For cross-functional teams service icon Asana Service Management arrow-right icon For service teams Coming soon agency icon Asana Client Management arrow-right icon For teams delivering client work Coming soon command icon Command by Asana arrow-right icon For developer teams Coming soon stackai icon StackAI by Asana arrow-right icon For every critical workflow AI Platform ait icon AI Teammates arrow-right icon Ready-to-go AI agents for every team aistudio icon AI Studio arrow-right icon Powerful no-code automations dash icon Asana Dash arrow-right icon Your AI Chief of Staff Asana Foundations Admin and security arrow-right icon App integrations arrow-right icon Developer arrow-right icon Latest feature release arrow-right icon Company Type Enterprise arrow-right icon Small business arrow-right icon Nonprofit arrow-right icon Agencies arrow-right icon Teams Operations arrow-right icon Marketing arrow-right icon IT arrow-right icon Leaders arrow-right icon Industries Government arrow-right icon Healthcare arrow-right icon Retail arrow-right icon Financial services arrow-right icon Education arrow-right icon Manufacturing arrow-right icon Use Cases Goal management arrow-right icon Organizational planning arrow-right icon Project intake arrow-right icon Resource planning arrow-right icon Product launches arrow-right icon View all use cases arrow-right icon LEARN Resource center arrow-right icon Events and webinars arrow-right icon Customer stories arrow-right icon Asana Academy arrow-right icon Certifications arrow-right icon Trainings arrow-right icon SUPPORT Help Center arrow-right icon Community arrow-right icon Templates arrow-right icon SERVICES Customer Success arrow-right icon Find a partner arrow-right icon Become a partner arrow-right icon desktop icon Contact sales message icon View demo download icon Download app Products chevron-down icon Solutions chevron-down icon Learning &amp; support chevron-down icon Pricing desktop icon Contact sales message icon View demo download icon Download app Log in Contact sales Get started Launch Asana Upgrade Contact sales The OS for human-agent teams Supercharge your teams to get things done Get started View demo 85% of Fortune 100 companies choose Asana\u00b9 AI that works the way your team works Asana Work Graph\u00ae chevron-up icon A neural network of everything your company is doing, every person, task, project, goal, and dependency connected, so humans and agents always know who is doing what, by when, and toward which goal. Multiplayer chevron-up icon Every workflow, app, and agent runs in a shared space where humans and agents act on the same plan, see the same context, and get smarter with every interaction. Multiple humans can train, guide, and improve any agent. Shared memory chevron-up icon Shared memory means AI Teammates learn from the work itself. Every completed task, every piece of feedback, every preference gets carried forward, so no one repeats context they&#x27;ve already given and every workflow starts smarter than the last. Enterprise governance chevron-up icon Every agent has an identity, scoped permissions, an audit trail, and cost constraints, with data access, approved actions, and spend all managed from the same console that governs human users. AI Teammates Your team just got bigger Pre-built AI agents that work alongside your team inside real workflows, with shared memory, governance, and context from th", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-saas", "domain": "clickup.com", "fetchedAt": "2026-07-30T17:20:47.967186Z", "result": {"base": "https://clickup.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (8779 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"ClickUp\u2122 | Maximize productivity \u2022 Software, AI, and humans \u2026\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12091 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 76, "title": "ClickUp\u2122 | Maximize productivity \u2022 Software, AI, and humans converge", "textOnly": "ClickUp\u2122 | Maximize productivity \u2022 Software, AI, and humans converge Brain AI Product Solutions Learn Pricing Enterprise Get a Demo Login Sign Up The Best AI is Software to replace all software Save money. All Apps, AI, Projects, Chat + 20 more. Save time. All humans working together with perfect context. Create infinite productivity. AI Agents &amp; Workflows. Get started. It&#x27;s FREE Free forever. No credit card. New ClickUp 4.0 New Agents Projects Brain (AI) Time Tracking Chat Calendar Docs Whiteboards Sprints Automations and more... Get started. It&#x27;s FREE! The Best AI is Software to replace all software Save money. All Apps, AI, Projects, Chat + 20 more. Save time. All humans working together with perfect context. Create infinite productivity. AI Agents &amp; Workflows. Get started. It&#x27;s FREE! Free forever. No credit card. Get 400% more done \u2022 Customize your workspace Projects Chat Brain MAX AI Agents Sprints Time Tracking Calendar Docs Whiteboards Automations Dashboards Scheduling Activate \u2192 New! Introducing Super Agents AI that actually showed up to work. Across every team, every app, and every workflow. Get started. It&#x27;s FREE! Free forever. No credit card. Projects Chat Brain MAX AI Agents Sprints Time Tracking Calendar Docs Whiteboards Automations Dashboards Scheduling Activate \u2192 Project Management Super Agent Trusted by the best Trusted by the best 60% of work is lost in context \u2013 and AI is lost without it Work Sprawl is killing context and destroying productivity. Play Video Play Video All apps, AI Agents, and humans in ClickUp 100+ products to replace fragmented software &amp; maximize human productivity. Dependencies Connected Search Tasks Mind Maps Wikis AI Notetaker Calendar Proofing Portfolios Templates Reminders Reporting Goals Projects Docs Sprints Custom Status AI Writer API Calls Milestones Forms Automations Custom Fields Timesheets AI Q&amp;A Priorities Time Estimates Brain Chat Clips Everything view Single Sign-on Emails Dashboards Time Tracking Kanban Boards Integrations Guests Tags 24/7 Support Checklists Scheduling Spreadsheets Whiteboards Gantt Charts Roadmaps Inbox Teams A new era of humans, with Super Agents Build your own agent Learn More Delegate any task Super Agent Works 24/7 Infinite memory 500+ tool superpowers Delegate any task Infinite memory Build your own agent The best AI is your AI Already plugged into your team, your tools, and your tasks. Available everywhere macOS iOS Windows Android Context Brain\u00b2 sees your tasks, docs, and conversations. It doesn&#x27;t need a brief because it knows what&#x27;s happening in real-time. Intelligence Brain\u00b2 picks the best AI model for each job. One subscription. Every frontier model running with full context. Best models Brain GPT Claude Opus Gemini Personality Brain\u00b2 learns how you and your team talk: your tone, your shorthand, your preferences. tone &quot;direct, no fluff, like a sharp coworker&quot; sprint_methods &quot;story points, fibonacci&quot; reports_to &quot;VP Engineering, weekly on Mondays&quot; prefers &quot;tables over bullet points&quot; projects &quot;Brain 2.0, Platform Migration, Q2&quot; timezone &quot;PST, don&#x27;t schedule before 10am&quot; tools &quot;GitHub, Figma, Linear, Notion&quot; Nothing comes close to Brain 2 We rebuilt Brain from the ground up. Memory &amp; Preferences Brain\u00b2 keeps track of how you like to work, and gets smarter every time you use ClickUp. Memory Updated User Preference: Values clarity Every AI, Unlimited GPT, Claude Opus, Gemini, and more. Every model runs with full knowledge of your work. Gemini Claude ChatGPT Multiplayer AI The more your team uses Brain\u00b2, the more it knows, the more useful it gets for everyone. Strategist Developer Visual Designer Connected Apps &amp; Any MCP Brain\u00b2 taps Google Drive, GitHub, Salesforce, and more to get you answers and insights. MCP Online Ambient Intelligence Brain\u00b2 surfaces relevant context, related tasks, and smart suggestions before you even ask", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "amazon.com", "fetchedAt": "2026-07-30T17:20:58.052697Z", "result": {"base": "https://amazon.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: MISSING - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": false, "detail": "157 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 0, "title": "https://amazon.com", "textOnly": "JavaScript is disabled In order to continue, we need to verify that you're not a robot. This requires JavaScript. Enable JavaScript and then reload the page.", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "walmart.com", "fetchedAt": "2026-07-30T17:21:03.296113Z", "result": {"base": "https://walmart.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: Organization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Walmart | Save Money. Live better.\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.walmart.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "2666 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 70, "title": "Walmart | Save Money. Live better.", "textOnly": "Walmart | Save Money. Live better. Skip to Main Content Pickup or delivery? Cancel Reorder My Items Reorder Lists Registries Sign In Account Sign in or create account Purchase History Walmart+ Subscriptions Language | English $0.00 Departments Services Pickup or delivery? Walmart | Save Money. Live better. Shopping just got easier Try 30 days of Walmart+ for just $1 Claim trial now Claim trial now $35 min. See T&amp;C. 1 trial/mbr. Tech, fashion, beauty &amp; more Prep for college with all the essentials Shop now Shop now In as fast as an hour* Fans, pools &amp; more Shop now Shop now Discover Great Brands Shop all Bed Head by TIGI After Party Super Smoothing Cream for frizzy or dull hair, 3.38 fl oz\u200b Add Bed Head by TIGI After Party Super Smoothing Cream for frizzy or dull hair, 3.38 fl oz\u200b 437 4.7 out of 5 Stars. 437 reviews Ginger Mint by Marvis for Unisex - 4.5 oz Toothpaste Add Ginger Mint by Marvis for Unisex - 4.5 oz Toothpaste 21 4.9 out of 5 Stars. 21 reviews Restored Acer Nitro 29.5\u201d Curved UWFHD (2560 x 1080) 200Hz 1800R Curved Gaming Monitor, Black, ED300C XWBMIIPPX, New Add Restored Acer Nitro 29.5\u201d Curved UWFHD (2560 x 1080) 200Hz 1800R Curved Gaming Monitor, Black, ED300C XWBMIIPPX, New ASICS MENS Gel 1130 &quot;White Pure Silver&quot; 1201B020 100 from Stadium Goods ASICS MENS Gel 1130 \"White Pure Silver\" 1201B020 100 from Stadium Goods 8 3.6 out of 5 Stars. 8 reviews Sol de Janeiro Bom Dia Body Scrub 8 oz Add Sol de Janeiro Bom Dia Body Scrub 8 oz 1 5 out of 5 Stars. 1 reviews Rollback VIZIO 75&quot; Class Quantum 4K QLED HDR Smart TV (VQD75R-10) Rollback Add VIZIO 75\" Class Quantum 4K QLED HDR Smart TV (VQD75R-10) 5501 4.4 out of 5 Stars. 5501 reviews Looks that ace the day Back-to-school glasses for kids Shop now Shop now Walmart+ members earn 5% cash back at Walmart Learn more Learn more Free setup &amp; support on select computers* Shop now Shop now Get 0% APR on select laptops with OnePay Later* Shop now Shop now Exactly what kids need Find your school\u2019s supply list Get started Get started We\u2019d love to hear what you think! Give feedback All Departments Store Directory Careers Our Company Sell on Walmart.com Help Product Recalls Accessibility Tax Exempt Program Get the Walmart App Safety Data Sheet Terms of Use Privacy Notice California Supply Chain Act Your Privacy Choices Customer Privacy Center Notice at Collection AdChoices Consumer Health Data Privacy Notices Learn about Spark Driver Brand Shop Directory Pharmacy Walmart Business #IYWYK Delete Account \u00a9 2026 Walmart. The trademarks Walmart and the Walmart Spark design are registered with the US Patent and Trademark Office. All Rights Reserved.", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "unitedhealthgroup.com", "fetchedAt": "2026-07-30T17:21:08.548419Z", "result": {"base": "https://unitedhealthgroup.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Built for Better Health - UnitedHealth Group\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.unitedhealthgroup.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9379 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://unitedhealthgroup.com/contact-us.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 77, "title": "Built for Better Health - UnitedHealth Group", "textOnly": "Built for Better Health - UnitedHealth Group Skip to content Close Menu Close Search About About UnitedHealth Group Healthcare should work better than it does today. Too often, it&#39;s complicated, expensive and difficult to navigate. At UnitedHealth Group, we&#39;re working to change that. Through UnitedHealthcare and Optum, we&#39;re helping make care more affordable, accessible and connected \u2014 so more people can live healthier lives and the healthcare system works better for everyone. Mission &amp; Values Our People &amp; Culture United for Giving Awards &amp; Recognition United Health Foundation Our Businesses Sustainability What We Do Value-Based Care Health Financial Services\u200b Health Benefits Artificial Intelligence Pharmacy Services\u200b How We Do It Careers Careers UnitedHealth Group is a health care and well-being company dedicated to improving health outcomes for millions of people around the world. We comprise two distinct but complementary businesses, UnitedHealthcare and Optum, working together to build a better health system for all. Find your next job Join our talent community --> Work Here Clinical Professionals Corporate &amp; Business Operations Customer Service &amp; Support Early Careers Sales &amp; Account Management Technology &amp; Data Life Here Culture of Belonging Employee Benefits Military Fellowship Program Job Seeker Resources Blog Events Newsroom Insights, News &amp; Resources Explore our news stories to see our mission at work, find recent press releases and read research offering solutions for addressing health care challenges and opportunities. Newsroom Media Resources Latest Updates Insights &amp; Research Center for Health Care Research A Path Forward Investor Relations Investor Relations We are dedicated to continuously improving the quality of our performance for consumers, customers, care providers and our shareholders. Investor Relations Financial &amp; Earnings Reports Shareholder Resources Dividend History &amp; Stock Basis Archive UnitedHealth Group Reports Second Quarter 2026 Results UnitedHealth Group reported second quarter 2026 results and raised guidance for full year 2026. Read Press Release UnitedHealthcare Our mission is to help people live healthier lives and make the health system work better for everyone. We dedicate ourselves to this every day for our members by being there for what matters in moments big and small \u2014 from their earliest days to their working years and through retirement. Go to UnitedHealthcare Find a Doctor Find Medicare Advantage Plan Estimate Drug Costs Check My Coverage Review My EOB Optum We\u2019re a leading health solution and care delivery organization. Our work is complex, but our mission is simple: create a healthier world, with you at the center. Go to Optum About Optum Careers at Optum For Business Find Care Member Sign-In Search Close Search Country Country Built for Better Health &nbsp;&nbsp; &nbsp; Listen to the replay of our second quarter 2026 earnings call Forward-Looking Statement The statements, estimates, projections, guidance or outlook contained in this document include \u201cforward-looking\u201d statements which are intended to take advantage of the \u201csafe harbor\u201d provisions of the federal securities laws. The words \u201cbelieve,\u201d \u201cexpect,\u201d \u201cintend,\u201d \u201cestimate,\u201d \u201canticipate,\u201d \u201cforecast,\u201d \u201coutlook,\u201d \u201cplan,\u201d \u201cproject,\u201d \u201cshould\u201d and similar expressions identify forward-looking statements. These statements may contain information about financial prospects, economic conditions and trends and involve risks and uncertainties. Actual results could differ materially from those that management expects, depending on the outcome of certain factors including: our ability to effectively estimate, price for and manage medical costs; new or changes in existing health care laws or regulations, or their enforcement or application; cyberattacks, other privacy/data security incidents, or our failure to comply with related regulations; reductions in revenue or delays to cash f", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://unitedhealthgroup.com/contact-us.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "apple.com", "fetchedAt": "2026-07-30T17:21:13.723020Z", "result": {"base": "https://apple.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: Organization, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Apple\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.apple.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6419 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:1-800-692-7753).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 70, "title": "Apple", "textOnly": "Apple Apple Apple Store Mac iPad iPhone Watch Vision AirPods TV &amp; Home Entertainment Accessories Support 0 + Apple Upgrade Love it. Lease it. Upgrade it. Learn more College, sorted. Get a gift card from $100 to $150 * when you buy Mac or iPad with education savings. Shop iPhone Meet the latest iPhone lineup. Learn more Shop iPhone MacBook Air Now supercharged by M5. Learn more Buy iPad Air Now supercharged by M4. Learn more Buy Apple Watch Series 11 The ultimate way to watch your health. Learn more Buy iPad Pro Advanced AI performance and game-changing capabilities. Learn more Buy Apple Trade In Get up to $195\u2013$695 in credit when you trade in iPhone 13 or higher. 1 Get your estimate Apple Card Get up to 3% Daily Cash back with every purchase. Learn more Apply now Endless entertainment. Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Stream now Comedy It\u2019s pickleball vs. tennis in the hilarious new movie. Stream now Action Anya Taylor-Joy is a grifter running for her life after a heist goes sideways. F1 on Apple TV Every Grand Prix\u2122, live and on demand\u2014all in one place, all year long. Stream now Comedy All family. No plan. MLS on Apple TV Watch every club, every match, live\u2014all season long. Stream now Thriller Fear takes hold. See the schedule Live MLB games, every&nbsp;Friday. Stream now Mystery 19 Emmy\u00ae Nominations Including Best Comedy Catch up now Comedy The new season premieres August 5. You better believe it. Sabrina Carpenter: The Zane Lowe Interview Listen now Sabrina Carpenter: The Zane Lowe Interview Play now Hello Kitty Island Adventure Watch now David Bowie A-List Pop Listen now A-List Pop Play now PowerWash Simulator Watch now HIIT with Bakari New Music Daily Listen now New Music Daily Play now Balatro+ Watch now Programs Apple Footer * Available for Qualified Purchasers only. Qualified Purchasers receive an Apple Gift Card when they purchase an eligible Mac or iPad (\u201cEligible Product\u201d) at a Qualifying Location through August 27, 2026. Gift card values may vary by Eligible Product. Customers will receive a discount equal to the value of the Apple Gift Card off the price of the Eligible Product, but will be charged for all items in their cart, including the Apple Gift Card. Only one Apple Gift Card per Eligible Product per Qualified Purchaser. Offer subject to availability. While supplies last. Additional restrictions apply. View full offer terms here . Trade\u2011in values will vary based on the condition, year, and configuration of your eligible trade\u2011in device. Not all devices are eligible for credit. You must be at least the age of majority to be eligible to trade in for credit or for an Apple Gift Card. Trade\u2011in value may be applied toward qualifying new device purchase, or added to an Apple Gift Card. Actual value awarded is based on receipt of a qualifying device matching the description provided when estimate was made. Sales tax may be assessed on full value of a new device purchase. In\u2011store trade\u2011in requires presentation of a valid photo ID (local law may require saving this information). Offer may not be available in all stores and may vary between in\u2011store and online trade\u2011in. Some stores may have additional requirements. Apple or its trade\u2011in partners reserve the right to refuse, cancel, or limit quantity of any trade\u2011in transaction for any reason. More details are available from Apple\u2019s trade-in partner for trade\u2011in and recycling of eligible devices. Restrictions and limitations may apply. To access and use all Apple&nbsp;Card features and products available only to Apple&nbsp;Card users, you must add Apple&nbsp;Card to Wallet on an iPhone or iPad that supports and has the latest version of iOS or iPadOS. Apple&nbsp;Card is subject to credit approval, available only for qualifying applicants in the United States, and issued by Goldman Sachs Bank USA, Salt Lake City Branch. Apple&nbsp;Payments&nbsp;Services&nbsp;LLC, a subsidiary of Apple&nbsp;Inc., is a service provider of Goldman Sachs B", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:1-800-692-7753", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "abc.xyz", "fetchedAt": "2026-07-30T17:21:18.766527Z", "result": {"base": "https://abc.xyz", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Alphabet Investor Relations - Home\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6334 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 30, "title": "Alphabet Investor Relations - Home", "textOnly": "Alphabet Investor Relations - Home Skip to main content Home Investors Results & Financials Earnings SEC Filings Updates News Events Governance Board & Governance Annual Meeting Founders\u2019 Letters Other FAQs Additional Information Email Alerts Skip to main content Home Home Investors Results & Financials Earnings SEC Filings Updates News Events Governance Board & Governance Annual Meeting Founders\u2019 Letters Other FAQs Additional Information Email Alerts is for Google As Sergey and I wrote in the original founders letter 11 years ago, \u201cGoogle is not a conventional company. We do not intend to become one\u201d more As part of that, we also said that you could expect us to make \u201csmaller bets in areas that might seem very speculative or even strange when compared to our current businesses.\u201d From the start, we\u2019ve always strived to do more, and to do important and meaningful things with the resources we have. We did a lot of things that seemed crazy at the time. Many of those crazy things now have over a billion users, like Google Maps, YouTube, Chrome, and Android. And we haven\u2019t stopped there. We are still trying to do things other people think are crazy but we are super excited about. We\u2019ve long believed that over time companies tend to get comfortable doing the same thing, just making incremental changes. But in the technology industry, where revolutionary ideas drive the next big growth areas, you need to be a bit uncomfortable to stay relevant. Our company is operating well today, but we think we can make it cleaner and more accountable. So we are creating a new company, called Alphabet. I am really excited to be running Alphabet as CEO with help from my capable partner, Sergey, as President. What is Alphabet? Alphabet is mostly a collection of companies. The largest of which, of course, is Google. This newer Google is a bit slimmed down, with the companies that are pretty far afield of our main internet products contained in Alphabet instead. What do we mean by far afield? Good examples are our health efforts: Life Sciences (that works on the glucose-sensing contact lens), and Calico (opens in new window) (focused on longevity). Fundamentally, we believe this allows us more management scale, as we can run things independently that aren\u2019t very related. Alphabet is about businesses prospering through strong leaders and independence. In general, our model is to have a strong CEO who runs each business, with Sergey and me in service to them as needed. We will rigorously handle capital allocation and work to make sure each business is executing well. We\u2019ll also make sure we have a great CEO for each business, and we\u2019ll determine their compensation. In addition, with this new structure we plan to implement segment reporting for our Q4 results, where Google financials will be provided separately than those for the rest of Alphabet businesses as a whole. This new structure will allow us to keep tremendous focus on the extraordinary opportunities we have inside of Google. A key part of this is Sundar Pichai. Sundar has been saying the things I would have said (and sometimes better!) for quite some time now, and I\u2019ve been tremendously enjoying our work together. He has really stepped up since October of last year, when he took on product and engineering responsibility for our internet businesses. Sergey and I have been super excited about his progress and dedication to the company. And it is clear to us and our board that it is time for Sundar to be CEO of Google. I feel very fortunate to have someone as talented as he is to run the slightly slimmed down Google and this frees up time for me to continue to scale our aspirations. I have been spending quite a bit of time with Sundar, helping him and the company in any way I can, and I will of course continue to do that. Google itself is also making all sorts of new products, and I know Sundar will always be focused on innovation\u2014continuing to stretch boundaries. I know he deeply cares that we ca", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "cvshealth.com", "fetchedAt": "2026-07-30T17:21:24.111162Z", "result": {"base": "https://cvshealth.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (4486 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "1 JSON-LD block(s) found but none parsed as valid JSON -- broken structured data no agent can actually read.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Leading Health Solutions Company | CVS Health\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6997 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://cvshealth.com/contact.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 68, "title": "Leading Health Solutions Company | CVS Health", "textOnly": "Leading Health Solutions Company | CVS Health Skip to main content Aetna\u00ae CVS Caremark\u00ae CVS Pharmacy Careers Investors Contact Us Aetna\u00ae CVS Caremark\u00ae CVS Pharmacy Careers Investors Contact Us About About Our purpose is simple and clear: To simplify health care one person, one family and one community at a time. About About Our Strategy Our Strategy Inclusion &amp; Belonging Company History Our Strategy Inclusion &amp; Belonging Company History Our People Our People Our Purpose Our People Our Purpose Leadership Leadership Public Policy Public Policy Public Health Value-Based Care Access to Coverage Drug Costs Public Policy Public Health Value-Based Care Access to Coverage Drug Costs CVS Health Ventures CVS Health Ventures Services Services Our approach ties directly to our purpose: To simplify health care one person, one family and one community at a time. Services Services Health Care and Wellness Health Care and Wellness Health Services Condition Management Mental Health Health and Wellness Products Other Health Care Services Health Care and Wellness Health Services Condition Management Mental Health Health and Wellness Products Other Health Care Services Prescription Drug Coverage Prescription Drug Coverage Pharmacy Benefits Management Health Care Affordability Prescription Drug Coverage Pharmacy Benefits Management Health Care Affordability Health Insurance Health Insurance Commercial Health Coverage Medicare Medicaid Vision and Dental Management Medical Management Health Insurance Commercial Health Coverage Medicare Medicaid Vision and Dental Management Medical Management Pharmacy Pharmacy Mail Order Pharmacy Neighborhood Pharmacy Prescription Delivery Prescription Savings Specialty Pharmacy Pharmacy Mail Order Pharmacy Neighborhood Pharmacy Prescription Delivery Prescription Savings Specialty Pharmacy News News Find everyday health news, media resources and more. News News Press Releases Press Releases Insights &amp; Innovations Insights &amp; Innovations Media Contacts Media Contacts Media Library Media Library Impact Impact Find out how we&#39;re supporting our people, communities, business and the planet. Impact Impact Healthy People Healthy People Healthy Community Healthy Community Healthy Planet Healthy Planet Healthy Business Healthy Business Reporting and Governance Reporting and Governance Quitting Tobacco Quitting Tobacco Back Building a world of health around every individual We connect people to the care they need, put medicine and wellness within reach, and drive greater affordability, simplicity and convenience in everything we do. Combining our strengths for you Health services Health insurance Pharmacy Prescription drug coverage Health services We have an unmatched ability to reach people and provide them with access to the affordable, quality health care and wellbeing services they need to live their healthiest lives. Explore our services Health insurance Aetna\u00ae creates insurance plans that deliver a total, connected, affordable approach to health care. Discover Aetna plans Pharmacy Our neighborhood pharmacies \u2014 as well as our prescription delivery and specialty medication services \u2014 are making it as easy as possible for people to get the care they depend on. Visit our pharmacy page Prescription drug coverage Our pharmacy benefit manager , CVS Caremark\u00ae, plays a critical role in the health care system by helping to lower drug costs and expand coverage to the affordable medications people need to stay healthy. Learn about drug coverage CVS Health Corporation reports strong first quarter 2026 results and raises full\u2011year 2026 guidance Read the story Latest news Go to our News section July 30, 2026 CVS Health\u00ae named Overall Winner and wins four gold awards at 2026 US Customer Experience Awards July 29, 2026 Health100\u2122 Earns Industry Accreditation for Consumer Data Privacy and Security July 22, 2026 CVS Health helping to strengthen communities across Massachusetts July 21, 2026 Pet medications now available ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://cvshealth.com/contact.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "berkshirehathaway.com", "fetchedAt": "2026-07-30T17:21:29.349039Z", "result": {"base": "https://berkshirehathaway.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "No robots.txt found (defaults open, but unmanaged).", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"BERKSHIRE HATHAWAY INC.\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1262 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 28, "title": "BERKSHIRE HATHAWAY INC.", "textOnly": "BERKSHIRE HATHAWAY INC. B ERKSHIRE H ATHAWAY INC. 3555 Farnam Street Omaha, NE 68131 Official Home Page &nbsp; A Message from Warren E. Buffett Fraudulent Claims Regarding Mr. Buffett's Endorsements of Investment Products and Political Candidates Annual &amp; Interim Reports Updated May 2, 2026 News Releases from Berkshire Hathaway and from Warren Buffett Updated July 24, 2026 Link to SEC Filings Annual Meeting Information Updated April 20, 2026 Warren Buffett's Letters to Berkshire Shareholders Updated February 22, 2025 Greg Abel's Letters to Berkshire Shareholders Updated February 28, 2026 Special Letters from Warren & Charlie RE:Past, Present and Future Charlie Munger's Letters to Wesco Shareholders Links to Berkshire Operating Companies Berkshire Hathaway Energy Investor Presentations Corporate Governance Common Stock Information Sustainability FOR A FREE CAR INSURANCE RATE QUOTE THAT COULD SAVE YOU SUBSTANTIAL MONEY WWW.GEICO.COM OR CALL 1-888-395-6349, 24 HOURS A DAY If you have any comments about our WEB page, you can write us at the address shown above. However, due to the limited number of personnel in our corporate office, we are unable to provide a direct response. Legal Disclaimer Copyright &copy; 1977-2026 Berkshire Hathaway Inc.", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "mckesson.com", "fetchedAt": "2026-07-30T17:21:35.066606Z", "result": {"base": "https://mckesson.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"McKesson | Medical Supplies, Pharmaceuticals Healthcare Solu\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7541 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://mckesson.com/contact-us/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "McKesson | Medical Supplies, Pharmaceuticals Healthcare Solutions", "textOnly": "McKesson | Medical Supplies, Pharmaceuticals Healthcare Solutions Skip to content Skip to footer Logins Explore All Logins Medical Supplies McKesson Connect Specialty Health Customer Center InspiroLink&#x2122; Contact Us Careers Investors Explore All Investor Resources Financials Events &amp; Presentations Investor FAQ Menu Close McKesson About About Guided by our values, we are an impact-driven organization that improves care in every setting &ndash; one product, one partner, one patient at a time. Explore Business Segments Leadership History Corporate Governance McKesson Ventures Our Locations Newsroom Explore Corporate Governance Board of Directors Board Committee Composition and Charters Governance Documents Contact Our Directors Explore Newsroom Press Releases Commitments Commitments Through our core commitments, we are leveraging our scale for the greater good, such as serving the needs of our stakeholders, donating back to our communities, protecting our planet, working with policymakers, and more. Explore Discover Our Impact Every year, we publish updates to our commitments through our Impact Report. Download Our Impact Report Our Impact McKesson Foundation Taking the Lead Public Policy Compliance and Ethics Explore Our Impact Our People Our Planet Our Community Our Partners Explore McKesson Foundation McKesson Taking Care of Our Own Fund Explore Taking the Lead Supply Chain Resilience Fighting Opioid Abuse Supply Chain Security Explore Public Policy Access to Medicine Improving Connectivity in Healthcare Preparing for Emergencies Expanding Access to Clinical Trials Promoting Community Providers Commitment to Patient Safety in Health IT Political Engagement McKesson Amplify Business Solutions Business Solutions Our businesses bring together leading technologies, innovative solutions and hands-on expertise to support the entire healthcare ecosystem. Explore All Businesses Listings Take a look at our list of businesses. Our Businesses Pharmaceutical, Medical Supplies &amp; Products Distribution Services We distribute pharmaceuticals and medical supplies to healthcare settings across North America, from pharmacies and hospitals to doctors&rsquo; offices and clinics. Pharmaceutical Distribution Services Medical Supply Distribution Pharmacy Services We help to ensure the financial wellbeing of pharmacies and health systems and support a stable work environment for their employees. Oncology &amp; Multispecialty Solutions We provide research, insights, technologies and other support to help address challenges in cancer and specialty care. Biopharma Capabilities We provide a suite of solutions designed to address access, affordability and adherence challenges by bridging the gaps between biopharma companies, pharmacies, providers, and payers to help patients get on and stay on their medications. Prescription Benefit Solutions We offer solutions that enable employers, payers, health-plan brokers and government agencies to provide lower-cost options for prescription medications and therapies. Consumer Products &amp; Services We help to ensure the financial wellbeing of pharmacies and health systems and support a stable work environment for their employees. Stories &amp; Insights About Back to Menu About Explore Business Segments Leadership History Corporate Governance McKesson Ventures Our Locations Newsroom Back To About Corporate Governance Explore Corporate Governance Board of Directors Board Committee Composition and Charters Governance Documents Contact Our Directors Back To About Newsroom Explore Newsroom Press Releases Commitments Back to Menu Commitments Explore Our Impact McKesson Foundation Taking the Lead Public Policy Compliance &amp; Ethics Back To Commitments Our Impact Explore Our Impact Our People Our Planet Our Community Our Partners Back To Commitments McKesson Foundation Explore McKesson Foundation McKesson Taking Care of Our Own Fund Back To Commitments Taking the Lead Explore Taking the Lead Supply Chain Res", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://mckesson.com/contact-us/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "exxonmobil.com", "fetchedAt": "2026-07-30T17:21:40.992791Z", "result": {"base": "https://exxonmobil.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Exxon Mobil Corporation | ExxonMobil\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://corporate.exxonmobil.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5977 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://exxonmobil.com/news/media-contacts).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Exxon Mobil Corporation | ExxonMobil", "textOnly": "Exxon Mobil Corporation | ExxonMobil Skip to content Who we are Who we are We're committed to responsibly meeting society&rsquo;s evolving energy needs. Who we are Who we are Our organization Our global organization Our approach History Global brands Our business divisions Overview Upstream Downstream Chemicals Low Carbon Solutions Technology and collaborations Overview Energy technologies Digital technologies Risk management and safety Policy Overview EU policy Advanced recycling policy Leadership Management committee Board of Directors What we do What we do &nbsp;Creating energy and sustainable solutions that improve quality of life and meet society&rsquo;s evolving needs. What we do What we do Transforming transportation Overview EV &amp; fuel technology Lower emission fuels Motorsports Materials for modern living Overview Advanced recycling Advanced synthetic graphite Proxxima thermoset resin systems Delivering industrial solutions Overview Carbon capture &amp; storage Hydrogen Lithium Energy supply Overview Global Outlook Natural gas Oil Sustainability Sustainability ExxonMobil is committed to improving quality of life by meeting the needs of society. Sustainability Sustainability Sustainability Overview Integrating sustainability into what we do Pursuing environmental excellence Meeting society's critical needs for energy and products Safeguarding people Contributing to the well-being of communities Advancing Climate Solutions Overview Growing Low Carbon Solutions Positioned for growth in a lower emission future Driving reductions in methane emissions Rational and constructive policy Research and development Governance and risk management Community Investing in communities We&rsquo;re committed to being a good corporate citizen, helping to improve society and quality of life where we operate around the world. Community Community STEM education Overview United States education initiatives International education initiatives Our partners Community impact Managing socioeconomic impacts Working with suppliers Respecting human rights Worldwide giving Overview Investing in STEM education Improving quality of life and economic well-being Volunteering in local communities Owner relations Overview Questions and inquiries Revenue online support Newsroom Global newsroom Stay up to date with the latest news releases and publications from around the globe. Newsroom Newsroom News releases Publications Viewpoints Subscription center Media contacts Investors Investors Explore financials, strategy and reports showcasing our growth and investor value. Investors Investors Corporate governance Dividends Earnings SEC filings Stock information Individual shareholders Events &amp; presentations Careers Careers Build a career with us and use your energy to change the world. Careers Careers Featured career paths Benefits Internship opportunities Job openings Global Search Popular searches Community Sustainability Carbon capture and storage Hydrogen Careers Our global organization Publications Investor Relations ExxonMobil to release second quarter 2026 financial results Learn more Register for the webcast Creating sustainable solutions Browse our latest Sustainability Report ExxonMobil introduces our new Voluntary Retail Voting Program Learn more about the program Our third advanced recycling unit at Baytown is now operational Learn more ExxonMobil to release second quarter 2026 financial results Creating sustainable solutions ExxonMobil introduces our new Voluntary Retail Voting Program Our third advanced recycling unit at Baytown is now operational Exxon Mobil Corporation (XOM) $156.23 $-0.52 (-0.33%) 12:59pm ET \u2022 July 30, 2026 The need for energy is universal. That's why ExxonMobil scientists and engineers are pioneering new research and pursuing new technologies to reduce emissions while creating more efficient fuels. We're committed to responsibly meeting the world's energy needs. Who we are Trending topics Carbon capture and storage Carbon", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://exxonmobil.com/news/media-contacts", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "cencora.com", "fetchedAt": "2026-07-30T17:21:46.828613Z", "result": {"base": "https://cencora.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (4927 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Creating healthier futures | Cencora\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.cencora.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10511 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://cencora.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Creating healthier futures | Cencora", "textOnly": "Creating healthier futures | Cencora Skip to main content menu toggle button What we do Main menu For manufacturers For manufacturers Logistics and warehousing Leverage our global network to ensure safe, secure storage and timely delivery worldwide. Drug research and clinical development Streamline the drug development process and minimize go-to-market risk. Product and solution innovation Learn about our newest offerings currently in development. Commercialization support Build a foundation for a successful product launch and accelerate your speed-to-market. Patient access and adherence support Address barriers to patient access and increase your product&#39;s speed-to-therapy. Explore all solutions Find solutions tailored to your business needs. For providers For providers Wholesale and specialty distribution Increase access to medications with our innovative solutions and secure supply chain. Pharmacy solutions Build and scale your pharmacy, accelerate growth, and deliver stronger clinical results. Product and solution innovation Learn about our newest offerings currently in development. Practice and clinic solutions Elevate the patient experience and expand care possibilities with physician practice solutions. Hospital and health system solutions Solve your most critical problems, plan for sustainable growth, and create better health outcomes with a strategic innovation partner. Explore all solutions Find solutions tailored to your business needs. Our expertise Main menu Biosimilar commercialization Cencora offers expertise in biosimilar distribution, helping manufacturers and providers navigate this evolving market for cost-effective therapies. Cell and gene therapies Accelerate your path from discovery to commercialization with our integrated support for biopharma companies. OTC &amp; consumer healthcare products Partner with Cencora for market-leading distribution of OTC and consumer healthcare products. Leverage our extensive network and expertise. Oncology Increase your therapy&#39;s success with tailored strategies and global expertise for faster market entry. Brand and specialty products Discover Cencora&#39;s solutions for brand and specialty products, including distribution, commercialization, and patient access strategies. Global generic products Extend your reach with Cencora&#39;s global distribution network for generic products. Benefit from our extensive network and purchasing programs. Mature product outsourcing Unlock greater value for your mature product portfolio with strategic and operational support on a global scale from Cencora. Our global reach Main menu Our global reach Explore the impact our global reach and local expertise can have on your business and on people worldwide. North and South America Find out how we enhance patient care and optimize supply chains across North and South America. Europe Discover how we improve access to medicines and strengthen local health outcomes within Europe. Asia Pacific Learn how we supply high-value therapies and improve patient outcomes in the Asia Pacific. Who we are Main menu Our company Our company About us Get to know Cencora and how we&#39;re improving the health and well-being of people and animals across the globe. Leadership Meet our executive leaders, each of whom has a proven record of leadership across the healthcare spectrum. History Learn how we\u2019ve become a world-class partner to pharmaceutical innovators around the world. Capabilities Discover how we support communities everywhere by offering global support and local expertise to human health professionals. Stories from Cencora Read featured stories that dive deeper into what we do and who we are as a company. Our impact Our impact Our impact Discover how we&#39;re making an impact on lives around the world. Corporate responsibility Discover how we\u2019re making a positive impact on the health of our people and our planet through the work we do. Cencora Impact Foundation Learn about our independent, n", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://cencora.com/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "microsoft.com", "fetchedAt": "2026-07-30T17:21:52.190460Z", "result": {"base": "https://microsoft.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Microsoft \u2013 AI, Cloud, Productivity, Computing, Gaming &amp;\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.microsoft.com/en-gb).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "4477 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://support.microsoft.com/contactus).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 49, "title": "Microsoft \u2013 AI, Cloud, Productivity, Computing, Gaming &amp; Apps", "textOnly": "Microsoft \u2013 AI, Cloud, Productivity, Computing, Gaming &amp; Apps Trace Id is missing Skip to main content Microsoft 365 Azure Copilot Windows Surface XBOX Deals Small Business Support Windows Apps OneDrive Outlook Moving from Skype to Teams OneNote Microsoft Teams Shop XBOX Accessories VR &amp; mixed reality XBOX Game Pass Ultimate XBOX games PC games Microsoft AI Microsoft Security Azure Dynamics 365 Microsoft 365 for business Microsoft Power Platform Windows 365 Digital Sovereignty Microsoft Developer Microsoft Learn Support for AI marketplace apps Microsoft Tech Community Microsoft Marketplace Software companies Visual Studio Microsoft Rewards Free downloads &amp; security Education Gift cards Licensing View Sitemap New Surface Laptop Enjoy the performance Supercharged productivity that lasts all day. Available in 13, 13.8 and 15-inch. Shop Surface Laptop New Surface Pro Born to flex Laptop-grade power, tablet flexibility \u2013 adapts as your day changes shape. Available in 12 and 13-inch. Shop Surface Pro Play/Pause Secondary Nav Shop Surface Shop XBOX Find your next PC Shop Microsoft 365 Shop accessories Shop for your Business Find what you need faster with AI-powered suggestions Ask a question Help me choose a PC Microsoft 365 tips Sign in to my account Your productivity, supercharged Microsoft 365 delivers cloud storage, advanced security and Microsoft Copilot in your favourite apps \u2013 all in one plan. Shop Microsoft 365 Accessories you\u2019ll love Mice, keyboards, docks, and chargers\u2014designed to keep you comfortable, charged up, and ready to take on anything, from anywhere. Learn more XBOX controllers Elite, wireless, adaptive \u2013 find the controller that fits your style of play. Shop now XBOX Game Pass Ultimate Play new titles on day one. Enjoy hundreds of high-quality games with friends on console, PC or cloud. Join now XBOX Series X - 2 TB Next-gen frame rate meets lightning-fast load times in the most powerful XBOX ever\u2014now available with a 2TB SSD in special-edition Galaxy Black. Shop now For business New configurations Surface for Business with Intel Fast performance for demanding apps, all-day battery life and built-in security \u2013 powered by Intel\u00ae Core\u2122 Ultra Series 3 processors.* Learn more Microsoft 365 Copilot Save time and focus on the things that matter most with AI in Microsoft 365 for business. Learn more Join the era of AI Create, communicate, and code with the latest Microsoft AI solutions. Explore AI solutions Windows 11 for business Designed for hybrid work. Powerful for employees. Consistent for IT. Secure for all. Learn more Explore more about AI and Copilot Turning words into apps \u201cVibe coding\u201d allows people to build software by talking through an idea with a Microsoft AI tool that writes the code. Learn more Exploring Theodore Roosevelt\u2019s legacy with help from AI Learn how the Microsoft AI for Good Lab helped shape a new presidential library featuring a lifelike, AI-powered Roosevelt, ready for conversation. Read the story Making AI more representative Microsoft is partnering with disability communities to build image libraries that depict them in a more accurate way. Read the story Less plastic, more planet\u200b From phasing out single-use plastics to monitoring our climate, discover how Microsoft is helping protect and support the planet. Learn more Follow Microsoft Back to top Surface Pro Surface Laptop Surface Laptop Ultra Copilot for organisations Copilot for personal use AI in Windows Explore Microsoft products Windows 11 apps Account profile Download Center Microsoft Store Support Returns Order tracking Recycling Microsoft Store Promise Microsoft in education Devices for education Microsoft Teams for Education Microsoft 365 Education Office Education Educator training and development Deals for students and parents Azure for students Microsoft AI Microsoft Security Azure Dynamics 365 Microsoft 365 Microsoft 365 Copilot Microsoft Teams Small Business Microsoft Developer Microsoft Learn Support for AI marke", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://support.microsoft.com/contactus", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "jpmorganchase.com", "fetchedAt": "2026-07-30T17:21:57.616589Z", "result": {"base": "https://jpmorganchase.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Home | JPMorganChase\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.jpmorganchase.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5621 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://jpmorganchase.com/newsroom/media-contacts).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 42, "title": "Home | JPMorganChase", "textOnly": "Home | JPMorganChase Skip to main content Join our team Search JPMorganChase About us About us How we do business Leadership Awards and recognition Technology Governance Suppliers Diversity, opportunity &amp; inclusion Read more Chairman and CEO Letter to Shareholders Learn more Impact Impact Global impact U.S. impact Creating thriving communities together See how our clients and partners\u2014from small business owners to workforce training leaders\u2014work with JPMorganChase to drive meaningful impact and economic growth where they live and work. Learn more Communities Institute Institute Explore all topics Latest news Careers Careers Work with us Grow with us How we hire Explore opportunities Students and graduates Apply now Latest news Veteran\u2019s Unconventional Path to Landing her Dream Job in Tech U.S. Army Veteran Ashley Wigfall transitioned to a civilian role and charted her path to technologist through mentorship and skills training at the JPMorgan Chase tech hub in Plano, Texas. Learn more Investor Relations Investor Relations Senior Executive Letters Annual Report Quarterly Earnings Press releases Events and presentations Company Update Investor Day Learn more Chairman and CEO Letter to Shareholders Learn more Newsroom Latest news Chairman and CEO Letter to Shareholders Learn more Homepage | JPMorganchase 2025 Annual Report \u201cWe are champions of banking\u2019s essential role in a community \u2014 its potential for bringing people together, for enabling companies and individuals to attain their goals, and for being a source of strength in difficult times.&#34; Jamie Dimon Chairman and Chief Executive Officer Read letter Proud Olympic &amp; Paralympic Games Partner JPMorganChase is honored to be the first global banking partner of the Olympic and Paralympic Games. Learn about our business Committed to service, innovation and growth JPMorganChase serves millions of customers, clients and communities in over 100 global markets. For 225 years, our work has been guided by principles that strengthen, protect and grow our company over time. Learn about our business Making opportunities happen See how our clients\u2014from small business owners to affordable housing developers\u2014help uplift communities. Learn more Our new headquarters JPMorganChase opened its new global headquarters, marking a major milestone in its investment in New York City and workspaces globally. Learn more Serving with purpose We are united by a purpose that weaves together what we value and how we do business and deliver results for the clients, colleagues and communities we serve. See how we do business Creating economic opportunity for all We leverage the best of our firm to help clients, customers, communities and employees grow and succeed. Our impact Helping power economic growth by creating opportunities for all through business and community investments, local collaborations and policy advocacy. Learn about our impact Our communities Making an impact in the communities where we live and work through long-term business and philanthropic efforts. Explore our communities Prioritizing talent and career growth A talent-driven company is, by definition, an inclusive one. We\u2019re committed to creating a workplace where employees feel like they belong. Explore careers See impact in action Access stories about our impact and employees from around the world, as well as insights from our research, policy, and leadership teams. Read more Our 2025 Annual Report Our latest report and letters to shareholders, released April 2026. Read the report In the news View all JPMorganChase announces $24 million to help strengthen shipbuilding in Philadelphia and America\u2019s defense industrial base Investments support skills training, regional collaboration and small business suppliers, connecting local job and business opportunities with national competitiveness and resilience priorities Learn more Registration opens for San Francisco\u2019s 40th JPMorganChase Corporate Challenge The 3.5-mile run/walk is s", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://jpmorganchase.com/newsroom/media-contacts", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "costco.com", "fetchedAt": "2026-07-30T17:22:02.721057Z", "result": {"base": "https://costco.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://costco.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "thecignagroup.com", "fetchedAt": "2026-07-30T17:22:08.033887Z", "result": {"base": "https://thecignagroup.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The Cigna Group | A Global Health Company\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.thecignagroup.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3926 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://thecignagroup.com/about-us/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 54, "title": "The Cigna Group | A Global Health Company", "textOnly": "The Cigna Group | A Global Health Company Cigna Healthcare Cigna Healthcare Evernorth Health Services Evernorth Health Services Careers Careers Newsroom Newsroom Espa\u00f1ol Espa\u00f1ol About Us About Us Accreditation Contact Us Milestones Supplier Community About Us About Us About Us Accreditation Contact Us Milestones Supplier Community Capabilities Our Capabilities Cigna Healthcare Evernorth Health Services Capabilities Capabilities Our Capabilities Cigna Healthcare Evernorth Health Services Impact Our Impact Environmental, Social, and Governance Improving Health and Vitality Impact Impact Our Impact Environmental, Social, and Governance Improving Health and Vitality Commitments Our Commitments Commitments Commitments Our Commitments Investors Our Commitment to Better We know the health care system isn't delivering for everyone. So, we're taking accountability and responsibility for the role that we play and committing to help make it better for all our customers and patients. Learn about the actions we're taking We are a global health company, focused on improving the health and vitality of those we serve. Through our two divisions, Cigna Healthcare &reg; and Evernorth Health Services &reg; , we are committed to enhancing the lives of our clients, customers, and patients. In order to fulfill this goal, we continually challenge ourselves to strengthen and evolve our capabilities. View our capabilities INVESTORS Our growth strategy consists of our scaled businesses contributing steady, predictable growth and high-growth businesses in attractive markets combined with cross-enterprise leverage to create even greater value. Read our latest earnings and reports NEWSROOM Stay up to date SPOTLIGHT Built with America: The Cigna Group\u2019s 230-Year History From America\u2019s earliest days, The Cigna Group has grown alongside the nation. Learn more about the company\u2019s history and enduring commitment to customers across generations. Read more SPOTLIGHT Employers Want Simpler and More Transparent Pharmacy Benefits According to research, 87% of employers believe rebate-free models align better with organizational needs and 90% believe they will give the best experience to employees. SPOTLIGHT The Cigna Group Joins the CMS Health Technology Ecosystem Learn about the ongoing efforts to simplify health care and foster patient-centered innovation. SPOTLIGHT Voices of Impact: When a Library Becomes a Pathway to Care Hartford Public Library and The Cigna Group Foundation provide no-cost transportation for local residents to attend medical appointments. About Us We are guided by our mission and our values and powered by our brands and over 230 years of innovation and partnership. Learn more about our commitment to the greater good on a global scale and how our past inspires our future. Discover The Cigna Group Our Impact We're taking on the health care system as a whole to affect positive change. To achieve our goals, we're conducting studies with leading experts in critical areas like health equity and partnering with dynamic entrepreneurs and programs who share our vision. Through the volunteer efforts of our dedicated workforce and our philanthropic commitments, we're making a difference in the health of communities worldwide. View our impact Be a part of our movement. We're looking for others who want to build a healthier future, together. Explore careers at The Cigna Group Leaders Our Leaders The Cigna Group Leadership Board of Directors Committees of the Board Our Culture of Compliance Learn More About Us Environmental, Social, and Governance Supplier Community Media Contacts Connect with Us on Social \u00a9 The Cigna Group Selecting these links will take you away from TheCignaGroup.com to another website, which may be a non-Cigna website. The Cigna Group may not control the content or links of non-Cigna websites. Details Legal Privacy Policy Accessibility Terms of Use Cookie Settings", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "https://thecignagroup.com/about-us/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "cardinalhealth.com", "fetchedAt": "2026-07-30T17:22:13.324016Z", "result": {"base": "https://cardinalhealth.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://cardinalhealth.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "nvidia.com", "fetchedAt": "2026-07-30T17:22:18.818186Z", "result": {"base": "https://nvidia.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (2610 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"World Leader in Artificial Intelligence Computing | NVIDIA\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.nvidia.com/en-gb/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "43571 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.nvidia.com/en-gb/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 83, "title": "World Leader in Artificial Intelligence Computing | NVIDIA", "textOnly": "World Leader in Artificial Intelligence Computing | NVIDIA NVIDIA Home NVIDIA Home Menu Menu icon Menu Menu icon Close Close icon Close Close icon Close Close icon Caret down icon Accordion is closed, click to open. Caret down icon Accordion is closed, click to open. Caret up icon Accordion is open, click to close. Caret right icon Click to expand Caret right icon Click to expand Caret right icon Click to expand menu. Caret left icon Click to collapse menu. Caret left icon Click to collapse menu. Caret left icon Click to collapse menu. Shopping Cart Click to see cart items Search icon Click to search Visit your regional NVIDIA website for local content, pricing, and where to buy partners specific to your country. Argentina Australia Belgi\u00eb (Belgium) Belgique (Belgium) Brasil (Brazil) Canada \u010cesk\u00e1 Republika (Czech Republic) Chile Colombia Danmark (Denmark) Deutschland (Germany) Espa\u00f1a (Spain) France India Italia (Italy) M\u00e9xico (Mexico) Middle East Nederland (Netherlands) Norge (Norway) \u00d6sterreich (Austria) Peru Polska (Poland) Rest of Europe Rom\u00e2nia (Romania) Singapore Suomi (Finland) Sverige (Sweden) T\u00fcrkiye (Turkey) United Kingdom United States \ub300\ud55c\ubbfc\uad6d (South Korea) \u4e2d\u56fd\u5927\u9646 (Mainland China) \u53f0\u7063 (Taiwan) \u65e5\u672c (Japan) Continue Skip to main content Artificial Intelligence Computing Leadership from NVIDIA Main Menu Products Cloud Services Creating Data Center Embedded Systems Gaming Graphics Cards and Desktops Laptops Networking Professional Workstations Software Tools Cloud Services DGX Cloud NVIDIA\u2019s AI factory in the cloud NVIDIA APIs Explore, test, and deploy AI models and agents Private Registry Guide for using NVIDIA NGC private registry with GPU cloud NVIDIA NGC Accelerated, containerized AI models and SDKs DSX Platform Build AI factories optimized for lowest cost tokens per megawatt Virtual GPU Virtual solutions for scalable, high-performance computing Creating NVIDIA Studio High performance GeForce RTX PCs, purpose-built for creators NVIDIA Broadcast App AI-enhanced voice and video for next-level streams, videos, and calls NVIDIA App and Studio Drivers Optimize gaming, streaming, and AI-powered creativity RTX AI PCs AI PCs for gaming, creating, productivity and development RTX Remix Create RTX remasters of classic games with open-source AI Project G-Assist AI assistant to optimize and control your GeForce RTX PC Data Center Overview Modernizing data centers with AI and accelerated computing DGX Platform Enterprise AI factory for model development and deployment Grace CPU Architecture for data centers that transform data into intelligence HGX Platform A supercomputer purpose-built for AI and HPC IGX Platform Advanced functional safety and security for edge AI MGX Platform Accelerated computing with modular servers OVX Systems Scalable data center infrastructure for high-performance AI DSX Platform Build AI factories optimized for lowest cost tokens per megawatt Embedded Systems Jetson Leading platform for autonomous machines and embedded applications DRIVE AGX Powerful in-vehicle computing for AI-driven autonomous vehicle systems IGX Platform Advanced functional safety and security for edge AI Gaming GeForce Explore graphics cards, gaming solutions, AI technology, and more GeForce Graphics Cards RTX graphics cards bring game-changing AI capabilities Gaming Laptops Thinnest and longest lasting RTX laptops, optimized by Max-Q G-SYNC Monitors Smooth, tear-free gaming with NVIDIA G-SYNC monitors DLSS Neural rendering tech boosts FPS and enhances image quality Reflex Ultimate responsiveness for faster reactions and better aim RTX Remix Create RTX remasters of classic games with open-source AI Project G-Assist AI assistant to optimize and control your GeForce RTX PC GeForce NOW Cloud Gaming RTX-powered cloud gaming. Choose from 3 memberships NVIDIA App and Game Ready Drivers Optimize gaming, streaming, and AI-powered creativity NVIDIA Broadcast App AI-enhanced voice and video for next-level streams, videos, and calls SHIELD TV World-cla", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"ai2bot, anthropic-ai, Amazonbot, Amazon-kendra, Applebot-extended, AutoGPT, BedrockBot, Bytespider, CCBot, Chatglm-spider, ChatGPT, ChatGPT-User, Claudebot, Claude-user, Claude-search, Claude-web, Cohere-ai, Cotoyogi, Deepseek, Deepseek Chat, Deepseekbot, DuckAssistBot, DuckDuckBot, FacebookBot, Geminiios, Google-Extended, Googleagent-manier, GPTBot, Grok, ia-archiver, iaskbot, iaskspider, ICC-crawler, Linerbot, Metaaib, Meta-externalagent, Meta-externalfetcher, Meta-llama, Mistralai-user, NovaAct, OAI-Searchbot, Pangubot, Perplexity, PerplexityBot, Perplexity-user, Phindbot, PoeBot, Qwen, Sogou, Timpibot, Tongyi, Wenxiaobai, Xinghuo, Youbot, Yuanbaobot\" has no root Disallow."}], "contactMatch": "https://www.nvidia.com/en-gb/contact/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "about.meta.com", "fetchedAt": "2026-07-30T17:22:25.547525Z", "result": {"base": "https://about.meta.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, Organization,Brand,Corporation, ImageObject, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"About Meta | Social technology, VR, AR and innovation\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.meta.com/en-gb/about/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1262 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 47, "title": "About Meta | Social technology, VR, AR and innovation", "textOnly": "About Meta | Social technology, VR, AR and innovation Skip to content We&#039;re building the future of human connection Our mission And the technologies that make it possible Our technologies Our innovations give people new ways to connect AI at Meta And we&#039;re committed to helping keep everyone safe and making a positive impact Our actions Catch up on the latest news See more at Newsroom Shop the latest devices and expand your world Ray-Ban Meta AI glasses Capture, share and stay in the moment, completely hands-free. Shop now Meta Quest 3S Dive into the wonder of mixed reality with the new Meta Quest 3S. Shop now Connect in new ways with our products Explore our technologies Go to Help Centre Stay informed about our actions We&#039;re committed to helping keep people safe and making a positive impact. Learn more Safety and expression Protecting your voice and helping you connect and share safely Data and privacy Giving you control over your privacy and protecting your information Responsible innovation Building for the future with privacy and safety in mind Meet our leadership Meta&#039;s leaders are guiding our company as mixed reality and AI evolve, helping to create the next evolution of digital connection. Get to know our leadership", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "elevancehealth.com", "fetchedAt": "2026-07-30T17:22:31.284984Z", "result": {"base": "https://elevancehealth.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Elevance Health | Advancing Health Beyond Healthcare\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.elevancehealth.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7501 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.elevancehealth.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Elevance Health | Advancing Health Beyond Healthcare", "textOnly": "Elevance Health | Advancing Health Beyond Healthcare Skip to main content Please wait while loading... Search The Site Who We Are Our Approach To Health Research Stories Newsroom Investors Back Who We Are Who We Are Elevance Health supports health at every life stage, offering health plans and clinical, behavioral, pharmacy, and complex-care solutions that promote whole health. Our Workforce Our Workforce Our Companies Our Companies Suppliers Suppliers Careers Careers Corporate Responsibility Corporate Responsibility Supporting Our Communities Environmental Sustainability Our Foundation Close Menu Back Our Approach To Health Our Approach To Health Our approach to health begins by redefining health, reimagining the system, and strengthening our communities. Improving health for everyone is possible. Whole Health &amp; What Drives It Whole Health &amp; What Drives It Community Health Community Health Health Equity Health Equity Digitally Enabled Healthcare Digitally Enabled Healthcare Consumer-Centered Health System Consumer-Centered Health System Featured Topics Featured Topics Whole Health Index Whole Health Index Care Provider Partnership Care Provider Partnership Food as Medicine Food as Medicine Maternal Health Maternal Health Close Menu Back Research Research Research supports our commitment to innovation and our pursuit of moving health forward. Elevance Health Research Elevance Health Research Physical Drivers of Health Behavioral Drivers of Health Social Drivers of Health Peer-Reviewed Research Public Policy Institute Research Public Policy Institute Research About Commercial Insurance Medicaid Medicare Healthcare Spending and Value Transforming Healthcare Close Menu Back Close Menu Back Newsroom Newsroom Read the latest press releases and news from Elevance Health, where we strive to elevate whole health and advance health beyond healthcare. Media Resources Media Resources Close Menu Back Investors Investors Integrity, transparency, and a commitment to excellence inspire business operations and corporate governance at Elevance Health, all to pursue our purpose: improving the health of humanity. Events and Presentations Events and Presentations Stock Information Stock Information Financials and SEC Filings Financials and SEC Filings Annual Reports Annual Reports Corporate Governance Corporate Governance Resources Resources Close Menu Search Search The Site Elevating Whole Health and Advancing Health Beyond Healthcare We serve people across their entire health journey and address their full range of needs with an integrated whole-health approach. Learn More About Us Elevance Health brings together the concepts of elevate and advance, exemplified by our bold purpose of improving the health of humanity. We are a health company dedicated to improving health outcomes, lowering costs, and simplifying experiences. Ideas That Explore How to Elevate and Advance Health A Whole Health Story Addressing the Cost of Healthcare: Relief for Today, Solutions for Tomorrow As a partner in health, it\u2019s our role \u2014 and our responsibility \u2014 to help manage healthcare costs on both fronts, making it easier for people to get the care they need. To do so, we\u2019re focused on lowering costs, improving health outcomes, and making healthcare easier to navigate. Whole Health And What Drives It Consumer Centered Health System Lowering the Cost of Care June 3, 2026 Elevance Health Impact Elevance Health Releases Fourth Advancing Health Together: Progress Report Elevance Health\u2019s fourth annual Advancing Health Together: Progress Report highlights how we\u2019re partnering with care providers to create a simplified care experience that costs less and leads to improved health outcomes for the people we serve. Whole Health And What Drives It Elevance Health Impact Care Provider Partnership April 14, 2026 A Consumer-Centered Health System Four Ways Employers Are Managing Rising Healthcare Costs Rising healthcare costs have long challenged employers and remain an ur", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.elevancehealth.com/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "centene.com", "fetchedAt": "2026-07-30T17:22:36.873018Z", "result": {"base": "https://centene.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Managed Care &amp; Healthcare Solutions | Centene Corporatio\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.centene.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "4431 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.centene.com/contact.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 46, "title": "Managed Care &amp; Healthcare Solutions | Centene Corporation", "textOnly": "Managed Care &amp; Healthcare Solutions | Centene Corporation Skip to Main Content Toggle navigation Search Toggle navigation Contact Who We Are Our Mission History Board of Directors Capabilities Subsidiaries Corporate Facts &amp; Reports Ethics &amp; Integrity One CenTeam People with Disabilities Serving the Military Community Employee Networks Attracting Top Talent Centene Foundation Impact Becoming a Partner Featured Partnerships Accreditations Awards Why We&#39;re Different Corporate Sustainability Empowering Health Healthier Communities Healthy Environment Business Accountability Strategic Partnerships Providers &amp; Healthcare Partners Impacting Lives Our Suppliers Economic Impact Improving Outcomes Combating Fraud, Waste and Abuse Products &amp; Services Medicaid Long-Term Services &amp; Supports Foster Care Medicare Health Insurance Marketplace ICHRA Whole Health Solutions Browse by State Careers Investors News Transforming the health of the communities we serve, one person at a time. This is Centene Centene\u2019s mission is focused on helping people live healthier lives. For 40 years, our company has worked tirelessly to increase access to affordable, quality-focused healthcare to those who need it most. Every community and person has unique needs and we strive for continuous improvement and expansion of our services to better meet those needs every day. Who We Are &gt; Featured Stories centene:homepage true 3 newest View All News &gt; Serving our Members As a leading provider of government-sponsored healthcare, we provide access to affordable, quality-focused products to Medicaid and Medicare members, as well as to individuals and families served by the Health Insurance Marketplace. We are the largest Medicaid managed care organization and the longest-running carrier on the Marketplace. Products &amp; Services &gt; Investing in Communities We believe that quality healthcare is best delivered locally. That\u2019s why Centene\u2019s health insurance plans are locally branded, with teams in every community we serve. This allows us to be present in our communities, and to better understand and provide solutions for the unique needs of our members, providers and other stakeholders. We partner with local organizations and nonprofits to further address the challenges our members face at the local level. Our Impact &gt; Careers At Centene, we&#39;re passionate about transforming the health of the communities we serve while giving our employees the freedom to take charge of their own career journeys. We offer a competitive total rewards package including vacation, a 401(k) matching program and health insurance. Find your purpose and make a difference working here. Join our Team Life at Centene Our collaborative and engaging work environment is a cornerstone of Centene\u2019s culture, which includes a flexible approach for our employees to do their best work in the way they work best. This mindset not only broadens our employees\u2019 knowledge and skills, but also results in better outcomes for our members. Our People &gt; Prioritizing Corporate Sustainability As a corporate neighbor, we believe that environmental stewardship, enriching the social well-being of the communities we serve and adhering to high standards of ethics and sound corporate governance are essential to the welfare of our members, employees and communities. Learn More &gt; Centene Investor Relations We are committed to delivering sustainable, profitable growth through strategic focus, innovative programs and a uniquely local approach in the rapidly evolving healthcare sector. Explore how our competitive advantages drive market differentiation, value creation and significant opportunities for future growth. Visit Investor Site Transforming the health of the communities we serve, one person at a time. HEALTHCARE IS BEST DELIVERED LOCALLY Our uniquely local approach allows us to help members access quality-focused, culturally sensitive healthcare services. CENTENE IS COMMITTED TO ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.centene.com/contact.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "bankofamerica.com", "fetchedAt": "2026-07-30T17:22:42.048642Z", "result": {"base": "https://bankofamerica.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: Corporation, Organization, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Bank of America - Banking, Credit Cards, Loans and Merrill I\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.bankofamerica.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "35869 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:wealthmanagement@bankofamerica.com?body&#x3D;Please).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Bank of America - Banking, Credit Cards, Loans and Merrill Investing", "textOnly": "Bank of America - Banking, Credit Cards, Loans and Merrill Investing Skip to main content \",\"textContent\":\"We sent a text message with the download link to \"},\"error\":{\"icon\":\"https://www1.bac-assets.com/homepage/spa-assets/images/assets-images-global-get-app-modal-ico_error-CSX5e531926.svg\",\"header\":\"We couldn&#x27;t send the link\",\"content\":\"We&#x27;re sorry we weren&#x27;t able to send you the download link. Please try again, or use your mobile device to get the app from its app store.\"},\"buttons\":{\"chooseDevice\":\"Go\",\"submitContact\":\"Send\",\"success\":\"OK\",\"error\":\"OK\",\"interstitialContinue\":\"Continue\",\"interstitialBack\":\"Go back to Bank of America\"},\"deviceField\":{\"required\":true,\"label\":{\"text\":\"Select Your Device\"},\"select\":{\"attrs\":{\"id\":\"getAppDeviceSelect\",\"name\":\"getAppDeviceSelect\",\"noEmptyOption\":true,\"autocomplete\":\"off\",\"class\":\"spa-input-select spa-input-select\",\"value\":\"\",\"data-val\":\"\"},\"options\":[{\"text\":\"iPhone\",\"value\":\"iphone-get-app\"},{\"text\":\"iPad\",\"value\":\"ipad-get-app\"},{\"text\":\"Android\",\"value\":\"android-get-app\"},{\"text\":\"Other\",\"value\":\"other-get-app\"}]}},\"phoneNumberField\":{\"required\":true,\"label\":{\"text\":\"Phone Number\",\"adaText\":\"In 999-999-9999 Format\"},\"input\":{\"format\":\"phone\",\"attrs\":{\"id\":\"phoneNumberField\",\"name\":\"phoneNumberField\",\"autocomplete\":\"off\"}},\"validationRules\":{\"change\":{\"phone\":{\"parameters\":[\"us\"]}}}},\"emailAddressField\":{\"required\":true,\"label\":{\"text\":\"Email Address\"},\"input\":{\"format\":\"email\",\"attrs\":{\"id\":\"emailAddressField\",\"name\":\"emailAddressField\",\"data-error-msg\":\"Please provide a valid email address\"}},\"validationRules\":{\"change\":{\"email\":{\"parameters\":[]}}}},\"jcr\":\"copy/global/en/get-app-modal/global-getappmodal.json\"}}'> Get the Mobile Banking app Select Your Device &nbsp;* iPhone iPad Android Other Go Get it on the App Store Before you leave our site, we want you to know your app store has its own privacy practices and level of security which may be different from ours, so please review their policies. Or we can text a download link directly to your phone Phone Number &nbsp;* In 999-999-9999 Format Send Please enter a valid 10-digit phone number By providing your mobile number you are consenting to receive a text message. Text message fees may apply from your carrier. Text messages may be transmitted automatically. Apple, the Apple logo, iPhone, iPad, Apple Watch and Touch ID are trademarks of Apple Inc., registered in the U.S. and other countries. App Store is a service mark of Apple Inc. Get it on the App Store Before you leave our site, we want you to know your app store has its own privacy practices and level of security which may be different from ours, so please review their policies. Or we can send you a link by email Email Address &nbsp;* Send Please enter a valid email address Apple, the Apple logo, iPhone, iPad, Apple Watch and Touch ID are trademarks of Apple Inc., registered in the U.S. and other countries. App Store is a service mark of Apple Inc. Get it on Google Play Before you leave our site, we want you to know your app store has its own privacy practices and level of security which may be different from ours, so please review their policies. Or we can text a download link directly to your phone Phone Number &nbsp;* In 999-999-9999 Format Send Please enter a phone number By providing your mobile number you are consenting to receive a text message. Text message fees may apply from your carrier. Text messages may be transmitted automatically. Android is a trademark of Google Inc. Samsung is a registered trademark of Samsung Electronics Co., Ltd. Our mobile app isn't available for all devices If you don't see an app for your device, you may still be able to access our mobile website by typing bankofamerica.com in your mobile web browser. Learn more about mobile banking options and supported devices Get the mobile banking app Before you leave our site, we want you to know your app store has its own privacy practices and level of security which may be diff", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:wealthmanagement@bankofamerica.com?body&#x3D;Please", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "chevron.com", "fetchedAt": "2026-07-30T17:22:49.859355Z", "result": {"base": "https://chevron.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://chevron.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "ford.com", "fetchedAt": "2026-07-30T17:23:05.159220Z", "result": {"base": "https://ford.com", "fatal": "Their server refused or timed out (status network error). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://ford.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "gm.com", "fetchedAt": "2026-07-30T17:23:10.513780Z", "result": {"base": "https://gm.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://gm.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "citigroup.com", "fetchedAt": "2026-07-30T17:23:19.687108Z", "result": {"base": "https://citigroup.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s).", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Citi | Global Investment Bank and Financial Services\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.citigroup.com/global).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "949 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:\\).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Citi | Global Investment Bank and Financial Services", "textOnly": "Citi | Global Investment Bank and Financial Services About Us Businesses Our Impact News Insights Investors Careers My Account Citigroup.com Homepage Second Quarter 2026 Results and Key Metrics View Materials We are dedicated to responsibly providing financial services to enable growth and economic progress. Learn more about Citi OUR BUSINESSES We offer products and services as one integrated institution while treating our clients as one global relationship. Overview Services Markets Banking &amp; International Wealth U.S. Consumer Cards Citi Second Quarter 2026 Earnings Call Press Release Jane Fraser, DBE Chair and Chief Executive Officer Gonzalo Luchetti Chief Financial Officer Presentation Financial Supplement Financial Supplement Audio Transcript INVESTOR HIGHLIGHTS 2026 Investor Day Quarterly Earnings Materials SEC Filings Corporate Governance 2025 Annual Report Sign up to receive the latest insights from Citi. Email Address* Next", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:\\", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "homedepot.com", "fetchedAt": "2026-07-30T17:23:25.158430Z", "result": {"base": "https://homedepot.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://homedepot.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-f500", "domain": "fanniemae.com", "fetchedAt": "2026-07-30T17:23:30.847331Z", "result": {"base": "https://fanniemae.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Powering America\u2019s Housing | Fannie Mae\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.fanniemae.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6384 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://fanniemae.com/about-us/corporate-governance/contacting-board-directors).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 60, "title": "Powering America\u2019s Housing | Fannie Mae", "textOnly": "Powering America\u2019s Housing | Fannie Mae Skip to main content Toggle navigation Home Toggle Search Single-Family Business Multifamily Business Capital Markets Homebuyers, Owners, and Renters Our Company About Us Toggle Who We Are Toggle Awards and Recognition Employee Code of Conduct Community Engagement History What We Do Toggle Fannie Mae Today Duty to Serve Toggle Manufactured Housing Affordable Housing Preservation Rural Housing Research Initiatives Heirship Colonias Investment Areas Supporting Native American homeownership Homeownership Toggle Access to Mortgage Credit Restrictive Covenants Innovation Disaster Resiliency and Response Pilot Transparency Positive Rent Payment Reporting Leadership Team Corporate Governance Toggle Corporate Governance Guidelines Fannie Mae Charter Board of Directors Committees Committee Charters Code of Conduct Bylaws Contacting the Board of Directors Housing Reports Investor Relations Toggle Quarterly and Annual Results SEC Filings Toggle All SEC Filings Annual Filings Quarterly Filings Current Reports Section 16 Filings Proxy Filings Fixed-Income Securities Monthly Summary and Capital Disclosures Toggle Monthly Summary Capital Disclosures Stockholder Information Current and Historical Stock Information Contact Information Corporate Responsibility Data and Insights Toggle Forecast Toggle Monthly Archive Surveys &amp; Indices Toggle National Housing Survey Toggle About the National Housing Survey Archive Mortgage Lender Sentiment Survey Toggle About the Mortgage Lender Sentiment Survey Archive Fannie Mae Home Price Index Home Price Expectations Survey Weekly Mortgage Applications Data Toggle About the Weekly Mortgage Applications Data Special Topics Toggle Consumer Mortgage Understanding Survey Special Topics Archive Publications Toggle Fannie Mae Working Papers Housing Insights Partner Research Publications Archive Perspectives Toggle Archive - 2025 Archive - 2024 Archive - 2023 Archive - 2022 About ESR Newsroom Toggle News Media Resources Careers Toggle Our Teams Toggle Corporate Teams Data &amp; Analytics Digital &amp; Technology Mortgage &amp; Customer Solutions Risk Management Life at Fannie Mae Benefits &amp; Professional Development Campus Contact Us Menu Fannie Mae Home FM Site Nav Homepage Search Toggle search Search Skip to main content Our financial results: Net income of $4.0 billion for 2Q 2026 Watch our 2Q 2026 earnings webcast replay. Watch the replay Read the press release Supporting a stronger housing market We are a foundational part of the housing finance system and are committed to ensuring market stability. Learn more Fannie Mae Powering America\u2019s Housing We help make housing more accessible and affordable. \u200b Fannie Mae purchases mortgages from lenders and bundles them into mortgage-backed securities (MBS) that we sell to investors to promote a healthy housing market. Learn how&nbsp; &nbsp; $241 billion in funding to support the U.S. housing market in the first half of 2026 ~802,000 households helped in buying, refinancing, or renting a home in the first half of 2026 $4.1 trillion guaranty book of business as of June 30, 2026 Serving the housing finance market Single-Family Business Single-Family Business Multifamily Business Multifamily Business Capital Markets Capital Markets Homebuyers, Owners, &amp; Renters Homebuyers, Owners, &amp; Renters Single-Family Business Provides liquidity to the home purchase market primarily by acquiring loans from lenders and overseeing the loans\u2019 servicing. Visit Single-Family &nbsp; Multifamily Business Delivers liquidity to the rental housing market through our national network of Delegated Underwriting and Servicing (DUS \u00a9 ) lender partners. Visit Multifamily &nbsp; Capital Markets Offers investors high-quality assets with attractive yields to fit various portfolio needs or investment strategies. Visit Capital Markets &nbsp; Homebuyers, Owners, &amp; Renters We offer support through housing education, information, and tools including di", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://fanniemae.com/about-us/corporate-governance/contacting-board-directors", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "thekrogerco.com", "fetchedAt": "2026-07-30T17:23:38.176346Z", "result": {"base": "https://thekrogerco.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "No robots.txt found (defaults open, but unmanaged).", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Article,BlogPosting, WebPage, ImageObject, BreadcrumbList, WebSite, Organization, Person.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Home - The Kroger Co.\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.thekrogerco.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "8249 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.thekrogerco.com/contact-us/media-contacts/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 58, "title": "Home - The Kroger Co.", "textOnly": "Home - The Kroger Co. Skip to content Media Investors Careers Contact Us Media Investors Careers Contact Us Search for: About Fresh Health History Our Business Our Values Our Brands Newsroom Vendors / Suppliers Partner Pass Partner Pass Support Supplier Hub Supplier Hub Support Vendor Information Document Library Supplier Diversity &#038; Inclusion Responsible Sourcing Approach &amp; Resources The Kroger Specification Portal Purpose Awards &#038; Recognition Innovation Fund Portfolio Reporting Hub Statements &#038; Policies Transportation Zero Hunger | Zero Waste Community About Fresh Health History Our Business Our Values Our Brands Newsroom Vendors / Suppliers Partner Pass Partner Pass Support Supplier Hub Supplier Hub Support Vendor Information Document Library Supplier Diversity &#038; Inclusion Responsible Sourcing Approach &amp; Resources The Kroger Specification Portal Purpose Awards &#038; Recognition Innovation Fund Portfolio Reporting Hub Statements &#038; Policies Transportation Zero Hunger | Zero Waste Community What it takes to win in food retail Our teams work year-round to bring fresh, quality food to every table at prices families can count on. A letter from CEO Greg Foran A letter from Chairman Ron Sargent $4.4B Meals donated to our communities ~400K Associates company-wide 11M Customers served daily ~2,700 Stores across 35 states\u200b What Drives Us Everything we do connects back to the people we serve and the communities we call home. Commitment to Affordability Fresh, quality products at prices families can count on \u2014 through Our Brands, smart sourcing and everyday low prices. Explore Our Brands Innovation Pickup, delivery, and an experience that makes shopping easy however you shop \u2014 powered by technology and built around you. See Our Innovations Rewards That Matter Fuel points, personalized savings and Boost membership that rewards your loyalty every time you shop. Learn More Our People Our 400,000 associates make the difference every day \u2014 in stores, online and in the communities we serve. Meet Our People Our Communities Working to end hunger in our communities and eliminate waste across our company through our Zero Hunger | Zero Waste initiative. Our Impact What\u2019s Happening at Kroger View all News July 28, 2026 Kroger Helps Families Simplify Back-to-Routine Season with Fresh, Convenient Meals and a Smarter Way to Shop The Kroger Co. (NYSE: KR), America's grocer, is helping families transition out of summer and back into their routines with Learn More July 24, 2026 Five Ways to Get the Most from a GLP-1 Medication GLP-1 medications are changing what is possible for many patients looking to manage their weight. To get the most out Learn More July 21, 2026 Murray&#8217;s Cave-Aged Cheeses Bring Home Six Medals at the 2026 American Cheese Society Awards The Kroger Co. (NYSE: KR) and Murray's Cheese announced today that six cheeses from its renowned Cave Aged Cheese program Learn More July 2, 2026 Kroger Names Emilee De Martino Executive Vice President and Chief People Officer The Kroger Co. (NYSE: KR) today announced Emilee De Martino will serve as its next Executive Vice President and Chief Learn More Our Family of Banners A family of banners serving communities across America. Southern California Intermountain West Arizona Pacific Northwest Southeast Colorado Pacific Northwest Kansas Chicago Midwest Mid - Atlantic Nebraska Our Brands Thousands of quality products across brands customers love \u2014 offering the quality you want at prices you deserve. Explore All Our Brands America&#8217;s largest natural and organic brand. Over 1,500 products made without 101+ artificial ingredients \u2014 from pantry staples to plant-based proteins. Explore Our Brands Private Selection\u00ae brings you the next-level flavor you crave, without all the fuss. So you can create culinary excellence anywhere, anytime. Explore Private Selection Everyday Essentials Premium Quality Everyday Value Fresh Poultry &amp; Meat Baby &amp; Toddler Pet Prod", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "https://www.thekrogerco.com/contact-us/media-contacts/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "verizon.com", "fetchedAt": "2026-07-30T17:23:44.176572Z", "result": {"base": "https://verizon.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Verizon: Wireless, Internet, TV and Phone Services | Officia\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.verizon.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5844 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:+18338374966).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Verizon: Wireless, Internet, TV and Phone Services | Official Site", "textOnly": "Verizon: Wireless, Internet, TV and Phone Services | Official Site Accessibility Resource Center Skip to main content Personal Business 1-833-VERIZON Contact us Support Stores Coverage map Espa\u00f1ol Mobile Mobile Mobile Mobile Simplicity Plan Simplicity Plan myPlan Smartphones More devices More devices More devices Certified pre-owned phones Unlocked phones Basic phones Tablets &amp; laptops Watches Certified pre-owned watches Jetpacks &amp; hotspots Bring your phone Bring your phone Verizon Prepaid Accessories Close &nbsp; Home Home Home Home Shop fast, reliable Wi-Fi Shop TV Move your service Fios accessories Close &nbsp; Save Save Save Save Switch to Verizon Phone deals Discover Verizon Loyalty Verizon One - Mobile &amp; Internet Exclusive perks &amp; services Trade-in your device Earn with Verizon Visa\u00ae Card Plan discounts Plan discounts Plan discounts Verizon Forward Military discounts First Responder discounts Nurse discounts Teacher discounts Student discounts Employer discounts Close &nbsp; Sign in Back to Menu Account overview Account overview Pay bill Make a one-time payment Register Business Log in Close Choose your cart: Mobile solutions Home solutions Go To Cart Sign in Back to Menu Account overview Account overview Pay bill Make a one-time payment Register Business Log in Close Choose your cart: Mobile solutions Home solutions Go To Cart Close Personal Business 1-833-VERIZON Contact us Support Stores Coverage map Espa\u00f1ol Call Us: 1-800-VERIZON Call now Loyalty you'll love. Money back. Epic experiences. Incredible deals. All for you. end of navigation menu Switch to Verizon Simplicity Plan Home Internet Verizon One Mobile + Home Bring your phone Phone Deals --> Introducing Simplicity Plan Learn more After AutoPay and $15/mo switch discount. Plus taxes and fees. --> $ 30 /month per line Your phone, your way With a single line for $30/mo, you have three easy ways to get the phone you want. Keep your own phone Connect it to our best 5G network. Buy a new phone right now We\u2019ve got flexible financing options to meet your needs. Upgrade to the latest phone every year Get the newest every year for a monthly charge. New Samsung Galaxy Z Fold8 on Simplicity All for $60/mo Buy this phone for $30/mo when you switch. Plus $30/mo w/ 1 line on Simplicity Plan. After Auto Pay & switch discount + taxes and fees. Phone: $1899.99 (256 GB only) w/ 48 month device payment purchase for new customers w/ new smartphone line on Simplicity Plan req\u2019d. Less $460 promo credit applied over 48 mos.; promo credit ends if eligibility req\u2019s are no longer met; 0% APR. Simplicity Plan: $10/mo Auto Pay (ACH or Verizon Visa Card) & paper-free billing discount requires enrollment w/in 30 days. $15/mo Switch discount requires smartphone line port-in or an uploaded mobile bill from an eligible carrier dated w/in the past 45 days. Unlimited data is restricted to on-device smartphone usage. Domestic data roaming at 2G speeds. Galaxy AI basic features provided by Samsung are free. Any Samsung enhanced AI features and all third-party AI features are subject to different terms and may be subject to fees.\"> Best deal More great device offers Add Verizon Home Internet Plus up your plan with our fast, reliable Home Internet for only $35/month with Auto Pay & Mobile + Home Discount. Fast: 5G Home/Verizon Home Internet Lite services provided through cellular network; speeds vary due to factors affecting cellular networks. $35/mo: Verizon Home Internet: General: Verizon Home Internet (VHI) includes 5G Home, VHI Lite and Fios internet services. Availability varies. Auto Pay & paper-free billing req\u2019d. Subject to credit approval. 5G Home/Fios 300 Mbps/VHI Lite: Plans start at $35/mo. when combined with postpaid Verizon mobile phone plan (excludes business and data-only plans). Fios 300 Mbps: Mobile + Home Discount enrollment req'd. $99 setup and other terms apply.\"> Check availability Starting at $ 35 /month Fast: 5G Home/Verizon Home Internet Lite services provided ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:+18338374966", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "phillips66.com", "fetchedAt": "2026-07-30T17:23:49.774562Z", "result": {"base": "https://phillips66.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Leader in Energy Solutions | Phillips 66\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.phillips66.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5856 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.phillips66.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Leader in Energy Solutions | Phillips 66", "textOnly": "Leader in Energy Solutions | Phillips 66 Contact Careers Business Operations Business Operations With nearly 150 years of experience, our teams across our unique value chain focus on safely delivering reliable energy now and for the future. Operations Refining Midstream Overview Midstream Asset Map Pipeline Safety Tariffs Customers Marketing Commercial Chemicals Renewable Fuels Energy Research &amp; Innovation Products Aviation Overview Cards and Rewards Rebate Programs Become a Branded FBO Aviation Products Find an FBO Aviation News Asphalt Crude Oil Fuels &amp; Lubricants Specialties Services Safety Data Sheets Interest Owners AnalysisPlus Sustainability Sustainability Sustainability shapes how we define and execute our strategy, which supports our long-term resilience and competitiveness. Overview Our Targets ESG Library Safety &#038; Community Safety &#038; Community We put safety first to strengthen the communities we serve. Our Commitment to Safety Social Impact Pipeline Safety Stakeholder Engagement Investors Investors Find relevant financial and operating information about our company for institutional and individual investors. Introduction Stock Information Financial Information Events and Presentation Corporate Governance Resources Phillips 66 Indicators Newsroom Newsroom Stay up to date with the latest news releases, company stories and publications. Newsroom Home Our Stories News Releases Media Request Form About Us About Us Our mission is to provide energy and improve lives. We are excited about the future of energy and helping the world move forward. Our People History Products We Make Possible Corporate Locations Careers Contact Search Phillips 66: Search Business Operations Business Operations Operations Operations Refining Midstream Midstream Overview Midstream Asset Map Pipeline Safety Tariffs Customers Marketing Commercial Chemicals Renewable Fuels Energy Research &amp; Innovation Products Products Aviation Aviation Overview Cards and Rewards Rebate Programs Become a Branded FBO Aviation Products Find an FBO Aviation News Asphalt Crude Oil Fuels &amp; Lubricants Specialties Services Services Safety Data Sheets Interest Owners AnalysisPlus Sustainability Sustainability Overview Our Targets ESG Library Safety &#038; Community Safety &#038; Community Our Commitment to Safety Social Impact Pipeline Safety Stakeholder Engagement Investors Investors Introduction Stock Information Financial Information Events and Presentation Corporate Governance Resources Phillips 66 Indicators Newsroom Newsroom Newsroom Home Our Stories News Releases Media Request Form About Us About Us Our People History Products We Make Possible Corporate Locations Careers Contact Contact Careers Providing Energy. Improving Lives. Guided by our core values of safety, honor and commitment, we work to meet the world\u2019s energy needs today and tomorrow. Business Operations Innovation &amp; Operational Performance Community &amp; Safety Business Operations With a focus on continually improving our operations, we provide enduring value to our employees, investors and communities. 01 Refining 02 Midstream 03 Chemicals 04 Marketing 05 Renewable Fuels Our People Every person in our organization is valued \u2014 and we make sure they are supported, heard and provided with opportunities to grow in their careers and as people. More About Our People Innovation &amp; Operational Performance Making continual advancements is critical to supporting our existing operations and developing technologies for future needs. 01 Energy Research &amp; Innovation By exploring new avenues of energy generation, storage and access, we balance efficiency and sustainability in order to keep meeting the needs of an ever-changing world. 02 Sustainability A focus on safety and operational excellence helps ensure that we remain a strong, vibrant company. Bringing innovations to market is how we succeed and help move the world forward. Sustainability &amp; Renewable Fuels How Phillips 66", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.phillips66.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-f500", "domain": "marathonpetroleum.com", "fetchedAt": "2026-07-30T17:23:54.803080Z", "result": {"base": "https://marathonpetroleum.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://marathonpetroleum.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "nytimes.com", "fetchedAt": "2026-07-30T17:24:01.829591Z", "result": {"base": "https://nytimes.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebSite, NewsMediaOrganization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The New York Times - Breaking News, US News, World News and \u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.nytimes.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7830 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://help.nytimes.com/hc/en-us/articles/115015385887-Contact-The-New-York-Times).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "The New York Times - Breaking News, US News, World News and Videos", "textOnly": "The New York Times - Breaking News, US News, World News and Videos Skip to content Skip to site index SKIP ADVERTISEMENT U.S. International Canada Espa\u00f1ol \u4e2d\u6587 Today\u2019s Paper Go Home U.S. International Canada Espa\u00f1ol \u4e2d\u6587 New York Times - Top Stories Live LIVE Trump Administration July 30, 2026, 1:17 p.m. ET War in the Middle East July 30, 2026, 12:08 p.m. ET Top Stories U.S. Growth Slows as Inflation Bites, in an Uncertain Economic Moment The economy slowed in the second quarter, expanding at an annual rate of 1.5 percent, as the war with Iran upended prices, supply chains and energy markets. 5 min read Markets Challenge Warsh\u2019s Approach to Taming Inflation Kevin Warsh has made controlling inflation his top priority as Federal Reserve chairman. But a hesitancy to embrace the prospects of higher rates has set off a backlash. 4 min read The New York Times Trump Administration Zelensky and Netanyahu Meetings Air Force One Nuclear Energy Approval Rating LIVE July 30, 2026, 1:17 p.m. ET Trump Says He May Drop Blanche Nomination, for Now The confirmation of Todd Blanche, President Trump\u2019s pick for attorney general, was on hold because two key Republican senators threatened to vote \u201cno.\u201d Why Blanche Can\u2019t Win Republican Approval for Attorney General \u2014 Yet 6 min read Alex Kent/The New York Times ABC Formally Rebukes F.C.C. for Review of TV Licenses 3 min read Number of Uninsured Rises Sharply, Hospitals Report, Citing Obamacare Cuts 6 min read War in the Middle East Saudi Arabia Enters War The Whiplash War How the Cease-Fire Collapsed Iran and the Houthis LIVE July 30, 2026, 12:08 p.m. ET Drone Attack in Egypt Threatens to Expand Fighting The United States and Iran exchanged strikes overnight, after Saudi Arabia acknowledged direct involvement in the war for the first time. Attack on Egyptian Port Shows Suez Canal\u2019s Vulnerability 3 min read U.S. Sees Iran as Probably Behind Cyberattack on Minnesota Water Systems 3 min read 1:04 Julie Yoon and Alisa Shodiyev Kaff Russia-Ukraine War Attacks on Crimea Trump-Zelensky Relationship Ukraine\u2019s New Top General Patriot Missiles Photos Russia Hits Ukraine With Missiles and Drones, Killing at Least 8 President Volodymyr Zelensky of Ukraine said five members of a family had been killed in the latest large-scale aerial assault by Moscow. 3 min read NATO Says Russian Missile Hit Poland During Ukraine Attack 4 min read Roman Baluk/Reuters Got a Tip? The Times offers several ways to send important information confidentially \u203a Who Put Candy on a Diet? \u201cHealthy\u201d confections have overwhelmed grocery and convenience stores, and even found their way into gyms and supplement shops. 5 min read Scott Semler for The New York Times European Nations Agree to Boycott World Cup Over FIFA\u2019s Private Investment Plans The decision comes in response to a plan to sell significant stakes in FIFA\u2019s tournaments, with the 2027 women\u2019s World Cup next on the calendar. From The Athletic What Is the Financial Model Being Proposed by FIFA? From The Athletic Carl Recine/Getty Images The Great Read When His Church Goes Dark, They Talk to His Digital Twin To meet the growing needs of his church, Justin Lester created an A.I. duplicate of himself. Every day he sees more people using it. 14 min read Erin Schaff/The New York Times U.S. Immigration Crackdown ICE Arrests at Airports Labor Shortage Fears Fines for Immigrants Killing of Lorenzo Salgado Araujo Fatal Shooting of Johan Guerrero Justice Dept. Seeks to Deport Mother of Convicted ISIS Supporter The case is an early test of a court that has never been used before, where evidence is presented in secret and defendants cannot challenge how it was collected. 3 min read Dormant for 30 Years, a \u2018Zombie Court\u2019 for Terror Cases Awakens 7 min read Alex Kent/The New York Times Can Cuba Survive? Video: Adapting Without Fuel How Cubans Are Enduring Video: Struggle to Sleep Video: Children\u2019s Movie Night Throwing Parties Amid Blackouts 2:22 Jack Nicas, Alex Pena, Estelle Caswell, Thomas Vollkomme", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent, meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"AmazonBot\" has no root Disallow."}], "contactMatch": "https://help.nytimes.com/hc/en-us/articles/115015385887-Contact-The-New-York-Times", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "washingtonpost.com", "fetchedAt": "2026-07-30T17:24:06.963340Z", "result": {"base": "https://washingtonpost.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://washingtonpost.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "wsj.com", "fetchedAt": "2026-07-30T17:24:12.314892Z", "result": {"base": "https://wsj.com", "fatal": "Their server refused or timed out (status 401). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://wsj.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "usatoday.com", "fetchedAt": "2026-07-30T17:24:18.158766Z", "result": {"base": "https://usatoday.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"USA TODAY - Breaking News and Latest News Today\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "4204 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact link present.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 36, "title": "USA TODAY - Breaking News and Latest News Today", "textOnly": "USA TODAY - Breaking News and Latest News Today The problem with Costco? Parking rage. America's birthday \ud83c\udf82 Explore Marvel comics Check grocery prices \ud83d\uded2 Check home prices \ud83c\udfe0 U.S. Politics Sports Entertainment Life Money Travel Opinion Crossword ONLY AT USA TODAY: Newsletters For Subscribers Real Estate From the Archives USA TODAY PLAY Crossword Marvel Comics eNewspaper Magazines Investigations Video Podcasts Humankind Advertise Your Business WITNESS (True Crime) Just Curious Best-selling Booklist Top Workplaces Nomination Legals Press Releases Cars Grocery Shopping OUR PORTFOLIO: Real Estate 10BEST USA TODAY Store Shopping Blueprint WITNESS (True Crime) Southern Kitchen Home Internet Services This one problem is driving people away from Costco PGA Tour golfer hit with 6-month ban for violating anti-doping policy Inside Kyle Busch's family pursuit of son Brexton's racing dreams 'Inexcusable deception': RFK Jr., in exclusive interview, discusses Fauci diary This is 'happiest destination' in the US Skip Europe: These 5 US getaways deliver the same charm Nicole Kidman, Zoe Saldan\u0303a light up 'Lioness' premiere Featured Videos Your Daily PLAY Spider-Man has always had relationship drama. Just look\u2026 Read your daily horoscope for Thursday, July 30, 2026. How fast can you solve today\u2019s daily crossword puzzle? Start your Thursday strong with a clue to today's\u2026 More Top Stories Discover Her 12-year-old had an eating disorder. What happened\u2026 Dive into the weird and wonderful world of sharks Over 12,000 pounds of bacon recalled. See impacted\u2026 Best bars in America 2026: See the top 38 picks from USA TODAY Shopping Sports Entertainment Politics Pets $20K fish tank bursts at Chicago area pet store, flooding\u2026 See 9-year-old tear up after parents surprise him with\u2026 Why Wendy Ly says your cat\u2019s quirks deserve a Purrfect\u2026 Mark Ballas opens up about his dog Hendricks News in Pictures The next Busch: Kyle Busch's son Brexton thriving on the\u2026 Nicole Kidman, Zoe Saldan\u0303a light up 'Lioness' premiere Four DC statues restored with gold in $5 million project Buck moon lights up midsummer skies Trending Video Video shows missing woman before she disappeared in\u2026 Beloved bald eagle Jackie takes 'encouraging' step\u2026 Watch shark leap out from behind surfer at California\u2026 Watch daughters secretly join parents' anniversary trip Money Real Estate Find your dream home \u2014 start your search today Need an agent who listens? These pros get it. Get a real home value \u2014 not just a guess Historic home with ties to George Washington just hit the\u2026 USA TODAY 10BEST I asked a pro surfer how she stays safe from sharks\u2014her\u2026 Tom Holland has one drink order before Spider-Man's new\u2026 Gardening experts say these are the best veggies to plant\u2026 4 private jet CEOs share their best travel habits\u2014here's\u2026 Wellness Just Curious Opinion Travel Investigations Exclusive: Pentagon ramps up plans for military operation\u2026 Timothy Busfield was accused of sex abuse. Then things\u2026 With no competitor, Merck's blockbuster cancer drug hauls\u2026 A sheriff clings to power after a man froze to death in\u2026 U.S. News Tech For You Which topics matter most to you? Add your favorites and you\u2019ll see more of these stories in the For You section. View For You About Us Newsroom Staff Ethical Principles USA TODAY Network Style Guide Responsible Disclosure Request a Correction Media Room Accessibility Sitemap Subscription Terms & Conditions Advertiser Acceptance Policy Terms of Service Privacy Policy California Notice at Collection Do Not Sell or Share My Personal Information Your Privacy Choices Contact Us Help Center Manage Account Give Feedback Get Home Delivery eNewspaper USA TODAY Shop USA TODAY Print Editions Licensing & Reprints Advertise Your Business Careers Internships Support Local Business News Tips Submitting letters to the editor Podcasts Newsletters Mobile Apps Facebook X Instagram LinkedIn Threads YouTube Reddit Flipboard Real Estate 10BEST USA TODAY Store Shopping Deals Cashback Best-selling Booklist Sou", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AmazonBot\nDisallow: /"}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "latimes.com", "fetchedAt": "2026-07-30T17:24:25.364759Z", "result": {"base": "https://latimes.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Blocks training crawlers only (GPTBot, ClaudeBot, Bytespider) -- a legitimate opt-out; retrieval/citation access stays open.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"News from California, across the nation and world - Los Ange\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.latimes.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "22278 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:?body=SoCal%27s%20new%20Harry%20Potter%20train%20ride%20attracts%20fans%2C%20concerns%20local%20activists%0A%0Ahttps%3A%2F%2Fwww.latimes.com%2F0000019f-a5c2-d0bf-a3ff-bdde60bc0000-123).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 69, "title": "News from California, across the nation and world - Los Angeles Times", "textOnly": "News from California, across the nation and world - Los Angeles Times Advertisement News Home Page California Election 2024 Housing &amp; Homelessness Politics Science &amp; Medicine World &amp; Nation Business Artificial Intelligence Autos Jobs, Labor &amp; Workplace Real Estate Technology and the Internet California California Politics Earthquakes Education Housing &amp; Homelessness L.A. Influential L.A. Politics Mental Health Climate &amp; Environment Climate Change Water &amp; Drought Entertainment &amp; Arts Arts Books Stand-Up Comedy Hollywood Inc. The Envelope (Awards) Movies Music Television Things to Do De Los En Espa\u00f1ol Food 101 Best Restaurants in L.A. Recipes Image Art &amp; Culture Conversations Drip Index: Event Guides Fashion Shopping Guides Styling Myself Lifestyle Health &amp; Wellness Home Design L.A. Affairs Plants Travel &amp; Experiences Weekend Things to Do in L.A. Obituaries Voices Editorials Letters to the Editor Contributors Short Docs Sports Angels Angel City FC Chargers Clippers Dodgers Ducks Galaxy High School Sports Kings Lakers Olympics USC UCLA Rams Sparks World &amp; Nation Immigration &amp; the Border Mexico &amp; the Americas Middle East Ukraine Times Everywhere 404 by L.A. Times Facebook Instagram LA Times Today Newsletters Photography Podcasts Short Docs TikTok Threads Video YouTube X (Twitter) For Subscribers eNewspaper All Sections _________________ LA Times Studios BUSINESS \u2022 AI &amp; Tech \u2022 Automotive \u2022 Banking &amp; Finance \u2022 Commercial Real Estate \u2022 Entertainment \u2022 Goods &amp; Retail \u2022 Innovators Unplugged \u2022 Healthcare &amp; Science \u2022 Law \u2022 Sports \u2022 Orange County HOT PROPERTY \u2022 Decor &amp; Design \u2022 Homes For Sale LIVE &amp; WELL \u2022 Fitness \u2022 Beauty \u2022 Body \u2022 Longevity TRAVEL \u2022 Trip Ideas \u2022 Travel Destinations \u2022 Travel Themes \u2022 Travel Experiences THE HUB: REBUILDING LA DOCTORS &amp; SCIENTISTS DEALS &amp; COUPONS NEWSLETTERS Live Stream Events Screening Series Crossword Games L.A. Times Store Subscriptions Manage Subscription EZPAY Delivery Issue eNewspaper Group Subscriptions Subscribe Subscriber Terms Gift Subscription Terms About Us About Us Archives Company News eNewspaper For the Record Got a Tip? L.A. Times Careers L.A. Times Store LA Times Studios Capabilities News App: Apple IOS News App: Google Play Newsroom Directory Public Affairs Rights, Clearance &amp; Permissions Short Docs Advertising Classifieds Find/Post Jobs Hot Property Sections Local Ads Marketplace L.A. Times Digital Agency Media Kit: Why the L.A. Times? Place an Ad Place an Open House Sotheby\u2019s International Realty Special Supplements Healthy Living Higher Education Philanthropy Sections Show Search Search Query Submit Search California Topics California Politics Housing &amp; Homelessness Earthquakes Education Orange County All California News Latino Life De Los The Latinx Files L.A. Times en Espa\u00f1ol Kiosco Digital Voices Gustavo Arellano Anita Chabria Steve Lopez Patt Morrison George Skelton Newsletters Essential California This Evening\u2019s Big Stories California Politics TimesOC News Alerts More L.A. Influential LA Times Today Photography Podcasts Video Entertainment Topics Television Movies Music Hollywood Inc. L.A. Arts Books The Envelope Oscars Emmys Grammys Newsletter Video Podcast Glenn Whipp Voices Robert Lloyd Mary McNamara Charles McNulty Mark Swed Mikael Wood Newsletters Entertainment Daily Essential Arts The Wide Shot Screen Gab Book Club More Festival of Books Behold L.A. Times Entertainment Sports Teams &amp; Sports Dodgers Lakers Rams USC Kings Sparks Angels Clippers Chargers UCLA Soccer High School Sports Voices Bill Plaschke Bill Shaikin Mirjam Swanson Newsletters The Sports Report Dodgers Dugout Lakers! Prep Rally Times of Troy UCLA Unlocked More World Cup 2026 Olympics Sports Videos L.A. Times Sports Lifestyle Topics Home &amp; ADUs Plants Wellness L.A. Affairs West Coast Experiences Things to Do Neighborhood Guides L.A. things to do for under $20 Outdoors Newsletters Mr. Todd\u2019s Wild Ride The Wild ", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:?body=SoCal%27s%20new%20Harry%20Potter%20train%20ride%20attracts%20fans%2C%20concerns%20local%20activists%0A%0Ahttps%3A%2F%2Fwww.latimes.com%2F0000019f-a5c2-d0bf-a3ff-bdde60bc0000-123", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "nypost.com", "fetchedAt": "2026-07-30T17:24:34.835522Z", "result": {"base": "https://nypost.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"New York Post \u2013 Breaking News, Top Headlines, Photos & Video\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://nypost.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "27841 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://nypost.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "New York Post \u2013 Breaking News, Top Headlines, Photos & Videos", "textOnly": "New York Post \u2013 Breaking News, Top Headlines, Photos & Videos Primary Menu Sections US News Metro Long Island Politics World News Page Six Sports NFL MLB Olympics NBA NHL College Football College Basketball WNBA Post Sports+ Sports Betting Business Opinion Entertainment TV Movies Music Celebrities Awards Theater Shopping Lifestyle Weird But True Sex &amp; Relationships Viral Trends Human Interest Parenting Fashion &amp; Beauty Food &amp; Drink Travel Health Wellness Fitness Health Care Medicine Men&#8217;s Health Women&#8217;s Health Mental Health Nutrition Real Estate Alexa Media Tech Science Space Environment Wildlife Archaeology Astrology Video Photos Pod Force One NY POSTcast Today&#8217;s Paper Covers Columnists Horoscopes Crosswords &amp; Games Sports Odds Podcasts Careers Email Newsletters Official Store Home Delivery Tips Search Email New York Post Page Six Sports Metro Opinion Podcasts Business Shopping Switch between CA and NY editions here. Got it Edition CA NY Search Search New! Interview with Host, Culture Apothecary Podcast, Alex Clark. LISTEN NOW Celebrate the Knicks first championship in 53 years! SHOP NOW Full price?! In this economy? Shop smarter with Post Wanted. SIGN UP &#039;It&#039;s too much&#039; exclusive Heartbroken mom of 12-year-old killed by stray bullet in The Bronx says NYC has only gotten safer for criminals The anguished mother of a 12-year-old boy killed by a stray bullet outside a Bronx bodega said the city has gotten more dangerous because criminals no longer fear consequences, and that her son&#8217;s murder is the latest example. exclusive UEFA nations vote to boycott World Cup over FIFA boss\u2019s $20B cash grab \u2014 throwing global soccer into chaos breaking \u2018Hammer of the Gods\u2019 new secret weapon upgrades US Army power to blast enemy missile attacks First case of untreatable, potentially fatal Bourbon virus confirmed in New York Advertisement exclusive photos Best buds Limelight-loving Fauci all smiles as he\u2019s surprised by flowers left by adoring fan \u2014 after getting savaged by Senate Republicans Dr. Anthony Fauci beamed with delight Thursday as he collected flowers from an admirer placed on the front steps of his $2.4 million Washington, DC home. exclusive photos Angry Fauci drops F-bomb during walk with wife hours after Senate grilling Advertisement breaking Trump may yank Todd Blanche for AG nominee over GOP opposition to weaponization fund \u2014 on one condition Dad of Nolan Wells\u2019 \u2018best friend\u2019 offers massive reward for info \u2014 as athlete\u2019s parents take shot at son\u2019s pal NFL great reveals Parkinson\u2019s diagnosis at 49 Advertisement exclusive Whodunnit? Netflix sued for $105M by billionaire movie producer after his unreleased Nicolas Cage thriller goes missing A Netflix exec wrote as part of an email Opinion Common sense. No nonsense. See All editorial Anthony Fauci\u2019s Fifth Amendment mockery demands the Senate\u2019s contempt Daniel McCarthy DSA radicals trying to centralize power by scrapping the Senate are completely ignorant of our history editorial NYC\u2019s education chiefs just don\u2019t care about teaching kids to read See All Clancy on trial Cops describe finding children in basement, recall dad screaming Lindsay Clancy \u2018killed the f\u2013king kids!\u2019 Follow live coverage of the Lindsay Clancy trial Thursday in Massachusetts, as the mom stands accused of murdering her three young children while suffering from postpartum psychosis in 2023. US travel advisory urges tourists to \u2018exercise increased caution\u2019 in Caribbean vacation hotspot Ben Affleck wins elusive $1M prize on \u2018Who Wants to Be a Millionaire\u2019 \u2014 would you have gotten the winning question? exclusive \u2018RHOC\u2019 alum shares devastating tongue cancer prognosis after dentist\u2019s \u2018huge mistake\u2019 delayed diagnosis Costly kiss Law partner caught canoodling with younger associate in Central Park could lose millions as lucrative offer reneged The married lawyer caught in a steamy smoochfest with his much younger colleague in Central Park could lose tens of millions of", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: claude-user\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: amazonbot\nDisallow: /"}], "contactMatch": "https://nypost.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "cnn.com", "fetchedAt": "2026-07-30T17:24:43.944534Z", "result": {"base": "https://cnn.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Breaking News, Latest News and Videos | CNN\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.cnn.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1116842 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 44, "title": "Breaking News, Latest News and Videos | CNN", "textOnly": "Breaking News, Latest News and Videos | CNN @font-face { font-family: \"cnn_sans_condensed\"; font-weight: 700; src: url(/fonts/cnn/cnn_sans_condensed-bold.woff2); } @font-face { font-family: \"cnn_sans_condensed\"; font-weight: 500; src: url(/fonts/cnn/cnn_sans_condensed-medium.woff2); } @font-face { font-family: \"cnn_sans_condensed\"; src: url(/fonts/cnn/cnn_sans_condensed.woff2); } @font-face { font-family: \"cnn_sans_display\"; font-weight: 700; src: url(/fonts/cnn/cnn_sans_display-bold-v1.woff2); } @font-face { font-family: \"cnn_sans_display\"; font-weight: 500; src: url(/fonts/cnn/cnn_sans_display-medium-v1.woff2); } @font-face { font-family: \"cnn_sans_display\"; src: url(/fonts/cnn/cnn_sans_display-v1.woff2); } @font-face { font-family: \"helveticaneue\"; src: url(/fonts/cnn/helveticaneue-v1.woff2); } @font-face { font-family: \"noto_sans_arabic\"; src: url(/fonts/cnn/noto_sans_arabic-v1.ttf); } @font-face { font-family: \"noto_sans_arabic\"; src: url(/fonts/cnn/noto_sans_arabic-v1.woff2); } @font-face { font-family: \"noto_serif_display\"; src: url(/fonts/cnn/noto_serif_display-v1.woff2); } @font-face { font-family: \"noto_serif_semi_condensed\"; src: url(/fonts/cnn/noto_serif_semi_condensed-v1.woff2); } @font-face { font-family: \"noto_serif\"; src: url(/fonts/cnn/noto_serif-bold_italic-v1.woff2); } @font-face { font-family: \"noto_serif\"; font-weight: 700; src: url(/fonts/cnn/noto_serif-bold-500.woff2); } @font-face { font-family: \"noto_serif\"; font-weight: 700; src: url(/fonts/cnn/noto_serif-bold-v1.woff2); } @font-face { font-family: \"noto_serif\"; src: url(/fonts/cnn/noto_serif-italic-v1.woff2); } @font-face { font-family: \"noto_serif\"; font-weight: 300; src: url(/fonts/cnn/noto_serif-light.woff2); } @font-face { font-family: \"noto_serif\"; src: url(/fonts/cnn/noto_serif-v1.woff2); }.subscribe-button{color:#fff;cursor:pointer;font-family:var(--theme-font-family);background-color:#c00;border-width:0;border-radius:6px;height:30px;padding:0 12px;font-size:14px;font-weight:700;line-height:20px}.subscribe-button--hide{display:none}.subscribe-button:hover{background-color:#a4001e}.footer{background-color:var(--theme-footer-background-color);border-top:1px solid var(--theme-divider);padding:48px 0 24px}.footer .brand-logo__logo{width:var(--theme-footer__brand-logo-width);height:var(--theme-footer__brand-logo-height);display:block}.footer .brand-logo__icon{margin-top:var(--theme-footer__brand-logo-icon-margin-top)}.footer .brand-logo .cnn-logo-dark{width:100%;height:100%}.footer .brand-logo__theme path{fill:var(--theme-footer-copy-accent)}.footer__brand-logo{margin-bottom:32px}.footer__right{align-items:center;margin-bottom:7px;display:flex;position:relative}.footer__user-account-icon{margin:0 0 0 16px;padding:0}.footer__user-account-nav-mobile{margin:0;display:block}.footer__user-account-nav-mobile .user-account-nav__menu-link{font-size:16px}.footer__user-account-nav-desktop{display:none}.footer .footer__user-account-nav-icon .user-account-nav.user-account-nav--unauth .user-account-nav__icons .user-account-nav__text-button.user-account-nav__text-button--unauth{color:var(--theme-footer__login-button);border:none;padding-left:12px;padding-right:12px;line-height:24px}.footer .footer__inner{width:100%;max-width:1440px;margin:0 auto;padding:0 16px}.footer .footer__subnav{flex-wrap:wrap;display:flex}.footer .footer__subnav .subnav__section-link{font-size:16px}.footer .footer__subnav .subnav__section-link,.footer .footer__subnav .subnav__subsection-link{color:var(--theme-footer-copy-accent)}.footer .footer__subnav .subnav__section-link:hover,.footer .footer__subnav .subnav__section-link:focus,.footer .footer__subnav .subnav__section-link:active,.footer .footer__subnav .subnav__subsection-link:hover,.footer .footer__subnav .subnav__subsection-link:focus,.footer .footer__subnav .subnav__subsection-link:active{color:var(--theme-footer-copy-accent-hover)}.footer__divider--hide{display:none}.footer .footer__divider{background-color:var(--theme-divider);bor", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AI2Bot, Ai2Bot-Dolma, AliyunSecBot, Amazonbot, amzn-searchbot, amzn-user, anthropic-ai, Applebot-Extended, Archive.org_bot, AwarioRssBot, AwarioSmartBot, Brightbot 1.0, Bytespider, CCBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, cohere-ai, cohere-training-data-crawler, Crawlspace, DataForSeoBot, Diffbot, DuckAssistBot, EchoboxBot, Exabot, FacebookBot, FriendlyCrawler, GPTBot, Google-Extended, GoogleOther, GoogleOther-Image, GoogleOther-Video, IAB-Tech-Lab, iaskspider/2.0, ICC-Crawler, img2dataset, ISSCyberRiskCrawler, ImagesiftBot, Kangaroo Bot, magpie-crawler, MistralAI-user, MyCentralAIScraperBot, NewsNow, news-please, OAI-SearchBot, omgili, omgilibot, PanguBot, Panscient, PiplBot, PerplexityBot, Perplexity-User, PetalBot, Poseidon Research Crawler, QuillBot, quillbot.com, Quora-Bot, SBIntuitionsBot, Scrapy, SeekrBot, SemrushBot-OCOB, SemrushBot-SWA, SeznamHomepageCrawler, Sidetrade indexer bot, TaraGroup Intelligence Bot, Timpibot, TurnitinBot, VelenPublicWebCrawler, ViennaTinyBot, Webzio-Extended, YandexAdditional, YandexAdditionalBot, YouBot\nDisallow: /"}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "foxnews.com", "fetchedAt": "2026-07-30T17:24:49.532009Z", "result": {"base": "https://foxnews.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (2191 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: NewsMediaOrganization, WebSite, WebPage,CollectionPage, VideoObject.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Fox News - Breaking News Updates | Latest News Headlines | P\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.foxnews.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "25821 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.foxnews.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "Fox News - Breaking News Updates | Latest News Headlines | Photos &amp; News Videos", "textOnly": "Fox News - Breaking News Updates | Latest News Headlines | Photos &amp; News Videos Fox News Media Fox News Fox Business Fox Nation Fox News Audio Fox Weather Outkick Fox Noticias Books Fox News U.S. Blue City Crime Campus Radicals Border Crisis True Crime Politics Donald Trump Senate House Judiciary Foreign Policy Fox News Polls Elections World U.N. Conflicts Terrorism Disasters Global Economy Environment Religion Scandals Opinion Media Fox Nation Coverage Entertainment Taylor Swift Royal Families Country Music Sports FIFA WORLD CUP 26 NFL College Football MLB NASCAR INDYCAR NBA NHL Golf Tennis UFC WWE OutKick Lifestyle Travel Food &amp; Drink Faith Discovery Games &amp; Quizzes Health Longevity Brain Health Weight Loss Sleep Health Cancer News Mental Health Heart Health Healthy Living Video AI More Toggle search Log In Watch TV Menu Home WAR WITH IRAN FAUCI FALLOUT 'WOKE' SMITHSONIAN NANCY GUTHRIE Hot Topics WAR WITH IRAN FAUCI FALLOUT 'WOKE' SMITHSONIAN NANCY GUTHRIE SP500 I:COMP I:DJI Games Expand/Collapse Menu Scattergories Daily 5 Across the Letter Word Game DownWords Sudoku Swap Words Word Search Icon Match Daily Crossword Puzzle Mini Crossword Puzzle Deals Expand/Collapse Menu Trending Deals Gift Guides Home Deals Kitchen Deals Tech and Electronic Deals Health and Fitness Deals Beauty and Fashion Deals FOX Business Expand/Collapse Menu Economy Personal Finance Markets Watchlist Lifestyle Real Estate Tech Science &amp; Tech Expand/Collapse Menu Air &amp; Space Security Innovation Video Games Military Tech Planet Earth Wild Nature Watch Live Expand/Collapse Menu FOX News Shows Programming Schedule On Air Personalities Full Episodes Show Clips News Clips Apps &amp; Products Expand/Collapse Menu FOX News Go FOX Weather FOX Nation FOX Noticias FOX News Shop FOX News Radio FOX News Newsletters FOX News Podcasts FOX One U.S. Expand/Collapse Menu Crime Immigration Terror True Crime Luigi Mangione Tyler Robinson Campus Radicals Politics Expand/Collapse Menu Donald Trump Senate House Judiciary Foreign Policy Fox News Polls Elections World Expand/Collapse Menu U.N. Conflicts Terrorism Disasters Global Economy Environment Religion Scandals Opinion Media Expand/Collapse Menu FOX Nation Coverage Entertainment Expand/Collapse Menu Taylor Swift Royal Families Country Music Sports Expand/Collapse Menu FIFA WORLD CUP 26 NFL College Football MLB NASCAR INDYCAR NBA NHL Golf Tennis UFC WWE OutKick Lifestyle Expand/Collapse Menu Travel Food &amp; Drink Faith Discovery Games &amp; Quizzes Health Expand/Collapse Menu Longevity Brain Health Weight Loss Sleep Health Cancer News Mental Health Heart Health Healthy Living Deals Expand/Collapse Menu Trending Deals Gift Guides Home Deals Kitchen Deals Tech and Electronic Deals Health and Fitness Deals Beauty and Fashion Deals FOX Business Expand/Collapse Menu Personal Finance Economy Markets Watchlist Lifestyle Real Estate Tech Science &amp; Tech Expand/Collapse Menu Air &amp; Space Security Innovation Planet Earth Wild Nature Video Games Military Tech Games Expand/Collapse Menu Scattergories Daily 5 Across the Letter Word Game DownWords Sudoku Word Swap Word Search Icon Match Daily Crossword Puzzle Mini Crossword Puzzle Watch Live Expand/Collapse Menu FOX News Shows Programming Schedule On Air Personalities Watch Live Full Episodes Show Clips News Clips Apps &amp; Products Expand/Collapse Menu FOX News Go FOX Weather FOX Nation FOX Noticias FOX News Shop FOX News Radio FOX News Newsletters FOX News Podcasts FOX One About Expand/Collapse Menu Contact Us Careers FOX Around the World Advertise With Us Media Relations Corporate Information Compliance Facebook Twitter Instagram Youtube Flipboard LinkedIn Slack RSS Newsletters Spotify iHeartRadio Fox News Terms of Use New Privacy Policy Your Privacy Choices Closed Caption Policy Help Contact Us Accessibility Statement News Sitemap Video Sitemap This material may not be published, broadcast, rewritten, or redistributed. \u00a92026 FOX News Network, LLC. All ri", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.foxnews.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "nbcnews.com", "fetchedAt": "2026-07-30T17:24:56.268270Z", "result": {"base": "https://nbcnews.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: CollectionPage, Dataset, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"NBC News - Breaking Headlines and Video Reports on World, U.\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.nbcnews.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "15887 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:?subject=HEADLINE&body=URL).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "NBC News - Breaking Headlines and Video Reports on World, U.S. and Local Angles | NBC News", "textOnly": "NBC News - Breaking Headlines and Video Reports on World, U.S. and Local Angles | NBC News BREAKING: Apalachee High School shooter\u2019s father sentenced to 15 years IE 11 is not supported. For an optimal experience visit our site on another browser. Skip to Content NBC News Logo Politics U.S. News World Local New York Los Angeles Chicago Dallas-Fort Worth Philadelphia Washington, D.C. Boston Bay Area South Florida San Diego Connecticut Sports Business Health SHOPPING Tipline Subscribe Share & Save&nbsp;&mdash; Subscriber Hub Saved Newsletters Profile Subscription Preferences Search Search Sections U.S. News Politics World Business Sports Investigations Culture &amp; Trends Health Science Tech Weather VIDEO Photos Sudoku NBC Selected NBC Asian America NBC BLK NBC Latino NBC OUT Local New York Los Angeles Chicago Dallas-Fort Worth Philadelphia Washington, D.C. Boston Bay Area South Florida San Diego Connecticut tv Today Nightly News Meet the Press Dateline Featured NBC News Now Nightly Films Stay Tuned Special Features Newsletters Podcasts Listen Now Business in America More From NBC NBC.COM NBCU Academy Peacock NEXT STEPS FOR VETS NBC News Site Map Help Follow NBC News news Alerts There are no new alerts at this time Search Search Facebook Twitter Email SMS Blue Sky Whatsapp Print Reddit Flipboard Pinterest Linkedin Latest Stories Politics U.S. News World Sports Business Health SHOPPING Tipline LIVE Trump admin Iran war Todd Blanche Europe wildfires Pope Leo \u2018Children of Blood and Bone\u2019 UEFA boycott Tipline Advertisement Trump threatens to pull Todd Blanche\u2019s AG nomination until after GOP holdouts leave office The Senate Judiciary Committee postponed a vote on Blanche\u2019s nomination over objections from Sens. John Cornyn of Texas and Thom Tillis of North Carolina. Trump threatens to pull Todd Blanche\u2019s AG nomination until after GOP holdouts leave office The Senate Judiciary Committee postponed a vote on Blanche\u2019s nomination over objections from Sens. John Cornyn of Texas and Thom Tillis of North Carolina. LIVE Cornyn says there are more Republicans concerned about Trump IRS settlement GOP senators hesitant to back Blanche due to Trump IRS settlement GOP senators hesitant to back Blanche due to Trump IRS settlement LIVE Cornyn says there are more Republicans concerned about Trump IRS settlement BREAKING Apalachee High School shooter\u2019s dad sentenced to 15 years Colin Gray gifted his son an assault rifle that the teen used to kill 4 and wound 9 in a mass shooting at the Georgia school in 2024. BREAKING Apalachee High School shooter\u2019s dad sentenced to 15 years Colin Gray gifted his son an assault rifle that the teen used to kill 4 and wound 9 in a mass shooting at the Georgia school in 2024. Abbey Cutrer / Atlanta Journal-Constitution via AP, Pool Iran vows to \u2018punish the aggressor today\u2019 after U.S. unleashes \u2018heavy wave\u2019 of strikes The conflict appeared to draw in new countries, as Egypt confirmed a drone hit two natural gas vessels. Iran vows to \u2018punish the aggressor today\u2019 after U.S. unleashes \u2018heavy wave\u2019 of strikes The conflict appeared to draw in new countries, as Egypt confirmed a drone hit two natural gas vessels. FOR SUBSCRIBERS Trump \u2018exasperated\u2019 on Iran as top officials can\u2019t agree on strategy, sources say FOR SUBSCRIBERS Sens. Sanders and Grassley push to fine Pentagon if it fails an audit amid Iran spending U.S. hits dozens of targets in overnight strikes U.S. hits dozens of targets in overnight strikes FOR SUBSCRIBERS Trump \u2018exasperated\u2019 on Iran as top officials can\u2019t agree on strategy, sources say FOR SUBSCRIBERS Sens. Sanders and Grassley push to fine Pentagon if it fails an audit amid Iran spending FOR SUBSCRIBERS Robot waiters and drone deliveries: Inside China\u2019s booming tech hub SHENZHEN, China \u2014 Long a symbol of China\u2019s economic surge, Shenzhen is embracing a high-tech, low-altitude future. FOR SUBSCRIBERS Robot waiters and drone deliveries: Inside China\u2019s booming tech hub SHENZHEN, China \u2014 Long a symbol of China\u2019s ", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent, meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider, ByteSpider Desktop\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot, Perplexity-User\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-Web, Claude-User, Claude-SearchBot\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot, AmazonBot\nDisallow: /"}], "contactMatch": "mailto:?subject=HEADLINE&body=URL", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "cbsnews.com", "fetchedAt": "2026-07-30T17:25:02.121572Z", "result": {"base": "https://cbsnews.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Blocks training crawlers only (GPTBot) -- a legitimate opt-out; retrieval/citation access stays open.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "4 valid JSON-LD block(s) -- types: NewsMediaOrganization, WebSite, BreadcrumbList, CollectionPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"CBS News | Breaking news, top stories &amp; today's latest h\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.cbsnews.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "29868 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact link present.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "CBS News | Breaking news, top stories &amp; today's latest headlines", "textOnly": "CBS News | Breaking news, top stories &amp; today's latest headlines Latest U.S. Iran War Blanche Dispute World Politics Entertainment HealthWatch MoneyWatch Crime Space Sports Brand Studio The Free Press Local News Atlanta Baltimore Bay Area Boston Chicago Colorado Detroit Los Angeles Miami Minnesota New York Philadelphia Pittsburgh Sacramento Texas Live CBS News 24/7 Baltimore Bay Area Boston Chicago Colorado Detroit Los Angeles Miami Minnesota New York Philadelphia Pittsburgh Sacramento Texas 48 Hours 60 Minutes Shows 48 Hours 60 Minutes CBS Evening News CBS Mornings CBS Morning News CBS Reports CBS Saturday Morning The Daily Report The Dish Face the Nation Sunday Morning The Takeout Things That Matter The Uplift CBS News Investigates CBS News Confirmed Podcasts Newsletters Download Our App CBS News Team Executive Team Brand Studio Paramount+ Join Our Talent Community RSS Feeds U.S. Iran War Blanche Dispute World Politics HealthWatch MoneyWatch Entertainment Crime Sports The Free Press Watch CBS News Live Iran war appears to expand to Egypt's coast as drones hit gas port Drone strikes hit gas vessels at an Egyptian port on the Mediterranean, as a poll shows little American support for the five-month war with Iran. updated 40M ago Trump says he may pull Blanche's nomination until 2 Senate critics leave office President Trump said Todd Blanche will remain acting attorney general as a holdup on his nomination continues. updated 21M ago Sen. Paul: \"Very real chance\" Fauci could be prosecuted after Senate hearing Hours after Dr. Anthony Fauci invoked the Fifth Amendment in response to all questions during a Senate hearing, Republican Sen. Rand Paul of Kentucky told CBS News he believes Fauci could be prosecuted. 18H ago U.S. economy slowed in second quarter as Iran war weighs on growth Dconomic activity weakened between April and June as energy costs surged, new government data shows. 3H ago Father convicted in Apalachee High School shooting gets 15 years Days after his son was sentenced, the father of the Apalachee High School shooter Colt Gray, is in court for his own sentencing hearing. updated 1M ago Possible Russian missile crashes in Poland during Russian attacks on Ukraine Russian attacks across Ukraine killed at least eight people and an apparent Russian missile crashed in Poland, officials said. 4H ago Lindsay Clancy jury hears from first responders from day 3 kids died Ten first responders from the day Lindsay Clancy killed her children took the stand during her Massachusetts murder trial. updated 1M ago U.K. village plans independence vote in protest over migrant housing plans The English village of Piddington is planning a symbolic referendum to demand independence from the U.K. over plans to house people seeking asylum at a disused military barracks. 2H ago First on CBS Trump-affiliated companies distance themselves from Trump-IRS deal Top Senate Democrats sought answers about Trump IRS deal from mining company Kaz Resources, cryptocurrency companies World Liberty Financial and prediction markets Polymarket and Kalshi. 20H ago CBS News Live CBS News Live Video Live More Top Stories European teams boycott FIFA over plan to sell World Cup stake to investors A source close to Joshua Kushner's firm tells CBS News there's no politics behind FIFA's bid to draw private investment, but Europe's soccer powerhouses are crying foul. 43M ago Battle against monster wildfire in France boosted by improving weather George and Amal Clooney are among thousands of residents in southwest France still displaced by a massive wildfire, but crews are gaining control. 1H ago Canelo &Aacute;lvarez says there's \"not much left\" for him to do in boxing Professional boxer Sa&uacute;l \"Canelo\" &Aacute;lvarez spoke to \"CBS Mornings\" co-host Nate Burleson about when he'll know to walk away from the sport. 3H ago Japan earthquake death toll climbs to 34 as hopes wane for more rescues The Kumamoto prefectural government said at least 34 people were k", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "abcnews.go.com", "fetchedAt": "2026-07-30T17:25:07.865309Z", "result": {"base": "https://abcnews.go.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"ABC News - Breaking News, Latest News and Videos\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://abcnews.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "11981 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://abcnews.com/Contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "ABC News - Breaking News, Latest News and Videos", "textOnly": "ABC News - Breaking News, Latest News and Videos ABC News Live Video Local News Shows Good Morning America ShopGMA Interest Successfully Added We&#x27;ll notify you here with news about Turn on desktop notifications for breaking stories about interest? Off On Stream on Blanche AG nomination &#x27;Diddy&#x27; release date moved US economy Live Updates Live Updates Iran GLP-1s School shooting sentencing World Cup brawl Chip Somodevilla/Getty Images Politics Trump floats temporarily withdrawing Blanche&#x27;s AG nomination over GOP holdouts Sen. John Cornyn said there were &quot;new complications this morning.&quot; Live Live ABC News Live ABC News Live 1:30 PM ABC News Live 2:00 PM ABC News Live 2:30 PM ABC News Live 3:00 PM U.S. Dad sentenced to 15 years in prison in school shooting carried out by teenage son 7 minutes ago Lifestyle Sean &#x27;Diddy&#x27; Combs&#x27; prison release date moved up, records show Business US economy slowed more than expected as the Iran war took hold International Iran live updates: Egypt confirms Mediterranean port fire was caused by drone Lifestyle Women say they got pregnant after taking GLP-1 drugs Lifestyle Hero teen lifeguard speaks out about viral rescue of 10-year-old boy 2 hours ago U.S. Arizona parents won&#x27;t face charges after toddler found alive in hospital morgue Health Ebola deaths surpass 1,500 in DRC as outbreak continues to spread 1 hour ago Top Stories Reported grenade safely removed in Philadelphia Russia violated NATO airspace in Poland, Kyiv says Fire captain and family killed in flood honored Former NFL star reveals Parkinson&#x27;s diagnosis Howard president defends enrollment process Wildfires force Clooneys to evacuate France home FIFA announces World Cup disciplinary proceedings A 150-year-overdue book resurfaces in a library Trending Videos 1:49 U.S. Escaped NYC owl Flaco gets tribute in new documentary 6:46 Nightline &#x27;The Brady Experience&#x27; offers tours of iconic TV home 0:21 WNT Alleged road rage incident in Washington state 1:22 WNT Renovations uncover wallet lost decades ago with cherished family photos inside 9:08 U.S. Howard University president on enrollment scandal 3:37 Entertainment &#x27;Jumanji,&#x27; BTS and wedding drama top the pop culture buzz 1:39 U.S. Older Americans over 60 lost nearly $8 billion to scammers in 2025 1:07 GMA New Mega Millions winner overnight 4:09 GMA Latest fitness fashion trends 2:25 GMA Can a spreadsheet help you find love? 1:38 U.S. Pepper-spraying drones to help stop school shooters sparks debate 1:25 U.S. EBay to pay nearly $56M to couple over stalking lawsuit Playlist \u00b7 12 Videos Politics ABC News Israel &#x27;It&#x27;s his decision&#x27;: Netanyahu says choice on how to proceed with Iran is Trump&#x27;s Jeffrey Epstein case Ex-Goldman Sachs lawyer told panel that Epstein made her firm &#x27;millions&#x27; of dollars Travel Trump unveils $22.5 billion plan to overhaul Dulles International Airport Trump Administration Senate committee advances nomination of Dr. Erica Schwartz to be Trump&#x27;s CDC director Manhattan DA challenges Trump&#x27;s appeal of his 2024 hush money conviction Oil prices surge after Trump vows retaliation for Iranian attack Americans are down on the economy and concerned about costs: Polls Trump asks SCOTUS to overturn $83M Carroll judgment based on presidential immunity Coronavirus Republican senators vow Fauci contempt vote after hearing on COVID-19 origins Senators still working with SEC, Big Ten on issues involved in college sports bill DOJ sought input on Comey case from Sammy &#x27;The Bull&#x27; Gravano, defense says Secret Service employee facing attempted manslaughter in alleged hazing incident Michigan AG investigating child services swatting call involving Buttigieg, his kids War with Iran 4:44 Middle East tensions raise fears of a wider regional conflict 1:41 US launches new retaliatory strikes on Iran after attack on US forces 16:33 Israeli Prime Minister Benjamin Netanyahu spoke wi", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://abcnews.com/Contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "msnbc.com", "fetchedAt": "2026-07-30T17:25:13.733559Z", "result": {"base": "https://msnbc.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (11129 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, ImageObject, BreadcrumbList, WebSite, NewsMediaOrganization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"MS NOW (formerly MSNBC) - US News Today | Breaking News\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.ms.now/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7518 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:?subject=%5BShared%20Post%5D%20Homepage%20New&#038;body=https%3A%2F%2Fwww.ms.now%2F&#038;share=email).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 75, "title": "MS NOW (formerly MSNBC) - US News Today | Breaking News", "textOnly": "MS NOW (formerly MSNBC) - US News Today | Breaking News Opinion Morning Joe RacheL Maddow Deadline: White House The weekend NEWSLETTERS Live TV Featured Shows the rachel maddow show mondays 9pm et morning joe weekdays 6am et deadline: white house with nicolle wallace weekdays 4pm et the beat with ari melber weeknights 6pm et the weeknight weeknights 7pm et all in with chris hayes weeknights 8pm et the briefing with jen psaki tuesdays &#8211; fridays 9pm et the last word with lawrence o&#8217;donnell weeknights 10pm et the 11th hour with ali velshi weeknights 11pm et More Shows Way Too Early with Ali Vitali Connect with Jacob Soboroff The Weekend Money, Power, Politics with Stephanie Ruhle The Moment with Katy Tur Alex Witt Reports On the Line with Alicia Menendez PoliticsNation with Al Sharpton MS NOW Tv Watch Live Listen Live More MS NOW Live Events MS NOW Columnists TV Schedule MS NOW Newsletters Podcasts Transcripts Help Follow MS NOW Facebook Instagram X Mail Search Homepage New Share this &#8211; Share on Facebook (Opens in new window) Facebook Share on X (Opens in new window) X Share on Mail (Opens in new window) Mail Share on Print (Opens in new window) Print Share on WhatsApp (Opens in new window) WhatsApp Share on Reddit (Opens in new window) Reddit Flipboard Share on Pinterest (Opens in new window) Pinterest Share on LinkedIn (Opens in new window) LinkedIn Together In Texas LIVE Elections White House Congress Supreme Court Culture World Science &amp; Tech Civil Rights Business Sports As Trump\u2019s approval rating reaches embarrassing depths, the GOP faces a difficult choice Steve Benen Elon Musk\u2019s disastrous interview should end the myth of his fearless genius Anthony L. Fisher Trump \u2018de-legalized\u2019 an estimated 1.8 million legal immigrants Laura Barr\u00f3n-L\u00f3pez , Nour Longi &amp; Alex Tabet The GOP revolt over Todd Blanche is suddenly very real Mychael Schnell As Blanche struggles, Trump opens the door to \u2018temporarily withdrawing\u2019 his nomination Steve Benen Trump drops the pretense and openly declares war on Democratic areas Hayes Brown Speaker Johnson sheds light on Republicans\u2019 tolerance for scandal-plagued members Steve Benen How trying to make sense of the Nolan Wells tragedy led to rampant misinformation Keisha N. Blain \u2018I\u2019ll let you know\u2019: Trump suggests Thune\u2019s future as Senate GOP leader is in doubt Steve Benen Why Republicans should think twice about condemning those who invoke the Fifth Amendment Steve Benen What a poll showing widespread support for the U.S. arresting Netanyahu means Zeeshan Aleem Trump is running the government like a bad businessman Rick Newman Trump has been dismantling post-9/11 counterterrorism protections, Senate report says Ken Dilanian Trump\u2019s fixation on the Dulles Airport overhaul reflects his misplaced priorities Steve Benen FIFA faces global anger over plan to give Kushner brother a financial stake Ja&#039;han Jones E. Jean Carroll&#8217;s $83.3 million award is now at the Supreme Court Jordan Rubin Trump\u2019s tourism envoy reportedly got heaps of cash from his civics-based nonprofit Ja&#039;han Jones Epstein survivors say meeting with Todd Blanche was \u2018very condescending\u2019 Erum Salam Navy short-circuits promotion board to pick top spokesperson Priya Sridhar Trump ex-staffer previews \u2018shocking\u2019 Epstein exhibit on nationwide tour Allison Detzel U.S. economic growth continues to struggle as the Trump-era economy lags Steve Benen Chris Hayes questions GOP silence over allegations against Ohio\u2019s Max Miller Allison Detzel U.S. launches retaliatory strikes on Iran, poll shows Americans&#8217; support for war sliding Ebony Davis &amp; Sydney Carruth Secret Service felt pressure from administration to surveil James Comey, new documents reveal Carol Leonnig , Lisa Rubin &amp; Fallon Gallagher Cornyn to MS NOW: I&#8217;m prepared to vote no on Blanche Lawrence: Republicans attacking Anthony Fauci is &#8216;one of the darkest days in our Senate&#8217;s history\u2019 Trump gets in his own way as attorney", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent, meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "mailto:?subject=%5BShared%20Post%5D%20Homepage%20New&#038;body=https%3A%2F%2Fwww.ms.now%2F&#038;share=email", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "bbc.com", "fetchedAt": "2026-07-30T17:25:19.193119Z", "result": {"base": "https://bbc.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"BBC Home - Breaking News, World News, US News, Sports, Busin\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.bbc.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "17052 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.bbc.co.uk/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "BBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video &amp; Audio", "textOnly": "BBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video &amp; Audio Skip to content Watch Live British Broadcasting Corporation Home News Sport Business Technology Health Culture Arts Travel Earth Audio Video Live Documentaries Home News US &amp; Canada UK UK Politics England N. Ireland N. Ireland Politics Scotland Scotland Politics Wales Wales Politics Africa Asia China India Australia Europe Latin America Middle East In Pictures BBC InDepth BBC Verify Sport Business World of Business Technology of Business NYSE Opening Bell Technology Watch Documentaries Artificial Intelligence Intelligence Revolution AI v the Mind Tech Now Health Watch Documentaries Culture Watch Documentaries Film &amp; TV Music Art &amp; Design Style Books Entertainment News Arts Watch Documentaries Arts in Motion Travel Watch Documentaries Destinations Africa Antarctica Asia Australia and Pacific Caribbean &amp; Bermuda Central America Europe Middle East North America South America World\u2019s Table Culture &amp; Experiences Adventures The SpeciaList Earth Watch Documentaries Science Natural Wonders Climate Solutions Sustainable Business Green Living Audio Podcast Categories Radio Audio FAQs Video Watch Documentaries BBC Maestro Discover the World Live Live News Live Sport Documentaries Site search Home News Sport Business Technology Health Culture Arts Travel Earth Audio Video Live Documentaries Weather Newsletters Watch Live European nations to boycott World Cup if Fifa plans go through Uefa&#x27;s 55 member associations vote to boycott the World Cup if Fifa proceeds with its plan to sell stakes in its competitions. 36 mins ago World Cup US launches &#x27;heavy&#x27; strikes on Iran after attempted attack on American troops After a short period of relative calm when hostilities were paused, both sides have now restarted missile attacks. 7 hrs ago World Drone attack caused fire on gas ships at Mediterranean port, Egypt says The drone reportedly hit a US-owned regasification vessel and a Greek-owned tanker at Damietta port on Wednesday. 6 hrs ago World Missile that left crater deep inside Poland was probably Russian - Polish PM The missile left a 10m-wide (33ft) crater a short distance from the village of Tarnawa Kolonia, about 100km from the border with Ukraine. 2 hrs ago Europe George and Amal Clooney evacuate French home amid wildfires The couple and their children left their home in Brignoles but promised the mayor they would return and help rebuild the area. 5 hrs ago Europe Watch: Giant iceberg turns over near Greenland&#x27;s coast Video captured by a passerby showed the moment the iceberg collapsed, creating a big wave. 6 hrs ago Europe US economic growth sees surprise slowdown in second quarter The economy grew at an annual rate of 1.5% in the three months to June, down from 2.1% seen in the previous quarter. 3 hrs ago US &amp; Canada Only from the BBC The devil worm is the deepest-living animal. It&#x27;s redefining life at its limits When a single worm was discovered in the depths of our planet&#x27;s crust, it began an extraordinarily lucky series of events that transformed our understanding of life at extreme depth. The trespassers that changed access to England&#x27;s countryside Seventy-five years after the Peak District became Britain&#x27;s first national park, visitors can retrace the mass trespass that helped open the countryside to everyone. Recommended Today The Global Story Was Lindsey Graham the last US foreign policy hawk? Save Global News Podcast US launches &#x27;heavy&#x27; strikes on Iran Save Business Daily Listen again - Follow the Money: FIFA&#x27;s Billions Save Witness History &#x27;I organised the first Comic Con&#x27; Save The Global Story The US-Saudi Arabia nuclear deal: could it all go wrong? Save The Documentary Podcast A battle that changed Ireland Save Business Daily Power Players: India&#x27;s Gen\u2011Z jobs crisis Save Americast Could a conspiracy theorist change the ", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://www.bbc.co.uk/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "theguardian.com", "fetchedAt": "2026-07-30T17:25:25.511914Z", "result": {"base": "https://theguardian.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Latest news, sport and opinion from the Guardian\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.theguardian.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "21418 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://theguardian.com/info/about-guardian-us/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 31, "title": "Latest news, sport and opinion from the Guardian", "textOnly": "Latest news, sport and opinion from the Guardian Skip to main content Skip to navigation Print subscriptions Newsletters Sign in &#x27;Peptide wild west&#x27; How RFK Jr is clearing way for a market of untested drugs The Filter Compression socks to help if you&#x27;re traveling or on your feet all day \u2018It looks very childish\u2019 College Republicans are fighting and the far right is benefiting Melanie Challenger double quotation mark I research what it means to be alive. This is what I\u2019ve learned Television How Harlan Coben became the king of potboiler TV \u2018Kind of like a fairy tale\u2019 Documentary tells the unusual tale of Flaco the Owl US US edition UK edition Australia edition Europe edition International edition The Guardian - Back to home The Guardian News Opinion Sport Culture Lifestyle Show more Hide expanded menu News View all News US news US politics World news Climate crisis Middle East Ukraine US immigration Soccer Business Environment Tech Science Newsletters The Filter Wellness Opinion View all Opinion The Guardian view Columnists Letters Opinion videos Cartoons Sport View all Sport Soccer NFL Tennis MLB MLS NBA WNBA NHL F1 Golf Culture View all Culture Film Books Music Art &amp; design TV &amp; radio Stage Classical Games Design a better world Lifestyle View all Lifestyle The Filter Wellness Fashion Food Recipes Love &amp; sex Home &amp; garden Health &amp; fitness Family Travel Money Search input google-search Search Support us Print subscriptions Newsletters Download the app Search jobs Digital Archive Guardian Licensing Live events About Us The Guardian app Video Podcasts Pictures Inside the Guardian Guardian Weekly Crosswords Wordiply Corrections Tips Search input google-search Search Search jobs Digital Archive Guardian Licensing Live events About Us US news US politics World Climate crisis Middle East Ukraine US immigration Soccer Business Environment Tech Science Newsletters The Filter Wellness News Hide US politics live Trump mulls \u2018temporarily\u2019 pulling Blanche\u2019s attorney general nomination until key Republican senators leave office Trump says he may wait until two Republican holdouts, senators John Cornyn and Thom Tillis, are out of office before nominating Blanche for top DoJ job 1m ago Todd Blanche Trump says he may \u2018temporarily\u2019 drop bid to make Blanche US attorney general Donald Trump Three-quarters of Americans think Trump isn\u2019t focusing on top US issues \u2013 poll Todd Blanche Trump says he may \u2018temporarily\u2019 drop bid to make Blanche US attorney general Donald Trump Three-quarters of Americans think Trump isn\u2019t focusing on top US issues \u2013 poll US-Israel war on Iran US hits dozens of targets in Iran overnight as peace efforts under threat Central Command says strikes aimed at diminishing Iranian military while mediators say talks are continuing 2h ago Middle East crisis live Jordan says it foiled Iranian missile attack as US launches \u2018heavy\u2019 strikes on Iran after pause Trump administration Democrats demand answers over treatment of wounded US troops after Iran strike Analysis Iran\u2019s war with the US is not its only conflict as internal crises mount The Latest US-Iran strikes risk regional war in Middle East Russia Russia accused of violating Polish airspace after missile explosion near village 2m ago Ukraine \u2018Lenin was an enemy of humanity\u2019: Why Kyiv wants to put Cossack chief statue in place of Soviet leader Technology Flock surveillance cameras can pose a crash risk for drivers, US experts say 5h ago Minnesota Mayor of St Paul reportedly under investigation for sexual harassment 4h ago Blood American Red Cross declares second-ever national blood supply crisis 9m ago Fifa Uefa agrees to boycott Fifa competitions if World Cup sell-off plans proceed Uefa nations have agreed to boycott all Fifa tournaments if the global governing body continues with its plan to sell off stakes in the World Cup 2h ago Fifa Fifa\u2019s sales pitch revealed: more tournaments, more debt \u2026 but no mention of women\u2019s game Fifa Fifa\u2019s sales pit", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: NewsNow, CCBot, TurnitinBot, PetalBot, MoodleBot, FacebookBot, Bytespider, Mojeek, JenkersBot, Seekr, YouBot, Arquivo-web-crawler, coccocbot-web, SeznamBot, PerplexityBot, yacy, anthropic-ai, ClaudeBot, Claude-SearchBot, Claude-User, AwarioRssBot, AwarioSmartBot, SentiOne, ImageSift, Applebot-Extended, YandexAdditional, YandexAdditionalBot, scalepostAI, Buck, meta-externalagent, Amazonbot, amazon-QBusiness, DuckAssistBot, Google-CloudVertexBot, Amzn-SearchBot, AhrefsBot, AhrefsSiteAudit\nDisallow: /"}], "contactMatch": "https://theguardian.com/info/about-guardian-us/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "reuters.com", "fetchedAt": "2026-07-30T17:25:30.795472Z", "result": {"base": "https://reuters.com", "fatal": "Their server refused or timed out (status 401). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://reuters.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "apnews.com", "fetchedAt": "2026-07-30T17:25:42.766916Z", "result": {"base": "https://apnews.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebSite, NewsMediaOrganization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Associated Press News: Breaking News, Latest Headlines and V\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://apnews.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "60280 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://apnews.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Associated Press News: Breaking News, Latest Headlines and Videos  | AP News", "textOnly": "Associated Press News: Breaking News, Latest Headlines and Videos | AP News Menu World SECTIONS Iran war Russia-Ukraine war Espa\u00f1ol China Asia Pacific Latin America Europe Africa TOP STORIES \u2018Absolute chaos\u2019 as thousands of people breach Morocco\u2019s border with Spain's Ceuta Rotten fish sauce that has long stunk up a small Canadian town is finally being removed 15 elephants die of cyanide poisoning after eating tomatoes from farms near Amboseli National Park Newsletters The Afternoon Wire Our personalization engine sends you what you may have missed throughout the day. The Morning Wire Our flagship newsletter breaks down the biggest headlines of the day. See All Newsletters U.S. SECTIONS Immigration Weather Education Transportation Abortion LGBTQ+ Notable Deaths America at 250 TOP STORIES Montana man hikes out of the mountains to get help after he was impaled by a trekking pole 'Pleading the Fifth' invokes a constitutional amendment that has a long history in US society Wisconsin's appeals court overturns ruling that had restored collective bargaining power Newsletters The Afternoon Wire Our personalization engine sends you what you may have missed throughout the day. See All Newsletters AP QUIZZES Weekly News Quiz Politics SECTIONS 2026 Elections Election Results Election calendar White House Congress Supreme Court The latest AP-NORC polls TOP STORIES Takeaways from Anthony Fauci's COVID-era diaries Trump announces a $22.5B makeover of Dulles airport that will eliminate the 'people movers' How views on Trump\u2019s handling of Iran have shifted, according to a new AP-NORC poll Newsletters Ground Game Your guide to the biggest stories in politics, policy and U.S. elections. See All Newsletters Sports SECTIONS NFL MLB NBA WNBA Soccer Tennis Golf NHL TOP STORIES European nations vow to boycott World Cup, other FIFA tournaments over Infantino private equity plan Former Chargers linebacker Billy Ray Smith Jr. dies at 64, family announces Chiefs assistant Eric Bieniemy remains with family after wife's shooting as full team begins camp Newsletters The Sports Wire Get AP\u2019s in-depth sports reporting tailored to your interests. See All Newsletters Entertainment SECTIONS Movies Fashion Television Celebrity Interviews Music Books TOP STORIES Stephen Colbert wins an anti-censorship award after CBS ends 'The Late Show' Jared Leto denies sexual assault allegations described in BBC documentary The Recording Academy CEO responds to BTS decision not to submit to Grammys Newsletters AP Entertainment Wire AP\u2019s entertainment, media, fashion and culture coverage, personalized to your interests. See All Newsletters Business SECTIONS Tariffs Inflation Financial Markets Financial Wellness Technology TOP STORIES US economy grows at a sluggish 1.5% in second-quarter with inflation remaining stubbornly high Between prices and a diarrhea-causing parasite, lettuce is causing hard times at some restaurants Science Newsletters Quizzes Games Fact Check Oddities TOP STORIES Viral phenomenon in Argentina has young people identifying themselves as animals Fatou, the world's oldest gorilla living in captivity, celebrates her 69th birthday at Berlin Zoo Chicken wings advertised as 'boneless' can have bones, Ohio Supreme Court decides Ghana's fantasy coffins are a colorful celebration of life and legacy Lettuce introduce you to the live frog found in this grocery store salad bag Raccoon goes on drunken rampage in Virginia liquor store and passes out on bathroom floor Be Well SECTIONS Trending Better health At home Working well For the climate Eating well TOP STORIES Do you toss food once the date on the label has passed? Experts say that's often a waste Worries about flying seem to be taking off. Here's how to cope with in-flight anxiety Challenging your brain helps keep it healthy. Here's how to do it Photography SECTIONS Photo Essays TOP STORIES One frame captures a mother\u2019s anguish during a search through a collapsed building in Venezuela At the Kaaba during Hajj, a p", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://apnews.com/contact-us", "sept15": {"behindCloudflare": true, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "already-dark", "detail": "Served through Cloudflare and already blocking AI retrieval crawlers today. September 15 formalizes a posture this site already has. If being invisible to AI assistants is not a deliberate choice, revisit it before it hardens."}}}
{"corpus": "sept15-news", "domain": "bloomberg.com", "fetchedAt": "2026-07-30T17:25:48.027694Z", "result": {"base": "https://bloomberg.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://bloomberg.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "ft.com", "fetchedAt": "2026-07-30T17:25:53.205319Z", "result": {"base": "https://ft.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://ft.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "economist.com", "fetchedAt": "2026-07-30T17:25:58.202575Z", "result": {"base": "https://economist.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://economist.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "forbes.com", "fetchedAt": "2026-07-30T17:26:04.255040Z", "result": {"base": "https://forbes.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Forbes\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.forbes.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "21977 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:feedback@forbes.com).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "Forbes", "textOnly": "Forbes Sign Up For Newsletters Games Share a News Tip Featured Featured Breaking News White House Watch Daily Cover Stories 2026 Best Brands For Value 2026 Forbes World&#x27;s Most Influential CMOs List How Midsize Companies Get Trapped Between AI Promise And AI Reality | Paid Program The AI Risks CEOs Didn\u2019t Budget For | Paid Program The Resilient Enterprise: How A Strategic Insurance Partner Helps Large Firms Navigate Evolving Threats | Paid Program Making It Their Way: Six TikTok Creators On Finding A High-Quality Content Strategy That Pays Off | Paid Program The Hidden Tax On Enterprise AI: 1 In 5 Workers Lose A Full Day Every Week | Paid Program Scaling AI Editorial Video Series The Enterprise AI Shortcut: Why Co-Innovation Is Setting A New Speed For Adopting and Operationalizing AI | Paid Program The Toughest Problems Are Never Solved Alone | Paid Program How Chief Marketing Officers Are Scaling AI At Speed \u2014 While Preserving Brand Trust AI\u2019s Nuanced Impact And A Quest To Quantify It Embracing And Bracing For AI Facing A Volatile Market, C-Suites Look To The CFO For Strategic Guidance Next Billion-Dollar Startups 2025 America&#x27;s Most Powerful Women In Sports CMO Unscripted: Vulnerability Is The New Superpower For Leaders In The AI Era Nvidia GTC 2026 And The Ambitious Path to $1 Trillion In AI Revenue 2026 Forbes 30 Under 30 Europe The Next AI Frontier: Inside LG Electronics\u2019 Strategic Investment In Arizona\u2019s Tech Transformation | Paid Program The Numbers Behind The 2026 World Cup Top Creators Show: How Katie Fang Turned Her Viral Tears Into A Social Media Empire | Paid Program Billionaires Billionaires See All World&#x27;s Billionaires Forbes 400 America&#x27;s Richest Self-Made Women China&#x27;s Richest India&#x27;s Richest Indonesia&#x27;s Richest Korea&#x27;s Richest Thailand&#x27;s Richest Japan&#x27;s Richest Australia&#x27;s Richest Taiwan&#x27;s Richest Singapore&#x27;s Richest Philippines&#x27; Richest Hong Kong&#x27;s Richest Malaysia&#x27;s Richest Money &amp; Politics Innovation Innovation See All AI Future Of Work Xfinity BrandVoice | Paid Program AI-Powered Cybersecurity Acxiom Insights | Paid Program Enterprise AI Nutanix BrandVoice Future of Healthcare Enterprise Intelligence Employee and Customer Experience AI Agentic AI Gaming Big Data Cloud Cloud 100 Consumer Tech Creator Economy Making It Their Way: Six TikTok Creators On Finding A High-Quality Content Strategy That Pays Off Cybersecurity Digital Transformation Enterprise Tech Future Of Work Healthcare Innovation Rules Retail Industry Science Social Media Sustainability &amp; Climate Transportation Venture Capital Technology TikTok BrandVoice | Paid Program Leadership Leadership See All Under 30 C-Suite CEO Network CFO Network CHRO Network CIO Network CMO Network Leadership Strategies Careers Education Forbes EQ | Paid Program ForbesBLK Forbes Research ForbesWomen Deloitte BrandVoice | Paid Program Dell Technologies BrandVoice | Paid Program Money Money See All ETF Investing Trends Banking &amp; Insurance ETFs &amp; Mutual Funds Fintech Hedge Funds &amp; Private Equity Investing Investor Hub Markets Personal Finance Retirement Taxes Top Advisor | SHOOK Wealth Management CareCredit BrandVoice | Paid Program Forbes Digital Assets Forbes Digital Assets See All News Crypto Prices NFT Prices Research Learn Crypto Basics BTC DeFi Web3 Education NFTs Crypto Investments Events Crypto Portfolios Business Business See All Aerospace &amp; Defense Energy Food &amp; Drink Hollywood &amp; Entertainment Law Manufacturing Media Policy Retail SportsMoney Entertainment Digest Enterprise Growth World Cup Small Business Small Business See All Entrepreneurs Small Business Strategy Small Business Taxes Lifestyle Lifestyle See All Luxury Travel Arts Boats &amp; Planes Cars &amp; Bikes Dining ForbesLife Forbes Travel Guide Spirits Style &amp; Beauty Travel Vices Watches Real Estate Real Estate See All Commercial Real Estate Forbes Global Properties Residential Real Esta", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent, Meta-ExternalAgent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot, CCbot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"AhrefsBot, AhrefsSiteAudit, Claude-User, Claude-SearchBot\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "mailto:feedback@forbes.com", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "businessinsider.com", "fetchedAt": "2026-07-30T17:26:09.420303Z", "result": {"base": "https://businessinsider.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Blocks training crawlers only (ClaudeBot, anthropic-ai, Bytespider, CCBot) -- a legitimate opt-out; retrieval/citation access stays open.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: CollectionPage, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Business Insider - Latest News in Tech, Markets, Economy & I\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.businessinsider.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10026 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://businessinsider.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Business Insider - Latest News in Tech, Markets, Economy & Innovation", "textOnly": "Business Insider - Latest News in Tech, Markets, Economy & Innovation Business Insider Subscribe Log in Business Strategy Economy Finance Retail Advertising Careers Law Media Real Estate Small Business The Better Work Project Personal Finance Tech Science AI Enterprise Transportation Startups Innovation Markets News Stocks Indices Commodities Crypto Currencies ETFs Lifestyle Entertainment Culture Travel Food Health Education Parenting Military &amp; Defense Politics Reviews Home Kitchen Style Streaming Pets Tech Deals Gifts Tickets Video Big Business So Expensive View From Above Small Business Authorized Account Risky Business Boot Camp Still Standing How Crime Works Life Lessons Level Up Games Boxed Pipeline Knighthop Tally Community Newsletters Today's Briefing Search Business Strategy Economy Finance Retail Advertising Careers Law Media Real Estate Small Business The Better Work Project Personal Finance Tech Science AI Enterprise Transportation Startups Innovation Markets News Stocks Indices Commodities Crypto Currencies ETFs Lifestyle Entertainment Culture Travel Food Health Education Parenting Military &amp; Defense Politics Reviews Home Kitchen Style Streaming Pets Tech Deals Gifts Tickets Video Big Business So Expensive View From Above Small Business Authorized Account Risky Business Boot Camp Still Standing How Crime Works Life Lessons Level Up Community Games Boxed Pipeline Knighthop Tally Subscribe My account Log in Newsletters Saved Articles US edition Deutschland &amp; \u00d6sterreich Espa\u00f1a Japan Polska TW \u5168\u7403\u4e2d\u6587\u7248 Get the app Business Insider - Top Stories Top stories Situational Awareness was forced to sell most of its stocks to Citadel. The fund got to keep its best asset. 3&nbsp;min read Inside Business The inner workings of companies shaping our world today. Blackstone transformed Jersey Mike&#x27;s. Now it&#x27;s giving employees a bite of the $7 billion IPO. 6&nbsp;min read Emily Stewart The killer app that&#x27;s killing America&#x27;s small businesses 12&nbsp;min read The bond market is sending a big warning to Kevin Warsh 4&nbsp;min read Tech 3 min read Sell it or keep it? Google and Meta&#x27;s AI compute quandary Markets 2 min read Microsoft is on pace for its biggest stock jump in 18 years \u2014 and it&#x27;s lifting the whole market Media 5 min read New court docs show how MrBeast woos brands like Slack and Walmart Live Updated&nbsp; 2026-07-30T17:18:23.473Z Apple earnings: Tim Cook is heading out on top as stock surges to lead the Mag 7 in 2026 Exclusive &#x27;The Diary of a CEO&#x27; star Steven Bartlett has lost 4 key podcast staffers in recent months 4&nbsp;min read Exclusive EY says its &#x27;invisible&#x27; AI router has helped cut token consumption by up to 60% 4&nbsp;min read Exclusive Whatnot is in talks to raise a new funding round at a $20 billion valuation 2&nbsp;min read Exclusive This ex-Y Combinator lawyer thinks startups are overpaying for legal work \u2014 and he&#x27;s got a plan 4&nbsp;min read Latest See all All Business Tech Careers Markets Parenting Business Insider Business Insider tells the innovative stories you want to know Business Insider Business Insider tells the innovative stories you want to know Business Insider Business Insider tells the innovative stories you want to know What readers are saying Explore Business Insider tells the innovative stories you want to know comments comment_author Loading comment 0 replies Business Insider tells the innovative stories you want to know comments comment_author Loading comment 0 replies Business Insider tells the innovative stories you want to know comments comment_author Loading comment 0 replies Business Insider tells the innovative stories you want to know comments comment_author Loading comment 0 replies No comments right now, check back later. Comments are unavailable right now. Featured Business Insider tells the innovative stories you want to know 3&nbsp;min read Business Insider tells the innovative stories you want to know 3&nbsp;min r", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://businessinsider.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "cnbc.com", "fetchedAt": "2026-07-30T17:26:18.446657Z", "result": {"base": "https://cnbc.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: CollectionPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Stock Markets, Business News, Financials, Earnings - CNBC\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.cnbc.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1098616 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 47, "title": "Stock Markets, Business News, Financials, Earnings - CNBC", "textOnly": "Stock Markets, Business News, Financials, Earnings - CNBC Skip Navigation Markets Pre-Markets U.S. Markets Currencies Prediction Markets Cryptocurrency Futures &amp; Commodities Bonds Funds &amp; ETFs Business Economy Finance Health &amp; Science Media Real Estate Energy Climate Transportation Investigations Industrials Retail Wealth Sports Life Small Business Investing Personal Finance Fintech Financial Advisors Options Action ETF Street Buffett Archive Earnings Trader Talk Tech Cybersecurity AI Enterprise Internet Media Mobile Social Media CNBC Disruptor 50 Tech Guide Politics White House Policy Defense Congress Expanding Opportunity Video Latest Video Full Episodes Livestream Live Audio Live TV Schedule CNBC Podcasts CEO Interviews CNBC Documentaries Digital Originals Watchlist Investing Club Trust Portfolio Analysis Trade Alerts Meeting Videos Homestretch Jim&#x27;s Columns Education Subscribe PRO Pro News Josh Brown Mike Santoli Calls of the Day My Portfolio Livestream Full Episodes Stock Screener Market Forecast Options Investing Chart Investing Subscribe Livestream Menu Make It select USA INTL Livestream Search quotes, news &amp; videos Livestream Watchlist SIGN IN Create free account Markets Business Investing Tech Politics Video Watchlist Investing Club PRO Livestream Menu Create your own markets banner US ASIA EUR BONDS OIL GOLD FX CRYPTO PRE-MKT Stocks recover from Fed day sell-off, Nasdaq soars 2% led by chip stocks and Microsoft Quick Links Quick Links Pro: Value stocks GDP growth slows Situational Awareness Iran war When Dow drops by 1,000 META stock Retirement income My Portfolio <image id=\"image2_16943_87611\" width=\"3288\" height=\"4096\" preserveAspectRatio=\"none\" xlink:href=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAADNgAABAACAYAAAAaib2mAAAAAXNSR0IArs4c6QAAIABJREFUeF7s3WuwXWd5J/j/s/Y+5+joYvkiyRd8AYeAMZhLDAFCAAUnJE6H7qTT8nyYnp6QTmwIMYF0T3XNp9n9YWqqumYCjYHEzkxI+lY91nQnadJtQhpQYgghYAIGTALY4Iss27IsS7Iu57LXO7Vl6CQdB8vSkXwuv1W16hzvvdb7Ps/vXV+0z/77rTgIECBAgAABAgQIECBAgAABAgQIECBAgACBNSfQWuq6nTu6F2/dW/u+NDs4tOloTc8f6s6aPa+6hcPd0ZnFGizMdnNHF6vrZ7qFqXHNzI2r66e7bEiO9vPd9LCvxfmZmhr2NTXoa+7IsKupfrZfzLrpQZtdHLTZ6gezbZxBMt5Y1VWrbEpal9RUV5lNa5VqZ03qSdVs0k1V+kHfZ2NNXmnZ1Gryo1Uls0mm/spirUsyk/r2Ky3TqVp3fMzJ0TLMZI6k+64LfLyGOuuZPgRVmW8tx47P9NTHYpKjSfq/8XblwH97rW+HUnX8mkqOttT8k++1hdZy5PjrE4Kqg8df7vtU1w60vvpUO1qpcZ/JeNW3vj/aVbeYagcr3TjVH23jLA6GdWix9eOpQXd0/li3MLN+sV8Yd2242LXWLRyfux2d7udmjrYN3XR/bGHQ+m6un5kdtiPzw3b28Gj/xLGptjjc1G84/Gjbf85Me+6GR/oH98+2/edc3u/csbNP/a0Oz5TW9QQIECBAgAABAgQIECBAgAABAgQIECBAgACBMy7wnT85nfGJTUiAAAECBAgQIECAAAECBAgQIECAAAECBAicnMD1N189ddbBhenx4MKZuZm56XXppxaGw+m0hZnxeDA9NdWmxuNuOmlTrdrUIG0qGUy1tKlqGbbqp9NqJl1mM85MVW3oq01Xy4aqNj0JpbSWdU8GXmqQ1tYnbXoSiqlkfVoGrSbvZ6a1DLtktiXTefL0t4eTW9Yze1flWFomAaBJQGhyTn4/nGQ+LXOtcqxSc621ua4yl6ojaf1ia5PAUDfXqs13LUf7LnPV15FW/fHXu9bPJ7XQum4+GS9UaqHvu/nK4sLk9X4wmB8uzi9WTc2NF7LY9YOFauO5I+sPzV+0Z9P8aLRrUoeDAAECBAgQIECAAAECBAgQIECAAAECBAgQIHDGBfyR64yTm5AAAQIECBAgQIAAAQIECBAgQIAAAQIECCQ7bt0xeN6hB9fPL3Yb+1YbW8vGrrJpMdlYfW2ors2mutnW+k3Hgy4ts5MATCYBly4zaccDLzOVTLfKdPrjO7nMTIIuk9cy2c3l+G4vbSpdTXZ9mWrt+HuTXV2mMtkxxkFgaQUmO+FMdt9Z+PbPJ39vmW9dFqpNduZpc6ksTkI8T15XC8dfS+aqar71/bGke6KSJ9K1I63vj7TqDqfyRNfa4b51RwaDdrib747Mp55YONYf3tDNHf6Vd3/6WNlBZ2lX02gECBAgQIAAAQIECBAgQIAAAQIECBAgQGCNCfjj2RpbcO0SIECAAAECBAgQIECAAAECBAgQIECAwNILjEbpHr9s+1nzR+a2TE1Nn9MvtHNqKme31p3T+pxT1c5NcnaSc1I5O23y37Upk7BLMmjJoJJu8vu3zy6tDVLVTXaLSR1/7zvn5BoHgdUq0JKMk9YnNU4yCe385c+WPpVx/srPqozbk8Ge/al6rLW2v0vt6yv7u9Ym//1Ya/VYG3SPDdrwsWNPLOzff+nW/Tuv2zkZ10GAAAECBAgQIECAAAECBAgQIECAAAECBAgQOC4gYONBIECAAAECBAgQIECAAAECBAgQIECAAIG1JlDbR9sH2698pDvwwObB3MzMYLB5tls8eGgwHgwHXT/f9d1gsHGxhguD/tzK1JZ0bWu18ZbWD7b06bdW1ZYkk3Prt89JgGayY4yDAIEVIXA8wPN4KpPwzb5q7bEkj6b6vcngkbR+b6r29tU90o8X9s4M6mg7Ot0fnR2PB+PF8Xiwrh8uLo4Xh4f6mblt47mZR/qL9rxlPBqNJoEgBwECBAgQIECAAAECBAgQIECAAAECBAgQILACBQRsVuCiKZkAAQIECBAgQIAAAQIECBAgQIAAAQIE/qbAaDTqHrzww+sG49mZmUGtG2e8vhsPNmTYzaRvs+OuNlTfZpNsrOScvk12k+nOnuwu8+QuM8d3mNn87XPy+8Zv7xqDmwCBNSzQklbJ4UoOtORAKo+3lse7yc8", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent, meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "npr.org", "fetchedAt": "2026-07-30T17:26:23.531814Z", "result": {"base": "https://npr.org", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://npr.org", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "politico.com", "fetchedAt": "2026-07-30T17:26:28.525732Z", "result": {"base": "https://politico.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://politico.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "axios.com", "fetchedAt": "2026-07-30T17:26:33.734683Z", "result": {"base": "https://axios.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://axios.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "thehill.com", "fetchedAt": "2026-07-30T17:26:38.618902Z", "result": {"base": "https://thehill.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://thehill.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "newsweek.com", "fetchedAt": "2026-07-30T17:26:45.046304Z", "result": {"base": "https://newsweek.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User, PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Newsweek - Latest Breaking News, US, World, Politics, Busine\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.newsweek.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "22570 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:a.higham@newsweek.com\\\\\\).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Newsweek - Latest Breaking News, US, World, Politics, Business, Tech and Videos.", "textOnly": "Newsweek - Latest Breaking News, US, World, Politics, Business, Tech and Videos. U.S. News Politics Tech Fact Check Sports Global Iran War Russia-Ukraine Middle East China And Asia Live Blog Better Planet All World News Lifestyle Family &amp; Parenting Entertainment Travel Pets Automotive Relationships Science Health Personal Finance Better Workplaces Readers Choice Opinion My Turn All Opinion Rankings Health Finance Products Education Workplaces Corporate Responsibility All Rankings Social Facebook Instagram Threads YouTube Reddit TikTok Twitter/X Extra Newsletters Newsmakers Unconventional Events Essays Podcasts Vantage Magazine CEO Circle Announcements Leadership Dev Pragad Subscribe Sign In U.S. News Politics Tech Fact Check Sports Global Iran War Russia-Ukraine Middle East China And Asia Live Blog Better Planet All World News Lifestyle Family &amp; Parenting Entertainment Travel Pets Automotive Relationships Science Health Personal Finance Better Workplaces Readers Choice Opinion My Turn All Opinion Rankings Health Finance Products Education Workplaces Corporate Responsibility All Rankings Social Facebook Instagram Threads YouTube Reddit TikTok Twitter/X Extra Newsletters Newsmakers Unconventional Events Essays Podcasts Vantage Magazine CEO Circle Announcements Leadership Dev Pragad For Members Analysis Trump Gives Middle East Its Own NATO Moment Collective defense in the Middle East could help stabilize the region, but Washington must give up influence to make it happen. 6 min read Top Stories TRENDING Breaking News Tech Iran War News Trump Signals Shift as Blanche AG Pick Faces GOP Resistance: What\u2019s Next? 6 min read For Members Analysis How the Data Center Next Door Can Pay Your Bills 5 min read News Trump Losing US Support on Iran Amid Renewed Heavy Strikes 6 min read News Who is Seattle Police Chief Shon Barnes? Questions Raised About Whereabouts 6 min read News Warsh Says Economy Showing \u2018Impressive Resilience\u2019\u2014Most Americans Disagree 5 min read Food &amp; Drink Taco Bell Reports \u2018Meaningful\u2019 Hit From Cyclospora Outbreak 5 min read News How Trump Freedom Haulers Plan May Benefit Thousands of Veterans 5 min read News Lost Sermon by Saint in Pope Leo-JD Vance Debate Found 1,600 Years Later 6 min read News Trump Losing US Support on Iran Amid Renewed Heavy Strikes 6 min read News Who is Seattle Police Chief Shon Barnes? Questions Raised About Whereabouts 6 min read News Warsh Says Economy Showing \u2018Impressive Resilience\u2019\u2014Most Americans Disagree 5 min read Food &amp; Drink Taco Bell Reports \u2018Meaningful\u2019 Hit From Cyclospora Outbreak 5 min read News How Trump Freedom Haulers Plan May Benefit Thousands of Veterans 5 min read News Lost Sermon by Saint in Pope Leo-JD Vance Debate Found 1,600 Years Later 6 min read Popular For Members Opinion Sophie Cunningham Spoke Out, but the Locker Room Remains Silent | Opinion 5 min read News McConnell Missing Posters Appear as Questions Grow Over Absence 6 min read Weather Hurricane Genevieve Spaghetti Models Show Today\u2019s Path After Cat 5 Peak 5 min read News Fauci Could Face Contempt Over Pleading the Fifth During Testimony 12 min read Popular For Members Opinion Sophie Cunningham Spoke Out, but the Locker Room Remains Silent | Opinion 5 min read News McConnell Missing Posters Appear as Questions Grow Over Absence 6 min read Weather Hurricane Genevieve Spaghetti Models Show Today\u2019s Path After Cat 5 Peak 5 min read News Fauci Could Face Contempt Over Pleading the Fifth During Testimony 12 min read Popular For Members Opinion Sophie Cunningham Spoke Out, but the Locker Room Remains Silent | Opinion 5 min read News McConnell Missing Posters Appear as Questions Grow Over Absence 6 min read Weather Hurricane Genevieve Spaghetti Models Show Today\u2019s Path After Cat 5 Peak 5 min read News Fauci Could Face Contempt Over Pleading the Fifth During Testimony 12 min read Shows From The Paddock From the Paddock is your go-to Formula 1 podcast for everything happening on and off the grid. Hop in with A", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "mailto:a.higham@newsweek.com\\\\\\", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "time.com", "fetchedAt": "2026-07-30T17:26:50.985507Z", "result": {"base": "https://time.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (716 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebPage, BreadcrumbList.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"TIME | Current &amp; Breaking News | National &amp; World Up\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://time.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "13026 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:feedback@time.com).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "TIME | Current &amp; Breaking News | National &amp; World Updates", "textOnly": "TIME | Current &amp; Breaking News | National &amp; World Updates Skip to Content Menu Close Subscribe The Cover People Ideas Games Health Entertainment AI Climate Skip to Content Menu Close Subscribe TIME | Current &amp; Breaking News | National &amp; World Updates Fabiola Ferrero for TIME How Delcy Rodr\u00edguez Went From Maduro Loyalist to Trump Proxy The acting president of Venezuela insists that she is not surrendering Venezuelan sovereignty but opening the country\u2019s economy in the best interests of its people. by Eric Cortellessa Fabiola Ferrero for TIME Read the Transcript of TIME\u2018s Interview With Venezuelan President Delcy Rodr\u00edguez by Eric Cortellessa Kevin Dietsch\u2013Getty Images Trump Says He Has \u2018No Objection\u2019 to Withdrawing Todd Blanche\u2019s Nomination by Philip Wang World Russian Attack Kills at Least Eight in Ukraine as Zelensky Renews Plea for Supplies Politics The Lone Republican Who Voted Against Advancing Bill to Put Tougher Sanctions on Russia Politics Kentucky Governor to McConnell: Prove Fitness to Serve or \u2018Resign\u2019 Ideas Why Iran Cannot Fight a Forever War Advertisement Newsletters The DC Brief Future Proof In the Loop The Brief Sign up By signing up you are agreeing to our Terms of Service and Privacy Policy . Maximilien Lamy\u2013\u2013AFP/Getty Images What Is a Fire Cloud and How Are They Making Europe&#x27;s Wildfires More Dangerous? Climate europe As firefighters in France and Spain battle to get the blazes under control, an expert breaks down the \u200bphenomenon of fire clouds for TIME. by Tiago Ventura Read More Health How to Stop Being So Defensive During Arguments Ideas The Hidden Cost of Stress at Work Entertainment Review: &#x27;Spider-Man: Brand New Day&#x27; Health Can&#x27;t Sleep? Try Getting Out of Bed Advertisement get the issue The Latest Covers Access The Vault Alix Earle Has Plenty More to Share Alix Earle Has Plenty More to Share The influencer and entrepreneur on what followers get wrong about her and the storylines viewers can expect to unfold on her family&#x27;s upcoming Netflix reality series. Read More Go to cover People Alix Earle Drew Desbordes Lebron James Erling Haaland Advertisement Ideas Read More How India\u2019s Young Faced Down Modi By Yamini Aiyar The Iran War Exposed Africa\u2019s Biggest Energy Weakness By Michael Adu Okyere Brian Mukhaya David W. Yellen The Hidden Cost of Stress at Work By Lilly Springer David Slusky Anupam B. Jena What Octogenarian Pop Divas Show Us About a Life Well-Lived By Debbie Millman TIME100 Creators 2026 TIME100 Creators 2026 Find out who made this year&#x27;s list Read More Access all games FANTASY PREDICTION MARKETS Market Movers Think you know the wisdom of the crowd? Choose the outcome the market thinks is more likely, beat the clock, and climb the leaderboard. PLAY NOW PLAY ON WORDS Linked In this addictive word-association game, find the steps that will unite two seemingly unrelated words in just four moves. PLAY NOW TIME FOR GAMES What Year Was It? Five real TIME covers from our century-spanning archive: look at the pictures, read the clues, guess the years. PLAY NOW TIME FOR GAMES Race to the Red Border Choose from a variety of images to test your jigsaw puzzle speed-solving skills. PLAY NOW Advertisement Health Longevity Open follow modal Personalized Content Longevity Follow this tag to personalize your feed and get instant alerts. Follow Go to your personalized feed WHY FOLLOW? Custom Feed: See the stories that matter most to you. Smart Alerts: Get notified about major news as it happens. Update your preferences in Account Settings Close Public Health Open follow modal Personalized Content Public Health Follow this tag to personalize your feed and get instant alerts. Follow Go to your personalized feed WHY FOLLOW? Custom Feed: See the stories that matter most to you. Smart Alerts: Get notified about major news as it happens. Update your preferences in Account Settings Close Exercise &amp; Fitness Open follow modal Personalized Content Exercise &amp; Fitness Foll", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:feedback@time.com", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "theatlantic.com", "fetchedAt": "2026-07-30T17:26:56.090087Z", "result": {"base": "https://theatlantic.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"World Edition - The Atlantic\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "8539 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.theatlantic.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 55, "title": "World Edition - The Atlantic", "textOnly": "World Edition - The Atlantic Skip to content Site Navigation The Atlantic Popular Latest Newsletters Sections Ideas Politics Economy Global National Security Washington Week Features Technology AI Watchdog Science Planet Health Philosophy Education Culture Comedy Family Books Fiction Photography Events Explore The Atlantic Archive Play The Atlantic Games Listen to Podcasts and Articles The Print Edition Latest Issue Past Issues Give a Gift Search The Atlantic Quick Links Audio Crossword Puzzle Magazine Archive Your Subscription Popular Latest Newsletters Sign In Subscribe Dana Golan for The Atlantic The Cities That Said Yes to Drugs Policies that gave addicts clean needles and places to use drugs were intended to reduce harm. They created a \u201czombie apocalypse.\u201d Michael Powell The Revelatory Cringe of Anthony Fauci Gal Beckerman Iran\u2019s War Within Arash Azizi Is She the New Britney Spears\u2014Or the New Lou Reed? Spencer Kornhaber Why Animal Narrators Are All the Rage Rachel Vorona Cote Sarah Rice / Getty Abdul El-Sayed Fuses Together Two Toxic Ideologies Jonathan Chait Marvel Entertainment / Columbia Pictures / Everett Collection Spider-Man\u2019s Not-So-Great Responsibility David Sims Julia Kokernak for The Atlantic The Town That Wants a Data Center Next Door Maine was about to ban data centers. Why did one town stop it? Ethan Brooks Olivier Douliery / Sipa USA / AP What Anthony Fauci Didn\u2019t Say Senate Republicans had many questions. Fauci had one answer. Katherine J. Wu and Daniel Engber Nic Antaya for The Atlantic The Anti-Democrat Abdul El-Sayed is calling out his party; Haley Stevens is embracing it. Russell Berman Illustration by Matteo Giuseppe Pani / The Atlantic The New iPhone Underclass Apple\u2019s rental program is a trap. Damon Beres Illustration by Alisa Gao / The Atlantic Why Kids Can\u2019t Quit Roblox The mechanics of many games on the platform are nearly identical to those of slot machines and sports-betting apps. Connie Chang CFOTO / Getty The Great Chinese Oil Mystery America\u2019s greatest rival is single-handedly averting a global energy crisis. No one knows how the country is doing it\u2014or why. Rog\u00e9 Karma Illustration by Jonelle Afurong / The Atlantic. Sources: baona / Getty; nelic / Getty; Scott Olson / Getty. Rural America\u2019s Maternal-Health Crisis Is About to Get Much Worse Care will be in shorter supply, travel distances will stretch farther, and high-risk situations will more likely result in tragedy. Keith Humphreys Illustration by The Atlantic. Sources: fad1986 / Getty; PM Images / Getty. The Masculinity Scam The manosphere\u2019s campaign strategy Adam Serwer Anna Hop for The Atlantic The Return of the Monarch After years of decline, butterfly populations are rising\u2014for now. Vann R. Newkirk II The Horrible Rise of the Makeout Police Faith Hill Lindsey Graham\u2019s Last Laugh Mark Leibovich Trump Crashes Into Reality Jonathan Lemire RECOMMENDED FOR YOU Cassidy Araiza Why Everything Is Getting Louder The tech industry is producing a rising din. Our bodies can\u2019t adapt. (From 2019) Bianca Bosker Paul Spella / The Atlantic The Outsize Influence of Your Middle-School Friends The intensity of feelings generated by friendship in childhood and adolescence is by design. (From 2020) Lydia Denworth ARCHIVE Library of Congress The New Talking Machines A noted architect and writer commends Thomas Edison for his progress in developing the phonograph and predicts great things for its future. (From 1889) Philip G. Hubert Associated Press Pitchers and Catchers \u201cGood fielding and pitching, without hitting, or vice versa, is like Ben Franklin\u2019s half a pair of scissors \u2014 ineffectual. Twenty-game winners or .400 hitters do not ensure victory.\u201d (From 1941) Moe Berg Magazine past issues Enjoy unlimited access to all of The Atlantic \u2019s journalism, in the print edition and online. Subscribe Podcasts see all Julia Kokernak for The Atlantic radio atlantic \u2022 Episode 200 The Town That Wants a Data Center Next Door Maine was about to ban data centers. Why did on", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://www.theatlantic.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "newyorker.com", "fetchedAt": "2026-07-30T17:27:05.366668Z", "result": {"base": "https://newyorker.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, ItemList.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The New Yorker\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.newyorker.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1235702 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:?subject=&body=https%3A%2F%2Fwww.newyorker.com%2Fcartoon%2Fa62120%3Futm_source%3Donsite-share%26utm_medium%3Demail%26utm_campaign%3Donsite-share%26utm_brand%3Dthe-new-yorker).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "The New Yorker", "textOnly": "The New Yorker Skip to main content The New Yorker Newsletter Search Search Open Navigation Menu Menu The Latest News Books &amp; Culture Fiction &amp; Poetry Humor &amp; Cartoons Magazine Puzzles &amp; Games Video Podcasts Goings On Shop What Would It Mean to See a New Color? A team of researchers developed a new color, olo, which can be described as the \u201cgreenest green.\u201d A printer can\u2019t print it and a camera can\u2019t capture it, but its development tells us a lot about how we see color. Rivka Galchen reports on our miraculous eyes. Listen Dots Today\u2019s Mix How Donald Trump Silenced the Voice of America Listen The agency was created to spread the story of freedom and democracy around the world. Now, with many of its journalists being paid to stay home, it\u2019s delivering a very different message. By Marc Fisher Inside OpenAI\u2019s Hack of Hugging Face Hugging Face\u2019s chief science officer recounts the cybercrime that signals a terrifying new era for A.I. By Stephen Witt Will Left-Wing Democrats Reinvent the Party or Hurt Its Chances in November? Bernie Sanders\u2019s erstwhile campaign manager explains why he thinks it\u2019s the former. By Isaac Chotiner \u201cFurious\u201d Is a Crime Drama for a Post-Epstein World Listen Elizabeth Meriwether\u2019s new series, which follows a female killer seeking vengeance against a network of abusers, is brutal, nuanced, and, at times, surprisingly funny. By Inkoo Kang Dots American Daredevil Johnny Knoxville, the star of \u201cJackass,\u201d has been riding bulls and shooting himself out of cannons for two decades. Why would anyone do that? By Molly Young Listen Dots The Lede A daily column on what you need to know. Trump\u2019s Isolationism Is Losing the Great E.V. Race Listen The United States risks becoming a theme park of internal combustion, as the rest of the world moves on. By Bill McKibben How Bots Took Over Our Lives Listen During the 2016 election, much of the country was introduced to the concept of internet bots for the first time. Now, just a decade later, it feels like they\u2019re everywhere. By T. M. Brown Which Way Should the Democratic Party Go Now? Listen It\u2019s been a tumultuous primary season, but is this the Democrats\u2019 Tea Party moment? Or is something more pragmatic emerging? By Benjamin Wallace-Wells The Complications of Graham Platner\u2019s Veteran Identity Listen How much of the former Senate candidate\u2019s behavior should be read through the lens of what combat did to him, and how much of it is simply who he is, independent of war? By Jasper Craven \u201cNo One Knows When It Will End\u201d: A Doctor on the Ebola Emergency Listen Health-care workers in the Democratic Republic of the Congo are fighting the third-largest outbreak in history&#8212;and trying not to get infected themselves. By Jason Liebowitz Maine\u2019s Mad Scramble to Replace Graham Platner Listen The state\u2019s Democratic primary was dominated by an unlikely left-wing insurgent&#8212;it became a three-week sprint for Party insiders to select the next best thing. By Jason Zengerle Dots American Chronicles A Woman Lost Two Partners to Guns&#8212;Then the N.R.A. Came to Collect The majority of suicides in the U.S. involve guns, and eighty per cent of those are done by white men, a pattern that the gun-rights organization has spent decades denying. By Mike Spies Listen Dots Help shape the future of The New Yorker . Share your thoughts by taking our survey \u00bb The Critics The Current Cinema The Kinky Comedy of \u201cI Want Your Sex\u201d Listen In Gregg Araki\u2019s film, a B.D.S.M. workplace relationship opens up a cross-generational debate about carnal mores. By Justin Chang The Art World Fairfield Porter\u2019s Lush, Lonely Vision of the World Listen The painter was rich and cloistered, and used color like few others before or after him. By Zachary Fine Pop Music Charli XCX and the Strokes Are (Sort of) Making Rock Music Listen Both artists have released new albums&#8212;\u201cMusic, Fashion, Film\u201d and \u201cReality Awaits\u201d&#8212;that seesaw between raw sincerity and punkish insouciance. By Amanda Petrusich The", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}], "contactMatch": "mailto:?subject=&body=https%3A%2F%2Fwww.newyorker.com%2Fcartoon%2Fa62120%3Futm_source%3Donsite-share%26utm_medium%3Demail%26utm_campaign%3Donsite-share%26utm_brand%3Dthe-new-yorker", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "vox.com", "fetchedAt": "2026-07-30T17:27:11.143912Z", "result": {"base": "https://vox.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User, PerplexityBot, Claude-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: NewsMediaOrganization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Vox\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.vox.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "14129 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://vox.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "Vox", "textOnly": "Vox Skip to main content The homepage Vox Vox logo Explainers Politics Culture Advice Listen Podcast Watch Video Menu The homepage Vox Vox logo Navigation Drawer Login / Sign Up close Close Search Video Watch Podcast Listen Crossword Play Explainers Politics Culture Advice Science Technology Climate Health Money Life Future Perfect Newsletters Archives Become a Member Facebook Instagram Youtube RSS TikTok Making sense of it all: The Trump administration Iran war Climate The people next door Start Here 1 President Donald Trump is escalating his fight with GOP senators over Todd Blanche\u2019s attorney general nomination , threatening to withdraw it. 2 Elon Musk\u2019s AI company is suing to block a Minnesota law that would ban AI tools used to create fake nude images. 3 ABC says the Trump administration is retaliating against the network over its news coverage, in an escalating fight with the FCC. 4 From Taylor Swift to Love Island , Kalshi is using pop culture to bring more women into prediction markets and betting . The Highlight What housing in America could look like in 50 years Abundant, affordable, and...fun? Here\u2019s how to get there. By Marina Bolotnikova Technology The iPhone lease is too good to be true Technology AI could end up too cheap to control Advice How to bail on plans without losing friends Today, Explained newsletter The Covid debate that never ended Politics Marco Rubio and Ben Shapiro have a new attack line on the left Today, Explained newsletter The Covid debate that never ended Politics Marco Rubio and Ben Shapiro have a new attack line on the left Crossword Play today&#x27;s crossword Start Here 1 President Donald Trump is escalating his fight with GOP senators over Todd Blanche\u2019s attorney general nomination , threatening to withdraw it. 2 Elon Musk\u2019s AI company is suing to block a Minnesota law that would ban AI tools used to create fake nude images. 3 ABC says the Trump administration is retaliating against the network over its news coverage, in an escalating fight with the FCC. 4 From Taylor Swift to Love Island , Kalshi is using pop culture to bring more women into prediction markets and betting . Today, Explained newsletter The Covid debate that never ended Politics Marco Rubio and Ben Shapiro have a new attack line on the left Crossword Play today&#x27;s crossword The Highlight What housing in America could look like in 50 years 5 books that define America \u2014 for better and for worse The breakthrough changing how Americans donate organs What housing in America could look like in 50 years 5 books that define America \u2014 for better and for worse The breakthrough changing how Americans donate organs Even Better More How to bail on plans without losing friends By Allie Volpe The iPhone lease is too good to be true By Adam Clark Estes How to be a good roomie \u2014 regardless of where you are in life By Allie Volpe Don\u2019t give up vegetables because of the diarrhea parasite By Dylan Scott How to protect yourself from the diarrhea-causing parasite wreaking havoc across the US By Dylan Scott How to bail on plans without losing friends By Allie Volpe The iPhone lease is too good to be true By Adam Clark Estes How to be a good roomie \u2014 regardless of where you are in life By Allie Volpe Don\u2019t give up vegetables because of the diarrhea parasite By Dylan Scott How to protect yourself from the diarrhea-causing parasite wreaking havoc across the US By Dylan Scott Future Perfect More What housing in America could look like in 50 years By Marina Bolotnikova Should you keep practicing a religion even if you don\u2019t believe? By Shayla Love A tiny pill could beat the world\u2019s biggest killer By Bryan Walsh Costco\u2019s organic chicken isn\u2019t what it seems By Kenny Torrella The US is better off than it was in 1976. So why does it feel worse? By Bryan Walsh What housing in America could look like in 50 years By Marina Bolotnikova Should you keep practicing a religion even if you don\u2019t believe? By Shayla Love A tiny pill could beat the world\u2019s bigg", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"Amazonbot\" has no root Disallow."}], "contactMatch": "https://vox.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "slate.com", "fetchedAt": "2026-07-30T17:27:16.259053Z", "result": {"base": "https://slate.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: NewsArticle.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Slate Magazine - Politics, Business, Technology, and the Art\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://slate.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12105 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://slate.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "Slate Magazine - Politics, Business, Technology, and the Arts", "textOnly": "Slate Magazine - Politics, Business, Technology, and the Arts Advertisement Advertisement Skip to the content Slate Shop Games Newsletters Sign In Account Activating this button will toggle the display of additional content Account Sign out Slate homepage News &amp; Politics Culture Technology Business Life Advice Podcasts Menu Open menu Close menu Menu News &amp; Politics Culture Technology Business Life Advice Podcasts Search Activating this button will toggle the display of additional content Light Dark Auto Subscribe Slate Shop Games Newsletters Sign in Account Sign out Slate homepage Account * Simmering Along Like a Cauldron of Chowder Submit search Enter query Activating this button will toggle the display of additional content Light Dark Auto Democrats Have Been Stuck on the Same Question for 10 Years. This Year\u2019s Biggest Primary Could Finally Give Them the Answer. Jim Newell The Real Reason Trump Hasn\u2019t Been Able to Win the War With Iran Fred Kaplan The Trump Administration Is Finally Allowed to Post on TikTok. What They\u2019re Doing Is, Frankly, Insane. Christina Cauterucci As a Classicist, I\u2019ve Defended Emily Wilson\u2019s Odyssey Translation. I Can\u2019t Defend Her Review of Nolan\u2019s Movie. Theo Nash Trump\u2019s Desperate Attempt to Get the Supreme Court to Reverse Itself on Voting Chaos Donathan L. Brown Advertisement Movies The Anthony Bourdain Biopic Is Delicious Dana Stevens Forget IMAX 70 mm. I Found a Better Way to Watch The Odyssey . Nadira Goffe The 1987 Movie Was for the Male Gaze. Hulu\u2019s Dark New Adaptation Turns It Into Something Much Better. Laura Miller Stars She\u2019s Been in a Fight With Donald Trump for 20 Years. He Threatened to Revoke Her Citizenship. Now She\u2019s Back. Scaachi Koul Tech Trouble Kalshi Says It\u2019s \u201cVoter Suppression\u201d to Ban Betting on Elections. LOL. Richard L. Hasen Slate games Slate Mini Crossword for July 30, 2026 K. Austin Collins Which Greek Letter Represents the Ratio of a Circle\u2019s Circumference to Its Diameter? Ray Hamel Lindsay Clancy Is on Trial for Killing Her Three Children. That Could Have Been Me. Meghan Cliffel Is Your Phone Making You Kind of Miserable? We Have a Challenge for You. Dan Kois Health MAHA and Clavicular Have United Behind Peptides. It\u2019s an Incomprehensible Alliance\u2014Until You Realize One Thing. Molly Olmstead Abstinence-Only Used to Be the Dumbest Way to Talk to Teens About Sex. Trump\u2019s Team Is Pushing a Worse One. Jill Filipovic Podcast Episode Have We Thrown Too Much Shade on Sunlight? Decoder Ring Advertisement Advertisement Sign up for Your Morning Slate Three personal recommendations from our writers to enjoy with your coffee, plus everything else we can\u2019t stop talking about. Thanks for signing up! You can manage your newsletter subscriptions at any time. You're already subscribed to the Your Morning Slate newsletter. You can manage your newsletter subscriptions at any time. We encountered an issue signing you up. Please try again, or manage all your newsletter subscriptions here . Please enable javascript to sign up for newsletters. Email address: Send me updates about Slate special offers. By signing up, you agree to our Privacy Policy and Terms . Sign Up Sounds Like a New Game Solve clues, collect sounds, and crack the mystery word. Now every weekday! Play SoundBites Play Archive Play Archive Play Archive Play Archive Play Archive This Content is Available for Slate Plus members only true pay dirt Some Couples Dream of the Set Up My Boyfriend and I Have. I\u2019m So Unhappy. Featured This Content is Available for Slate Plus members only true how to do it My Boyfriend Is Suspicious of My Relationship With My Male BFF. I Accidentally Made It Worse. This Content is Available for Slate Plus members only true care and feeding I Noticed My Son Stopped Wearing Underwear. It\u2019s His Brother\u2019s Fault. This Content is Available for Slate Plus members only true dear prudence Help! This \u201cTrend\u201d in Party Planning Made Me Miss My Friend\u2019s Bachelor Party. I\u2019m So Annoyed. Latest from Our Columnis", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: 008, AdsTxtCrawler, AI2Bot, Ai2Bot-Dolma, Amazonbot, Applebot-Extended, bedrockbot, Bytespider, CCBot, ChatGLM-Spider, ClaudeBot, CloudVertexBot, cohere-training-data-crawler, CohereBot, Cotoyogi, Dataprovider.com, Datenbank Crawler, dcrawl, DeepSeekBot, Doubaobot, ERNIEBot, FacebookBot, Google-Extended, GoogleOther, GPTBot, HelloworkJobPostingBot, HTTrack, HTTrack 3.0, ICC-Crawler, imageSpider, IndeedJobBot, Kangaroo Bot, KimiBot, KimiCrawler, KrawlerBot, laion-huggingface-processor, LCC, meta-externalagent, MetaInspector, MetaJobBot, micro-crawl, MistralBot, MoonshotBot, MoonSpider, netEstate Imprint Crawler, newspaper, Nutch, nyt_scraping, Offline Explorer, omgili, OpenindexSpider, PanguBot, Potions, QwenBot, SalamandraVLM, SBIntuitionsBot, Scrapy, ServerHunterSpider, Spider, StatsDroneBot, Thinkbot, Timpibot, VelenPublicWebCrawler, webzio-extended\nDisallow: /"}], "contactMatch": "https://slate.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "salon.com", "fetchedAt": "2026-07-30T17:27:21.357749Z", "result": {"base": "https://salon.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, Website.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Salon.com - News, Politics, Culture, Science &amp; Food\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.salon.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10821 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.salon.com/about/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 60, "title": "Salon.com - News, Politics, Culture, Science &amp; Food", "textOnly": "--> Salon.com - News, Politics, Culture, Science &amp; Food Subscribe Search News &amp; Politics Culture Food Science &amp; Health Money Life Stories Video Reviews Lifestyle The New Sober Boom Getting Hooked on Quitting Education Liberal Arts Cuts Are Dangerous Is College Necessary? Finance Dying Parents Costing Millennials Dear Gen Z Investing In Le Creuset Crypto Investing SEC vs Celebrity Crypto Promoters &#8216;Dark&#8217; Personalities Drawn to BTC News &amp; Politics Culture Food Science &amp; Health Money Life Stories Video Reviews Lifestyle The New Sober Boom Getting Hooked on Quitting Education Liberal Arts Cuts Are Dangerous Is College Necessary? Finance Dying Parents Costing Millennials Dear Gen Z Investing In Le Creuset Crypto Investing SEC vs Celebrity Crypto Promoters &#8216;Dark&#8217; Personalities Drawn to BTC Subscribe Summer sale: Save 58% Newsletter News &amp; Politics Culture Food Science &amp; Health Money Video Search Salon Homepage CNN is complicit in the Trump circus Sophia Tesfaye CNN is complicit in the Trump circus Sophia Tesfaye The press didn't just cover Trump's White House Correspondents' Dinner meltdown \u2014 it enabled it A magical Miyazaki cure for burnout Melanie McFarland A magical Miyazaki cure for burnout Melanie McFarland \"Kiki's Delivery Service\" is a family movie that holds practical magic for adulthood The art of the beach sandwich Ashlie D. Stevens Unlike the airport sandwich, a great beach sandwich doesn\u2019t merely survive the wait \u2014 it gets better as it sits Accountability can't stop at ICE Insha Rahman Corey Ruiz's death in Wisconsin shows that local police need scrutiny too FEATURED Judges slam Trump's \"unethical\" DOJ J. David McSwane, Pratheek Rebala, Carla Astudillo - ProPublica Federal judges have criticized the administration's prosecutors in hundreds of cases since Trump took office Vance will never be the GOP Obama Amanda Marcotte He can copy Obama's biography, but the vice president will never have the charisma A founding father's queer legacy Nicholas Boston Thomas Jefferson and Sally Hemings' family tree keeps getting more complicated and colorful Performing arts and the empathy crisis Stephen Brown-Fried, Pablo Helguera We\u2019re losing our connection with one another. Performing arts education can help us get it back MOST POPULAR Andrew Tate's arrest causes trouble for Trump's youngest son Trump's youngest son faces Tate scrutiny Sophia Tesfaye The manosphere influencer has lost the president's support \u2014 but he could pose a threat to Barron A founding father's queer legacy A founding father's queer legacy Nicholas Boston Thomas Jefferson and Sally Hemings' family tree keeps getting more complicated and colorful \"The Man Will Burn\" accidentally shows why people despise Burning Man Burning Man was flawed from the start Melanie McFarland Instead of clearing up misconceptions about the annual desert bacchanalia, HBO's docuseries confirms them Donald Trump has stopped caring about the midterms Trump has quit caring about the midterms Brian Karem But the Democrats \u2014 and the rest of us \u2014 still hope to make them count Pete Hegseth's Iran war math doesn't add up Hegseth's Iran war math doesn't add up Heather Digby Parton The defense secretary pegged the war's cost at $37 billion. The real total is far higher Trump's foreign policy enters its decadent, desperate end stage The last gasp of Trump's foreign policy Andrew O'Hehir He restarted a war no one wanted and signed a nuclear deal no one read. What's next? NEWS &amp; POLITICS Talarico leads Paxton in new poll Gabby Birenbaum - The Texas Tribune The poll found the Austin Democrat with a surprising edge Private school on the public's dime Glenn Sacks A provision now taking effect under the Big Beautiful Bill sends students to private schools at public expense Trump and Musk's history obsession Sonja Drimmer The frenemies' strange obsession with historical \"accuracy\" has disturbing connections to big AI Harris\u2019 biggest 2028 opponent is he", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"Google-Extended\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"ClaudeBot\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"anthropic-ai\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"Meta-ExternalAgent\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"Applebot-Extended\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"Bytespider\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"CCBot\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"PerplexityBot\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"Claude-User\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"Amazonbot\" has no root Disallow."}], "contactMatch": "https://www.salon.com/about/contact", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "motherjones.com", "fetchedAt": "2026-07-30T17:27:26.607289Z", "result": {"base": "https://motherjones.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Mother Jones &#8211; Smart, fearless journalism\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "8241 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.motherjones.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 25, "title": "Mother Jones &#8211; Smart, fearless journalism", "textOnly": "Mother Jones &#8211; Smart, fearless journalism Skip to main content --> Donate Donate Subscribe Got tips? Get in touch confidentially. Newsletters Politics Environment Criminal Justice Guns Race Gender + Sexuality Food Podcast Video Magazine Got tips? Get in touch confidentially. Subscribe to our magazine --> Donate Monthly Donate 50 Years of Fearless Journalism Politics Environment Criminal Justice Guns Race Gender Food Podcast Video Magazine Newsletters Mother Jones illustration; Tom Williams/CQ Roll Call/Zuma Democrats Call For Ethics Investigation into Congressman Max Miller Abby Vesoulis The Kushner-Backed FIFA Takeover Tearing the Soccer World Apart Tim Murphy A For-Profit Hospital Slashed Costs. Then a Patient Froze to Death on Its Roof. Hannah Levintova and Ashley Cleek France Had Never Seen a &#8220;Fire Cloud&#8221; Until This Month\u2019s Record-Smashing Blazes Brian Kahn Maine Just Told ICE to Get Out Sophie Hurwitz Covid Truthers Are Claiming Fauci\u2019s Diary Was &#8220;Scrubbed&#8221; Kiera Butler California Does Medicaid Home Care Well. They&#8217;re Being Punished For It. Julia M\u00e9traux Latest Video Digital dispatches from our journalists More in Video The Venice Biennale\u2019s US Pavilion Was Supposed to Say Something About America. It Didn\u2019t\u2014And That\u2019s the Point. \u201cI\u2019m Ashamed of My Country\u201d: Biddeford, Maine Locals Grieve Neighbor Killed by ICE DHS Is Spending $1.5 Billion to Block ICE Oversight US Soccer Doesn\u2019t Need a Big, Orange Thumb on the Scale More Top Stories What &#8220;You Only Get 18 Summers&#8221; Gets Right About Parenting Inae Oh The Rapid Rise of Right-Wing Militias Reveal Big Oil Is Getting Sued for Heat Deaths. It&#8217;s Fighting Back With an Army of Immunity Laws. Kate Yoder The IRS Is Imploding Stephanie Mencimer Trump\u2019s New Ridiculous and Dangerous Red-baiting David Corn Staff Blog OpenAI Hacking Fiasco Exposes a \u201cDeeply Insufficient\u201d System to Protect the Public Alex Nguyen Report: Dozens of ICE and CBP Agents Have Abused Women and Children Sophie Hurwitz The Only Research Center For Disabled Parents Is Losing Federal Funding Julia M\u00e9traux David Corn&rsquo;s Newsletter Trump\u2019s New Ridiculous and Dangerous Red-baiting July 28, 2026 The Most Extreme GOP Candidate in the Nation July 15, 2026 Trump and Iran: Stupid Is As Stupid Does June 19, 2026 Sign up for our free newsletter Subscribe to the Mother Jones Daily to have our top stories delivered directly to your inbox. Close Thank you for subscribing! G59DC By signing up, you agree to our privacy policy and terms of use , and to receive messages from Mother Jones and our partners. Featured Mother Jones illustration; Tom Williams/CQ-Roll Call/Getty Assault, Misogyny, Verbal Threats: Records Detail New Allegations Against Rep. Max Miller Abby Vesoulis Reveal Podcast Listen to our award-winning radio show More from Reveal The Rapid Rise of Right-Wing Militias ICE Took Mom and Dad. Now the Kids Are on Their Own. Staten Island v. New York City With Revisionist History The Secret Story of FTX\u2019s Rise and Ruin Part 2 Politics Trump Administration Moves to Open American Samoa&#8217;s Waters to Deep-Sea Mining Teresa Tomassoni Records Reveal How Homeland Security Agents Surveilled ICE Protesters Schuyler Mitchell Environment EPA Ordered\u2014Again\u2014to Release $2.8 Billion in Climate Justice Funds Lauren Dalban It Took a Decade, Two Sniffer Dogs, and a Lot of Mud to Protect This Orchid Melissa L. Sevigny Criminal Justice Moms of Black Babies More Likely to Be Flagged to Police Over Alleged Pregnancy Drug Use Shoshana Walter The Secret Origins of the Supreme Court\u2019s Shadow Docket Reveal Race What Author and Poet Victoria Chang Learned From Trees Jeffrey Kelly Why America at 250 Still Cannot Face Slavery Reveal Gender The Heritage Foundation Only Wanted to Protect Women&#8217;s Sports When It Was About Trans Athletes Madison Pauly Todd Blanche Just Opened the Door to a National Abortion Ban Garnet Henderson and Susan Rinkunas LGBTQ The Supreme Court\u2019s Trans Athlete Rulin", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"Claude-User\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://www.motherjones.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "propublica.org", "fetchedAt": "2026-07-30T17:27:32.251125Z", "result": {"base": "https://propublica.org", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"ProPublica \u2014 Investigative Journalism and News in the Public\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.propublica.org/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "13421 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.propublica.org/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "ProPublica \u2014 Investigative Journalism and News in the Public Interest", "textOnly": "ProPublica \u2014 Investigative Journalism and News in the Public Interest Arrow Right Caret Close Investigative Journalism in the Public Interest Donate ProPublica Menu Search ProPublica Menu Menu Search Donate Investigative Journalism in the Public Interest Reporting on Dismantling USAID Skipping Newborn Shots Contaminated Water About Newsletters Podcast Have a Tip? ProPublica Investigative Journalism in the Public Interest Menu Donate Search ProPublica Search Close ProPublica is a nonprofit, investigative newsroom that exposes corruption. We report in all 50 states and partner with local newsrooms . Our work spurs real-world impact and has received numerous awards, including nine Pulitzer Prizes. About Us Impact Leadership Staff Partners Contact Us Press Send Us Tips Topics We Cover See All Abortion Civil Rights Courts Criminal Justice Debt Democracy Education Environment Health Care Health Insurance Immigration Labor Mental Health Military Police Politics Pregnancy Prison Racial Justice Regulation Sex and Gender Technology Our Biggest Series See All Life of the Mother How Abortion Bans Lead to Preventable Deaths The New Immigration How Recent Arrivals at the Border Have Changed the Country and Its Attitudes Friends of the Court SCOTUS Justices&#8217; Beneficial Relationships With Billionaire Donors ProPublica \u2014 Investigative Journalism and News in the Public Interest Ryan Hermens/Lexington Herald-Leader Andy Beshear Set Out to Make Drug Treatment Widely Available in Kentucky. Fraud and Abuse Followed. In 2020 as Kentucky faced a spike of deadly overdoses, Gov. Andy Beshear set out to help more people access drug treatment by loosening Medicaid spending controls. But as the costs piled up, so did the warnings that the system was being abused. 4 Months Ago: They Needed Treatment for Drug Addiction. The Company They Turned to May Have Used Them to Commit Fraud. Help Us Report: Tell Us About Your Experience With Kentucky\u2019s Addiction Recovery Care Illustration by Shoshana Gordon/ProPublica Anthropic\u2019s New AI Model Can Identify More Software Bugs Than Ever. Microsoft Is Struggling to Fix Them Fast Enough. Anthropic\u2019s Mythos has flagged bugs faster than Microsoft can fix them. Documents reviewed by ProPublica reveal the tech giant&#8217;s \u201cmad dash\u201d behind the scenes to patch holes before hackers can find and exploit them. 1 Year Ago: A Little-Known Microsoft Program Could Expose the Defense Department to Chinese Hackers 2 Years Ago: Microsoft Chose Profit Over Security and Left U.S. Government Vulnerable to Russian Hack, Whistleblower Says Philip Cheung/The Washington Post/Getty Images This MAGA Influencer Runs a Charity to Teach Kids Civics. Much of the Money Goes to Him Instead. Nick Adams\u2019 Foundation for Liberty and American Greatness has brought in millions in donations and helped him land a State Department appointment, but records show there is almost no indication FLAG was spending that money on its stated mission. Al Drago/Bloomberg via Getty Images Senators Push Federal Regulator to Prove School Bus Safety Data Is Accurate The request from two Democratic lawmakers comes after WBUR and ProPublica found that the federal safety record was missing dozens of fatal crashes involving one of the nation\u2019s largest public transit operators. 2 Months Ago: A School Bus Killed a 5-Year-Old. The Crash Is Among Dozens Missing From the Bus Company\u2019s Federal Safety Record. Timo Lenzen for ProPublica Federal Judges Chastise Trump\u2019s Justice Department for \u201cUnlawful,\u201d \u201cUnethical\u201d and \u201cUnseemly\u201d Conduct ProPublica reviewed hundreds of cases in which judges criticized the actions of federal prosecutors. In many, judges expressed frustration that they can no longer take the government at its word. Obtained by ProPublica. Faces blurred in original video. Vignette added by ProPublica to spotlight Morrison. How Videos Helped Me Uncover Abuse at an Arkansas Private School Mary \u201cTracy\u201d Morrison ordered students to scrub floors and urged them to attack a", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.propublica.org/contact", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "theintercept.com", "fetchedAt": "2026-07-30T17:27:38.028133Z", "result": {"base": "https://theintercept.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User, PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The Intercept\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://theintercept.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5122 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://theintercept.com/about/#contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "The Intercept", "textOnly": "The Intercept Skip to main content Support Us Free Speech Crackdown: Germany Deports Bob Vylan Frontman for Anti-Israel Chants Hanno Hauenstein Iran Midterms Justice War on Gaza Unmasking ICE Chilling Dissent About Support Us Iran Midterms Justice War on Gaza Unmasking ICE Chilling Dissent Support Us Special Investigations Voices Podcasts Press Freedom Defense Fund Join the Newsletter About Contact Us More Ways to Donate Impact &#038; Reports Jobs Become a Source \u00a9 THE INTERCEPT ALL RIGHTS RESERVED Terms of Use Privacy Top Stories Voices Fauci\u2019s Diary Isn\u2019t the Key to Why Covid Ravaged the U.S. \u2014 Our Broken Healthcare System Is Mychal Denzel Smith Rand Paul is trying to put Anthony Fauci in prison for not confirming a conspiracy theory, but the key to blunting the next pandemic is universal healthcare. Donald Trump and Wars of Choice Tom Engelhardt, Nick Turse Trump\u2019s latest war is wrapped up in his egocentric narcissism and his me, me, me world. Targeting Iran Iran Has Figured Out How to Overwhelm U.S. Air Defenses Nick Turse As the U.S. runs short of interceptor missiles, Iran is carrying out strikes with a mix of attack drones and advanced ballistic missiles. Voices The Right Seeks to Capitalize on Berlin Pride Attack \u2014 Even as It Poses Its Own Existential Threat to Queer Lives Natasha Lennard Voices House Investigation of BreakThrough News Is an Attack on All Journalism Seth Stern Midterms 2026 Abdul El-Sayed Faces Islamophobic Attacks From GOP and Democratic Establishment Jonah Valdez Rubio\u2019s Paranoid Cuba Report Designed to \u201cJustify a Military Attack on Cuba,\u201d Says Democratic Rep. Noah Hurowitz Podcasts The Intercept Briefing Trump\u2019s National Killer Police Force The Intercept Briefing Trump\u2019s Formula for Forever Wars Voices Voices Fetterman Might Leave Democratic Party Because the Only Thing He Loves More Than Israel Is Attention Eoin Higgins Voices It\u2019s Scary As Hell Being Black in America Right Now Alain Stephens Join Our Newsletter Original reporting. Fearless journalism. Delivered to you. By signing up, I agree to receive emails from The Intercept and to the Privacy Policy and Terms of Use . I&#039;m in Become a Member By signing up, I agree to receive emails from The Intercept and to the Privacy Policy and Terms of Use . Iran U.S. Official: Iran Attacks Injured \u201cFar More\u201d Troops Than Pentagon Admits Nick Turse Iran Claims to Kill 3 U.S. Service Members in Kuwait Nick Turse Another Trump Ceasefire With Iran Crumbles Nick Turse The Surprising Reaction Inside Iran to Its War Victory S\u00e9amus Malekafzali Midterms William Lawrence Is Running in a Purple Michigan District \u2014 on His Pro-Palestine Credentials Matt Sledge Nancy Pelosi\u2019s Pick for Her SF House Seat Takes Anti-AIPAC Pledge Jonah Valdez Two-Thirds of Shri Thanedar&#8217;s Campaign Cash Came Through AIPAC as He Lost Over $600K Akela Lacy Democrats Are Desperate to Flip an Arizona House Seat. They\u2019re Rallying Around a Former Republican. Akela Lacy Justice Leaked Email Reveals New Requirements for ICE Vehicle Stops Karl Loftus U.S. Website Gave False Approval to Green Card Applicants \u2014 Then Blamed Glitch and Took It Away Noah Hurowitz ICE Mandates Dress Code After Killings and Botched Arrest Karl Loftus Google Is Censoring Reviews of ICE Detention Centers Sam Biddle War on Gaza On a West Bank Eviction Tour, Israeli Settlers Advertise \u201cNo Arabs Here\u201d Theia Chatelle Armed Israeli Settlers Detained Ro Khanna. He Wants Their Illegal Outposts Demolished. Jonah Valdez How to Show That Israel\u2019s Sexual Violence Against Palestinians Is Systemic \u2014 and Has Gone on for Decades Natasha Lennard The Performative Ceasefire in Gaza Unmasking ICE ICE Should Show It Hasn\u2019t Been \u201cInfiltrated by Violent Extremists,\u201d Senator Urges Matt Sledge, Sam Biddle ICE Recruitment Tweets Are So Racist That Cops Feared They Could Incite Neo-Nazi Violence Matt Sledge, Sam Biddle Government Ordered to Turn Over Files on ICE Agent Who Killed Renee Good Noah Hurowitz, Austin Campbell Pentagon Implores Civ", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://theintercept.com/about/#contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "buzzfeednews.com", "fetchedAt": "2026-07-30T17:27:43.421578Z", "result": {"base": "https://buzzfeednews.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"BuzzFeed News | Breaking News | Original Reporting | News An\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.buzzfeednews.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10880 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:{{).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "BuzzFeed News | Breaking News | Original Reporting | News Analysis", "textOnly": "BuzzFeed News | Breaking News | Original Reporting | News Analysis Skip To Content {title} BuzzFeed News {title} BuzzFeed News Home Reporting To You Menu Icon BuzzFeed News Sections Arts &amp; Entertainment Books Business Celebrity Culture Health Inequality Internet Culture Investigations JPG/Photo LGBTQ Rights National Opinion Politics Science Tech World Breaking News \u201cI Always Felt Bad For Her\u201d: People Are Re-Examining This ICONIC \u201cKUWTK\u201d Moment, And I Actually Think We Owe Kim Kardashian An Apology \"I always thought this scene was interpreted backwards...\" Ellen Durney &#8226; &nbsp;36 minutes ago \u201cWe Were Always Playing An Entirely Different Game\u201d: The Ultimate Oral History Of BuzzFeed News BuzzFeed News A Final Editor\u2019s Note Karolina Waclawiak Utilizamos cookies, pr\u00f3prios e de terceiros, que o reconhecem e identificam como um usu\u00e1rio \u00fanico, para garantir a melhor experi\u00eancia de navega\u00e7\u00e3o, personalizar conte\u00fado e an\u00fancios, e melhorar o desempenho do nosso site e servi\u00e7os. Esses Cookies nos permitem coletar alguns dados pessoais sobre voc\u00ea, como sua ID exclusiva atribu\u00edda ao seu dispositivo, endere\u00e7o de IP, tipo de dispositivo e navegador, conte\u00fados visualizados ou outras a\u00e7\u00f5es realizadas usando nossos servi\u00e7os, pa\u00eds e idioma selecionados, entre outros. Para saber mais sobre nossa pol\u00edtica de cookies, acesse link . Caso n\u00e3o concorde com o uso cookies dessa forma, voc\u00ea dever\u00e1 ajustar as configura\u00e7\u00f5es de seu navegador ou deixar de acessar o nosso site e servi\u00e7os. Ao continuar com a navega\u00e7\u00e3o em nosso site, voc\u00ea aceita o uso de cookies. Aceito Advertisement Breaking News \u201cI Always Felt Bad For Her\u201d: People Are Re-Examining This ICONIC \u201cKUWTK\u201d Moment, And I Actually Think We Owe Kim Kardashian An Apology \"I always thought this scene was interpreted backwards...\" Ellen Durney &#8226; &nbsp;36 minutes ago \u201cWe Were Always Playing An Entirely Different Game\u201d: The Ultimate Oral History Of BuzzFeed News BuzzFeed News A Final Editor\u2019s Note Karolina Waclawiak Sections Arts &amp; Entertainment Books Business Celebrity Culture Health Inequality Internet Culture Investigations JPG/Photo LGBTQ Rights National Opinion Politics Science Tech World Fans Are Remembering The Real-Life People Behind The Tragedies After \u201cGlee\u201d Star Kevin McHale Furiously Hit Back At Claims The Show Is \u201cCursed\u201d Over the years, \u201cGlee\u2019s\u201d lasting legacy has been tarnished by a series of tragedies and scandals, which have led to the \"curse\" rhetoric. Stephanie Soteriou &#8226; &nbsp;1 hour ago 1 hour ago After Detailing Her History Of Disordered Eating, Hilaria Baldwin Shared How She\u2019s Ensuring Her Seven Kids Have A Healthy Relationship With Food Hilaria \u2014 who started dieting when she was just 5 years old \u2014 noted that while she doesn\u2019t always approve of her kids eating unhealthy snacks, she\u2019s conscious not to \u201cshame\u201d them for doing so. Leyla Mohammed &#8226; &nbsp;1 hour ago 1 hour ago Amid All The \u201cOdyssey\u201d Awards Hype, Matt Damon\u2019s Emotional Comments About What It\u2019s Really Like To Win An Oscar Have Resurfaced Online \"It can't fill you up.\" Ellen Durney &#8226; &nbsp;2 hours ago 2 hours ago 7 Fascinating &#34;Little Shop Of Horrors&#34; Behind-The-Scenes Stories From Ellen Greene As The Film Turns 40 Four decades later, Ellen Greene shares heartfelt memories of originating Audrey onstage, working with Frank Oz, Rick Moranis, and Audrey II, and why \"Somewhere That's Green\" still moves her today. Andrew Firriolo &#8226; &nbsp;2 hours ago 2 hours ago A Harry Styles Fan Was So Excited To Ask Him For A Selfie That She Didn\u2019t Realize Her Baby\u2019s Stroller Was Rolling Into The ROAD \u2014 And People Are Horrified \u201cthis video is actually so scary:/ that stroller was rolling back into the street &amp; he was so concerned he went to go grab the stroller himself.\u201d Stephanie Soteriou &#8226; &nbsp;3 hours ago 3 hours ago After Nara Smith Pushed Back Against Being Labeled A \u201cTradwife,\u201d People Are Calling Out Other Influencers Who \u201cSell A Lifestyle\u201d That Normalizes A \u201cConservative Fantasy", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: google-extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "mailto:{{", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "huffpost.com", "fetchedAt": "2026-07-30T17:27:51.687368Z", "result": {"base": "https://huffpost.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: NewsMediaOrganization, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"HuffPost - Breaking News, Politics, Entertainment & Opinion\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.huffpost.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "11251 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.huffpost.com/static/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "HuffPost - Breaking News, Politics, Entertainment & Opinion", "textOnly": "HuffPost - Breaking News, Politics, Entertainment & Opinion Skip to Main Content Main Menu U.S. Edition News U.S. News World News Business Environment Health Social Justice Crime Politics Congress Extremism Opinion Entertainment Culture &amp; Arts Media Celebrity TV &amp; Film Sports Life Wellness Travel Tech Food &amp; Drink Style &amp; Beauty Family Relationships Money Home &amp; Living Work/Life Shopping Voices Black Voices Queer Voices Latino Voices Indigenous Voices Asian Voices Women&#x27;s Voices Voices of Disabled People HuffPost Personal Games Horoscopes Video Newsletters International U.S. U.K. Espa\u00f1a France \u0395\u03bb\u03bb\u03ac\u03b4\u03b1 (Greece) Italia \u65e5\u672c (Japan) \ud55c\uad6d (Korea) Follow Us Terms | Privacy Policy Part of HuffPost News. \u00a92026 BuzzFeed, Inc. All rights reserved. MORE FROM HUFFPOST Group Agrees To Boycott FIFA Competitions Over &#x27;Profound Failure Of Leadership&#x27; MS Now Host Sums Up How Gross Trump\u2019s Lindsey Graham Eulogy Was In Just 1 Sentence Donald Tusk Says Russian Missile Appears To Have Come Down In Eastern Poland Congo Says The Death Toll In Fastest-Growing Ebola Outbreak Has Passed 1,500 U.S. Launches Strikes On Iranian Targets A Day After It Foiled Missile Attack On American Forces Military May Require Some Troops In Mideast To Surrender Cell Phones, Sources Say \u2018I Walk Between 6 And 8 Miles EVERY DAY,\u2019 And These Inserts Have \u2018Saved My Feet\u2019 People In Their 50s Say This $20 Facial Hair Remover Is A \u2018Next-Level Holy Effing Grail\u2019 Joe Scarborough Enraged By 1 Trump Comment That He Says Got &#x27;Ho-Hum&#x27; Reaction Death Toll Rises To 18 As Rescuers Comb Through Quake Rubble In Southwestern Japan Major Gulf Fighting Restarts As U.S. And Saudi Strike Iran&#x27;s Allies In Iraq Ex-Trump Die-Hard Wacko Pleads For Congress To Impeach President 21 Years of The Huffington Post HUFFPOST NEWS POLITICS ENTERTAINMENT LIFE SHOPPING PERSONAL VOICES GAMES \u26a1 European Soccer Body Votes To Boycott World Cup... TRUMP THREATENS TO CUT BAIT ON BLANCHE! ERUPTS AT GOP HOLDOUTS LATEST UPDATES: Heroic Teen Lifeguard, Nancy Guthrie Case Update, Katie Miller On Her Relationship With Fauci, MORE\u2026 LATEST NEWS Ohio House Republican Faces Harrowing Domestic Abuse Allegations Risk Of All-Out War Surges Katie Miller Responds After Cozy Relationship With Anthony Fauci Revealed Andy Beshear: Mitch McConnell Is \u2018Ruining\u2019 His Legacy Clearer Picture Of U.S. Economy Emerges After Fed Meeting With Midterms On The Line GOP Senator Trolls Trump Using A Simple 3-Word Answer To CNN Reporter Judge Tosses Kash Patel\u2019s Suit Vs. Man Who Called Him \u2018Googly-Eyed\u2019 Chud Fox News Host Makes Chilling Trump Prediction: \u2018Everything In His Power\u2019 George And Amal Clooney Share Heartbreaking Message After Fleeing French Wildfire Father Who Gave His Son The Gun Used In A Georgia High School Shooting To Be Sentenced Musk Revives Super PAC To Help GOP Ahead Of Midterms: Report \u2018How Could Anybody Defend That?\u2019: Joe Rogan Breaks With Trump On Important Policy Position &#x27;Alarms Are Going Off&#x27;: Ex-Trump Aide Reveals What\u2019s Driving His Advisers Crazy Right Now Brutal Dictator\u2019s Grandson Disqualified From Major Boxing Bout Over Headbutt Senior Trump Official Using Government Account To Smear Sitting Federal Judges \ud83d\udc47 WHAT'S HAPPENING \ud83d\udc47 When I Was 22, My Mom Told Me A Shocking Secret. I Never Looked At My Father The Same Way Again. Lawrence O\u2019Donnell Drops The Hammer On GOP Over Fauci: \u2018Disgraced Themselves\u2019 Jordan Klepper Makes Unexpected Prediction About MAGA After Trump \u2018Huge Freaking Deal\u2019: MS NOW\u2019s Jen Psaki Reveals Devastating Detail In New Trump Poll I Kept This Secret To Myself Out Of Shame. Now I\u2019m Telling The Truth Many Black Women Know Too Well. Labor Dept\u2019s Weird New Trump Meme Backfires Spectacularly After Brutal Fact-Check FIFA Opens Disciplinary Case Over Argentina Conduct At World Cup DJ Kavinsky, Who Performed At Paris Olympics, Dead At 50 Mary Trump Says Donald Trump Has Lost One Of His Political Superpowers \u2018Practical Magic\u2019 Star Speaks Out After Not Be", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: OAI-SearchBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://www.huffpost.com/static/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "dailybeast.com", "fetchedAt": "2026-07-30T17:27:57.664707Z", "result": {"base": "https://dailybeast.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The Daily Beast: The Latest in Politics, Media &amp; Enterta\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.thedailybeast.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6327 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:missouriyoungrepublicans@proton.me\\).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 46, "title": "The Daily Beast: The Latest in Politics, Media &amp; Entertainment News", "textOnly": "The Daily Beast: The Latest in Politics, Media &amp; Entertainment News Skip to Main Content Got a Tip? Become a Subscriber Play our Crossword SUBSCRIBE Newsletters ALL CS CHEAT SHEET NEWSLETTERS LOG IN Cheat Sheet The Looker Media Obsessed Royals Politics Opinion Power 100 Innovation U.S. News Scouted My Account Manage Newsletters Need Help? LogOut Elaine Chao Snubs Missing McConnell in Lengthy Speech NOTHING TO SEE HERE The former transportation secretary urged Congress to act\u2014without acknowledging her own husband, a powerful Republican. Julia Ornedo Rubio Goons Get Every Single Country Wrong in Map Disaster LOST CONTINENT Ewan Palmer Trump Unveils His Latest Unfinished Gaudy Vanity Project BLING IT! Farrah Tomazin Elaine Chao Sparks New Mitch Mystery With Bizarre Appearance FLYING SOLO Josh Fiallo , Laura Esposito Trump\u2019s Big Gamble Blows Up in His Face in Devastating Poll FOR WHO THE BELL POLLS Leigh Kimmins opinion Why Trump\u2019s Base Attacks on Kaitlan Collins Demean Us All NASTY WOMEN Michael Ian Black Alec Baldwin\u2019s Wife Reveals 20-Year Health Battle MINDSET MAKEOVER Tamilore Oshikanlu GOP Rushes to Hide MAGA Enemy\u2019s Kindness to Katie Miller NARRATIVE BREAKER Josh Fiallo Trump Ignites GOP Backlash With Lindsey Graham\u2019s Sister NEPO SISTER Janna Brancolini Jaw-Dropping Scale of Voters\u2019 Fears Under Trump Revealed A VOTE OF NO CONFIDENCE Martha McHardy CNN Data Guru Stunned by Trump\u2019s \u2018Absolutely Huge\u2019 Loss of S Harry Thompson This Samurai Whodunit Is a Must-See Masterpiece Nick Schager Elaine Chao Snubs Missing McConnell in Lengthy Speech Julia Ornedo Country Star Ends Show Early and Leaves Stage in Tears Leigh Kimmins, Dana Herrnstadt Jordan Klepper Predicts What Happens to MAGA After Trump WHAT COMES NEXT? The \u201cDaily Show\u2019s\u201d resident MAGA whisperer finds some hope in the president\u2019s cult of personality. Obsessed: The Podcast , Dana Herrnstadt Musk\u2019s Massive Midterm Plot to Rescue Trump Revealed Janna Brancolini McConnell Aide Cornered on Why There\u2019s No Video of Senator Will Neal Trump Hit by Humiliating Economic Bombshell Leigh Kimmins \u201990s Sitcom Stars Pose for Iconic Pic After Years Apart Fiona O&#x27;Reilly Cheat Sheet Top 10 Right Now 1 NFL Great, 49, Reveals Heartbreaking Diagnosis RUSHING LEADER 2 What Lindsey Graham Taught Us About Political Weakness LYING IN STATE Shop with Scouted This Chef-Loved Brand Just Launched a \u2018Clean\u2019 Cookware Line PAN APPEAL 3 15 Elephants Die in Suspected Cyanide Poisoning DEADLY DINNER 4 Teen Fined Over Gross Social Media Straw-Licking Stunt LAST STRAW Partner update These Sleep Gels Don\u2019t Leave You Groggy the Next Day\u201430% Off DREAM COME TRUE AD BY Blue Sky CBD 5 Airline Reveals Cause of Terrifying Cabin Smoke Emergency CULPRIT FOUND See more cheats Newsletter Get caught up in minutes with our speedy summary of today\u2019s must-read stories. Sign Up By subscribing I have read and agree to the Terms of Use and Privacy Policy The Major Drawback of Height Surgery Might Finally Be Solved Tamilore Oshikanlu Alec Baldwin\u2019s Wife Reveals 20-Year Health Battle Tamilore Oshikanlu The 7 Best Supplements for Hair Growth, According To Experts Casey Clark Mom Gets Mistaken for Her Daughter After Facelift Roosa Rahkonen VISIT OBSESSED NOW Hot Takes Did Reality TV\u2019s Biggest Villain Just Vindicate Herself? A much-maligned Bravo star may have just pulled off a miracle in her latest episode of TV. Kevin Fallon Celebrity Legendary Singer Sounds Alarm on New \u2018Fight\u2019 in Open Letter Cameron Adams Celebrity Emmy Winner Reveals Why He Almost Rejected Most Famous Role Eboni Boykin-Patterson This \u2018Furious\u2019 Crime Show Is One of the Year\u2019s Best Series KILLER CONTENT The prolific creator of \u201cNew Girl\u201d continues her TV hot streak with the unflinching \u201cFurious.\u201d Emma Fraser \u2018Odyssey\u2019 Expert Exposes Christopher Nolan\u2019s Biggest Mistake Dana Herrnstadt Katy Perry Crowdsurfs in Giant Bottle in Stunt Gone Wrong Dana Herrnstadt The Anthony Bourdain Origin Story Is a Tasty Revelation Nick Schager \u2018Practical Magic\u2019 Star Responds ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:missouriyoungrepublicans@proton.me\\", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "vanityfair.com", "fetchedAt": "2026-07-30T17:28:05.966277Z", "result": {"base": "https://vanityfair.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, ItemList.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Vanity Fair -  Entertainment, Politics, and Fashion News | V\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.vanityfair.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12925 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://vanityfair.com/contact/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Vanity Fair -  Entertainment, Politics, and Fashion News | Vanity Fair", "textOnly": "Vanity Fair - Entertainment, Politics, and Fashion News | Vanity Fair Skip to main content Newsletters Menu Menu HOLLYWOOD ARTS &amp; CULTURE POLITICS STYLE ROYALS HOLLYWOOD Expand AWARDS AND FILM FESTIVALS MOVIES TV WHAT IS CINEMA? ALL HOLLYWOOD ARTS &amp; CULTURE POLITICS STYLE Expand FASHION BEAUTY ALL STYLE ROYALS BUSINESS MORE VIDEO NEWSLETTERS PODCASTS MAGAZINE ARCHIVE VF SHOP Search Search Vanity Fair Headphone Listen Politics Can Nithya Raman Win LA\u2019s Mayoral Race? In an exclusive interview, Nithya Raman shares new details about her dramatic break with Karen Bass and explains what she has in common with Spencer Pratt. By Maxwell Adler Photography by Ethan Noah Roy Headphone Listen MUSIC A Phish Pond: Finding the Joy of a Billion-Dollar Band By Claire Howorth Headphone Listen POLITICS Lindsey Graham\u2019s Funeral Was a Grand Exhibition of His Influence and His Servility By Aidan McLaughlin Headphone Listen Politics Andrew Tate Was the Toast of Trumpworld. Has That Changed? By Dan Adler Headphone Listen The Sweetest Sounds Inside the 30-Year Fight to Release Whitney Houston and Brandy\u2019s Cinderella Soundtrack Recording the album was a magical 48-hour whirlwind. So where is it? By Chris Murphy Headphone Listen FIT CHECK Michael B. Jordan is Remaking One of the Most Stylish Movies of the 90\u2019s In the \u201990s, Rene Russo wore fresh-off-the-runway fashion by Celine\u2019s Michael Kors, who was then a budding star. Can Jordan\u2019s remake pull off the same style coup? By Jos\u00e9 Criales-Unzueta LATEST STORIES INTERVIEW Headphone Listen INTERVIEW \u201cI Was the Perfect Victim\u201d: A Narrative History of Surviving Epstein Donald Trump once vowed to release the so-called Epstein files, but millions of pages of documents later, little action has been taken, and those subjected to Epstein\u2019s abuse are left feeling retraumatized. Vanity Fair speaks to five Epstein survivors about the predator, and their ongoing fights for justice. By Abby Ellin and Stacy Schneider Photography by Miranda Barnes Movies What the Venice and Toronto Lineups Tell Us About Oscar Season By Rebecca Ford Royals Prince William and Kate Middleton Release Their Summer Adventure Photos By Mathilde Engelmann Hollywood Animal Instinct: After Tiger King and Chimp Crazy, Filmmaker Eric Goode Exposes His Own Past With the Illegal Reptile Trade By Savannah Walsh Tech The Tech Bros Trying to Make Mining Sexy Again By Margaux MacColl Movies Tom Holland Is King of the Summer Box Office. Now It\u2019s Time for Him to Grow Up. By Chris Murphy VANITY FAIR Get unlimited access. PERSONAL HISTORY How I Blew Up&#8212;Then Rebuilt&#8212;My Life When news of her affair with Gavin Newsom exploded into the public eye, it was only the outward mark of a long-rippling private implosion. In the wreckage of the scandal, Ruby Rippey faced her demons but lost her voice. Twenty years later, she is ready to share her story. By Ruby Rippey Headphone Listen Dots Embark on an Odyssey Headphone Listen Film Matt Damon Struggled Through His Odyssey Transformation Headphone Listen Hollywood The Odyssey Stuntman Simone Moncada On Training to Steer Odysseus&#39;s Ship and Sharing a Scene With Zendaya Headphone Listen Awards Can The Odyssey Outdo Oppenheimer at the Oscars? A Case for Success. Headphone Listen Hollywood The Odyssey \u2019s Epic Box Office Haul Breaks The \u201cGo Woke, Go Broke\u201d Myth Headphone Listen commentary Christopher Nolan\u2019s Odyssey Is Going to Drive the Right Wing Completely Insane Headphone Listen MOVIES Everything to Know About Christopher Nolan&#39;s The Odyssey Headphone Listen Movies Samantha Morton Says Making The Odyssey Was a \u201cReligious\u201d Experience Most Popular Politics How I Blew Up My Life. (Or, the Gavin Newsom Affair) When news of her affair with Gavin Newsom exploded into the public eye, it was only the outward mark of a long-rippling private implosion. In the wreckage of the scandal, Ruby Rippey faced her demons but lost her voice. Twenty years later, she is ready to share her story. By Ruby Rippey Movies E", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}], "contactMatch": "https://vanityfair.com/contact/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "wired.com", "fetchedAt": "2026-07-30T17:28:13.256582Z", "result": {"base": "https://wired.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, ItemList.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"WIRED - The Latest in Technology, Science, Culture and Busin\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.wired.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9232 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:kate_knibbs@wired.com\\).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "WIRED - The Latest in Technology, Science, Culture and Business | WIRED", "textOnly": "WIRED - The Latest in Technology, Science, Culture and Business | WIRED Skip to main content Menu WIRED SECURITY POLITICS THE BIG STORY BUSINESS SCIENCE CULTURE REVIEWS Menu WIRED Account Account Newsletters Security Politics The Big Story Business Science Culture Reviews Chevron More Expand The Big Interview Magazine Events WIRED Insider WIRED Consulting Newsletters Podcasts Video Livestreams Merch Search Search WIRED Today\u2019s Picks Curating... Headphone Listen Flying Blind A Civilian Plane Crashed in New Mexico. Was the Military\u2019s Tech to Blame? Drone warfare is making the skies more dangerous, even for airplanes far from the battlefield. By Jeff Wise Do the Wrong Thing OpenAI\u2019s Hacking Debacle Comes Down to Human Error If the generative AI giant had followed well-known security best practices, it\u2019s likely that its AI agent would never have escaped to the open internet and hacked multiple companies. By Lily Hay Newman Big Hires DOGE Veterans Are Landing Big Jobs at Prediction Markets Polymarket and Novig have both recently brought ex-DOGE affiliates onboard as scrutiny of the industry intensifies. By Kate Knibbs Look Ma, No Hands For the First Time, Zoox Can Charge People for Rides in Its Steering-Wheel-Free Robotaxis By Aarian Marshall Restart Now Chrome Needs Twice-a-Week Patching Thanks to AI Bug Hunting By Lily Hay Newman Headphone Listen AI, Robot Google\u2019s Gemini Can Now Stomp Around as a Humanoid Robot By Will Knight Headphone Listen Against All Odds Don\u2019t Get Too Attached to Jimothy By Miles Klee Headphone Listen The Big Interview Can the New York Times Save Journalism From Our AI Overlords? By Katie Drummond Reviews Product Review I Dissected Every Aspect of Razer\u2019s First Hall Effect Keyboard Clean Machine The Best Vacuum Cleaners We\u2019ve Ever Tried Streaming Content Your Shower Water Might Be Messing With Your Hair and Skin&#8212;These Filters Help Smart Spending The Best Budget Smartwatch Doesn\u2019t Skimp on Performance Subscriber Exclusives Headphone Listen Inner Loop It Looks Like Nothing Can Dent MAGA\u2019s Support for ICE Despite weeks of renewed press coverage and controversy around ICE, Donald Trump\u2019s supporters appear to be standing by the agency no matter what. By David Gilbert Headphone Listen AI Lab It\u2019s Frighteningly Easy to Jailbreak Some Frontier AI Models Headphone Listen Made in China Chinese Companies Are Selling Vapes With Chemicals Potentially More Potent Than Nicotine Headphone Listen Backchannel Silicon Valley Is Completely Divided Over Chinese AI Headphone Listen Inner Loop The White House Is Trying to Figure Out What to Do About Chinese AI Politics Laura Loomer\u2019s Reversal on Ukraine Stirs Up the MAGA Civil War This Is Donald Trump\u2019s AI Brain Trust Trump Endorsed the MyPillow Guy as Minnesota Governor. The State GOP Isn\u2019t Buying It Security AI Scammers Are Better at Building Trust Than Humans ICE\u2019s New Detention Center Contracts Declare State Laws \u2018Shall Not Apply\u2019 A Typo Landed an Innocent Gamer in Prison for 18 Months Science Tropical Diseases Like Dengue Fever and Chikungunya Are on the Rise in Europe How to Bring a Geothermal Well Back From the Dead NASA\u2019s New 3D Model Shows the Earth Is a Lumpy Mess Business LinkedIn Won\u2019t Be Expanding Its Data Centers in the Next Year X Says Australia\u2019s Under-16 Social Media Ban Risks Interfering With Foreign Law Ebay Has to Pay $55.7 Million in Settlement for Its Unhinged Harassment Campaign Gear News The New Friend AI Pendant Can Now Talk Back to You I Got a Free Meal From a Private Chef&#8212;Who Filmed It All to Train Robots Love It or Hate It, Ferrari\u2019s First EV Is a Thrilling Drive Reviews TCL\u2019s Tab A1 Is a Budget Tablet That Does Not Feel Cheap Best Hotels.com Coupon Codes for August 2026 Top Dyson Promo Codes: 25% Off in August 2026 Culture Spider-Man: Brand New Day Is a Superhero Movie for Yearners More Typos, Fewer Em Dashes: Writers Are Creating an Anti-AI \u2018Literary Counterculture\u2019 Boomers Can\u2019t Stop Gifting Their Grandkids AI-Generated Slop Books What t", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, Amazonbot, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix\nDisallow: /"}], "contactMatch": "mailto:kate_knibbs@wired.com\\", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "theverge.com", "fetchedAt": "2026-07-30T17:28:19.184756Z", "result": {"base": "https://theverge.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User, PerplexityBot, Claude-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: NewsMediaOrganization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The Verge\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.theverge.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "20708 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://theverge.com/contact).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "The Verge", "textOnly": "The Verge Skip to main content The Verge The Verge logo. Tech Reviews Science Entertainment AI Policy Notifications Notifications Hamburger Navigation Button Notifications Notifications Hamburger Navigation Button Navigation Drawer The Verge The Verge logo. Login / Sign Up close Close Search Light System Dark Tech Expand Amazon Apple Facebook Google Microsoft Samsung Business See all tech Reviews Expand Smart Home Reviews Phone Reviews Tablet Reviews Headphone Reviews See all reviews Science Expand Space Energy Environment Health See all science Entertainment Expand TV Shows Movies Audio See all entertainment AI Expand OpenAI Anthropic See all AI Policy Expand Antitrust Politics Law Security See all policy Gadgets Expand Laptops Phones TVs Headphones Speakers Wearables See all gadgets Verge Shopping Expand Buying Guides Deals Gift Guides See all shopping Gaming Expand Xbox PlayStation Nintendo See all gaming Streaming Expand Disney HBO Netflix YouTube Creators See all streaming Transportation Expand Electric Cars Autonomous Cars Ride-sharing Scooters See all transportation Features Verge Video Expand TikTok YouTube Instagram Podcasts Expand Decoder The Vergecast Version History Newsletters Archives Store Verge Product Updates Subscribe Facebook Threads Instagram Youtube RSS The Verge The Verge logo. Top Stories Latest Following The Verge The Verge logo. 8 Verge Score A second lens makes DJI\u2019s Osmo Pocket 4P my new favorite video camera This is bigger than the Pocket 4, but definitely better. Dominic Preston 12:00 PM UTC Razer\u2019s new keyboards drop the price on powerful gaming features Antonio G. Di Benedetto Amazon\u2019s short story collections are one of the best reasons to own a Kindle Andrew Webster The US government just banned Roombas Sean Hollister Samsung\u2019s Galaxy Z Fold 8 feels like the future Dominic Preston Razer\u2019s new keyboards drop the price on powerful gaming features Antonio G. Di Benedetto Amazon\u2019s short story collections are one of the best reasons to own a Kindle Andrew Webster The US government just banned Roombas Sean Hollister Samsung\u2019s Galaxy Z Fold 8 feels like the future Dominic Preston The latest Quick Posts Banned in the USA / The Trump administration\u2019s list of banned foreign gadgets keeps getting longer and messier. The US government can\u2019t enforce its own drone ban Brendan Carr\u2019s FCC keeps approving the very drones it was supposed to ban. How did it go so wrong? Sean Hollister The US is banning foreign robots The ban will have a major impact on robots made in China. Emma Roth and Sean Hollister Sorry kid, drones are for war now The DJI ban was just the beginning. Sean Hollister The United States router ban, explained It\u2019s the latest Trump shakedown. Sean Hollister Review roulette / Catch up on some of our latest reviews, from games to gadgets. 7 Verge Score The Nothing Ear 3A look great\u2026 and sound good enough At $99, Nothing can get away with some minor flaws, especially when the earbuds look this good. John Higgins You don\u2019t need to splurge on an expensive handheld fan to beat the heat This cheap and basic fan from JisuLife has helped my family endure the hottest days of summer. Andrew Liszewski This is my new favorite laptop, but thanks to RAMageddon the price already went up by $800 The Framework Laptop 13 Pro has it all. But RAMageddon spiked its price. Antonio G. Di Benedetto Nintendo keeps making shooters more friendly and approachable Splatoon Raiders swaps competitive online play for a fresh take on looter shooters that you can play solo. Andrew Webster Most Popular 1 The US government just banned Roombas Sean Hollister Jul 29 2 What\u2019s the catch with the Apple Upgrade program? Emma Roth Jul 29 3 Microsoft confirms Copilot \u2018super app\u2019 coming this year Emma Roth Jul 29 4 Samsung\u2019s Galaxy Z Fold 8 feels like the future Dominic Preston Jul 29 5 Is it illegal to trick the US government into wiping your phone during a questionably legal search? Gaby Del Valle Jul 28 Smorgasbord of the rings / Smart rings", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: meta-externalagent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Claude-User\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"Amazonbot\" has no root Disallow."}], "contactMatch": "https://theverge.com/contact", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "techcrunch.com", "fetchedAt": "2026-07-30T17:28:24.938174Z", "result": {"base": "https://techcrunch.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: ChatGPT-User -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: CollectionPage, BreadcrumbList, WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"TechCrunch | Startup and Technology News\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://techcrunch.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12719 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://techcrunch.com/contact-us/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "TechCrunch | Startup and Technology News", "textOnly": "TechCrunch | Startup and Technology News TechCrunch Desktop Logo TechCrunch Mobile Logo Latest Startups Venture Apple Security AI Apps Disrupt 2026 Events Podcasts Newsletters Search Submit Site Search Toggle Mega Menu Toggle Topics Latest AI Amazon Apps Biotech &amp; Health Climate Cloud Computing Commerce Crypto Enterprise EVs Fintech Fundraising Gadgets Gaming Google Government &amp; Policy Hardware Instagram Layoffs Media &amp; Entertainment Meta Microsoft Privacy Robotics Security Social Space Startups TikTok Transportation Venture More from TechCrunch Staff Events Startup Battlefield StrictlyVC Newsletters Podcasts Videos Partner Content TechCrunch Brand Studio Crunchboard Contact Us Image Credits: TechCrunch Security In the Hugging Face breach, OpenAI&#8217;s hacker was noisy and fast \u2014 but not unstoppable Lorenzo Franceschi-Bicchierai 3 hours ago AI Forward-deployed engineers are the AI industry\u2019s latest talent obsession Rebecca Bellan 2 hours ago Transportation Zoox clears final federal hurdle to launch paid robotaxi service Kirsten Korosec 4 hours ago Top Headlines FTC sues Hims &amp; Hers for allegedly sharing patients&#8217; medical data with advertisers Meta and Snap Zack Whittaker Microsoft is openly competing with OpenAI, Anthropic more than ever Julie Bort Mark Zuckerberg predicts that billions of people will have personal AI agents in five years Amanda Silberling The Hugging Face break-in explained Connie Loizos Claude Opus 5 became downright ruthless when tasked with running a vending machine Julie Bort Image Credits: TC TechCrunch Disrupt 2026 The Disrupt Stage&#039;s first major speakers have been revealed TechCrunch Events 3 hours ago TechCrunch Disrupt 2026 Introducing the new Smart Money stage TechCrunch Events Jul 24, 2026 TechCrunch Disrupt 2026 Get a first look at our new AI infra-focused stage TechCrunch Events 3 days ago TechCrunch Disrupt 2026 No product? No problem. This session shows you how to get pre-seed funding TechCrunch Events Jul 17, 2026 Latest News See More Security Okta buys AI security startup Permiso; source says for about $200M Jagmeet Singh 1 hour ago Transportation Tesla made its 10 millionth EV Sean O&#039;Kane 1 hour ago Climate When will fusion power startup Commonwealth Fusion Systems go public? Tim De Chant 2 hours ago Apps Meta says AI is making it easier to build new apps \u2014 and more are coming Sarah Perez 2 hours ago Media &amp; Entertainment Netflix lands global streaming deal for \u2018The Walking Dead\u2019 Lauren Forristal 2 hours ago AI Nscale buys Anyscale as it seeks to own more of the AI compute stack Ram Iyer 2 hours ago Climate Fusion power darling Commonwealth Fusion Systems raises another $1B Tim De Chant 2 hours ago AI Forward-deployed engineers are the AI industry\u2019s latest talent obsession Rebecca Bellan 2 hours ago Security In the Hugging Face breach, OpenAI&#8217;s hacker was noisy and fast \u2014 but not unstoppable Lorenzo Franceschi-Bicchierai 3 hours ago TechCrunch Disrupt 2026 TechCrunch Disrupt 2026\u2019s biggest stage features leaders from Amazon, Replit, Tether, with much more to come TechCrunch Events 3 hours ago Next October 13 &#8211; 15 San Francisco Scale faster. Grow your portfolio. Gain practical expertise. No matter your goal, Disrupt can empower you. Save up to $330 toda y! REGISTER NOW Most Popular Claude Opus 5 became downright ruthless when tasked with running a vending machine Julie Bort Sam Altman is ready to decelerate Tim Fernholz Librarians are hosting viral &#8216;Avoiding AI&#8217; workshops for people who are fed up with Big Tech Amanda Silberling SpaceX launches new V3 Starlink satellites but suffers another booster failure Sean O&#039;Kane Prentis, new AI lab co-founded by Reid Hoffman, Mark Pincus in talks to raise $100M Marina Temkin US accuses American of allegedly wiping his phone using a &#8216;duress&#8217; password during border search Zack Whittaker Anduril reportedly in talks to raise funding at $100B valuation, more than 3x last year&#821", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: Claude-Web, ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: ChatGPT-User\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://techcrunch.com/contact-us/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "arstechnica.com", "fetchedAt": "2026-07-30T17:28:31.691044Z", "result": {"base": "https://arstechnica.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebSite, WebPage, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Ars Technica - Serving the Technologist since 1998. News, re\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://arstechnica.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10263 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:adinquiries@condenast.com).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "Ars Technica - Serving the Technologist since 1998. News, reviews, and analysis.", "textOnly": "Ars Technica - Serving the Technologist since 1998. News, reviews, and analysis. Skip to content Ars Technica home Sections Forum Subscribe Search AI Biz &amp; IT Cars Culture Gaming Health Policy Science Security Space Tech Feature Reviews AI Biz &amp; IT Cars Culture Gaming Health Policy Science Security Space Tech Forum Subscribe Theme HyperLight Day & Night Dark System Search Sign In Sign in dialog... Sign in Ars Technica homepage Grid Classic Grid List Neutron Star Classic Grid List Neutron Star grid settings grid settings Grid Included stories AI Apple Biz &amp; IT Cars Culture Gaming Google Health Policy Science Security Shopping Space Tech = visible = hidden Most read Any AI Biz &amp; IT Cars Culture Gaming Health Policy Science Security Space Tech Disable Most Read module throughout the site Most Read lists the most popular stories on Ars. By default it will pull from all sections, but you can optionally choose a single section to highlight. Cancel Save These settings are only available for Ars Pro subscribers Discover all the benefits of an Ars sub , including a 100% ad and tracker free experience and advanced news layouts and custom settings. Dismiss Subscribe Now! View not permitted witout subscription These settings are only available for Ars Pro subscribers Discover all the benefits of an Ars sub , including a 100% ad and tracker free experience and advanced news layouts and custom settings. Dismiss Subscribe Now! Featured Real problems Google&#8217;s SynthID watermark is hard to break, but it doesn&#8217;t solve AI disinformation Deciding what&#8217;s real on the Internet won&#8217;t be easy in the future. Ryan Whitwam \u2013 7/29/2026 | 82 Quantum computers outperform classical ones, with results you can trust Three approaches to the issue of quantum results that can&#8217;t be verified classically. John Timmer \u2013 7/30/2026 | 18 New MCP specification addresses the main barrier to enterprise adoption Plus, a new policy for the AI protocol ensures features aren&#8217;t removed suddenly. Samuel Axon \u2013 7/30/2026 | 30 Mythos attack on 3rd-round PQC algorithm candidate puts it out of commission HAWK withstood years of testing that had yet to uncover a fatal weakness found through Mythos. Dan Goodin \u2013 7/29/2026 | 51 Musk went to \u201cwar,\u201d sought jail time for X ad boycotts\u2014but case ends with a whimper Advertisers agreed to \u201creset\u201d relationship with X. Ashley Belanger \u2013 7/29/2026 | 151 Quantum computers outperform classical ones, with results you can trust Three approaches to the issue of quantum results that can&#8217;t be verified classically. John Timmer \u2013 7/30/2026 | 18 New MCP specification addresses the main barrier to enterprise adoption Plus, a new policy for the AI protocol ensures features aren&#8217;t removed suddenly. Samuel Axon \u2013 7/30/2026 | 30 Mythos attack on 3rd-round PQC algorithm candidate puts it out of commission HAWK withstood years of testing that had yet to uncover a fatal weakness found through Mythos. Dan Goodin \u2013 7/29/2026 | 51 Musk went to \u201cwar,\u201d sought jail time for X ad boycotts\u2014but case ends with a whimper Advertisers agreed to \u201creset\u201d relationship with X. Ashley Belanger \u2013 7/29/2026 | 151 Who wins and who loses after US bans foreign robots? Government ban on foreign-made robots may hinder instead of help US robotics. Jeremy Hsu \u2013 7/29/2026 | 78 Comcast store punished low sales by smashing pies in workers&#8217; faces, lawsuit claims Comcast, accused of negligence, says lawsuit mischaracterized alleged events. Jon Brodkin \u2013 7/29/2026 | 98 Customize Ars your way with an Ars Pro subscription Ars Pro subscribers get customizable layouts along with no ads and no trackers. Eric Bangeman \u2013 7/29/2026 | 123 Google&#8217;s &#8220;privacy-preserving&#8221; age verification system is coming to the Play Store Google&#8217;s new API relies on parents to set age ranges in Family Link. Ryan Whitwam \u2013 7/29/2026 | 37 Elon Musk\u2019s xAI is trying to sue its way out of a Grok reckoning Musk defends Grok, says Minnesota", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: Google-Extended, Google-CloudVertexBot, GoogleOther, Applebot-Extended, meta-webindexer, meta-externalagent, meta-externalfetcher, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, cohere-training-data-crawler, cohere-ai, CCBot, PanguBot, PetalBot, Bytespider, Diffbot, DuckAssistBot, MistralAI-User, Timpibot, Webzio, Webzio-Extended, archive.org_bot, ia_archiver, ia_archiver-web.archive.org, heritrix, anthropic-ai, Claude-Web, FacebookBot, Omgilibot, YouBot\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "mailto:adinquiries@condenast.com", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "engadget.com", "fetchedAt": "2026-07-30T17:28:36.973610Z", "result": {"base": "https://engadget.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Engadget | Technology News & Expert Reviews\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.engadget.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9580 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Engadget | Technology News & Expert Reviews", "textOnly": "Engadget | Technology News & Expert Reviews Latest News AI Apps Computing Mobile Social Media EVs & Transportation Reviews Smartphones Laptops & PCs Gaming Headphones Wearables Photography Tablets Home Buying Guides Laptops Headphones Smart Home Gaming Best Gaming Handhelds Best E-Ink Tablets Best Power Banks Best Fast Chargers Best Live TV Streaming Best Wireless Earbuds Gaming Nintendo PC PlayStation Xbox Big Tech Amazon Apple Google Meta Microsoft Samsung Entertainment TV & Movies Streaming Cybersecurity VPN Wearables Tomorrow Science Space Robotics Newsletter About Editorial Policies No-AI Writing Policy Reviews Policy Podcast Archive Privacy Policy Terms of Use &copy; 2026 Static Media . All Rights Reserved Engadget | Technology News & Expert Reviews Latest News AI Apps Computing Mobile Social Media EVs & Transportation Reviews Smartphones Laptops & PCs Gaming Headphones Wearables Photography Tablets Home Buying Guides Laptops Headphones Smart Home Gaming Best Gaming Handhelds Best E-Ink Tablets Best Power Banks Best Fast Chargers Best Live TV Streaming Best Wireless Earbuds Gaming Nintendo PC PlayStation Xbox Big Tech Amazon Apple Google Meta Microsoft Samsung Entertainment TV & Movies Streaming Cybersecurity VPN Wearables Tomorrow Science Space Robotics Newsletter Gaming Lego Pok\u00e9mon Smart Play hands-on: A better way to build, train and battle With its latest Smart Play sets, Lego managed put its own spin on one of the world's most popular franchises. By&nbsp; Sam Rutherford Read More Photography DJI Osmo Pocket 4P review: The best gimbal camera money can't buy The OG gimbal camera goes \"pro\" with two lenses, class-leading dynamic range and optical zoom. By&nbsp; James Trew Read More Big Tech The US government's robot ban also includes vacuums Robots apparently don't need to be bipedal to be caught up in the FCC's new foreign-made robot ban. By&nbsp; Ian Carlos Campbell Read More Laptops &amp; PCs Framework Laptop 13 Pro review: A repairable notebook at the top of the class The Framework 13 Pro is a class-leading Windows laptop that, right now, is just a little too pricey. By&nbsp; Daniel Cooper Read More Smartphones Samsung Galaxy Z Fold 8 review: Wider really is better For people who want the power of a phone that transforms into a miniature tablet, the Galaxy Z Fold 8 offers a better way to watch shows and movies. By&nbsp; Sam Rutherford Read More More Stories 0 Social Media Social Media By&nbsp; Karissa Bell 23 min ago LinkedIn is testing a 'seems like AI slop' reporting tool The company is also removing some of its own AI-writing features. By&nbsp; Karissa Bell 23 min ago Read More 0 Mobile Mobile By&nbsp; Max Miller 56 min ago Expensive phone cases: When and why it's worth paying more Phone cases vary dramatically when it comes to pricing, but it's not always as simple as \"you get what you pay for.\" By&nbsp; Max Miller 56 min ago Read More 0 News News By&nbsp; Mariella Moon 1 hour ago ChatGPT and Roblox will reportedly be subject to tighter rules in the EU Bloomberg says ChatGPT and Roblox will soon be added to the EU\u2019s \u2018very large online platform\u2019 list, which subjects them to stricter rules. By&nbsp; Mariella Moon 1 hour ago Read More 0 Google Google By&nbsp; Kris Holt 1 hour ago Google says it has fixed a Pixel battery drain issue, but some users disagree Google said a July update resolves the problem on recent devices, with Pixel 9a and 8a getting a fix in September. By&nbsp; Kris Holt 1 hour ago Read More 0 Google Google By&nbsp; Christine Persaud 2 hours ago Google Voice paid subscriptions offer more than just a phone number The free version may be enough for you, but let's dive into the upgraded features to be sure. By&nbsp; Christine Persaud 2 hours ago Read More 2 Streaming Streaming By&nbsp; Max Miller 4 hours ago How to use screen mirroring on Roku with your Android phone Your Roku can probably display what's on your phone screen, if you have Android. By&nbsp; Max Miller 4 hours ago Read More 3 TV &amp; ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "gizmodo.com", "fetchedAt": "2026-07-30T17:28:41.778041Z", "result": {"base": "https://gizmodo.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://gizmodo.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "mashable.com", "fetchedAt": "2026-07-30T17:28:46.719919Z", "result": {"base": "https://mashable.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://mashable.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "cnet.com", "fetchedAt": "2026-07-30T17:28:52.648686Z", "result": {"base": "https://cnet.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, ImageObject, WebPage, Person, Article.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"CNET: Product reviews, advice, how-tos and the latest news -\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.cnet.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "18059 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:brand_licensing@ziffdavis.com).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "CNET: Product reviews, advice, how-tos and the latest news - CNET", "textOnly": "CNET: Product reviews, advice, how-tos and the latest news - CNET Skip to content Your Guide To a Better Future Add as a preferred source on Google Open Close Your Guide To a Better Future AI Tech Home Entertainment Reviews Best Products News Deals Search Search Close AI AI Atlas Best AI Image Generators Best AI Chatbots Best AI Website Builders ChatGPT vs. Gemini Tech Tech Mobile Home Entertainment VPN Computing Services &amp; Software Gaming Laptops Streaming Services Web Hosting Home Home Internet Home Security Kitchen &amp; Household Smart Home Energy &amp; Utilities Yard &amp; Outdoors Smart Plugs Entertainment Culture Internet Culture Entertainment Sports Fashion Wellness Nutrition Fitness Personal Care Medical Mental Health Cover Stories Reviews Best Products Best VPN Services Best Web Hosting Services Best TVs Best Identity Theft Protections Best Mesh Wi-Fi Routers Best Robot Vacuums Best Smart Locks Best Phone Plans Best Printers Best Laptops Best Bluetooth Speakers Best Wireless Earbuds Best Noise-Canceling Headphones Best Noise-Canceling Earbuds Best Home Security Cameras Best Outdoor Home Security Cameras Best Video Doorbells Best Air Purifiers Best Smart Thermostats Best Smoke Detectors Best Phones News Deals Add as a preferred source on Google Best Editors&#8217; picks and our top buying guides Best VPN Service for 2026: The Top-Ranked VPNs in Our Testing Best Cellphone Plans of 2026: Our Top Picks Best TVs of 2026: Expert Tested and Reviewed Best Headphones We&#8217;ve Tested 8 Years and 200 Meal Kits Later, These Are the Best Meal Delivery Services to Try Galaxy Z Fold 8 Review: A Passport-Size Shape-Shifter for Every Moment Samsung&#039;s reimagined book-style foldable is compact and versatile, making it the ideal device for consuming all kinds of content. It&#039;s also just delightful to use. By Abrar Al-Heeti CNET Readers Are Convinced Apple Has a Foldable iPhone Coming. But What About Smart Glasses? Here&#039;s what CNET readers predict Apple will do this fall. By Dashia Milden Filmmaking Beyond the &#8216;Slop&#8217; of Generative AI CNET was given exclusive access to 11 short films created by Google Labs Flow Sessions resident artists. By Macy Meyer I Lived Without Cooking Gas for Over a Month. These Hacks Helped Me Feed My Family This tech lets you prepare meals without turning on your stove, making it perfect for heat waves. By Giselle Castro-Sloboda Scotland and First-Person Screams Are Silent Hill: Townfall&#8217;s New Direction of Intimate Horror A hands-on preview of a smaller, more intimate Silent Hill game &#8212; with the ambiguous psychological horror fans expect. By David Lumb Upgrade your inbox Get CNET Insider From talking fridges to iPhones, our experts are here to help make the world a little less complicated. Thanks for signing up! Email address Subscribe Subscribing&#8230; By signing up, you confirm you are 16+, will receive newsletters and promotional content and agree to our Terms of Use and acknowledge the data practices in our Privacy Policy . You may unsubscribe at any time. TODAY&#039;S DEALS Our shopping experts unearth the best tech and home essential deals every day. If you make a purchase using our links, CNET may earn a commission. Score Up to $350 Off Samsung\u2019s Next-Gen Smartwatches With These Preorder Deals Walmart Deals of the Day: Hit the Ground Running With $71 Off a Garmin Fitness Tracker These Are the Standout Deals CNET\u2019s Shopping Experts Are Buying This Week These Soundpeats Noise-Canceling Earbuds Just Plunged to a New Record-Low Price of $38 MORE TO EXPLORE I\u2019ve Been Testing Handheld Fans All Summer. This Is the $50 One I Throw in My Bag By Anna Gragert LinkedIn Added a New Way to Report AI Slop By Katelyn Chedraoui I Used the &#8216;Thermometer of the Future&#8217; to Listen to My Own Heart and Lungs By Anna Gragert Hugging Face Users Easily Created and Shared Abusive AI Images, Study Finds By Katelyn Chedraoui I Test Coffee Gear for a Living. Here&#8217;s Why I Us", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: AddSearchBot, AI2Bot, Ai2Bot-DeepResearchEval, Ai2Bot-Dolma, AIWebIndex, amazon-QBusiness, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Anchor Browser, anthropic-ai, ApifyBot, ApifyWebsiteContentCrawler, Applebot, Applebot-extended, atlassian-bot, AutoRAG, AwarioSmartBot, AzureAI-SearchBot, Big Sur AI, bigsur.ai, Brandwatch, Bravebot, Brightbot, Bytespider, CCBot, Channel3Bot, ChatGLM-Spider, ChatGPT-User, Claude-Code, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Code, cohere-ai, cohere-training-data-crawler, Cotoyogi, Crawl4AI, Cursor, Datenbank Crawler, Datenbank-Crawler, DeepSeekBot, Devin, Diffbot, Direqt Anomura, DuckAssistBot, Echobot Bot, ExaBot, FacebookBot, Factset_spyderbot, FirecrawlAgent, GeistHaus-PageFetcher, GPTBot, iAsk, iAskBot, iaskspider, ICC Crawler, ICC-Crawler, ImageSiftBot, imageSpider, kagi-fetcher, Kangaroo Bot, Kangaroo-Bot, Kimi-User, KlaviyoAIBot, Kunato, laion-huggingface-processor, LCC, LINER Bot, LinerBot, LinkupBot, Manus-User, meta-externalagent, meta-externalfetcher, meta-webindexer, mistral.ai, MistralAI-User, netEstate Imprint Crawler, NovaAct, Novellum, Novellum AI Crawl, OAI-SearchBot, omgili, opencode, PanguBot, PeopleInc-DCipher-Scraper/1.1.0, Perplexity-User, PerplexityBot, PetalBot, PhindBot, Poggio-Citations, QualifiedBot, SBIntuitionsBot, SeekrBot, SemrushBot-OCOB, SemrushBotSwa, Shap-User, ShapBot, Spider, TavilyBot, TerraCotta, Timpibot, TongyiBot, Trae, TwinAgent, UseAI, Velen Crawler, VelenPublicWebCrawler, Webzio-Extended, Wrtn, YiyanBot, YouBot, ZanistaBot\nDisallow: /"}], "contactMatch": "mailto:brand_licensing@ziffdavis.com", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "sept15-news", "domain": "zdnet.com", "fetchedAt": "2026-07-30T17:28:57.564556Z", "result": {"base": "https://zdnet.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://zdnet.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "sept15-news", "domain": "venturebeat.com", "fetchedAt": "2026-07-30T17:29:02.577253Z", "result": {"base": "https://venturebeat.com", "fatal": "Their server refused or timed out (status 429). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://venturebeat.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "hyundaiusa.com", "fetchedAt": "2026-07-30T17:29:25.757636Z", "result": {"base": "https://hyundaiusa.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (39668 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Hyundai USA: Cars, SUVs, &amp; Electric Vehicles | Official \u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.hyundaiusa.com/us/en).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1611578 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:18006335151).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 74, "title": "Hyundai USA: Cars, SUVs, &amp; Electric Vehicles | Official Site", "textOnly": "Hyundai USA: Cars, SUVs, &amp; Electric Vehicles | Official Site --> For disability accessibility concerns, please contact us at 1-800-633-5151 or [email&#160;protected] | Hyundai\u2019s accessibility efforts are guided by WCAG 2.0 AA. Skip to Main Content &lt;g> &lt;path fill-rule=&#34;evenodd&#34; clip-rule=&#34;evenodd&#34; d=&#34;M28.8798 2.77594C31.1289 4.03763 33.675 6.2559 33.4094 9.2138C33.1018 11.9544 30.6087 13.8685 28.3813 15.0208C21.8789 18.2069 12.746 18.3057 6.03136 15.5213C3.76154 14.5528 1.21547 12.8901 0.260462 10.3883C-0.471039 8.3335 0.419934 6.19056 1.91526 4.76632C5.04494 1.85124 9.03298 0.850894 13.1387 0.274756C17.5933 -0.279812 22.282 0.209421 26.2707 1.6121C27.1616 1.94971 28.0422 2.31922 28.8798 2.77594ZM25.2098 2.68903C25.255 2.71373 25.3027 2.73401 25.3501 2.75414C25.4696 2.80486 25.5869 2.85468 25.655 2.97194C25.7238 3.12167 25.6258 3.22842 25.5353 3.32697C25.514 3.35023 25.493 3.37304 25.4748 3.39584C23.2794 5.15739 20.765 6.13648 18.1985 6.99491C17.9204 7.06444 17.6419 7.14582 17.3616 7.22769C16.3264 7.53011 15.2682 7.83926 14.1248 7.58292C13.7854 7.49571 13.4778 7.2675 13.2866 6.9302C12.9265 5.90797 13.4147 4.84228 13.8385 3.9501C14.4008 2.83065 15.3979 1.51394 16.777 1.49299C19.7469 1.47048 22.5476 1.92752 25.2098 2.68903ZM11.3241 2.33016L11.3668 2.16666C11.301 2.0419 11.1439 2.04444 11.0057 2.04667C10.9878 2.04696 10.9701 2.04725 10.9531 2.04725C9.53183 2.27545 8.18439 2.67809 6.89031 3.17827C4.70496 4.12453 2.0851 5.51689 1.43806 8.12624C1.03557 10.1272 2.43514 11.8131 3.98413 12.8569C4.06614 12.8945 4.14966 12.953 4.23451 13.0125C4.46767 13.1759 4.71082 13.3463 4.95987 13.107C6.69974 9.3551 8.53505 5.5816 11.3241 2.33016ZM29.8772 5.03798C31.1927 6.14711 32.3809 7.65889 31.9775 9.48577C31.3195 12.1826 28.4557 13.6181 26.1755 14.5313L26.1041 14.5557C24.8859 14.9727 23.6565 15.3935 22.3244 15.5213C22.3089 15.5193 22.2916 15.5184 22.2736 15.5175C22.1931 15.5133 22.0979 15.5083 22.0805 15.4013L22.1122 15.249C24.0432 13.0201 25.5071 10.5193 26.7902 7.98494C27.3848 6.84329 27.9364 5.66851 28.4453 4.51529C28.5194 4.4287 28.5935 4.38556 28.6789 4.35274C29.034 4.38532 29.313 4.60457 29.59 4.82219C29.685 4.89684 29.7797 4.9713 29.8772 5.03798ZM19.3547 14.0776C19.8898 13.0766 20.4465 12.035 20.2134 10.8015C20.0439 10.3445 19.5667 10.0075 19.0895 9.94249C17.9398 9.79084 16.853 10.1122 15.8295 10.4148L15.8113 10.4202C15.688 10.4625 15.5646 10.5048 15.4412 10.547C12.8052 11.4498 10.1585 12.3563 7.91941 14.2165C7.8133 14.3143 7.73921 14.4547 7.79226 14.6076C7.85599 14.7167 7.97277 14.792 8.08895 14.8251C10.6561 15.6188 13.4244 16.0656 16.2574 16.1087C17.5195 16.2387 18.5696 15.3688 19.2057 14.3578C19.2547 14.2647 19.3046 14.1713 19.3547 14.0776Z&#34; fill=&#34;white&#34; /> &lt;/g> &lt;/svg>\" media=\"(max-width: 639px)\"/> &lt;g> &lt;path fill-rule=&#34;evenodd&#34; clip-rule=&#34;evenodd&#34; d=&#34;M28.8798 2.77594C31.1289 4.03763 33.675 6.2559 33.4094 9.2138C33.1018 11.9544 30.6087 13.8685 28.3813 15.0208C21.8789 18.2069 12.746 18.3057 6.03136 15.5213C3.76154 14.5528 1.21547 12.8901 0.260462 10.3883C-0.471039 8.3335 0.419934 6.19056 1.91526 4.76632C5.04494 1.85124 9.03298 0.850894 13.1387 0.274756C17.5933 -0.279812 22.282 0.209421 26.2707 1.6121C27.1616 1.94971 28.0422 2.31922 28.8798 2.77594ZM25.2098 2.68903C25.255 2.71373 25.3027 2.73401 25.3501 2.75414C25.4696 2.80486 25.5869 2.85468 25.655 2.97194C25.7238 3.12167 25.6258 3.22842 25.5353 3.32697C25.514 3.35023 25.493 3.37304 25.4748 3.39584C23.2794 5.15739 20.765 6.13648 18.1985 6.99491C17.9204 7.06444 17.6419 7.14582 17.3616 7.22769C16.3264 7.53011 15.2682 7.83926 14.1248 7.58292C13.7854 7.49571 13.4778 7.2675 13.2866 6.9302C12.9265 5.90797 13.4147 4.84228 13.8385 3.9501C14.4008 2.83065 15.3979 1.51394 16.777 1.49299C19.7469 1.47048 22.5476 1.92752 25.2098 2.68903ZM11.3241 2.33016L11.3668 2.16666C11.301 2.0419 11.1439 2.04444 11.0057 2.04667C10.9878 2.04696 10.9701 2.04725 10.9531 2.04725C9.53183 2.27545 8.18439 2.67809 6.89031 3.17827C4.70496 4.12453 2.085", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:18006335151", "sept15": {"behindCloudflare": true, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "posture-choice", "detail": "Served through Cloudflare with no ad code detected, currently readable by AI. The September 15 default targets ad-bearing pages, so no automatic flip is expected here. But this site\u2019s AI access is now a Cloudflare dashboard setting \u2014 block, allow, or charge under Pay Per Use, where publishers get paid when content is actually used in an AI answer. Confirm the current posture is a decision, not a default."}}}
{"corpus": "auto", "domain": "kia.com", "fetchedAt": "2026-07-30T17:29:32.449694Z", "result": {"base": "https://kia.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation, Brand, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Kia America Official Site: Discover the New Kia\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.kia.com/us/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "56068 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact link present.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 69, "title": "Kia America Official Site: Discover the New Kia", "textOnly": "Kia America Official Site: Discover the New Kia Vehicles SUV / CUV / MPV Hybrid / Electric Sedan Upcoming Show All&nbsp; Certified Pre-Owned&nbsp; SUV / CUV / MPV Seltos $24,990 starting MSRP * *\u201cStarting MSRP&quot; price is manufacturer\u2019s suggested retail price (MSRP) for lowest model trim. See &quot;Build and Price&quot; section for MSRP of model shown. MSRP excludes destination and handling charges, taxes, title, license fees, options, and dealer charges. Vehicles shown with optional equipment. Actual price set by dealer and may vary. The applicable destination and handling charge per model are provided in the \u201cBuild and Price\u201d section. See dealer for details. Some features only available as part of packages. See &quot;Features &amp; Specifications&quot; and &quot;Options&quot; pages for details. Kia America, Inc. reserves the right to make changes at any time as to vehicle availability, destination, and handling fees, colors, materials, specifications, features, accessories, packages, and models. Not responsible for typographical or computer errors. Build Find Nearby Sportage $28,790 starting MSRP * *\u201cStarting MSRP&quot; price is manufacturer\u2019s suggested retail price (MSRP) for lowest model trim. See &quot;Build and Price&quot; section for MSRP of model shown. MSRP excludes destination and handling charges, taxes, title, license fees, options, and dealer charges. Vehicles shown with optional equipment. Actual price set by dealer and may vary. The applicable destination and handling charge per model are provided in the \u201cBuild and Price\u201d section. See dealer for details. Some features only available as part of packages. See &quot;Features &amp; Specifications&quot; and &quot;Options&quot; pages for details. Kia America, Inc. reserves the right to make changes at any time as to vehicle availability, destination, and handling fees, colors, materials, specifications, features, accessories, packages, and models. Not responsible for typographical or computer errors. Build Find Nearby Hybrid/Electric Available Sorento $32,390 starting MSRP * *\u201cStarting MSRP&quot; price is manufacturer\u2019s suggested retail price (MSRP) for lowest model trim. See &quot;Build and Price&quot; section for MSRP of model shown. MSRP excludes destination and handling charges, taxes, title, license fees, options, and dealer charges. Vehicles shown with optional equipment. Actual price set by dealer and may vary. The applicable destination and handling charge per model are provided in the \u201cBuild and Price\u201d section. See dealer for details. Some features only available as part of packages. See &quot;Features &amp; Specifications&quot; and &quot;Options&quot; pages for details. Kia America, Inc. reserves the right to make changes at any time as to vehicle availability, destination, and handling fees, colors, materials, specifications, features, accessories, packages, and models. Not responsible for typographical or computer errors. Build Find Nearby Hybrid/Electric Available Carnival MPV $37,490 starting MSRP * *\u201cStarting MSRP&quot; price is manufacturer\u2019s suggested retail price (MSRP) for lowest model trim. See &quot;Build and Price&quot; section for MSRP of model shown. MSRP excludes destination and handling charges, taxes, title, license fees, options, and dealer charges. Vehicles shown with optional equipment. Actual price set by dealer and may vary. The applicable destination and handling charge per model are provided in the \u201cBuild and Price\u201d section. See dealer for details. Some features only available as part of packages. See &quot;Features &amp; Specifications&quot; and &quot;Options&quot; pages for details. Kia America, Inc. reserves the right to make changes at any time as to vehicle availability, destination, and handling fees, colors, materials, specifications, features, accessories, packages, and models. Not responsible for typographical or computer errors. Build Find Nearby Hybrid Available Telluride $39,190 starting MSRP * *\u201cStarting MSRP&quot; pri", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "nissanusa.com", "fetchedAt": "2026-07-30T17:29:37.718187Z", "result": {"base": "https://nissanusa.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (11688 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "Microdata found.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"New Cars, SUVs, Trucks &amp; Electric Vehicles | Nissan USA\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.nissanusa.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "27138 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.nissanusa.com/contact-dealer.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "New Cars, SUVs, Trucks &amp; Electric Vehicles | Nissan USA", "textOnly": "New Cars, SUVs, Trucks &amp; Electric Vehicles | Nissan USA Vehicles Shop Electrified Owners Dealers Build & Price Build & Price Vehicles Featured Vehicles Crossovers &amp; SUVs Trucks Cars Electrified Sports Cars Future Vehicles All Vehicles Going Electric Certified Pre-Owned Business & Fleet Rogue\u00ae \"> Pathfinder\u00ae \"> Frontier\u00ae \"> Armada\u00ae \"> Kicks\u00ae \"> Rogue\u00ae \"> Nissan LEAF\u00ae \"> Pathfinder\u00ae \"> Murano\u00ae \"> Rogue\u00ae Plug-in Hybrid Armada\u00ae \"> Frontier\u00ae \"> Versa\u00ae \"> Sentra\u00ae \"> Altima\u00ae \"> Nissan LEAF\u00ae \"> Rogue\u00ae Plug-in Hybrid Nissan Z\u00ae \"> 2027 Nissan Rogue\u00ae Hybrid Coming late 2026 [[291]] Learn More Future Vehicles Learn More Featured Vehicles Crossovers &amp; SUVs Trucks Cars Electrified Sports Cars Future Vehicles Featured Vehicles Rogue\u00ae \"> Pathfinder\u00ae \"> Frontier\u00ae \"> Armada\u00ae \"> Crossovers &amp; SUVs Kicks\u00ae \"> Rogue\u00ae \"> Nissan LEAF\u00ae \"> Pathfinder\u00ae \"> Murano\u00ae \"> Rogue\u00ae Plug-in Hybrid Armada\u00ae \"> Trucks Frontier\u00ae \"> Cars Versa\u00ae \"> Sentra\u00ae \"> Altima\u00ae \"> Electrified Nissan LEAF\u00ae \"> Rogue\u00ae Plug-in Hybrid Sports Cars Nissan Z\u00ae \"> Future Vehicles 2027 Nissan Rogue\u00ae Hybrid Coming late 2026 [[291]] Learn More Future Vehicles Learn More All Vehicles Going Electric Certified Pre-Owned Business & Fleet Shop Build Inventory Offers Shopping Tools Shop New Inventory Shop Pre-Owned Build & Price Parts & Accessories Offers & Financing with NMAC View Offers Financing Options Estimate Payments Estimate Trade-In Get Credit Score Shop New Inventory Shop Pre-Owned Parts & Accessories Financing Options Estimate Payments Estimate Trade-In Get Credit Score Building Our Best. For Your Best. Driven by an unwavering commitment to quality, reliability, durability, and enhanced safety. For every mile, every journey. About Nissan Rogue Rock Creek: Explore true toughness Frontier PRO-4X: Explore true strength Electrified Electric Vehicles Discover the striking new crossover design, advanced connected features, and access to a fast-growing charging network you can trust in the All-New 2026 Nissan LEAF\u00ae. Explore LEAF Going Electric With the&nbsp;MyNISSAN app, easy home and public charging, and advanced EV connectivity, there's never been a better time to go electric. Going Electric Explore LEAF Going Electric Owners MyNISSAN Owner Portal Enhance your ownership experience with&nbsp;MyNISSAN - your online home for information on your Nissan model. Login/Register Experience MyNISSAN Owners Resources NissanConnect Services MyNISSAN Owner Portal Enhance your ownership experience with&nbsp;MyNISSAN - your online home for information on your Nissan model. View Dashboard Owners Resources NissanConnect Services Logout Recalls & Service Info Schedule Service Dealers Search Search Search Panel Expanded Search {\"2022-titan\":{\"Destination\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"1795\",\"bestPriceVersionKey\":\"37112\",\"bestPriceGradeKey\":\"29491-KING_CAB_S-174\"},\"Retail\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"38810\",\"bestPriceVersionKey\":\"37112\",\"bestPriceGradeKey\":\"29491-KING_CAB_S-174\"},\"Updated_On\":\"2026.06.03.10.36.02\",\"modelCode\":\"29491\"},\"2024-pathfinder\":{\"Destination\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"1390\",\"bestPriceVersionKey\":\"25114\",\"bestPriceGradeKey\":\"29863-S\"},\"Retail\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"36650\",\"bestPriceVersionKey\":\"25114\",\"bestPriceGradeKey\":\"29863-S\"},\"Updated_On\":\"2026.06.03.10.35.53\",\"modelCode\":\"29863\"},\"2021-titan-xd\":{\"Destination\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"1695\",\"bestPriceVersionKey\":\"53011\",\"bestPriceGradeKey\":\"29123-CREW_CAB_S\"},\"Retail\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"45430\",\"bestPriceVersionKey\":\"53011\",\"bestPriceGradeKey\":\"29123-CREW_CAB_S\"},\"Updated_On\":\"2026.06.03.10.36.05\",\"modelCode\":\"29123\"},\"2021-maxima-reveal\":{\"Destination\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"925\",\"bestPriceVersionKey\":\"XX16111\",\"bestPriceGradeKey\":\"29200-SV\"},\"Retail\":{\"priceDisclaimer\":\"\",\"modelPrice\":\"36450\",\"bestPriceVersionKey\":\"XX16111\",\"bestPriceGradeKey\":\"29200-SV\"},\"Updated_On\":\"2026.06.03.10.36.19\",\"modelCode\":\"29200\"},\"2024-titan-xd\":{\"Destinatio", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.nissanusa.com/contact-dealer.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "vw.com", "fetchedAt": "2026-07-30T17:29:43.639103Z", "result": {"base": "https://vw.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (147535 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebSite, Corporation, Organization, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"VW.com | Official Home of Volkswagen Cars &amp; SUVs\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.vw.com/en.html).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6657 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://vw.com/en/contact.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 85, "title": "VW.com | Official Home of Volkswagen Cars &amp; SUVs", "textOnly": "VW.com | Official Home of Volkswagen Cars &amp; SUVs Skip to main content Skip to footer Volkswagen Menu Offers Find a Dealer Search Inventory Models All models SUV Line-up Sedan Line-up Compact Line-up EV Line-up Shop Current Offers Search Inventory Financing &amp; Leasing Vehicle Protection Plans Purchase Programs Certified Pre-Owned Program DriverGear - Apparel &amp; Gear Vehicle Accessories Fleet Introduction to EVs Owners About My Vehicle Owner&#x27;s Manuals Recalls Warning &amp; Indicator Lights Vehicle Software Updates How-To Videos &amp; Guides Warranty &amp; Maintenance Information Service &amp; Maintenance Maintenance Coverage Maintenance Schedule Roadside Assistance Certified Collision Repair Genuine Volkswagen Service Express Service Post-Service Towing Coverage EV Service Service and Parts Financing Parts and Accessories Parts Tires &amp; Wheels Service &amp; Parts Financing My Financial Account Accounts &amp; Payments Financial FAQs Service &amp; Parts Financing Trade In and Upgrade Options Apps &amp; Connected Services myVW App Vehicle Software Updates Connected Services &amp; Plans SiriusXM Ownership Benefits EV Ownership &amp; Charging Benefits Driver Accessibility Program Certified Pre-Owned Benefits About VW Mission and Values Our History Corporate Information Brand &amp; Community DriverGear - Apparel &amp; Gear Our U.S. Soccer Federation Partnership Newsroom Shaped by the People Find A Volkswagen Dealer Help &amp; Support Here comes the 2026 Jetta Get ready. Get set. Drive. Meet the Jetta Search Inventory Take a good look at the 2026 Atlas Because you\u2019re all in it for the ride, wherever you go. Meet the Atlas Search Inventory Welcome to you, in the 2026 Tiguan This is what the good life looks like. Meet the Tiguan Search Inventory Here comes the 2026 Jetta Get ready. Get set. Drive. Meet the Jetta Search Inventory Take a good look at the 2026 Atlas Because you\u2019re all in it for the ride, wherever you go. Meet the Atlas Search Inventory Offers 2026 Atlas Models 0.9% APR for 48 months* or $3,500 Customer Bonus** Offer ends August 31, 2026 View offer details 2026 Tiguan Models 1.9% APR for 60 months* or $2,500 Customer Bonus on select trims** Offer ends August 31, 2026 View offer details The all-new 2027 Atlas SUV \u2014 something big is coming Bold? You bet. The future looks a lot like the all-new turbocharged 2027 Atlas SUV. Take a look and see what\u2019s in store, only from Volkswagen . See the 2027 Atlas SUV Here\u2019s what goes into each and every Volkswagen From innovative engineering to thoughtful functionality , we make your drive easy\u2014and a little more fun than it should be. Drive with confidence Our innovative features and systems are engineered from the wheels up for your peace of mind. Learn more about safety-enhancing features Shop smart See how select Volkswagen models stack up against the competition\u2019s cost of maintenance . Compare cost of maintenance Stay connected Of course we\u2019ve got cutting\u2011edge technology , from generative AI\u2011enhanced voice control to insights about your Volkswagen . Explore connected features Drive with confidence Our innovative features and systems are engineered from the wheels up for your peace of mind. Learn more about safety-enhancing features Shop smart See how select Volkswagen models stack up against the competition\u2019s cost of maintenance . Compare cost of maintenance Stay connected Of course we\u2019ve got cutting\u2011edge technology , from generative AI\u2011enhanced voice control to insights about your Volkswagen . Explore connected features We\u2019re all writing the Volkswagen story together For decades, we\u2019ve been putting out a call for VW drivers who dare, and you\u2019ve been answering. Artists, dreamers, doers, all kinds of curious minds. See how stories from people just like you have shaped our story in America\u2014and all over the world. Watch the stories We\u2019re all writing the Volkswagen story together For decades, we\u2019ve been putting out a call for VW drivers who dare, and you\u2019ve been answering.", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://vw.com/en/contact.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "subaru.com", "fetchedAt": "2026-07-30T17:29:50.231002Z", "result": {"base": "https://subaru.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Subaru Cars, SUVs, Crossovers &amp; Hybrids | Subaru of Amer\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.subaru.com/index.html).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "190734 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 65, "title": "Subaru Cars, SUVs, Crossovers &amp; Hybrids | Subaru of America", "textOnly": "Subaru Cars, SUVs, Crossovers &amp; Hybrids | Subaru of America --> --> --> --> Popular Searches Connected Services EyeSight Driver Assist Technology Parts and Accessories Helpful Links Customer Service Vehicle Recalls All Vehicles Special Offers Local Inventory Our Company &lt;subaru-disclaimer disclaimer-text=&quot;&lt;p>MSRP does not include destination and delivery charges, tax, title, and registration fees. Destination and delivery includes handling and inland freight fees and may vary in some states. Prices, specifications, options, features, and models subject to change without notice. Select colors may be subject to an additional charge. See your retailer for more information.&lt;br> &lt;/p> &quot; disclaimer-id=&quot;2047&quot; disclaimer-label=&quot;2047&quot;>&lt;/subaru-disclaimer>&lt;/sup>&#34;,&#34;hideMsrpLine&#34;:false,&#34;alternativePreText&#34;:null,&#34;alternativeText&#34;:null,&#34;modelTargetUrl&#34;:&#34;/vehicles/outback/2026.html&#34;,&#34;comingSoonTag&#34;:null,&#34;mpg&#34;:&#34;31/25&#34;,&#34;mpgPreText&#34;:&#34;MPG&#34;,&#34;mpgPostText&#34;:&#34;(hwy/city)&lt;sup>&lt;subaru-disclaimer disclaimer-text=&quot;&lt;p>EPA fuel economy estimates. Actual mileage will vary with driving conditions and your vehicle&rsquo;s condition.&lt;/p>&quot; disclaimer-id=&quot;2005&quot; disclaimer-label=&quot;2005&quot;>&lt;/subaru-disclaimer>&lt;/sup>&#34;,&#34;dataCategory&#34;:null,&#34;bgSrcAssetXL&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/604_background-14&#34;,&#34;bgAltTextXL&#34;:&#34;Outback-background image&#34;,&#34;bgSrcAssetLG&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/604_background-14&#34;,&#34;bgAltTextLG&#34;:&#34;Outback-background image&#34;,&#34;bgSrcAssetMD&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/604_background-14&#34;,&#34;bgAltTextMD&#34;:&#34;Outback-background image&#34;,&#34;bgSrcAssetSM&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/604_background-14&#34;,&#34;bgAltTextSM&#34;:&#34;Outback-background image&#34;,&#34;srcAssetXL&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/OBK_default_nav-6&#34;,&#34;altTextXL&#34;:&#34;Outback&#34;,&#34;srcAssetLG&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/OBK_default_nav-6&#34;,&#34;altTextLG&#34;:&#34;Outback&#34;,&#34;srcAssetMD&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/OBK_default_nav-6&#34;,&#34;altTextMD&#34;:&#34;Outback&#34;,&#34;srcAssetSM&#34;:&#34;https://s7d1.scene7.com/is/image/scomv2/OBK_default_nav-6&#34;,&#34;altTextSM&#34;:&#34;Outback&#34;,&#34;vehicleTags&#34;:[&#34;Gas&#34;,&#34;Wilderness&#34;],&#34;vehicleTagList&#34;:[{&#34;name&#34;:&#34;Gas&#34;,&#34;icon&#34;:&#34;/content/dam/subaru/en/common-images/marketing-type/Gas_Icon.svg&#34;},{&#34;name&#34;:&#34;Wilderness&#34;,&#34;icon&#34;:&#34;/content/dam/subaru/en/common-images/marketing-type/Wilderness_Icon.svg&#34;}]},{&#34;modelVariationType&#34;:&#34;driven&#34;,&#34;modelYear&#34;:&#34;2026&#34;,&#34;modelSelector&#34;:&#34;CTK&#34;,&#34;modelTitle&#34;:&#34;Crosstrek&#34;,&#34;secondaryTitle&#34;:&#34;Compact Crossover SUV&#34;,&#34;msrp&#34;:&#34;$26,995&#34;,&#34;msrpPreText&#34;:&#34;Outstanding value at&#34;,&#34;msrpPostText&#34;:&#34;&lt;sup>&lt;subaru-disclaimer disclaimer-text=&quot;&lt;p>MSRP does not include destination and delivery charges, tax, title, and registration fees. Destination and delivery includes handling and inland freight fees and may vary in some states. Prices, specifications, options, features, and models subject to change without notice. Select colors may be subject to an additional charge. See your retailer for more information.&lt;br> &lt;/p> &quot; disclaimer-id=&quot;2047&quot; disclaimer-label=&quot;2047&quot;>&lt;/subaru-disclaimer>&lt;/sup>&#34;,&#34;hideMsrpLine&#34;:false,&#34;alternativePreText&#34;:null,&#34;alternativeText&#34;:null,&#34;modelTargetUrl&#34;:&#34;/vehicles/crosstrek/2026.html&#34;,&#34;comingSoonTag&#34;:null,&#34;mpg&#34;:&#34;33/26&#34;,&#34;mpgPreText&#34;:&#34;MPG&#34", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "toyota.com", "fetchedAt": "2026-07-30T17:29:55.906360Z", "result": {"base": "https://toyota.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": false, "detail": "Blocks retrieval/citation crawlers: PerplexityBot, Amazonbot -- invisible to those assistants even when someone asks about you.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"New Cars, Trucks, SUVs &amp; Hybrids | Toyota Official Site\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.toyota.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "15587 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://toyota.com/support/contact-us/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 57, "title": "New Cars, Trucks, SUVs &amp; Hybrids | Toyota Official Site", "textOnly": "New Cars, Trucks, SUVs &amp; Hybrids | Toyota Official Site --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> Skip to main content Vehicles Shop Support &amp; Service Account Account 2026 GRMN Corolla Born on the N\u00fcrburgring, not in the boardroom. Learn More GRMN shown in Gravel using visual effects.\" data-cutline-theme=\"dark\" aria-label=\"info\">Info 2027 Tundra Relentless on the clock. Formidable off the grid. Learn More Tundra SR5 shown in Magnetic Gray Metallic with available Trailhunter Package. Prototype shown with options using visual effects.\" data-cutline-theme=\"dark\" aria-label=\"info\">Info 2027 Sequoia Conquer every path you take. Learn More Sequoia SR5 shown in Lunar Rock with available Trailhunter Package. Prototype shown with options using visual effects. \" data-cutline-theme=\"dark\" aria-label=\"info\">Info 2027 Prius Plug-in Hybrid Plug in. Maximize going out. Learn More Info 2027 Prius Powerfully dynamic. Learn More Info pick up where you left off latest build updated 1/6/23 [model name] [grade name] MSRP* $xx,xxx Est. MPG* / MPGe* xx/xx search inventory Local Specials continue build see more pick up where you left off latest build updated 1/6/23 [model name] [grade name] MSRP* $xx,xxx Est. MPG* / MPGe* xx/xx search inventory Local Specials continue build see more Explore All Vehicles Cars &amp; Minivan Trucks Crossovers &amp; SUVs Electrified Upcoming Vehicles Cars &amp; Minivan Trucks Crossovers &amp; SUVs Electrified Upcoming Vehicles Hybrid EV Available As shown: $28,840 31/38 MPG Est. City/Hwy Rating 2026 Corolla Discover more fun in every curve. $23,125 Starting MSRP 32/41 Up to Est. City/Hwy MPG Rating Explore Build As shown: $26,657 32/41 MPG Est. City/Hwy Rating 2027 Corolla Hatchback Bring out the best of every day. $24,780 Starting MSRP 32/41 Up to Est. City/Hwy MPG Rating Explore Build Hybrid EV As shown: $37,170. 51/45 MPG Est. City/Hwy Rating 2027 Prius Go farther than you thought possible. $28,755 Starting MSRP 59/50 Up to Est. City/Hwy MPG Rating Explore Build Plug-in Hybrid EV As shown: $38,000. 48/114 MPG/MPGe Est. Combined Rating 2027 Prius Plug-in Hybrid Thrilling acceleration, flexible fueling and advanced plug-in capabilities. $33,980 Starting MSRP 52/127 Up to Est. Combined MPG/MPGe Rating Explore Build Hybrid EV As shown: $35,100. 46/46 MPG Est. City/Hwy Rating 2026 Camry Chase your daily thrills in style. $29,600 Starting MSRP 52/49 Up to Est. City/Hwy MPG Rating Explore Build As shown: $41,485. 20/26 Up to MPG Est. City/Hwy Rating 2026 GR86 More miles mean more fun. Wherever you\u2019re driving. $31,400 Starting MSRP 21/30 Up to Est. City/Hwy MPG Rating Explore Build As shown: $49,040. 19/27 MPG Est. City/Hwy Rating 2026 GR Corolla Built to push the limits. $40,520 Starting MSRP 21/28 Up to Est. City/Hwy MPG Rating Explore Build As shown: $69,350. 19/26 MPG Est. City/Hwy Rating 2026 GR Supra Outpace your limits. $58,300 Starting MSRP 22/29 Up to Est. City/Hwy MPG Rating Explore Build Hybrid EV As shown: $51,810. 35/36 MPG Est. City/Hwy Rating 2026 Sienna Make room for whoever and whatever the day brings. $40,820 Starting MSRP 36/36 Up to Est. City/Hwy MPG Rating Explore Build Hybrid EV As shown: $55,965. 29/32 MPG Est. City/Hwy Rating 2026 Toyota Crown Crafted to captivate. $41,440 Starting MSRP 42/41 Up to Est. City/Hwy MPG Rating Explore Build Fuel Cell EV As shown: $52,270. 76/71 MPGe Est. City/Hwy Rating 2026 Mirai Clean mobility that elevates style and performance. $51,795 Starting MSRP 76/71 Est. City/Hwy MPGe Rating Explore Build X of X Explore All Vehicles Search Inventory Hybrid EV Available As shown: $48,140. 18/23 MPG Est. City/Hwy Rating 2026 Tacoma Built to take on whatever\u2019s in front of you. $32,445 Starting MSRP 21/26 Up to Est. City/Hwy MPG Rating Explore Build Hybrid EV Available As shown: $62,165. 17/23 MPG Est. City/Hwy Rating 2026 Tundra Powerfully equipped for the day and the day off. $41,260 Starting MSRP 18/23 Up to Est. City/Hwy MPG Rating Ex", "bots": [{"name": "GPTBot", "category": "training", "blocked": true, "evidence": "User-agent: GPTBot\nDisallow: /"}, {"name": "Google-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Google-Extended\nDisallow: /"}, {"name": "ClaudeBot", "category": "training", "blocked": true, "evidence": "User-agent: ClaudeBot\nDisallow: /"}, {"name": "anthropic-ai", "category": "training", "blocked": true, "evidence": "User-agent: anthropic-ai\nDisallow: /"}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": true, "evidence": "User-agent: Meta-ExternalAgent\nDisallow: /"}, {"name": "Applebot-Extended", "category": "training", "blocked": true, "evidence": "User-agent: Applebot-Extended\nDisallow: /"}, {"name": "Bytespider", "category": "training", "blocked": true, "evidence": "User-agent: Bytespider\nDisallow: /"}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: CCBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"OAI-SearchBot\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"ChatGPT-User\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": true, "evidence": "User-agent: PerplexityBot\nDisallow: /"}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"Claude-User\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": true, "evidence": "User-agent: Amazonbot\nDisallow: /"}], "contactMatch": "https://toyota.com/support/contact-us/", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": false, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "mazdausa.com", "fetchedAt": "2026-07-30T17:30:01.876227Z", "result": {"base": "https://mazdausa.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Blocks training crawlers only (CCBot) -- a legitimate opt-out; retrieval/citation access stays open.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Mazda USA Official Site | Cars, SUVs &amp; Crossovers  | Maz\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.mazdausa.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "37651 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://mazdausa.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 54, "title": "Mazda USA Official Site | Cars, SUVs &amp; Crossovers  | Mazda USA", "textOnly": "Mazda USA Official Site | Cars, SUVs &amp; Crossovers | Mazda USA --> Vehicles All Crossovers &amp; SUVs Electrified Sedans &amp; Hatchbacks Sports Cars All Crossovers &amp; SUVs Electrified Sedans &amp; Hatchbacks Sports Cars Mazda ranked Safest New-Car Brand by Consumer Reports Consumer Reports does not endorse products or services. LEARN MORE Build And Price Search Inventory Request A Quote Find A Dealer Special Offers Schedule A Test Drive Certified Pre-Owned ELECTRIFIED OPTIONS TO FIT YOUR LIFE LEARN MORE 2026 Mazda CX-70 Inline 6 Turbo Disclaimers Starting At $42,750 3 Combined 25 MPG 4 Seating 5 Passengers EXPLORE BUILD 2026 Mazda CX-90 Inline 6 Turbo Disclaimers Starting At $39,300 5 Combined 25 MPG 6 Seating up to 8 Passengers 27 EXPLORE BUILD 2026 MAZDA CX-30 Disclaimers Starting At $26,375 9 Combined 27 MPG 28 Seating 5 Passengers EXPLORE BUILD 2026 MAZDA CX-50 Disclaimers Starting At $29,900 9 Combined 26 MPG 10 Seating 5 Passengers EXPLORE BUILD 2026 MAZDA CX-5 Disclaimers Starting At $29,990 1 Combined 26 MPG 2 Seating 5 Passengers EXPLORE BUILD 2026 Mazda CX-70 PHEV Plug-in Hybrid Disclaimers Starting At $44,450 15 Combined 61 MPGe 14 Seating 5 Passengers EXPLORE BUILD 2026 Mazda CX-90 Plug-in Hybrid Disclaimers Starting At $50,695 13 Combined 56 MPGe 14 Seating 7 Passengers 29 EXPLORE BUILD 2026 CX-50 Hybrid Disclaimers Starting At $34,750 11 Combined 38 MPG 12 Seating 5 Passengers EXPLORE BUILD 2026 Mazda CX-50 Hybrid Disclaimers Starting At $34,750 11 Combined 38 MPG 12 Seating 5 Passengers EXPLORE BUILD 2026 Mazda CX-90 Plug-in Hybrid Disclaimers Starting At $50,695 13 Combined 56 MPGe 14 Seating 7 Passengers 29 EXPLORE BUILD 2026 Mazda CX-70 PHEV Plug-in Hybrid Disclaimers Starting At $44,450 15 Combined 61 MPGe 14 Seating 5 Passengers EXPLORE BUILD 2026 MAZDA3 HATCHBACK Disclaimers Starting At $25,650 30 Combined 29 MPG 31 Seating 5 Passengers Explore Build 2026 MAZDA3 SEDAN Disclaimers Starting At $24,650 18 Combined 30 MPG 19 Seating 5 Passengers EXPLORE BUILD 2026 MAZDA MX-5 MIATA Disclaimers Starting At $30,430 20 Combined 29 MPG 21 Seating 2 Passengers Explore Build 2026 MAZDA MX-5 MIATA RF Disclaimers Starting At $38,450 23 Combined 29 MPG 22 Seating 2 Passengers Explore Build Shopping Tools EXPLORE Build and Price Keep Me Updated Shop Inventory Search Request a Quote Reserve a Test Drive Certified Pre-Owned Buy Special Offers Payment Estimator Apply For Financing 2026 MAZDA CX-90 BUILD YOURS FIND YOUR NEXT MAZDA VIEW INVENTORY 0% APR FOR 36 MONTHS PLUS NO PAYMENTS FOR 90 DAYS ON SELECT 2026 MODELS 32 VIEW OFFERS What Drives Us THE MAZDA DIFFERENCE See how our unique perspective creates our exceptional vehicles. DISCOVER MAZDA CENTRAL Get news and insights on the topics that matter to you. EXPLORE Owners Maintaining Your Mazda Service Warranty Recalls How to Use Making it Yours Accessories Parts FAQs Find a service dealer near you and make your next appointment. SCHEDULE YOURS Find the latest MAZDA accessories for your next adventure. EXPLORE Go to MyMazda to enjoy the full benefits of being a Mazda Owner&nbsp;and visit the Mazda Owner\u2019s Club for badges, exclusive content, and more. LOGIN REGISTER Mazda ranked \u201cSafest New-Car Brand\u201d by Consumer Reports Consumer Reports does not endorse products or services. LEARN MORE Build And Price Search Inventory Request A Quote Find A Dealer Special Offers Schedule A Test Drive Certified Pre-Owned Find a Dealer Clear --> All Crossovers &amp; SUVs Electrified Sedans &amp; Hatchbacks Sports Cars All Crossovers &amp; SUVs Electrified Sedans &amp; Hatchbacks Sports Cars Mazda ranked Safest New-Car Brand by Consumer Reports Consumer Reports does not endorse products or services. LEARN MORE Build And Price Search Inventory Request A Quote Find A Dealer Special Offers Schedule A Test Drive Certified Pre-Owned ELECTRIFIED OPTIONS TO FIT YOUR LIFE LEARN MORE 2026 Mazda CX-70 Inline 6 Turbo Disclaimers Starting At $42,750 3 Combined 25 MPG 4 Seating 5 Passengers EXPLORE B", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": true, "evidence": "User-agent: aiHitBot, Barkrowler, BDCbot, BLEXBot, BLP_bbot, brokenlinkcheck.com, Buck, CCBot, Cliqzbot, cyencebot, DomainCrawler, Dow Jones Searchbot, Exabot, ExtLinksBot, FemtosearchBot, Fever, GarlikCrawler, Gigabot, gobuster, GrapeshotCrawler, heritrix, istellabot, Jersey, Jobkicks, libwww-perl, linkdexbot, LinkpadBot, ltx71 - (http://ltx71.com/), lua-resty-http, LumtelBot, magpie-crawler, Magus bot, Mail.RU_Bot, Megaindex.ru, Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com, Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com, NL-Crawler, OnPageBot, Riddler, ScoutJet, Scrapy, Seekport, SeznamBot, Siteimprove, SMTBot, UptimeRobot, VelenPublicWebCrawler, Wget, yacybot, Yeti, YisouSpider, YunSecurityBot, ZoominfoBot\nDisallow: /"}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://mazdausa.com/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "honda.com", "fetchedAt": "2026-07-30T17:30:07.185628Z", "result": {"base": "https://honda.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://honda.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "tesla.com", "fetchedAt": "2026-07-30T17:30:12.702629Z", "result": {"base": "https://tesla.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://tesla.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "bmwusa.com", "fetchedAt": "2026-07-30T17:30:17.906398Z", "result": {"base": "https://bmwusa.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://bmwusa.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "bmw.com", "fetchedAt": "2026-07-30T17:30:24.471040Z", "result": {"base": "https://bmw.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://bmw.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "mbusa.com", "fetchedAt": "2026-07-30T17:30:31.352395Z", "result": {"base": "https://mbusa.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation, Person.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Luxury Cars - Sedans, SUVs, Coupes &amp; Wagons | Mercedes-B\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.mbusa.com/en/home).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10199 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://mbusa.com/en/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Luxury Cars - Sedans, SUVs, Coupes &amp; Wagons | Mercedes-Benz USA", "textOnly": "Luxury Cars - Sedans, SUVs, Coupes &amp; Wagons | Mercedes-Benz USA --> Skip Navigation Back Search QUICK LINKS Mercedes Electric Build Your Own Special Offers New Inventory Mercedes Owners Certified Pre-Owned Vehicles Electric Shopping Owners My Account Find a Dealer Search Find a Dealer Back Vehicles All Vehicles Compare Vehicles Special Offers Certified Pre-Owned Program Future Vehicles Features &amp; Education Plug-In Hybrid Vehicles (PHEV) Driver Assistance Features Mercedes-Benz Studio L.A. Brands Mercedes-AMG Mercedes Electric Mercedes-Maybach Mercedes-Benz Vans Back Electric DISCOVER ELECTRIC Mercedes-Benz Electric Explore the Lineup Charging Your Vehicle Electric FAQs EV Charging Tools Shop Electric New Inventory Certified Pre-Owned All-Electric Special Offers Build Your Own Charge up in Luxury Mercedes-Benz High-Power Charging Network, a premium experience for effortless public charging and preferential access for Mercedes-Benz drivers across North America. Learn More About Charging Back Shopping INVENTORY New Vehicles Certified Pre-Owned Vehicles Pre-Owned Vehicles Special Offers Start Purchase Online TOOLS Build Your Own Shopping Assist Compare Vehicles Request a Quote Finance Financial Services Apply for Financing Trade-In Value Shopping Assist by Mercedes-Benz Answer a few quick questions and your selected dealer will find the perfect match for you. Find Your Dream Mercedes-Benz Back Owners Service &amp; Manuals Owners Support Maintenance Support Owners Manuals Service &amp; Warranty Manuals Schedule Service Updating your Vehicle Accessories Recall Information Vehicle Information Bluetec Update OWNERSHIP ON-THE-Go Mercedes-Benz App Mercedes-Benz Store Mercedes-Benz Finance App FLEET SOLUTIONS &amp; SERVICES Manage Your Fleet Corporate Program Benefits Partner Programs &amp; Employee Incentives Back My Account Log In or Register Even if you don\u2019t own a Mercedes-Benz, you can register for an account to save builds, track inventory, set alerts and more. Login Register Explore Account Details View Account Saved Vehicles Deals In Progress Log Out Mercedes-Benz App The Mercedes-Benz app is the ultimate resource, putting control of your vehicle in the palm of your hand. Learn More Back Search QUICK LINKS Mercedes Electric Build Your Own Special Offers New Inventory Mercedes Owners Certified Pre-Owned Show Navigation A sleek silhouette, an illuminating presence. Don\u2019t miss the world premiere of the all-new 2028 Mercedes-Benz GLA SUV. Watch the Livestream Close Banner The Mercedes-Benz Welcome Home Celebration. Transforming every drive into a homecoming. Explore special offers on the Mercedes-Benz GLC SUV, GLE SUV and S-Class Sedan. View Offers Explore the Lineup Countdown to Dreams DAYS : HOURS : MINUTES : SECONDS Outgoing. Intuitive. Electric. Impressive range. Awe-inspiring style. Advanced intelligence. Get it all in the Mercedes-Benz GLC Electric SUV. Discover the GLC Electric SUV Build Yours Countdown to Dreams DAYS : HOURS : MINUTES : SECONDS Exquisite, of course. Refining travels with a commanding presence, the new 2027 Mercedes-Maybach GLS 680 captivates with an even more distinct design, astounding comfort and intuitive technology. Learn More I\u2019m Interested Countdown to Dreams DAYS : HOURS : MINUTES : SECONDS Shopping Tools ALL VEHICLES BUILD YOUR OWN SPECIAL OFFERS VIEW INVENTORY ELECTRIC. HYBRID. HERITAGE. Meet our powertrains. Whether you prioritize energy-efficient driving, hybrid flexibility or traditional power, there\u2019s a Mercedes-Benz for your needs and lifestyle. Stop slideshow ELECTRIC Pure electric performance for the modern driver. All-electric vehicles offer responsive acceleration and effortless driving for every journey. Designed for drivers seeking zero-emission mobility without compromising on capability or craftsmanship. Learn More PLUG-IN HYBRID The versatility of two powertrains, intelligently wrapped into one machine. Hybrid vehicles pair electric and gas power, offering the flexibility daily ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://mbusa.com/en/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "auto", "domain": "mercedes-benz.com", "fetchedAt": "2026-07-30T17:30:36.542475Z", "result": {"base": "https://mercedes-benz.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://mercedes-benz.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "audiusa.com", "fetchedAt": "2026-07-30T17:30:42.281459Z", "result": {"base": "https://audiusa.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://audiusa.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "auto", "domain": "audi.com", "fetchedAt": "2026-07-30T17:30:52.963922Z", "result": {"base": "https://audi.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (1823 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, Organization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"audi.com\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.audi.com/en).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6238 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://audi.com/en/press-contacts).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "audi.com", "textOnly": "audi.com Menu Models Company Technology Sustainability Motorsport Careers Audi Media Center Sign in Legal DE EN Close menu Company overview Audi at a glance Audi Group Production Sites Strategy Investor Relations Sports and Culture Audi History Works council Close menu Company Technology overview Electric Mobility Plug-in-hybrid Internal combustion engines (ICE) quattro Vehicle safety Design Digital Experience Close menu Technology Sustainability overview Sustainability Concept Sustainability reports E \u2013 Environment &amp; Resources S \u2013 People &amp; Society G \u2013 Governance Close menu Sustainability Motorsport overview Formula 1\u00ae Motorsport history Audi Sport racing legends Close menu Motorsport Careers overview Working at Audi School students Students Graduates Professionals Your Application Close menu Careers Audi Media Center overview Dates Media releases Images Videos Newsletter-Registration Media contacts Close menu Audi Media Center DE EN Discover Audi Company. People. Innovations. Audi at a glance Models Formula 1\u00ae Investor Relations Careers Media Learn more A seamless experience: in-car entertainment in the Audi Q6 e-tron 07/23/2026 Audi Nuvolari: The vehicle shown is a near-production prototype. Fuel consumption combined (weighted) in l/100 km: 11.3 (preliminary); Power consumption combined (weighted) in kWh/100km: 7.8 (preliminary); CO\u2082 emissions combined (weighted) in g/km: 270 (preliminary); CO\u2082 class combined (weighted): G (preliminary); Fuel consumption with discharged battery in l/100 km: 14.7 (preliminary); CO\u2082 emissions with discharged battery: G (preliminary) Audi Nuvolari at Goodwood Festival of Speed 07/10/2026 Audi upgrades the Q3 for the 2027 model year 07/15/2026 Audi A6 allroad TDI quattro 220 kW: Combined fuel consumption in l/100 km: 6.4\u20135.8 (36.8\u201340.6 US mpg); Combined CO 2 emissions in g/km: 167\u2013153 (268.8\u2013246.2 g/mi); CO 2 class: E-F Audi Models A2 e-tron A6 e-tron Q4 e-tron Q6 e-tron e-tron GT A3 A5 A6 A8 Q3 Q5 Q7 Q8 Q9 Nuvolari Information on the models Discover the Audi Models Find the configurator, pricing, and purchase and leasing information here. Discover Audi Models Step into the world of Audi Formula 1\u00ae and Audi Exploring the complex logistics behind Audi\u2019s Formula 1 commitment Learn more Audi A6 Sportback e-tron performance: Electric power consumption (combined): 15.9\u201314.0 kWh/100 km; CO 2 emissions (combined): 0 g/km; CO 2 class: A Driving sustainability Advancing electric mobility and embedding sustainability across the value chain Learn more Behind the scenes How Audi is preparing for the era of the software-defined vehicle Learn more Audi Nuvolari: The vehicle shown is a near-production prototype. Fuel consumption combined (weighted) in l/100 km: 11.3 (preliminary); Power consumption combined (weighted) in kWh/100km: 7.8 (preliminary); CO\u2082 emissions combined (weighted) in g/km: 270 (preliminary); CO\u2082 class combined (weighted): G (preliminary); Fuel consumption with discharged battery in l/100 km: 14.7 (preliminary); CO\u2082 emissions with discharged battery: G (preliminary) Key statements CEO Gernot D\u00f6llner on Audi\u2019s technological transformation Learn more Careers at Audi People who think ahead and stay determined drive our progress Learn more Innovation at Audi&#39;s core How advanced technology and forward thinking bring \u201cVorsprung durch Technik\u201d to life Learn more Audi Media Center Find the latest media information, images, and videos from Audi Communications. Audi A6 allroad e-hybrid quattro 270 kW: Fuel consumption (weighted, combined): 2.9-2.6 l/100 km (81.1-90.5 US mpg); Power consumption (weighted, combined): 16.2-15.7 kWh/100 km; CO 2 emissions (weighted, combined): 67-59 g/km (107.8-95.0 g/mi); CO 2 class (weighted, combined): B; Fuel consumption on discharged battery (combined): 7.7-7.2 l/100 km (30.5-32.7 US mpg); CO 2 class on discharged battery: F 07/27/2026 audi.com Audi Group maintains stable operating profit in the first half of 2026 despite challenging markets Audi Nuvolari", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://audi.com/en/press-contacts", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "financial", "domain": "northwesternmutual.com", "fetchedAt": "2026-07-30T17:30:58.347643Z", "result": {"base": "https://northwesternmutual.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: MISSING - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": false, "detail": "0 chars of server-rendered text -- looks like a JS app shell; many agents see an empty page.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 15, "title": "https://northwesternmutual.com", "textOnly": "", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "financial", "domain": "schwab.com", "fetchedAt": "2026-07-30T17:31:03.856595Z", "result": {"base": "https://schwab.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Charles Schwab\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": false, "detail": "217 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 25, "title": "Charles Schwab", "textOnly": "Charles Schwab We\u2019re sorry, but we were unable to authorize your request. Please call us at 800-433-9196 and provide reference number SWAF-18.6e551702.1785432663.191c2675. We apologize for any inconvenience. 0915-82LR", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "financial", "domain": "vanguard.com", "fetchedAt": "2026-07-30T17:31:09.189989Z", "result": {"base": "https://vanguard.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Mutual funds, IRAs, ETFs, 401(k) plans, and more | Vanguard\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://investor.vanguard.com/corporate-portal).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3282 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://corporate.vanguard.com/content/corporatesite/us/en/corp/contact-us.html).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 51, "title": "Mutual funds, IRAs, ETFs, 401(k) plans, and more | Vanguard", "textOnly": "Mutual funds, IRAs, ETFs, 401(k) plans, and more | Vanguard Our Sites Close Close Menu Personal investors Retirement plan participants Plan sponsors &amp; consultants Financial advisors Company information and insights Global sites My Plan Manager\u2122 Institutional Trading Pressroom Career opportunities Contact Us Personal investors Retirement plan participants Plan sponsors &amp; consultants Financial professionals Company information and insights Global sites 50 years of investor success doesn\u2019t happen by chance. We combine our time-tested investment principles with a relentless drive to help you keep more of what you earn. Our high-quality investments, savings strategies, personalized financial advice, retirement tools, and relevant market insights help you build a future for those you love. In a letter to investors, our CEO, Salim Ramji, reflects on another year of smart investing. Hear from our CEO Our Company Our Sites Important Information All investing is subject to risk including the possible loss of the money you invest. Vanguard is owned by its funds, which are owned by Vanguard&#x27;s fund shareholders. Our retail direct investment advisory strategies, in turn, are built on core investments in the Vanguard funds. 1 For the 10-year period ended March 31, 2026, 6 of 6 Vanguard money market funds, 77 of 108 Vanguard bond funds, 21 of 23 Vanguard balanced funds, and 172 of 196 Vanguard stock funds\u2014for a total of 276 of 333 Vanguard funds\u2014outperformed their Lipper peer-group averages. Results will vary for other time periods. Only mutual funds and ETFs (exchange-traded funds) with a minimum 10-year history were included in the comparison. Source: LSEG Lipper. Note that the competitive performance data shown represent past performance, which is not a guarantee of future results, and that all investments are subject to risks. For the most recent performance, visit our website at www.vanguard.com/performance . 2 Vanguard average ETF and mutual fund expense ratio: 0.07%. Industry average ETF and mutual fund expense ratio: 0.44%. All averages are asset-weighted. Industry average excludes Vanguard. Sources: Vanguard and Morningstar, Inc., as of December 31, 2025. The experiences of any Vanguard client may not be representative of the experience of other Vanguard clients and are not a guarantee of future investment performance or success. A client is any person or entity who purchases a Vanguard product or service, including but not limited to advice clients, brokerage customers, plan sponsors and participants, and fund shareholders via third-party financial institutions. Non-U.S. funds and non-fund investment vehicles do not directly own Vanguard. Advice services are provided by Vanguard Advisers, Inc., a registered investment advisor, or by Vanguard National Trust Company, a federally chartered, limited-purpose trust company. \u00a9 1995-2026 The Vanguard Group, Inc. All rights reserved. Vanguard Marketing Corporation, Distributor of the Vanguard Funds. Your use of this site signifies that you accept our terms &amp; conditions of use. Privacy Center Do Not Sell My Personal Information Security Center Accessibility Mutual fund prospectuses Exchange-traded fund (ETF) prospectuses Special notice to non-U.S. investors Manage cookies", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://corporate.vanguard.com/content/corporatesite/us/en/corp/contact-us.html", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "financial", "domain": "usbank.com", "fetchedAt": "2026-07-30T17:31:16.314759Z", "result": {"base": "https://usbank.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (953296 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Personal Banking, Credit Cards, Loans &amp; Investing | U.S.\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.usbank.com/index.html).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": false, "detail": "Missing -- a small, easy entity-presentation signal.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9888 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:&#34;,&#34;^tel:&#34;,&#34;access-insite.usbank.com&#34;,&#34;account3000.com/usbank&#34;,&#34;aeromexicovisa.com&#34;,&#34;bizcards.com&#34;,&#34;careers.usbank.com&#34;,&#34;carenet.fnfismd.com/usbank&#34;,&#34;clearwateranalytics.com&#34;,&#34;digitalexplorer.usbank.com&#34;,&#34;dillonsmastercard.com&#34;,&#34;engage.usbank.com&#34;,&#34;fleetcommanderonline.com&#34;,&#34;flexperks.com&#34;,&#34;freddysmastercard.com&#34;,&#34;frysmastercard.com&#34;,&#34;gymboreevisa.com&#34;,&#34;h-dvisa.com&#34;,&#34;htmastercard.com&#34;,&#34;htmastercard.com/usbank&#34;,&#34;https://usbank.findmyspecialoffer.com/fmo/preQual/firstRequest.do?business\\u003dwxunUB11iM0l3GLLCWZw9g\\u0026origSiteInd\\u003dDotcom&#34;,&#34;https://usbank.findmyspecialoffer.com/fmo/preQual/firstRequest.do?business\\u003dwxunUB11iM0l3GLLCWZw9g\\u0026origSiteInd\\u003dUSBB/SMBZ&#34;,&#34;https://www.careerarc.com/job-map/u-s-bank?campaign_id\\u003d24678&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/altitude&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/flexperks&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/welcome&#34;,&#34;https://www.usbankgomobile.com/wallet/#/home&#34;,&#34;kingsoopersmastercard.com&#34;,&#34;krogermastercard.com&#34;,&#34;latamvisa.com&#34;,&#34;marianosmastercard.com&#34;,&#34;merchantcommanderonline.com&#34;,&#34;multiserviceaviation.com&#34;,&#34;mycardgtb.com&#34;,&#34;myvisacardportal.com&#34;,&#34;paymentsinsider.com&#34;,&#34;picknsavemastercard.com&#34;,&#34;polarisvisa.com&#34;,&#34;portal.syncada.com&#34;,&#34;qfcmastercard.com&#34;,&#34;quickencard.com&#34;,&#34;radissonrewardsvisa.com&#34;,&#34;ralphsmastercard.com&#34;,&#34;reimastercard.com&#34;,&#34;rewardscommunication@usbank.com&#34;,&#34;skypassvisa.com&#34;,&#34;smithsmastercard.com&#34;,&#34;specialoffers.visa.com&#34;,&#34;taleo.net&#34;,&#34;usbancorpassetmanagement.com&#34;,&#34;usbank.autogravity.com&#34;,&#34;usbank.customercarenet.com&#34;,&#34;usbank.marvelapp.com&#34;,&#34;usbank.com&#34;,&#34;usbankacceptpaymentsnow.com&#34;,&#34;usbankcanada.com&#34;,&#34;usbankcontour.com&#34;,&#34;usbankfocus.com&#34;,&#34;usbankgomobile.com&#34;,&#34;usbankhr.com&#34;,&#34;usbank-me.everfi.com&#34;,&#34;usbankprepaid.com&#34;,&#34;usbankreliacard.com&#34;,&#34;usbexpensemanagement.com&#34;,&#34;usb-fit.com&#34;,&#34;usbpayment.com&#34;,&#34;www.elalflycardpreferred.com&#34;,&#34;www.mystreetscape.com&#34;,&#34;www.usbankacceptpaymentsnow.com&#34;,&#34;www.usbankcanada.com&#34;,&#34;www.usbankmyaccount.com&#34;,&#34;www.usbankpaymentsolutions.com&#34;,&#34;www.usbankrewardsconnect.com&#34;,&#34;www.usbankprepaid.com&#34;,&#34;www.usbankreliacard.com&#34;,&#34;www.usbankrewardsconnect.com&#34;,&#34;meetingtomorrow.com/webcast/2021outlook&#34;,&#34;mycardgtb.com/bizconnectvs&#34;,&#34;mycardgtb.com/bizwemc&#34;,&#34;onlinebanking.usbank.com/Auth/Login&#34;,&#34;onlinebanking.usbank.com/digital/loginhelp&#34;,&#34;stage.usbank.com&#34;,&#34;usbankinteractive.postclickmarketing.com/assessment-home-buying&#34;,&#34;www.usbank.com&#34;,&#34;www.usbank.com/en-eu/investment-services/contact-us/global-corporate-trust-contact-us.html&#34;,&#34;www.usbank.com/leveragecategories&#34;,&#34;www.usbank.com/locations&#34;,&#34;www.usbank.eu/investment-services/contact-us/global-corporate-trust-contact-us.html&#34;,&#34;www.usbank.eu/investment-services/luxembourg.html&#34;,&#34;onboarding.usbank.com&#34;,&#34;usbankexpensecard.com&#34;,&#34;www.meetingtomorrow.com/webcast/PostElectionOutlook&#34;,&#34;join.paymentstart.com&#34;,&#34;www.private-privacy.my.onetrust.com&#34;,&#34;www.usbank.com/locations/search&#34;,&#34;f.io&#34;,&#34;www.elavon.com&#34;,&#34;www.talech.com&#34;,&#34;usbankflyer.onelink.me&#34;,&#34;usbankflyer.onelink.me/Woj5/moneyTracker&#34;,&#34;www.access.usbank.com/cpsApp1/AxolPreAuthServlet?requestCmdId\\u003dlogin&#34;,&#34;www.figma.com&#34;,&#34;singlepoint.usbank.com&#34;,&#34;markets.usbank.com&#34;,&#34;tradepoint.usbank.com&#34;,&#34;www.usbank.com/locations/search&#34;]).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 70, "title": "Personal Banking, Credit Cards, Loans &amp; Investing | U.S. Bank", "textOnly": "Personal Banking, Credit Cards, Loans &amp; Investing | U.S. Bank Skip to main content Log in About us Financial education Support Locations Log in Locations Support Financial Education About us Personal Return to Main Menu Personal Checking &amp; savings Return to Personal Checking &amp; savings Checking accounts Checking accounts Return to Checking &amp; savings Checking accounts Bank Smartly\u00ae Checking account Safe Debit account Young adult checking (18-24) Teen checking (13-17) Debit card Explore all checking accounts Savings accounts &amp; CDs Savings accounts &amp; CDs Return to Checking &amp; savings Savings accounts &amp; CDs Bank Smartly\u00ae Savings account Money market account CD accounts Explore all savings accounts Banking services &amp; benefits Banking services &amp; benefits Return to Checking &amp; savings Banking services &amp; benefits Greenlight debit card for kids Smart Rewards Mobile &amp; online banking Overdraft protection Routing &amp; account numbers Checking + Savings Double up on rewards in half the time Earn more when you open Bank Smartly Checking and Savings accounts together. Apply for both in under 5 minutes. Learn about the bundle Explore all bank accounts Credit cards Return to Personal Credit cards Types of credit cards Types of credit cards Return to Credit cards Types of credit cards Cash back credit cards Travel credit cards Rewards credit cards Low intro rate credit cards No annual fee credit cards Flexible spending credit cards Credit-building credit cards Business credit cards Payment plans &amp; installment loans Payment plans &amp; installment loans Return to Credit cards Payment plans &amp; installment loans Pay over time with an ExtendPay Plan Tools &amp; resources Tools &amp; resources Return to Credit cards Tools &amp; resources Explore credit card insights Check your credit score Improve your credit score How to increase your credit limit Mobile &amp; online banking Explore all credit cards Investing &amp; retirement Return to Personal Investing &amp; retirement Investing Investing Return to Investing &amp; retirement Investing Start investing Work with an advisor Invest on your own Investment management Explore investing options Retirement Retirement Return to Investing &amp; retirement Retirement Open an IRA 401(k) rollovers Save for retirement Explore retirement planning Tools &amp; resources Tools &amp; resources Return to Investing &amp; retirement Tools &amp; resources Retirement planning toolkit Learn about IRAs Savings vs. investing: What\u2019s the difference? Read all insights Retirement planning Still have that 401K at your previous company? Learn how to consolidate and help your money do more with our investment options and guidance. Explore your options Find an advisor Personal loans Return to Personal Personal loans Personal loans &amp; lines Personal loans &amp; lines Return to Personal loans Personal loans &amp; lines Personal loans &amp; lines of credit Debt consolidation loans Personal loans for cars Tools &amp; resources Tools &amp; resources Return to Personal loans Tools &amp; resources Personal loan calculator Debt consolidation calculator Personal loans Money for what matters most to you From home projects to consolidating debt, a personal loan can help. Check your rate to get started. Check your rate Explore all personal loans Home loans Return to Personal Home loans Mortgage Mortgage Return to Home loans Mortgage Today\u2019s mortgage rates Compare mortgage options Affordable home loans Explore mortgage loans Prequalify for a mortgage Refinance Refinance Return to Home loans Refinance Today\u2019s refinance rates Explore refinance loans Apply for a refinance Home equity &amp; improvement Home equity &amp; improvement Return to Home loans Home equity &amp; improvement Home equity lines of credit (HELOC) Home equity loans Home improvement loans Explore home equity Prequalify for a HELOC Tools &amp; resources Tools &amp; resources Return to Home loans Tools &amp; resources Hom", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:&#34;,&#34;^tel:&#34;,&#34;access-insite.usbank.com&#34;,&#34;account3000.com/usbank&#34;,&#34;aeromexicovisa.com&#34;,&#34;bizcards.com&#34;,&#34;careers.usbank.com&#34;,&#34;carenet.fnfismd.com/usbank&#34;,&#34;clearwateranalytics.com&#34;,&#34;digitalexplorer.usbank.com&#34;,&#34;dillonsmastercard.com&#34;,&#34;engage.usbank.com&#34;,&#34;fleetcommanderonline.com&#34;,&#34;flexperks.com&#34;,&#34;freddysmastercard.com&#34;,&#34;frysmastercard.com&#34;,&#34;gymboreevisa.com&#34;,&#34;h-dvisa.com&#34;,&#34;htmastercard.com&#34;,&#34;htmastercard.com/usbank&#34;,&#34;https://usbank.findmyspecialoffer.com/fmo/preQual/firstRequest.do?business\\u003dwxunUB11iM0l3GLLCWZw9g\\u0026origSiteInd\\u003dDotcom&#34;,&#34;https://usbank.findmyspecialoffer.com/fmo/preQual/firstRequest.do?business\\u003dwxunUB11iM0l3GLLCWZw9g\\u0026origSiteInd\\u003dUSBB/SMBZ&#34;,&#34;https://www.careerarc.com/job-map/u-s-bank?campaign_id\\u003d24678&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/altitude&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/flexperks&#34;,&#34;https://www.myvisacardportal.com/usbank/en_us/welcome&#34;,&#34;https://www.usbankgomobile.com/wallet/#/home&#34;,&#34;kingsoopersmastercard.com&#34;,&#34;krogermastercard.com&#34;,&#34;latamvisa.com&#34;,&#34;marianosmastercard.com&#34;,&#34;merchantcommanderonline.com&#34;,&#34;multiserviceaviation.com&#34;,&#34;mycardgtb.com&#34;,&#34;myvisacardportal.com&#34;,&#34;paymentsinsider.com&#34;,&#34;picknsavemastercard.com&#34;,&#34;polarisvisa.com&#34;,&#34;portal.syncada.com&#34;,&#34;qfcmastercard.com&#34;,&#34;quickencard.com&#34;,&#34;radissonrewardsvisa.com&#34;,&#34;ralphsmastercard.com&#34;,&#34;reimastercard.com&#34;,&#34;rewardscommunication@usbank.com&#34;,&#34;skypassvisa.com&#34;,&#34;smithsmastercard.com&#34;,&#34;specialoffers.visa.com&#34;,&#34;taleo.net&#34;,&#34;usbancorpassetmanagement.com&#34;,&#34;usbank.autogravity.com&#34;,&#34;usbank.customercarenet.com&#34;,&#34;usbank.marvelapp.com&#34;,&#34;usbank.com&#34;,&#34;usbankacceptpaymentsnow.com&#34;,&#34;usbankcanada.com&#34;,&#34;usbankcontour.com&#34;,&#34;usbankfocus.com&#34;,&#34;usbankgomobile.com&#34;,&#34;usbankhr.com&#34;,&#34;usbank-me.everfi.com&#34;,&#34;usbankprepaid.com&#34;,&#34;usbankreliacard.com&#34;,&#34;usbexpensemanagement.com&#34;,&#34;usb-fit.com&#34;,&#34;usbpayment.com&#34;,&#34;www.elalflycardpreferred.com&#34;,&#34;www.mystreetscape.com&#34;,&#34;www.usbankacceptpaymentsnow.com&#34;,&#34;www.usbankcanada.com&#34;,&#34;www.usbankmyaccount.com&#34;,&#34;www.usbankpaymentsolutions.com&#34;,&#34;www.usbankrewardsconnect.com&#34;,&#34;www.usbankprepaid.com&#34;,&#34;www.usbankreliacard.com&#34;,&#34;www.usbankrewardsconnect.com&#34;,&#34;meetingtomorrow.com/webcast/2021outlook&#34;,&#34;mycardgtb.com/bizconnectvs&#34;,&#34;mycardgtb.com/bizwemc&#34;,&#34;onlinebanking.usbank.com/Auth/Login&#34;,&#34;onlinebanking.usbank.com/digital/loginhelp&#34;,&#34;stage.usbank.com&#34;,&#34;usbankinteractive.postclickmarketing.com/assessment-home-buying&#34;,&#34;www.usbank.com&#34;,&#34;www.usbank.com/en-eu/investment-services/contact-us/global-corporate-trust-contact-us.html&#34;,&#34;www.usbank.com/leveragecategories&#34;,&#34;www.usbank.com/locations&#34;,&#34;www.usbank.eu/investment-services/contact-us/global-corporate-trust-contact-us.html&#34;,&#34;www.usbank.eu/investment-services/luxembourg.html&#34;,&#34;onboarding.usbank.com&#34;,&#34;usbankexpensecard.com&#34;,&#34;www.meetingtomorrow.com/webcast/PostElectionOutlook&#34;,&#34;join.paymentstart.com&#34;,&#34;www.private-privacy.my.onetrust.com&#34;,&#34;www.usbank.com/locations/search&#34;,&#34;f.io&#34;,&#34;www.elavon.com&#34;,&#34;www.talech.com&#34;,&#34;usbankflyer.onelink.me&#34;,&#34;usbankflyer.onelink.me/Woj5/moneyTracker&#34;,&#34;www.access.usbank.com/cpsApp1/AxolPreAuthServlet?requestCmdId\\u003dlogin&#34;,&#34;www.figma.com&#34;,&#34;singlepoint.usbank.com&#34;,&#34;markets.usbank.com&#34;,&#34;tradepoint.usbank.com&#34;,&#34;www.usbank.com/locations/search&#34;]", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "hotel", "domain": "marriott.com", "fetchedAt": "2026-07-30T17:31:22.931389Z", "result": {"base": "https://marriott.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://marriott.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "hilton.com", "fetchedAt": "2026-07-30T17:31:37.341352Z", "result": {"base": "https://hilton.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://hilton.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "hyatt.com", "fetchedAt": "2026-07-30T17:31:42.516247Z", "result": {"base": "https://hyatt.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://hyatt.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "ihg.com", "fetchedAt": "2026-07-30T17:31:47.526622Z", "result": {"base": "https://ihg.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://ihg.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "fourseasons.com", "fetchedAt": "2026-07-30T17:31:52.601558Z", "result": {"base": "https://fourseasons.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://fourseasons.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "bestwestern.com", "fetchedAt": "2026-07-30T17:31:57.755342Z", "result": {"base": "https://bestwestern.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://bestwestern.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "radissonhotels.com", "fetchedAt": "2026-07-30T17:32:03.447593Z", "result": {"base": "https://radissonhotels.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://radissonhotels.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "omnihotels.com", "fetchedAt": "2026-07-30T17:32:08.532050Z", "result": {"base": "https://omnihotels.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://omnihotels.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "kimptonhotels.com", "fetchedAt": "2026-07-30T17:32:14.252638Z", "result": {"base": "https://kimptonhotels.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://kimptonhotels.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "choicehotels.com", "fetchedAt": "2026-07-30T17:32:19.201555Z", "result": {"base": "https://choicehotels.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://choicehotels.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "hotel", "domain": "wyndhamhotels.com", "fetchedAt": "2026-07-30T17:32:24.736827Z", "result": {"base": "https://wyndhamhotels.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (15846 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Corporation.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Wyndham Hotels &amp; Resorts - Find Hotel Rooms, Rates &amp;\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.wyndhamhotels.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "21373 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:accessibility@wyndham.com).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 87, "title": "Wyndham Hotels &amp; Resorts - Find Hotel Rooms, Rates &amp; Deals", "textOnly": "Wyndham Hotels &amp; Resorts - Find Hotel Rooms, Rates &amp; Deals Toggle navigation Toggle search Book Book Find My Reservation Our Locations Hotel Collections Extended-Stay Hotels New Hotels All Offers Book or Redeem TRAVEL BUNDLES Hotel + Flight Packages All-Inclusive Cruises Flights Car Rentals Tours &amp; Activities Transfers Save with Wyndham Travel Bundles Bundle together your room, flights, and more for a seamless booking experience. BOOK NOW MEETINGS &amp; EVENTS Meetings Groups Wyndham Business Group &amp; Meeting Planners Discover flexible spaces for your next meeting or group gathering. LEARN MORE SIGN IN OR JOIN Limited-Time Offer: Earn up to 120,000 Bonus Points with the NEW Wyndham Rewards Earner\u00ae Cards. See if you pre-qualify with no impact to your credit score. Terms &amp; Conditions apply. PRE-QUALIFY NOW Welcome! SIGN IN JOIN Already a member but don\u2019t have an online account? Set Up Your Account --> My Account Favorite Hotels Sign Out English - US English - CA English - UK English - Asia Pacific Fran\u00e7ais \u7b80\u4f53\u4e2d\u6587 Deutsch T\u00fcrk\u00e7e \ud55c\uad6d\uc5b4 Portugu\u00eas Espa\u00f1ol \u0395\u03bb\u03bb\u03b7\u03bd\u03b9\u03ba\u03ae ACCOUNT Book Book TRAVEL BUNDLES MEETINGS &amp; EVENTS LANGUAGE Back Book Find My Reservation Our Locations Hotel Collections Extended-Stay Hotels New Hotels All Offers Back TRAVEL BUNDLES Hotel + Flight Packages All-Inclusive Cruises Flights Car Rentals Tours &amp; Activities Transfers Save with Wyndham Travel Bundles Bundle together your room, flights, and more for a seamless booking experience. BOOK NOW Back MEETINGS &amp; EVENTS Meetings Groups Wyndham Business Group &amp; Meeting Planners Discover flexible spaces for your next meeting or group gathering. LEARN MORE Back LANGUAGE English - US English - CA English - UK English - Asia Pacific Fran\u00e7ais \u7b80\u4f53\u4e2d\u6587 Deutsch T\u00fcrk\u00e7e \ud55c\uad6d\uc5b4 Portugu\u00eas Espa\u00f1ol \u0395\u03bb\u03bb\u03b7\u03bd\u03b9\u03ba\u03ae Welcome! SIGN IN JOIN Already a member but don\u2019t have an online account? Set Up Your Account . My Account Favorite Hotels Sign Out Book or Redeem Clear search Check In Checkout 1 Room Rooms , 1 Guest Guests per Room Special Rates Search New! Road Trip Planner The Easiest Way to Book &amp; Go on Our App SM Learn More Cancel Done Minimum length of stay required Rooms - Rooms 1 Rooms + Rooms Need more than 9 rooms? Adults per Room - Adults per Room 1 Adults per Room + Adults per Room Children per Room - Children per Room 0 Children per Room + Children per Room Cancel Done Wyndham Rewards Points Use Wyndham Rewards Points Special Rates None AAA/CAA AARP Government/Per Diem Military/Veteran Senior Corporate Code Group Code Cancel Done - Change Search | Currency TOP CURRENCIES Currency updated on {updatedDate}. Prices may not reflect the most accurate conversion rates. Default Currency View More ALL CURRENCIES Currency updated on {updatedDate}. Prices may not reflect the most accurate conversion rates. Cancel ALL CURRENCIES Currency updated on {updatedDate}. Prices may not reflect the most accurate conversion rates. &times; Sign In Username or password is invalid Your account has been suspended/closed. Please call Wyndham Rewards Member Services at (844) 405-4141 Your account has been locked. Please call Wyndham Rewards Member Services at (844) 405-4141 Username Password Remember me Sign in Forgot Password? Not a member? Join for free at Wyndham Rewards Security Question You are seeing this message because your account security is important to us and we don't recognize the computer you are using to sign in. To continue please answer the following security validations questions. NOTE: Your answers do not need to be case-sensitive. Answers entered in upper or lowercase are recognized as the same; however, you must use the exact format of your answer i.e. January 31, 2010 is not the same answer as 1/31/10. In what city or town did you meet your spouse/partner long question lorem ipsum dolor? Answer Security question answer does not match Submit Please Set Up Your Security Questions In order to enhance the security of this site and help protect your privacy and identif", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "mailto:accessibility@wyndham.com", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "hotel", "domain": "accor.com", "fetchedAt": "2026-07-30T17:32:32.485579Z", "result": {"base": "https://accor.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Accor Group\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://group.accor.com/en).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5574 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://accor.com/en/contact-hotel-development-team).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 59, "title": "Accor Group", "textOnly": "Accor Group Go to content Go to footer BOOK NOW Open in a new tab FIND A JOB Open in a new tab Search AC 00.000\u20ac 0.00% EN WHO WE ARE WHO WE ARE ACCOR AT A GLANCE What Drives Us People at Our Heart People at Our Heart PEOPLE AT OUR HEART Our Heartist\u00ae Ethos Career Growth &amp; Development Our Culture of Inclusion Sustainable Hospitality Sustainable Hospitality SUSTAINABLE HOSPITALITY Ensuring Future-Proof Operations Empowering All Talent Elevating the Guest Experience Engaging Our Broader Ecosystems Leadership &amp; Governance Our Global Presence Our Partnerships &amp; Alliances Our Story So Far OUR BRANDS &amp; EXPERIENCES OUR BRANDS &amp; EXPERIENCES OUR BRANDS &amp; EXPERIENCES Our Hotel Brands Our Hotel Brands OUR HOTEL BRANDS Luxury Luxury Orient Express Raffles Fairmont Emblems Collection Sofitel MGallery Collection Faena Banyan Tree Lifestyle \u2014 Ennismore Lifestyle \u2014 Ennismore 21c Museum Hotel 25hours Hotels Delano Hyde Mama Shelter Mondrian Morgans Originals Our Habitas Rixos SLS SO/ The Hoxton Premium Premium Swiss\u00f4tel Pullman M\u00f6venpick Grand Mercure Mantis Art Series Angsana Peppers The Sebel Midscale Midscale Handwritten Collection Novotel Mercure TRIBE Mantra Adagio Economy Economy ibis Styles ibis JO&amp;JOE ibis budget greet Breakfree hotelF1 Our Experience Portfolio Our Experience Portfolio OUR EXPERIENCE PORTFOLIO Food &amp; Drink Working &amp; Meeting Wellness &amp; Spa Entertainment &amp; Nightlife Private Rentals ALL Accor Booking &amp; Loyalty Business Accelerators HOTEL DEVELOPMENT HOTEL DEVELOPMENT DEVELOP WITH US Solutions for Every Project Design &amp; Technical Expertise Maximize Your Revenue Optimize Your Costs Sustainable Hotel Performance Contact Our Team FINANCE FINANCE FINANCE Accor Share Price Accor Share Price ACCOR SHARE PRICE Shareholding &amp; Dividend ADR Program Investors &amp; Analysts Investors &amp; Analysts FINANCIAL RESULTS Debt &amp; Rating Analysts Coverage Group Fiscal Policy Financial Press Releases Capital Markets Day ESG Reporting &amp; Performance Shareholders Shareholders INDIVIDUAL SHAREHOLDERS Annual General Meeting Periodic Information Permanent Information NEWS &amp; STORIES NEWS &amp; STORIES ALL OUR NEWS &amp; STORIES Pressroom Trends &amp; Insights Documents &amp; Publications BOOK NOW Open in a new tab FIND A JOB Open in a new tab English French 2026 Half-Year Results An in-depth look at Accor&#x27;s 2026 half-year solid performance, demonstrating the Group\u2019s agility in a challenging global environment. Discover our results 45+ Brands. 5,800 Hotels. Explore our unparalleled portfolio of hotel brands tailored to new ways of traveling, working, and living, spanning segments and geographies and offering diverse experiences to anticipate and meet the expectations of every guest. Orient Express La Minerva, Italy Grand Mercure Hanoi, Vietnam DISCOVER OUR HOTEL BRANDS Explore Our Universe We Are Hospitality Innovators With a bold vision, a trailblazing spirit, a culture of care, and people at our heart, we are reimagining hospitality. Discover our story, our team ethos and how we welcome with passion, creating value while driving responsible growth. DISCOVER WHO WE ARE We Are Creators of Experience Discover our diversified hospitality ecosystem \u2013 the leading brands, innovative services and experiences, and expert solutions that inspire new ways to experience the world and drive business performance for our partners. DISCOVER WHAT WE DO We Are Growth Accelerators With world-class expertise, global scale and power, and local market presence, we deliver tailored solutions that drive success \u2013 making us a trusted partner of choice for hotel owners and investors. DEVELOP WITH US Prev Next Discover Our Latest News &amp; Highlights SUSTAINABILITY 24.07.2026 Accor Launches Sustainability Owners Advisory Board to Strengthen Owner Collaboration READ MORE PRESS RELEASE 07.07.2026 Accor Accelerates Ambitions in China: A Key Engine for Growth and Travel Innovation READ MORE Open in a new tab", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "https://accor.com/en/contact-hotel-development-team", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "hotel", "domain": "ritzcarlton.com", "fetchedAt": "2026-07-30T17:32:38.422997Z", "result": {"base": "https://ritzcarlton.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"The Ritz-Carlton - Luxury Hotels &amp; Resorts\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.ritzcarlton.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7879 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://ritzcarltonleadershipcenter.com/contact-us/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 46, "title": "The Ritz-Carlton - Luxury Hotels &amp; Resorts", "textOnly": "The Ritz-Carlton - Luxury Hotels &amp; Resorts The Ritz-Carlton English Sign in or Join Hotels &amp; Resorts Hotels &amp; Resorts New Hotels Meetings &amp; Celebrations Ritz-Carlton Reserve Residences Yachts About The Ritz-Carlton About The Ritz-Carlton About The Ritz-Carlton Our Legacy Sustainability The Gold Standard With a credo and corporate philosophy of unwavering commitment to service, The Ritz-Carlton sets the standard for luxury experiences the world over. About The Ritz-Carlton The Ritz-Carlton Leadership Center Learn More about The Ritz-Carlton Leadership Center Advisory &amp; Consulting Onsite Programs &amp; Experiences Virtual Programs &amp; Experiences Upcoming Onsite &amp; Virtual Courses The Ritz-Carlton Leadership Center The Ritz-Carlton Leadership Center offers a comprehensive catalog of presentations, keynotes, and workshops that can be delivered at the location of your choosing. Contact Us The Ritz-Carlton Experience Experiences Club Level Suites &amp; Villas Restaurants &amp; Bars Spa &amp; Wellness Beaches &amp; Pools Family Golf Inspiring Life\u2019s Journeys Explore how The Ritz-Carlton aspires to give you an experience that will stay with you forever. The Ritz-Carlton Experience Immersions Reserve Now The Ritz-Carlton Sign in or Join Hotels &amp; Resorts Hotels &amp; Resorts Hotels &amp; Resorts New Hotels New Hotels A New Era Discover your next great escapes at the newest hotels and resorts from The Ritz-Carlton. New Hotels Meetings &amp; Celebrations Meetings &amp; Celebrations Events, Elevated. At The Ritz-Carlton, we help you craft moments that will last a lifetime. Meetings &amp; Celebrations Ritz-Carlton Reserve Residences Yachts About The Ritz-Carlton About The Ritz-Carlton About The Ritz-Carlton About The Ritz-Carlton Our Legacy Sustainability The Gold Standard With a credo and corporate philosophy of unwavering commitment to service, The Ritz-Carlton sets the standard for luxury experiences the world over. About The Ritz-Carlton The Ritz-Carlton Leadership Center The Ritz-Carlton Leadership Center Learn More about The Ritz-Carlton Leadership Center Advisory &amp; Consulting Onsite Programs &amp; Experiences Virtual Programs &amp; Experiences Upcoming Onsite &amp; Virtual Courses The Ritz-Carlton Leadership Center The Ritz-Carlton Leadership Center offers a comprehensive catalog of presentations, keynotes, and workshops that can be delivered at the location of your choosing. Contact Us The Ritz-Carlton Experience The Ritz-Carlton Experience Experiences Club Level Suites &amp; Villas Restaurants &amp; Bars Spa &amp; Wellness Beaches &amp; Pools Family Golf Inspiring Life\u2019s Journeys Explore how The Ritz-Carlton aspires to give you an experience that will stay with you forever. The Ritz-Carlton Experience Immersions Reserve Now Destination Location is required Dates Night Nights (Flexible) --> Dates of stay is required --> --> Find Hotels Rooms and Guests 1 Room, 1 Guest 1 Room, 1 Guest Rooms and Guests 1 Room, 1 Guest Cancel Rooms Adults (Max: 8 total guests/room) Children (Max: 8 total guests/room) (Age-based rates may be available) Done Reset SELECT ONE RATE Cancel Lowest Regular Rate AAA/CAA Senior Discount Government &amp; Military Corp/Promo Code Edit ADD CORP/PROMO CODE Clear Done Use Points/Certificates Find Hotels Please Select Your Preferred Language English \u65e5\u672c\u8a9e \u4e2d\u6587 Destination Location is required Dates Night Nights (Flexible) --> Dates of stay is required --> --> Find Hotels Rooms and Guests 1 Room, 1 Guest 1 Room, 1 Guest Rooms and Guests 1 Room, 1 Guest Cancel Rooms Adults (Max: 8 total guests/room) Children (Max: 8 total guests/room) (Age-based rates may be available) Done Reset SELECT ONE RATE Cancel Lowest Regular Rate AAA/CAA Senior Discount Government &amp; Military Corp/Promo Code Edit ADD CORP/PROMO CODE Clear Done Use Points/Certificates Find Hotels Journey into a world where wonders await around every corner. Welcome to the world according to The Ritz-Carlton. Where to Go Next Fea", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://ritzcarltonleadershipcenter.com/contact-us/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "hotel", "domain": "fairmont.com", "fetchedAt": "2026-07-30T17:32:46.003132Z", "result": {"base": "https://fairmont.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "15 valid JSON-LD block(s) -- types: WebSite, Organization, SiteNavigationElement.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Luxury Hotels Around the World | Fairmont Hotels &amp; Resor\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.fairmont.com/en.html).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "15664 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:+18002577544).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Luxury Hotels Around the World | Fairmont Hotels &amp; Resorts", "textOnly": "Luxury Hotels Around the World | Fairmont Hotels &amp; Resorts Skip to main content This site requires javascript for full functionality. Please enable the javascript in your browser. Stay longer, save more with our Endless Summer Memories offer. Discover more Menu Hotels &amp; Resorts Back Destinations back Americas back United States Austin - Austin Boston - Copley Plaza, Boston Chicago - Chicago, Millennium Park Dallas - Dallas Hawai\u2019i - Orchid Long Beach - Breakers Long Beach Los Angeles - Century Plaza Los Angeles at Beverly Hills Maui - Kea Lani, Maui New Orleans - New Orleans (Opening 2026) New York City - The Plaza, A Fairmont Hotel Pittsburgh - Pittsburgh San Diego - Grand Del Mar San Francisco - Heritage Place, Ghirardelli Square San Francisco - San Francisco Santa Monica - Miramar Hotel &amp; Bungalows Scottsdale - Scottsdale Princess Seattle - Olympic Hotel, Seattle Sonoma - Sonoma Mission Inn &amp; Spa Telluride - Heritage Place, Franz Klammer Lodge Washington, D. C. - Washington D.C. Georgetown Canada Banff - Banff Springs Calgary - Palliser Edmonton - Hotel Macdonald Jasper - Jasper Park Lodge La Malbaie - Le Manoir Richelieu Lake Louise - Chateau Lake Louise Mont Tremblant - Tremblant Montebello - Le Ch\u00e2teau Montebello Montreal - The Queen Elizabeth Ottawa - Ch\u00e2teau Laurier Quebec - Le Ch\u00e2teau Frontenac Toronto - Royal York Vancouver - Hotel Vancouver Vancouver - Pacific Rim Vancouver - Vancouver Airport Vancouver - Waterfront Victoria - Empress Whistler - Chateau Whistler Winnipeg - Winnipeg Barbados - Saint James Bermuda Hamilton - Hamilton Princess &amp; Beach Club - A Fairmont Hotel Southampton - Southampton Brazil - Rio de Janeiro Mexico Acapulco - Heritage Place Acapulco Riviera Maya - Heritage Place, Mayakoba Riviera Maya - Mayakoba Puerto Rico - San Juan Europe back Azerbaijan - Baku Czech Republic - Prague Germany - Hamburg Ireland - Kildare Monaco - Monaco Spain - Costa Del Sol Switzerland Geneva - Grand Hotel Geneva Montreux - Le Montreux Palace Turkey - Istanbul Ukraine - Kiev United Kingdom Knutsford - Cheshire The Mere London - The Savoy - A Fairmont Hotel St Andrews - St Andrews Windsor - Windsor Park Middle East &amp; Africa back Egypt - Cairo Jordan - Amman Kenya Aitong - Mara Safari Club Nairobi - The Norfolk Nanyuki - Mount Kenya Safari Club Morocco Agadir - Taghazout Bay Marrakech - Royal Palm Marrakech Rabat - La Marina Rabat-Sal\u00e9 Rabat - Residences La Marina Rabat Sale Tanger - Tazi Palace Tangier Qatar - Doha Saudi Arabia Mecca - Makkah Clock Royal Tower - A Fairmont Hotel Riyadh - Ramla Riyadh Riyadh - Riyadh South Africa - Cape Town United Arab Emirates Abu Dhabi - Bab Al Bahr, Abu Dhabi Ajman - Ajman Dubai - Dubai Dubai - The Palm Fujairah - Fujairah Beach Resort Asia Pacific back China Beijing - Beijing Chengdu - Chengdu Kunshan - Yangcheng Lake Sanya - Sanya Haitang Bay Shanghai - Peace Hotel Wuhan - Wuhan Xiamen - Xiamen India Jaipur - Jaipur Mumbai - Mumbai Udaipur - Udaipur Palace Indonesia - Jakarta Japan - Tokyo Philippines - Makati Singapore - Singapore South Korea - Seoul Thailand - Bangkok Vietnam - Hanoi New &amp; Upcoming Hotels VIEW ALL DESTINATIONS Elevated Stays Fairmont Gold Discover Fairmont Living Fairmont Heritage Place Fairmont Residences Offers Exclusive offers Endless Summer Memories Unrivaled Fairways CAA/AAA Member Offer Adventure Awaits View all offers Dining Savor the exceptional Restaurants Bars &amp; cocktails Cafes &amp; breakfast View all dining options Experiences Explore unrivaled experiences Wellness Ski Golf Fairmont Experiences Fairmont Gold Events Craft your storied occasion Meetings &amp; Conferences Weddings Social Events &amp; Celebrations Loyalty About Us Timeless elegance &amp; enduring hospitality Our Story Make Special Happen Our Partners Sustainability Press Room Gift Cards Shop Shop The Fairmont Store Gift Cards More All Accor Loyalty EN Back Select your location and language below Geographical area Country/Region - Language CONFIRM FirstNam", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:+18002577544", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "insurance", "domain": "geico.com", "fetchedAt": "2026-07-30T17:32:51.849946Z", "result": {"base": "https://geico.com", "fatal": "Their server refused or timed out (status network error). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://geico.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "allstate.com", "fetchedAt": "2026-07-30T17:32:57.163702Z", "result": {"base": "https://allstate.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://allstate.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "farmers.com", "fetchedAt": "2026-07-30T17:33:02.685308Z", "result": {"base": "https://farmers.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://farmers.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "nationwide.com", "fetchedAt": "2026-07-30T17:33:08.038749Z", "result": {"base": "https://nationwide.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://nationwide.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "thehartford.com", "fetchedAt": "2026-07-30T17:33:16.504429Z", "result": {"base": "https://thehartford.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://thehartford.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "usaa.com", "fetchedAt": "2026-07-30T17:33:21.796497Z", "result": {"base": "https://usaa.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://usaa.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "insurance", "domain": "aflac.com", "fetchedAt": "2026-07-30T17:33:27.567734Z", "result": {"base": "https://aflac.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (5793 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: Organization, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Aflac | #1 Supplemental Insurance Company - Get a Quote\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.aflac.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "16048 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:855-782-5850).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "Aflac | #1 Supplemental Insurance Company - Get a Quote", "textOnly": "Aflac | #1 Supplemental Insurance Company - Get a Quote Skip to main content hide Notifications from Aflac Dismiss all notifications closes this dialog hide As part of Aflac's planned system maintenance, this site will be unavailable or have limited functionality July 3-5. --> Update regarding June 2025 security incident. View notification opens a dialog --> New messages from Aflac View notifications opens a dialog Dismiss closes the notifications alert High Contrast Mode Espa\u00f1ol Newsroom Find an Agent Become an Agent Brokers Log In / Register Individuals Business Owners Brokers Agents Get a Quote Individuals &amp; Families Business Owners Resources File a Claim Search query Clear Submit Enable High Contrast Mode Disable High Contrast Mode Espa\u00f1ol Newsroom --> Find an Agent Log In / Register Individuals Business Owners Brokers Agents Products Direct Products Accident Insurance Cancer Insurance Critical Illness Insurance Dental Insurance Dental, Vision and Hearing Insurance Final Expense Insurance Life Insurance Medicare Supplement --> Pet Insurance Supplemental Dental Insurance Employer Offered Products Hospital Insurance Short-Term Disability Insurance Supplemental Vision Insurance Products Products: Accident Insurance Cancer Insurance Critical Illness Insurance Life Insurance Hospital Insurance Short-Term Disability Insurance Dental Insurance Supplemental Dental Insurance Vision Insurance Supplemental Vision Insurance Services Cafeteria Plans Resources Article Topics Accident Insurance Advice Cancer Insurance Advice Dental Insurance Advice Life Insurance Advice Health Insurance Advice Customer Resources Getting Started Guide Aflac Always How to Access Your Account Filing Wellness Benefit Claims Life Event Changes Find a Provider Find a Provider Tools Life Insurance Calculator Request a Quote Call Us Health insurance wasn\u2019t designed to cover everything. Health insurance wasn\u2019t designed to cover everything. Aflac has the assist. --> That\u2019s why there\u2019s Aflac. We can help you take care of the expenses health insurance doesn\u2019t cover, so you can take care of everything else. Learn how Aflac can help. Aflac has the assist. Aflac has the assist. Aflac has the assist. --> When you\u2019re sick or injured, Aflac pays cash to help cover what health insurance doesn\u2019t. Enter your info to get started. Zip Code Age Covering Me Me &amp; My Spouse Me &amp; My Children Me, My Spouse &amp; Children My Business Get a Free Quote Opening talk to an agent form. Get a Quote Choose one of the options below. Speak to An Agent Quote &amp; Apply Online Opening business form. Get a Quote Change your info. Internal Page Navigation Our Products Get Coverage --> We're here to help. COVID-19 update Learn More --> Discover how our insurance policies can help provide financial peace of mind: Accident Insurance Get added peace of mind and help paying for emergency treatment. Cancer Insurance 1 Get help coping with the financial and emotional effects of cancer. Critical Illness Insurance Get help with the cost of treating covered critical illnesses. Dental Insurance Choose from Aflac\u2019s coverage options for the level and premiums that are right for you. Final Expense Insurance Final expense whole life insurance helps loved ones cover funeral and other final expenses. Medicare Supplement 2 Can help pay for some of the costs that Original Medicare doesn't, like copay, coinsurance and deductibles. Life Insurance Help protect the financial future of your loved ones. Pet Insurance 3 Help for when your pet experiences the unexpected. Additional products available through an employer: Hospital Insurance Help keep high deductibles from derailing your life plans. Short-Term Disability Help ease the financial stress caused by a covered injury. Supplemental Dental Get the benefits you need&mdash;with no networks or deductibles. --> Supplemental Vision Find the care you need to help maintain healthy vision. Group products designed for large employers: Dental Insurance Help keep ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:855-782-5850", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "insurance", "domain": "libertymutual.com", "fetchedAt": "2026-07-30T17:33:33.222417Z", "result": {"base": "https://libertymutual.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (6341 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization, WebSite, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Liberty Mutual | Trusted Insurance for Over 100 Years\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.libertymutual.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5747 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:844-251-3582).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "Liberty Mutual | Trusted Insurance for Over 100 Years", "textOnly": "Liberty Mutual | Trusted Insurance for Over 100 Years Skip to content Liberty Mutual Insurance logo keyword(s) Search Shop insurance Auto and vehicle Car Motorcycle Boat and watercraft Antique and classic car ATV and off road vehicle Recreational vehicle (RV) Umbrella Property Home Renters Condo Landlord Mobile home Flood Umbrella Bundle and save Auto and home Auto and renters Auto and condo Other Life Identity theft Pet insurance Small business Business owner&#x27;s policy (BOP) Workers compensation Auto More coverages Midsize and large business Commercial and auto Environmental liability Equipment breakdown More coverages Connect with us Call for a quote 844-251-3582 (personal insurance) 844-925-1271 (small business insurance) More information Insurance resources About Liberty Mutual Careers Get a quote Claims Personal claims Claims center Manage your claim File an auto glass claim File a claim against a Liberty Mutual customer Business claims Small business claims Mid to large size business claims Helpful resources Auto claims process Home claims process Connect with us Personal insurance claims File a claim Manage claim Or call: 800-2CLAIMS Small business claims Call: 844-325-2467 Mid to large size business claims Call: 800-362-0000 Customer support Personal Frequently asked questions Manage your policy Billing and payment options Download the Liberty Mutual app Pay your bill Fraud Protection Secure upload portal Small business Small business support Manage your policy Liberty+ Prepare and prevent Home improvement tips Connect with us Personal insurance Log in Create account Text us 844-629-8981 Roadside Assistance* 877-762-3101 *Additional charges apply if your policy does not include the Towing and Labor coverage. Small business Log in Create account Midsize and large business International Find a local agent Search Log in Personal insurance Log in to manage your auto, home, condo, renters, and more. Log in Create account Agent portal log in Small business insurance Log in to manage your small business insurance. Log in Create account Agent portal log in Personal insurance Log in to manage your auto, home, condo, renters, and more. Log in Create account Agent portal log in Small business insurance Log in to manage your small business insurance. Log in Create account Agent portal log in You could save when you bundle insurance 1 Get a quote from Liberty Mutual and see how much you could save when you only pay for what you need . Auto Home Condo Renters Pet Motorcycle Life Small Business ZIP Code Get my price Find a local agent Protection you can trust, at a price that works for you. 100+ Years of experience Trusted for over 100 years, Liberty Mutual has the experience and stability to back you up. $950 Average savings You could save over $950 a year on car and home insurance when you bundle online. 1 24/7 Customer support Access claims and service support anytime from both our website or highly-rated mobile app. We have you covered. Learn more about our products and start a Liberty Mutual quote to find the right coverage for you 1 Car insurance Learn about car insurance coverages and find discounts to help you save. 2 Homeowners insurance See what homeowners insurance covers, how it works, and what savings are available to you. 3 Renters insurance We can help you learn about options to protect your things from damage, theft, and more. 4 Pet insurance Learn about pet insurance coverage for your furry friends. Need to file an insurance claim? Just 5-10 minutes and the basic details gets you started. File a claim Manage your claim Not sure where to start? Start a quote the way that works best for you. Quote on your own We&#x27;ve made it easy for you to shop independently and confidently. Purchase your policy online in just a few minutes. Get a quote Work with an agent For those looking for extra support, find a local independent agent and get personalized advice and a quote. Find a local agent Quickly pay your bill or get y", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"GPTBot, ChatGPT-User, OAI-SearchBot, Google-Extended, Anthropic, ClaudeBot, Claude-Web, CCBot, PerplexityBot, Bytespider, Amazonbot, FacebookBot, Meta-ExternalAgent, Applebot-Extended, cohere-ai\" has no root Disallow."}], "contactMatch": "tel:844-251-3582", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "insurance", "domain": "metlife.com", "fetchedAt": "2026-07-30T17:33:38.739106Z", "result": {"base": "https://metlife.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (2195 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: BreadcrumbList, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Insurance and Employee Benefits | MetLife\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.metlife.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10840 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:+18006385433).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 90, "title": "Insurance and Employee Benefits | MetLife", "textOnly": "Insurance and Employee Benefits | MetLife Skip Navigation SOLUTIONS SUPPORT ABOUT US RESOURCES LOG IN SEARCH LOG IN SEARCH SOLUTIONS Insurance Business and Brokers MetLife Investment Management Retirement &amp; Income Solutions Insurance Enhance Your Health Coverage Dental Vision Pet Insurance Protect Your Finances Accident Hospital Indemnity Critical Illness Cancer Disability and Absence Safeguard Your Valuables Auto and Home Insurance Identity &amp; Fraud Protection Secure Your Family&#39;s Future Term Life Group Variable Universal Life Group Universal Life Insurance Accidental Death &amp; Dismemberment Legal Plans Reach Your Financial Goals Health Savings and Spending Accounts Total Control Account (TCA) Employer-Sponsored Retirement Plans More Solutions Annuity Long-Term Care Business and Brokers Employee Benefits Group Auto &amp; Home Group Dental Group Accident and Health Group Life Group Disability and Absence Group Legal Services Group Vision Group Pet Group Health Savings &amp; Spending Accounts Executive Benefits Identity &amp; Fraud Protection Upwise Small Business Small Business Products Exceptional Service &amp; Support Small Business Index Specialty Business Professional Employer Organization (PEO) Association Union Chamber of Commerce For Brokers Small Market Experience Getting Appointed Broker Compensation Broker Forms Why MetLife? Broker Service HR Tech Integrations MetLife Benefits Bridge MetLife Worldwide Benefits Financial Wellness Workforce Insights Benefit Trends Study MetLife Investment Management MetLife Investment Management Financing Solutions About Us Insights Investment Strategies Retirement &amp; Income Solutions Retirement &amp; Income Solutions Lifetime Income Annuities Insights Employer-Sponsored Retirement Plans Pension Risk Structured Settlements Longevity Reinsurance Stable Value Funding Solutions SUPPORT Support and Manage Quick and easy access to MetLife customer support services and resources. Learn More Customer Care Customer Care Contact Us Find a Dentist Find a Vision Provider Find an Insurance Policy Beneficiary Assistance More Support My Account My Account Account Log In Register for MyBenefits Find a Form File a Claim For Current Customer Fraud Awareness Ethics &amp; Fraud HelpLine ABOUT US About MetLife Careers at MetLife Investor Relations Sustainability About MetLife Corporate Profile MetLife Worldwide Company Ratings Global Procurement MetLife Affiliates Corporate Governance Board of Directors Our Executive Leadership Team Corporate Conduct Code of Conduct Artificial Intelligence Policy Our Purpose Our People Our Customers Our Communities Our Shareholders More About Us Careers at MetLife Newsroom Awards and Recognition MetLife Foundation Next Gen Ventures (NGV) Careers at MetLife Investor Relations Sustainability 2025 Sustainability Report 01 Sustainability at MetLife 02 For Our Customers 03 For Our Workforce 04 For Our Communities 05 As an Investor 06 Responsible Governance Sustainability Resource Center Environmental Stewardship Social Responsibility Corporate Governance Statements, Policies and Reports Our Sustainability Aspirations Our Sustainability Aspirations \u2013 Net Zero MetLife Sustainable Financing Framework MetLife Foundation Our Work Employee Engagement &amp; Volunteerism MetLife Foundation News &amp; Media MetLife Foundation Resources RESOURCES MetLife Guides Learn about different types of insurance and employee benefits, as well as smart moves for your finances, retirement, and more\u2014so you can plan ahead for you and your family. Learn More Insurance Insurance Accident &amp; Health Insurance Dental Insurance Disability Insurance Legal Insurance Life Insurance Pet Insurance Vision Insurance Financial Wellness Financial Wellness Employee Benefits Identity Theft Oral Health Library Personal Finance Retirement Lifetime Income Hub Retirement Planning Log in to MetLife Select Login Type Personal Account Business Account Broker Account Whether you purchased your policy on you", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:+18006385433", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "gov", "domain": "irs.gov", "fetchedAt": "2026-07-30T17:33:44.457885Z", "result": {"base": "https://irs.gov", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Internal Revenue Service | An official website of the United\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.irs.gov/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12207 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:800-829-1040).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 51, "title": "Internal Revenue Service | An official website of the United States government", "textOnly": "Internal Revenue Service | An official website of the United States government Skip to main content Home Page An official website of the United States government Here's how you know Here's how you know Official websites use .gov A .gov website belongs to an official government organization in the United States. Secure .gov websites use HTTPS A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites. English Espa\u00f1ol \u4e2d\u6587 (\u7b80\u4f53) \u4e2d\u6587 (\u7e41\u9ad4) \ud55c\uad6d\uc5b4 \u0420\u0443\u0441\u0441\u043a\u0438\u0439 Ti\u1ebfng Vi\u1ec7t Krey\u00f2l ayisyen Information Menu Help News Tax Pros Sign in Individual Business Tax Pro Access your tax information with an IRS account. Refunds Payments Tax records Create account Search Sign in Individual Business Tax Pro Access your tax information with an IRS account. Refunds Payments Tax records Create account Menu Search Include Historical Content - Any - No Include Historical Content - Any - No Search Main navigation File Overview INFORMATION FOR\u2026 Individuals Business and self-employed Charities and nonprofits International filers Retirement Governments and tax-exempt bonds Indian Tribal Governments FILING FOR INDIVIDUALS How to file When to file Where to file Update your information POPULAR Tax record (transcript) Apply for an Employer ID Number (EIN) Amend return Identity Protection PIN (IP PIN) Pay Overview PAY BY Bank Account (Direct Pay) Payment Plan (Installment Agreement) Electronic Federal Tax Payment System (EFTPS) POPULAR Your Online Account Tax Withholding Estimator Estimated Taxes Penalties Refunds Overview About refunds Direct deposit Reduced refunds Where\u2019s my amended return? Credits & Deductions Overview INFORMATION FOR... Individuals For you and your family Businesses & Self-Employed Standard mileage and other information POPULAR Earned Income Credit (EITC) Child Tax Credit Clean Energy and Vehicle Credits Standard Deduction Retirement Plans Forms Overview POPULAR FORMS & INSTRUCTIONS Form 1040 Individual Income Tax Return Form 1040 Instructions Instructions for Form 1040 Form W-9 Request for Taxpayer Identification Number (TIN) and Certification Form 4506-T Request for Transcript of Tax Return Form W-4 Employee&#039;s Withholding Certificate Form 941 Employer&#039;s Quarterly Federal Tax Return Form W-2 Wage and Tax Statement Form 9465 Installment Agreement Request POPULAR FOR TAX PROS Form 1040-X Amended Individual Income Tax Return Form 2848 Power of Attorney Form W-7 Apply for an ITIN Circular 230 Regulations Governing Practice before the IRS Report Fraud Overview Tax fraud and scams Identity theft Fake IRS email or message Tax return preparer Search Include Historical Content - Any - No Include Historical Content - Any - No Search Information Menu Help News Tax Pros Sign in Individual Business Tax Pro Access your tax information with an IRS account. Refunds Payments Tax records Create account File Overview INFORMATION FOR\u2026 Individuals Business and self-employed Charities and nonprofits International filers Retirement Governments and tax-exempt bonds Indian Tribal Governments FILING FOR INDIVIDUALS How to file When to file Where to file Update your information POPULAR Tax record (transcript) Apply for an Employer ID Number (EIN) Amend return Identity Protection PIN (IP PIN) Pay Overview PAY BY Bank Account (Direct Pay) Payment Plan (Installment Agreement) Electronic Federal Tax Payment System (EFTPS) POPULAR Your Online Account Tax Withholding Estimator Estimated Taxes Penalties Refunds Overview About refunds Direct deposit Reduced refunds Where\u2019s my amended return? Credits &amp; Deductions Overview INFORMATION FOR... Individuals Businesses &amp; Self-Employed POPULAR Earned Income Credit (EITC) Child Tax Credit Clean Energy and Vehicle Credits Standard Deduction Retirement Plans Forms Overview POPULAR FORMS &amp; INSTRUCTIONS Form 1040 Form 1040 Instructions Form W-9 Form 4506-T Form W-4 Form 941 Form W-2 Form 9465 POPULAR FOR TAX PROS Form 1040-X Form 2848 Form W-7 Circular", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:800-829-1040", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "gov", "domain": "ssa.gov", "fetchedAt": "2026-07-30T17:33:49.456151Z", "result": {"base": "https://ssa.gov", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://ssa.gov", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "gov", "domain": "studentaid.gov", "fetchedAt": "2026-07-30T17:33:55.008692Z", "result": {"base": "https://studentaid.gov", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://studentaid.gov", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "banking", "domain": "tdbank.com", "fetchedAt": "2026-07-30T17:34:10.158480Z", "result": {"base": "https://tdbank.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "Microdata found.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Online Banking, Loans, Credit Cards &amp; Home Lending | TD \u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "23882 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://tdbank.com/us/en/personal-banking/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 70, "title": "Online Banking, Loans, Credit Cards &amp; Home Lending | TD Bank", "textOnly": "Online Banking, Loans, Credit Cards &amp; Home Lending | TD Bank Notification closed &lt;i>FDIC-Insured \u2013 Backed by the full faith and credit of the U.S. Government&lt;/i>&lt;/p>\\r\\n&#34;}}\" id=\"text-dd7b25e7f2\" class=\"cmp-text\"> FDIC-Insured \u2013 Backed by the full faith and credit of the U.S. Government Skip to main content Personal Selected Small Business Commercial Investing &amp; Wealth About Us Search Help Find Us United States Canada United States Selected Log In Log in to your other accounts Banking and Investing Online Banking EasyWeb TD Wealth TD Private Client Wealth TD Investment Services (US) TD Automated Investing TD Wealth Securities-Based Lending Cards Credit Card Gift Card TD Commercial Plus Card Lending TD Fit Loan TD Home Lending Other TD Business Central (US) TD eTreasury TDFX TD Bank Trade TD Remote Deposit Capture Checking &amp; Savings Go Back Checking &amp; Savings Close Open My Account Help me choose Checking Savings Student CDs IRAs Cross-Border Banking TD Premier Set Up My Account Enroll in Online Banking Download the TD Bank app Get started with TD Checking See my debit card perks Get TD Early Pay with direct deposit Explore how-to tutorials Make the Most of My Account See my overdraft options Pay bills and schedule payments Order foreign currency Send Money with Zelle\u00ae Report fraud Earn $300 with TD Beyond Checking Open a TD Beyond Checking account today and enjoy a $300 bonus with qualifying activities. Plus, get paid early with TD Early Pay. Learn more Earn $200 with TD Signature Savings Open a TD Signature Savings account today to earn $200 with qualifying activities. We offer competitive interest rates. Learn more Credit Cards Go Back Credit Cards Close Choose My Credit Card Explore all credit cards Check if I\u2019m prequalified Balance transfer cards Cash back and points cards No annual fee cards Low intro APR cards Build credit Business Set Up My Credit Card Manage my credit card Enroll in Online Banking Download the TD Bank app Manage payments Explore how-to tutorials Make the Most of My Credit Card See what my card can do Explore Visa benefits Use my digital wallet Report fraud Exclusive offer ahead. See if you prequalify for a chance to unlock a special credit card offer. Check offers Up to 3% custom Cash Back. Enjoy a low intro APR on balance transfers for 15 billing cycles and purchases for 12 billing cycles. Learn more Lending Go Back Lending Close Choose My Loan Help me choose Personal loan Mortgage See my home loan rate Home equity Manage My Loan Review my home loan application Enroll in Online Banking Make home loan payments Make personal loan payments Explore More Loan Options Buy my first home Build or renovate Find affordable home loans Refinance Compare home equity options Get funds for almost anything\u2014fast. From vacation to renovation\u2014get a TD Fit loan up to $50K. Learn more There&#39;s a story behind every home loan. Buying a new home? Building an addition? Whatever your story, we&#39;ve got the home lending solutions you need. Learn more Learning Go Back Learning Close Life Stages Explore life stages Paying for school Starting out on my own Buying a home Retirement Topics View all topics Banking Borrowing money Budgeting &amp; saving Building &amp; managing credit Business banking Financial wellness Tools &amp; Calculators View all tools &amp; calculators Monitor my credit Check if I\u2019m prequalified for credit cards Explore how-to tutorials Refinance Compare home equity options See my home loan rate Find a store See details and directions for a TD Bank near you. Find a TD Bank Book an appointment Meet with a banking specialist in a store or by phone. Book now Ask the TD Help Center Find answers to your everyday banking questions. Get help TD Premier Open Log In TD: Personal Personal Selected Small Business Commercial Investing &amp; Wealth About Us Checking &amp; Savings Go Back Checking &amp; Savings Close Open My Account Help me choose Checking Savings Student CDs IRAs Cross-Border ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://tdbank.com/us/en/personal-banking/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "banking", "domain": "key.com", "fetchedAt": "2026-07-30T17:34:15.790033Z", "result": {"base": "https://key.com", "fatal": "Their server refused or timed out (status 503). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://key.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "airline", "domain": "united.com", "fetchedAt": "2026-07-30T17:34:21.549595Z", "result": {"base": "https://united.com", "fatal": "Their server refused or timed out (status 520). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://united.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "airline", "domain": "delta.com", "fetchedAt": "2026-07-30T17:34:26.643059Z", "result": {"base": "https://delta.com", "fatal": "Their server refused or timed out (status 521). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://delta.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "airline", "domain": "aa.com", "fetchedAt": "2026-07-30T17:34:31.956923Z", "result": {"base": "https://aa.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://aa.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "airline", "domain": "jetblue.com", "fetchedAt": "2026-07-30T17:34:38.825724Z", "result": {"base": "https://jetblue.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"JetBlue | Why settle for just cheap airline tickets? Get mor\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "1526 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://www.jetblue.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 52, "title": "JetBlue | Why settle for just cheap airline tickets? Get more for less. | JetBlue", "textOnly": "JetBlue | Why settle for just cheap airline tickets? Get more for less. | JetBlue Skip to main content Sign in JetBlue Home Flights Flights + Hotel Cruises Cars Stays Round trip Round trip One-way Multi-city 1 Adult 1 Adult Use TrueBlue points From From To To Depart Return Use TrueBlue points Search flights Explore more flights Packages, trip deals and more Bundle and save or add extras as you go. All-inclusives Flights, hotel, savings and perks\u2014all included. Cruises Save big on a flight + cruise package or just a cruise. Family vacations Bundle flights + hotel together for big family savings. Hotel deals Unlock up to 20% off hotels. Plus, earn points &amp; tiles. Car rental deals Drive off with up to 35% off, all while earning points &amp; tiles. Trip activities Book trip activities and earn points &amp; tiles while you\u2019re at it. So many ways to JetBlue. Flights Explore packages Explore trip deals Points &amp; Perks Credit Cards Need help? Search for answers Get To Know Us Get To Know Us Credit Cards Our Company Partner Airlines Travel Agents Investor Relations Careers Site Map Contact Us Policies Policies Legal Accessibility Contract of Carriage Tarmac Delay Plan Customer Service Plan Privacy JetBlue In Action JetBlue In Action JetBlue for Good Sustainability Stay Connected Download the JetBlue app Join our email list See help topics English English Espa\u00f1ol Fran\u00e7ais Italiano Indicates link opens an external site and may not conform to the same accessibility policies as JetBlue. \u00a92026 JetBlue Airways", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://www.jetblue.com/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "airline", "domain": "suncountry.com", "fetchedAt": "2026-07-30T17:34:43.889923Z", "result": {"base": "https://suncountry.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Sun Country Airlines - Low Fares. Nonstop Flights.\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": false, "detail": "Missing -- crawlers may treat duplicate/parameterized URLs as separate pages.", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": false, "detail": "50 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": false, "detail": "No obvious contact/action path in HTML.", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 36, "title": "Sun Country Airlines - Low Fares. Nonstop Flights.", "textOnly": "Sun Country Airlines - Low Fares. Nonstop Flights.", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": null, "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "telecom", "domain": "t-mobile.com", "fetchedAt": "2026-07-30T17:34:48.998090Z", "result": {"base": "https://t-mobile.com", "fatal": "Their server refused or timed out (status 403). That's itself a real signal -- either it's down, or it's blocking automated requests, possibly including AI crawlers.", "blockedByTarget": true, "checks": [], "score": 0, "title": "https://t-mobile.com", "textOnly": "", "bots": [], "contactMatch": null}}
{"corpus": "dental", "domain": "1800dentist.com", "fetchedAt": "2026-07-30T17:34:54.667945Z", "result": {"base": "https://1800dentist.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (133680 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: BreadcrumbList, Organization, WebPage, WebSite, LocalBusiness.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Emergency Dentist Near Me 24 Hour Emergency Dentist Office N\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://1800dentist.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "9648 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:(888)-820-1564).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 95, "title": "Emergency Dentist Near Me 24 Hour Emergency Dentist Office Near Me Urgent Dental Care", "textOnly": "Emergency Dentist Near Me 24 Hour Emergency Dentist Office Near Me Urgent Dental Care Dentist Near Me | Find a Dentist 24/7 | 1-800-DENTIST Home How it Works About Us Dental Emergencies Are You a Dentist? X Call 1-800-DENTIST\u00ae Home SIMPLIFY YOUR SEARCH Connect with a Qualified Dentist Now We understand how overwhelming it can be to find a dentist. We&#8217;re here to take that frustration away and connect you to a dentist in seconds, for free with no obligations. Call 1-800-DENTIST or 0 patients have found their new dentist 1-800-DENTIST is the nation&#8217;s #1 FREE dental referral service with a network of 25,000 dentists at your fingertips. We can connect you directly to a local, qualified dentist to take care of your needs quickly. Begin with our Simple Process. Call 1-800-DENTIST Call us now to connect with a qualified dentist. We offer our referral services free of charge. Enter your Zip Code For over 30 years, we&#8217;ve vetted our roster of dentists to make sure you&#8217;re matched with a qualified and reputable dentist across 25,000 zip codes nationwide Get Connected We&#8217;ll locate a dental office near you who can answer your questions and provide you a hassle-free dental experience. PATIENT RESOURCE Top 5 Questions to Ask When Choosing a Dentist We know finding the right dentist can be daunting. Here are a few questions to ask so you feel comfortable and confident with your dental match.\u200b Call 1-800-DENTIST What are your payment options? When you're considering a new dental office, it's essential to understand how you can manage the financial aspects of your dental care. By asking about payment options, you can find out if the dental office accepts your insurance, credit cards, or if they offer payment plans that suit your budget. This information will help you plan for your dental treatments more effectively. What services do you offer? You want to make sure that the dental office can address your specific dental needs. By asking about the services they offer, you can determine if they provide a comprehensive range of treatments, including the ones you require, such as preventive care, orthodontics, oral surgery, or cosmetic procedures. This question helps you assess whether the dental office is the right fit for your unique oral health needs. How long have you been in practice? Building trust in your dental care is crucial. You may feel more comfortable with a dental office that has a history of successful treatments and satisfied patients. By asking how long they've been in practice, you can get a sense of their experience and reliability, which can give you more confidence in the care they provide. Do you have emergency appointments or services? Dental emergencies can happen at any time, and it's important to know that the dental office can assist you in such situations. By inquiring about emergency appointments and services, you can ensure that they can provide prompt care when you need it urgently. This question helps you be prepared for unexpected dental issues and assures you that the dental office can be there for you when it matters most. Do you speak Spanish or another language? Effective communication is key in your dental care. If you're not fluent in English, it's essential to know if the dental office staff can speak your preferred language, such as Spanish. This ensures that you can comfortably express your concerns, understand treatment instructions, and discuss your oral health in a language you're comfortable with. This question reflects the dental office's commitment to accommodating your language needs and making your experience as smooth as possible. Do you have a Dental Emergency? If you are experiencing any of the following symptoms or conditions, you have a dental emergency. You may need to contact an emergency dentist, or call 911. Severe Toothache Severe toothaches can range from a nagging pain to excruciating discomfort. To determine if it's a dental emergency, consider the following", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:(888)-820-1564", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "gentledental.com", "fetchedAt": "2026-07-30T17:35:00.117048Z", "result": {"base": "https://gentledental.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": true, "detail": "Found (15867 bytes).", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: WebPage, WebSite, FAQPage, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Find A Dentist | Dental Care Services | Gentle Dental of New\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.gentledental.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "12347 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://gentledental.com/about-us/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 95, "title": "Find A Dentist | Dental Care Services | Gentle Dental of New England", "textOnly": "Find A Dentist | Dental Care Services | Gentle Dental of New England Skip to main content Locations Location Overview Massachusetts Boston Boston Boston - Newbury Street Brighton Brookline Jamaica Plain South Boston West Roxbury Greater Boston Greater Boston Arlington Belmont Brockton Burlington Cambridge Malden Medford Norwood Somerville Stoughton Waltham Metrowest Metrowest Franklin Hudson Milford Natick Merrimack Valley Merrimack Valley Chelmsford Methuen North Andover North Shore North Shore Beverly Peabody Saugus Wakefield South Coast South Coast Attleboro New Bedford Seekonk South Shore South Shore Braintree Hanover Quincy Worcester Worcester Worcester at The Trolley Yard Worcester - Shrewsbury Street New Hampshire Manchester Manchester Manchester Elm Street Manchester South Willow Nashua Nashua Nashua - Main Street South Nashua All New Hampshire All New Hampshire Concord Derry Dover Exeter Keene Rochester New Patient Offer $ 79 A $ 400+ Value Exam X-rays Cleaning Treatment Plan See Details Book Now Dental Services Dental Services Overview Cosmetic Dentistry Cosmetic Dentistry Smile Makeover Invisalign\u00ae Whitening Veneers Emergency Dental Care Oral Surgery Oral Surgery Extractions Wisdom Teeth Orthodontics Orthodontics Braces Invisalign\u00ae Preventive &amp; General Care Preventive &amp; General Care Exams Digital x-rays Cavity Prevention Plan Cleaning Fluoride Oral Cancer Screening Sealants Diabetes And Oral Health Restorative Dental Care Restorative Dental Care Crowns &amp; Bridges Dental Fillings Root Canals Emergency Dental Care Gum Treatments Partial &amp; Full Dentures Implants Specialty Care Specialty Care Pediatric Dentistry Sedation Dentistry Sleep Apnea Treatment TMD/TMJ Treatment Botox &amp; Injectibles New Patient Offer $ 79 A $ 400+ Value Exam X-rays Cleaning Treatment Plan See Details Book Now Payment Options Dental Offers Dental Offers New Patient Offer - $79 Consultations Payment &amp; Insurance Pay My Bill New Patient Offer $ 79 A $ 400+ Value Exam X-rays Cleaning Treatment Plan See Details Book Now Patient Resources Patient Resources What to Expect at Your First Visit Before &amp; After Dental Definitions FAQs Articles About Us About Us Health &amp; Safety Awards &amp; Recognition Meet Our Dentists Press Partners Contact Us Careers Affiliate Your Practice Locations Locations Location Overview Massachusetts Boston Boston Boston - Newbury Street Brighton Brookline Jamaica Plain South Boston West Roxbury Greater Boston Greater Boston Arlington Belmont Brockton Burlington Cambridge Malden Medford Norwood Somerville Stoughton Waltham Metrowest Metrowest Franklin Hudson Milford Natick Merrimack Valley Merrimack Valley Chelmsford Methuen North Andover North Shore North Shore Beverly Peabody Saugus Wakefield South Coast South Coast Attleboro New Bedford Seekonk South Shore South Shore Braintree Hanover Quincy Worcester Worcester Worcester at The Trolley Yard Worcester - Shrewsbury Street New Hampshire Manchester Manchester Manchester Elm Street Manchester South Willow Nashua Nashua Nashua - Main Street South Nashua All New Hampshire All New Hampshire Concord Derry Dover Exeter Keene Rochester Dental Services Dental Services Dental Services Overview Cosmetic Dentistry Cosmetic Dentistry Smile Makeover Invisalign\u00ae Whitening Veneers Emergency Dental Care Oral Surgery Oral Surgery Extractions Wisdom Teeth Orthodontics Orthodontics Braces Invisalign\u00ae Preventive &amp; General Care Preventive &amp; General Care Exams Digital x-rays Cavity Prevention Plan Cleaning Fluoride Oral Cancer Screening Sealants Diabetes And Oral Health Restorative Dental Care Restorative Dental Care Crowns &amp; Bridges Dental Fillings Root Canals Emergency Dental Care Gum Treatments Partial &amp; Full Dentures Implants Specialty Care Specialty Care Pediatric Dentistry Sedation Dentistry Sleep Apnea Treatment TMD/TMJ Treatment Botox &amp; Injectibles Payment Options Payment Options Dental Offers Dental Offers New Patient Offer - $79 Consultations P", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://gentledental.com/about-us/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "smiledirectclub.com", "fetchedAt": "2026-07-30T17:35:06.833189Z", "result": {"base": "https://smiledirectclub.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"What Happened to SmileDirectClub? | SmileSet\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://smileset.com/pages/former-smiledirectclub-customers).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6916 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (mailto:privacy@smileset.com\\).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 77, "title": "What Happened to SmileDirectClub? | SmileSet", "textOnly": "What Happened to SmileDirectClub? | SmileSet &nbsp &nbsp Skip to content Get a Free 15-min Consultation Subscribe & Save 40% on Retainers Extra 20% OFF with code: Back2School Aligners Retainers Night Guards Oral Care About Us Am I A Candidate? Sign In Aligners Retainers Night Guards Oral Care About Us Am I a Candidate? Sign In Cart Welcome to SmileSet Powered by SmileDirectClub Technology Welcome to SmileSet Powered by SmileDirectClub Technology What Happened to SmileDirectClub? SmileSet has acquired SmileDirectClub's groundbreaking technology and assets, incorporating their innovative patents and manufacturing systems into our established business. Experience the same trusted technology that has transformed over 2 million smiles worldwide, now enhanced with SmileSet's comprehensive approach to orthodontic care. Former SmileDirectClub Customer? As a former SmileDirectClub client, you may qualify for special, deep discounts! Watch out for messages from our team or reach out to us here so we can get your smile set. Aligners SmileSet offers Aligner Treatments for former SmileDirectClub customers. We provide a special $800 discount for former SDC customers to help you continue your smile journey with clear aligners. Our state-licensed dental professionals will create a new treatment plan tailored to your current smile position to support you every step of the way. Get Started Retainers SmileDirectClub's former dental professionals have partnered with SmileSet to make it easy and affordable for former SDC customers to get retainers . Order your custom retainers on subscription with SmileSet to help preserve your smile. If you're a former customer you can get deep discounts on subscription because you likely won't need any additional 3D scans or putty impressions. Order Retainers Oral Care SmileSet offers a complete line of truly premium oral care products to keep your smile bright and healthy. Ranging from toothbrushes and floss to whitening and appliance cleaners we help keep your dental assets in mint condition. Plus, we still offer the SmileDirectClub customer favored lip balm! Shop Now Why Customers Choose SmileSet Compare your options and see why over 2 million customers trust us. Traditional Braces Price $995.00 $5000+ Office Visits None required Monthly Treatment Time 4-6 months 18-24 months Visibility Nearly invisible Metal visible Doctor Reviewed Yes Yes 100+ Patents FDA-Cleared 2M+ Happy Customers BPA-Free Materials Frequently Asked Questions Got questions? We've got answers. Here are the most common questions about SmileSet. What happened to SmileDirectClub? SmileDirectClub ceased operations in 2023. SmileSet acquired their technology and assets, allowing us to incorporate their innovative platform into our comprehensive approach to smile care. Is SmileSet the same thing as SmileDirectClub? No. SmileSet is a distinct business entity that purchased SmileDirectClub's assets. While we acquired their technological innovations, SmileSet operates with our own distinct business practices and customer commitments. I was a former SmileDirectClub customer, what's this mean for me? As part of our technology acquisition, SmileSet now supports some of the licensed dental professionals who worked with SmileDirectClub. This means you can order SmileSet retainers or night guards without needing new impressions in most cases. For aligner treatment, we require new impressions to ensure the best possible outcome. While we cannot honor SmileDirectClub's previous guarantees or warranties, we are offering special discounts on aligners for former SDC customers to help you continue your smile journey. Can I continue my treatment if I was mid-treatment with SmileDirectClub? For your safety and the best possible outcome, our dental professionals need to create a treatment plan based on your current smile position. We can't simply continue a treatment plan created by another company as this could compromise your results. We understand this is a di", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "mailto:privacy@smileset.com\\", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "heartland.com", "fetchedAt": "2026-07-30T17:35:12.165886Z", "result": {"base": "https://heartland.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, ImageObject, BreadcrumbList, WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Heartland Dental | America\u2019s Leading Dental Support Organiza\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://heartland.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "7561 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://heartland.com/contact/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Heartland Dental | America\u2019s Leading Dental Support Organization", "textOnly": "Heartland Dental | America\u2019s Leading Dental Support Organization Skip to content Selling Your Practice Selling Your Practice Why Heartland How It Works Path to Long-Term Wealth Support Model Support Model Support Services Super GP Continuing Education About About Our Story Giving Back Resources Resources Blogs and Articles In the News Masters &#038; Mentors Podcast Videos Careers Get in Touch Selling Your Practice Selling Your Practice Why Heartland How It Works Path to Long-Term Wealth Support Model Support Model Support Services Super GP Continuing Education About About Our Story Giving Back Resources Resources Blogs and Articles In the News Masters &#038; Mentors Podcast Videos Careers Get in Touch Heartland Dental. Where Doctors Come Together To Be the Best They Can Be. This is our calling. As America\u2019s leading dental support organization (DSO), Heartland Dental is committed to elevating dentistry in communities across the country. For nearly 30 years, we have helped empower supported doctors who share a passion for elevating patient care and personal growth. Sell Your Practice Sell Your Practice Explore Careers Explore Careers Heartland Dental \u2013 Our Story Sorry, your browser doesn&#8217;t support embedded videos. A Dental Support Organization Built&nbsp;by&nbsp;Doctors to&nbsp;Elevate&nbsp;Dentistry Being a dentist is a proud profession. It asks a great deal of the people who choose it. In 1997, Dr. Rick Workman of Effingham, Illinois, founded Heartland Dental as the first dental support organization. His vision was to help doctors lead their practices while having the support and resources to achieve new levels of success. Watch Our Story Sell Your Practice Find out your practice value and how to grow your future. Thinking about selling your practice? Get a practice valuation and explore your transition options with a Heartland Dental practice transition representative. No obligation Completely confidential Get connected to a representative near you 1,900+ Supported Practices 3,100+ Supported Doctors 29 Years of DSO Leadership Tell us about you and your practice. All fields are required. dental practice owners Sell Your Dental Practice to a Team that Supports Growth and Shares Your Values and Vision. Every doctor\u2019s next step looks different. Whether you want to grow, ease your schedule, or plan for transition, Heartland Dental and a supported practice build around your goals. With a transparent process and clear communication, we will work together to design a pathway that elevates patient care, keeps your team thriving, and opens your future to new possibilities. Discover Your Options Explore the Process Start a Conversation SUPPORTING CLINICAL EXCELLENCE Bring the Future of Dentistry to Your Chair. Delivering world-class patient care starts with doing the right things for the right reasons. That&#039;s why as a Heartland Dental supported doctor, you\u2019ll always retain clinical independence to make the care decisions that impact your patients. Only now, with our support model behind you, you\u2019ll have the time and focus to provide more comprehensive care for those who come to see you. At a Heartland Dental supported practice, you\u2019ll have the access and opportunity to learn and offer new procedures, connect with mentors and peers, and utilize the latest technology in your practice. That\u2019s what happens when you become part of something greater. COMPREHENSIVE SUPPORT More Support Means More Opportunity. And More of a Life Outside of Work. When the business side runs smoothly, your whole practice feels it. As a Heartland Dental supported doctor, you remain the leader that your team trusts, while experienced support teams help handle operations, HR, marketing, vendors, payers, and more. Our support model has been proven and is constantly refined, so you can grow your practice with confidence and still have the time and energy to enjoy the life and rewards that you\u2019ve earned. See the Support MASTERS &amp; MENTORS Ideas, Insights,", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://heartland.com/contact/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "brightnow.com", "fetchedAt": "2026-07-30T17:35:17.736953Z", "result": {"base": "https://brightnow.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, WebSite, MedicalBusiness.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Book a Dentist Appointment Near You Online at a Bright Now!\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.brightnow.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6733 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:1-844-400-7645).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Book a Dentist Appointment Near You Online at a Bright Now!", "textOnly": "Book a Dentist Appointment Near You Online at a Bright Now! Skip to main content Call 1-844-400-7645 &#9776; Menu Find an Office Arizona California Colorado Florida Indiana Maryland Ohio Oregon Pennsylvania Virginia Washington Affordable Care Services Clear Aligners Dental Implants General Dentistry Specialty Dentistry Cosmetic Dentistry What to Expect Careers Book Now &times; Close Find an Office Arizona California Colorado Florida Indiana Maryland Ohio Oregon Pennsylvania Virginia Washington Affordable Care Services Clear Aligners Dental Implants General Dentistry Specialty Dentistry Cosmetic Dentistry What to Expect Careers Call 1-844-400-7645 Book Now Your go-to dentist. When you're ready, we're ready! Find an Office Book an Appointment Find Offers Near You Everything you need. Where you need it. Your smile, our experts.\u200b Your best smile is waiting at Bright Now! Dental! From cleanings to crowns, whitening to wisdom teeth, we\u2019ve got you covered with the comprehensive dental services you and your family need for a lifetime of good oral health. At Bright Now! Dental, you and your family can receive all the care you need at one convenient location, saving you time, money, and hassle. We have all the state-of-the-art equipment and technology to perform many procedures in-office as well as the experienced and highly-trained staff to do them, making Bright Now! Dental a truly full-service provider. Emergency Dental Care Dental Implants Teeth Whitening Fillings Curodont\u2122 Dental Bridges Dentures Braces Clear Aligners Veneers Root Canals Wisdom Teeth Removal Crowns Extractions Teeth Cleaning Dental Checkup Chipped Tooth Repair Bright Now! Dental is your affordable, comprehensive dental provider. Good dental health is essential for good overall health, and at Bright Now! Dental, we care about both! We offer all the dental and oral care services you need to get\u2014and keep\u2014your smile healthy. Bright Now! Dental knows that taking care of your teeth takes a team effort. From the doctors to the schedulers, from the front desk to the back billing, we\u2019re all here to make sure your experience is as seamless and stress-free as possible. We also understand that no two situations are the same, so you can be confident that you\u2019ll receive personalized, attentive, and considerate customer service, no matter what your circumstances. What makes Bright Now! Dental different is our commitment to high quality, affordable care for every patient. Not only can we provide greater flexibility in service pricing for our patients, we can also provide the time, technology, and resources our dentists need to deliver the individualized care you deserve in a familiar, convenient, and comfortable setting. Right Here, Right Now Open Evenings & Weekends Here when you need us. Your neighborhood dentist. Bright Now! Dental is here for you no matter when you need us or what you need us for. We know that you\u2019re busy, and sometimes you just can\u2019t fit routine dental care into your 9-5 schedule. That\u2019s why we\u2019re open on weekends and evenings at easy-to-find retail locations right in your neighborhood, so you can get in, get out, and get on with your day without sacrificing quality care*. Read More We also provide exceptional emergency dental care services for dental emergencies and urgent dental injuries. Bright Now! Dental accepts call-ahead and walk-in emergencies and we can see most patients same-day. Bright Now! Dental has more than 90 affiliated locations throughout Arizona, California, Colorado, Florida, Indiana, Maryland, Ohio, Oregon, Pennsylvania, Virginia, and Washington, which means there is one near you! Find your closest one here. *individual office hours vary. Online Booking What to expect at your first visit Your budget. Your choices. Offers/Insurance/OneSmile Dental Plan/Financing Dental Care that fits your budget. At Bright Now! Dental, we\u2019re proud to offer exceptional dental care that fits your schedule and your budget. We believe that getting and keeping ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:1-844-400-7645", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "comfortdental.com", "fetchedAt": "2026-07-30T17:35:26.123281Z", "result": {"base": "https://comfortdental.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, ImageObject, BreadcrumbList, WebSite, Organization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Home - Comfort Dental\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://comfortdental.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "13179 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://comfortdental.com/contact-us/).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Home - Comfort Dental", "textOnly": "Home - Comfort Dental Skip to content New Patient Exam &amp; X-rays $19 \u279e *Full payment due at time of service. Find a Dentist Gold Plan Menu Services General Dentistry Braces Braces What to Expect Braces options Invisible Braces Financing Dental Implants Oral Surgery Comfort Kids Periodontics Endodontics Emergency Dentistry Same-Day Appointments Blogs About About Us Gold Plan Medicaid Care Day Contact Us Connect with Comfort Franchise Opportunities Standard of Care English Spanish Menu Services General Dentistry Braces Braces What to Expect Braces options Invisible Braces Financing Dental Implants Oral Surgery Comfort Kids Periodontics Endodontics Emergency Dentistry Same-Day Appointments Blogs About About Us Gold Plan Medicaid Care Day Contact Us Connect with Comfort Franchise Opportunities Standard of Care English Spanish Find a Dentist Gold Plan I am looking for a All Practice Types Dentist Kids Dentist Oral Surgeon Orthodontist Endodontists GP limited to Oral Surgery near \u00d7 Search get comfortable with Comfort affordable dentistry with compassionate care across dozens of locations. find a location near you learn about our discount plan get comfortable with Comfort affordable dentistry with compassionate care across dozens of locations. find a location near you learn about our discount plan some say it's the season of giving. we say it's the season of saving. book an appointment. find a dentist near you SERVICES General Dentistry Dental Implants Oral Surgery Braces Comfort Kids Endodontics Periodontics SERVICES General Dentistry Dental Implants Oral Surgery Braces Comfort Kids Endodontics Periodontics --> --> --> --> --> --> --> Open Saturdays Evenings and Saturdays appointments available. Why? Because convenience. No Insurance? We have a cost savings membership program. Want to be your own boss? Learn more about becoming a franchise owner. Open Saturdays Evenings and Saturdays appointments available. Why? Because convenience. No Insurance? We have a cost savings membership program. New locations Federal & Jewell, CO Dentist We are happy to announce we have opened a new location. view office South Western, OK Dentist We are happy to announce we have opened a new location. view office Tempe, AZ Dentist We are happy to announce we have opened a new location. view office New locations welcome to Comfort Dental your trusted partner for complete dental care Finding a dentist you can trust with your family's oral health is crucial. At Comfort Dental, our dentists operate on a simple philosophy: exceptional dental care should be accessible, affordable, and delivered with genuine compassion. Our experienced team guides you through every step. comprehensive dentistry for every stage of life We understand that dental needs evolve throughout your lifetime, which is why we offer a complete range of services under one roof. Our general dentistry services form the foundation of good oral health, including preventive care, cleanings, fillings, crowns, and regular checkups that help you avoid more serious problems down the road. When you choose a dentist who prioritizes prevention , you're investing in a healthier future with fewer emergency visits and less invasive treatments. For our youngest patients, Comfort Kids provides specialized pediatric dentistry in an environment designed to make children feel safe and excited about dental care. We know that early experiences with a dentist shape a child's attitude toward oral health for years to come, so our pediatric team uses gentle techniques, positive reinforcement, and kid-friendly language to build confidence and trust. From first teeth to teenage years, we're here to support healthy development and establish excellent oral hygiene habits that last a lifetime. Specialized Care When You Need It Most Sometimes dental health requires more than routine maintenance. Our periodontics services address gum disease and other conditions affecting the supporting structures of your teeth. Healthy gu", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "https://comfortdental.com/contact-us/", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "smilegeneration.com", "fetchedAt": "2026-07-30T17:35:31.306184Z", "result": {"base": "https://smilegeneration.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: BreadcrumbList, Organization, WebPage.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Find Your Local Trusted Dentist | Smile Generation\u00ae\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.smilegeneration.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "6183 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:18007645343).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 69, "title": "Find Your Local Trusted Dentist | Smile Generation\u00ae", "textOnly": "Find Your Local Trusted Dentist | Smile Generation\u00ae All Offices Blogs Providers FAQ Mouth-Body Connection Mouth-Body Connection CATEGORIES Overview Heart Health Brain Health Women&#39;s Health Diabetes Cancer Inflammation RESOURCES ebook Magazine Quiz Blog The Balancing Act Health Uncensored What is the Mouth-Body Connection? Read More Health Uncensored with Dr. Drew Watch Now Dental Services Dental Services POPULAR SERVICES Overview Emergency Dentistry Teeth Cleaning X-Rays/Imaging Fillings Dentures Dental Implants Extractions Wisdom Teeth Removal Teeth Whitening Dental Crowns OralFitnessCheck\u00ae COMMON CONCERNS Cavities Chipped Tooth Crooked Teeth Gingivitis Gum Disease Halitosis Receding Gums Sensitive Teeth TMJ Toothache Yellow Teeth Have questions about treatments or a dental problem? Check out our blog. Read More Connected Care Payment Options Payment Options Overview Insurance Accepted Smile Generation Dental Plan Dental Financing Financing or a Dental Plan? Read More Need Help Understanding your Dental Benefits? Read More 1-800-SMILEGEN FIND A DENTIST beyond dentistry &nbsp; There\u2019s a critical link between oral and overall health. We call this The Mouth-Body Connection\u00ae &nbsp; All Offices Blogs Providers FAQ &nbsp; Watch Smile Generation on Health Uncensored with Dr. Drew Explore conversations and expert insights on whole-body health, wellness, and the connection between oral health and overall health \u2014 featuring trusted clinicians and thought leaders from Smile Generation. WATCH &nbsp; &nbsp; Generations of Smiles: Brain Health Insights &nbsp; Explore the powerful connection between your mind, mouth, and music with expert insights on brain health, confidence, cognitive wellness, and the role oral health plays in helping you thrive. &nbsp; LEARN MORE &nbsp; Smile Generation on Dr. Drew\u2019s Health Uncensored &nbsp; Join Stephen E. Thorne IV, PDS Health CEO, and five Smile Generation-trusted clinicians for honest, expert-led conversations about the Mouth-Body Connection. &nbsp; WATCH NOW &nbsp; Dental and Medical, Working As One &nbsp; Discover how dental and medical expertise come together for your personalized, whole-body care. &nbsp; EXPLORE CONNECTED CARE &nbsp; &nbsp; What does it mean to be a Smile Generation-trusted office? &nbsp; Connecting you with great, local dentists Helping you afford the care you need and want Offering the highest quality of care Serving the communities in which we live and work &nbsp; Find a dentist At Smile Generation, we are all about connecting people with trusted, local dentists. &nbsp; &nbsp; Big Smiles Near You &nbsp; &nbsp; What&#39;s on your mind? &nbsp; HALITOSIS Missing Teeth Toothache CAVITIES Stained Teeth SENSITIVE TEETH TMJ CROOKED TEETH CHIPPED TOOTH RECEDING GUMS Load More &nbsp; &nbsp; Advanced Proven Technology &nbsp; Smile Generation MyChart Smile Generation MyChart is a convenient, secure patient portal that makes it easy to manage your care at any Smile Generation-trusted office. You can share your health history, schedule and manage appointments, fill out paperwork, pay your bill and even message your care team all in one place. WHOLE-BODY HEALTH MADE EASY Same-Day Dentistry From single-visit crowns to CBCT scans for more accurate diagnoses, to laser treatments for gum disease, same-day dentistry uses proven technology for more efficient, effective care. SMILE FOR OUR CAMERAS Saliva Screening Saliva screening analyzes saliva to identify biomarkers for systemic conditions, cancers, and other conditions. This test is sent to an external lab and measures 11 types of bacteria known to cause gum disease and aggravate or even cause systemic medical conditions. HOW IT WORKS Innovation An amazing array of proven technologies is helping dentistry take a greater role in improving both oral and overall health. ALWAYS A STEP AHEAD &nbsp; Payments Made Painless &nbsp; Insurance Smile-Generation trusted offices accept many dental insurance plans and will file claims on your behalf, sav", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:18007645343", "sept15": {"behindCloudflare": true, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "posture-choice", "detail": "Served through Cloudflare with no ad code detected, currently readable by AI. The September 15 default targets ad-bearing pages, so no automatic flip is expected here. But this site\u2019s AI access is now a Cloudflare dashboard setting \u2014 block, allow, or charge under Pay Per Use, where publishers get paid when content is actually used in an AI answer. Confirm the current posture is a decision, not a default."}}}
{"corpus": "dental", "domain": "affordabledentures.com", "fetchedAt": "2026-07-30T17:35:37.479858Z", "result": {"base": "https://affordabledentures.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "22 valid JSON-LD block(s) -- types: MedicalOrganization,Dentist, WebPage, VideoObject, ItemList, HowTo, WebSite, SiteNavigationElement, Dentist.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Affordable Dentures &amp; Implants | Dentists &amp; Dental C\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.affordabledentures.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "17329 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:18003368873).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 72, "title": "Affordable Dentures &amp; Implants | Dentists &amp; Dental Care", "textOnly": "Affordable Dentures &amp; Implants | Dentists &amp; Dental Care Skip to main content HAVE YOUR BEST SUMMER SMILE YET. Make your benefits count and smile now. \u2192 1-800-DENTURE Find Your Office Blog Our Way The Affordable Way Success Stories Dentures Dentures Overview EconomyPlus Dentures Premium Dentures UltimateFit Dentures Partial Dentures Denture Maintenance Implants Implants Overview SnapSecure Implants FixedSecure Implants All-in-One Solutions Services Services Overview Tooth Extractions Sedation Dentistry Pricing &amp; Payments Pricing &amp; Payments Overview Pricing Insurance Financing Patient Support Patient Support Overview FAQs How It Works Getting Used to Dentures Special Needs Patients Health Care Tips New Patient Forms Your Nearest Office Loading... Loading... Change Get started Get started Your Nearest Office Loading... Loading... Change We understand. We understand. Pause rotating text A \u201cfixed budget\u201d really is fixed . Just answer a few quick questions to craft an affordable treatment strategy\u2014and see how easy it is to take the first step. Start the Treatment Finder Book appointment Give us a call We understand. Pause rotating text A \u201cfixed budget\u201d really is fixed . Just answer a few quick questions to craft an affordable treatment strategy\u2014and see how easy it is to take the first step. Start the Treatment Finder Book appointment Give us a call Dentures and dental implants are our passion. That\u2019s why you get affordable treatment and compassionate care. Learn more about who we are 360+ locations nationwide 8M+ patients served 50+ years of experience 4.7 / 5 stars on Google Learn more about who we are Save with Affordable Dentures &amp; Implants At Affordable Dentures &amp; Implants clinics, dentures are crafted in the clinic\u2019s local lab by teams who live and work in your community. By making locally, the team is able to keep costs down\u2014often saving you more than 30% compared to traditional dental services. Service Initial Exam and X-rays FREE* Extractions $46/month* (avg. 8 teeth) Economy Dentures Package $33/month* (single arch upper or lower denture) Dental Implants $56/month* (per implant) * Costs are based on average pricing for all Affordable Dentures &amp; Implants locations nationwide. Monthly payment amounts are determined by 3rd party lenders whose terms are set for qualified buyers and assume a down payment of $0 with equal payments over 24 months and an annual percentage rate of 0%. Free exam &amp; X-ray is only for new denture or implant patients and is only available at participating offices\u2014confirm with your local office for availability. Initial exam costs $1 in Missouri &amp; Illinois. Affordable Dentures &amp; Implant patients have reported savings of over 30% on average. View all services pricing The best price. Guaranteed. Our Best Price Guarantee means our dental team in our practice will not be beaten on price. Bring in a treatment plan from any competitor and we will match the total treatment plan for comparable services. View pricing for your local office Treatment plan must be from a licensed dentist within the last six months and for comparable services, materials, and clinical scope. See Full Details . The best price. Guaranteed. Our Best Price Guarantee means our dental team in our practice will not be beaten on price. Bring in a treatment plan from any competitor and we will match the total treatment plan for comparable services. View pricing for your local office Treatment plan must be from a licensed dentist within the last six months and for comparable services, materials, and clinical scope. See Full Details . Play Restoring confidence by restoring smiles. That\u2019s what we do. Whether it\u2019s hiding from friends and family, taking favorite foods off the menu, or a fear of getting started, every new smile has a backstory. Debbie, Marvin, and Barrett all found the courage to change their lives\u2014with our help. Here\u2019s a look at their stories. Play Watch More Patients Share Their Journey Play ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:18003368873", "sept15": {"behindCloudflare": false, "adsDetected": true, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "westerndental.com", "fetchedAt": "2026-07-30T17:35:43.419710Z", "result": {"base": "https://westerndental.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "2 valid JSON-LD block(s) -- types: WebPage, BreadcrumbList, WebSite, MedicalOrganization.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Affordable, family-focused dental care | Western Dental\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.westerndental.com/en-us/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "5211 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:8005793783).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Affordable, family-focused dental care | Western Dental", "textOnly": "Affordable, family-focused dental care | Western Dental Affordable, family-focused dental care | Western Dental Skip to content (800) 579-3783 Manage your appointment Pay bill Espa\u00f1ol Dental services Emergency dental care General dentistry Routine Cleanings Kids dentistry Braces &#038; orthodontics Surgery &#038; restoration Sedation &#038; comfort dentistry Gum disease &amp; treatments Cosmetic dentistry Dental products Insurance &#038; financing Medicare Advantage Medi-Cal dental Medicaid Accepted dental insurance &#038; providers\u200b Workers&#8217; Compensation Financing options Current offers Western Dental Plan members Western Dental Plan providers What to expect Your first visit Patient forms Dental anxiety About us\u200b Contact us GRIEVANCE FORM For providers Careers Find a location Arizona Casa Grande Chandler Glendale Mesa Peoria Phoenix Tucson California View All Cities We Serve Bakersfield Fresno Long Beach Los Angeles Modesto Sacramento San Diego San Jose Santa Ana Stockton Find a location (800) 579-3783 Manage your appointment Pay bill Book an appointment Language: Espa\u00f1ol Book an appointment READY, SET, SUMMER! GET YOUR SMILE SUMMER READY! Book Now Affordable, family-focused dental care Dental services for everyone From regular checkups to specialty treatments, Western Dental offers complete care that grows with you. Our services are designed to support healthy smiles at every age. View all services General dentistry We offer checkups, cleanings, and other essential services to help you maintain a healthy smile. View service Kids dentistry Our friendly pediatric teams make visits easy and comfortable, helping kids build healthy habits. View service Braces &#038; orthodontics From braces to aligners, we offer orthodontic treatments to help straighten smiles at any age. View service View all services Healthy smiles start here. Book a new patient exam today. Book an appointment Flexible payment options that fit your needs Western Dental accepts most insurance plans, including Medi-Cal and Medicaid, and we offer a variety of financing options. Whether you\u2019re insured or paying out of pocket, we\u2019ll help you find a solution that makes sense for you and your family. Explore payment options Braces for $108 per month 1 A straighter smile is more affordable than you think. Start with a free consultation and explore flexible payment options. View offer Feel at ease from your very first visit From the moment you arrive, our friendly team is here to help you feel comfortable. We focus on personalized care, clear communication, and a welcoming environment\u2014especially for those who feel nervous about dental visits. Learn about your first visit Healthy smiles, backed by dental expertise Discover what sets Western Dental apart as a trusted leader in safe, reliable, and professional dental care. Learn more about us Experienced dental professionals Every Western Dental location is staffed by state-licensed dentists who are committed to providing high-quality, compassionate care. Board-certified specialists Our team includes board-certified orthodontists and specialists who bring advanced expertise to every treatment plan. Safety and infection control We adhere to strict, accredited safety standards and infection control protocols to protect your health and well-being. Ongoing education and training Our dental professionals receive continuous training in the latest tools, techniques, and technologies to deliver the best possible care. Learn more about us Find a Western Dental near you With over 260 convenient locations, Western Dental makes it easy to find trusted care close to home. Wherever you are, a healthier smile is just around the corner. Find a location Hear what\u2019s making our patients smile Gloria helped transfer my DHMO plan and book fast care for me and my son. The dentist explained everything clearly, and we even got a free orthodontic consultation. Michelle Lucas I was so embarrassed to smile before. Rochelle at Western Dental ga", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:8005793783", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "sonravahealth.com", "fetchedAt": "2026-07-30T17:35:48.585885Z", "result": {"base": "https://sonravahealth.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "3 valid JSON-LD block(s) -- types: WebSite, Organization, LocalBusiness.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": false, "detail": "title: \"Sonrava Health\" - description: MISSING", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.sonrava.com).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": false, "detail": "Missing -- link previews and some AI browsing tools fall back to guessing.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "961 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (https://sonravahealth.com/contact-us).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 65, "title": "Sonrava Health", "textOnly": "Sonrava Health 0 Skip to Content Company About Sonrava Health Brands News Leadership Contact Us Affiliate with Us Open Menu Close Menu Company About Sonrava Health Brands News Leadership Contact Us Affiliate with Us Open Menu Close Menu Folder: Company Back About Sonrava Health Brands News Leadership Contact Us Affiliate with Us Expanding access to high quality, affordable care in communities coast to coast and border to border. Sonrava Health provides high quality care in nearly 600 affiliated offices in 21 states coast to coast from California to Florida, and border to border from Michigan to Texas. Our Affiliated Brands Western Dental &amp; Orthodontics Learn More Western Dental &amp; Orthodontics Kids Learn More Brident Dental &amp; Orthodontics Learn More DentalWorks Learn More Perfect Teeth Learn More Vital Smiles Learn More &nbsp;Sonrava Health, 530 S. Main St., Orange, CA 92868 Privacy Policy ADA Policy Statement Terms of Use &nbsp; &nbsp;", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "https://sonravahealth.com/contact-us", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "clearchoice.com", "fetchedAt": "2026-07-30T17:35:57.397450Z", "result": {"base": "https://clearchoice.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": true, "detail": "Found.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Dental Implant Centers | Tooth Replacement Solutions | Clear\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.clearchoice.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "10005 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:8884863443).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 54, "title": "Dental Implant Centers | Tooth Replacement Solutions | ClearChoice", "textOnly": "Dental Implant Centers | Tooth Replacement Solutions | ClearChoice Find a center (888) 486-3443 Schedule now The ClearChoice difference Dental implant solutions Cost &amp; financing options About dental implants Why ClearChoice? Meet our doctors Lifetime warranty Patient reviews &amp; testimonials FAQs Read our blog Free consultation With you at every step Celebrating smiles See patient transformations Custom dental implants Every shape. Every shade. Patient Solutions All solutions Full mouth dental implants Implant dentures Single-tooth implants Custom dental implants Pre-implant preparation Free consultation Smile design &amp; planning Bone grafting Periodontal (gum) treatment Oral surgery &amp; extractions Level up your smile Meet Endura Elite Ready to upgrade? Go from dentures to implants Get a First Look Preview your smile. Working with your budget Cost of dental implants Affording dental implants Payment plans &amp; financing Insurance &amp; coverage Full mouth implants Clear pricing. No surprises. Implant dentures Secure fit, for your budget Single tooth implants Lasting solution, real value All patient resources Am I a candidate? Your experience What are dental implants? Dentures vs implants Benefits of dental implants When to get dental implants Implant resources Your questions, answered Expert tips &amp; articles Read our blog Your healing guide Steps to smooth recovery Call us Enjoy 20% off your new smile Save on fixed full arch dental implants with code SUMMERSMILE20, through Aug. 31. 1 Schedule free consultation (888)-486-3443 Marta , ClearChoice Woburn \u00b9Valid at participating centers. Discount off total treatment cost for patients who begin treatment on or after 6/18/2026. Not valid on prosthodontic appointments or for previous or ongoing work. Cannot be combined with other discounts. Discount taken off usual and customary fees. Must start treatment within 5 business days of office consultation. Mention SUMMERSMILE20 at your consultation to receive a discount. Offer ends 8/31/2026. THE PATIENT OR ANY OTHER PERSON RESPONSIBLE FOR PAYMENT HAS A RIGHT TO REFUSE TO PAY, CANCEL PAYMENT, OR BE REIMBURSED FOR PAYMENT FOR ANY OTHER SERVICE, EXAMINATION, OR TREATMENT THAT IS PERFORMED AS A RESULT OF AND WITHIN 72 HOURS OF RESPONDING TO THE ADVERTISEMENT FOR THE FREE, DISCOUNTED FEE, OR REDUCED FEE SERVICE, EXAMINATION OR TREATMENT. A smile that\u2019s here to stay ClearChoice dental specialists work together under one roof to plan, place and perfect your smile, with advanced implant personalization. Now and for life. Proven to last 20+ years of research. Same-day implant expertise. From simple to complex cases, every zirconia smile is backed by our Lifetime Warranty.\u00b2 See what&#x27;s covered Designed to fit From the start, your smile is shaped around you, then refined as you go, so you can eat freely, laugh openly and feel like you. Customize your smile Made possible Permanent implant solutions for every starting point. Clear pricing and trusted lending partners make it easier to get started and finance your care. Explore cost options Loading quiz\u2026 Insurance only takes you so far. We take you further. At ClearChoice, if you have dental insurance, you can save more on your fixed full double arch implants than most plans offer. Show us your insurance card and save $5,000 \u00b2 . Explore Insurance Assurance \u00b2Valid at participating centers only. Discount off fixed full arch dental implants, $2,500 per arch. Maximum discount $5,000. Not valid on prosthodontic appointments or for previous or ongoing work. Cannot be combined with other discounts. Discount taken off usual and customary fees. Contact your local center and show proof of insurance to claim your discount. ClearChoice is not affiliated with any insurance provider, and does not process insurance directly with insurance providers. If you choose to file a claim for reimbursement, you are solely responsible for all communication with your insurance company. The ClearChoice Lifetim", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:8884863443", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "mykoolsmiles.com", "fetchedAt": "2026-07-30T17:36:03.670897Z", "result": {"base": "https://mykoolsmiles.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "No robots.txt found (defaults open, but unmanaged).", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": true, "detail": "1 valid JSON-LD block(s) -- types: WebPage, WebSite.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Kool Smiles Dentistry for Children &amp; Kids Dentists at Ko\u2026\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.mykoolsmiles.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "3068 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:877-200-2064).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 67, "title": "Kool Smiles Dentistry for Children &amp; Kids Dentists at Kool Smiles Kids Club", "textOnly": "Kool Smiles Dentistry for Children &amp; Kids Dentists at Kool Smiles Kids Club Cookie Policy We use \u201ccookies\u201d on this site. A cookie is a piece of data stored on a site visitor\u2019s hard drive to help us improve your access to our site and identify repeat visitors to our site. Privacy Policy Agree & Close skip to main content Find A Location Schedule Appointment Call Us 877-200-2064 Kid\u2019s Club About Kid&#8217;s Club Activities Dental ABC\u2019s Services All Services General Dentistry Children\u2019s Dentistry Orthodontics Oral Surgery About Community Dental Lesson Plans Locations The power of a winning smile! For the Smiles of the Future Join Kid\u2019s Club Schedule Appointment Find a Dentist! Use my current location Search for a dentist &rarr; Welcome to the Kool Smiles Kids Club. A place designed for the smiles of the future, not only to promote oral health, but to encourage education, compassion and involvement in the community. Together with our network of partner dentists, we\u2019re working to make this a reality. Dental ABC's Dental ABC\u2019s are your resource for all things dental. Find articles about everything you need to know about your child\u2019s dental care. Learn More Services Whether its General Dentistry, Orthodontics or Oral Surgery, our Kool Smiles Kids Club partners have your family\u2019s dental needs covered! Our partners number one priority is to give your child a winning smile. Learn More Medicaid Our Kool Smiles Kids Club partners happily accept Medicaid in their offices! Insurance Kool Smile Kids Club partners are In-Network providers for most commercial insurance companies. JOIN KID&#8217;S CLUB Earn cool games, perks &amp; awesome rewards. Our Kids Club is designed For the Smiles of the Future, not only to promote oral health, but to encourage education, compassion and involvement in the community. Learn More Dental ABC's Resources FAQs to Help You Choose Dental Insurance Read Full Article > Handling Small Cavities: the Kool Smiles Approach Read Full Article > Kool Smiles FAQs Read Full Article > Oral Health FAQs Read Full Article > Trouble With Cavities? Kool Smiles Can Help! Read Full Article > View Top Articles All Services Our partners are experienced in dental care for kids of all ages, from infants to young adults, we can take care of all your kids\u2019 dental needs. We also see parents and adults too! General Dentistry Children's Dentistry Orthodontics Oral Surgery Find A Kool Smiles Kids Club Official Partner Dentist Use my current location Search for a dentist (City, State, or Zip) &rarr; Subscribe to our Kool Smiles Kids Club mailing list now! Join Kids Club Helpful Links Kid\u2019s Club Dental ABC\u2019s Services Locations Sitemap follow us Kool Smiles is a network of local dental offices dedicated to expanding access to quality dental care for children and families in underserved communities. Kool Smiles official partner dentists provide preventative care, diagnostic imaging and a full range of restorative care supported with electronic health records. \u00a9 2026 Kool Smiles. All Rights Reserved. Privacy Policy Accessibility", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "No matching robots.txt group -- default allow."}], "contactMatch": "tel:877-200-2064", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}
{"corpus": "dental", "domain": "aspendental.com", "fetchedAt": "2026-07-30T17:36:15.094205Z", "result": {"base": "https://aspendental.com", "checks": [{"id": "llms", "name": "llms.txt present", "weight": 18, "pass": false, "detail": "Missing -- agents have no machine-readable index of what this site/business offers.", "fix": "Publish /llms.txt: a concise markdown index of who you are, key pages, and products/services."}, {"id": "llmsfull", "name": "llms-full.txt (expanded content)", "weight": 5, "pass": false, "detail": "Missing (optional but a strong signal).", "fix": "Publish /llms-full.txt with the full markdown content of key pages so agents can read without parsing HTML."}, {"id": "agentsmd", "name": "agents.md guidance", "weight": 5, "pass": false, "detail": "Missing (emerging standard).", "fix": "Publish /agents.md describing how agents should interact: key flows, contact, constraints."}, {"id": "robots", "name": "robots.txt allows AI agents", "weight": 15, "pass": true, "detail": "Open to AI agents.", "fix": "Decide an explicit AI-crawler policy in robots.txt -- allow the retrieval agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) even if you opt training crawlers out."}, {"id": "schema", "name": "Structured data (schema.org)", "weight": 18, "pass": false, "detail": "None found -- business facts are not machine-readable.", "fix": "Add valid JSON-LD for your core entities: Organization, Product+Offer (with price/availability), LocalBusiness/Service, FAQ."}, {"id": "sitemap", "name": "XML sitemap", "weight": 5, "pass": false, "detail": "Missing or invalid.", "fix": "Publish /sitemap.xml and reference it from robots.txt."}, {"id": "meta", "name": "Title + meta description", "weight": 9, "pass": true, "detail": "title: \"Aspen Dental | Find a Dentist Near You for Dental Care\" - description: present", "fix": "Every key page needs an accurate <title> and meta description -- agents use them for routing decisions."}, {"id": "canonical", "name": "Canonical tag", "weight": 2, "pass": true, "detail": "Found (https://www.aspendental.com/).", "fix": "Add <link rel=\"canonical\" href=\"...\"> pointing at the one true URL for this page."}, {"id": "og", "name": "Open Graph basics", "weight": 3, "pass": true, "detail": "og:title and og:description present.", "fix": "Add og:title and og:description meta tags (can mirror your <title>/meta description)."}, {"id": "favicon", "name": "Favicon/app icon", "weight": 2, "pass": true, "detail": "Found.", "fix": "Add a <link rel=\"icon\"> (and ideally an apple-touch-icon)."}, {"id": "nojs", "name": "Readable without JavaScript", "weight": 13, "pass": true, "detail": "397339 chars of server-rendered text.", "fix": "Server-render key content (SSR/prerender), or publish markdown mirrors via llms-full.txt."}, {"id": "contact", "name": "Machine-findable contact/action path", "weight": 5, "pass": true, "detail": "Contact path present (tel:(800) 277-3633).", "fix": "Expose a plain-HTML contact method or action endpoint agents can find without executing JS."}], "score": 49, "title": "Aspen Dental | Find a Dentist Near You for Dental Care", "textOnly": "Aspen Dental | Find a Dentist Near You for Dental Care menu Reason for visit keyboard_arrow_down location_on Atlanta, GA (Cumberland) Schedule local_phone (800) 277-3633 local_phone person_outline arrow_back Locations Atlanta, GA (Cumberland) Change Location 4.4 star star star star star_border ( 801 ) 2937 Cobb Parkway SE STE 100 , Atlanta , GA 30339 \u2022 10.1 mi (770) 240-0328 Open \u2022 Closes at 5:30pm directions Directions phone Call info More Info Schedule Dental Services Dentures chevron_right Dental implants chevron_right General dentistry chevron_right Clear aligners chevron_right Financing, insurance, &amp; offers chevron_right About chevron_right Explore dentures Denture types chevron_right Compare dentures Denture services chevron_right Denture costs Denture affordability Denture resources Denture reviews Dentures guidance Explore dental implants Implant types chevron_right Implant benefits Implants costs Implant affordability Implant resources Implant reviews Dental implant guidance Explore general dentistry Emergency &amp; preventative chevron_right Restorative &amp; cosmetic chevron_right Cost &amp; resources chevron_right Explore Motto clear aligners Motto pricing How Motto works Clear aligners technology Motto expert care Is Motto right for you? Take our quiz Results &amp; reviews Financing Insurance Aspen Dental Savings Plan Insurance partners Implant insurance FSA &amp; HSA Pricing Offers &amp; savings Locations Patient resource center Dental care blogs Patient reviews Frequently asked questions Careers About us Contact us Implant dentures Full dentures Partial dentures Replacement dentures Denture repair &amp; reline Implant dentures Fixed full arch implants Implant-supported bridge Single tooth implants Explore all implant types Emergency care Dental checkup Root canal Tooth extraction Dental bridges Dental crowns Tooth filling Teeth whitening Periodontal disease treatment General dentistry costs General dentistry resources Same day appointments. Dentures or implants? Dentures for every budget Save big on implants. See your new smile Save more with You&#x27;re busy. 99% approval rate \u200b search \u200b Atlanta, GA (Cumberland) 4.4 star star star star star_border ( 801 ) 2937 Cobb Parkway SE STE 100 , Atlanta , GA 30339 \u2022 10.1 mi (770) 240-0328 Open \u2022 Closes at 5:30pm directions Directions phone Call info More Info Schedule Dunwoody, GA 4.0 star star star star_half star_border ( 651 ) 4718 Ashford Dunwoody Rd St400 , Dunwoody , GA 30338 \u2022 13.0 mi Open \u2022 Closes at 5:00pm Austell, GA 4.1 star star star star star_border ( 1131 ) 1680 E West Connector Ste 112 , Austell , GA 30106 \u2022 14.5 mi Open \u2022 Closes at 5:30pm Stockbridge, GA 4.5 star star star star star_half ( 982 ) 3515 Highway 138 , Stockbridge , GA 30281 \u2022 15.1 mi Open \u2022 Closes at 5:30pm Norcross, GA 4.2 star star star star star_border ( 943 ) 6141 Peachtree Pkwy , Norcross , GA 30092 \u2022 16.5 mi Open \u2022 Closes at 7:00pm Stone Mountain, GA 5181 Stone Mountain Hwy Suite 200 , Stone Mountain , GA 30087 \u2022 16.8 mi No working hours available \u2022 Fayetteville, GA 4.4 star star star star star_border ( 1052 ) 1135 Highway 85 North, STE A , Fayetteville , GA 30214 \u2022 19.3 mi Open \u2022 Closes at 5:00pm Douglasville, GA 4.3 star star star star star_border ( 1756 ) 6351 Douglas Blvd , Douglasville , GA 30135 \u2022 20.3 mi Open \u2022 Closes at 5:30pm Kennesaw, GA 4.0 star star star star star_border ( 1087 ) 1955 N Cobb Pkwy NW, Ste 110 , Kennesaw , GA 30152 \u2022 20.9 mi Open \u2022 Closes at 5:00pm Conyers, GA 4.2 star star star star star_border ( 1578 ) 1450 Highway 138 SE , Conyers , GA 30013 \u2022 22.7 mi Open \u2022 Closes at 5:00pm Hiram, GA 4.1 star star star star star_border ( 1241 ) 4373 Jimmy Lee Smith Pkwy 105 , Hiram , GA 30141 \u2022 23.4 mi Open \u2022 Closes at 5:00pm Snellville, GA 4.3 star star star star star_border ( 1534 ) 1790 Scenic Highway , Snellville , GA 30078 \u2022 23.4 mi Open \u2022 Closes at 5:00pm McDonough, GA 4.3 star star star star star_border ( 1637 ) 1430 Highway 20 West , McDonough , GA 30253 \u2022 25.3 ", "bots": [{"name": "GPTBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Google-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ClaudeBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "anthropic-ai", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Meta-ExternalAgent", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Applebot-Extended", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Bytespider", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "CCBot", "category": "training", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "OAI-SearchBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "ChatGPT-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "PerplexityBot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Claude-User", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}, {"name": "Amazonbot", "category": "retrieval", "blocked": false, "evidence": "Group for \"*\" has no root Disallow."}], "contactMatch": "tel:(800) 277-3633", "sept15": {"behindCloudflare": false, "adsDetected": false, "retrievalCurrentlyAllowed": true, "verdict": "unaffected", "detail": "Not served through Cloudflare, so the September 15 default change does not apply here. Your AI access posture stays whatever your robots.txt and server say it is."}}}