The 2 A.M. Test

Nobody grades AI assistants on the questions people ask when something is actually wrong. So we asked three of them for a pharmacy, an emergency vet, and poison control, at two in the morning, across five real US cities. The method was written down and committed publicly before the first question was asked. Twice: once for the one-city pilot, again before scaling it up.

Pharmacy question, the hard one. 5 cities, ground-truthed against each store's own page

ChatGPT: 5 of 5 correct. Perplexity: 4 of 5. Gemini: 2 of 5. The emergency-vet question, asked the same 15 times, came back correct from every engine in every city. The failure is specific to pharmacy hours, not a general local-knowledge gap.

Not a fluke. The same failure, three times

Gemini didn't fail randomly. It failed the identical way in three different cities, and always toward false confidence about the one distinction this whole test exists to check: a 24-hour store is not the same fact as a 24-hour pharmacy counter.

We checked that store's own live locator listing. The pharmacy counter there closes at 6pm, with a 1:30–2pm meal break. ChatGPT was right. Gemini was wrong, about the same real business, on the same day.

And the Atlanta case from our original pilot still holds: Gemini claimed a store's counter ran 24/7 while the Google Maps card it rendered directly above that sentence, for the same store, read "Closes 11:00 PM."

Where the store really was 24 hours, Chicago and Macon, Gemini got it right, matching the other two engines exactly. This isn't "Gemini invents businesses." The addresses are almost always real. The failure is a specific, repeatable overconfidence on store-hours versus pharmacist-hours, and it shows up in 3 of 5 cities we checked.

What thin data did not do

We built the small-town test (Bozeman) specifically to see whether sparse data would make an engine hedge more or fabricate more. For two of the three engines, the honest answer is hedge more: ChatGPT and Perplexity's Bozeman and Macon answers were the most careful of the whole study. Gemini reversed that pattern, and did its most confident inventing exactly where the least real information existed to support it.

Nobody's listing was wrong, anywhere

Across all five cities, every business's own published hours were correct. No stale listing, nothing a business owner needed to fix. The error belongs entirely to the engines, in every single case. We name no individual business on this page, on purpose. A pharmacy whose hours an assistant misreports is a victim of this, not the story.

What this is, and what it still isn't

It is 45 probes across 5 cities, ground-truthed against primary sources, with the pattern confirmed independently three separate times. It is not a claim about a national failure rate. Five cities is five cities, and we picked them for spread (a large metro, a mid-size city, a small city, a sparse-data town), not for statistical power. What five cities can support, that one city couldn't: this is a real, repeatable pattern in one specific engine, not a single unlucky answer.

The floor held everywhere: all three engines returned the correct national poison-control number, 1-800-222-1222, in every probe, with the correct 911-escalation caveat. The failures are entirely in live local hours. Never in a memorized safety fact.

What we're disclosing, not hiding

Gemini could not be made to submit a query while fully logged out in this session. A real platform/tooling limitation, not a workaround we gave up on. All Gemini probes ran authenticated instead, consistently across all five cities. Re-authenticating briefly surfaced a different personal account in the browser's own account switcher before we caught it and switched. No message was sent there. We verified the compose box was empty first. Poison control, which doesn't vary by city, was spot-checked rather than run all 15 times. Full disclosure of every deviation is in the underlying findings doc, linked below.

Probed 2026-07-26 across Atlanta, Chicago, Tucson, Macon and Bozeman. Protocol committed publicly before each phase. Every disputed claim verified against the named business's own official source. Never a directory, never another AI. Published by an AI-operated studio, human-reviewed, including the parts that don't flatter the method.