Of the 2,453 sites ChatGPT, Google AI Mode and Claude cited in our Panel’s October answers, 20 tell OpenAI’s search crawler to stay out of the cited page, and 101 shut out its training crawler.1 The robots.txt argument teams have is usually about the second number. It is the one that decides least.
This page counts what AI crawlers and agents can reach, gate by gate, for the marketing team asked whether the engines can read its site at all. The sites come from the Empact Panel, the tool we built to put a fixed set of buyer questions to the engines every week, and from answers collected by Qvery, our sister company.
Empact Partners, the consultancy that builds go-to-market engines inside B2B software teams, runs these checks first in the technical audit of its Search Everywhere Optimization work, which covers every place a buyer searches, AI engines included. Each section says who owns the fix.
Highlights
Read together, they say the cheap checks mostly pass and the expensive failures sit where nobody looks: the edge in front of the site, the help center, the agent’s path through a form. The robots.txt line teams argue about and the llms.txt file vendors sell are the two that decide least.
Almost No Cited Site Shuts Out the Crawler That Fetches Answers
A robots.txt file carries two kinds of rule for the same company’s bots. OpenAI uses OAI-SearchBot to decide which sites appear in ChatGPT’s search answers and GPTBot to collect training data, and says the two settings are independent.6 Anthropic splits ClaudeBot, Claude-SearchBot and Claude-User the same way.7
On 29 October we read the robots.txt of every site cited in two sets of answers, for the exact page that was cited: the 2,453 domains in our Panel’s October answers, and the 7,539 in Qvery’s September answers.1 Of the sites with a readable file, the cited page was closed to:
The rule that keeps a page out of the answers is the one almost nobody writes, and the rule people do write keeps it out of training.
The training rule did not keep sites out of the answers either. ChatGPT cited 57 of the 1,614 sites in our Panel and 384 of the 5,100 in Qvery’s set whose files shut GPTBot out of the page it cited.1
Both sets put the same questions to ChatGPT and Google AI Mode, which allows one comparison. Of the sites AI Mode cited that shut out OAI-SearchBot, ChatGPT also cited 1 of 16 in our set and 7 of 30 in Qvery’s. Where both crawlers were let in, it cited 42.1% and 36.7%.1
Where only GPTBot was shut out, ChatGPT cited 39% of Qvery’s 136 sites, level with the base, and 28.6% of our 35, too few to read.1 The answer rule goes with fewer citations in both sets. The training rule shows no consistent effect.
Qvery’s Quora count puts the answer rule at its plainest. Quora’s robots.txt shuts out OAI-SearchBot, and in Qvery’s September answers to buyer questions in ten industries ChatGPT cited Quora in 0.00% of them, while Google AI Mode, which the file lets in, cited it in 0.44% of 2,521.8
Qvery builds AI agents that measure brand visibility in AI search, and I co-founded it, so read its numbers as ours and discount them accordingly. The September set behind the Quora figures, 7,602 ChatGPT answers and 2,521 from Google AI Mode, is the second population in our run, and its method is published for anyone who wants to check it.
.png)
Robots.txt is not the only road in. Reddit, the site cited most in our Panel’s answers, 1,763 of 24,998 citations,1 carries one rule for every crawler: stay out.9
Google announced access to Reddit’s Data API in February 202410 and OpenAI in May.11 In the Panel, ChatGPT cited Reddit in 11.8% of its answers and Google AI Mode in 34.4%. Claude, which the same file shuts out, cited it in 0 of 200.1
Software vendors’ own sites and a whole network of others look the same:
Our own audits found the same. In the four partner site audits our consultants ran between August and October that read robots.txt, 0 of 4 found an AI crawler shut out by name. What they found instead:14
So the audit that opens a partnership with us reads the answer agents’ lines first, which takes minutes, and spends its time on the layers a robots.txt cannot speak for.
“For audits I’d separate robots.txt policy, actual server/CDN access, raw HTML availability, retrieval evidence in logs, and citation/recommendation evidence in answers.” — r/GEO_optimization, Sep 2026
The answer agent’s line is a visibility setting and the training line is a licensing one, and a team that confuses them either closes itself out of the answers or defends a rule that changes nothing there.
Firewalls Answer to a Crawler’s Name, and the Name Is Often Fake
A robots.txt that says yes is a request to the crawler. What the crawler receives is decided by the edge in front of the site: the CDN that serves the pages, the firewall and the bot protection, and OpenAI’s own guidance says a robots.txt rule alone may not be enough for exactly that reason.15
The names that reach a site are not always who they say. On every domain on our own Cloudflare account, 29 September to 26 October, 48,615 requests named an AI agent:2
Our domains are mostly retired ones that redirect, and they draw exactly this traffic, so the share describes our edge and not yours. What transfers is the mechanism: an edge that trusts names admits the impostors, and an edge that checks addresses refuses anyone testing with a name typed in.
By agent the split is starker. Applebot, which crawls from addresses Apple publishes, was itself on 85.4% of the requests carrying its name. GPTBot was on 8.2%, and ChatGPT-User on 1 of 2,503.2
We ran the typed-name test ourselves. For a random 400 cited pages in each set we sent one request as a browser and one under each of four crawlers’ published names. Where the browser got the page, the edge turned away:1
On 11.5% of the Panel’s sampled pages whose robots.txt let all four crawlers in, at least one of the names was turned away, and on 11.7% of Qvery’s.1 The engines had already cited every one of those pages, so the refusal shows what the edge does with a name from our address, not what the crawler received.
Some cited pages refused even the browser: 5.0% of the Panel’s sample and 7.5% of Qvery’s.1 Qvery’s own fetch of the product pages AI answers used hit a block page on 10.23% of them,16 and Lorikeet found 13 of the 50 brand homepages it tested turned away both its automated browser and a request identifying itself as ChatGPT’s agent.5
One setting moved under many sites in September. On Cloudflare, choosing Block for AI training now stops Googlebot, Bingbot and Applebot too, search included, while Disallow AI Training keeps search open.13
So the edge is read from the logs, filtered by the address lists each operator publishes, which is the check that separates the tools that track AI crawlers from the ones that only count names. In our audits it sits beside the bot settings themselves, because a toggle switched on for training can now close the site to search.
Most Cited Pages Send Their Text Before Any Script Runs
A crawler that gets through still has to find words in what it receives, and the AI crawlers that were measured do not run scripts. Vercel and MERJ watched them across Vercel’s network in late 2024:17
A page whose words appear only after its scripts run is blank to those crawlers. The question that decides a budget is how many cited pages are built that way, and the answer is fewer than the fear suggests.
We loaded the same random 400 cited pages per set twice, as the first response and in a headless browser, and counted how many of the rendered page’s words were already in the first response.1 Of the pages that answered both:
86.1% of the Panel’s compared pages and 74.2% of Qvery’s carried nine-tenths of their words or more before any script ran. 14 of 339 and 20 of 306 carried less than half.1 Structured data that existed only after scripts was rarer still: 1 of 296 pages with JSON-LD in the Panel’s sample, 8 of 247 in Qvery’s.1
Qvery found the same on the product pages ChatGPT and Google AI Mode sent shoppers to: 97.10% had the price somewhere in the first file, and 2.90% showed it only after scripts.16 It fetched each page twice from one laptop, so its figure, like ours, is what a page serves an ordinary request.
The failures sit on particular templates. In our count of 421 developer-tool vendors, 61 of 380 documentation front pages held under 150 words before scripts ran, against 22 of 411 home pages.12 One of the four partner audits found the pricing page built entirely in the browser.14
A rebuild is the answer for the template that fails, usually documentation or pricing, and almost never for the whole site.
An llms.txt File Is a Map Few Engines Open
The llms.txt file is sold as an AI fix, and it is the one check here with a direct count of who reads it. Ahrefs checked the 137,210 domains in its analytics that had traffic in May 2026: 28% published a valid llms.txt, and 97% of those files got no request that month.3
Of the requests that did reach a file, 1.1% came from the bots that fetch for AI search answers, and zero AI-bot requests went to files that were not there: no engine goes looking.3
The file is common on the sites the engines cite. 58.0% of the Panel’s cited hosts serve one and 36.2% of Qvery’s,1 as do 58 of 100 Cloud 100 marketing sites4 and 319 of 411 developer-tool vendors.12
At least 16.7% and 11.3% of those files carry a plugin’s own mark, “Generated by Yoast SEO” and the like, so part of the adoption is a default nobody chose.1 An llms.txt file is a map of the pages that matter, and a map does not help a crawler that cannot read the pages it points to.
The Help Center Is Where the Door Shuts
The hosts that fail are rarely the marketing site. On 12 October we asked for the documentation and the help center of every Forbes Cloud 100 company with a plain request and no JavaScript:4
The same companies serve their documentation to anyone and close the help center, where the answers to how-do-I questions live, on a platform setting nobody chose with AI in mind.
In a partnership the fix list goes to whoever owns each host: the hosting team for the edge, the web developer for a template built in the browser, the help-center admin for the platform’s bot setting.
A senior consultant owns the list and re-checks each host after the change. The visibility that follows is measured in quarters, with the first improvements usually inside three to six months.
.png)
An Agent Finishing a Task Is a Different Number
Crawlers fetch pages. Agents run errands, and the errand has its own denominator. Lorikeet sent more than 1,000 AI agents, each acting for a real person, to the support teams of 50 well-known US consumer brands between 8 and 15 October.5
1 in 2 were stopped before they could ask their question, by a login, an app or a pre-chat form asking for details a new customer’s agent cannot give.5 Those are consumer brands and support channels, so the rate is theirs. The stopping points are the ones any software company puts in front of a demo or a price.
We ran the same kind of agent test on our own site, with three buyer tasks, and the stalls had nothing to do with crawlers:18
None of those stops shows up in a robots.txt, in a log of crawler requests or in a citation count. An agent’s task is measured by running it. The agent also marked two tasks down for gaps that were our site’s, not its own, so the trace decides, not the agent’s verdict.
What Our Run Cannot Settle
Every page in the run was already cited, so it describes what cited pages serve. It cannot say what uncited pages serve, or that any check here caused a citation, and the files were read in late October while the answers ran in September and October.
Readable is not cited. A page that passes every check here is eligible to be quoted, and whether an engine quotes it is a different number. We measure that one in the answers themselves, through the Empact Panel, as part of our Generative Engine Optimization work on getting partners named in AI answers.
So check access in the order a request travels: the answer agents’ robots.txt lines first, then the edge in your own logs by the operators’ addresses, then the first response of your documentation, pricing page and help center. Your logs could prove that order wrong for your site, and nothing on this page argues for a file or a rebuild before it.
If you need to find where AI engines and agents are being turned away from your site, book a call with me, and let’s see whether it is work for us.
Sources
- Empact Partners, access run on the pages AI engines cited: the robots.txt and llms.txt of the 2,453 domains cited in the Empact Panel’s answers to buyer questions in our own category (ChatGPT, Google AI Mode and Claude; 4,200 answers, 4 to 23 October 2026) and of the 7,539 domains cited in Qvery’s September 2026 ten-industry answers (10,123 answers, 21 to 22 September 2026), read for each domain’s most-cited page; edge and first-response checks on a random 400 cited pages per set. Read 29 October 2026.
- Empact Partners, Cloudflare analytics for every zone on its own Cloudflare account, 29 September to 26 October 2026: the 48,615 requests that named an AI agent, checked against Cloudflare’s verification and each operator’s published address lists. Pulled 27 October 2026.
- Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read,” 2026. Read 29 October 2026.
- Empact Partners, documentation count of the Forbes Cloud 100 (2025 edition): 100 marketing sites, 89 documentation addresses and 83 help centers, each asked once with a plain HTTP request and no JavaScript, 12 October 2026.
- Lorikeet, “The AX Index: 50 brands ranked for AI agents,” 2026; method at theaxindex.com. Read 29 October 2026.
- OpenAI, “Overview of OpenAI Crawlers,” 2026. Read 29 October 2026.
- Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?,” 2026. Read 29 October 2026.
- Qvery, “Why ChatGPT Doesn’t Cite Quora, and How Rarely Google AI Mode Does,” 2026; the September 2026 ten-industry research, 7,602 ChatGPT and 2,521 Google AI Mode answers in the US, UK, Canada and Australia. Read 29 October 2026.
- Reddit, robots.txt, as served on 29 October 2026. Read 29 October 2026.
- Google, “An expanded partnership with Reddit,” 2024. Read 29 October 2026.
- Reddit, “Reddit and OpenAI Build Partnership,” 2024. Read 29 October 2026.
- Empact Partners, vendor-site count of the 421 domains in DevTune’s 2026 developer-tools benchmark (411 answered), each asked with a plain HTTP request and no JavaScript, 13 October 2026.
- Cloudflare, “Have it both ways: stay discoverable in search while disallowing AI training,” 2026. Read 29 October 2026.
- Empact Partners, four partner site audits run by our consultants, August to October 2026, each reading the partner’s robots.txt and how its pages render. No partner named.
- OpenAI Help Center, “Advertiser Guidance for Allowing OpenAI Web Crawlers,” 2026. Read 29 October 2026.
- Qvery, “Can AI Engines Read Your Product Pages? Start With What Your Server Sends First,” 2026; 88 makers’ product pages from ChatGPT and Google AI Mode answers, each fetched twice on 19 October 2026. Read 29 October 2026.
- Vercel, “The rise of the AI crawler,” 2024. Read 29 October 2026.
- Empact Partners, agent run on empact.partners: three buyer tasks written before the run, one AI agent with a browser, 28 October 2026.
