Skip to content

How To Check Which AI Bots Really Read Your Site

Pull a week of Cloudflare or host logs, check each AI crawler name against its operator’s addresses, and hand engineering a fix list.

On this page
  1. Pulling a Week of Requests From Cloudflare
  2. Reading a Week From Your Host’s Raw Logs
  3. Turning the Week Into Fixes With Owners
  4. Do AI Crawler Visits Lead to Citations?
  5. Frequently Asked Questions

Empact Newsletter

One email a month on what is working in AI search and SaaS go-to-market.

Check your email for a link to confirm your subscription.

You are already subscribed.

Check your email address and try again.

We could not save your subscription just now; please try again.

Your team has heard that AI engines cannot read your site, or that they read it all day, and someone wants robots.txt changed because of it. That file holds the instructions crawlers read first.

The usual evidence is a dashboard’s GPTBot count, which tallies requests by user agent, the line where a request names its own software. Nothing checks that line, so a scanner hunting for passwords can call itself GPTBot as easily as OpenAI’s crawler does.

We ran the check on twelve of our own domains behind one Cloudflare account, mostly retired sites that redirect visitors. Cloudflare’s sampled counts for 23 to 29 October 2026 came out like this 1:

13,442requests named an AI crawler in the week
6,438of them passed the address check
4,497asked for password or config files, and none passed
2,071named an assistant’s live fetch, and only 2 of them passed
Source: Empact Partners

Just over half the requests wearing an AI crawler’s name failed the check, every hunt for a password file among them. A dashboard counting names would have shown about a thousand times more assistant fetches than the two that passed. AI search work goes by answer engine optimization (AEO) or generative engine optimization (GEO). On the site it starts with one question: which AI crawlers get in, and what do they get?

Empact Partners works on SaaS sites for AI search visibility, and part of it is making sure Claude, ChatGPT and Google AI Mode can reach and read the pages. We would read a week of logs before anyone edits robots.txt.

An engineer can run Steps 1 to 3, and the reading starts at Step 4. To run it yourself, you need five things:

Pulling a Week of Requests From Cloudflare

Step 1: Create a read-only token

Cloudflare is a content delivery network (CDN): it answers every request before your server does, so every crawler that comes passes through it. Open My Profile, then API Tokens, and create a custom token with Analytics Read and Zone Read for your domain, which Cloudflare calls a zone.

If Step 3 answers “does not have permission”, the token lacks Analytics Read on that zone. An account-level token of ours failed that way, and a user token with both zone permissions worked.

Step 2: Find your zone ID

Replace paste-your-token-here with your token and example.com with your domain, then run both lines. The second prints the zone’s identifier (ID), 32 characters long.

Find the zone ID
export CF_API_TOKEN=paste-your-token-here
curl -s -H "Authorization: Bearer $CF_API_TOKEN" "https://api.cloudflare.com/client/v4/zones?name=example.com" | python3 -c "import sys, json; print(json.load(sys.stdin)['result'][0]['id'])"

Step 3: Run the week through the script

The script carries the twenty-seven AI agents eleven companies name on their own pages, as read on 30 October and 13 November. Read the last column: only where an operator publishes its addresses, as for twenty-one of them, can the script test a request’s claim.

Scroll horizontally to see all columns when the table is wider than this page.

Operator, Agents in the logs, What they do, Published addresses
OperatorAgents in the logsWhat they doPublished addresses
OpenAI 2GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBottraining, search, a user’s fetch, ad checksyes, one per agent
Anthropic 3 4ClaudeBot, Claude-SearchBot, Claude-User, Claude-Agenttraining, search, a user’s fetch, a browser it hosts for a useryes, but they prove only that an address is Anthropic’s
Perplexity 5PerplexityBot, Perplexity-Usersearch, a user’s fetchyes
Google 6 7Google-CloudVertexBot, Google-GeminiNotebook, Google-Agenta site owner’s agent build, a user’s fetch, agents acting for a useryes
Apple 8Applebotsearchyes
Meta 9meta-externalagent, meta-webindexer, meta-externalfetcher, meta-externaladstraining, search, a user’s fetch, adsno
Amazon 10Amazonbot, Amzn-SearchBot, Amzn-Usertraining, search, a user’s fetchyes, as web pages
Common Crawl 11CCBotan open archive many models train onyes
DuckDuckGo 12DuckAssistBota live fetch for AI answersyes
Mistral 13MistralAI-User, MistralAI-Index, MistralAI-Traininga user’s fetch, search, trainingyes, except training
ByteDanceBytespidercrawling, no operator pageno

We added Claude-Agent on 13 November, a name our first read missed, so the week’s counts predate it. Google-Extended and Applebot-Extended never appear in a log, by Google’s and Apple’s own account 6 8.

Save the script as ai_bot_week.py in an empty folder. It pulls every request that named an agent or carried Cloudflare’s AI label, checks each against the operator’s published addresses, and writes ai-bot-week.csv without addresses.

ai_bot_week.py
"""One week of AI-agent requests, each checked against the operator's own published address ranges.

Cloudflare:  CF_API_TOKEN=your-token python3 ai_bot_week.py cloudflare ZONE_ID 2026-10-23 2026-10-30
Railway:     python3 ai_bot_week.py railway host-week.jsonl
Writes ai-bot-week.csv (no IP addresses in it) and prints the week by agent.
Agent list read at each operator's own page on 30 October 2026 and re-checked on 13 November 2026.
Re-read them before you trust it.
"""
import csv, datetime, ipaddress, json, os, re, sys, urllib.request
from collections import Counter, defaultdict

G = "https://developers.google.com/static/crawling/ipranges/"
AGENTS = {  # token: (operator, what it does, the operator's address list or None)
    "GPTBot": ("OpenAI", "training", "https://openai.com/gptbot.json"),
    "OAI-SearchBot": ("OpenAI", "search index", "https://openai.com/searchbot.json"),
    "ChatGPT-User": ("OpenAI", "user fetch", "https://openai.com/chatgpt-user.json"),
    "OAI-AdsBot": ("OpenAI", "ads check", "https://openai.com/adsbot.json"),
    "ClaudeBot": ("Anthropic", "training", "https://claude.com/crawling/bots.json"),
    "Claude-User": ("Anthropic", "user fetch", "https://claude.com/crawling/bots.json"),
    "Claude-SearchBot": ("Anthropic", "search index", "https://claude.com/crawling/bots.json"),
    "Claude-Agent": ("Anthropic", "hosted browser", "https://claude.com/.well-known/sail"),
    "PerplexityBot": ("Perplexity", "search index", "https://www.perplexity.ai/perplexitybot.json"),
    "Perplexity-User": ("Perplexity", "user fetch", "https://www.perplexity.ai/perplexity-user.json"),
    "Google-CloudVertexBot": ("Google", "owner-requested crawl", G + "special-crawlers.json"),
    "Google-GeminiNotebook": ("Google", "user fetch", G + "user-triggered-fetchers.json"),
    "Google-Agent": ("Google", "user agent", G + "user-triggered-agents.json"),
    "Applebot": ("Apple", "search index", "https://search.developer.apple.com/applebot.json"),
    "meta-externalagent": ("Meta", "training", None),
    "meta-webindexer": ("Meta", "search index", None),
    "meta-externalfetcher": ("Meta", "user fetch", None),
    "meta-externalads": ("Meta", "ads", None),
    "Amazonbot": ("Amazon", "training", "https://developer.amazon.com/amazonbot/ip-addresses/"),
    "Amzn-SearchBot": ("Amazon", "search index", "https://developer.amazon.com/amazonbot/searchbot-ip-addresses/"),
    "Amzn-User": ("Amazon", "user fetch", "https://developer.amazon.com/amazonbot/live-ip-addresses/"),
    "CCBot": ("Common Crawl", "training", "https://index.commoncrawl.org/ccbot.json"),
    "DuckAssistBot": ("DuckDuckGo", "user fetch", "https://duckduckgo.com/duckassistbot.json"),
    "MistralAI-User": ("Mistral", "user fetch", "https://mistral.ai/mistralai-user-ips.json"),
    "MistralAI-Index": ("Mistral", "search index", "https://mistral.ai/mistralai-index-ips.json"),
    "MistralAI-Training": ("Mistral", "training", None),
    "Bytespider": ("ByteDance", "training", None),
}
INSIDE_ONLY = {"Anthropic"}  # its list proves a request is Anthropic's, never that one is not


def address_lists():
    nets = defaultdict(list)
    for url in {a[2] for a in AGENTS.values() if a[2]}:
        req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
        text = urllib.request.urlopen(req, timeout=60).read().decode("utf-8", "ignore")
        text = re.sub(r"(?is)<script.*?</script>", "", text)  # Amazon's pages print YOUR address in a script
        for m in re.findall(r"[0-9a-fA-F:.]{7,}/\d{1,3}|\b\d{1,3}(?:\.\d{1,3}){3}\b", text):
            try:
                net = ipaddress.ip_network(m, strict=False)
            except ValueError:
                continue
            for op in {a[0] for a in AGENTS.values() if a[2] == url}:
                nets[op].append(net)
    return nets


def verdict(op, ip, cloudflare_label, nets, operators_at_address):
    if cloudflare_label or any(ipaddress.ip_address(ip) in n for n in nets.get(op, [])):
        return "passed"
    if nets.get(op) and op not in INSIDE_ONLY:
        return "failed"
    if operators_at_address > 1:  # one address cannot be two companies' crawler
        return "failed"
    return "unresolved"


def cloudflare(zone, start, end):
    q = """query($z:String!,$a:Time!,$b:Time!,$f:[ZoneHttpRequestsAdaptiveGroupsFilter_InputObject!]){
      viewer{zones(filter:{zoneTag:$z}){httpRequestsAdaptiveGroups(limit:10000,
        filter:{datetime_geq:$a,datetime_lt:$b,OR:$f}){count dimensions{date clientRequestHTTPHost
        clientRequestPath userAgent clientIP edgeResponseStatus verifiedBotCategory}}}}}"""
    f = [{"userAgent_like": f"%{t}%"} for t in AGENTS] + [{"verifiedBotCategory_like": "AI%"}]
    day, rows = datetime.date.fromisoformat(start), []
    while day < datetime.date.fromisoformat(end):
        v = {"z": zone, "a": f"{day}T00:00:00Z", "b": f"{day + datetime.timedelta(days=1)}T00:00:00Z", "f": f}
        req = urllib.request.Request("https://api.cloudflare.com/client/v4/graphql", method="POST",
                                     data=json.dumps({"query": q, "variables": v}).encode(), headers={
                                         "Authorization": "Bearer " + os.environ["CF_API_TOKEN"],
                                         "Content-Type": "application/json"})
        res = json.load(urllib.request.urlopen(req, timeout=120))
        if res.get("errors"):
            sys.exit(res["errors"])
        groups = res["data"]["viewer"]["zones"][0]["httpRequestsAdaptiveGroups"]
        if len(groups) == 10000:
            print(f"{day}: 10,000 groups, the limit; split the day into hours")
        for g in groups:
            d = g["dimensions"]
            rows.append((d["date"], d["clientRequestHTTPHost"], d["clientRequestPath"], d["edgeResponseStatus"],
                         d["userAgent"], d["clientIP"], d["verifiedBotCategory"], g["count"]))
        day += datetime.timedelta(days=1)
    return rows


def railway(path):
    rows = []
    for line in open(path):
        if line.startswith("{"):
            r = json.loads(line)
            rows.append((r["timestamp"][:10], r["host"], r["path"].split("?")[0], r["httpStatus"],
                         r.get("clientUa", ""), r["srcIp"], "", 1))
    return rows


def main():
    rows = cloudflare(*sys.argv[2:5]) if sys.argv[1] == "cloudflare" else railway(sys.argv[2])
    nets, out, week, claims = address_lists(), Counter(), defaultdict(Counter), defaultdict(set)
    for r in rows:
        token = max((t for t in AGENTS if t.lower() in r[4].lower()), key=len, default=None)
        if token:
            claims[r[5]].add(AGENTS[token][0])
    for day, host, path, status, ua, ip, label, n in rows:
        token = max((t for t in AGENTS if t.lower() in ua.lower()), key=len, default=None)
        if token:
            op, kind, _ = AGENTS[token]
        elif label:
            op, kind = "(Cloudflare: " + label + ")", "unnamed"
        else:  # no AI name: keep it only if it came from an operator's own addresses
            op = next((o for o, ns in nets.items() if any(ipaddress.ip_address(ip) in x for x in ns)), None)
            if not op:
                continue
            kind = "unnamed, from " + op + "'s addresses"
        v = verdict(op, ip, label, nets, len(claims[ip]))
        out[(day, host, path, status, token or ua[:60], op, kind, label, len(claims[ip]), v)] += n
        week[token or op][v] += n
        week[token or op][f"status {status}"] += n
    with open("ai-bot-week.csv", "w", newline="") as fh:
        w = csv.writer(fh)
        w.writerow(["day", "host", "path", "status", "agent", "operator", "kind", "cloudflare_label",
                    "operators_at_address", "verdict", "requests"])
        w.writerows(k + (n,) for k, n in sorted(out.items()))
    for agent, c in sorted(week.items(), key=lambda x: -sum(v for k, v in x[1].items() if "status" not in k)):
        total = sum(v for k, v in c.items() if not k.startswith("status"))
        print(f"{agent:22} {total:7}  " + "  ".join(f"{k} {v}" for k, v in sorted(c.items())))


main()

Run it with your zone ID, the window’s first day and the day after its last, soon after the week ends, because it checks old requests against today’s address files:

Run the week
python3 ai_bot_week.py cloudflare ZONE_ID 2026-10-23 2026-10-30

If a day prints “10,000 groups, the limit; split the day into hours”, that day came back short, and splitting it means changing the script’s one-day step to hours. We reran the script as printed on 13 November, on one of our domains for 6 to 12 November: 14 seconds, and 726 of 861 named requests failed 1.

Step 4: Read each name against its address check

The script gives every request one of three outcomes:

Our twelve domains split like this:

Scroll horizontally to see all columns when the table is wider than this page.

Outcome, Requests, Share
OutcomeRequestsShare
Passed the address check6,43848%
Failed it6,90251%
Unresolved1021%
Named an AI agent13,442

Report the two kinds of failure separately. 4,920 requests, 37% of those named, came from outside every range their operator publishes. The other 1,982 named Anthropic or ByteDance crawlers, which no address file can rule out. They failed because one address claimed two companies’ crawlers, and at least one claim must be false 1.

Applebot sent 6,074 named requests and 5,598 passed, most of our passing total. Without Applebot, 840 of 7,368 passed. Name by name:

Share of Each Crawler Name’s Requests That Passed the Address Check

Our own domains through Cloudflare, 23 to 29 October 2026, names with 100 or more requests

View the data
LabelValue
Applebot 6,074 named92.2%
ClaudeBot 794 named44.1%
OAI-SearchBot 744 named31%
Amazonbot 565 named20.7%
GPTBot 577 named15.1%
Bytespider 589 named2.2%
PerplexityBot 483 named0.8%
ChatGPT-User 501 named0.4%
Claude-User 584 named0%
OAI-AdsBot 534 named0%
CCBot 495 named0%
DuckAssistBot 493 named0%
Perplexity-User 493 named0%
Claude-SearchBot 480 named0%
Only Applebot passed for nearly all its requests, and six of the fourteen crawler names never passed once.Source: Empact Partners

The chart shows eleven names from seven companies each landing 480 to 590 requests. Close totals prove nothing, but they sent us to the operators_at_address column, the count of companies one address claimed: 6,867 named requests came from an address claiming two or more.

Step 5: Read what each group asked for and what it got

This table splits the same requests by the path they asked for:

Scroll horizontally to see all columns when the table is wider than this page.

What was asked for, Passed, Failed, Unresolved
What was asked forPassedFailedUnresolved
Pages5,7121,70177
Secret or config files (.env, .git, backups)04,4970
App, admin, login or API paths05080
robots.txt4742924
Home page180540
Sitemap42530
Images, scripts, fonts30601

Of the 4,497 requests that used an AI crawler’s name to ask for password and config files, not one passed the address check.

Where an operator publishes no addresses, the path is the next clue: a request calling itself Bytespider asked for /@fs/home/ubuntu/.aws/credentials, a file no crawler needs. People who run their own servers read these requests the same way:

Tips from Reddit

What self-hosters see in the same requests

The background noise of scanners trying to find exploitable sites usually hit 404s on predefined paths and paths that don't take a lot of processing.

r/selfhosted, u/DeliciousJaffa, October 2026

These are not AI agents, they are spoofers. They search the entire internet for known vulnerabilities.

r/selfhosted, u/Famous-Ambassador-87, October 2026

Then read the status, the code the server sent back. Our domains redirect everyone, so every passing crawler got the 301 a person gets. All 229 requests refused with a 403 had failed, nearly all on app and admin paths of domains with no app.

Our bot settings were all off, and the Free plan’s analytics would not say which rule sent those 403s. The counts are sampled estimates. A rerun moved single rows by one, so round them in a report.

Step 6: Read the agents the script does not name

The script also keeps every request Cloudflare labels as AI, because names typed in by hand fall behind. That added 383 requests, 382 of them under names the script did not carry: PetalBot, Reflectionbot, ShapBot and GoogleOther, a generic crawler Google’s product teams use 6. Each is an allow-or-ignore decision.

The remaining one had an ordinary Chrome-on-Linux user agent and no crawler name. Cloudflare labelled it “AI Assistant”, so the script kept it, where a name filter would have dropped it. This route catches such an agent only when Cloudflare labels it.

Reading a Week From Your Host’s Raw Logs

Cloudflare sees only what it sits in front of. A host whose logs keep each request’s user agent and address feeds the same script, with the same three outcomes.

Step 7: Export the host’s week

On Railway, with its command-line tool signed in, pull the service’s request logs for the week, using your own project ID and service name.

Export the week from Railway
railway logs --http --project YOUR_PROJECT_ID --service web --environment production --since 2026-10-23T00:00:00Z --until 2026-10-30T00:00:00Z --lines 5000 --json > host-week.jsonl

Railway refuses --lines 10000 with “Error in beforeLimit - Invalid input” (still true on 13 November), so pull a busy week in parts. After a midweek redeploy, the earlier days sit with the older deployment.

Step 8: Run the same script on it

The same script reads the exported file:

Run the host week
python3 ai_bot_week.py railway host-week.jsonl

Our host route ran on one of our internal tools: 451 requests that week, none naming an AI agent or coming from an operator’s address. Its robots.txt tells every crawler to stay out, on purpose, and was fetched twice, both times under a desktop browser’s user agent. The log cannot say whether that rule kept AI crawlers away. It suits an internal tool and would hurt a marketing site.

Step 9: Send one AI read and find it in the log

A log with no AI requests may simply miss them, so test it first. Make up a path nobody links to, ask an AI assistant to open it, then pull that path and check it:

Pull one path and check it
railway logs --http --project YOUR_PROJECT_ID --service web --environment production --path /your-made-up-path --since 1h --json > calibration.jsonl
python3 ai_bot_week.py railway calibration.jsonl

We asked Gemini, through Google’s Antigravity command-line tool, to open such a path on our tool. Within a second the log held three requests, none with a crawler name from any operator’s page 1:

Scroll horizontally to see all columns when the table is wider than this page.

Request, User agent, Came from
RequestUser agentCame from
GETGoogle-Jetski-Antigravityan address in Google’s published crawler ranges
HEADGo-http-client/2.0the laptop that asked the question
GETGo-http-client/2.0the laptop that asked the question

None asked for robots.txt first, and Google says its user-triggered fetchers “generally ignore robots.txt rules” 7. On the host route the script keeps a request from an operator’s addresses under any name, so Google’s line passed. The other two came from our laptop under a code library’s name, and nothing marks them as AI, so a log undercounts AI reads.

A log also never shows what a crawler made of the page it got. The playbook below covers that side, so every release ships pages AI crawlers can reach and read.

Turning the Week Into Fixes With Owners

Before a row becomes a fix, read its address check and its path together:

A grid crossing the address check, passed, failed or unresolved, with what the request asked for, a page or a secret file, and the action each pairing calls for.
The address check decides how much a request counts, and the path decides what you do about it.

A crawler name alone never justifies a rule. Our week’s fixes, straight from the sheet:

Scroll horizontally to see all columns when the table is wider than this page.

Finding, Intended?, Owner, Check next week
FindingIntended?OwnerCheck next week
Crawlers that passed got the redirect everyone getsYes, the domains are retirednobodypassed rows still redirect
229 blocks, all on failed requests, mostly app and admin pathsHarmless, though the Free plan would not name the rulenobody, leave itno passed row among the blocks
4,497 named requests for secret files, all failedNothing to fix behind a redirectengineering, on a live sitethose paths answer 404
2,071 “live AI fetches” by name, 2 passedNot a site faultwhoever reports AI trafficAI counts drawn from passed rows, unresolved shown apart
383 AI-labelled requests the script did not nameNot a faultwhoever runs the checkeach name decided and dated in the script
No AI crawler on the hostThe rule is intended, its effect unknownnobodyrobots.txt unchanged

Half the rows say leave it alone, so mark each row intended or a fault before engineering sees it, or engineers chase scanners.

A live marketing site can show rows our retired domains could not:

Rows that need a fix on a live site

  • A search crawler such as OAI-SearchBot passed and got a 403 or a challenge. The CDN owner adds an exception for verified bots
  • A passing crawler got a 200 on a page whose text appears only after scripts run, a rendering job no CDN rule fixes
  • A training crawler nobody chose to allow or block, a policy call first: OpenAI says each of its settings “is independent of the others” 2
  • A robots.txt change rechecked the same day, when OpenAI says about 24 hours and Perplexity up to 24 2 5
  • A secret-file request that gets anything but a 404

We found the rule and rendering rows on four partner sites we reviewed between August and October, reading each robots.txt and what each page serves without scripts. None had blocked an AI crawler by name 14, the fix a bot count prompts, yet each had its own fault.

One robots.txt shut a section to every crawler, and one site had no robots.txt at all. One price loaded only after scripts ran, and one site’s canonical tags (naming a page’s preferred address) pointed at redirects.

My own rule from that week and those reviews:

Check the address before a count goes into a report or a block goes into a rule. Keep the unresolved rows visible instead of rounding them into a total. A rule written against a crawler’s name blocks the real crawler along with every scanner borrowing it.

Vlad ShvetsFounder & CEO @ Empact PartnersVlad Shvets on LinkedIn

Whether crawlers could read what they found is a separate check, and it needs no log access. Give the free GEO Audit below your website’s address, and it reports the robots rules your site sets for AI crawlers and what AI engines read without JavaScript.

Do AI Crawler Visits Lead to Citations?

A clean week shows which AI crawlers reached your pages and what each got back. It cannot show whether ChatGPT, Google AI Mode or Claude names you when a buyer asks, since a crawler can fetch a page daily with no answer quoting it. Citations are measured differently: ask the engines your buyers’ questions and read the sources they give.

We read those sources in the Empact Panel, a tool we built. Every week it puts the questions a partner’s buyers ask to Claude, ChatGPT and Google AI Mode, then saves each answer and the pages behind it.

One partner’s card shows what no log can: which of its own pages answers cite. Over 90 days, answers cited 184 of its pages, and 97 were neither landing pages nor blog posts, so a crawl check of the marketing pages alone misses most of them. Its own pages were 3.8% of all sources cited, and the rest came from other sites.

Empact Panel Citation Ownership card, one partner, 90 days: 3.8% citation ownership, 184 cited pages (ChatGPT 146, Google AI Mode 70, Claude 11), 56 landing pages, 31 blog posts, 97 other pages
Empact Panel example: the split by site section, with 97 of one partner’s 184 cited pages sitting outside its landing pages and blog.Empact Panel

So judge each fix twice, in the logs and in the answers. After any robots.txt or CDN change, wait a full day, re-read the operators’ pages and pull the next week. The answers take longer to move.

With your site, our first look would skip the bot names. It would go to what keeps pages from being read, such as refused search crawlers and pages whose text arrives only after scripts run.

What our crawl-access work covers

  • A week of your Cloudflare or host requests, each AI crawler name checked against its operator’s addresses where they are published
  • Your robots.txt and the text your key pages serve without scripts
  • Each recommendation with its first step
  • If the check should run every week, an AI agent that runs it on your logs, owned by one person on your team

Frequently Asked Questions

If GPTBot visits our site every day, does ChatGPT recommend us?

No. A GPTBot line shows that something using that name fetched a page, and GPTBot is OpenAI’s training crawler, while ChatGPT’s search answers rely on OAI-SearchBot. Even a search crawler that passes the address check only proves the page was available. Whether ChatGPT names you shows up when you ask it your buyers’ questions and read its sources, so keep crawler reads and recommendations on separate lines of any report.

Can we run this check if our host keeps no request logs?

Not for any week already gone, because a request nobody recorded cannot be recovered. On 30 October 2026 Webflow offered no request log, and Vercel’s log drain began recording only on the day it was switched on. Put Cloudflare in front of the site, even on its Free plan, or turn on your host’s log drain, wait seven full days, then run the steps. Before you trust a zero from a new log, send one AI read to a made-up path and confirm the log caught it.

Should we block requests that use an AI crawler’s name from the wrong address?

No, not by the name, because a scanner switches names faster than anyone updates a rule, and a rule against GPTBot also stops the real GPTBot. Close the paths those requests hunt for instead, so .env files, .git folders and backup archives answer 404 to everyone. If an engineer still wants a block, base it on the address check and the path together, then confirm next week that no passing request landed inside it.

Is it safe to let every AI crawler into a SaaS marketing site?

Decide each kind on its own. A site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, OpenAI says, though it can still appear as a navigational link. Training crawlers such as GPTBot are a call for leadership and legal. Fetchers acting for a person may skip robots.txt, by Google’s, Meta’s and Amazon’s account. A sensible default is search crawlers allowed and training decided in writing.

Does a 200 response mean the crawler read our page?

A 200 means the server answered. If your price or product description is drawn by scripts after the page loads, a crawler that runs no scripts got an empty frame with that 200. Open the page source, not the rendered page, and search it for the sentence you want quoted. If it is missing, a crawler that runs no scripts never received it. That is a rendering fix for engineering, and no robots.txt line changes it.

How long after a robots.txt change should we check the logs again?

At least a full day, and preferably the whole following week. OpenAI says a robots.txt change can take about 24 hours to reach its search systems, and Perplexity says up to 24 hours for its settings. A same-afternoon check shows crawlers still acting on the old file and sends the team after a fix that has not had time to land. A CDN rule applies from the next request, but crawlers come back on their own schedule, so judge it on a week too.

Why does Google-Extended never show up in our logs?

Because it has no request string. Google says it crawls with its existing user agent strings and reads Google-Extended only as a robots.txt control. Its log count is always zero, and the token does not affect inclusion in Google Search. To see Google, look for Googlebot and the fetchers on Google’s address files: the host route checks those by address under any name, while the Cloudflare route sees only listed names and Cloudflare’s AI label.

Which team changes what once the fixes are written up?

Whoever runs Cloudflare owns the rules and any exception for verified bots. The web lead owns robots.txt and the page templates, and any text that loads after scripts. Whoever reports AI traffic owns the counting rule, so dashboards count only passed requests. Marketing orders the rows by which pages buyers need first, and whoever wrote each row checks it again the following week.

Do the agent names in the script stay current on their own?

The address files do, since the script downloads them fresh each run. The names do not: they sit in the file, and operators keep adding them. OpenAI, Meta, Amazon, Mistral, Perplexity and Google each changed a page or an address file between our 27 and 30 October reads, and on 13 November we added Claude-Agent. Re-read each operator’s page before a run, date your copy of the names, and add any name Cloudflare labels as AI.

Do AI agents that browse like a person show up in our logs at all?

Some do. Anthropic’s hosted browser adds Claude-Agent to an ordinary browser’s user agent and publishes its addresses, so the script can check it. Others send a browser’s user agent with no name. Cloudflare labelled one on our domains that week, and our triggered AI read wrote two lines under a code library’s name from the asking laptop. Read any log’s count as missing some AI reads.

Sources (14)
  1. Empact Partners, "AI crawler requests on twelve of our own Cloudflare zones, 23 to 29 October 2026" (2026)Empact Partners' own data↩1↩2↩3↩4↩5↩6one Railway host week and one calibration fetch, 30 October 2026; the printed script rerun on one zone, 13 November 2026.
  2. OpenAI, "Overview of OpenAI Crawlers" (2026)↩1
  3. Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (2026)↩1
  4. Anthropic, "How do Anthropic-hosted browsers identify themselves when they browse the web on a user’s behalf?" (2026)↩1
  5. Perplexity, "Crawlers" (2026)↩1
  6. Google Search Central, "Google’s common crawlers" (2026)↩1↩2↩3
  7. Google Search Central, "Google User-Triggered Fetchers" (2026)↩1↩2
  8. Apple, "About Applebot" (2026)↩1↩2
  9. Meta, "Web Crawlers" (2026)↩1
  10. Amazon, "Web Search crawlers" (2026)↩1
  11. Common Crawl, "CCBot" (2026)↩1
  12. DuckDuckGo, "Is DuckAssistBot related to DuckDuckGo?" (2026)↩1
  13. Mistral AI, "Mistral crawlers" (2026)↩1
  14. Empact Partners, "four partner site audits run by our consultants, August to October 2026" (2026)Empact Partners' own data↩1each reading the site’s robots.txt and the text its pages serve without scripts. No partner named.

Ready
To Connect?

Let's Partner