Most AI crawlers fetch a page’s HTML and leave without running your analytics script, so your dashboard misses them. The only record is where the request lands, at your CDN’s edge or on your server, and every tool below reads it there. This year Screaming Frog added AI bot verification, and Cloudflare began verifying agents that sign their requests.
What separates the tools is whether they check that a bot is who it says it is, and whether they see an agent that looks like an ordinary browser. Any request can call itself GPTBot, and a tool that believes the name counts it as an AI crawl. On our own domains, nearly half the requests using an AI bot’s name came from addresses its operator rules out.
Empact Partners works inside B2B software marketing teams as a go-to-market consultancy, and we read request logs in our audits for Search Everywhere Optimization, SEO wherever buyers search. Of the tools we reviewed, six made the cut, each best for a different situation. Semrush’s log analyzer went Enterprise-only, and no page we read lists its AI bots.
| Tool | Best for | Checks identity | Price |
|---|---|---|---|
| Cloudflare | Sites on Cloudflare | Labels verified bots; counts go by name | Free |
| Screaming Frog | One-off log audits | If switched on | Free tier |
| Oncrawl | Agents split by purpose | Nine operators | Demo |
| Botify | Large catalogs | Most, not CCBot, Claude, or Bytespider | Demo |
| Conductor | Recrawl checks | Not documented | Demo |
| Known Agents | Hosted sites | Claimed | Free tier |
.png)
1. Cloudflare AI Crawl Control
Best for: sites already on Cloudflare that want to see and control AI crawlers without buying anything.
AI Crawl Control is the AI section of Cloudflare’s dashboard, available on every plan, the free one included. Cloudflare handles each request before your server does, so it sees crawls that never reach your server logs, including the ones it serves from cache. It can also allow or block each crawler there.
Cloudflare is also where our own numbers came from. It labels a request verified when the bot proves who it is, with a signature, a published address list, or reverse DNS. That label and each operator’s own list confirmed an estimated 15,515 genuine requests, most of them from Apple’s crawler.
The knock is the free plan’s crawler view. Cloudflare’s docs say it identifies crawlers by their user agent, the name each request gives itself. On our domains, most of those names were not verified. Check those counts against the verified label, which the analytics API returns on the free plan too.
Pricing
Pros and cons
What Reddit and TrustRadius reviewers say
TrustRadius rating: 9 out of 10 from 570 reviews and ratings, for Cloudflare as a whole, read 27 October 2026.
“We already use Cloudflare and I hadn’t even noticed the bot section, which summarizes all bot traffic and can block specific ones. Super easy and useful, and saved me a lot of time.” — r/webdev, Sep 2025
“That metric counts retrievals by AI crawlers and fetchers at the edge, machines reading your pages to answer questions, and they never show in Umami because analytics scripts only fire in browsers.” — r/SEO, Jul 2026
“I saw Meta-ExternalAgent in Cloudflare’s crawler statistics, but some other suspicious requests looked like ordinary browser traffic, and I don’t want to assume they all came from the same source without evidence.” — r/CloudFlare, Oct 2026
“AI crawlers/bots monitoring can also use a better UX and allow for better drill downs” — Growth engineer at a real estate company, TrustRadius, Jul 2025
Standout feature: AI bot controls
One dialog sorts AI bots into Search, Training, and Agent, and sets each group to allow, block, or block only on pages with ads. Most allow-or-block arguments come down to letting answer engines in and keeping training crawlers out. Here that is one setting instead of a robots.txt rule per crawler.
Verdict: on Cloudflare, start here. It costs nothing and sees requests your server never logs. Trust a count only as far as the verified label backs it. Off Cloudflare, Known Agents is the shorter route if your platform has its plugin or your CDN streams logs.
2. Screaming Frog Log File Analyzer
Best for: a one-off audit of raw server logs on one computer, with AI bots checked against their operators’ lists.
The Screaming Frog Log File Analyzer is a desktop app for Windows, macOS, and Linux, sold separately from the company’s SEO Spider crawler. You give it raw access logs, and it shows which bots requested which URLs, how often, and what status each one got back. Version 7.0, from April 2026, added Bot Verification for AI Bots.
The verification checks each request’s address against a list you point it at. It can also go by reverse DNS, by the network the address sits in, or by fixed ranges, and it files whatever fails as spoofed. On our domains, the operators’ own lists confirmed 14,770 of the 15,515 genuine requests.
The catch is that verification is opt-in and slows the import. Anthropic’s list is the trap: it vouches for the addresses on it but makes no claim to be complete, so a Claude request from an unlisted address could still be genuine. The tool marks those as spoofed regardless, which on our domains would make 6,932 Claude requests look fake.
Pricing
Pros and cons
What Reddit says
Reddit talks about Screaming Frog’s crawler far more than its log tool, and G2, Capterra, and TrustRadius list the crawler with no page for the log tool. The independent comment quoted here predates the tool’s AI features and is about reading Googlebot, the job it was built for.
“If you want to digg deeper, learn to use Screaming Frog Log Analyzer. Compare your crawl and Googlebot’s. That’s where the truth comes to light.” — r/SEO, Oct 2020
Standout feature: Bot Verification for AI Bots
Verification is the reason Screaming Frog is on this list. A log tool that counts names repeats the mistake the logs already contain, and this one can tell you which ChatGPT-User lines came from OpenAI and which only said so.
Verdict: the tool for a one-off audit of logs you can export. Switch verification on, and count a failed Anthropic match as unresolved before you report a number. Skip it if the team wants a dashboard to watch every week.
3. Oncrawl
Best for: SEO teams that want the AI agents in their logs sorted by purpose and named in detail.
Oncrawl is an SEO crawler and log analyzer sold through a demo. Its log monitoring groups each AI agent by purpose: ai search, ai user, or ai training. That split decides what you allow and what you block.
Its supported-bots list is the most detailed of the log analyzers here, down to agents their own operators do not document. It also says how it checks. For nine operators, among them OpenAI, Perplexity, Google, and Apple, it tests each address against the ranges that operator publishes. Everyone else is matched on the user agent alone.
Anthropic, Amazon, Meta, ByteDance, and Ai2 are on its AI list but not among the nine, and their names go unchecked. On our domains, 2,486 requests called themselves Amazonbot from outside Amazon’s list. Another 6,932 called themselves Claude from addresses not on Anthropic’s list, and by its docs Oncrawl counts all of them as real.
Pricing
Pros and cons
What Reddit, G2, and Capterra reviewers say
None of the Oncrawl reviews we found on Reddit, G2, or Capterra mentions its AI reports. They praise the log analysis itself and disagree about the price.
G2 rating: 4.5 out of 5 from 22 reviews, read 27 October 2026.
Capterra rating: 3.8 out of 5 from 4 reviews, read 27 October 2026.
“The biggest impact OnCrawl has had for us is log monitoring and, more broadly, the technical side.” — SEO project manager at a retail enterprise, G2, Jun 2026
“The kicker is their log file analyzer which competitors charge much more for.” — VP of marketing at a pharmaceuticals company, Capterra, Dec 2020
“You should try onCrawl, it’s slightly cheaper and the UI is way faster which is one of the main reason why I switched.” — r/bigseo, Oct 2019
“OnCrawl isn’t really entry. They’re owned by BrightEdge so you’re looking at like 4 figures minimum. Try Screaming Frog instead.” — r/TechSEO, Oct 2023
Standout feature: AI bot hits by bot kind
The overview splits AI hits into user, training, and search, for pages and for resources separately. Its labels show how much of that traffic is an assistant fetching a page because someone asked it something. That share is what most teams weigh before they block anything.
Verdict: the pick for a team that already runs log analysis and wants the purpose split done for it. Skip it if most of your AI requests claim to be Claude or Amazonbot, because those are exactly the ones it takes on trust.
4. Botify
Best for: large catalog sites that want AI bot crawls joined to the pages that earn search traffic.
Botify is an enterprise SEO platform, and LogAnalyzer is the part that reads your server logs, refreshed daily. It names GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, and Meta-ExternalAgent among the AI bots it monitors beside the search crawlers.
Botify’s AI Discovery overview joins two data sets. It sets the pages that earn Google Search Console impressions beside the pages ChatGPT fetched while answering. A catalog team picks a period and sees which ranking pages ChatGPT skipped.
Botify’s own help center names its weak spot. It can only authenticate bots that share their addresses, and it names Bytespider, CCBot, ClaudeBot, and Claude-Web as bots it cannot validate. Common Crawl and Anthropic now publish address lists. On our domains, almost every request calling itself CCBot, 2,468 of 2,544, came from outside Common Crawl’s list.
Pricing
Pros and cons
What Reddit, Capterra, and TrustRadius reviewers say
The Botify reviews we read predate its AI bot reports or skip them. Those about its log analyzer praise it, and most warn that it costs a lot or is best kept for very large sites.
Capterra rating: 4.3 out of 5 from 4 reviews, read 27 October 2026.
TrustRadius rating: 9.2 out of 10 from 8 reviews and ratings, read 27 October 2026.
“Personally love Botify. We have found enormous value in the log file analyzer. I understand they are releasing some more in depth self directed learning/certification as well. As people have noted, it’s technical and takes some getting used to but definitely worth it.” — r/bigseo, Oct 2019
“Very expensive product when using features like log analyzer and keyword research.” — VP at an IT services company, Capterra, Jul 2019
“Botify is great for large websites because the crawler can handle millions of pages and a ton of data. If your site only has a couple thousand or hundred thousand pages, there are probably cheaper tools that can be used to crawl your site and you may not need as much in-depth reporting.” — Marketing manager at a company of 10,001+ employees, TrustRadius, Sep 2021
Standout feature: AI Discovery
For a catalog, the number to watch is the share of pages with Search Console impressions that ChatGPT also fetched while answering. The overview shows it beside three counts: URLs with Search Console impressions, URLs ChatGPT fetched while answering, and URLs that got clicks from ChatGPT.
Verdict: the one to buy if your catalog runs to millions of URLs and you already pay for enterprise SEO. Skip it if all you need is which bots came, because that question is free at the edge.
5. Conductor Monitoring
Best for: content teams that want to see when bots using AI crawlers’ names return to a page after it changes.
Conductor Monitoring watches a site around the clock for changes, and its Log File Analysis adds which bots requested each page. It takes logs only from a CDN stream: a Cloudflare Worker or Logpush, Akamai, CloudFront, or Fastly.
Its strength is the page view. Each URL’s record shows how often each crawler visits and when it last came, with OpenAI and Perplexity beside Google and Bing. After a rewrite, it tells you whether something using an AI crawler’s name has asked for the page again.
Where it falls short is identity. Its FAQ lists each operator’s published addresses, but nothing we read says it checks incoming requests against them. On our domains, only about one in ten requests that used an OpenAI agent’s name came from an address OpenAI lists. Before you buy, ask how Conductor would have counted the other nine.
Pricing
Pros and cons
What Capterra reviewers say
The Conductor reviews we read, on Reddit, G2, Capterra, and TrustRadius, never mention its log file analysis or its AI crawler data. Capterra’s Conductor page also carries reviews of ContentKing, the product Conductor Monitoring grew out of, and those cover its site monitoring rather than its logs.
Capterra rating: 4.7 out of 5 from 118 reviews, for all of Conductor, ContentKing included, read 27 October 2026.
“The possibility to create segments for high priority pages with instant alerts when something changes in these segments prevents high-impact mistakes.” — Global SEO lead in travel and tourism, Capterra, May 2021
“Compared to other website crawlers, Content King is somewhat lacking in the number of different things it can check for.” — Regional head of SEO at a health and fitness company, Capterra, Dec 2021
Standout feature: Search Engine Activity
On each page’s record, the panel shows crawl frequency and the last visit for OpenAI, Perplexity, Google, and Bing, with a visits-per-day chart beside it. Rewrite a page on Monday and the panel shows when OpenAI’s and Perplexity’s crawlers next ask for it, as far as their names can tell you.
Verdict: the right tool if your team rewrites pages and your CDN can stream logs, because it shows, page by page, when each AI crawler’s name last turned up. Check that your plan includes the AI crawler view. Skip it if you need to prove which bots were genuine, since nothing in its documentation says it can.
6. Known Agents
Best for: small teams on hosted platforms that want the widest list of agents, free to start.
Known Agents used to be Dark Visitors, and the old address redirects to it. It began as a public list of AI agents, now sells analytics on top, and tracks more than 2,000 crawlers, scrapers, and agents. It connects through a platform integration, such as its WordPress plugin, or a CDN log stream from Fastly or Akamai.
Known Agents puts browser-using agents in a category of their own, called AI Agent. These are agents that do tasks for a user in a real browser. The company says its Agent Identification API can tell a genuine agent from one using a borrowed name. Oncrawl, Botify, and Conductor name Google’s browsing agent but do not group such agents.
What is missing is the method. The pages we read make that promise without saying how the check works. Test it on your own traffic before you rely on it. The free tier is large enough to try.
Pricing
Pros and cons
Reddit posts about Known Agents or Dark Visitors share its list and say nothing about the product, and its G2 page has no reviews yet, so this entry has no user voices.
Standout feature: Agent Directory
The directory is public and free, and lists every agent it tracks with its operator, its category, and what it is for. It is the quickest way to find out what an unfamiliar name in your logs is before you decide whether to block it.
Verdict: the start for a hosted site with a plugin or a log stream and no Cloudflare. Before you report its numbers, check a few requests yourself and make sure its verified results match.
Start Where Your Requests Already Land
These six do not compete for one job, so there is no winner here. Find your situation and start there.
Whichever you pick, ask what it does with a request that claims to be an AI bot and comes from somewhere else. Between 29 September and 26 October, 48,615 requests to our domains carried an AI bot’s name.
That figure comes from Cloudflare’s sampled analytics, so every request count we give is an estimate. Most of the domains are retired sites that redirect, so the shares describe ours alone. Checked against Cloudflare’s verified label and each operator’s list as it stood on 27 October, the requests fell three ways.
A small crowd sent most of these requests. Just 176 addresses each claimed, without proof, to be bots from two or more companies. Together they sent two thirds of the total, 32,509 requests, and nearly all of the ruled-out ones. Of those addresses, 111 each used 14 different bot names.
No person browses like that, and a report that trusts names files every one of those requests as AI crawling.
When Empact Partners runs this check in a technical audit, you give us read access to your CDN’s analytics or a month of server logs, and we read the records. You get four findings back.
One limit holds for every report here. A logged request means something asked for the page, whatever it got back. It says nothing about whether any answer quoted or linked the page.
Whether the answers then recommend you is separate work. We get a partner’s product discussed in the threads and third-party pages AI engines read, then measure every week which brands ChatGPT, Google AI Mode, and Claude name when a buyer asks. We call that work Generative Engine Optimization.
If you want to know which AI agents really reach your site and what to do about them, pick a time on my calendar, and we can work out together whether we are the right people for it.
