Training crawlers
GPTBot, ClaudeBot, Google-Extended
Collect your pages as training material for the next model version. The heaviest readers, and the ones most owners consider blocking.
See which AI scrapers read your website
GPTBot, ClaudeBot and PerplexityBot visit your pages more often than you think. PromptRadar logs every visit from 24 AI crawlers, so you see what they read, what they skip and how often they come back.
Three kinds of AI scrapers, three different signals
AI search indexers
OAI-SearchBot, PerplexityBot, Claude-SearchBot
Fill the index behind AI search answers. These visits are how an answer learns your page exists, so they can turn into citations.
Live answer fetchers
ChatGPT-User, Perplexity-User, Claude-User
Fetch a page while a person is asking a question. A visit here means someone is reading about you inside an AI conversation right now.
How it works
1
Connect your website
Install the WordPress plugin, a middleware for Next.js, Cloudflare, Node.js or Python, or a cron script over your Nginx or Apache logs. It runs on your server, so your visitors get no extra JavaScript and no cookies. A built-in test confirms the install in seconds.
2
AI bots get logged
Only requests from the 24 known AI crawlers are recorded. Normal visitors and classic search bots never enter the log, so the feed is pure AI signal.
3
Read the log
See visits per bot as they happen, the pages each bot reads, which pages AI answers use, the sitemap pages bots skip, broken pages they hit and which bots your robots.txt blocks. About ninety days of history per website.
AI Scrapers FAQ
What are AI scrapers?
AI scrapers are bots that AI companies send to read website pages. Some collect pages to train models, such as GPTBot from OpenAI and ClaudeBot from Anthropic. Some fill the index behind AI search answers, such as PerplexityBot. Some fetch a page live while a person is asking a question.
Which AI scrapers does PromptRadar track?
PromptRadar recognizes 24 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Meta, Amazon, Apple, ByteDance and others. Plain Googlebot is deliberately excluded: it is a classic search crawler and its volume would drown the AI signal.
How does the tracking work?
A small hook on your server reports AI bot requests to PromptRadar. There is a WordPress plugin, a Shopify Hydrogen adapter and snippets for Cloudflare, Vercel, TanStack and any other server. Nothing runs in your visitors' browsers: no JavaScript, no cookies, no effect on page speed. One honest limit: responses served from a CDN cache never reach your server, so those hits cannot be counted.
Should I block AI scrapers?
That is a trade-off, and the log makes it an informed one. Training crawlers use your pages to train models. Search and assistant crawlers are how AI answers find and cite you: block those and you disappear from the answers your buyers read. Once you see which bots read which pages, you can decide per bot instead of guessing.
Is AI scraper tracking free?
Crawler logs are included with paid PromptRadar plans. Every view queries live analytics infrastructure that stores about ninety days of bot visits per website.
Do AI scraper visits mean AI mentions?
A visit is not a mention, but being read comes first: a page an AI crawler never fetched cannot be cited. Pair the crawler log with PromptRadar AI Visibility to see whether the pages bots read turn into recommendations.
Know what AI reads on your website
The crawler log shows every AI bot visit, page by page. Being read comes before being recommended.