PreferiumChapter 04 of 04
Who fetches the pages, and what they feed.
Answer crawlers and training crawlers are not the same thing. The log keeps them apart, so you see which fetch can actually become a citation.
6
agents in the log
1,018
fetches, last 7 days
12
pages in coverage
0
errors, last 7 days
Two kinds of crawlers
Two kinds of crawlers, and only one of them is urgent.
The edge · agent traffic livelast 60 min1,180 calls
Answer crawlersUrgent
OAI-SearchBotnow412
PerplexityBot24 s208
ChatGPT-User41 s143
Claude-User58 s96
Fetches the page while the answer is being written. It arrives right before a citation, so a spike here is the signal that something is about to happen.
Training crawlersLong term
GPTBotnow254
Google-Extended141 s187
ClaudeBot203 s61
Harvests pages for the next model. Important in the long run, but it moves nothing in the answers your clients are given today.
The request, read at the edgebot · path · response
nowOAI-SearchBotGET /bike-service200 · answer
3 sChatGPT-UserGET /contact200 · answer
9 sGoogle-ExtendedGET /blog/sources-in-ai-answers200 · training
17 sClaude-UserGET /guides/what-is-a-citation200 · answer
And how we knowThe category reads crawler visits out of server logs after the fact. We read them off the request itself, at the edge, as it comes in — the same place the HTML is rewritten. It is not a tracking script, because a crawler runs none.
From published to cited
From published, to cited.
Every page passes through four stages. The funnel shows how many make it all the way, and exactly where the rest drop off.
Published22 · 100%
Discovered19 · 86%−3 never crawled
Answer-crawled14 · 64%−5 no answer crawl
Cited9 · 41%−5 not cited
Median published → cited
8 days
From the page going live to its first citation.
Share cited
41%
9 of 22 published pages are a source in an answer today.
Biggest leak
5 pages
Fetched by training bots, never by an answer crawler.
The engine’s job
Close it
The findings go straight into the loop — find, fix, deploy.
Want to see it on a client site?
Pick one of your client sites. We run it through the health checks, show you what the engine would fix first, and walk you through the panel your clients would see under your brand.
Agency agreementNo lock-in on your dataEvery change exportablePricing on the pricing page