Skip to content
PreferiumJoin the waitlist

Chapter 04 of 05

See which AI crawlers reach client pages.

Read a log of crawler visits to each page. See which requests collect pages for search, fetch content for an AI answer or gather training data.

See crawler traffic
A visit is not a citation
Understand crawler access
  1. Crawler visits
  2. Page responds
  3. Visit is logged
Platform facts
types: search, answers and training
3
tracking scripts needed
0
which crawler fetched what, and when
Per page
logged as requests reach the site
Edge

Why AI crawlers visit.

Three request types. Each tells you something different.

  1. Search crawlers

    Pages collected for search indexes used by AI services.

    A successful fetch does not confirm indexing

    Details and how to use it

    Search crawlers retrieve pages for search indexes used by AI services. The request log shows which pages they accessed and the response they received.

    Can the client’s content be discovered?

    Investigate important pages with little or no crawler activity.

  2. Answer-time retrieval

    A page fetched while an AI service answers a question.

    Retrieval and citation are separate observations

    Details and how to use it

    Some AI services fetch a page during a user interaction. This shows retrieval activity; it does not prove that the final answer cited the page.

    Was a page fetched while answering a question?

    Check citation tracking to see whether an answer also linked to the page.

  3. Training crawlers

    Web content collected for model development.

    A visit does not confirm inclusion in training data

    Details and how to use it

    Training crawlers collect web content for model development. Their visits are classified separately from search and answer-time requests.

    Which requests are for model development?

    Keep training requests separate from search and answer-time activity.

Inspect a crawler request.

See the agent, requested page, purpose and server response in one log.

Example request logSample
  1. Time09:42
    AgentOAI-SearchBot
    Page requestedGET /bike-service
    PurposeSearch indexing
    ResponseHTTP 200
  2. Time09:39
    AgentPerplexity-User
    Page requestedGET /pricing
    PurposeAnswer-time fetch
    ResponseHTTP 200
  3. Time09:37
    AgentGPTBot
    Page requestedGET /about
    PurposeModel training
    ResponseHTTP 200

What this log confirms

Page access, recorded without a browser tracking script.

What a request cannot confirm

Indexing, AI citations and use in training need separate evidence. Each request type above explains its limit.

See how citations are measured

Published, requested or cited?

Track delivery, crawler access and citations separately. A successful request does not prove a citation.

Sample 22 pages from Northwind Cycles, a fictional bike shop.

Published pages22 · 100%
All pages in this example
Any crawler request19 · 86%
3 pages with no observed request
Search or answer-time request14 · 64%
5 pages seen only by training crawlers
Cited in tracked answers9 · 41%
5 fetched pages not cited in this sample

Each bar compares one signal with the same 22 published pages. Requests and citations are measured separately.

How to read this sample
Time to first citation
8 days
Median for the nine cited pages in this example.
Share cited
41%
9 of the 22 pages appear as sources in the tracked answers.
Training-only requests
5 pages
No search or answer-time request was observed for these pages.
Next action
Check access
Review crawler access and readable HTML before choosing a supported fix.
Founding partners

Build your agency on Preferium.

Partner registration opens by invitation from the waitlist, and there is no date yet. Read the agency agreement and how partner billing works before you decide.

Join the waitlist Talk to us

Privacy choices

Choose which optional technologies Preferium AS may use. All of them are off until you choose.

Analytics: Google Analytics 4 counts page views. Google may receive the page address, referrer, network address, browser and device details, and online identifiers. Browser storage: _ga, _ga_*: Up to 730 days. Renewed on activity. The browser may shorten the storage period.

Provider: Google Ireland Limited. Google may transfer data to the United States.

See the cookie notice, the privacy notice and theterms.

Necessary technologies Used for site functions

Preferium AS and Cloudflare deliver the site and protect forms against abuse. A local preference remembers if you pause animation. When optional tracking is available, the site can also remember your documented privacy choices.

Consent receipt
Provider: Preferium AS. HttpOnly receipt that documents and retrieves your consent choice. Name in the browser: __Host-preferium_consent. Storage period: The receipt is valid for up to 180 days without rolling renewal.
Local privacy choices and pending rejections
Provider: Preferium AS. Local storage of privacy choices and pending rejections. The entry alone can never allow optional technologies; a valid server receipt is required. Name in the browser: preferium-consent-v2. Storage period: Until the entry is overwritten or the browser site data is cleared. No automatic timed deletion is configured.
Privacy choice synchronization between tabs
Provider: Preferium AS. The latest message that synchronizes privacy choices and pending rejections between tabs. The message can only close optional technologies and trigger a new server check. Name in the browser: preferium-consent-sync-v1. Storage period: Until the entry is overwritten or the browser site data is cleared. No automatic timed deletion is configured.
Motion pause preference
Provider: Preferium AS. Session storage restores the requested accessibility preference between pages in this tab. Nothing is sent to a server. Name in the browser: preferium-motion-paused. Storage period: Until this browser tab session ends.
Cloudflare Turnstile
Provider: Cloudflare. Abuse protection that loads only on forms where Turnstile is necessary. Storage period: Short-lived control value tied to a form submission.
Analytics

Helps us understand how the site is used, when you consent.

Provider: Google Ireland Limited. The data may include the page address, referrer, network address, browser/device, online identifiers and usage events.

_ga, _ga_*
Processes site usage for aggregated analytics after specific consent. Storage period: Up to 730 days. Renewed on activity. The browser may shorten the storage period.

You can withdraw your choice via Privacy choices. That stops further optional loading but does not recall data already sent to Google. We attempt to delete known first-party values; the browser may prevent deletion of third-party values.

How Google uses and is responsible for data · How Google uses information from partner sites · Google privacy policy