Skip to content
PreferiumJoin the waitlist
Chapter 04 of 05

See which AI crawlers reach your clients’ pages.

Read a log of crawler visits to each page. See which requests collect pages for search, fetch content for an AI answer or gather training data.

Understand crawler access
  1. Crawler visits
  2. Page responds
  3. Visit is logged
Illustrative workflow
3
types: search, answers and training
0
tracking scripts needed
Per page
which crawler fetched what, and when
Edge
logged as requests reach the site
Crawler intelligence

Understand the purpose behind each request.

Start with the request type, then inspect the pages and responses in the log.

Search crawlersCan the client’s content be discovered?

Search crawlers retrieve pages for search indexes used by AI services. The request log shows which pages they accessed and the response they received.

Investigate important pages with little or no crawler activity.

Answer-time retrievalWas a page fetched while answering a question?

Some AI services fetch a page during a user interaction. This shows retrieval activity; it does not prove that the final answer cited the page.

Check citation tracking to see whether an answer also linked to the page.

Training crawlersWhich requests are for model development?

Training crawlers collect web content for model development. Their visits are classified separately from search and answer-time requests.

Keep training requests separate from search and answer-time activity.

Illustrative workflow
Search indexing
Search crawler
Client pageRequest recorded

A successful fetch does not confirm indexing

Answer-time retrieval
AI assistant
Page requestedRequest recorded

Retrieval and citation are separate observations

Training activity
Training crawler
Client pageRequest recorded

A visit does not confirm inclusion in training data

Request log

Inspect crawler activity by page.

See which agent requested a client page, when it happened and what the site returned.

Example request log

Sample
  1. Time09:42
    AgentOAI-SearchBot
    Page requestedGET /bike-service
    PurposeSearch indexing
    ResponseHTTP 200
  2. Time09:39
    AgentPerplexity-User
    Page requestedGET /pricing
    PurposeAnswer-time fetch
    ResponseHTTP 200
  3. Time09:37
    AgentGPTBot
    Page requestedGET /about
    PurposeModel training
    ResponseHTTP 200

What this log confirms

Read at the edge, without a browser tracking script.

A request does not prove

  • Search indexing
  • An AI citation
  • Use in model training
See how citations are measured ↗
Coverage and citations

Published, requested or cited?

A published page, a crawler request and a citation are different signals. Compare each one separately.

Sample 22 pages from Northwind Cycles, a fictional bike shop.

Published pages22 · 100%
All pages in this example
Any crawler request19 · 86%
3 pages with no observed request
Search or answer-time request14 · 64%
5 pages seen only by training crawlers
Cited in tracked answers9 · 41%
5 fetched pages not cited in this sample

Percentages use all 22 published pages. Crawler requests and citations in AI answers are measured separately.

How to read this sample
Time to first citation
8 days
Median for the nine cited pages in this example.
Share cited
41%
9 of the 22 pages appear as sources in the tracked answers.
Training-only requests
5 pages
No search or answer-time request was observed for these pages.
Next action
Check access
Review crawler access and readable HTML before choosing a supported fix.

Want to see it on a client site?

Pick one client site for a walkthrough of the engine and your branded panel.

  1. Check the site

    Review its health checks.

  2. See the first fixes

    Understand what the engine would improve.

  3. Explore your panel

    See the client experience under your brand.

Join the waitlistTalk to us
Agency agreementNo lock-in on your dataEvery change exportableBilling model on the pricing page
  1. Connect a client site
  2. Set the control level
  3. Run under your brand

Privacy choices

Choose which optional technologies Preferium AS may use. All of them are off until you choose.

Analytics: Google Analytics 4 counts page views. Google may receive the page address, referrer, network address, browser and device details, and online identifiers. Browser storage: _ga, _ga_*: Up to 730 days. Renewed on activity. The browser may shorten the storage period.

Provider: Google Ireland Limited. Google may transfer data to the United States.

See the cookie notice, the privacy notice and theterms.

Necessary technologies Used for site functions

Preferium AS and Cloudflare deliver the site and protect forms against abuse. A local preference remembers if you pause animation. When optional tracking is available, the site can also remember your documented privacy choices.

Consent receipt
Provider: Preferium AS. HttpOnly receipt that documents and retrieves your consent choice. Name in the browser: __Host-preferium_consent. Storage period: The receipt is valid for up to 180 days without rolling renewal.
Local privacy choices and pending rejections
Provider: Preferium AS. Local storage of privacy choices and pending rejections. The entry alone can never allow optional technologies; a valid server receipt is required. Name in the browser: preferium-consent-v2. Storage period: Until the entry is overwritten or the browser site data is cleared. No automatic timed deletion is configured.
Privacy choice synchronization between tabs
Provider: Preferium AS. The latest message that synchronizes privacy choices and pending rejections between tabs. The message can only close optional technologies and trigger a new server check. Name in the browser: preferium-consent-sync-v1. Storage period: Until the entry is overwritten or the browser site data is cleared. No automatic timed deletion is configured.
Motion pause preference
Provider: Preferium AS. Session storage restores the requested accessibility preference between pages in this tab. Nothing is sent to a server. Name in the browser: preferium-motion-paused. Storage period: Until this browser tab session ends.
Cloudflare Turnstile
Provider: Cloudflare. Abuse protection that loads only on forms where Turnstile is necessary. Storage period: Short-lived control value tied to a form submission.
Analytics

Helps us understand how the site is used, when you consent.

Provider: Google Ireland Limited. The data may include the page address, referrer, network address, browser/device, online identifiers and usage events.

_ga, _ga_*
Processes site usage for aggregated analytics after specific consent. Storage period: Up to 730 days. Renewed on activity. The browser may shorten the storage period.

You can withdraw your choice via Privacy choices. That stops further optional loading but does not recall data already sent to Google. We attempt to delete known first-party values; the browser may prevent deletion of third-party values.

How Google uses and is responsible for data · How Google uses information from partner sites · Google privacy policy