Main crawler
Reads sites connected by their owner or agency. Honours robots.txt, including Crawl-delay.
Crawler
PreferiumAIEdgeBot
Crawler and site checks by Preferium AS for Preferium AI Edge. Identify a request in your logs, check its source or stop future visits.
What it is and why it visited
Preferium AI Edge reads a site's current content so it can optimize what search engines and AI assistants receive. Its main crawler identifies itself with this exact user agent:
PreferiumAIEdgeBot/1.0 (+https://preferium.com/bot)Requests reach a site for one of three reasons:
Free AI scan
Checks a public page submitted by a visitor. The domain does not need to be connected.
Platform checks
Checks connected sites for delivery, uptime, page differences and broken links.
Crawling rules and limits
- Respects robots.txt. The crawler reads and honours your
/robots.txt, includingCrawl-delay, for thePreferiumAIEdgeBotuser agent and the*wildcard. Only the main crawler reads robots.txt: single-page checks do not use thePreferiumAIEdgeBotrules. If a visit seems mistaken, block the main crawler and contact us. - Polite by design. It paces requests, limits concurrency and backs off automatically on
429and403responses. - Read-only. It fetches public pages. It never submits forms, logs in or attempts to change anything.
- Scoped. It crawls a bounded number of pages per run on a schedule, not continuously.
Identify a request
User agents by request type
| Request | User agent |
|---|---|
| Crawling pages, sitemaps and robots.txtThe main crawler. Reads and honours robots.txt rules for PreferiumAIEdgeBot and the * wildcard, including Crawl-delay. | PreferiumAIEdgeBot/1.0 (+https://preferium.com/bot) |
| Rendering JavaScript-heavy pages, and re-checking the live page after a deployA headless browser run on Cloudflare Browser Rendering. It loads the page as a browser would, without images, media or fonts. | PreferiumAIEdgeBot/1.0 (browser-rendering) |
| Checking that the origin answers before a site goes liveOne request to confirm the edge can reach and rewrite the origin. | PreferiumOriginHealth/1.0 |
| Uptime checks of a connected siteThe home page is fetched as a browser and as an AI crawler, so an outage that affects only one of them is noticed. | Mozilla/5.0 (PreferiumUptime)GPTBot/1.2 |
| Cloaking guard, crawler view and the free AI scanThe same page is fetched as a person and as GPTBot, and the two are compared. These requests come from Preferium, not from OpenAI: they will not match OpenAI’s published addresses. | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot) |
| Checking links for errorsRequests the links found on a connected site to report broken ones. | Mozilla/5.0 (compatible; PreferiumLinkCheck/1.0; +https://preferium.com) |
The platform also looks up public facts on Wikipedia and Wikidata as Preferium-AI-Edge/1.0 (https://preferium.com; tech@preferium.no). These requests go to Wikimedia, never to your site.
How to verify it
A user agent alone cannot prove a request is ours.
- Network
- Requests use Cloudflare’s network, not a dedicated Preferium IP range.
- Signature
- Requests are not signed.
- Verification
- There is no separate verification method yet.
Need us to check a request?
Send its time, requested address and user agent to info@preferium.com.
How to block it
Stop crawling
Add this to your /robots.txt to disallow the crawler entirely:
User-agent: PreferiumAIEdgeBot
Disallow: /The crawler obeys it on its next fetch.
Disconnect the service
Blocking the crawler does not disconnect the site: to stop Preferium AI Edge completely, the site must be disconnected in the dashboard that connected it, and the DNS record pointed back to the original host.
Connected through an agency?
Many sites are connected by an agency that resells Preferium AI Edge under its own brand.
If your agency runs your site's SEO, these requests most likely come from that service, and the agency controls the connection: ask them to pause or disconnect the site. The original DNS record is kept on file when a site is connected, so the agency can restore it exactly.
If you cannot tell which agency connected your site, write to us with the domain at info@preferium.com (subject: Crawler question). We will help you reach the agency responsible. This inbox is not a sales line; we answer about the crawler. For a question about personal data, use the privacy request form.