Main crawler
Reads sites connected by their owner or agency. Honours robots.txt, including Crawl-delay.
Crawler
PreferiumAIEdgeBot
The web crawler and site checks operated by Preferium AS for Preferium AI Edge. If you saw one of the user agents below in your server logs, this page explains what it is, why it visited and how to stop it.
What it is
Preferium AI Edge reads a site's current content so it can optimize what search engines and AI assistants receive. Its main crawler identifies itself with this exact user agent:
PreferiumAIEdgeBot/1.0 (+https://preferium.com/bot)Why it visited your site
Free AI scan
Checks a public page submitted by a visitor. The domain does not need to be connected.
Platform checks
Checks connected sites for delivery, uptime, page differences and broken links.
Only the main crawler reads robots.txt. Single-page checks are not governed by its rules. If a visit seems mistaken, block the main crawler and contact us.
Every request type and its user agent
The crawler is not the only request the platform sends. These are all of them, with the exact user agent each one carries:
Exact user agents and request details
| Request | User agent |
|---|---|
| Crawling pages, sitemaps and robots.txtThe main crawler. Reads and honours robots.txt rules for PreferiumAIEdgeBot and the * wildcard, including Crawl-delay. | PreferiumAIEdgeBot/1.0 (+https://preferium.com/bot) |
| Rendering JavaScript-heavy pages, and re-checking the live page after a deployA headless browser run on Cloudflare Browser Rendering. It loads the page as a browser would, without images, media or fonts. | PreferiumAIEdgeBot/1.0 (browser-rendering) |
| Checking that the origin answers before a site goes liveOne request to confirm the edge can reach and rewrite the origin. | PreferiumOriginHealth/1.0 |
| Uptime checks of a connected siteThe home page is fetched as a browser and as an AI crawler, so an outage that affects only one of them is noticed. | Mozilla/5.0 (PreferiumUptime)GPTBot/1.2 |
| Cloaking guard, crawler view and the free AI scanThe same page is fetched as a person and as GPTBot, and the two are compared. These requests come from Preferium, not from OpenAI: they will not match OpenAI’s published addresses. | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot) |
| Checking links for errorsRequests the links found on a connected site to report broken ones. | Mozilla/5.0 (compatible; PreferiumLinkCheck/1.0; +https://preferium.com) |
Only the main crawler reads robots.txt. The other requests are single-page checks, not crawls, and are not governed by the PreferiumAIEdgeBot rules. Separately, the platform looks up public facts on Wikipedia and Wikidata as Preferium-AI-Edge/1.0 (https://preferium.com; tech@preferium.no); those requests go to Wikimedia, never to your site.
How it behaves
- Respects robots.txt. The crawler reads and honours your
/robots.txt, includingCrawl-delay, for thePreferiumAIEdgeBotuser agent and the*wildcard. - Polite by design. It paces requests, limits concurrency and backs off automatically on
429and403responses. - Read-only. It fetches public pages. It never submits forms, logs in or attempts to change anything.
- Scoped. It crawls a bounded number of pages per run on a schedule, not continuously.
How to verify it
A user agent alone cannot prove a request is ours.
- Network
- Requests use Cloudflare’s network, not a dedicated Preferium IP range.
- Signature
- Requests are not signed.
- Verification
- There is no separate verification method yet.
Need us to check a request?
Send its time, requested address and user agent to info@preferium.com.
How to block it
Add this to your /robots.txt to disallow the crawler entirely:
User-agent: PreferiumAIEdgeBot
Disallow: /The crawler obeys it on its next fetch. Blocking the crawler does not disconnect the site: to stop Preferium AI Edge completely, the site must be disconnected in the dashboard that connected it, and the DNS record pointed back to the original host.
Connected through an agency?
Many sites are connected by an agency that resells Preferium AI Edge under its own brand.
If your agency runs your site's SEO, these requests most likely come from that service, and the agency controls the connection: ask them to pause or disconnect the site. The original DNS record is kept on file when a site is connected, so the agency can restore it exactly.
If you cannot tell which agency connected your site, write to us with the domain. We will help you reach the agency responsible. For a question about personal data, use the privacy request form.
Contact
Questions about these requests: info@preferium.com (subject: Crawler question). This inbox is not a sales line; we answer about the crawler.