Crawler access
Blocked or missing pages.
What this means: Crawler access
An AI crawler is blocked, gets a 404, or never reaches a page that matters.
Each night, the engine finds supported SEO issues, publishes fixes within your rules and checks the live HTML. Changes are delivered without editing the client’s CMS.
Supported fixes for existing pages, delivered within your rules and checked after publication. No new articles.
Your publishing rules apply at every step
Your control level determines what can publish. Risky fixes get a safety review; you control visible rewrites per site. Every step is logged.
Review site health, prioritize findings and enable supported fixes. Each delivered change is recorded in the activity log.
What an issue can look like Illustrative examples
Blocked or missing pages.
An AI crawler is blocked, gets a 404, or never reaches a page that matters.
Missing structured page information.
A page a model would cite has no schema to cite from.
Missing or unclear search descriptions.
Title or description is missing, empty, or does not say what the page answers.
Two addresses. No clear original.
Two addresses show the same thing, and neither is marked as the original.
Business details that do not match.
Name, address and phone number differ between pages, so the model is unsure who the client is.
Pages that need another look.
A page that used to be cited has sat untouched and is losing ground.
Pages with no path leading to them.
A page no other page points to is a page the model rarely finds.
Discovery instructions that disagree.
Sitemap, robots and IndexNow disagree about what exists.
Explore the 13 check areas. Scheduled crawls process pages in batches.
Checks cover 13 areas, from page titles to crawler access.
Finding an issue does not mean it can be fixed automatically.Find missing, duplicate or poorly sized titles.
Find missing, duplicate or poorly sized descriptions.
Check the main heading and heading structure.
Find thin content, duplicate pages and heavy resources.
Find broken links and pages that are hard to reach.
Check alternative text and image file sizes.
Find inconsistent or overly long page addresses.
Check which version of a page should be indexed.
Check schema markup and missing fields.
Check the tags used for social previews.
Find settings or errors that prevent discovery.
Check language and regional page relationships.
Check HTTPS, discovery files and crawler access.
Additional technical coverage: 24 extra checks from an independent second engine, plus an external crawl for duplicates, redirect loops and non-indexable pages.
Inspect findings by page and priority, with an assistant that can explain the site’s data.
See the whole-site score, what passes and what needs attention.
Explore the site tree. Spot orphan pages and content buried too deep.
Review recommendations in priority order, with the site’s data behind each finding.
Help with your client’s data
Your questionWhere should I start?
Review high-priority findings, then open the affected pages.
Page findingsRead-only. It explains the data and links you to the relevant screen. It cannot publish changes.
Answers use the client’s actual site data.
Ordered steps link to the screen where you work.
Review upcoming automation and the reason behind it.
Search pages, sites and settings from the keyboard.
Browse common fixes below, or explore the complete capability catalog.
Explore The complete open list of 358 capabilities across 24 areas in thecapability catalog: 328 capabilities in 23 areas are client-facing. 30 capabilities in 1 area are agency and white-label functions.
What search engines and models read first. Most of it is fixed in the HTML that is served, without anyone changing how the client publishes; the rest is checked and reported.
Open an item to see what it covers.
Write unique titles and descriptions for the subject each page covers. Add them where they are missing.
Give meaningful images descriptive alt text that explains their subject.
Loading (LCP), interactivity (INP) and visual stability (CLS), measured from real Chrome visitors and reported per page.
Preload hints and critical CSS for the first screen, lazy loading below the fold, and picture markup for AVIF or WebP — applied where the client’s host can serve them.
Google reads the mobile version first. PageSpeed audits check and report mobile problems; the viewport setting is corrected where the page already carries one.
The map and the rulebook the engines get: where the pages are, and where not to look.
Says which page is the original, so variants of the same page are not read as duplicate content.
The language machines speak. Price, availability and ratings are marked up only when the page actually shows them.
The breadcrumb trail is marked up, so the engine can show the path to the page instead of just the URL.
One main heading per page and logical subheadings. You control visible rewrites per site.
The 301 and 302 redirects you create are live at the edge immediately, and a server that redirects to its own internal address is sent back to the client’s domain.
What gets indexed and what does not, with a filter so only valid instructions go out. Clearing a stray noindex goes through a safety review first.
Every page declares its correct language, validated, and matching the version actually served.
The certificate for the connected site is issued and renewed automatically, and an expiring certificate on the client’s own server is flagged before it takes the site down.
Eight checks find missing return links, invalid language codes and broken targets between language versions. The launch offering includes writing hreflang tags at the edge.
Search engines that support IndexNow are told the moment a page changes, instead of waiting for the next crawl.
How the page reads when shared on Facebook, LinkedIn and X: title, description and card type — added where missing. An existing share image is kept; an image address that might not exist is never invented.
Modern security headers are added to every response served, without touching the client’s server setup.
Touch icon, tile color and theme color, so the site looks right when someone saves it to their phone.
Rank Math, Yoast, All in One SEO and Squirrly are detected and their competing schema is replaced by one merged graph, so machines do not get two different answers.
Which valuable schema types the site still lacks, page by page — out of 31 types the engine can write.
Double slashes, uppercase letters and overlong paths are flagged before they become an indexing problem.
Internal links, outbound links, scripts, stylesheets and images are tested for errors and reported, so you decide what gets redirected.
Mark a page permanently gone, so search engines and AI crawlers drop it — instead of redirecting it forever.
Legal name, logo and official profile links marked up and tied to the Wikidata identifier, so machines can verify who the client actually is.
Which queries the client can actually win, and which page should answer each of them. Measured and reported; the findings feed the rewrites.
Open an item to see what it covers.
No guessing. Search data shows which words customers use, including misspellings and local variants.
One clear lead term per page, which the title and heading rewrites are written around.
Related terms that make the topic unambiguous — write about Apple, and iPhone and iOS help the engine know which Apple.
Fewer searches per phrase, but visitors who know what they want.
Every keyword gets its own home, so five pages do not fight over the same term.
The client site laid over a competitor’s to see where content is missing.
Which terms do they rank for that the client lacks? The holes in their strategy become the client’s work list.
Link text tells the engine what the target page is about. “Read about running shoes” beats “click here”.
The result page changes — map packs, videos, images, AI answers. The features are tracked, so you see when the picture changes.
Which terms bring the most traffic, and which are on the way up. The cold facts.
How the pages connect, and how the models understand what belongs to what.
Open an item to see what it covers.
Pages that should link to each other, found by meaning and reported. The related-links box can add them to the page once visible rewrites are on.
Questions, answers and steps marked up only where the page actually contains them.
Pages that are too alike are found and reported; a canonical can mark the one that counts.
The same product in several colors or sizes points to one original.
The smoke detector: if traffic drops or an important page disappears, you hear about it.
Visible rewrites deliberately skip everything in the menu, footer and sidebar.
Purely decorative images are recognized and skipped — no description is written or forced.
Correct text direction for Arabic, Hebrew and similar languages, also in text the engine inserts.
Map boxes for physical addresses are built safely from coordinates instead of pasted third-party code.
The signals that tie the client’s name to a place and a field — and to other sources that mention them.
Open an item to see what it covers.
Name, address and phone marked up the same way on every page, so the engine is never in doubt about who the client is.
The client’s Wikidata identifier tied to the pages, so the engine connects the site to the entity it already knows.
Who links to the client? Weak links are surfaced and new opportunities found.
Whether the client is listed in the leading local directories in its market — the listings that confirm the business is real.
Reporting against GA4 and Search Console, and on who the models cite — with the numbers that actually moved.
Open an item to see what it covers.
The report sums up what was changed and what moved, per client — white-label, ready to forward.
Measured against the client’s own accounts, not invented numbers. Everything reported can be checked.
Scheduled prompts are sent to the live engines with web search, in the client’s market, and the answer is read: who is cited.
Read straight from the request at the edge, not from a tracking script — a crawler runs none.
How often the client shows up, how often someone clicks, and on which queries.
Every page is followed on its own, so you see which ones lift and which ones stand still.
The report separates changes that were made from changes that actually moved a number.
The engine works on its own; control stays with you the whole way, in real time.
Open an item to see what it covers.
Every change listed with its time, the page it applied to and what was changed.
The panel shows the current state at all times — not yesterday’s report.
Disagree? Withdraw it yourself. The previous state is served again on the next request.
Want to keep the change but phrase it differently? Edit it in the panel.
Want to look before anything goes live? Set the client to Review everything, and every change waits for your approval.
The source code in the publishing system is never modified. Everything is delivered in the HTML response at the edge.
The work happens in the markup the engines read. The design visitors see stays exactly as the client wants it.
Set a control level per client. It determines which enabled changes can publish automatically and which require your team’s approval.
Enabled changes publish within your rules.
Automate technical work. Review visible text.
Your team approves each change before it goes live.
Choose a level per site. Enable visible rewrites separately. Menus, footers and sidebars stay untouched.
See checks and rollbackVisible text publishes automatically only on Full autopilot, when enabled and within your rules.
Risky fixes need a safety review before automatic publication and are flagged for your team.
After publication
Automatic work. Clear next steps for you.
Crawls revisit existing pages and discover new ones. Unfinished batches continue in later runs.
Compare the live page with the expected version. Restore changes that were overwritten.
Changes made in your panel reach visitors without republishing the original site.
Actions outside automatic work wait for your review.
Daily recommendations are ranked by likely impact.
A fictional bike-service page, before and after supported changes to its HTML. The client’s original site stays in place.
Home | Northwind Cycles
Bike Service in Seattle — Book Today
Empty
Service for every bike. Reply within one day.
Missing
Bike service in the workshop
Changes are delivered in the HTML response. The client’s CMS stays unchanged.
<title>Home | Northwind Cycles</title><meta name="description" content=""><img src="workshop.jpg"><title>Bike Service in Seattle — Book Today</title><meta name="description" content="Service for every bike. Reply within one day."><link rel="canonical" href="https://northwindcycles.com/bike-service"><meta property="og:title" content="Bike Service in Seattle — Book Today"><img src="workshop.jpg" alt="Bike service in the workshop"><script type="application/ld+json">{"@type":"Service"}</script>Our changes are already in the HTML when it arrives. Here’s why we chose this approach over browser-only fixes and crawler-specific delivery.
The optimized page arrives ready to read.
We apply changes before sending the HTML. On server-rendered sites, visitors and crawlers receive the same optimized page.
LimitationIf a crawler does not run the script, it misses those changes and reads the original HTML.
LimitationA separate rendered page is selected for detected crawlers. Delivery depends on that detection step.
For JavaScript-only sites: Preferium also needs dynamic rendering to make the page readable to crawlers.
These cards compare delivery methods, not every mode each vendor offers. OTTO's limitation above applies to its browser pixel, not its documented edge mode. Alli's card describes its documented crawler-rendering approach. It is not a measured failure rate.
Vendor documentation checked 3 October 2026: OTTO rendering modes, Alli crawler rendering and Alli DNS integration.
A real browser checks core page elements after deployment. See what triggers an automatic rollback and what needs your attention.
A real browser returns after deployment, once the edge serves the new version.
Compare title, meta description, H1 and structured data. Missing or empty elements can trigger rollback.
The change is rolled back automatically. An alert tells you what was withdrawn.
The checks also flag a lost page body or most of its content. Other edge changes remain available for manual rollback.
Add concise on-page answers when visible rewrites are enabled for the site.
Short summary
Workshop servicing for your bike. Contact the shop to choose a time.
Direct answer
Tell the shop which bike you ride and what needs attention. Agree on a service and a time before bringing it in.
Key takeaways
Answer blocks
Summaries, direct answers and key points help readers find what they need.
Visible rewrites must be enabled. Your publishing rules still apply.
Labels follow the page’s language, in 21 languages, including right-to-left text for Arabic and Hebrew.
Up to seven internal links chosen by meaning, not word match — helps both readers and crawlers find related pages.
For physical addresses, built safely from coordinates instead of pasted third-party code.
Bullet points are rewritten to be complete, and tables get a caption and column headers machines can understand.
A short plain-language explanation above code examples, and captions that say what the image actually shows.
Clear labels and helpful placeholder text — and “click here” becomes text that says where the link goes.
The titles of nearby pages are shown to the AI as it writes, so duplicates are avoided immediately instead of being found in an audit weeks later.
See what Preferium generates and which AI-use choices belong to your agency.
Generated by Preferium
Files and signals help crawlers find pages and understand the site.
Discovery support, not a guarantee of indexing or AI citations.
Chosen by your agency
Declare how AI may use the content.
Your declaration communicates your choice. It does not force a crawler to comply.
| File or signal | What it does | Handled by |
|---|---|---|
| /sitemap.xml | Every page with sensible priorities; blocked and non-indexable pages are left out. | Preferium |
| /robots.txt | AI crawlers are allowed by default, and the client’s own rules are respected. | Preferium |
| /llms.txt | A site overview following the llms.txt proposal — in short and full versions. | Preferium |
| /agent.json | Machine-readable manifest describing the site for AI agents. | Preferium |
| /agents.txt | The same description in text form, for agents that look for it. | Preferium |
| Content signal | A clear declaration of what AI may do: appear in search, answer questions, train — your choice. | Your agency |
| /feed.xml · Atom | Content feed generated from the client’s pages. | Preferium |
| Web app manifest | So the site behaves like an app when saved to a phone. | Preferium |
| /.well-known/security.txt | Published security contact, so researchers can reach the client the right way. | Preferium |
| IndexNow | Search engines are notified the moment a page changes, instead of waiting for the next crawl. | Preferium |
Your agency chooses the content signal. Preferium generates the other discovery outputs. Sitemap priorities and update frequency can be set per page; thin or private pages are excluded.
Measure real visitor performance, review supported loading fixes and connect language versions.
Core Web Vitals use real Chrome visits. PageSpeed audits inspect performance and page quality.
Supported fixes prioritize the first screen, use modern image formats and defer images below it.
Check language links and regional codes. Use a glossary to protect words that must stay unchanged.
If an origin fetch fails or times out, an eligible cached copy can serve public pages. A usable copy must already exist.
Pick one client site for a walkthrough of the engine and your branded panel.
Review its health checks.
Understand what the engine would improve.
See the client experience under your brand.
One connection. A clear way back.