TL;DR: Most websites have a search box. When a visitor types into it, the site builds a results page, and that page gets its own web address. For nearly twenty years Google’s official advice was to keep those pages hidden from Google. On July 31, in episode 113 of Google’s Search Off the Record podcast, Google’s John Mueller confirmed the advice is no longer in the rulebook. PPC Land and Search Engine Roundtable both reported it, and Google’s spam policies page no longer mentions search pages. In the same episode Mueller said to keep hiding them anyway. Your site needs no changes this week.
First, what are we talking about?
Two terms, because nothing below makes sense without them.
Your site’s search pages. Not Google’s results — your own. Someone types “blue jacket” into your shop’s search box, your site builds a results page for that phrase, and that page has an address, usually something like yoursite.com/search?q=blue+jacket. Type something else and you get a different address. There’s no limit to it. Every phrase anyone could type produces another page.
robots.txt. A small text file at the root of a website, listing instructions for the automated programs that read the web. It tells them which parts to skip. A line reading Disallow: /search means “stay out of there”. That line is what the old rule was about.
The question in this story is whether you should let Google wander into the endless supply of pages your own search box can produce.
What changed
Since Google’s original 2007 guidelines, “block your internal search results” sat in the rulebook as near enough a requirement. It was the kind of item that turns up on a technical audit checklist with an official citation beside it. It isn’t there any more. Mueller’s words, via PPC Land’s write-up: “Nowadays, we don’t have that listed in the search policies.”
Google didn’t say these pages are now welcome. It didn’t change how anything works, and it didn’t withdraw the technical guidance explaining why the pages caused trouble. What was probably always practical advice has stopped being written down as a rule.
Why you should still block them
Mueller gave two reasons in the same episode. Both are about how the machinery behaves, not about what the rulebook says.
It wastes Google’s attention on your site. Google doesn’t read every page on the internet. Its guidance on crawl budget calls the web “a nearly infinite space, exceeding Google’s ability to explore every publicly accessible URL”, and tells site owners to consolidate duplicates so crawling goes to unique content rather than unique addresses. Your search box can produce addresses without end. Time Google spends working through them is time it doesn’t spend on your product and article pages. Mueller put it plainly: “It’s just like, you’re being very inefficient.”
Strangers can put words on your site. A search page displays whatever was typed into it. That has long been a route for getting spam onto a respectable domain and from there into search results — gambling, pharmaceuticals, adult content. Mueller noted it can get a site flagged as hacked in Search Console no matter what the guidelines say.
The bit nobody’s mentioning
There’s a newer reason to keep the door shut, and it didn’t exist in 2007.
Google’s crawler is well-behaved. It has a documented limit on how much it will fetch, and Google has every reason to be efficient about it. The crawlers now gathering material for AI answers are a more mixed population, and here’s the catch: an instruction addressed to Google only applies to Google.
If your robots.txt blocks the search pages under a heading that names Googlebot specifically, rather than one meaning “everyone”, then GPTBot, ClaudeBot, PerplexityBot and whatever launches next can walk straight into the space Google itself declines to enter.
That’s a server-load problem, at a moment when crawler defaults across the web are being rewritten. It’s also a quality problem. An AI system reading your search pages is learning about your business from thin, repetitive, auto-generated copies of your catalogue instead of the pages you actually wrote. These crawlers mostly take the page exactly as served, so whatever they reach first is what they learn from.
What to actually do
For most sites, nothing. Check it, then move on.
- Confirm the search pages are still blocked, and check who the instruction is addressed to. One meaning “all crawlers” covers the current population. One naming only Google doesn’t.
- Use robots.txt rather than a
noindextag here. Anoindextag says “you may fetch this, just don’t list it”, so Google still spends the visit fetching it. A robots.txt block means the visit never happens. That difference is spelled out in Google’s crawl budget documentation. - Don’t delete a blocking rule nobody can explain. If no one remembers why it’s there, write down why, don’t remove it.
- Update your audit template, not your website. If your checklist cites this as an official Google requirement, that citation is now wrong even though the advice still holds. Fix the wording before a client finds it.
- Don’t confuse this with proper category pages. A curated listing page with a stable address is a different animal from an open search box, and can deserve to be in Google on its own merits. This rule was only ever about the search box.
Notice how much of that list is checking rather than doing. Someone has to open every client’s robots.txt, work out which crawler each rule is addressed to, decide whether the AI crawlers are covered, and repeat it the next time the documentation shifts. Across a portfolio that is a morning’s work for a change that produces nothing visible, which is precisely why it gets skipped.
That is the part Preferium automates. Its 47 automated checks crawl every page and score the site, the crawl-instruction layer included, and the system finds, fixes, deploys and verifies on its own, with a real browser re-opening the live page after each title, meta, H1 and schema deploy. If you would rather keep a hand on it for something like a robots.txt change, nothing ships without your approval. More on how the system works.
Key takeaways
- The rule left the documentation, not reality. Mueller confirmed on Search Off the Record episode 113 (July 31, 2026) that blocking your site’s own search pages is no longer listed in Google’s rulebook.
- He advised keeping them blocked anyway: they waste the limited attention Google gives your site, and they let strangers display spam on your domain.
- Treat it as a paperwork change, not permission. The practical case against letting these pages into Google hasn’t moved.
- Check who your blocking rule is addressed to. An instruction naming only Google won’t stop GPTBot, ClaudeBot or PerplexityBot crawling the same endless space.
- robots.txt beats a
noindextag here — withnoindex, Google still visits the page before ignoring it. - Fix your audit checklist’s wording before a client notices the rule you cited no longer exists.
