Google tells platforms to noindex shared and generated pages

Google updated its guide to preventing user-generated spam on 26 October, and the scope has widened. The document used to cover comment forms, other user-generated content inputs and hosting platforms. It now also covers “hosting platforms and tools that let users publish shared pages, generated content, or web apps”, and it carries a new section headed “Only index content meant for Search”.

The advice in that section is short. Any service that lets people make shareable pages, generated outputs or interactive tools should work out whether those pages are meant to be discovered in search or only opened by someone holding the link. Pages made for sharing, collaboration or practical use rather than public discovery should, on Google's advice, be kept out of the index by default, using the noindex rule (in a robots meta tag or an X-Robots-Tag HTTP header) or a robots.txt crawl block.

Google's changelog entry describes the update as guidance on “blocking indexing of link-shared content and monitoring abuse in Search Console”, and gives the reason as helping platforms “keep user-created pages that aren't meant for Search out of the index and detect spam earlier”.

Decide, per page type, whether shared pages and web apps belong in Google

The wording points at a kind of product that barely existed when the original guide was written: a tool where a user types a prompt or fills in a form and gets a page at a public URL. AI assistants that publish a generated document, no-code builders that spin up a site per user, quote and report generators, form builders, shared dashboards, whiteboards and document editors with a share button. Each of them creates URLs at a rate set by the user, not the operator, and each of those URLs is a candidate for Google's index unless someone has said otherwise.

Google's spam policies already cover scaled content abuse, defined as many pages generated for the primary purpose of manipulating rankings rather than helping users, and the policy ends with a plain instruction: if you host such content, exclude it from Search. The updated guide is the operational version of that line. A spammer who can create thousands of pages on your platform in an afternoon is borrowing your domain's standing to rank them, and the platform wears the consequences.

Plenty of Australian businesses that would never call themselves a platform are in scope. A customer portal that generates a public link per invoice or statement. A franchise system that stamps out a landing page per territory. A trades quoting tool that emails a link to every estimate. A review or Q&A feature on an ecommerce site. A community forum bolted onto a SaaS product. Each one is a page factory, and the guide now asks the operator to decide, per page type, whether the output belongs in Google.

rel="ugc" on untrusted links

The rest of the guide is retained and worth re-reading alongside the new section. Google suggests keeping a record of sign-ups and interactions and looking for spam patterns: how long a form took to complete, how many requests came from the same IP range, which user agents were used at sign-up, and what usernames or form values were chosen. Those signals can feed a reputation system.

Because comment spammers want their content in search engines, Google recommends adding a noindex robots meta tag to posts from new users with no reputation, then lifting it once the user has earned some. Links inside untrusted content should carry a nofollow or ugc rel attribute. Manual approval for suspicious interactions, blocklists for repeat offenders, and reCAPTCHA or similar checks at sign-up round out the list.

None of that is new. What is new is the default. The old advice was to noindex the risky users. The new advice is to noindex the risky page types, and let indexing be the exception you choose rather than the state you inherit.

Check regularly for spikes in the Page Indexing report

The monitoring section has also been expanded. Google says to verify the site or subdomain in Search Console so that manual action and security issue alerts arrive by email, and to keep an eye on the Page Indexing and Search performance reports for sudden jumps in indexed pages, impressions or search terms that have nothing to do with what the platform is for.

It goes further than Search Console. Google now recommends measuring, at the platform level, how many pages are created and indexed, and alerting when those numbers look wrong. Its two examples are a sudden jump in new public pages and a few accounts producing a big slice of the indexable URLs. Those are internal numbers, so they need an internal dashboard, and they can show a problem before Google's own reports do. Earlier this month Search Console's performance report was reported to be running more than 64 hours behind, which is a reminder that a spike you notice in Search Console may already be old news on your server.

The remaining advice is familiar: look for redirects, large ad sections, spammy keywords and big blocks of encoded JavaScript; use the site: operator or Google Alerts; watch server logs for traffic spikes; test URLs against the Safe Browsing API; and run confidence checks such as whether thousands of overnight interactions from an unexpected country make sense for your audience.

Which first, noindex or robots.txt?

Google offers two tools and the choice matters. A robots meta tag or X-Robots-Tag header tells Google not to index a page it has crawled. A robots.txt rule stops Google crawling the page at all. For the noindex rule to work, the page must not be blocked by robots.txt, because Google has to fetch the page to see the tag.

So the right pick depends on where you are starting. If user pages are already in the index, use noindex and leave them crawlable until they drop out, then consider blocking. If the page type has never been crawled and you want to save the crawl, robots.txt is simpler. Bear in mind that robots.txt saves crawling but does not guarantee the URLs stay out of Google: a blocked URL that is linked from elsewhere can still appear in results without a description, so if the goal is to keep pages out of the index, noindex is the reliable option. For generated files that are not HTML, such as a PDF quote or an exported report, the X-Robots-Tag header is the only way to apply noindex, and it is worth remembering how Google indexes and ranks PDF files before assuming they are invisible.

This is the third update in a week to Google's documentation on how pages are crawled and indexed. On 21 October Google refreshed its page on controlling what you share with Google Search to list the robots meta tag, data-nosnippet, X-Robots-Tag and the Search generative AI control in one place. On 23 October it clarified its URL structure guidance on using hyphens rather than underscores to separate words. The spam guide change on 26 October fits the pattern: Google is tidying up the levers site owners have, and reminding platforms to use them.

Audit user-made URLs against the Search performance report

• Open the Page Indexing report and compare the count of indexed URLs with the number of pages you intend to rank. If the first number is several times the second, find out what is filling the gap.
• List every feature on your site or product that creates a URL on behalf of a user or customer: portals, quotes, reports, share links, profiles, reviews, forum threads, generated documents.
• For each one, decide whether it should rank. If the answer is no, or not sure, add noindex by default and allow indexing only for the page types you choose.
• Add rel="ugc" or rel="nofollow" to links inside anything a user can write.
• Put a counter on pages created per user per day and alert on outliers. Google's own example is a small number of accounts producing a large share of your indexable URLs.
• In the Search performance report, sort by query and look for terms that have nothing to do with your business. Those are the pages a spammer built on your domain.

Google is not saying user pages must never be indexed. Some platforms earn real organic traffic from public templates, community answers and shared documents, and nothing in the guide forbids that. The operative words are “by default”. Indexing a user-created page should be a decision the platform makes on purpose, for page types it has looked at, rather than a side effect of shipping a share button.