Google can briefly keep important 4xx URLs in its index

Google Search can hold an important URL in its index for a short time after the page starts returning a 4xx error, Google's Gary Illyes said in a LinkedIn comment on or around 5 November. The listing survives in a stripped form: a bare link with the last title Google recorded, and no snippet or favicon.

The protection is temporary and Illyes tied it to the page's importance. Google's own crawling documentation still says a previously indexed URL that returns a 4xx code is removed from the index. For a business whose enquiry pages break by accident, the lesson is to fix the status code the same day and treat any reprieve as luck.

Google briefly protected VLC's URL after a 418

The comment answered SEOs asking why the VLC media player site had lost its favicon in Google's results. Illyes said VLC's server had been responding with HTTP 418, the joke "I'm a teapot" status, and that Google treats 418 the way it treats most other 4xx responses, which is as a page that does not exist.

A not-found page would normally drop out. Illyes said VLC is "arguably one of the most important pages on the internet", so Search "temporarily protected the URL from falling out of the index right away".

By the time he replied, VLC's site was returning 200 OK with content again, and he expected the problem to sort itself out soon. Google's results at the time also showed VLC's home page sitting second, under its download page, though Illyes did not connect the two.

Protected URLs lose their snippet and favicon in results

Protection is a narrow favour. Illyes said that once Google can no longer read the page it has nothing to display except "just the link itself and the last good title we had". The snippet goes, the favicon goes, and the title can be stale.

For a page that relies on its snippet to win the click, that is a worse result even though the URL is still listed. A rank tracker would still report the URL as present and miss that the listing had lost everything that earns a click.

Indexed pages returning 4xx codes get removed

Google's page on how HTTP status codes affect its crawlers sets out the standard handling. Google does not use content from URLs that return 4xx codes, a URL that was indexed and now returns 4xx is removed from the index, and Google's systems "stop using the URL over time". Crawling of the URL also slows gradually.

Every 4xx code except 429 is treated the same way. A 403 from a firewall, a 401 from a login prompt and a 404 from a deleted page all tell Google the content is gone. The document says exotic codes such as 418 are not covered, which is why the VLC case needed a Google engineer to explain it.

Server errors get gentler handling. Google says 5xx responses and 429 make its crawlers slow down, already indexed URLs are kept for a while and only dropped if the errors persist, and the crawl rate climbs back once the server returns 2xx again. That difference is why Google's advice to return a 503 status for sites taken offline to patch is worth following: a 503 keeps index entries intact for a while, where a 404 or 403 starts the removal.

Will Google keep my broken page indexed?

Illyes framed the protection as a response to VLC's importance. He did not say how Google decides which URLs qualify, how long the protection runs, or whether an ordinary business site would ever receive it. Google's documentation does not describe the mechanism at all.

So the safe assumption for a plumber in Brisbane or an online store in Melbourne is the documented one. A service page that starts returning 403 or 404 will be removed, and Google will come back to check it less often.

Firewall rules can serve Googlebot errors that owners miss

Pages rarely start returning 4xx because somebody deleted them. More often a deploy changes a URL pattern, a security plugin or CDN firewall rule starts answering Googlebot with 403, a staging password is left on a live section and returns 401, or a rate limit hands crawlers a 429.

The owner looking at the site in a browser often sees nothing wrong, because the rule only fires for bots or for traffic from certain countries. Googlebot sees the error on every visit.

Test enquiry pages with URL Inspection

This is the first check in any SEO audit we run, and it takes minutes.

• List the URLs that bring in enquiries or sales: the home page, the main service or category pages, and the landing pages behind Google Ads campaigns. Keep the list short enough that someone will check it.
• Fetch each one with a tool that shows the HTTP status code, and test as Googlebot where you can, since firewall rules often treat bots differently from browsers. Search Console's URL Inspection tool shows how Google fetched a page.
• Put uptime monitoring on the list that checks for a 200 status and for real content on the page. Google reports a 200 response carrying an error message or an empty body as a soft 404 in Search Console.
• Re-run the check after every deploy, plugin update, CDN change or firewall rule. Blocking rules aimed at scrapers are a common way a working site starts serving 403s to Googlebot.
• Watch Search Console, which generates errors for 4xx and 5xx responses and failed redirects. The Page Indexing report is the same place Google pointed platforms to when it told them to noindex shared and generated pages, and a sudden jump in 4xx pages there is the signal to act on.
• When you check the Performance report for a dip on an affected URL, compare impressions with clicks. The early November click spikes on some sites looked automated and can hide or exaggerate a real change.
• If a page did break, restore the original URL with its original content and a 200 status. Illyes expected VLC's listing to repair itself once the site served 200 OK again, and that is the cleanest recovery path for any page.

Pick the ten URLs that make the business money and check their status codes today. Add the same check to whatever list gets run after a server, CDN or plugin change, because a 403 served only to Googlebot never shows up in a browser.