Google publishes guide to allowing Google-Agent through firewalls
Google published a technical guide on 17 November explaining how to let Google-Agent through a content delivery network, web application firewall or bot-detection service. Google-Agent is the name for agents hosted on Google infrastructure that browse the web and carry out actions on a user's behalf, and Google says the guide is for publishers who want those agents to reach their site.
The agent itself is not new. It has been listed among Google's user-triggered fetchers since earlier this year. The new page tells site owners how to recognise its requests, how to verify them, and why an allow rule that looks correct can still fail.
For an Australian business running a shop or booking system behind a security service, the question is whether that service is already treating Google-Agent as hostile automation. Google's troubleshooting section lists six reasons a request can still be blocked after an allow rule is added, which gives a fair picture of where the problems sit.
robots.txt is unlikely to stop Google-Agent
Google splits its automated traffic into common crawlers and user-triggered fetchers. Googlebot and the other common crawlers build the search index and always obey robots.txt. User-triggered fetchers do a job a person has asked for inside a Google product, and because a user requested the fetch, Google says they generally ignore robots.txt rules.
Google-Agent is in the second group, alongside tools such as Gemini Notebook, Google Read Aloud and the Site Verifier that checks Search Console ownership tokens. Google says user-triggered fetchers generally ignore robots.txt, so a disallow rule is not a reliable way to control Google-Agent. The place to control it is the firewall or bot-detection layer.
It also has nothing to do with Google-Extended, the robots.txt token that governs whether Google uses crawled content to train or ground Gemini. Google-Extended has no user agent of its own and does not affect a site's inclusion or ranking in Google Search; our guide to how Gemini answers questions covers where that token fits.
User agent and IP checks identify Google-Agent
Every Google-Agent request carries a User-Agent header containing the string Google-Agent. Google publishes a mobile form and a desktop form. The mobile string presents as a Nexus 5X running Android 6.0.1 with a Chrome version placeholder; the desktop string is a Linux Chrome signature. Both include "compatible; Google-Agent;" followed by a pointer to Google's documentation.
Google uses the same W.X.Y.Z placeholder it uses for Googlebot, where it stands for a Chrome version that changes over time, and advises wildcards rather than a fixed version when filtering logs.
The second check is the source IP. Google-Agent uses the ranges published in a file called user-triggered-agents.json, which is separate from the common-crawlers.json ranges Googlebot uses and from the two other user-triggered fetcher files. Reverse DNS for user-triggered fetchers resolves to a gae.googleusercontent.com or google-proxy hostname, depending on whether the fetcher is Google owned or user owned.
The third check is still experimental. Google is testing the Web Bot Auth protocol under the identity agent.bot.goog and signing Google-Agent requests with HTTP Message Signatures, the standard defined in RFC 9421. A signed request carries a Signature-Agent header set to that identity plus Signature and Signature-Input headers labelled g. Google hosts the public keys needed to verify them in a well-known directory under the same domain.
Google's rule set is to confirm the user agent, confirm the IP, verify the signature where one is present, and then skip bot challenges for requests that pass. Some CDNs and bot-detection services already verify Google-Agent on their own, so Google says to check with your provider for the exact steps before writing custom rules.
Put the Google-Agent allow rule above AI agent blocks
The troubleshooting list is the most useful part of the guide, because it describes how a correctly written allow rule still fails.
Rule priority comes first. Google says the allow rule has to evaluate before broad blocks on cloud or data centre ASNs, managed bot rules, and category-level AI agent block policies. If a one-click AI agent block sits above a verified-bot allowance, the block wins.
Interactive challenges are the second trap. An allow rule that only lifts HTTP blocks leaves JavaScript challenges and CAPTCHAs in place, and Google specifically calls out standalone CAPTCHA widgets that appear on search or form submission. An agent that loads the homepage and then hits a CAPTCHA on the enquiry form is, for a lead-driven business, blocked where it counts.
Shared IP rate limits are another trap. Concurrent requests from many different users leave Google's infrastructure through the same egress pool, so a per-IP limit tuned for one human will trip on legitimate agent traffic. Google says to account for or exempt the verified ranges.
Two more are housekeeping. Keep IP-based rules synced with the latest user-triggered-agents.json, and make sure load balancers and proxies pass through the original client IP, the User-Agent header and, for signed requests, the three signature headers. Strip any of those in transit and verification fails before your rule ever runs.
Blocked cart and booking endpoints stop Google-Agent finishing tasks
The remaining item is the one most relevant to a business site. Google says allowing Google-Agent has to cover actions beyond the initial homepage load: on-site search, filtering, clicking into detail pages, add-to-cart, booking, and the XHR, GraphQL or api.* endpoints those actions call behind the scenes.
This is the difference between a crawler and an agent. Googlebot reads pages. Google-Agent is built to complete a task a user asked for, which on a retail or services site means working through a product filter, choosing a time slot or submitting a form. Allowing the page that loads the booking widget while blocking the API endpoint behind it produces an agent that can read the page and do nothing on it.
Google's page does not name the products that use Google-Agent, only that the agents run on Google infrastructure and act on a user's request. The direction is visible elsewhere: in the US, AI Mode already sends home service searches to direct booking links on platforms such as Jobber and Housecall Pro. Whether a Google agent can do something similar on an independent site depends in part on whether that site lets Google-Agent through.
Google's checklist for letting Google-Agent through
The decision before any configuration work is whether you want agent traffic at all. Google frames the guide as help for publishers who want to enable Google-Agent, which leaves the choice with the site owner. The same decision comes up for the other assistants, and our guide to Claude and Copilot search covers which crawlers to allow for those two.
If the answer is yes, the steps follow Google's own checklist:
- Search server or CDN logs for Google-Agent in the User-Agent string, with a wildcard on the Chrome version. Requests that were challenged or blocked suggest the agent is already trying to reach you and failing.
- Check whether your provider has a built-in verified-bot entry for Google-Agent. Google says some services verify it automatically; if yours does, use that rather than custom rules.
- Otherwise build the rule from Google's three checks: user agent contains Google-Agent, IP falls within user-triggered-agents.json, and the signature verifies where the headers are present.
- Place that rule above any ASN, managed bot or AI agent category blocks, and make it skip JavaScript challenges and CAPTCHAs, including the ones on search and forms.
- Extend it to the API endpoints behind your cart, booking calendar and enquiry forms.
- Exempt the verified IP ranges from per-IP rate limiting and set a reminder to refresh the ranges.
For a business that takes bookings or sells online, the cheapest first step is the log search. A run of 403 responses or challenge pages against that user agent is a sign the agent may already be trying to reach you; check the source IPs against user-triggered-agents.json to confirm. Mojo Dojo includes this check in technical SEO audits for sites that sit behind a CDN or bot-protection service, because the rules that keep scrapers out are the same rules that can keep a customer's agent out.




