
As of November 2026, Google has published the answer it used to leave to guesswork: whether traffic from its own agents should be let through a site's edge. On November 17, 2026, Google added a technical guide explaining how to allow Google-Agent in a content delivery network, a web application firewall or a bot-detection service. For a Singapore team whose ai automation for business work ends on a site it owns, that guide is now the checklist the edge configuration has to pass.
Key takeaways:
Google's crawling changelog records that on November 17, 2026 it published a technical guide on making sure Google-Agent has access to a website, for publishers that want to enable the agent. The guide covers how the agent identifies its requests, how to allow that traffic in a CDN, firewall or bot-detection service, and troubleshooting.
Read the timeline in order. On March 20, 2026, Google added the Google-Agent user agent and the IP ranges for user-triggered agents to its documentation, and said the agent would roll out over the following weeks. The November 17, 2026 guide answers what that rollout left with site owners, and frames itself narrowly: guidance for publishers who want the agent to reach them, not a requirement for every site.
An agent hosted on Google's infrastructure fetches a page because a person asked it to do something, and the guide describes Google-Agent as navigating the web and performing actions on that request. From a firewall's point of view it looks like an automated client: no browser session a human started on your domain, no cookie your site set, a user agent most analytics tools do not classify.
Bot management treats the pattern as suspicious because most traffic matching it is scraping. A managed rule challenges it, a category rule blocks it, a cloud or data-centre ASN block catches it again, and together those defaults also stop the agent a customer asked to use your site. The aim Google states is to prevent legitimate requests from being blocked while keeping existing controls, which is narrower than switching protection off.
What the guide does not give you is scale: Google does not say how many requests its agent loses to blocked sites. The direction is clear all the same. An agent that can search, filter, add to cart and book is part of the journey, and a block there is a lost transaction rather than a lost impression.

The allow rule belongs at the edge, in front of the origin. For most ai automation for business teams the rule sits in a console they rarely open, so the placement matters: the edge evaluates the request before any page is served, and the site cannot grant access to a request the edge has rejected.
Google gives three ways to identify the agent: a User-Agent header containing the string Google-Agent, an address in the user-triggered-agents.json file Google publishes, and, in an experiment, a signature checked against a published key set.
Where a provider does not verify Google-Agent on its own, the guide sets out four rules in order:
The fourth rule is the one half-applied: a team adds an allow rule, sees the challenge still fire, and concludes the rule failed.
Google's experiment uses Web Bot Auth. Signed requests name the identity https://agent.bot.goog in a Signature-Agent header, label their Signature and Signature-Input headers with the letter g, and are checked against the HTTP Message Signatures standard, RFC 9421, using public keys in a well-known directory under agent.bot.goog.
Signing may spread, or the labels may change as the protocol settles. If your edge verifies signatures already, switch it on; if you built the check, keep the key source configurable.
Google's troubleshooting section reads as a list of places where a correct rule still loses:

The person who owns the consequence is rarely the person who owns the setting. In most Singapore setups the site is built and hosted by an agency, a website builder or group IT, so the marketing lead hears about a block from a sales report rather than a console. The change needs one conversation this week with whoever holds the CDN account.
Start with the journeys where an agent does something a person would otherwise do: adding to a cart, picking a clinic slot, filling a B2B enquiry form. That request arrives without a session your analytics can attribute, so a block leaves no referral and no error. For teams running web application development in-house, the edge rule and the application's own bot logic must agree, or one undoes the other.
Allowing the agent is a decision per site, not a company-wide policy. This is the table for that conversation.
| Your site | Allow Google-Agent? | What to check first |
|---|---|---|
| Online store with a cart | Yes, scoped | Bot rules on cart, checkout and the api.* endpoints behind them |
| Booking or appointment flow | Yes, scoped | The challenge that fires on date selection and form submission |
| Brochure site with an enquiry form | Yes, after checking | Shared-IP rate limits that can throttle a small egress pool |
| Member area behind a login | Rarely | Which pages an agent may read, and whether account actions need approval |
| Staging, admin or internal tools | No | Keep the block; allow only production URLs a journey uses |
| Licence-restricted or paywalled content | Decide per licence | Whether the terms permit agent access at all |
The member area row needs the most thought, particularly for a site that already runs ai automation for business flows into a logged-in area. Allowing a read and allowing an action are different permissions: readable data can be opened, while sending a message or placing an order should sit behind an agreed approval and a log. That split also answers the usual objection, because a scraper copying the user agent still fails the IP check.
Topkee builds and runs sites, measures how search engines and AI systems read them, and runs agents of its own, so this update lands in two places.
The first is the crawl layer. Weber sites get an automatically generated robots.txt that can be edited in the Robots.txt document editor and checked at the domain root, for example www.domain.com/robots.txt. That file is often mistaken for the allow switch here, and it is not one: it instructs a crawler that has already reached the site, and a request stopped at the edge never reads it.
The second is measurement. Topkee's website diagnosis checks load speed, mobile compatibility, content originality, backlink quality and user experience, with periodic automated re-checks, and an agent reachability item belongs in that routine. Topkee's AI browsing analysis covers retrieval sources and cited pages, along with basic site readability, and its documentation states the limit plainly: it is not the same as actual site visits. Real crawler access, referral traffic and conversion have to be joined from site logs or analytics.
The short reading for a client site: Topkee does not sell a Google-Agent allowlist, a CDN or a WAF, and nothing in its product material claims to. What it supplies is the part most teams skip, a check of whether the site is readable and a measurement of what AI systems retrieve and cite, re-run on a schedule.
A working rule leaves traces at the origin, where the request either arrives or does not:
Two mistakes are common. The first is checking robots.txt or a rank tracker and calling the job done; neither sees an edge block. The second is treating a visibility dashboard as proof of visits. If the log shows nothing, work backwards through the traps above rather than guess.

Three things will change this answer. The signing experiment comes first: if Web Bot Auth becomes the normal way Google's agents identify themselves, checking a signature will replace trusting a user agent string. The IP file comes second, because a rule that does not resync slowly stops matching. Provider support comes third, since a CDN that verifies Google-Agent natively removes most of the work above.
For the content side of the shift, AI Agent Traffic and Your SEO Plan covers what agent visits mean for a plan. Retail teams watching agent-driven checkout can follow Boost Google Ads conversions with UCP & AI tools. Teams building their own endpoints will find web application development the place where the edge rule and the application meet.
No. Robots.txt tells a crawler what it may index once it has reached the site. A request blocked at the edge never gets that far.
No. Googlebot is the search crawler documented for indexing. Google-Agent is the name for agents hosted on Google's infrastructure that navigate the web and act on a user's request, with its own user agent and IP range file.
Read the origin access log for Google-Agent requests and their status codes. Visibility panels and rank trackers do not show edge blocks.
The rule is scoped to a user agent, the published IP ranges and, where present, a signature. A scraper copying the user agent string fails the IP check, and the other bot controls stay in place for everything outside the scope.

Article Author