Singapore
Google-Agent: allow Google's agent through CDN and WAF

Google-Agent: allow Google's agent through CDN and WAF

As of November 2026, Google has published the answer it used to leave to guesswork: whether traffic from its own agents should be let through a site's edge. On November 17, 2026, Google added a technical guide explaining how to allow Google-Agent in a content delivery network, a web application firewall or a bot-detection service. For a Singapore team whose ai automation for business work ends on a site it owns, that guide is now the checklist the edge configuration has to pass.

Key takeaways:

  • The guidance names three identifiers: the Google-Agent user agent, an IP range file Google publishes, and an optional signed request.
  • Google-Agent is not new. Google added the user agent to its documentation on March 20, 2026; the November 17, 2026 change is the allow-traffic guidance on top of it.
  • A request stopped at the edge never reaches robots.txt, so a permissive robots.txt says nothing about agent access.

What changed on November 17, 2026

Google's crawling changelog records that on November 17, 2026 it published a technical guide on making sure Google-Agent has access to a website, for publishers that want to enable the agent. The guide covers how the agent identifies its requests, how to allow that traffic in a CDN, firewall or bot-detection service, and troubleshooting.

Read the timeline in order. On March 20, 2026, Google added the Google-Agent user agent and the IP ranges for user-triggered agents to its documentation, and said the agent would roll out over the following weeks. The November 17, 2026 guide answers what that rollout left with site owners, and frames itself narrowly: guidance for publishers who want the agent to reach them, not a requirement for every site.

Why Google published the guidance

An agent hosted on Google's infrastructure fetches a page because a person asked it to do something, and the guide describes Google-Agent as navigating the web and performing actions on that request. From a firewall's point of view it looks like an automated client: no browser session a human started on your domain, no cookie your site set, a user agent most analytics tools do not classify.

Bot management treats the pattern as suspicious because most traffic matching it is scraping. A managed rule challenges it, a category rule blocks it, a cloud or data-centre ASN block catches it again, and together those defaults also stop the agent a customer asked to use your site. The aim Google states is to prevent legitimate requests from being blocked while keeping existing controls, which is narrower than switching protection off.

What the guide does not give you is scale: Google does not say how many requests its agent loses to blocked sites. The direction is clear all the same. An agent that can search, filter, add to cart and book is part of the journey, and a block there is a lost transaction rather than a lost impression.

An AI assistant on a shopper's phone halted at a closed gate in front of a lit online store.

How to allow Google-Agent traffic in your CDN or WAF

The allow rule belongs at the edge, in front of the origin. For most ai automation for business teams the rule sits in a console they rarely open, so the placement matters: the edge evaluates the request before any page is served, and the site cannot grant access to a request the edge has rejected.

Google gives three ways to identify the agent: a User-Agent header containing the string Google-Agent, an address in the user-triggered-agents.json file Google publishes, and, in an experiment, a signature checked against a published key set.

Where a provider does not verify Google-Agent on its own, the guide sets out four rules in order:

  1. Confirm that the User-Agent header contains Google-Agent.
  2. Confirm that the request's IP address is in user-triggered-agents.json.
  3. If a signature is present, verify the Signature and Signature-Input headers for the agent identity Google publishes.
  4. For requests that pass verification, skip the bot challenge.

The fourth rule is the one half-applied: a team adds an allow rule, sees the challenge still fire, and concludes the rule failed.

Check a signed request before trusting it

Google's experiment uses Web Bot Auth. Signed requests name the identity https://agent.bot.goog in a Signature-Agent header, label their Signature and Signature-Input headers with the letter g, and are checked against the HTTP Message Signatures standard, RFC 9421, using public keys in a well-known directory under agent.bot.goog.

Signing may spread, or the labels may change as the protocol settles. If your edge verifies signatures already, switch it on; if you built the check, keep the key source configurable.

Traps that survive a correct allow rule

Google's troubleshooting section reads as a list of places where a correct rule still loses:

  • Rule priority. An allow rule helps only if it evaluates before broad ASN blocks, managed bot rules and category-level AI agent block policies.
  • Interactive challenges. In-page CAPTCHA widgets fired on search or form submission also stop the agent.
  • Actions beyond the homepage. On-site search, filtering, add-to-cart and booking need the same treatment, including the XHR, GraphQL or api.* endpoints behind them.
  • Shared IP limits. Concurrent requests from many users leave through a shared pool, so a per-IP limit tuned for one visitor can throttle them all.
  • IP range updates. Rules built on the file go stale when Google edits it, so sync them on a schedule.
  • Proxy headers. Load balancers can strip the client IP, the User-Agent header and the signature headers.
A small glowing marker passing through a gap above stacked translucent barriers in a narrow corridor.

Who the change touches in a Singapore team

The person who owns the consequence is rarely the person who owns the setting. In most Singapore setups the site is built and hosted by an agency, a website builder or group IT, so the marketing lead hears about a block from a sales report rather than a console. The change needs one conversation this week with whoever holds the CDN account.

Start with the journeys where an agent does something a person would otherwise do: adding to a cart, picking a clinic slot, filling a B2B enquiry form. That request arrives without a session your analytics can attribute, so a block leaves no referral and no error. For teams running web application development in-house, the edge rule and the application's own bot logic must agree, or one undoes the other.

Which sites should allow Google-Agent

Allowing the agent is a decision per site, not a company-wide policy. This is the table for that conversation.

Your siteAllow Google-Agent?What to check first
Online store with a cartYes, scopedBot rules on cart, checkout and the api.* endpoints behind them
Booking or appointment flowYes, scopedThe challenge that fires on date selection and form submission
Brochure site with an enquiry formYes, after checkingShared-IP rate limits that can throttle a small egress pool
Member area behind a loginRarelyWhich pages an agent may read, and whether account actions need approval
Staging, admin or internal toolsNoKeep the block; allow only production URLs a journey uses
Licence-restricted or paywalled contentDecide per licenceWhether the terms permit agent access at all

The member area row needs the most thought, particularly for a site that already runs ai automation for business flows into a logged-in area. Allowing a read and allowing an action are different permissions: readable data can be opened, while sending a message or placing an order should sit behind an agreed approval and a log. That split also answers the usual objection, because a scraper copying the user agent still fails the IP check.

What Topkee's site work and AI-visibility measurement imply

Topkee builds and runs sites, measures how search engines and AI systems read them, and runs agents of its own, so this update lands in two places.

The first is the crawl layer. Weber sites get an automatically generated robots.txt that can be edited in the Robots.txt document editor and checked at the domain root, for example www.domain.com/robots.txt. That file is often mistaken for the allow switch here, and it is not one: it instructs a crawler that has already reached the site, and a request stopped at the edge never reads it.

The second is measurement. Topkee's website diagnosis checks load speed, mobile compatibility, content originality, backlink quality and user experience, with periodic automated re-checks, and an agent reachability item belongs in that routine. Topkee's AI browsing analysis covers retrieval sources and cited pages, along with basic site readability, and its documentation states the limit plainly: it is not the same as actual site visits. Real crawler access, referral traffic and conversion have to be joined from site logs or analytics.

The short reading for a client site: Topkee does not sell a Google-Agent allowlist, a CDN or a WAF, and nothing in its product material claims to. What it supplies is the part most teams skip, a check of whether the site is readable and a measurement of what AI systems retrieve and cite, re-run on a schedule.

How to prove the allow change worked

A working rule leaves traces at the origin, where the request either arrives or does not:

  1. Read the access log for requests whose User-Agent contains Google-Agent, and note the timestamps.
  2. Check the status codes those requests received, and whether they cluster on one path or spread across the journey.
  3. Repeat on the endpoints behind the journey, including search, filtering and any api.* route.
  4. Compare volumes before and after, over a window long enough to cover a normal week.
  5. Only then open Search Console, remembering its generative AI report covers AI Overviews and AI Mode impressions rather than agent requests.

Two mistakes are common. The first is checking robots.txt or a rank tracker and calling the job done; neither sees an edge block. The second is treating a visibility dashboard as proof of visits. If the log shows nothing, work backwards through the traps above rather than guess.

An engineer at a dim desk reads scrolling server logs on a monitor, notebook of timestamps beside the keyboard.

What to watch next

Three things will change this answer. The signing experiment comes first: if Web Bot Auth becomes the normal way Google's agents identify themselves, checking a signature will replace trusting a user agent string. The IP file comes second, because a rule that does not resync slowly stops matching. Provider support comes third, since a CDN that verifies Google-Agent natively removes most of the work above.

For the content side of the shift, AI Agent Traffic and Your SEO Plan covers what agent visits mean for a plan. Retail teams watching agent-driven checkout can follow Boost Google Ads conversions with UCP & AI tools. Teams building their own endpoints will find web application development the place where the edge rule and the application meet.

A checklist to run today

  1. Find out who controls the CDN, WAF or bot-management configuration.
  2. Confirm whether any rule blocks AI agents by category, and whether it runs before an allow rule.
  3. Add the allow rule for the Google-Agent user agent and the published IP file, above the broad blocks.
  4. Check challenge settings and list the endpoints behind your main journey.
  5. Review per-IP rate limits and put the IP file sync on a schedule.
  6. Read the origin log a week later and keep the evidence.

FAQ

Does robots.txt allow Google-Agent?

No. Robots.txt tells a crawler what it may index once it has reached the site. A request blocked at the edge never gets that far.

Is Google-Agent the same as Googlebot?

No. Googlebot is the search crawler documented for indexing. Google-Agent is the name for agents hosted on Google's infrastructure that navigate the web and act on a user's request, with its own user agent and IP range file.

How do I know the change worked?

Read the origin access log for Google-Agent requests and their status codes. Visibility panels and rank trackers do not show edge blocks.

Will allowing Google-Agent let scrapers through?

The rule is scoped to a user agent, the published IP ranges and, where present, a signature. A scraper copying the user agent string fails the IP check, and the other bot controls stay in place for everything outside the scope.

Sources

  1. "Google Search Help Document On How To Allow Google-Agent Traffic", Search Engine Roundtable, 2026-11-17. https://www.seroundtable.com/google-agent-bot-42559.html
  2. "Allow Google-Agent traffic", Google Crawling documentation, 2026-11-17. https://developers.google.com/crawling/docs/crawlers-fetchers/allow-google-agent
  3. "Changelog", Google Crawling documentation, 2026-11-17. https://developers.google.com/crawling/docs/changelog
Category:
Share to:
Date: 2026-11-18
Pang Sir

Article Author

Pang Sir