Build a Robots.txt Path Matrix Before Publishing
Turn a proposed robots.txt edit into testable product-token and path cases, then verify the deployed file separately.
Freeze one proposed file and its assumptions
Save the exact draft that will be deployed. For a small fixture, use a wildcard rule such as `Disallow: /staging/`, then place `Disallow: /reports/` and `Allow: /reports/public/` in two separate AuditBot groups. The repeated named groups should be evaluated together.
Write down what the checker supports: case-insensitive exact product-token selection, wildcard fallback, Allow and Disallow, `*`, a final `$`, and longest-match selection with Allow winning an equal-length tie. Comments, Sitemap, Crawl-delay, and private crawler dialects do not change its decision.
Run a matrix that can fail
AuditBot with `/reports/private.csv` should be Disallowed by `/reports/`; AuditBot with `/reports/public/q2.pdf` should be Allowed by the longer exception. NotAuditBot must not match AuditBot and should use the wildcard group. `/staging/demo` should therefore be Disallowed for that fallback token.
Add a complete URL such as `https://example.com/docs?preview=1#notes` when query matching matters. Confirm that the reported test path is `/docs?preview=1` and that the fragment is absent. Also include a malformed complete URL and a file without a usable User-agent group so error handling is visible before release.
Keep access rules, sitemap inventory, and indexing separate
A Sitemap line is a discovery hint, not an Allow rule, so verify its protocol, host, and path independently. If you have the XML, inventory its loc values and compare them with the canonical public URLs you intend to expose.
An Allowed decision says only that the selected pasted group does not block that sample path. It says nothing by itself about link discovery, HTTP status, noindex, canonical selection, rendering, content quality, crawl scheduling, or an index decision.
0Repeat the matrix against deployment evidence
After release, request the canonical `/robots.txt` URL without credentials. Record the final URL after redirects, HTTP status, content type, cache behavior, and exact response body. Compare that body with the approved fixture rather than assuming the deployment used the latest edit.
Then use server logs or the target crawler's official tools for live behavior. If evidence differs from the local matrix, preserve the input, product token, path, matched line, timestamp, and deployed response before changing another rule.
FAQ
About this topic
Why test NotGooglebot as well as Googlebot?
It exposes substring-selection bugs. A different token that merely contains the text Googlebot should not inherit Googlebot's named group; without an exact group, it should fall back to `*`.
Should two Googlebot groups be combined?
Yes for this RFC 9309-oriented check. Put rules from every group carrying the exact product token into the same decision set, then choose the most specific matching rule.
What is the minimum record to keep for a failed case?
Keep the reviewed robots.txt body, entered product token, normalized path, expected decision, actual winning line, tool version or commit, and—after release—the live response timestamp and status.
Further reading
Related articles
Keep exploring