Robots.txt & AI Crawler Checker

robots.txt matching rules

  1. Pick the group. Find groups whose User-agent equals the crawler's product token, case-insensitively (GPTBot = gptbot). Multiple matching groups are merged. If none match, use User-agent: *. If there's no * group, everything is allowed.
  2. Find matching rules. A rule matches if its path is a prefix of the URL path (plus query). * matches any characters; $ anchors the end.
  3. Longest wins. The rule with the most characters wins. If an Allow and a Disallow are equally long, Allow wins.
  4. Nothing matches = allowed. Disallow: with an empty value allows everything.

Examples

User-agent: *
Disallow: /shop/
Allow: /shop/public/
Disallow: /*.pdf$
PathResultWhy
/shop/cartblockedDisallow: /shop/
/shop/public/aallowedAllow: /shop/public/ is longer
/docs/x.pdfblocked/*.pdf$
/docs/x.pdf?v=2allowed$ requires the path to end in .pdf

When robots.txt can't be fetched

4xx (except 429): crawl freely. 5xx, 429 or network failure: assume complete disallow. Crawlers may cache robots.txt for up to 24 hours, and must parse at least 500 KiB.

Test a URL · API: GET /v1/robots-check?url=&user_agent=.

Source: RFC 9309.