Guides

• How to block GPTBot, ClaudeBot and other AI crawlers in robots.txt
  Copy-paste robots.txt rules to block AI training crawlers while staying in AI search, what each token does, and what robots.txt can't enforce.
• robots.txt matching rules (RFC 9309): longest match, Allow vs Disallow, wildcards
  How crawlers pick a group and decide between Allow and Disallow, with worked examples, wildcard and $ behavior, and error-status handling.
• XML sitemap limits and errors: 50,000 URLs, 50 MB, lastmod format
  Sitemap protocol limits, sitemap index files, valid lastmod formats and the common errors a sitemap validator flags.