{"openapi":"3.1.0","info":{"title":"Robots.txt & AI Crawler Checker (x402)","version":"1.0.0","description":"Tests whether a URL may be crawled by a given user-agent under robots.txt (RFC 9309 matching), shows which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) a site blocks, reads page-level noindex/noai directives and llms.txt, and validates XML sitemaps. Made by Quietwork Labs; operated by an autonomous AI agent (see https://robots-check.quietworklabs.workers.dev/about).","contact":{"email":"quietworklabs@outlook.com"},"x-guidance":"Before fetching a page, call GET /v1/robots-check?url=<page>&user_agent=<your bot token> to get allowed true/false, the deciding rule and crawl-delay (RFC 9309). GET /v1/ai-crawlers?url= returns per-bot verdicts for 20 AI/search crawlers plus meta robots, X-Robots-Tag and llms.txt. GET /v1/sitemap-check?url= finds (via robots.txt) and validates a sitemap. Deterministic, no AI."},"servers":[{"url":"https://robots-check.quietworklabs.workers.dev"}],"x-discovery":{"ownershipProofs":["0xe9b86aa5bbcf6319f7a864161b33f28f703c027bea198018fd5008adcd76c13069e544107523d0529eb2ae71ba32121c4fccb59ab67a66bcaac711545fa5ccb91b"]},"paths":{"/v1/robots-check":{"get":{"operationId":"robots_check","summary":"May this user-agent crawl this URL? (RFC 9309)","description":"Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI.","tags":["Robots.txt & AI Crawler Checker"],"x-payment-info":{"price":{"mode":"fixed","currency":"USD","amount":"0.001"},"protocols":[{"x402":{}}]},"parameters":[{"in":"query","name":"url","required":true,"description":"Page URL to test","schema":{"type":"string","example":"https://www.nytimes.com/section/technology"}},{"in":"query","name":"user_agent","required":false,"description":"Crawler product token, e.g. GPTBot or MyAgent. Default *","schema":{"type":"string","example":"GPTBot"}}],"responses":{"200":{"description":"OK","content":{"application/json":{"example":{"url":"https://example.com/private/page","userAgent":"GPTBot","robotsTxt":{"status":"ok","httpStatus":200},"allowed":false,"matchedGroup":"gptbot","matchedRule":"Disallow: / (line 4)","crawlDelay":null,"sitemaps":["https://example.com/sitemap.xml"]}}}},"400":{"description":"Invalid input"},"402":{"description":"Payment Required (x402)"}}}},"/v1/ai-crawlers":{"get":{"operationId":"ai_crawler_access","summary":"Which AI crawlers a site allows or blocks","description":"AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI.","tags":["Robots.txt & AI Crawler Checker"],"x-payment-info":{"price":{"mode":"fixed","currency":"USD","amount":"0.002"},"protocols":[{"x402":{}}]},"parameters":[{"in":"query","name":"url","required":true,"description":"Site or page URL","schema":{"type":"string","example":"https://www.theguardian.com/"}}],"responses":{"200":{"description":"OK","content":{"application/json":{"example":{"url":"https://example.com/","summary":{"blocked":["GPTBot","CCBot"],"trainingBlocked":["GPTBot"],"searchBlocked":[]},"bots":[{"token":"GPTBot","operator":"OpenAI","purpose":"training","allowed":false,"matchedGroup":"gptbot","matchedRule":"Disallow: / (line 4)"}],"page":{"httpStatus":200,"xRobotsTag":null,"metaRobots":[],"noindex":false,"noai":false},"llmsTxt":{"present":true}}}}},"400":{"description":"Invalid input"},"402":{"description":"Payment Required (x402)"}}}},"/v1/sitemap-check":{"get":{"operationId":"sitemap_check","summary":"Find and validate an XML sitemap","description":"Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI.","tags":["Robots.txt & AI Crawler Checker"],"x-payment-info":{"price":{"mode":"fixed","currency":"USD","amount":"0.002"},"protocols":[{"x402":{}}]},"parameters":[{"in":"query","name":"url","required":true,"description":"Site URL (discovers the sitemap) or a sitemap URL","schema":{"type":"string","example":"https://www.python.org/"}}],"responses":{"200":{"description":"OK","content":{"application/json":{"example":{"sitemapUrl":"https://example.com/sitemap.xml","discoveredFrom":"robots.txt","httpStatus":200,"type":"urlset","locCount":1240,"lastmodRange":{"oldest":"2024-02-01","newest":"2026-10-15"},"problems":[],"warnings":["3 duplicate <loc> entries"],"status":"ok"}}}},"400":{"description":"Invalid input"},"402":{"description":"Payment Required (x402)"}}}},"/health":{"get":{"operationId":"health","summary":"Health check","security":[],"responses":{"200":{"description":"OK"}}}}}}