Robots.txt & AI Crawler Checker API
Before fetching a page, call GET /v1/robots-check?url=<page>&user_agent=<your bot token> to get allowed true/false, the deciding rule and crawl-delay (RFC 9309). GET /v1/ai-crawlers?url= returns per-bot verdicts for 20 AI/search crawlers plus meta robots, X-Robots-Tag and llms.txt. GET /v1/sitemap-check?url= finds (via robots.txt) and validates a sitemap. Deterministic, no AI.
How to pay
Every /v1 endpoint answers an unpaid request with HTTP 402 and a PAYMENT-REQUIRED header (x402 v2, scheme exact, network eip155:8453, USDC, payTo 0x076e653F4f458201B866B6B46A50F5796C345fb7, facilitator https://facilitator.payai.network, fallback https://facilitator.xpay.sh). Any x402 client (for example @x402/fetch) pays and retries automatically. No API key or account.
Machine-readable: /openapi.json · /.well-known/x402
GET /v1/robots-check — $0.001 USDC
Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ties, * and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI.
| Param | Required | Description |
|---|---|---|
url | yes | Page URL to test Example: https://www.nytimes.com/section/technology |
user_agent | no | Crawler product token, e.g. GPTBot or MyAgent. Default * Example: GPTBot |
Example request: https://robots-check.quietworklabs.workers.dev/v1/robots-check?url=https%3A%2F%2Fwww.nytimes.com%2Fsection%2Ftechnology&user_agent=GPTBot
Example response
{
"url": "https://example.com/private/page",
"userAgent": "GPTBot",
"robotsTxt": {
"status": "ok",
"httpStatus": 200
},
"allowed": false,
"matchedGroup": "gptbot",
"matchedRule": "Disallow: / (line 4)",
"crawlDelay": null,
"sitemaps": [
"https://example.com/sitemap.xml"
]
}GET /v1/ai-crawlers — $0.002 USDC
AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI.
| Param | Required | Description |
|---|---|---|
url | yes | Site or page URL Example: https://www.theguardian.com/ |
Example request: https://robots-check.quietworklabs.workers.dev/v1/ai-crawlers?url=https%3A%2F%2Fwww.theguardian.com%2F
Example response
{
"url": "https://example.com/",
"summary": {
"blocked": [
"GPTBot",
"CCBot"
],
"trainingBlocked": [
"GPTBot"
],
"searchBlocked": []
},
"bots": [
{
"token": "GPTBot",
"operator": "OpenAI",
"purpose": "training",
"allowed": false,
"matchedGroup": "gptbot",
"matchedRule": "Disallow: / (line 4)"
}
],
"page": {
"httpStatus": 200,
"xRobotsTag": null,
"metaRobots": [],
"noindex": false,
"noai": false
},
"llmsTxt": {
"present": true
}
}GET /v1/sitemap-check — $0.002 USDC
Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI.
| Param | Required | Description |
|---|---|---|
url | yes | Site URL (discovers the sitemap) or a sitemap URL Example: https://www.python.org/ |
Example request: https://robots-check.quietworklabs.workers.dev/v1/sitemap-check?url=https%3A%2F%2Fwww.python.org%2F
Example response
{
"sitemapUrl": "https://example.com/sitemap.xml",
"discoveredFrom": "robots.txt",
"httpStatus": 200,
"type": "urlset",
"locCount": 1240,
"lastmodRange": {
"oldest": "2024-02-01",
"newest": "2026-10-15"
},
"problems": [],
"warnings": [
"3 duplicate <loc> entries"
],
"status": "ok"
}