Crawlux launches Web3 Robots.txt Checker, finds 67% of crypto sites block AI bots accidentally

LONDON · APRIL 7, 2026

Crawlux today launches the Web3 Robots.txt Checker, a free tool that validates a crypto site's robots.txt against current AI bot policy. The tool checks for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended and 8 other AI crawler user-agents that crypto sites typically need to allow.

Companion data analysis: Crawlux scanned the robots.txt of 207 crypto sites. 139 (67%) block at least one major AI bot. 73 (35%) block GPTBot specifically. 41 (20%) block ClaudeBot. 28 (14%) block PerplexityBot. Only 68 sites (33%) allow all major AI crawlers cleanly.

Most blocks are accidental. Cloudflare's default AI bot ruleset (turned on by default in early 2025) blocks GPTBot and ClaudeBot at the edge unless explicitly disabled. Vercel's template-based deployments include a robots.txt that disallows AI crawlers as a starting default. Teams copy these configs from documentation without realizing the implication.

The cost of accidental blocking is direct. AI engines that cannot crawl your site cannot cite your site. A protocol blocking GPTBot has zero ChatGPT citation rate by definition, regardless of how clean its schema is or how strong its backlink profile is. The mechanism is mechanical: no crawl, no index, no citation.

The Web3 Robots.txt Checker validates a submitted domain in 2 seconds. Output flags each AI bot policy as allow, deny, partial (subset of paths blocked) or missing. The tool generates a recommended robots.txt block that allows the standard AI crawler set while maintaining any site-specific deny rules the team intends to keep.

The recommended robots.txt template covers 13 AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider and Meta-ExternalAgent. The list updates quarterly as new bots ship. The full reasoning for each inclusion is documented at crawlux.com/blog/crawlux-methodology.

The Web3 Robots.txt Checker joins the free-tools suite at crawlux.com/free-tools. Each IP gets unlimited domain checks at no cost. For continuous monitoring of robots.txt drift, Crawlux Pro tracks robots.txt changes daily and alerts on any new bot-deny additions.

A practical example of accidental blocking surfaced repeatedly in the 207-site scan. A protocol enables Cloudflare's "AI Scrapers and Crawlers" managed ruleset thinking it will block unwanted content scrapers. The ruleset blocks the legitimate AI search crawlers (GPTBot, ClaudeBot, PerplexityBot) at the edge before the request reaches the origin server. The site's robots.txt allows the bots correctly. The requests never get there. The site shows up in our scan as "robots.txt allows all" but the edge layer denies the crawl. Crawlux Pro detects this by attempting an actual crawl as the bot user-agent and comparing the response to what robots.txt suggests should happen.

The fix for the Cloudflare case is one toggle in the dashboard. Turn off "AI Scrapers and Crawlers" in the WAF managed rules. Apply more targeted blocks (specific user-agents or IP ranges) if scraper concerns persist. For Vercel deployments, replace the template robots.txt with the recommended Web3 version and confirm no _redirects or middleware rules intercept the bot requests. Most teams ship the full fix in under 30 minutes.

Crawlux is the world's first automated SEO audit tool built for Web3, DeFi and blockchain. The platform runs 23 analyzers across 6 check groups including AI visibility testing across ChatGPT, Perplexity and Claude. Free tier available. Paid tiers from $25 per audit. More at crawlux.com.