fix: add missing ai crawlers and correct omgili spelling#307
Open
hlassiege wants to merge 1 commit into
Open
Conversation
The AiBots list had not kept up with the crawlers vendors actually run. ClaudeBot, the crawler Anthropic uses today, was absent: the list only carried the legacy Claude-Web and anthropic-ai names, so blockAiBots did not block Anthropic at all. Same for OAI-SearchBot, meta-externalagent, Amazonbot, MistralAI-User, DeepSeekBot and others. Omigili/OmigiliBot were misspellings of omgili/omgilibot (Webz.io) and matched no real user agent. Names now follow the ai.robots.txt reference list, restricted to AI crawlers. Search crawlers stay out by design, so blockAiBots never costs search visibility: Googlebot, Applebot (Siri, Spotlight) and facebookexternalhit (Open Graph previews) are deliberately absent, while Google-Extended and Applebot-Extended are listed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
commit: |
Author
|
FYI, the PR has been made with Claude. But I checked the problem and validated the fix on my own project |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🔗 Linked issue
None, found while auditing the generated
robots.txtof a downstream project.❓ Type of change
📚 Description
AiBotshad not kept up with the crawlers vendors actually run, soblockAiBots: truesilently under-delivers today.ClaudeBotwas missing. The list only carries the legacyClaude-Webandanthropic-ainames, soblockAiBots: truedoes not block Anthropic's current crawler at all. This is the one that matters most: users enabling the option reasonably believe Claude is covered, and it is not. Same story forOAI-SearchBot,meta-externalagent,Amazonbot,MistralAI-User,DeepSeekBotand a few others.Omigili/OmigiliBotare misspellings. The real Webz.io agents areomgili/omgilibot(note the transposedi). Matching is case-insensitive, but this is an extra letter, not a case difference, so both entries match no real user agent. Replaced with the correct spelling.Names now follow ai.robots.txt, which seems to be the de-facto reference list.
Search crawlers stay out, by design
I kept the list to AI-only agents so that
blockAiBotsnever costs a site its search visibility or its social previews. Three deliberate omissions, now documented in a comment so they don't get "fixed" later:Googlebot— only Google's AI surfaces are listed, so the site stays indexedApplebot— powers Siri and Spotlight; only its AI counterpartApplebot-Extendedbelongs herefacebookexternalhit— fetches Open Graph previews; blocking it kills link previews on Facebook and WhatsAppScope
This is a curated subset, not all ~180 entries of ai.robots.txt: I left out the coding agents (Cursor, Devin, ...) and generic scrapers, which feel closer to
NonHelpfulBotsthan toblockAiBots. Happy to trim or extend to taste.GoogleOtheris the one judgment call I would flag: ai.robots.txt lists it and Google documents it as serving R&D, but it is more general-purpose than the rest, so say the word if you would rather drop it.No test added: the change is data-only, and an inline snapshot of the full list would need updating on every future addition for little value. Glad to add one if you disagree.
📝 Checklist