1. What is the “AI Block” feature?
The AI Block is a one-click toggle in your SEO settings that automatically inserts Disallow: / rules into your robots.txt to block a wide range of AI-driven crawlers and scraping bots.
2. Why should I enable AI Block?
Protect Content: Prevents AI services from scraping your proprietary content for model training.
Improve Performance: Reduces unnecessary bot traffic, leading to faster page loads for real users.
Lower Costs: Keeps cloud bandwidth and server-usage fees in check by filtering out high-volume crawlers.
Preserve SEO Budget: Ensures search engines focus their crawl budget on the pages that matter most to your site.
3. How do I turn on the AI Block?
Go to SEO → Crawler-Blocker in your dashboard.
Flip the “AI Block” switch to On.
Click Save.
No manual edits torobots.txtare needed.
5. Will enabling AI Block affect Google or other essential search engines?
No. The AI Block specifically targets known AI and scraping bots. Standard search engine crawlers (e.g., Googlebot, Bingbot) remain unaffected and will continue to index your public content as usual.
6. Does the AI Block interfere with any of our other SEO settings?
No. It operates independently and only manages the robots.txt entries related to AI and scraping bots. All other SEO configurations remain intact.
7. Which user-agents does the AI Block cover?
When enabled, it blocks all of the following by default:
Amazonbot
anthropic-ai
Applebot-Extended
AwarioRssBot & AwarioSmartBot
Bytespider
CCBot
ChatGPT-User
ClaudeBot & Claude-Web
cohere-ai
DataForSeoBot
Diffbot
DuckAssistBot
FacebookBot
FriendlyCrawler
Google-CloudVertexBot & Google-Extended
GPTBot
ImagesiftBot
magpie-crawler
Meta-ExternalAgent & Meta-ExternalFetcher
NewsNow & news-please
OAI-SearchBot
omgili & omgilibot
peer39_crawler
PerplexityBot
Quora-Bot
Scrapy
Timpibot
TurnitinBot
YouBot