How-to Block AI Web Crawlers

Managing who can access your website is an important task for any site owner in today’s AI driven internet. While people and also important for your business search engines like Google and Bing are great visitors, there are many AI web crawlers, bots and scrapers that might be accessing or downloading your site for various reasons, some of which you might prefer to prevent. Blocking AI web crawlers can help protect your content, reduce server load, and maintain control over your intellectual property. This guide will walk you through two straightforward methods to achieve this: using robots.txt and configuring your .htaccess file.

Video Tutorial: Blocking AI Web Crawlers

Why Block AI Web Crawlers?

You might be wondering why you’d want to block certain bots. Here are a few common reasons:

• Protecting content from being scraped for AI model training
• Reducing bandwidth usage and server resources
• Preventing unwanted data collection by competitors or unknown entities
• Maintaining privacy and control over your website’s information

Understanding how to block AI web crawlers gives you more authority over your online presence.

Blocking AI Web Crawlers with robots.txt

The robots.txt file is a simple text file that lives in the root directory of your website. It’s like a polite request to web crawlers (also known as user-agents), telling them which parts of your site they should and shouldn’t access. Most reputable bots will respect these instructions. If you want to prevent specific AI bots from crawling your entire site, you can add entries to your robots.txt file.

How to Implement robots.txt Blocking

You’ll need to create or edit a file named robots.txt in the main folder of your website (e.g., yourwebsite.com/robots.txt). To block a specific AI web crawler from your entire site, you’d use a format like this:

User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Dataprovider.com Disallow: / User-agent: DotBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: MistralAI-User Disallow: / User-agent: anthropic-ai Disallow: / User-agent: AhrefsBot Disallow: / User-agent: YandexBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: claude-web Disallow: / User-agent: meta-externalagent Disallow: / User-agent: GPTBot Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: Google-Extended Disallow: /

Each User-agent line identifies a specific bot. The Disallow: / line tells that particular bot not to access any part of your website. This is an effective way to control many common AI web crawlers.

Remember, robots.txt is a suggestion, not a strict enforcement. Malicious bots might ignore it, but most legitimate AI bots and web crawlers will respect your blocking rules.

Blocking AI Web Crawlers with .htaccess

For more robust control, especially against bots that might ignore robots.txt, you can use your server’s .htaccess file. This method actively denies access based on the User-Agent string reported by the bot. This is a more forceful way to block AI web crawlers and can be very effective.

Understanding .htaccess Blocking

The .htaccess file is a powerful configuration file that controls how your web server (Apache, in most cases) behaves for specific directories. By adding rules to this file, you can instruct the server to deny access to requests originating from certain User-Agents.

How to Implement .htaccess Blocking

You’ll need to locate or create a file named .htaccess in your website’s root directory. Be very careful when editing this file, as incorrect entries can make your website inaccessible. Always back up your current .htaccess file before making changes.

Here’s the code you can add to your Apache server .htaccess file to block a comprehensive list of AI web crawlers:

<IfModule mod_rewrite.c> RewriteEngine On RewriteCond %{HTTP_USER_AGENT} "claude-web|anthropic-ai|OAI-SearchBot|MistralAI-User|AhrefsSiteAudit|Amazonbot|Applebot-Extended|archive\.org_bot|Baiduspider|BLEXBot|Bytespider|CCBot|ChatGPT-User|CipaCrawler|ClaudeBot|coccocbot|Dataprovider\.com|DeepSeekBot|DeuSu|Dispatch|DotBot|EasouSpider|Feedly|GarlikCrawler|Google-CloudVertexBot|GPTBot|ia_archiver|Inoreader|Krzana|LexxeBot|linkdexbot|Mail\.RU|Mail\.RU_Bot|Meta-ExternalAgent|MJ12bot|Nutch|PaperLiBot|PerplexityBot|Perplexity-User|PetalBot|SearchmetricsBot|SemrushBot|SeznamBot|SiteExplorer|Sogou|SurveyBot|YandexBot|ZoominfoBot" [NC] RewriteRule ^.*$ - [F,L] </IfModule>

• <IfModule mod_rewrite.c> and </IfModule> These ensure the rules only apply if the mod_rewrite module is enabled on your server.
• RewriteEngine On This activates the rewrite engine.
• RewriteCond %{HTTP_USER_AGENT} “…” [NC] This line checks the User-Agent string of the incoming request. The long string between the quotes is a list of User-Agents to block, separated by a pipe | symbol. The [NC] means “no case,” so it matches regardless of name capitalization.
• RewriteRule ^.*$ – [F,L] If a User-Agent matches the condition, this rule executes.
• [F] means “Forbidden” and will return a 403 error to the bot.
• [L] means “Last rule” and stops processing further rules.

Using .htaccess provides a more direct and server-level denial of access, making it a very effective method to stop unwanted AI web crawlers from getting to your site.

Choosing the Right Blocking Method

Both robots.txt and .htaccess offer ways to block AI web crawlers. For a simple and polite request, robots.txt is often sufficient for well-behaved bots. For a more aggressive and server-enforced blocking, especially against bots that might not respect robots.txt, the .htaccess method (or even using Web Application Firewall) is highly recommended. Many website owners use a combination for a more comprehensive control.

Regularly reviewing your website’s access logs can help you identify new or unlisted AI web crawlers that you might want to add to your blocking rules.