humancollab · Answers

Should I block AI crawlers like GPTBot in robots.txt?

It depends on what you want. Blocking a crawler stops that company from reading your site for that purpose, which can also mean its AI knows less about you. Most AI companies use separate crawlers for training and for live search, so you can choose per purpose.

How

1. Search and live answers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
2. Model training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
3. If you want AIs to recommend you, allow at least the search crawlers; decide on training separately.

Limits

• robots.txt is a request; well-behaved crawlers follow it, others may not.
• Blocking training today does not remove what models already learned.

EnglishTürkçeEspañolPortuguêsFrançaisDeutsch日本語中文العربيةहिन्दी

Machine documentation: /llms.txt