ai crawlers.

AI crawlers such as GPTBot, ClaudeBot and Google-Extended collect web content for AI models and AI search. Learn how AI crawlers read your website and how to allow or block them.