Don’t block the bots. Build th... Note
Fast Company

Don’t block the bots. Build the gate

Most media companies are blocking AI company crawlers, viewing them as a drain on resources without providing significant human traffic. This binary approach of "block or don't block" is too simplistic for the complex issue of AI integration. While blocking unauthorized scrapers is necessary, publishers should consider allowing specific types of AI bots, like retrieval bots, which fetch information for user queries. Failing to appear in AI-generated answers cedes authority to competitors, making presence in AI summaries a matter of establishing authority rather than immediate traffic gain. Publishers should balance allowing AI to discover their content with protecting their most valuable assets and requiring deeper engagement for access.Treating AI bots strategically means understanding their different functions, such as training, search, and retrieval. Retrieval bots, essential for real-time AI answers, are generally low-impact and can be managed with rate limits or cached content. The current market shift is less about content for training models and more about licensing content for delivering immediate answers to users. Publishers can make their content machine-readable and accessible through controlled channels, rather than solely relying on licensing deals.Creating AI-ready archives and controlled retrieval layers empowers publishers to negotiate from a position of strength. Instead of solely blocking, publishers should build "gates" with clear rules, differentiating content for discovery versus access. By managing metadata, snippets, and access controls, publishers can ensure AI is aware of their value without giving it away entirely. The ultimate goal is to be legible to AI systems without making content free, positioning publishers as trusted sources when AI needs to provide answers.
CdXz5zHNQW_tJ9V4iivjU.jpeg