Search Engine Journal Follow Cloudflare Will Write Your Robots.txt, And It Has A Point Bot Preference Sync sets AI crawler policy per category, not per crawler. Allow GPTBot and block Bytespider, and no setting describes what you do. https://www.searchenginejournal.com/cloudflare-will-write-your-robots-txt-and-it-has-a-point/589262/ searchenginejournal.com SEO and SMM News on Bluesky @seo.at.thenote.app bsky.app
