Cloudflare Blocks AI Crawlers by Default, Launches Pay Per Crawl Model
Introduction
Cloudflare, which services about 20% of websites, announced Tuesday that it is now blocking web-scraping AI bots from accessing those sites by default. Unless a website explicitly turns off the default, an AI crawler will need to obtain permission from the website to scrape its content. Website owners can choose whether they want AI crawlers to access their content, and decide how AI companies can use it, Cloudflare explained in a statement. AI companies can now clearly state their purpose, whether their crawlers are used for training, inference, or search, to help website owners decide which crawlers to …