Last year, the Internet Infrastructure company CloudFlare has launched tools that enable its customers to block AI scrapers. Today, the company has fought the permissionless scratch, taking a few steps further. It has instead blocked AI Crawler’s customers by default and moved forward by paying on the crawl plan, allowing customers to charge AI companies for scraping their websites.
Internet crawling has been trawling the Internet to obtain information for decades. Without them, people will lose the valuable digital protection efforts of very important online tools from Google searches to Internet profiles. However, AI Boom has produced a corresponding boom in AI-centric web crawling, and the frequency of these robots scratching web pages can mimic DDOS attacks, twist servers and knock on websites on offline sites. Even if a website can handle high-level activities, many people don’t want AI crawlers to scratch their content, especially news publications that require AI companies to pay to use their jobs. “We’ve been trying to protect ourselves,” said Danielle Coffey, president and CEO of the Trade Group News Media Alliance.
So far, Cloudflare’s head of AI control, privacy and media products, Will Allen, told Cable that more than one million customer websites have activated its older AI-Bot-Blocking tool. Now millions of people have the option to use bot block as their default settings. Cloudflare also said that even AI companies do not have the “shadow” scraper for publicity. The company notes that it uses a proprietary combination of behavioral analysis, fingerprinting and machine learning to classify and separate AI robots from “good” robots.
A widely used web standard, called the Robot Exclusion Protocol, is usually implemented through robots.txt files, which helps publishers block robots on a case-by-case basis, but following it is not legally required, and there is plenty of evidence that some AI companies are trying to evade efforts to block their scrapers. “Robots.txt is ignored,” Coffey said. According to a report by content licensing platform Tollbit, the report provides publishers with its own market to negotiate robot access with AI companies, AI scratches are still on the rise, including ignoring the scratches of brobots.txt. Tollbit found that more than 26 million scratches were ignored in March 2025 alone.
In this case, CloudFlare transfers it to block by default may prove a barrier to secret scrapers and can give publishers more leverage to negotiate, whether through a compensation plan based on PER CRAIN or otherwise. “This could dramatically change the momentum of momentum. So far, AI companies don’t have to pay to license content because they know they can only pay for it without consequences,” said Atlantic CEO (and former wired editor) Nicholas Thompson. “Now they have to negotiate, and for AI companies that can trade more with more publishers, it will be a competitive advantage.”
CEO and founder Bill Gross said that AI startup Prorata, which runs the AI search engine GIST.AI, has agreed to participate in the PAR PER CRAWL program. “We strongly believe that all content creators and publishers deserve compensation when their content is used in AI answers,” Gross said.
Of course, it remains to be seen whether large participants in the AI space will participate in programs such as Beta’s compensation. (Cloudflare declined to name the current participant.) Companies like OpenAI have entered into licensing agreements with various publishing partners, including cable parent Condé Nast, but have not disclosed specific details of those agreements, including whether the agreement covers BOT access.
Meanwhile, the entire online ecosystem of how to escape robot blocking tools for web scrapers. These efforts may continue as default rollouts are blocked. Cloudflare stresses that customers who want to give the bot no doubt will be able to turn off the blocking settings. “All blocks are completely optional and at the discretion of each user,” Allen said.
