Patreon Tackles AI Scraping with New Blockade
In a bid to safeguard creators’ content from AI scraping, Patreon, the membership platform for artists and writers, has strengthened its measures to directly block AI training bots. The company has partnered with Cloudflare, a leading internet infrastructure provider, to implement a comprehensive solution to combat AI scraping.
According to Patreon, AI scraping has become increasingly sophisticated since it first introduced measures to deter AI crawlers in 2023. The platform’s paywall has long restricted access to creators’ content, but the introduction of new discovery tools like a redesigned Home Feed and Quips, a tweet-like feature, has exposed more content to crawlers.
Cloudflare’s AI Crawl Control Technology
Patreon is extending its existing collaboration with Cloudflare to utilize the company’s AI Crawl Control technology. This technology will enable Patreon to update its AI policies and enforcement tools, moving beyond simply asking AI crawlers not to scrape content via the robots.txt file.
As AI agents become more powerful and popular, creators deserve a say in how their work is used by AI companies, according to Patreon’s product chief Drew Rowny. Patreon has a different vision: creators should be able to grow their audience and control how their work is used.
When testing the features, individual AI training crawlers’ weekly attempts to access Patreon dropped from ‘thousands of attempts to zero.’ This indicates that the AI scrapers were ignoring Patreon’s robots.txt file and scraping the site despite its requests.
Patreon’s new measures will allow bots that index pages and organize information to send users back to the platform. This approach prioritizes creators’ content while still enabling useful bots to access the site.
The Importance of Consent in AI Scraping
Patreon emphasizes that consent shouldn’t depend on whether a scraper chooses to behave. The company’s blog post explains that stricter measures are necessary to protect creators’ content from AI scraping.
Patreon’s collaboration with Cloudflare reflects the growing concern among online publishers and content creators about how AI is ingesting their work for training purposes. Cloudflare now offers tools that allow website publishers to restrict AI bots, including a marketplace that lets websites charge AI bots for scraping, dubbed Pay Per Crawl.
Earlier this month, Cloudflare changed its policies to block ‘mixed-use’ crawlers by default on any pages that host ads. This move acknowledges the need for stricter controls on AI scraping and the importance of creators’ consent.
