A couple of weeks ago, I wrote up some of my thoughts on AI replacing traditional search on the internet. My main fear of it is that it would devastate publishers who rely on traffic to earn money from their content and thereby causing the whole industry to essentially collapse.
Fortunately, I’m not the only one to see the problem and, even more fortunately, someone in a position to do something about it has also realized it and has committed to do something about it. It turns out Matthew Prince, CEO of Cloudflare, has introduced a new Cloudflare feature to help publishers:
Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites.
[…]
Matthew Prince, CEO of Cloudflare, said the feature would ensure that the Internet as we know it will survive “the age of AI.”
“Original content is what makes the Internet one of the greatest inventions in the last century, and it’s essential that creators continue making it,” Prince said. “AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone.”
Some participating publishers expressed optimism in the press release that Cloudflare’s pay-per-crawl feature could potentially stop the endless scraping that publishers defending copyrights have alleged represents wide-scale theft.
Ars Technica
I think this is a fantastic idea. Cloudflare is in the unique position to make an initiative like this a success and perhaps even the standard. Publishers do need to get paid in order to continue to produce content.
You might think that AI firms might be against such a thing, but some are even on board with it:
For Cloudflare’s plan to work, AI companies must sign up, too. However, while some AI companies may not see the incentive, Cloudflare has confirmed that it has partnered with AI companies on the initiative, which may benefit from having a simple interface to negotiate with content creators.
Cloudflare suggested its AI partners benefit from “long-term collaboration” with creators whose updated content will help AI products stay relevant. They also can stop wasting money scraping poor quality data sources, a Cloudflare blog said.
“Without ongoing contributions from content creators, AI systems risk becoming outdated, biased, or less reliable—ultimately diminishing user trust and the value of AI products,” the blog said. “Cloudflare is working with AI companies to give them more signals, and ultimately improve the quality and relevance of content they can access. A healthy, sustainable ecosystem of original content is critical for AI innovation and relevance.”
Ars Technica
The emphasis is mine. It seems that at least some AI companies see the benefit of being able to negotiate access to publisher’s content through one channel rather than with each publisher separately. Aside from that, I can also imagine some AI companies are interested in preventing the onslaught of copyright violation lawsuits that have been working their way through the court systems of multiple countries.
Overall, I’m excited to see where this goes, if anywhere. It’s still very early with the process of trying to figure out where this is all going, so concepts like this will come and go and the ones that stick around will evolve.
There is plenty more to read about the topic on Ars Technica which I have quoted above. I can recommend it.
Otherwise, if you are a Cloudflare user and interested in the program, here is the link to the signup form for the beta: https://www.cloudflare.com/paypercrawl-signup/.
