A couple of days ago, I stumbled upon an interesting article on TechDirt about how many websites have started to wall themselves off from the open internet in order to prevent AI bots from scraping their content. I have thoughts about it.
Generally, I’ve been a big fan of this idea simply because I’ve been afraid of what’s going to happen when people who create content online no longer get as many readers or viewers. However, when I say I’ve been a fan of the idea, I don’t mean that I want paywalls or logins, but rather more technical solutions to stave off the hoards of training bots coursing through cyberspace at the moment.
The article from TechDirt raised a very good point though in that, by adding barriers, we are essentially walling off large swaths of the internet just to hurt AI companies. If we take that too far, we are violating the very princple we are trying to protect.
A longtime open internet activist recently asked me whether I’d reversed my position on internet openness and copyright because of AI. The question caught me off guard—until I realized what he was seeing. Across the tech policy world, people who spent decades fighting for an open, accessible internet are now cheering as that same internet gets locked down, walled off, and restricted. Their reasoning? If it hurts AI companies, it must be good.
This is a profound mistake that threatens the very principles these advocates once championed.[…]
The problem isn’t just ideological—it’s practical. We’re watching the construction of a fundamentally different internet, one where access is controlled by gatekeepers and paywalls rather than governed by open protocols and user choice. And we’re doing it in the name of stopping AI companies, even though the real result will be to concentrate even more power in the hands of those same large tech companies while making the internet less useful for everyone else.
The shift toward a closed internet shifted into high gear, to some extent, with Cloudflare launching its pay-per-crawl feature. I will admit that when I first saw this announcement, it intrigued me. It would sure be nice for Techdirt if we suddenly started getting random checks from AI companies for crawling the more than 80k articles we’ve written that are then fueling their LLMs.
But, also, I recognize that even having 80k high-quality (if I say so myself) articles is probably worth… not very much. LLMs are based on feeding billions of pieces of content—articles, websites, comments, pdfs, videos, books, etc—into a transformer tool to make the LLMs work. Any individual piece of content (or even 80k pieces of content) is actually not worth that much. So, even if Cloudflare’s system got anyone to pay, the net effect for almost everyone online would be… tiny.
TechDirt
Since my last posts about the subject, I have noticed a fascinating phenomenon about the source of traffic on my blogs. The amount of non-bot visitors coming from ChatGPT, Gemini and others has risen sharply, contradicting the pattern I thought I would actually see. In fact, I would almost argue that I’ve seen an increase in traffic with a lot of it coming from AI chats, but I haven’t crunched any numbers, so I can only say how it feels to me.
Most of the traffic increase I’ve seen has been on my blog The Beskirted Man. That is not only by far my most popular blog with 400-500 visitors a day, but I think it’s because of the unusual subject matter I discuss there that I’ve seen an increase in citations in AI chats and thus an increase in traffic. There is a huge number of tech blogs that discuss similar topics to what I post here, so I’m not surprised that I’m not seeing as many citations from this blog.
In any case, I’m cautiously optimistic now that the impact on bloggers and other creators won’t be as bad as I thought it would be. Of course, I’m not earning any money from my blogs, so that is one concern I don’t have, but maybe those who do won’t feel the pinch as much as I thought.
