One possible future for websites is pervasive gatekeepers

It's not news that "AI" crawlers are an ongoing plague and that broadly the plague is getting worse. There seem to be more and more crawlers, and they're getting past more precautions. One possible future for the web in the face of this is websites with less features, at least for the general public, because fewer features means less to crawl and less crawler impact. Another possible web future is one of pervasive gatekeepers that must be satisfied to allow you access to many or most websites.

That future is already partially present today, in the form of Cloudflare and a few other similar services, and another version of this future is appearing as more sites use tools like Anubis. Cloudflare is gatekeeping as a service and Anubis is gatekeeping as software, but both create the same core issue, which is that a lot of websites will be blocked if you can't satisfy them. This gives these gatekeepers a lot of power over how the web behaves, both for browsers (and other web clients) and for web servers.

The effect on web clients is obvious. If you don't run JavaScript acceptably fast or accept HTTP cookies, you're probably not getting past the gatekeeper. Gatekeepers effectively narrow the options for accessing the web as they exclude things that they can't verify or that look too suspicious. The exclusion can be explicit, such as Anubis not working with JavaScript disabled, or it can be implicit, such as how people browsing through Tor apparently very frequently have to pass tedious Cloudflare CAPTCHA checks. And if you're not a browser, the gatekeeper may shut you out entirely, closing the door on various sorts of new web software, new web search engines, or whatever.

The effect on web servers and web services is more subtle. Even though you can work around them, there are certain patterns of how your website or web application works that work well with gatekeepers and others that don't. To put it simply, you need to be able to put something in front of your website and then you need to make your web server and other software 'see through' the forwarding that the gatekeeper is doing to recover the original IP address of the request and so on. Or you can ignore the original IP address and count on the gatekeeper to tell you whatever you later want to know (something I suspect that people are already doing with Cloudflare traffic reports).

I don't like a web future where most everything is shut behind a few gatekeepers, with those gatekeepers holding great practical power to dictate how people and things on the web have to operate. But I think that this is the most likely web future if things stay as they are today, because people are going to do something to deal with their problems and right now, gatekeepers are the easiest solution for people to use, especially for small websites. Small websites aren't in the business of dealing with the current crawler plague; outsourcing it to someone else who is in that business is the obvious and simple approach.

(A corollary is that the gatekeepers don't have to be perfect, they only have to be clearly better than nothing when initially deployed and not cause obvious problems that makes people step away from using them. Of course being effective is better for gatekeepers for various reasons; if the gatekeepers aren't effective enough, people will have to do something else.)