Long story short, my VPS, which I’m forwarding my servers through Tailscale to, got hammered by thousands of requests per minute from Anthropic’s Claude AI. All of which being from different AWS IPs.
The VPS has a 1TB monthly cap, but it’s still kinda shitty to have huge spikes like the 13GB in just a couple of minutes today.
How do you deal with something like this?
I’m only really running a caddy reverse proxy on the VPS which forwards my home server’s services through Tailscale. "
I’d really like to avoid solutions like Cloudflare, since they f over CGNAT users very frequently and all that. Don’t think a WAF would help with this at all(?), but rate limiting on the reverse proxy might work.
(VPS has fail2ban and I’m using /etc/hosts.deny for manual blocking. There’s a WIP website on my root domain with robots.txt that should be denying AWS bots as well…)
I’m still learning and would really appreciate any suggestions.
I did find this github link as the first search result, looks interesting, thanks for letting me know the term “tar pit”.
If you’re looking to stop them from wasting your traffic, do not use a tarpit. The whole point of it is that it makes the scraper get stuck on your server forever. That means you pay for the traffic the scraper uses, and it will continually rack up those charges until the people running it wise up and ban your server. The question you gotta ask yourself is, who has more money, you or the massive AI corp?
Tarpits are the dumbest bit of anti-AI tech to come out yet.
There’s more than one style of tar pit. In this case you obviously wouldn’t want to use an endless maze style.
What you want to do in this case is send them through an HA proxy that would redirect them on user agent, whenever they come in as Claude you send them over to a box running on a Wanem process at modem speeds.
They’ll immediately realize they’ve got a hug of death going on and give up.
I don’t quiet understand how this is deployed. Hosting this behind a dedicated subdomain or path kind of defeats the purpose as the bots are still able to access the actual website no problem.
The trick is distinguishing them by behavior and switching what you serve them
How would I go about doing that? This seems to be the challenging part. You don’t want false positives and you also want replayability.
If you’ve already noticed incoming traffic is weird, you try to look for what distinguishes the sources you don’t want. You write rules looking at the behaviors like user agent, order of requests, IP ranges, etc, and put it in your web server and tells it to check if the incoming request matches the rules as a session starts.
Unless you’re a high value target for them, they won’t put endless resources into making their systems mimic regular clients. They might keep changing IP ranges, but that usually happens ~weekly and you can just check the logs and ban new ranges within minutes. Changing client behavior to blend in is harder at scale - bots simply won’t look for the same things as humans in the same ways, they’re too consistent, even when they try to be random they’re too consistently random.
When enough rules match, you throw in either a redirect or an internal URL rewrite rule for that session to point them to something different.
there is also https://forge.hackers.town/hackers.town/nepenthes
Now I just want to host a web page and expose it with nepenthes…
First, because I’m a big fan of carnivorous plants.
Second, because it let’s you poison LLMs, AI and fuck with their data.
Lastly, because I can do my part and say F#CK Y0U to those privacy data hungry a$$holes !
I don’t even expose anything directly to the web (always accessible through a tunnel like wireguard) or have any important data to protect from AI or LLMs. But just giving the opportunity to fuck with them while they continuously harvest data from everyone is something I was already thinking off but didn’t knew how.
Thanks for the link !