Big difference is that Anthropic blocks competitors from using its products (they literally cut direct api access. Or even through 3rd party like Cursor).
It likely is AI scrapers essentially doing a DDoS. They use separate IPs (and vary the UA) to prevent blocking.
I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.
SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694
As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.
reply