Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Recently I needed to block some scrapers to execessive load on a server, and here are some that I identified:

BOTS=( "semrushbot" "petalbot" "aliyunsecbot" "amazonbot" "claudebot" "thinkbot" "perplexitybot" "openai.com/bot" )

This was really just emergency blocking and it included more than 1500 IP addresses.

Here's Amazon's page about their bot with more information including IP addresses

https://developer.amazon.com/amazonbot



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: