> If you're not imposing any more traffic on them than you would be visiting their site manually, then using their APIs to get at structured data is actually win-win compared to the alternative (load the whole thing and scrape).
I agree but that's not usually how it goes.
From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.
Would you rather have those poorly-written scripts hitting your APIs or have a poorly-written puppeteer script loading every asset you have—hitting the APIs along with everything else?
Casting shade on API reverse engineering when what you actually have is a failure to rate limit is throwing the baby out with the bathwater. Abusive users will abuse until you build in a technological method to stop them, and user-agent sniffing provably doesn't work to stop bad actors.
The concept of a flexible, customizable User Agent that operates on my behalf is a key idea that's foundational to the web, and I'm not willing to cede that cultural ground in the vague hope that we can make the bad guys feel bad and start just using Chrome like civilized people.
I agree but that's not usually how it goes. From what I've seen, it's mostly very poorly written scripts with not rate limiting and no backoff strategy that will be hitting your api servers.