Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

And on the web side, fingerprinting is rampant and there are JS challenges in cloudflare, imperva, etc which make it trickier. Frustrating to run a whole browser with a virtual screen, load the whole page which is ofc like 15mb of JS and other trash, just to do a very simple thing.

Granted, smaller fish like the ones OP is referring to generally don't have aggressive anti automation measures in place, so it can be easy...but generally these techniques don't work if the operator has put the proper measures in place.



take a look at https://xhr.dev/, a product I built to avoid bot detection from things like cloudflare, imperva, aws waf, and others


What does the $500 a month get me? Infinite resources to scrape all of LinkedIn?


>self host (Docker): $60k/yr

lmao ok


Frustrating? Yeah! but it works SO great! I especially like Playwright in this context, it can do pretty much anything and is a joy to use.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: