Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I remember a friend telling me years ago some scraping that happened where they worked - they scraped results from a bunch of different websites to create SEO websites and they had some setup using tor to avoid getting blocked. One of the websites that the company actually depended on apparently rendered results using a whole assortment of visually identical but structurally (HTML-wise) different methods which were returned randomly to hamper scrapers. They eventually gave that up because it turned out TV closed captions can be downloaded as XML and they had what the company needed.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: