I was working at a recruiting startup years ago and we got a cease & desist letter from LI asking us to stop crawling their public data.
Initially crawling them was very easy (we had ~100m profiles). With time crawling became harder and harder on the technical front. They obviously invested millions of $ in technology to stop being crawled.
Initially crawling them was very easy (we had ~100m profiles). With time crawling became harder and harder on the technical front. They obviously invested millions of $ in technology to stop being crawled.