The Chinese AI scrapers/bots are killing quite a bit of the regular web now. Yis... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		JohnTHaller 79 days ago \| parent \| context \| favorite \| on: End of an era for me: no more self-hosted git The Chinese AI scrapers/bots are killing quite a bit of the regular web now. YisouSpider absolutely pummeled my open source project's hosting for weeks. Like all Chinese AI scrapers, it ignores robots.txt. So forget about it respecting a Crawl-delay. If you block the user agent, it would calm down for a bit, then it would just come back again using a generic browser user agent from the same IP addresses. It does this across 10s of thousands of IPs.

mono442 79 days ago | [–]

Just block the whole China, India and similar countries.

AlexeyBelov 77 days ago | | [–]

You'd need to block the US as well, as most such traffic comes from there. Which is not really reasonable.

kevin_thibedeau 79 days ago | | [–]

Start blocking /16s.

olivia-banks 76 days ago | [–]

I had no idea Yisou was still around, or did somebody buy them? I'm not as up to date on Chinese tech as I should be.

Consider applying for YC's Summer 2026 batch! Applications are open till May 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact