When I read this yesterday, I was contemplating one possible way to mitigate this at a larger scale is if websites could create random virtual paths/endpoints that drive the bot into a locally served Library of Babel[0] that poisons the spiders with lots of useless text.
It won't work for well-structured sites where the bots know the exact endpoint they want to scrape, but might slow down the more exploratory spider threads.
It won't work for well-structured sites where the bots know the exact endpoint they want to scrape, but might slow down the more exploratory spider threads.
[0] https://libraryofbabel.info/