I think you're being a bit alarmist, the most likely attack is that someone compromises one password and then logs into a higher value site with it. They can't do it in this case.

That's not going to happen with that scheme.

As long as you're not also using this for email/banks, it's not that silly, I use a similar scheme myself, it means you can log in from computers that aren't your own to certain services without having to install anything or carry round a bit of paper.