> Everyone who has used vi(m) has copied and pasted text in while in command mode and done who knows what.

try to set mousemode (vim) if vim is compiled with x11-support.