UTF-8 adds a fair amount of work to every operation; UTF-16 and -32 multiply the amount of data transferred.

But in practice, I'd expect non-ASCII to be associated with more expensive features, which has more of an effect.