Yes, it's studied. Rust and metal have different frictional properties, and the rotors produce different torques when the brakes are applied. This results in a sort of uneven shuddering when you use the brakes if they're heavily corroded.
It's not immediately deadly, but your brakes are one of the most important safety systems on the vehicle. They should be in good working order.
Totally, I'm not thinking for a second that you are wrong there. I'm not saying rust on brakes isn't bad. What I'm questioning is whether rust buildup is an actual issue happening for most EV drivers right now.
I'm genuinely curious because, for no good reason, I take pride in the fact that most trips I don't even touch the brake pedal other than to come to a complete stop and only when parking, etc where you can't regen all the way to a stop.
It's not immediately deadly, but your brakes are one of the most important safety systems on the vehicle. They should be in good working order.