I totally feel the same way too (edit: but almost certainly not worse than how it feels for women). Having worked in both mixed and segregated teams, I much prefer having a good mix. It feels much more natural and normal.