I also find moving hand to mouse is a really frustrating context switch, so I ended up getting a roller mouse (https://www.contourdesign.com/product/rollermouse-red/) to use along with my main mouse. The idea is for lots of small things my hands stay at the keyboard but my thumbs can move the mouse so I don't incur a context switch.