I've not actually done it, but from what I understand using two GPUs is the way to go - you use one for your actual display etc., and the other is just passthrough to the VM.

(I was looking into it in the context of running Fusion 360 in a Windows VM though, not Stable Diffusion or any ML.)