If you don't care about isolation... to do so requires a VM to securely isolate and that is a significant startup time and resource usage over a wasm module.
VMs are also full of side channels. Depending on how much isolation is a concern, you need to own the host.
I don't trust VMs particularly more than containers in this respect: Containers have a lot of attack surface, but VMs also have a lot of complicated in the code in the kernel, in addition to having complicated emulated device drivers and a large silicon-based attack surface.
