They use wayland compositor (gamescope) under linux (even under X11), in their new SteamOS UI and it's already working a lot better. Plus they can do other awesome stuff with it like completely having control of where and how the game outputs its buffer. So it can fix games with bad multi-monitor behaviour, force game to render in their native aspect ratio / resolution etc... it's just great and makes gaming experience feel more like a console where things (usually) just work.
Gamescope + mangohud + steam overlay working together is very big part of SteamOS and Steam Deck's success, and it does zero runtime modification or hooking into the child process. I can imagine all the trouble you run with the hooking eventually and all the hacks it must have piled up.
They use gamescope under SteamOS, not Linux in general where you already have another compositor. Running all games under a nested compositor (gamescope) would have some perf impact (even if a small one) as well as an unexpected cursor acceleration profile.
True. I personally have setup my distro to launch directly into gamescope optionally for SteamOS like experience. On desktop client they still either hook or use vulkan layer (latter often in proton, thanks to dxvk). However I would not be surprised if they started using nested compositor even on desktop.
Gamescope + mangohud + steam overlay working together is very big part of SteamOS and Steam Deck's success, and it does zero runtime modification or hooking into the child process. I can imagine all the trouble you run with the hooking eventually and all the hacks it must have piled up.