While it has nothing to do with how responsive your mouse feels, as that is measured in milliseconds while CAS latency is measured in nanoseconds, there has indeed been a small regression with DDR5 memory compared to the 3 previous generations. The best DDR2-4 configurations could fetch 1 word in about 6-7 ns while the best DDR5 configurations take about 9-10 ns.
RAM latency doesn't affect mouse response in any perceptible way. The fastest gaming mice I know of run at 8000Hz, so that's 125000ns between samples, much bigger than any CAS latency. And most mice run substantially slower.
Maybe your old PC used lower-latency GUI software, e.g. uncomposited Xorg instead of Wayland.
I only felt it on Windows, maybe tht is due to the special USB mouse drivers Microsoft made? Still motion-to-photon latency is really lower on my DDR3 PCs, would be cool to know why.
Dan Luu actually measured latency of older computers (terminal, input latency), and compared it to modern computers. It shows older computers (and I mean previous century-wise old) have lower input latency. This is much more interesting than 'feelings', especially when discussing with other people.
Bandwith increases, but if you only need a few bytes DDR3 is faster.
Also slower speed means less heat and longer life.
You can feel the speed advantage by just moving the mouse on a DDR3 PC...