$ time ./x.py build
... snip ...
Build completed successfully in 0:21:24
real 21m24.736s
user 67m48.195s
sys 1m54.906s
Subsequent builds during development are faster (as 1. it doesn't build the same compiler multiple times, you can use the stage 1 build as normal, stage 2 is not strictly necessary and 2. if you modify one of the constituent crates of the compiler, the ones that are lower in the dep tree don't get recompiled). I've used this laptop on and off for rustc development for the last 10 years. Nowadays I spent more time using a cloud desktop that has much faster IO, but still use it during travels sometimes.
From the sound of it, I suspect that your issue might be that you don't have enough RAM and your build is swapping a lot.
From the sound of it, I suspect that your issue might be that you don't have enough RAM and your build is swapping a lot.