Coming from nixos, I didn't want to use brew but I found nix-darwin to be poorly documented and supported.
Several upgrades wrecked havoc on my system, a lot of things didn't work as expected, I had to package a lot of stuff; eventually I just bit the bullet and started using homebrew
I sort of get what you're saying and I've had a few sporadic issues myself, once or twice I had to reinstall (although might have been my mishandling), but overall I can't see a better solution than this. Plus I can share the configs with Linux.
Several upgrades wrecked havoc on my system, a lot of things didn't work as expected, I had to package a lot of stuff; eventually I just bit the bullet and started using homebrew