Any ID system that isn't just totally run be each individual themselves is ceding power to someone.
Whether you need an ID to do certain things or are tracked doing certain things is also a very separate issue.
What is stopping "them" is that "them" in liberal democracies (as a technical term) isn't free to do whatever they please nor beyond control/recall/etc. If you want to live in a society, there will be rules, implicit or explicit, on how people interact, delegate, etc.
Do they also have a right to build a database of every time I utilize my ID? What's stopping them?
I think there is already a large group of people who would prefer to live in a society without ceding that much power to a centralized authority.