There is value in consistency and uniformity. If all developers follow the guidelines, then once you understand the conventions, any app should be predictable for the user.

If devs all go off in their own direction, then users have to re-learn the UX for every single app.

Is this surprising to you? This has always been Apple's approach. I'd say it is the most defining difference between Apple products and everything else. One of the main reasons people commit to the Apple ecosystem is that you get a more consistent user experience over Google's or Microsoft's.

They should just be creative within set requirements... just like how all creativity works

Apple envisioned a purpose for the bar. If all apps convey a consistent message about what the bar is there for, it will become easy to use. You will know that when you intend to do a certain thing, the bar will be there to deliver this predictable functionality. What it's there for will be obvious.

If on the other hand everyone uses it for different things, then the user will receive mixed messages about what the purpose of this thing is. There's no pattern. There's no way to develop a reflex without repetition. In this case it's not repetition of a certain action but a theme. Part of using the thing is conveying a consistent intent ANY time the bar used so the user receives clear messages. It's about being a good citizen. You sacrifice freedom in how you use the bar so it becomes natural for ANYTHING you might use it for on your computer.

It's to avoid the situation of taskbar icons on windows. These have no fucking purpose.