Well, by "better" I mean that what I wear is usually intentional and has some thought put into it; whereas a lot of hackers will actually make a point of declaring that they put no effort at all into what they chose to wear.
And that's fine! If you don't care then you don't care. There's nothing wrong with that.
I just see it as weird that people use that notion of "not caring" as a badge of honor. My question to those people would be, why do you think it's weird that I do actually care?