I know this might sound shallow but the first time I meet someone, I actually judge that person based on the way he or she dresses (if I suspect that he could afford dressing properly). It tells me whether they care about themselves, whether they have a good taste. Ultimately yes, in my experience, there's a correlation between "spiffy clothing" and "trustworthiness".
And I'm convinced that most people do the same, whether they admit it or not.
Fashion is clearly very subjective but it is also a visual language. If the way you dress within a context is communicating that you don't care about yourself, you aren't dressed properly.
I wear what is comfortable precisely because I care about myself. By your logic, dressing in any manner other than shorts and t-shirts would either be communicating that I don't care about myself, or would be a lie.
People have to trust what you say and what you do. How you present yourself partly determines how people perceive and trust you.