Fashion is clearly very subjective but it is also a visual language. If the way you dress within a context is communicating that you don't care about yourself, you aren't dressed properly.
I wear what is comfortable precisely because I care about myself. By your logic, dressing in any manner other than shorts and t-shirts would either be communicating that I don't care about myself, or would be a lie.
