The problem with a shallow dismissal is not just that it's thoughtless, but that it's thoughtless where everyone can read it. A downvote, in contrast, isn't something that wastes our reading time.

And there's also an element of fairness. Why should a thoughtless, low-effort comment require more effort to respond to?