I don't think that's a bad idea. However, anyone who has been around here long enough to accumulate 1000 karma (or whatever it is now) should know better than to downvote anodyne comments with which one mildly disagrees.
I think people are getting emotional about certain subjects and lashing out by downvoting any comments that provoke rational discussion on those subjects. In other words, downvoting exactly the kind of comments pg is trying to foster in HN.
This would make a lot of sense. Flagging clearly says "this comment is not appropriate on this site", while downvoting usually means simply "I don't agree with this comment".
Downvoting also means "this comment should be this high", however the problem is that there's o indication when a comment is at score of 1, so people would inadvertently push it into gray area.
