One thing I'd like to add though is that I do think there is an additional piece being discarded irrationally. They tend to highly undervalue everything you're describing. Humans aren't Vulcans. By being so obsessed with the risks of paperclip-maximizing-robots they devalue the risks of humans being the irrational animals they are.
This is why many on the left criticize them for being right wing. Not because they are, well some might be, but because they are incredibly easy to distract from what is being communicated by focusing too much on what is being said. That might be a bad phrasing but what I mean is that when you look at this piece from last year about prison sentence length and crime rates by Scott Alexander[0] nothing he says is genuinely unreasonable. He's generally evaluating the data fairly and rationally. Some might disagree there but that's not my point. My point is that he's talking to a nonexistant group. The right largely believes that punishment is the point of prison. They might _say_ the goal is to reduce crime, but they are communicating based on a set of beliefs that strongly favors punitive measures for their own sake. This causes a piece like that to miss the forest through the trees and can be seen by those on the left as functionally right wing propaganda.
Most people are not rational. Maybe some day they will be but until then it is dangerous to assume and act as if they are. This makes me see the rationalists as actually rather irrational.
One thing I'd like to add though is that I do think there is an additional piece being discarded irrationally. They tend to highly undervalue everything you're describing. Humans aren't Vulcans. By being so obsessed with the risks of paperclip-maximizing-robots they devalue the risks of humans being the irrational animals they are.
This is why many on the left criticize them for being right wing. Not because they are, well some might be, but because they are incredibly easy to distract from what is being communicated by focusing too much on what is being said. That might be a bad phrasing but what I mean is that when you look at this piece from last year about prison sentence length and crime rates by Scott Alexander[0] nothing he says is genuinely unreasonable. He's generally evaluating the data fairly and rationally. Some might disagree there but that's not my point. My point is that he's talking to a nonexistant group. The right largely believes that punishment is the point of prison. They might _say_ the goal is to reduce crime, but they are communicating based on a set of beliefs that strongly favors punitive measures for their own sake. This causes a piece like that to miss the forest through the trees and can be seen by those on the left as functionally right wing propaganda.
Most people are not rational. Maybe some day they will be but until then it is dangerous to assume and act as if they are. This makes me see the rationalists as actually rather irrational.
0: https://www.astralcodexten.com/p/prison-and-crime-much-more-...