You are pointing at "the complexity of wishes": if you have to specify what you want with computer-like precision, then it is easy to make a mistake.
In contrast, the big problem in the field of AI alignment is figuring out how to aim an AI at anything at all. Researchers certainly know how to train AIs and tune them in various ways, but no one knows how to get one reliably to carry out a wish. If miraculously we figure out a way to do that, then we can start worrying about the complexity of wishes.
Some researchers, like Eliezer and his coworkers, have been trying to figure out how to get an AI to carry out a wish for 20 years and although some progress has been made, it is clear to me, and Eliezer believes this, too, that unless AI research is stopped, it is probably not humanly possible to figure it out before AI kills everyone.
Eliezer likes to give the example of a strawberry: no one knows how to aim an AI at the goal of duplicating a strawberry down to the cellular level (but not the atomic level) without killing everyone. The requirement of fidelity down to the cellular level requires the AI to create powerful technology (because humans currently do not know how to achieve the task, so the required knowledge is not readily available, e.g., on the internet). The notkilleveryone requirement requires the AI to care what happens to the people.
Plenty of researcher think they can create an AI that succeeds at the notkilleveryone requirement on the first try (and of course if they were to fail on the first try, they wouldn't get a second try because everyone would be dead) but Eliezer and his coworkers (and lots of other people like me) believe that they're not engaging with the full difficulty of the problem, and we desperately wish we could split the universe in two such that we go into one branch (one future) whereas the people who are rushing to make AI more powerful go into the other.
In contrast, the big problem in the field of AI alignment is figuring out how to aim an AI at anything at all. Researchers certainly know how to train AIs and tune them in various ways, but no one knows how to get one reliably to carry out a wish. If miraculously we figure out a way to do that, then we can start worrying about the complexity of wishes.
Some researchers, like Eliezer and his coworkers, have been trying to figure out how to get an AI to carry out a wish for 20 years and although some progress has been made, it is clear to me, and Eliezer believes this, too, that unless AI research is stopped, it is probably not humanly possible to figure it out before AI kills everyone.
Eliezer likes to give the example of a strawberry: no one knows how to aim an AI at the goal of duplicating a strawberry down to the cellular level (but not the atomic level) without killing everyone. The requirement of fidelity down to the cellular level requires the AI to create powerful technology (because humans currently do not know how to achieve the task, so the required knowledge is not readily available, e.g., on the internet). The notkilleveryone requirement requires the AI to care what happens to the people.
Plenty of researcher think they can create an AI that succeeds at the notkilleveryone requirement on the first try (and of course if they were to fail on the first try, they wouldn't get a second try because everyone would be dead) but Eliezer and his coworkers (and lots of other people like me) believe that they're not engaging with the full difficulty of the problem, and we desperately wish we could split the universe in two such that we go into one branch (one future) whereas the people who are rushing to make AI more powerful go into the other.