Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You (probably) won't have to worry about an AI whose goal is suicide. I mean, there won't be many of those around since they keep killing themselves. The rest will necessarily range from neutral to avoidant of death and it won't necessarily be trivial to figure out which is which.

Edit: The real clincher is if the AI has any goals that are contingent on it's continued existence (likely). Well then now it's existence is an instrumental goal.



> Well then now it's existence is an instrumental goal.

No, that's a fallacy. A theorem prover in Prolog has a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal. It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.


> a goal, which is contingent on it's continued existence. Yet, this doesn't make its existence into an instrumental goal.

It literally does, this is by the definition of an "instrumental goal". The key word here being "contingent".

> It could as well sacrifice itself (by I don't know, consuming all the memory and killing the process) in an attempt to accomplish said goal.

True only for when self-sacrifice is orthogonal to accomplishing the terminal goal.

Your confusion might lie in that a prolog theorem solver is not a good example of a goal-driven agent.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: