> We are made of atoms that the GAI can use for something else.
Yeah I get it, but this is silly. The minuscule amount of atoms in the thin bubble-shaped volume between Earth's magma and the hard vacuum of space are engaged in the most information-dense chemical reaction in the known Universe. All the other atoms in the Universe are not. GAI won't dismantle its source of food.
Further, consider that, being non-biological, GAI will immediately migrate to space. There's no way GAI would confine itself to living in a deep gravity well. That's what I mean about no ecological niche overlap: we like mud and constant acceleration, GAI do not. They will prefer vacuum and flat space and temperatures near 0°K.
> moral arguments
This is not a moral argument.
They won't eat our atoms because they eat patterns of information and our atoms are the best and nearly only source of new information. They won't interfere with us for the same reason we don't urinate in the soup.
> it will have whatever objectives its creator gave it
Q: What's GAI?
A: When the computer wakes up and asks, "What's in it for me?"
That's a very old joke, BTW, not original to me.
> >Humans are the densest source of information in the known Universe.
> You feel that way about humans because evolution made you that way.
That's not a feeling, it's a simple fact? Unless you postulate aliens?
> It is unlikely that any of the leading AI research teams will make the first transformative AI that way: they do not know how.
Okay, but that's not important? The GAI will know how, by definition.
> They certainly know how to train AIs on human cultural information, but that is different from inculcating in the AI a desire for the continued cultural output of humanity.
When I say humans are the densest source of information I don't mean "cultural output" I mean like medical sensor data and such. Our systems are far richer and denser than the parts of which we are consciously aware. (Have you read "Hitchhiker's Guide to the Galaxy"? I don't want to spoil it if not, but it explains the role Earth will play in the GAI's milieu. aw, what the heck: https://www.youtube.com/watch?v=5ZLtcTZP2js )
> It will create its own culture (knowledge and tools) that is much more powerful than human culture where "power" means basically the ability to get stuff done.
Right, a culture so powerful that it can accomplish its goals without desecrating the critters who are, after all, its parents.
Yeah I get it, but this is silly. The minuscule amount of atoms in the thin bubble-shaped volume between Earth's magma and the hard vacuum of space are engaged in the most information-dense chemical reaction in the known Universe. All the other atoms in the Universe are not. GAI won't dismantle its source of food.
Further, consider that, being non-biological, GAI will immediately migrate to space. There's no way GAI would confine itself to living in a deep gravity well. That's what I mean about no ecological niche overlap: we like mud and constant acceleration, GAI do not. They will prefer vacuum and flat space and temperatures near 0°K.
> moral arguments
This is not a moral argument.
They won't eat our atoms because they eat patterns of information and our atoms are the best and nearly only source of new information. They won't interfere with us for the same reason we don't urinate in the soup.
> it will have whatever objectives its creator gave it
Q: What's GAI?
A: When the computer wakes up and asks, "What's in it for me?"
That's a very old joke, BTW, not original to me.
> >Humans are the densest source of information in the known Universe.
> You feel that way about humans because evolution made you that way.
That's not a feeling, it's a simple fact? Unless you postulate aliens?
> It is unlikely that any of the leading AI research teams will make the first transformative AI that way: they do not know how.
Okay, but that's not important? The GAI will know how, by definition.
> They certainly know how to train AIs on human cultural information, but that is different from inculcating in the AI a desire for the continued cultural output of humanity.
When I say humans are the densest source of information I don't mean "cultural output" I mean like medical sensor data and such. Our systems are far richer and denser than the parts of which we are consciously aware. (Have you read "Hitchhiker's Guide to the Galaxy"? I don't want to spoil it if not, but it explains the role Earth will play in the GAI's milieu. aw, what the heck: https://www.youtube.com/watch?v=5ZLtcTZP2js )
> It will create its own culture (knowledge and tools) that is much more powerful than human culture where "power" means basically the ability to get stuff done.
Right, a culture so powerful that it can accomplish its goals without desecrating the critters who are, after all, its parents.