> The fact that there's no foolproof way to distinguish instruction tokens from data tokens is not careless
Repeat that over to yourself again, slowly.
> it's a fundamental epistemological constraint that human communication suffers from as well
Which is why reliability and security in many areas increased when those areas used computers to automate previously-human processes. The benefit of computer automation isn’t just in speed: the fact that computer behavior can easily be made deterministically repeatable and predictable is huge as well. AI fundamentally does not have that property.
Sure, cosmic rays and network errors can compromise non-AI computer determinism. But if you think that means AI and non-AI systems are qualitatively the same, I have a bridge to sell you.
> Saying that "software engineers figured out these things decades ago" is deep hubris
They did, though. We know how to both increase the likelihood of secure outcomes (best practices and such), and also how to guarantee a secure behavior. For example: using a SQL driver to distinguish between instruction and data tokens is, indeed, a foolproof process (not talking about injection in query creation here, but how queries are sent with data/binds).
People don’t always do security well, yes, but they don’t always put out their campfires either. That doesn’t mean that we are not very sure that putting out a campfire is guaranteed to prevent that fire burning the forest down. We know how to prevent this stuff, fully, in most non-AI computation.
>> The fact that there's no foolproof way to distinguish instruction tokens from data tokens is not careless
> Repeat that over to yourself again, slowly.
Try using less snark.
And if you have a fundamental breakthrough in AI that gets around this, and demonstrates how "careless" AI researchers have been in overlooking it, then please share.
My point is that the fact that it is not solved makes the use of AI tools a careless choice in situations which benefit from non-AI systems which can distinguish instructions from data, behave deterministically, and so on.
Repeat that over to yourself again, slowly.
> it's a fundamental epistemological constraint that human communication suffers from as well
Which is why reliability and security in many areas increased when those areas used computers to automate previously-human processes. The benefit of computer automation isn’t just in speed: the fact that computer behavior can easily be made deterministically repeatable and predictable is huge as well. AI fundamentally does not have that property.
Sure, cosmic rays and network errors can compromise non-AI computer determinism. But if you think that means AI and non-AI systems are qualitatively the same, I have a bridge to sell you.
> Saying that "software engineers figured out these things decades ago" is deep hubris
They did, though. We know how to both increase the likelihood of secure outcomes (best practices and such), and also how to guarantee a secure behavior. For example: using a SQL driver to distinguish between instruction and data tokens is, indeed, a foolproof process (not talking about injection in query creation here, but how queries are sent with data/binds).
People don’t always do security well, yes, but they don’t always put out their campfires either. That doesn’t mean that we are not very sure that putting out a campfire is guaranteed to prevent that fire burning the forest down. We know how to prevent this stuff, fully, in most non-AI computation.