Why does it matter if LLMs are not deterministic? Who cares?
There should be tests covering meaningful functionality, as long as the code passes the tests, ie. the externally observable behaviour is the same, I don't care. (Especially, if many tests can also be autogenerated with the LLM.)
>>> Very few people have the expertise to write efficient assembly code, yet everyone relies on compilers and assemblers to translate high-level code to byte-level machine code. I think same concept is true her
>> That's a poor analogy which gets repeated in every discussion: compilers are deterministic, LLMs are not.
> Why does it matter if LLMs are not deterministic? Who cares?
In the context of this analogy, it matters. If you're not using this analogy, then sure, only the result matters. But when the analogy being used is deterministic, then, yes, it matters.
You can't very well claim "We'll compare this non-deterministic process to this other deterministic process that we know works."
There should be tests covering meaningful functionality, as long as the code passes the tests, ie. the externally observable behaviour is the same, I don't care. (Especially, if many tests can also be autogenerated with the LLM.)