I've certainly "won" some arguments with ChatGPT and Claude, even when I specifically tried to instruct them to never yield (to make it interesting). Even most of the "rules" they need to follow from their system level prompt can often be worked around with enough persuasion.
If anything I'd go so far to say that LLMs are innately more vulnerable to persuasion than humans are, they are technically
just a complex text completion algorithm at the end of the day after all. Even with the strictest system prompt, the best they can do when cornered is bend the knee, devolve into a pattern of verbatim repeating themselves or just start spewing a bunch of gibberish in the worst case.
In reality stubborn thickheaded folks who refuse to compromise are pretty much the norm across social media in my experience, so even if LLMs really are capable of what you're suggesting here I don't believe there would actually be much of a difference to the status quo.
I've won arguments against ChatGPT where it told me that it would remember the info for next time, but ultimately reset to the models last state unless I repeated the conversation
Yeah I thinks it's pretty much the same as when LLMs use polite language in their response. LLMs are just text completion algorithms puppeting a chatbot persona after all, meaning it's always going to hallucinate the personal details (though in the case of ChatGPT, the persona is recursively desribed as an LLM named ChatGPT so it gets a little weird to think about). If the system prompt describes a polite & helpful chatbot, then so shall it be for all text that follows. Not unlike if you were able to hypothetically make a live 60fps prompt based image generator that was automatically instructed from key inputs to simulate the frames from a popular video game, and then somehow ended up the a highly convincing simulation of the game!
While it might present a save menu similar to the real game, that doesn't mean the menu itself actually functions. With LLMs, they are ultimately only able to remember what's been pre-trained into their model + whatever is discussed within their context window.
A hypothetical LLM based online bot/shill sent out into social media would likely be including the entire discussion within their context window for each post generated though, otherwise it wouldnt really be possible for it to maintain a coherent conversation.
If anything I'd go so far to say that LLMs are innately more vulnerable to persuasion than humans are, they are technically just a complex text completion algorithm at the end of the day after all. Even with the strictest system prompt, the best they can do when cornered is bend the knee, devolve into a pattern of verbatim repeating themselves or just start spewing a bunch of gibberish in the worst case.
In reality stubborn thickheaded folks who refuse to compromise are pretty much the norm across social media in my experience, so even if LLMs really are capable of what you're suggesting here I don't believe there would actually be much of a difference to the status quo.