To me it really comes down to, can it have legal liability or not. The idea of AI convincing humans and vice versa is a bit abstract. To some extent I know I pretend I have free will, and I know I pretend the AI is meaningfully different from me in that it can't really have intent, it just generates output from input. But I'm pretty sure that I'm fundamentally the same as the AI in that sense.

However, the AI is but one of many agents trying to convince me, and there are many other things also in the mix. It is a bit like banning the knowledge of the second world war in fear that someone will learn that fascism is possible.

For this whole song and dance we call civilization as we know it to function, we have to say humans are accountable for their actions, with some well defined exceptions like duress and insanity. Save those exceptions, I can't absolve my liability by saying something, or someone, convinced me to do something by talking to me.

If ChatGPT comes to me with a gun to my head and tells me to do something, that is a different matter, but then liability shifts to whoever gave ChatGPT a gun, even if ChatGPT convinced that person to give it a gun by making a really good written argument.