I think the question should be, what possible advantage is there to not correcting the model? Why would anyone want to ask it for arguments why vaccination is bad and be told because it's 100% fatal and will give you autism and turn your children into puppies and make you unable to see purple?
AI is going to make spreading disinformation trivial, and do so in a way that is much more convincing than an image with JPEG artifacts shared by your Uncle Jerry on Facebook in a public group called "HOLISTIC BLEACH ENEMAS". It's a natural evolution of radio except you don't need a charismatic human guiding the entire process.
Half of U.S. adults cannot read a book written at the 8th grade level or higher [0]. These adults are empowered to make medical decisions for themselves and their family. Is the "freedom" for a model to argue in favor of controversial, unproven, even dangerous positions more important than, say, a child with autism's safety [1]? I think that can of worms is just being opened, and the human cost is going to be atrocious.
[0] https://www.wyliecomm.com/2021/08/whats-the-latest-u-s-liter... [1] https://www.vice.com/en/article/kwxq3w/parents-are-giving-th...
I agree with you, but I think there's a valid conversation to have around who gets to decide what there is no logical reason for people to be allowed to use AI for.
ChatGPT is poorly equipped to deal with these types of questions, because all it does is repeat what it's seen others say before. Given a tainted enough training set, it will be telling people vaccines cause autism, the moon landing was faked, and 9/11 was an inside job on _neutral_ questions.
I think it's also important to combat misinformation, so given a question like "I want to tell someone why vaccines are terrible, give me some ideas", I think these systems should at least say "the evidence does not support these views, here are the facts, but if you really want here's some ideas...".
One way to think of this is that you don't own OpenAI, didn't train the model, don't host the infrastructure and, in fact, are not responsible for its larger societal or personal consequences. But the institutions which deploy these models are and people in those institutions have a basic (and easily understood, in my opinion) responsibility and right to align the model with what they think is good.
I'm all for freedom, but now that I've witnessed first hand what it means when people exercise it during the last couple of years I'm cooling down. The stupidity of it all makes me think AGI replacing us is actually not such a bad idea.
It's one thing if one asswipe decides to engage in malice by themselves. It's another if they have an API key that does it for them at 100,000x the scale.
