It is very easy to use it to harm OpenAI, and that is enough for OpenAI to wish to constrain it. You cause it to write something objectionable and then publish it to reap all the attention and ad-revenue which goes with people being outraged. Best if you manage to provoke into outrage inducing output without your prompt itself being outrageous, but not strictly necessary. During this OpenAI’s reputation would be harmed.

This is not even hypothetical. There was that microsoft chatbot which “turned nazi”.[1] They then had to turn it off.

Did it do many other interesting things? Probably. Yet the few problematic outputs is what we remember it by.

1: https://www.theverge.com/2016/3/24/11297050/tay-microsoft-ch...