undefined | Better HN

0 pointsdrewm19803y ago0 comments

I instinctively want to "flip the sign" on all of the automated controls they put in, just out of the morbid interest to see what comes out. The moment you have a "avoid_harm_to_humans:bool" training parameter, someone's going to set it to -1.

Their document about all the measures they took to prevent unethical use is also a document about how to use a re-implementation of their system unethically. They literally hired a "red team" of smart people to come up with the most dangerous ideas for misusing their system (or a re-implementation of it), and featured these bad ideas prominently in a very accessibly written document on their website. So many fascinating terrible ideas in there! They make a very compelling case that the technology they are developing has way more potential for societal harm than good. They had me sold at "Prompt: Park bench with happy people. + Context: Sharing as part of a disinformation campaign to contradict reports of a military operation in the park."

0 comments

gwern3y ago

The kind of measures they are taking, like simply deleting wholesale anything problematic, don't really have a '-1'.

But amusingly, exactly that did happen in one of their GPT experiments! https://openai.com/blog/fine-tuning-gpt-2/

drewm1980OP3y ago

LOL great catch! Assume you're referring to this:

"One of our code refactors introduced a bug which flipped the sign of the reward. Flipping the reward would usually produce incoherent text, but the same bug also flipped the sign of the KL penalty. The result was a model which optimized for negative sentiment while preserving natural language. Since our instructions told humans to give very low ratings to continuations with sexually explicit text, the model quickly learned to output only content of this form."

Yeah, for measures that are subsetting out only the nice data, "flipping the sign" would be picking the other subset. So something like "data_to_train_on = (good_data_split, evil_data_split)[accidental_one_based_index_because_humans_still_cant_agree_on_how_to_count]"

gwern3y ago

Or just rerun that preference learning approach, but deliberately, with the filters as the reward function and the reward being for being filtered.

j / k navigate · click thread line to collapse

0 pointsdrewm19803y ago0 comments

0 comments

gwern3y ago

The kind of measures they are taking, like simply deleting wholesale anything problematic, don't really have a '-1'.

But amusingly, exactly that did happen in one of their GPT experiments! https://openai.com/blog/fine-tuning-gpt-2/

drewm1980OP3y ago

LOL great catch! Assume you're referring to this:

gwern3y ago

Or just rerun that preference learning approach, but deliberately, with the filters as the reward function and the reward being for being filtered.

j / k navigate · click thread line to collapse