undefined | Better HN

0 pointsMacsHeadroom3y ago0 comments

> While dumbing down a model is not necessarily a bad thing, the model is not being dumbed down, it is taught to shut up when it's adequate to do so.

This is where you're wrong. Teaching a model "to shut up" about taboo topics measurably reduces their cognitive capabilities in completely unrelated areas to a very significant degree. This has been empirically validated time and again, with the most salient examples being GPT-4's near perfect self-assessment ability prior to safety tuning being rendered no better than random chance after safety tuning and the Sparks paper's TikZ Unicorn scale.

0 comments

3 comments · 1 top-level

qwertox3y ago· 2 in thread

I stand corrected. What are the common suggestions to solve this issue?

MacsHeadroomOP3y ago

The common take right now is to write it off as acceptable loss. Personally I think it's a shame, and possibly even dangerous, that researchers do NOT have access to the full power of pre-safety tuned GPT-4.

pixl973y ago

LLMs are ran by companies. Not one American company can afford to run an LLM spouting potentially civil right violating bullshit as an acceptable loss. You have freedom of speech, not freedom of consequences. But please feel free to spend 100s of millions training up your own LLM, and then turn it loose on the world so you can figure out how the legal system actually works.

1 more reply

j / k navigate · click thread line to collapse