undefined | Better HN

0 pointsCaptainOfCoit7mo ago0 comments

> A lot of times when I ask for a source,

They'll do pretty much everything you ask of them, so unless the text actually come from some source (via tool calls, injecting content into the context or other way), they'll make up a source rather than doing nothing, unless prompted otherwise.

0 comments

instagib7mo ago

On my llm, I have a prompt that condenses down to:

For every line of text output, give me a full MLA annotated source. If you cannot then say your source does not exist or you are generating information based on multiple sources then give me those sources. If you cannot do that, print that you need more information to respond properly.

Every new model I mess with needs a slightly different prompt due to safeguards or source protections. It is interesting when it lists a source that I physically own and their training data is deteriorated.

vbezhenar7mo ago

They could make up source, but ChatGPT is an actual app with complicated backend, not dumb pipe between textedit and GPU. Surely they could verify on server side every link they output to user before including it in the answer. I'm sure Codex will implement it in no time!

therein7mo ago

They surely can detect it, but what are they going to do after detecting it? Loop the last job with a different seed and hope that the model doesn't lie through its teeth? They won't be doing it because the model will gladly generate you a fake source on the next retry too.

doikor7mo ago

This is actually harder then most think. The chances of your app doing this check being bot detected/blocked is very high.

(unless you are Google etc which are specifically let in to get the article indexed into search)

jacobgkau7mo ago

Maybe they should be trained on the understanding that making up a source is not "doing what you ask of them" when you ask for a source. It's actually the exact opposite of the "doing what you asked, not what you wanted" trope-- it's providing something it thinks you want instead of providing what you asked for (or being honest/erroring out that it can't).

ssivark7mo ago

Think for a second about what that means... this is a very easy thing to do IFF we already had a general purpose intelligence.

How do you make an LLM understand that it must only give factual sources? Just some naive RL with positive reward on the correct sources and negative reward on incorrect sources is not enough -- there are obscenely many more hallucinated sources possible, and the set of correct sources is a set of insanely tiny measure.

fluoridation7mo ago

"Easy". You make the model distinguish between information and references to information. Information may be fabricated (for example, a fictional book is mostly composed of lies) but references are assumed to be factual (a link does point to something and is related to something). Factual information is true only to the degree that it is conveyed exactly, so the model needs to be able to store and reproduce references verbatim.

Of course, "easy" is in quotes because none of this is easy. It's just easier than AGI.

Paradigma117mo ago

Wrong, just ask it about some non existent famous historical person and it will most likely tell you it didnt exist.

j / k navigate · click thread line to collapse