Bob and Juliet? Rag vs. Finetuning an LLM (opens in new tab)

I believe that retrieval-augmented generation is the right path to generative AI within organizations, at least for the next few years. Trying to directly fine-tune an LLM on your data also runs into issues with enforcing access permissions.

However, instead of simply being a post-processing step at the end of an IR pipeline, LLMs will eventually sandwhich the IR system, along the lines of the [Demonstrate, Search, Predict framework](https://arxiv.org/abs/2212.14024) by Khattab et al.

I've seen a lot of this, and other problems with fine tuning, that result in real business outcomes. For example:

  - I've seen research orgs have a *lot* of difficult to fine tuning out something that might be taken as "fact" based on the original training data.  As science advances, but the training may be based on (and reinforced through) several years of suddenly outdated info due to new research, it's really difficult to "fine tune that old knowledge out".  Meanwhile, RAG from what I've seen seems to use the language model for *language/structure* but the retrieval for the facts/details when done correctly.
  - I've seen a fair number of copyright issues with fine tuning, because it's really difficult to get a LLM to not return something that looks like the original source except to change its form.  Meanwhile, RAG focuses the output on whatever data you put in

It's of course possible to trick a RAG system into producing either of these issues -- especially if the prompt doesn't attempt to account for them -- but it seems to me to be much harder