To safely deploy generative AI in health care, models must be open source (opens in new tab)

Recently there has been a trend in calling models with weights and code available "open source" even if the training data is not available. For safe deployment in health care and other safety critical fields, transparency on the training data and process are vital too, which means developing clear terminology for models full transparency! Even this article title suffers from this ambiguity.

How does open source improve safety if we simply don't have the analytical tools to intuitively reason about LLMs?

You can't use this to prove that the model will always behave correctly (or desirably). At best, you can build test-suites to empirically check that it kinda-sorta appears to be doing the right thing most of the time. Which you can just as easily do with a black-box model.

It's not that I'm against openness. I just don't see how you can posit that it gets us close enough to safety.

It seems like all of their criticisms can be easily applied to essentially any technology used or medical company relied on.

For example: >In the rush to deploy off-the-shelf proprietary LLMs, however, health-care institutions and other organizations risk ceding the control of medicine to opaque corporate interests. Medical care could rapidly become dependent on LLMs that are difficult to evaluate, and that can be modified or even taken offline without notice should the service be deemed no longer profitable

Even:

>LLMs often generate ... convincing outputs that are false

is already a problem the medical community has to address with existing tests.

Or:

>Another problem specific to proprietary LLMs is that companies’ dependency on profits creates an inherent conflict of interest that could inject instability into the provision of medical care.

Seemingly applies to essentially the entirety of medical supplies and medications.

I mean opensource is nice, but that's not actually going to make healthcare safer.

Whatever flavour of AI needs to be deterministic, which llama, et al are not. even if you turn the temperature right down.

As others have pointed out, its the training set that actually makes a model behave, hence why models are freely given away by large companies.

As mentioned in another comment, the problem is that Open Source does not necessarily apply to all aspects of models. Open code allows everyone access to the "source" of an application. It does not mean the information that the code stores, when used, is also open to viewing.

In models, the training data (dataset) is frequently "closed", where it is not open to viewing. That's just the default behavior of publishing models. You don't need the dataset to use the model. The weights or tensors may be "open" in that we can see them, but they are fairly "not worth viewing" if we don't know the nature of the relationships between the tensors.

If we were able to figure out relationships between the tensors, and the dataset was not made open, then there might be a debate on whether or not certain use of that extracted or "transfer" knowledge is allowed.

For a "model" to be fully "open", it must publish the data it was trained on, the code it used to train itself, and its tensors or weights must not be encrypted or disallow establishing relationships in the weights.

It would seem to me, that the data is the more important part of the equation, and the health care providers, being the holders of this data, and also needing to find new revenue streams, want to profit from this.

With federated learning and homomorphic encryption, can we satisfy both parties?

I agree that open source (or source available) is better, in particular the weights and code, the training data is immaterial. But I think a lot of this is pretty naive. The "best" model is the best and it's unlikely to come from some idealistic consortium. And the data they have is virtually irrelevant, as every company that thinks they have a great trove of data finds out. My recommendation would be to use whatever the leading source available models is (one of the big llamas?) and focus on the guardrails needed to make it a helper for medicine. Reinventing the wheel is a bad idea.

To deploy generative AI in healthcare someone has to pay for the salaries of a lot of people to do the work. That means there needs to be a business model.

I am not sure who will take an AI through regulatory procedures if it is open source and there is no way to make money from it.

Open source is a useful tool for research yes. More of it would be nice.

But I don’t understand how or why anyone is going to go through all the hurdles of deploying technology if all of it is open source.

Maybe an open source enthusiast can explain to me how that is supposed to work?

Just like an MRI machine is open source? I'm not sure if the authors have thought any of this through.