undefined | Better HN

0 pointscausal1y ago0 comments

"Open weights" is a more appropriate term but I'll point out that these weights are also largely inscrutable to the people with the code that trained it. And for licensing reasons, the datasets may not be possible to share.

There is still a lot of modifying you can do with a set of weights, and they make great foundations for new stuff, but yeah we may never see a competitive model that's 100% buildable at home.

Edit: mkolodny points out that the model code is shared (under llama license at least), which is really all you need to run training https://github.com/meta-llama/llama3/blob/main/llama/model.p...

0 comments

7 comments · 7 top-level

stavros1y ago

"Open weights" means you can use the weights for free (as in beer). "Open source" means you get the training dataset and the methodology. ~Nobody does open source LLMs.

9 more replies

aerzen1y ago

LLAMA is an open-weights model. I like this term, let's use that instead of open source.

1 more reply

ab5tract1y ago

If you can’t share the dataset, under what twisted reality are you fine to share the derivative models based on those unsharable datasets?

In a better world, there would be no “I ran some algos on it and now it’s mine” defense.

1 more reply

yangcheng1y ago

latest llama 3.1 is in a different repo, https://github.com/meta-llama/llama-models/blob/main/models/... , but yes, the code is shared. It astonishing that in software 2.0 era, powerful applications like llama has only hundreds of lines of code, and most work hidden in training data. Source code alone is no longer that informative as Software 1.0

danielrhodes1y ago

For models of this size, the code used to train them is going to be very custom to the architecture/cluster they are built on. It would be almost useless to anybody outside of Meta. The dataset would be more a lot more interesting, as it would at the very least show everybody how they got it to behave in certain ways.

twelvechairs1y ago

Open training data would be great too.

If you have open data and open source code you can reproduce the weights

1 more reply

ajxlasA1y ago

Really? I have to check out the training code again. Last time I looked the training and inference code were just example toys that were barely usable.

Has that changed?

j / k navigate · click thread line to collapse