' Why would pretraining a 1.1B model for so long make sense? Doesn't it contradict the Chinchilla Scaling Law?
Above is the training loss curve taken from the Llama 2 paper. Here I quote from that paper: "We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation". That is why we believe pretraining a 1.1B model for 3T tokens is a reasonable thing to do. Even if the loss curve does not go down eventually, we can still study the phenomenon of saturation and learn something from it.'
It is something I have been wondering about: why did Meta not keep the training process going on while the loss curves seemed to go down? Could they conceivably release a Llama 2.1 being checkpoints taken a month after 2.0 was 'cut'? Maybe the expected gain is too small compared to what can be gained with fine/instruct tuning afterward anyway?
Because choosing the LR decay requires knowing the # of steps in advance. LR is too small after the 2T tokens, and changing it afterwards doesn't tend to help.
If I remember correctly, it's because the main reason they trained multiple models was to show a scaling trend. Each model was trained using a chinchilla-optimal mix of model size, cpu amount, and parameter size. The point was to provide an empirical scaling law that could possibly be extrapolated to estimate the performance of more expensive models, like imagine a billion dollar model for which the model size, data size, and cpu amount is picked in the chinchilla optimal ratios.
On small models the chinchilla optimal scaling stops training the model even when the model is still improving.
The problem comes when people are actually using these small llama models rather than treating them as just data points. If you are actually using these models, what you want is one that is trained forever on as many tokens and training time as possible.
Get together 5 people in that position and it's less than a week's income for the group. That sounds doable as a hobby for those lucky people.
More realistically, it's within range for a grant, or use of someone else's hardware if they aren't using it, as the sibling comment from wongarsu said.
Also cloud vendors sometimes give out large batches of credits to startups and such as marketing incentive to get future customers.
That's a lot for a hobby, but small enough that it might be running on a university machine (the TinyLlama devs provide a way to cite them and all seem to work or study at Singapore University of Technology) or could be sponsored (no indication of that now, but "people made an awesome model in our cloud" is good advertisement). Government grants or grants in general also aren't out of the question, especially for a topic with this much hype.
They are training the model on 3000/22=136 times the value of the chinchilla scale. It will be interesting to see how much it will improve after way beyond this value.
?? A 3060 or a slightly bigger AMD/Intel GPU can stream llama 7B about as fast as someone can read, if not faster. A somewhat bigger consumer GPU can batch it and serve dozens of users.
I use 13B finetunes on my 2020 14" laptop all the time, with 6GB of VRAM and 16GB of CPU RAM.
I have seen many people on HN say this, and I can't help but wonder why the optimized, quantized llama implementations are flying under the radar.
That's the thing: you need a whole GPU per concurrent user, this is insanely expensive if you want to run it as part of a SaaS (which is what most for-profit want to do). Of course running models locally is much better in almost every regard, but nobody is gonna be a billionaire with that…
And I'm not sure what you mean by inference latency being infeasible. Most people using thsss models at home don't even bother with the 7B and go straight to 13B because it's easy to run too and much smarter. And any cloud gpu can run 13B.
Also, when are we going to start seeing open weights MOE models being released?
2- The only 2 i know of are airoboros[1] and Hydra which is still in progress.
[0] https://x.com/ggerganov/status/1698667093711880687?s=46&t=Jp...
Hydra, is this it? https://github.com/SkunkworksAI/hydra-moe
The "main" training step using huge amounts of inputs is called pre-training. The idea is that after that pre-training, you might fine tune the model for your specific use case.
2T not saturating on a 7B is very different from 3T on a 1B.
It’s my understanding that the entire race to ever-more parameters was driven by that.
Newer large datasets like the ones used here optimize for diversity. (e.g. SlimPajama is a heavily-deduped dataset)
Yeah, the line keeps going down as the model gets bigger. What's your point? That there's a hump in the middle?
AFAIU slim pajama is about 627B tokens, and Starcoder:
> approximately 250 Billion tokens.
Ed: I see TFA says:
> Combined Dataset Size - Around 950B tokens
> Total Tokens During Training - 3 trillion (slightly more than 3 epochs/1430k steps)
... but I'm not seeing how one becomes three? That's more like 1 trillion than 3 trillion tokens?
