OpenLLaMA 7B Training Completed to 1T Tokens (opens in new tab)

(huggingface.co)

58 pointsjncraton3y ago3 comments

3 comments

3 comments · 2 top-level

fancyfredbot3y ago· 1 in thread

This is great. Based on the throughout of 2200 tokens/sec and the 1,000,000,000,000 tokens used to train this was at least $183k worth of compute (that's based on the three year committed use rate). And now we can have it for free!

thawab3y ago

The price for training their 7B, as stated by MosaicML[0] and Falcon 7B, is roughly the same.

[0] https://twitter.com/MosaicML/status/1660738892306485248

mdaniel3y ago

be sure to read the warning in their repo: https://github.com/openlm-research/open_llama#loading-the-we...

> Please note that it is advised to avoid using the Hugging Face fast tokenizer for now, as we’ve observed that the auto-converted fast tokenizer sometimes gives incorrect tokenization

j / k navigate · click thread line to collapse