Infrastructure setup and open-source scripts to train 70B model from bare metal (opens in new tab)

Loved this and the detail - thank you. It’s the best inside detail on the engineering work behind these models I’ve ever read.

Two things I’m curious about- first, what, if any difference would you imagine in training a 400b parameter model? It seems that you have plenty of vram across the cluster, but I want to know what you think.

Second, do you think this sort of architecture is the end game for model training? It seems sooo fragile. Are there better shared training mechanisms/architectures? Are there better cluster geometries?

Thanks again - great read.

What happened to the Minecraft-like 3d world your team built? Did you guys pivot?

> outperformed zero-shot GPT-4o

Cool stuff! Does this do RLHF or just pretraining? If the latter, how did you manage to beat GPT 4?

Nice. Tx for the write up

> If even a single one of the over 12,000 connections was a little flaky, it could slow down the entire training run

It's an unusual enough sentence to be remarkable and I was like "I read this exact same sentence before". Indeed, this and most of the writeup appeared on Twitter, LinkedIn, Reddit it seems word-by-word. Is this just spam ?

https://x.com/imbue_ai/status/1805629547473518695

https://reddit.com/r/learnmachinelearning/comments/1dobgbs/t...

https://www.linkedin.com/posts/mattboulos_training-a-70b-mod...

100M in GPUs and they are futzing with dell boxes that have bad Ethernet ports.

Interesting to hear about all the problems they ran into!

Looks correct. They raised $200m from NVIDIA which I presume is in pure GPUs. https://news.crunchbase.com/ai-robotics/new-ai-unicorn-imbue...

> Am I right in understanding, that's over $100 Million worth of GPUs?

Ha! I guess most or many of the readers (who don't have that much of funding) should jump to the next HN submission

thank you for listening!

im not used to conducting these kinds of interviews and felt out of my depth. please suggestions questions that you felt should have been asked but werent.

Are you asking why pay extra for a CPU and RAM? Not everything can be done on a GPU, for example, .png decompression. If you really analyzed your training code and preprocessed your data substantially you could probably get away with very lightweight CPU/RAM resources but I think the reality is that it's such a minor contribution of cost to the overall system (GPU are expensive) that wasting development cycles on that degree of optimization isn't strictly necessary. When you're a hyperscaler you are likely chasing those fractions of a percent of cost efficiency though. To use my original example, you would likely want to preprocess your .png to either .webp (multi-threaded lossless) or .jpeg (lossy), but likely it wouldn't make sense to turn it into a GPU decompressible format as you would save on CPU cost during training but would pay more in storage (and maybe transfer) cost.

Edit: To be more clear, if the CPU work is bottlenecking training, you want to optimize that as much as possible by preprocessing your data/tweaking training scripts. What I'm discussing here is the gap between "fast enough" and "faster":

CPU is not fast enough for training < CPU is exactly fast enough for training < CPU is faster than needed for training

Cause when you have quarter million dollars of GPU on each machine, it is dumb to worry about few thousand for the controlling hardware. Too risky to use something new.

Another problem is that all the hardware, drivers, and experience for GPU are on PC. It would take a lot of work to get running on ARM since would be starting from scratch. Then more work to get it stable. All to save a little on processor.

Keeping the GPUs feed is a actually a rather demanding job for deep learning training. I do not have experience with LLM/NLP, but for image and audio workloads one can struggle to reach full utilization of even a RTX2/3/4xxx GPU with a typical 4-8 core CPU. It does not take much to be bottlenecked by the CPU and/or IO.

Not likely to work. Not many (any) hobbyists can get 400gbps network throughput between each other's GPUs...

Voltage Park is a cloud provider. This is no different from renting barebone infra from AWS, GCP or Azure.

Except Voltage Park, being smaller, is probably more willing to provide some customized setup.

Indeed, they may even see it as a learning opportunity for when they rent similar setups to other customers.

It probably isn't the answer but should be - LAN party.

Either training the next model or inference for the already trained one. In some cases, you might even offer it as a service.