> DeepSeek-R1-Distill models are fine-tuned based on open-source models, using samples generated by DeepSeek-R1.
The amount of confusion on the internet because of this seems surprisingly high. DeepSeek R1 has 670B parameters, and it's not easy to run it on local hardware.
There are some ways to run it locally, like https://unsloth.ai/blog/deepseekr1-dynamic which should let you fit the dynamic quant into 160GBs of VRAM, but the quality will suffer.
Also MLX attempt on a cluster of Mac Ultras: https://x.com/awnihannun/status/1881412271236346233
Ollama and LM Studio cause so much confusion because people simply use their pre-packaged models and not exploring and comparing to what else is available on the market (HuggingFace).
LM Studio though allows you to browse and download straight from Huggingface (assuming GGUF), so people could spend more time looking for models, but I don't think many have the interest to do what many of us do, download 10s of models and compare them against each other to find the best one for our use case.
According to this, you can fit it on a CPU-only setup (no GPUs) with 2 x AMD EPYC CPUs and 24 x 32GB DDR5-RDIMM RAM. About $6000 MSRP for the rig. Doubt you are going to get very many tokens/sec out of it though (6-8, according to the author).
> The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs.
but realistically, >=671GB of VRAM to run at full precision on GPU, or >=131G VRAM to run the most heavily quantized version[1], or >=671GB RAM, and a dose of patience to run on CPU.
There are a lot of low quant ways to run in less RAM, but the quality will be worse. Also, running a distill is not the same thing as running the larger model, so unless you have access to an 8xGPU server with lots of VRAM (>$50k), cpu inference is probably your best bet today.
If the new M4 Ultra Macs have 256GB unified RAM as expected, then you may still need to connect 3 of them together via Thunderbolt 5 in order to have enough RAM to run the Q8 model. Assuming that the speed of that will be faster than the EPYC server, but will need to test empirically once that machine is released.
No… it downloads the 7B model by default. If you think that is large, then you better hold on to your seat when you try to download the 671B model.
I ended up downloading it in case it ever gets removed off the internet for whatever reason. Who knows, if VRAM becomes much cheaper in 10 years I might be able to run it locally without spending a fortune on GPUs!
The other ones are fine-tunes of LLama 3.3 and Qwen2 which have been additionally trained on outputs of the big "Deepseek V3 + R1" model.
I'm happy people are looking into selfhosting models, but if you want to get an idea of what R1 can do, this is not a good way to do so.
Here's how to run deepseek-r1:14b (DeepSeek-R1-Distill-Qwen-14B) and set it to 8k context window:
ollama run deepseek-r1:14b
/set parameter num_ctx 8192
/save deepseek-r1:14b-8k
ollama serveI remember back in late 80s (I was a kid) when we had MSDOS and Windows was just starting. We had all kinds of crazy stuff like Quattro Pro, and later in early 90s we had Wolf3D and all sort of software that was limited due to hardware advances.
With the current state of LLMs... imagine how it is going to look like in 10/15 years when the hardware race goes raging again.
It is also very interesting the position of the different countries: China is in a great position economically and with the amount of talent it has; in addition to its culture and political direction (centralized government that finds it easy to direct policies quickly). Meanwhile, the US is in a more fragile stance, with lots of internal fighting and even pushing [intelligent] people out of the country.
The previous race was to get to the moon (US vs Russia), it seems we now are watching a race to AGI.
Oh, good grief. Amazing how people find ways of squeezing Orange Man Bad into anything and everything.
PS - I am not going to stay awake at night worrying that ICE will be deporting hordes of illegal aliens working at US AI companies.
If you want the optionality of using the full model, there are private hosted models that can be connected in cheaply for those use cases into a "one place for all the models" locally.