undefined | Better HN

0 pointsTepix1y ago0 comments

Interesting, what RAM do you use exactly? 24x 16GB DDR5-6000 DIMMs? It seems that those boards only support up to DDR5-4800: https://geizhals.de/?cat=mbsp3&xf=4921_2%7E493_24x+DDR5+DIMM...

Does the core count matter or can you get away with the smallest 2x EPYC 9015 configuration? What are "good speeds"?

0 comments

4 comments · 2 top-level

DrPhish1y ago· 2 in thread

I use 24 sticks of ddr5-4800, which gets me up to 9t/s on deepseek 2.5 at q8. 48 threads was optimal in llama.cpp. I would like to move to epyc 9005 chips and ddr5-6000, but it is cost prohibitive with CPUs still over $10k each on eBay.

I followed the guide at https://rentry.co/miqumaxx/

TepixOP1y ago

How many cores do your CPUs have? Are you using the 64 core EPYC 9334 mentioned in the linked page? Do that many cores provide a speedup versus having fewer cores?

DrPhish1y ago

Yes I have the same engineering sample. I can only keep 48 cores fed with max memory bandwidth given current llama.cpp architectural constraints

TepixOP1y ago

Looks like we will soon get boards supporting 24x DDR5-6000 for the EPYC 9005 CPUs.

j / k navigate · click thread line to collapse