undefined | Better HN

0 pointszingar6h ago0 comments

Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?

0 comments

1 comments · 1 top-level

nok22kon2h ago

paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit

https://arxiv.org/abs/2212.09720

j / k navigate · click thread line to collapse