undefined | Better HN

0 pointspdyc2mo ago0 comments

can you elaborate? you can use quantized version, would context still be an issue with it?

0 comments

2 comments · 2 top-level

A usable quant, Q5_KM imo, takes up ~26GB[0], which leaves around ~6-7GB for context and running other programs which is not much.

context is always an issue with local models and consumer hardware.

j / k navigate · click thread line to collapse