undefined | Better HN

0 pointsenlyth3y ago0 comments

I've been told the 4 bit quantization slows it down, but don't quote me on this since I was unable to benchmark at 8 bit locally

In any case, you're right it might not be as significant, however, the quality of the output increases with 8/16bit, and running 65B is completely impossible on 24GB

0 comments

riku_iki3y ago

It's not impossible, there are several projects which load model layer by layer for execution from the disk or ram, but it will be much slower.

j / k navigate · click thread line to collapse

0 pointsenlyth3y ago0 comments

I've been told the 4 bit quantization slows it down, but don't quote me on this since I was unable to benchmark at 8 bit locally

In any case, you're right it might not be as significant, however, the quality of the output increases with 8/16bit, and running 65B is completely impossible on 24GB

0 comments

riku_iki3y ago

It's not impossible, there are several projects which load model layer by layer for execution from the disk or ram, but it will be much slower.

j / k navigate · click thread line to collapse