undefined | Better HN

0 pointsOtternonsenz6h ago0 comments

Is there any hope for people that cant even run 27B parameters, Qwen3.6 or otherwise? Are there any quantized models that do well with tool calling at smaller parameter sizes?

I do not have a crazy rig, a modest gaming one at that, but in trying to understand more about agents and their capabilities, I am SOL with my 16 GB of RAM and 8GB of VRAM. I can get most small, non tool calling models to perform well, but I've had major issues with anything over 9B doing anything more than reasoning (egregiously slow at higher parameter counts).

And so far, I cant get even Pi to extend itself or do any meaningful work with any of the models I currently can get to run.

0 comments

8 comments · 4 top-level

fumeux_fume6h ago· 2 in thread

I suspect with those specs, you're not in the game right now for reliably using local models for code generation. The easiest way in is a MacBook with at least 32GB of RAM. This should be able to run a 4bit quantization of qwen 3.6 using the MLX format really well.

OtternonsenzOP5h ago

Now that I’m dipping more into this space, am gonna see what I can upgrade with the motherboard I have, but RAM pricing as it is, I’ll need to be smart about when I upgrade.

I very much appreciate the frank response, as it makes me feel less defeated at knowing my understanding of how it should work is not the full issue, hahaha

fumeux_fume4h ago

M series macs are usually used for running these LLMs locally because the GPU and CPU share the same pool of RAM at very low latency. If you upgrade your RAM on a different kind of chipset without the Unified Memory Architecture, then it'll be much slower to produce all the tokens you need. Just another data point to add to your upgrade equation.

fluoridation5h ago· 2 in thread

I think at 16 GB you'd struggle to run the regular development tools nowadays, forget about any interesting inference.

OtternonsenzOP5h ago

Fully agreed, and my hope is as open models grow and change, that getting some amount of this working on Pro-sumer hardware will be more attainable.

But certainly seems like we are a few years away from that, sadly.

Am I also screwed in being able to train my own small model or adjust another one with such a non-workhorse PC?

fluoridation5h ago

Training requires even beefier hardware than inference.

jboss104h ago

I have 8GB VRAM but 32GB RAM. Qwen 3.6 35B runs nicely.

You should look at gemma-4-26B-A4B. 16+8=24gb and Q4 is about 16GB. Not much context left, but might run.

jboss104h ago

I have 8GB VRAM, but 32GB sys ram. I can run qwen 3.6 35B at 30 tok/s. I also use pi, and it's smart enough to extend itself(multishot and maybe a few tries)

For you, you could try gemma-4-26B-A4B

j / k navigate · click thread line to collapse

0 comments

8 comments · 4 top-level

fumeux_fume6h ago· 2 in thread

OtternonsenzOP5h ago

Now that I’m dipping more into this space, am gonna see what I can upgrade with the motherboard I have, but RAM pricing as it is, I’ll need to be smart about when I upgrade.

I very much appreciate the frank response, as it makes me feel less defeated at knowing my understanding of how it should work is not the full issue, hahaha

fumeux_fume4h ago

fluoridation5h ago· 2 in thread

I think at 16 GB you'd struggle to run the regular development tools nowadays, forget about any interesting inference.

OtternonsenzOP5h ago

Fully agreed, and my hope is as open models grow and change, that getting some amount of this working on Pro-sumer hardware will be more attainable.

But certainly seems like we are a few years away from that, sadly.

Am I also screwed in being able to train my own small model or adjust another one with such a non-workhorse PC?

fluoridation5h ago

Training requires even beefier hardware than inference.

jboss104h ago

I have 8GB VRAM but 32GB RAM. Qwen 3.6 35B runs nicely.

You should look at gemma-4-26B-A4B. 16+8=24gb and Q4 is about 16GB. Not much context left, but might run.

jboss104h ago

I have 8GB VRAM, but 32GB sys ram. I can run qwen 3.6 35B at 30 tok/s. I also use pi, and it's smart enough to extend itself(multishot and maybe a few tries)

For you, you could try gemma-4-26B-A4B

j / k navigate · click thread line to collapse