undefined | Better HN

0 pointspantulis1y ago0 comments

> Can you quote tps?

LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.

0 pointspantulis1y ago0 comments

> Can you quote tps?

LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.

0 comments

7 comments · 3 top-level

kristianp1y ago· 2 in thread

Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat

Thanks for your suggestion!

I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.

Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?

jychang1y ago· 2 in thread

That's pretty terrible. I'm getting 18tok/sec Gemma 3 27b QAT on a M1 Max 32gb macbook.

I was running the mac in low power mode!!! Getting 24tps now.

Yeah, I know. Not sure if this due to something in LLM Studio or whatever.

Gah, turns out I was running the Mac in low power mode!

I get 24tps in LM Studio now with gemma-3-27b-it-qat.

j / k navigate · click thread line to collapse