undefined | Better HN

0 pointssimonw1y ago0 comments

The tool you are using may set a default max output size without you realizing. Ollama has a num_ctx that defaults to 2048 for example: https://github.com/ollama/ollama/blob/main/docs/faq.md#how-c...

0 comments

3 comments · 1 top-level

nico1y ago· 2 in thread

Been playing with that, but doesn’t seem to have much effect. It works very well to limit output to smaller bits, like setting it to 100-200. But above 2-4k the output seems to never get longer than about 1 page

Might try using the models with mlx instead of ollama to see if that makes a difference

Any tips on prompting to get longer outputs?

Also, does the model context size determine max output size? Are the two related or are they independent characteristics of the model?

simonwOP1y ago

Interestingly the Gemma 3 docs say: https://ai.google.dev/gemma/docs/core/model_card_3#:~:text=T...

> Total output context up to 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size per request, subtracting the request input tokens

I don't know how to get it to output anything that length though.

nico1y ago

Thank you for the insights and useful links

Will keep experimenting, will also try mistral3.1

edit: just tried mistral3.1 and the quality of the output is very good, at least compared to the other models I tried (llama2:7b-chat, llama2:latest, gemma3:12b, qwq and deepseek-r1:14b)

Doing some research, because of their training sets, it seems like most models are not trained on producing long outputs so even if they technically could, they won’t. Might require developing my own training dataset and then doing some fine tuning. Apparently the models and ollama have some safeguards against rambling and repetition

1 more reply

j / k navigate · click thread line to collapse