undefined | Better HN

Skip to content

Top Best Ask Show New Jobs

0 pointsnextaccountic2d ago0 comments

> The model invents new categories (e.g. apartments) and doesn’t stick to the provided list of allowed categories

Can this specific failure mode be solved by providing a grammar that the output must adhere to? (Not sure if Qwen has this feature, it's used for eg. to ensure the output is parseable json)

0 comments

6 comments · 3 top-level

nl2d ago· 3 in thread

It can.

It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself.

Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking).

nextaccounticOP1d ago

Just one question. If I'm running a local model, can I do something other than just a context free grammar? Does it makes sense to have something more general, or it would be just too slow?

I guess the only hard constraint is to not have backtracking, right? To not waste previously emitted tokens

aesthesia2d ago

Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a </think> token, then do constrained decoding, right?

Sure, but does llama-cpp support that?

thomascountz2d ago

Yes, you can use constrained decoding like logit masking to force all invalid tokens in the vocabulary to -inf, and effectively be removed from selection. I believe llama.cpp exposes this by accepting a formatted grammar.

mijoharas2d ago

This was my thought as well. I'm surprised that it's not being used here (afaict)

j / k navigate · click thread line to collapse