undefined | Better HN

0 pointscurrymj1y ago0 comments

Generative is a technical term, meaning that a system models a full joint probability distribution.

For example, a classifier is a generative model if it models p(example, label) -- which is sufficient to also calculate p(label | example) if you want -- rather than just modeling p(label | example) alone.

Similar example in translation: a generative translation model would model p(french sentence, english sentence) -- implicitly including a language model of p(french) and p(english) in addition to allowing translation p(english | french) and p(french | english). A non-generative translation model would, for instance, only model p(french | english).

I don't exactly understand what this judge meant by "generative", it's presumably not the technical term.

0 comments

jll291y ago

Your definition of "generative" as a statistical term is correct.

However, and annoyingly so, recently the general public and some experts have been speaking of "generative AI" (or GenAI for short) when they talk about large language models.

This creates the following contradiction:

- large language models are called "generative AI"

- large language models are based on transformers, which are neural networks

- neural networks are discriminative models (not generative ones like Hidden Markov Models)

- discriminative models are the oppositve of generative models, mathematically

So we may say "Generative AI is based on discriminative (not generative) classifiers and regressors". [as I am also a linguist, I regret this usage came into being, but in linguistics you describe how language is used, not how it should be used in a hypothetical world.]

References

- Gen AI (Wikipedia) https://en.wikipedia.org/wiki/Generative_artificial_intellig...

- Discriminative (Conditional) Model (Wikipedia) https://en.wikipedia.org/wiki/Discriminative_model

echoangle1y ago

Do you have some kind of dictionary where I can find this definition? Because I don’t really understand how that can be the deciding factor of „generative“, and the wiki page for „generative AI“ also seems to use the generic „AI that creates new stuff“ meaning.

By your definition, basically every classifier with 2 inputs would be generative. If I have a classifier for the MNIST dataset and my inputs are the pixels of the image, does that make the classifier generative because the inputs aren’t independent from each other?

currymjOP1y ago

if you have an MNIST classifier that just takes in images, and spits out a probability of digits 1-9, that wouldn't necessarily be generative, if it is only capable of modeling P(which digit | all pixels).

But many other types of model would give you a joint distribution P(which digit, all pixels), so would be generative. Even if you only used it for classification.

https://en.wikipedia.org/wiki/Generative_model

I guess these days "generative" must mean "it is used to generate outputs that look like the training data".

But until recently, the meaning had to do with the information in the model, not how it's used.

zozbot2341y ago

You can derive the latter information (the joint distribution), given the former and a prior over "all pixels"-like data. So, the defining feature of "generative" models is that they feature a prior over their input data?

Nimitz141y ago

Generative models model the data, whether that is p(x) or p(x,y) or (x,y,z) etc.

currymjOP1y ago

Yes, though maybe not explicitly written down.

j / k navigate · click thread line to collapse

0 comments

jll291y ago

Your definition of "generative" as a statistical term is correct.

However, and annoyingly so, recently the general public and some experts have been speaking of "generative AI" (or GenAI for short) when they talk about large language models.

This creates the following contradiction:

- large language models are called "generative AI"

- large language models are based on transformers, which are neural networks

- neural networks are discriminative models (not generative ones like Hidden Markov Models)

- discriminative models are the oppositve of generative models, mathematically

References

- Gen AI (Wikipedia) https://en.wikipedia.org/wiki/Generative_artificial_intellig...

- Discriminative (Conditional) Model (Wikipedia) https://en.wikipedia.org/wiki/Discriminative_model

echoangle1y ago

currymjOP1y ago

But many other types of model would give you a joint distribution P(which digit, all pixels), so would be generative. Even if you only used it for classification.

https://en.wikipedia.org/wiki/Generative_model

I guess these days "generative" must mean "it is used to generate outputs that look like the training data".

But until recently, the meaning had to do with the information in the model, not how it's used.

zozbot2341y ago

Nimitz141y ago

Generative models model the data, whether that is p(x) or p(x,y) or (x,y,z) etc.

currymjOP1y ago

Yes, though maybe not explicitly written down.

j / k navigate · click thread line to collapse