Microgpt explained interactively (opens in new tab)

The rate they are posting new articles on random subjects is also a pretty indicative of a content mill.

In 3 days they've covered machine learning, geometry, cryptography, file formats and directory services.

I didn't get that sense from the prose; it didn't have the usual LLM hallmarks to me, though I'm not enough of an expert in the space to pick up on inaccuracies/hallucinations.

The "TRAINING" visualization does seem synthetic though, the graph is a bit too "perfect" and it's odd that the generated names don't update for every step.

ISWYDT

One of the downsides of using an expert LLM to write for you is that they know all that perfectly well, even if you don't, and aren't too bothered by such a chunk. It's like reading any Wikipedia article on mathematics... This is the kind of thing that people are documenting in the LLM-user literature in creating an illusion of expertise (or 'illusion of transparency'). Because the LLM explains it so fluently, you feel like you understand, even though you don't. Hence new phrases like 'cognitive debt' to try to deal with it.

(This is also why people like cramming or lectures rather than quizzing or spaced repetition, because they produce a certain 'illusion of depth' https://gwern.net/doc/psychology/cognitive-bias/illusion-of-... ).

This answer makes sense if you know that LLMs have layers, if you don't this answer is not super informative.

If I were to describe this to a nontechnical person, I would say:

LLMs are big stacks of layers of "understanders" that each teach the next guy something.

Imagine you are making a large language model that has 4 layers. Each layer will talk to it's immediate neighbor.

The first layer will get the bare minimum, in the LLM's of today, that's groups of letters that are common to come up together, called "tokens". This layer will try to derive a bit of meaning to tell the next layer, such as grouping of letters into words.

The next layer may be a little bit more semantic, for example interpreting that the word "hot" immediately followed by the word "dog" maps to a phrase "hot dog".

The layer after that, becoming a bit more intelligent given it's predecessors have already had some chances at smaller interpretations may now try to group words into bigger blobs, such as "i want a hot dog" as one combined phrase rather than a set of separated concepts.

The final layer may do something even more intelligent afterward, like realize that this is a quote in a book.

The point is that each layer tries to add a little meaning for the next layer.

I want to stress this: the layers do not actually correspond to specific concepts the way I just expressed, the point is that each layer adds a bit more "semantic meaning" for the next layer.

I think the big frustration I've had in learning modern ML is that the entire owl is just so complicated. A poor explainer reads like "black box is black boxing the other black box", completely undecipherable. A mediocre-to-above-average explanation will be like "(loosely introduced concept) is (doing something that sounds meaningful) to black box", which is a little better. However, when explanations start getting more accurate, you run into the sheer volume of concepts/data transforms taking place in a transformer, and there's too much information to be useful as a pedagogical device.

I tried to include tooltips in some places that go into more depth, but I understand there's a jump. I'm not sure what will be the best way to go about it tbh

I have no skin in the game here, but this seems a bit "sharp-edged", do you have something against the guy? He just seems deep into his influencer/retired hobbyist arc to me...

Agree, same as Carmack. They're suit and tie types now.

Is this AI generated?

Tesla FSD isn't a delusion. There are people using it to successfully do long distance drives across the USA right now, without interventions. Dunno how much credit Karpathy gets for that, but the tech works.

What if you just use words