This is quite a beautiful, strange (estranging?) clause - at least in the sense that we (or I) constantly struggle to find meaning and patterns in what might simply be plain noise (apophenic beauty?). It’s a similar form of intrigue that I and I think others often experience when reading the outputs of LLMs operating in the high-temperature regime, though of course we are just talking about embedding/embedding inversion here.
On a human level though, it makes me wonder why you picked that phrase. Did you roll dice in front of a dictionary? Play madlibs? Were they the first words that came to your mind? Or perhaps you went through several iterations to come up with the perfectly meaningless combination? Or perhaps you simply spilled your hot chocolate on your favorite pair of pants or dress while getting out of the car this morning (or perhaps as a child) and the memory has stuck with you… who knows! Only you!
In any case, my original point was simply that these interstitial points in embedding spaces can become ways of referring to or communicating ideas that we simply do not have the words for but which are none-the-less potentially useful in a communication between two entities that both have the ability to come to some roughly shared understanding of what is being referred to or expressed by that point in the embedding space. Regular languages of course invent new words all the time, and yet the points those new words map to in the embedding space always existed (eh not a great example because the shape of the embedding space might change as new words/tokens are introduced to the lexicon but I think the idea holds). Perhaps new words or phrases will come about to bring some point back into textual space; or perhaps that point will remain solely in the shared lexicon of the algorithmic systems using the latent space to communicate ideas. Again, who knows!
For instance, consider the midpoint of a segment connecting two ideas, or the centroid of any simplex in the embedding space… if we assume that there is some sort of well-defined semantic structure in the space, is it necessarily the case that the centroid must refer to something which equally represents all of the nodes, a kind of lowest-common semantic denominator? Obviously if the semantic structure only holds over local regions but breaks down globally this is not the case, but if all the points are within a region of relatively sound semantic structure, that seems plausible. We know what happens when you do a latent space traversal for a VAE which generates images, and it can be quite beautiful and strange (or boring and familiar by 2024, depending on your perspective), but some similarly weird process might be possible with embedding space traversals, if only we could some how phenomenologically if not linguistically decode those interpolating points.
> concepts which have essentially no meaning
This is a pretty strange idea to try to wrap your head around.
