I was more making a distinction between embeddings and bag of words which are very very sparse matrices. The embedding dimensionality will not be anywhere near as high so this level of sparsity is a minor inconvenience.

Edit: also CPUs for this, yikes…