undefined | Better HN

0 pointscs7025y ago0 comments

The model is sparsely-gated, not sparse. The individual experts in each mixture of experts are dense layers but they're sparsely activated, i.e., on each forward pass only some of them are conditionally used.

As to comparing parameter counts, I disagree with you. I think it's perfectly OK to compare parameter counts for different kinds of models. It would also be perfectly OK to compare, say, computational efficiency per parameter in each forward pass (which for this model is impressive), but that wasn't the focus on my comment above.

Finally, you're right that I didn't mention all the interim parameter counts that we have seen below 600B in all transformer variants. The list would have been way too long had I tried to include every figure!

0 comments

gwern5y ago

Probably the most relevant comparison here would be a mix of wallclock-hours and FLOPS. The MoE may be inefficient on a parameter level, but it may be the most efficient way to convert FLOPS into model power (sort of like how you currently do better making models wider than deeper - experts are the ultimate 'width').

cs702OP5y ago

It depends on your goal. If you want to measure the number of "artificial synapses" or connections, total parameters is the right figure to use, because each weight is one such connection. If you want to measure the computational cost of training or inference, then wallclock-hours and FLOPs would be better figures.

The 100's of trillions of connections (synapses) in the human brain are sparsely used -- i.e., your entire brain doesn't light up in response to every single stimulus. But we still talk about 100's of trillions of synapses when we refer to the size of the human brain's connectome. It's a perfectly valid way of measuring model size.

More to your point, the authors measure the computational cost of training in Table 3 of the paper in TPU-core-years for the various mixture-of-expert models, and compare them to an always-densely-used variant.

ma2rten5y ago

Fair enough, sparse usually means weights are sparse and not activations.

Obviously you can compare parameter count if you really want to, but from a technical point of view training a densely activated model is a much bigger feat. Also, I have personally spoken to one of the authors of this paper and they said sparsely activated models tend to well better on tasks that require knowledge but not tasks that require intelligence (e.g. GLUE).

cs702OP5y ago

I agree, training a dense model with the same number of parameters would be much a bigger feat.

Otherwise, as I mentioned elsewhere on this page, we routinely describe the size of the human brain in terms of numbers of synapses (connections), even though they are sparsely activated. Only a small subset of your brain 'lights up' for a given input. Number of parameters (connections) is a perfectly sensible way to measure model size.

Anyway, I expect we will see both much larger sparsely and densely activated models going forward. We live in interesting times :-)

j / k navigate · click thread line to collapse

0 pointscs7025y ago0 comments

0 comments

gwern5y ago

cs702OP5y ago

ma2rten5y ago

Fair enough, sparse usually means weights are sparse and not activations.

cs702OP5y ago

I agree, training a dense model with the same number of parameters would be much a bigger feat.

Anyway, I expect we will see both much larger sparsely and densely activated models going forward. We live in interesting times :-)

j / k navigate · click thread line to collapse