undefined | Better HN

0 pointsdnautics5y ago0 comments

Note that the example here is forward differentiation and most machine learning these days is backward propagation

0 comments

4 comments · 2 top-level

oxinabox5y ago· 1 in thread

Note that this applies backwards also. Of the 7 ADs demoed, only 1 was forward. The other 6 were reverse mode. All gave same result

dnauticsOP5y ago

Of course, sorry didn't mean to imply that AD was useful only in one direction. Haha I guess I was caught only reading 1/7 of the article!!

rockinghigh5y ago· 1 in thread

Backward propagation computes the gradient of the loss function with respect to the weights. A gradient is a vector or tensor of derivatives. You need to differentiate the loss function somehow to evaluate this vector.

dnauticsOP5y ago

That's not strictly speaking true. You also calculate the gradient of the input. Otherwise, you wouldn't be able to backpropagate over more than one layer. It's just useful because you can do both easily, where with forward mode getting derivatives with respect to weights is considerably more expensive.

j / k navigate · click thread line to collapse