undefined | Better HN

0 pointsaria11y ago0 comments

In practice you only need to do $H^{-1} g$; L-BFGS stores the {s_k} and {y_k} vectors which allow you to do the $H^{-1} g$ directly rather than needing to ever form the hessian or its inverse. There are techniques that aren't BFGS-based which approximate the hessian rather than the inverse and in that case you'd be better off solving.

0 comments

lliiffee11y ago

What I mean is that, if just doing regular Newton, I'm not sure I've ever seen an example using the interface you show. I think that the vast majority of Newton's method implementations pass H rather than H^-1, and get the search direction via "solve_H_g(H,g)" rather than "solve_iH_g(iH,g)". It takes cubic time to compute H^-1 from H (after which you can compute (H^-1)g) and it takes cubic time to solve for x such that H*x=g, but the rather is said to be more stable, and so preferred.

I know this is irrelevant to the main part of the post, which is to explain LBFGS, I'm just genuinely interested if there are applications where its better to pass the inverse of H rather than H itself for some reason.

ariaOP11y ago

The interesting part of BFGS is that it directly approximates H^{-1} rather than H.

j / k navigate · click thread line to collapse

0 pointsaria11y ago0 comments

0 comments

lliiffee11y ago

ariaOP11y ago

The interesting part of BFGS is that it directly approximates H^{-1} rather than H.

j / k navigate · click thread line to collapse