I can only state that KAN paper used LBFGS and that LBFGS is remarkably different from SGD.

One of the differences is a dynamic learning rate guided by approximation of the local curvature.