Hopfield Networks Is All You Need (opens in new tab)

Brief abstract for the lay person (like me):

1. Hopfield Networks are also known as "associative memory networks", a neural network model developed decades ago by a guy named Hopfield.

2. It's useful to plug these in somehow as layers in Deep Neural Networks today (particularly, in PyTorch).

I hate non-informative titles!

Off-topic, but does anyone know what Jekyll theme this is? Absolutely beautiful formatting and color scheme.

I’ve seen a lot of efforts to add a notion of associative memory into neural networks. Have any exciting applications of such architectures been publicised?

Trending as John Hopfield scheduled to present his "biologically plausible" response to the Modern Hopfield Network at ICLR next week:

Large Associative Memory Problem in Neurobiology and Machine Learning

https://arxiv.org/abs/2008.06996

MHN seem ideal for prediction problems based purely on data, such as chemical reactions and drug discovery:

Modern Hopfield Networks for Few- and Zero-Shot Reaction Prediction

https://arxiv.org/abs/2104.03279

“Sooner or later, everything old is new again.” -Steven King

Quoting: "We introduce a new energy function and a corresponding new update rule which is guaranteed to converge to a local minimum of the energy function."

Is this a minimum in a local area or local in the range of some function? I could see perhaps that'd being an advantage if you happen to know that local part of the range

In contrast we're usually looking for global min/max say with annealing algorithms. How is local is better in the context of this paper than global?

I looked at the paper but it was way over my head.

Can anyone explain it in simpler terms to a person who barely understands attention models and has no idea what associative memory means here?

Nice paper! I used Hopfield networks in the 1980s. I hope that I can clear a few hours of time this week to work through this. I admit that for machine learning, that I have fell into the “deep learning for everything pit” in the last six or seven years. Probably because DL is what I usually get paid for.

This reminded me of a very old fun side project of mine [1] that had made me look at neural networks from a different perspective.

[1] https://github.com/milosgajdos/gopfield

If I understood them correctly, they store all the training samples and then select one most similar to a given input.

Are*

Amazing