- tags
- Neural networks
Hopfield networks are a kind of recurrent neural network with binary threshold nodes.
Definition
Nodes have indexes \(i \in \{1, \cdots, n\}\) and are in state \(s_i \in \{-1, 1\}\). Nodes have connections between them, characterized by a weight \(w_{ij}\). Each node also has an associated threshold \(\theta_i\) such that
\[ s_i \leftarrow \begin{cases} +1 & \text{if}\ \sum_j w_{ij} s_j \geq \theta_i, \newline -1 & \text{otherwise}. \end{cases} \]
Energy
A Hopfield network has an associated energy value \[ E = - \frac{1}{2} \sum_{i,j} w_{ij} s_i s_j + \sum_i \theta_i s_i \] which makes it part of the Ising models.
Hopfield networks and attention
See Hopfield Networks is All You Need (Ramsauer et al. 2020)
Bibliography
- Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, et al.. . "Hopfield Networks Is All You Need". Arxiv:2008.02217 [cs, Stat]. http://arxiv.org/abs/2008.02217. See notes