Neural networks as dynamical systems

Neural networks, Dynamical systems

Neural networks can be seen as dynamical systems in different contexts.

Recurrent networks

With Recurrent neural networks, the continuous dynamical system analogy is very striking. These networks evolve progressively in time by updating an internal state with a fixed algorithm. Usually the state dynamics are not studied because the recurrent networks is designed to complete some fixed task.

The notion of attractor can be defined for such networks, making them related to the notion of attractor networks.

Training as a dynamical system

For a neural network with parameters \(\theta\) trained with example pairs \((x_i, y_i)\), the parameters trajectory \(\theta_0, \theta_1, \ldots, \theta_n\) until convergence can be interpreted as the evolution of a dynamical system.

One may study the various attractors of that dynamical system and how the training examples affect them.

← Back to Notes