Neural network training as development in program space
In such a framework, the goal of training the neural network is to reach a form of attractor further optimization steps don’t change the state of the neural network.
This attractor should correspond to useful functional properties for the network, a measured by a cost function. Meta-learning corresponds to learning the evolution function itself to make the dynamical system converge to better attractors faster.
A neural network is a program, an algorithm. Its parameters specify a sequence of steps from input data to output prediction. Training a neural network is like moving in the algorithmic space towards programs with better performance according to a given cost function.