- tags
- Recurrent neural networks
- source
- (Voelker et al. 2019)

## Summary

This paper introduces the LMU recurrent cell. This cell is based on a similar-ish idea to LSTM to maintain a memory hidden state. The main idea of the paper is to make this memory satisfy a set of first order ordinary differential equations.

\begin{equation} \theta \dot{m}(t) = Am(t) + Bu(t) \end{equation}

This system has a solution which represents sliding windows of \(u\) via Legendre polynomials. This new unit is tested on a range of tasks. A memory only task, a permuted MNIST task and a dynamical chaotic system prediction task.

## Comments

Unfortunately, my understanding of this paper is slightly limited. The approach is interesting and has good properties, however the new RNN cell is tested on a small set of task that independently demonstrate useful properties but not all of them together (e.g. good MNIST prediction + long-term dependency).

## Bibliography

- Aaron Voelker, Ivana Kajić, Chris Eliasmith. . "Legendre Memory Units: Continuous-time Representation in Recurrent Neural Networks". In
*Advances in Neural Information Processing Systems 32*, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d\textquotesingle Alché-Buc, E. Fox, and R. Garnett, 15544–53. Curran Associates, Inc.. http://papers.nips.cc/paper/9689-legendre-memory-units-continuous-time-representation-in-recurrent-neural-networks.pdf.