Scaling down Deep Learning by Greydanus, S. (2020)

tags
Neural networks
source
(Greydanus 2020)

Summary

This paper introduces a minimalist 1D version of the MNIST dataset for studying some basic properties of neural networks. The authors simplify the MNIST dataset by assigning a 1D glyph to each digit. These glyphs are padded, translated, sheared and blurred to build a dataset of multiple different objects.

The figure from the paper shown below illustrates this dataset’s construction:

1D simple MNIST

This dataset is shown to be a suitable tool to study very small neural networks and their properties. The authors discuss:

Comments

This is an interesting paper, which for once isn’t about getting bigger models or better results than the state of the art. The small MNIST dataset may lead to interesting insights into the functioning of neural networks but I’m not sure that will be enough for solving the big questions about deep learning. The secrets of those huge recent models might elude us for a while simply because we don’t have the resources (mathematical or computational) to study them.

Bibliography

. . “Scaling down Deep Learning”. arXiv:2011.14439 [Cs, Stat], November.


← Back to Notes