Drinking from a Firehose: Continual Learning with Web-scale Natural Language by Hu, H., Sener, O., Sha, F., & Koltun, V. (2020)

tags
Continual learning
source
(Hu et al. 2020)

Summary

This paper focuses on the problem of (self-)supervised continual learning with deep neural networks. The Firehose dataset introduced by the authors is a large database of timestamped tweets. The goal is to learn a language model for each user from the dataset, which is called Personalized online language learning (POLL).

The authors also introduce a new extension of gradient descent for continual learning. It is based on a replay buffer to retain information about past examples and a validation buffer used to choose the ideal number of gradient steps at each steps.

This new gradient method called ConGraD outperforms Online GD on most settings studied in the paper.