Transformers

tags

Neural networks

Transformers are a neural network architecture based on a mechanism called Attention.

They have been particularly successful for NLP applications.

Backlinks

• Language modeling
• Notes on: Information-Theoretic Probing with Minimum Description Length by Voita, E., & Titov, I. (2020)
• Notes on: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention by Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020)