- tags
- Neural networks
Transformers are a neural network architecture based on a mechanism called Attention.
They have been particularly successful for NLP applications and also penetrated other fields of machine learning such as Computer vision.
Transformers
Links to this note
- Byte-pair encoding
- Language modeling
- Notes on: Information-Theoretic Probing with Minimum Description Length by Voita, E., & Titov, I. (2020)
- Notes on: Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention by Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021)
- Notes on: Pretrained Transformers as Universal Computation Engines by Lu, K., Grover, A., Abbeel, P., & Mordatch, I. (2021)
- Notes on: The geometry of integration in text classification RNNs by Aitken, K., Ramasesh, V. V., Garg, A., Cao, Y., Sussillo, D., & Maheswaranathan, N. (2020)
- Notes on: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention by Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020)
- Self-supervised learning