- Transformers, GPT
- (Rae et al. 2022)
This model is very similar to GPT-2 but uses RSNorm instead of LayerNorm and relative positional encoding rather than absolute positional encoding.
- Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, et al.. . "Scaling Language Models: Methods, Analysis & Insights from Training Gopher". arXiv. DOI.