Architecture
RoFormer: Enhanced Transformer with Rotary Position Embedding
Zhuiyi Technology·April 20, 2021
Jianlin Su, Yu Lu, Shengfeng Pan
View on arXivTL;DR
Introduces rotary position embeddings (RoPE), now the standard way LLMs encode position.
Why it matters
RoPE quietly became near-universal in modern open and closed LLMs.