Efficient Transformers Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Papers on making transformers faster and cheaper — from linear attention and sparse patterns to kernel and state-space alternatives.

Efficient Transformers: 10 key papers

1. Efficient Transformers: A Survey
   Tay et al. arXiv 2020.
2. Longformer: The Long-Document Transformer
   Beltagy et al. arXiv 2020.
3. Big Bird: Transformers for Longer Sequences
   Zaheer et al. arXiv 2020.
4. Reformer: The Efficient Transformer
   Kitaev et al. arXiv 2020.
5. Linformer: Self-Attention with Linear Complexity
   Wang et al. arXiv 2020.
6. Rethinking Attention with Performers
   Choromanski et al. arXiv 2020.
7. Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
   Katharopoulos et al. arXiv 2020.
8. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
   Dao et al. arXiv 2022.
9. Long Range Arena: A Benchmark for Efficient Transformers
   Tay et al. arXiv 2020.
10. Synthesizer: Rethinking Self-Attention in Transformer Models
    Tay et al. arXiv 2020.

Related reading lists

• Transformers for Sequential Decision Making
• Long-Context & Memory Transformers
• State Space Models & Structured SSMs
• Time Series Forecasting & Temporal Models
• In-Context RL & Prompting
• Knowledge Distillation
• Linear MDPs & Linear Function Approximation
• Kernel Methods & RKHS