Long-Context & Memory Transformers Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Papers on extending context and adding memory to transformers — from Transformer-XL and Memorizing Transformers to retrieval and extrapolation.

Long-Context & Memory Transformers: 10 key papers

1. Lost in the Middle: How Language Models Use Long Contexts
   Liu et al. arXiv 2023.
2. Efficient Streaming Language Models with Attention Sinks
   Xiao et al. arXiv 2023.
3. Extending Context Window of Large Language Models via Positional Interpolation
   Chen et al. arXiv 2023.
4. YaRN: Efficient Context Window Extension of Large Language Models
   Peng et al. arXiv 2023.
5. LongNet: Scaling Transformers to 1,000,000,000 Tokens
   Ding et al. arXiv 2023.
6. Ring Attention with Blockwise Transformers for Near-Infinite Context
   Liu et al. arXiv 2023.
7. RoFormer: Enhanced Transformer with Rotary Position Embedding
   Su et al. arXiv 2021.
8. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
   Press et al. arXiv 2021.
9. Memorizing Transformers
   Wu et al. arXiv 2022.
10. Unlimiformer: Long-Range Transformers with Unlimited Length Input
    Bertsch et al. arXiv 2023.

Related reading lists

• Efficient Transformers
• Memory-Augmented RL & Recurrent Policies
• State Space Models & Structured SSMs
• Meta-Reinforcement Learning
• Transformers for Sequential Decision Making
• Retrieval-Augmented Generation
• In-Context RL & Prompting
• Time Series Forecasting & Temporal Models