Knowledge Distillation Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Foundational and modern papers on knowledge distillation — from Hinton's original formulation to distribution-aware and data-free methods.

Knowledge Distillation: 10 key papers

1. Distilling the Knowledge in a Neural Network
   Hinton et al. arXiv 2015.
2. FitNets: Hints for Thin Deep Nets
   Romero et al. arXiv 2014.
3. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
   Zagoruyko and Komodakis. arXiv 2016.
4. A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning
   Yim et al. 2017.
5. Born Again Neural Networks
   Furlanello et al. arXiv 2018.
6. Relational Knowledge Distillation
   Park et al. arXiv 2019.
7. Contrastive Representation Distillation
   Tian et al. arXiv 2019.
8. Knowledge Distillation: A Survey
   Gou et al. arXiv 2020.
9. Self-training with Noisy Student improves ImageNet classification
   Xie et al. arXiv 2019.
10. TinyBERT: Distilling BERT for Natural Language Understanding
    Jiao et al. arXiv 2019.

Related reading lists

• Graph Neural Networks & Message Passing
• Multi-Task & Transfer Reinforcement Learning
• In-Context RL & Prompting
• Efficient Transformers
• Exploration in Reinforcement Learning
• Privacy & Differential Privacy in Machine Learning
• Optimization for Deep Learning
• Vision-Language Models & Multimodal Learning