Hierarchical RL & Options Framework Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

Temporal abstraction turns long-horizon problems into manageable subproblems. From Sutton's options to modern subgoal and skill-discovery methods, this collection highlights how hierarchy enables exploration, transfer, and long-term credit assignment.

Hierarchical RL & Options Framework: 10 key papers

1. FeUdal Networks for Hierarchical Reinforcement Learning
   Vezhnevets et al. arXiv 2017.
2. The Option-Critic Architecture
   Bacon et al. arXiv 2016.
3. Data-Efficient Hierarchical Reinforcement Learning
   Nachum et al. arXiv 2018.
4. Diversity is All You Need: Learning Skills without a Reward Function
   Eysenbach et al. arXiv 2018.
5. Latent Space Policies for Hierarchical Reinforcement Learning
   Haarnoja et al. arXiv 2018.
6. Strategic Attentive Writer for Learning Macro-Actions
   Vezhnevets et al. arXiv 2016.
7. Learning Multi-Level Hierarchies with Hindsight
   Levy et al. arXiv 2017.
8. Dynamics-Aware Unsupervised Discovery of Skills
   Sharma et al. arXiv 2019.
9. Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
   Nachum et al. arXiv 2019.
10. Sub-policy Adaptation for Hierarchical Reinforcement Learning
    Li et al. arXiv 2019.

Related reading lists

• Intrinsic Motivation & Empowerment
• Hindsight Experience Replay & Relabeling
• Time Series Forecasting & Temporal Models
• Bayesian Optimization & Global Optimization
• Neural Architecture Search
• Data Augmentation for Reinforcement Learning
• Block MDPs & Latent State Discovery
• Zero-Shot RL & Generalization