Block MDPs & Latent State Discovery Reading List

Curated by Mouhssine Rifaki | Stanford Electrical Engineering | Last updated August 2026

When observations are rich but the underlying state is discrete and hidden, Block MDPs provide a clean model for representation learning in RL. This list covers decodability, optimism with function approximation, and provable exploration under latent structure.

Block MDPs & Latent State Discovery: 10 key papers

1. Provably efficient RL with Rich Observations via Latent State Decoding
   Du et al. arXiv 2019.
2. Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learning
   Misra et al. arXiv 2019.
3. Learning the Linear Quadratic Regulator from Nonlinear Observations
   Mhammedi et al. arXiv 2020.
4. Invariant Causal Prediction for Block MDPs
   Zhang et al. arXiv 2020.
5. Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
   Agarwal et al. arXiv 2021.
6. DeepMDP: Learning Continuous Latent Space Models for Representation Learning
   Gelada et al. arXiv 2019.
7. Learning Domain Invariant Representations in Goal-conditioned Block MDPs
   Han et al. arXiv 2021.
8. Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approach
   Zhang et al. arXiv 2022.
9. Agnostic Reinforcement Learning with Low-Rank MDPs and Rich Observations
   Dann et al. arXiv 2021.
10. Representation Learning for Online and Offline RL in Low-rank MDPs
    Uehara et al. arXiv 2021.

Related reading lists

• Linear MDPs & Linear Function Approximation
• Causal Discovery & Structure Learning
• Representation Learning for RL
• Low-Rank Structure in Machine Learning
• Goal-Conditioned Reinforcement Learning
• Safe Reinforcement Learning & Constrained MDPs
• Causal Inference & RL
• Robust & Risk-Sensitive RL