Schedule and Syllabus

Unless otherwise specified the course lectures and meeting times are:

Tuesday, Thursday 4:30-5:50
Location: NVIDIA Auditorium
EventDateDescriptionCourse Materials
Lecture Jan 9 Introduction to NLP and Deep Learning
Suggested Readings:
  1. [Linear Algebra Review]
  2. [Probability Review]
  3. [Convex Optimization Review]
  4. [More Optimization (SGD) Review]
Lecture Jan 11 Word Vectors 1
Suggested Readings:
  1. [Word2Vec Tutorial - The Skip-Gram Model]
  2. [Distributed Representations of Words and Phrases and their Compositionality]
  3. [Efficient Estimation of Word Representations in Vector Space]
A1 released Jan 11 Assignment #1 released [Assignment #1][ Written Solutions ]
Lecture Jan 16 Word Vectors 2
Suggested Readings:
  1. [GloVe: Global Vectors for Word Representation]
  2. [Improving Distributional Similarity with Lessons Learned fromWord Embeddings]
  3. [Evaluation methods for unsupervised word embeddings]
Lecture Jan 18 Neural Networks
Suggested Readings:
  1. cs231n notes on [backprop] and [network architectures]
  2. [Review of differential calculus]
  3. [Natural Language Processing (almost) from Scratch]
  4. [Learning Representations by Backpropagating Errors]
Review Jan 19 Python Refresher [ slides ]
Lecture Jan 23 Backpropagation and Project Advice
[slides] [lecture notes]
Suggested Readings:
  1. [Derivatives, Backpropagation, and Vectorization]
  2. [Yes you should understand backprop]
Lecture Jan 25 Introduction to TensorFlow
[slides] [lecture code]
Suggested Readings:
  1. [TensorFlow Basic Usage]
A1 Due Jan 25 Assignment #1 due
A2 Released Jan 25 Assignment #2 released [Assignment #2] [ Written Solutions ]
Lecture Jan 30 Dependency Parsing
Suggested Readings:
  1. Joakim Nivre. 2004. Incrementality in Deterministic Dependency Parsing. Workshop on Incremental Parsing.
  2. Danqi Chen and Christopher D. Manning. 2014. A Fast and Accurate Dependency Parser using Neural Networks. EMNLP 2014.
  3. Sandra K├╝bler, Ryan McDonald, Joakim Nivre. 2009. Dependency Parsing. Morgan and Claypool. [Free access from Stanford campus, only!]
  4. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally Normalized Transition-Based Neural Networks. ACL 2016.
  5. Marie-Catherine de Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D. Manning. 2014. Universal Stanford Dependencies: A cross-linguistic typology. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). Revised version for UD v1.
  6. Universal Dependencies website
Lecture Feb 1 Recurrent Neural Networks and Language Models
Suggested Readings:
[N-gram Language Models and Perplexity]
[The Unreasonable Effectiveness of Recurrent Neural Networks]
[Recurrent Neural Networks Tutorial]
[Sequence Modeling: Recurrent and Recursive Neural Nets]
DFP Released Feb 1 Default Final Project released
Lecture Feb 6 Vanishing Gradients, Fancy RNNs
Suggested Readings:
[Understanding LSTM Networks]
[Vanishing Gradients Example]
Review Feb 8 Midterm Review
This year's midterm will be most similar to practice midterm 3 (the first two are from cs224d).
[practice midterm 1] [with solutions]
[practice midterm 2] [with solutions]
[practice midterm 3] [with solutions]
Project Proposal Due Feb 8 Final Project proposal due Final Project Proposal
A2 Due Feb 8 Assignment #2 due
Alternate Midterm Feb 9 Alternate Midterm
A3 Released Feb 13 Assignment #3 released Assignment #3 [ Written Solutions ]
Midterm Feb 13 In-class midterm Location: Memorial Auditorium, Time: 4:30 - 5:50pm
[Midterm Solutions]
Lecture Feb 15 Machine Translation, Seq2Seq and Attention
Suggested Readings:
[Statistical Machine Translation slides (see lectures 2/3/4)]
[Statistical Machine Translation Book]
[BLEU metric]
[Original sequence-to-sequence NMT paper (also describes beam search)]
[Earlier sequence-to-sequence speech recognition paper (includes detailed beam search alg)]
[Original sequence-to-sequence + attention paper]
[Guide to attention and other RNN augmentations]
[Massive Exploration of Neural Machine Translation Architectures]
Lecture Feb 20 Advanced Attention
Suggested Readings:
[A Deep Reinforced Model for Abstractive Summarization]
[Get To The Point: Summarization with Pointer-Generator Networks]
[BlackOut: Speeding up Recurrent Neural Network Language Models with very Large Vocabularies]
[Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models]
[Quasi-Recurrent Neural Networks]
Lecture Feb 22 Transformer Networks and CNNs
Suggested Readings:
[Attention Is All You Need]
[Layer Normalization]
[Convolutional Neural Networks for Sentence Classification]
[Improving neural network3s by preventing co-adaptation of feature detectors]
[A Convolutional Neural Network for Modelling Sentences]
Lecture Feb 27 Coreference Resolution
Suggested Readings:
[Learning Anaphoricity and Antecedent Ranking Features for Coreference Resolution]
[Improving Coreference Resolution by Learning Entity-Level Distributed Representations]
[End-to-end Neural Coreference Resolution]
[Coreference Demo]
A3 Due Feb 27 Assignment #3 due
Milestone Due Feb 28 Final project milestone due Project Milestone
Lecture Mar 1 Tree Recursive Neural Networks and Constituency Parsing
Lecture Mar 6 Advanced Architectures and Memory Networks
Lecture Mar 8 Reinforcement Learning for NLP Guest Lecture
Suggested Readings:
[A Deep Reinforced Model for Abstractive Summarization]
[DCN+: Mixed Objective and Deep Residual Coattention for Question Answering]
[Deep Reinforcement Learning for Dialogue Generation]
Lecture Mar 13 Semi-supervised Learning for NLP
Suggested Readings:
[Semi-Supervised Sequence Learning]
[Learned in Translation: Contextualized Word Vectors]
[Deep Contextualized Word Representations]
[Adversarial Training Methods for Semi-Supervised Text Classification]
Lecture Mar 15 Future of NLP Models, Multi-task Learning and QA Systems
Final Project Due Mar 18 Final project due
Poster Presentation Mar 21 Final project poster presentations
McCaw Hall at the Alumni Center
McCaw Hall at the Alumni Center
More Details