Logistics

Content

What is this course about?

Natural language processing (NLP) is one of the most important technologies of the information age, and a crucial part of artificial intelligence. Applications of NLP are everywhere because people communicate almost everything in language: web search, advertising, emails, customer service, language translation, medical reports, etc. In recent years, Deep Learning approaches have obtained very high performance across many different NLP tasks, using single end-to-end neural models that do not require traditional, task-specific feature engineering. In this course, students will gain a thorough introduction to cutting-edge research in Deep Learning for NLP. Through lectures, assignments and a final project, students will learn the necessary skills to design, implement, and understand their own neural network models. This year, CS224n will be taught for the first time using PyTorch rather than TensorFlow (as in previous years).

Previous offerings

This course was formed in 2017 as a merger of the earlier CS224n (Natural Language Processing) and CS224d (Natural Language Processing with Deep Learning) courses. Below you can find archived websites and student project reports.

CS224n Websites: Winter 2018 / Winter 2017 / Autumn 2015 / Autumn 2014 / Autumn 2013 / Autumn 2012 / Autumn 2011 / Winter 2011 / Spring 2010 / Spring 2009 / Spring 2008 / Spring 2007 / Spring 2006 / Spring 2005 / Spring 2004 / Spring 2003 / Spring 2002 / Spring 2000
CS224n Lecture Videos: Winter 2017
CS224n Reports: Winter 2018 / Winter 2017 / Autumn 2015 and earlier
CS224d Reports: Spring 2016 / Spring 2015

Prerequisites

Reference Texts

The following texts are useful, but not required. All of them can be read free online.

If you have no background in neural networks but would like to take the course anyway, you might well find one of these books helpful to give you more background:


Coursework

Assignments (54%)

There are five weekly assignments, which will improve both your theoretical understanding and your practical skills. All assignments contain both written questions and programming parts.

Final Project (43%)

The Final Project offers you the chance to apply your newly acquired skills towards an in-depth application. Students have two options: the Default Final Project (in which students tackle a predefined task, namely textual Question Answering) or a Custom Final Project (in which students choose their own project). Examples of both can be seen on last year's website.

Important information

Practicalities

Participation (3%)

We appreciate everyone being actively involved in the class! There are several ways of earning participation credit, which is capped at 3%:

Late Days

Regrade Requests

If you feel you deserved a better grade on an assignment, you may submit a regrade request on Gradescope within 3 days after the grades are released. Your request should briefly summarize why you feel the original grade was unfair. Your TA will reevaluate your assignment as soon as possible, and then issue a decision. If you are still not happy, you can ask for your assignment to be regraded by an instructor.

Credit/No credit enrollment

If you take the class credit/no credit then you are graded in the same way as those registered for a letter grade. The only difference is that, providing you reach a C- standard in your work, it will simply be graded as CR.


Schedule

Lecture slides will be posted here shortly before each lecture. If you wish to view slides further in advance, refer to last year's slides, which are mostly similar.

The lecture notes are updated versions of the CS224n 2017 lecture notes (viewable here) and will be uploaded a few days after each lecture. The notes (which cover approximately the first half of the course content) give supplementary detail beyond the lectures.

This schedule is subject to change.

Date Description Course Materials Events Deadlines
Tue Jan 8 Introduction and Word Vectors
[slides] [notes]
Gensim word vectors example:
[zip] [preview]
Suggested Readings:
  1. Word2Vec Tutorial - The Skip-Gram Model
  2. Efficient Estimation of Word Representations in Vector Space (original word2vec paper)
  3. Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)
Assignment 1 out
[zip] [preview]
Thu Jan 10 Word Vectors 2 and Word Senses
[slides] [notes]
Suggested Readings:
  1. GloVe: Global Vectors for Word Representation (original GloVe paper)
  2. Improving Distributional Similarity with Lessons Learned from Word Embeddings
  3. Evaluation methods for unsupervised word embeddings
Additional Readings:
  1. A Latent Variable Model Approach to PMI-based Word Embeddings
  2. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
  3. On the Dimensionality of Word Embedding.
Fri Jan 11 Python review session
[slides]
1:30 - 2:50pm
Skilling Auditorium [map]
Tue Jan 15 Word Window Classification, Neural Networks, and Matrix Calculus
[slides] [matrix calculus notes]
[notes (lectures 3 and 4)]
Suggested Readings:
  1. CS231n notes on backprop
  2. Review of differential calculus
Additional Readings:
  1. Natural Language Processing (Almost) from Scratch
Assignment 2 out
[zip] [handout]
Assignment 1 due
Thu Jan 17 Backpropagation and Computation Graphs
[slides]
[notes (lectures 3 and 4)]
Suggested Readings:
  1. CS231n notes on network architectures
  2. Learning Representations by Backpropagating Errors
  3. Derivatives, Backpropagation, and Vectorization
  4. Yes you should understand backprop
Tue Jan 22 Linguistic Structure: Dependency Parsing
[slides]
[scrawled-on slides]
[notes]
Suggested Readings:
  1. Incrementality in Deterministic Dependency Parsing
  2. A Fast and Accurate Dependency Parser using Neural Networks
  3. Dependency Parsing
  4. Globally Normalized Transition-Based Neural Networks
  5. Universal Stanford Dependencies: A cross-linguistic typology
  6. Universal Dependencies website
Assignment 3 out
[zip] [handout]
Assignment 2 due
Thu Jan 24 The probability of a sentence? Recurrent Neural Networks and Language Models
[slides]
[notes (lectures 6 and 7)]
Suggested Readings:
  1. N-gram Language Models (textbook chapter)
  2. The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview)
  3. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.1 and 10.2)
  4. On Chomsky and the Two Cultures of Statistical Learning
Tue Jan 29 Vanishing Gradients and Fancy RNNs
[slides] [notes (lectures 6 and 7)]
Suggested Readings:
  1. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.3, 10.5, 10.7-10.12)
  2. Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
  3. On the difficulty of training Recurrent Neural Networks (proof of vanishing gradient problem)
  4. Vanishing Gradients Jupyter Notebook (demo for feedforward networks)
  5. Understanding LSTM Networks (blog post overview)
Assignment 4 out
[zip] [handout] [Azure Guide] [Practical Guide to VMs]
Assignment 3 due
Thu Jan 31 Machine Translation, Seq2Seq and Attention
[slides] [notes]
Suggested Readings:
  1. Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)
  2. Statistical Machine Translation (book by Philipp Koehn)
  3. BLEU (original paper)
  4. Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)
  5. Sequence Transduction with Recurrent Neural Networks (early seq2seq speech recognition paper)
  6. Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)
  7. Attention and Augmented Recurrent Neural Networks (blog post overview)
  8. Massive Exploration of Neural Machine Translation Architectures (practical advice for hyperparameter choices)
Tue Feb 5 Practical Tips for Final Projects
[slides][notes]
Suggested Readings:
  1. Practical Methodology (Deep Learning book chapter)
Thu Feb 7 Question Answering and the Default Final Project
[slides][notes]
Project Proposal out
[instructions]

Default Final Project out [handout] [github repo]
Assignment 4 due
Tue Feb 12 ConvNets for NLP
[slides][notes]
Suggested Readings:
  1. Convolutional Neural Networks for Sentence Classification
  2. A Convolutional Neural Network for Modelling Sentences
Thu Feb 14 Information from parts of words: Subword Models
[slides]
Assignment 5 out
[zip (requires Stanford login)] [handout]
Project Proposal due
Tue Feb 19 Modeling contexts of use: Contextual Representations and Pretraining
[slides]
Suggested readings:
  1. Smith, Noah A. Contextual Word Representations: A Contextual Introduction. (Published just in time for this lecture!)
Thu Feb 21 Transformers and Self-Attention For Generative Models
(guest lecture by Ashish Vaswani and Anna Huang)
[slides]
Suggested readings:
  1. Attention is all you need
  2. Image Transformer
  3. Music Transformer: Generating music with long-term structure
Fri Feb 22 Project Milestone out
[instructions]
Assignment 5 due
Tue Feb 26 Natural Language Generation
[slides]
Thu Feb 28 Reference in Language and Coreference Resolution
[slides]
Tue Mar 5 Multitask Learning: A general model for NLP? (guest lecture by Richard Socher)
[slides]
Project Milestone due
Thu Mar 7 Constituency Parsing and Tree Recursive Neural Networks
[slides][notes]
Suggested Readings:
  1. Parsing with Compositional Vector Grammars.
  2. Constituency Parsing with a Self-Attentive Encoder
Tue Mar 12 Safety, Bias, and Fairness (guest lecture by Margaret Mitchell)
[slides]
Thu Mar 14 Future of NLP + Deep Learning
[slides]
Sun Mar 17 Final Project Report due [instructions]
Wed Mar 20 Final project poster session
[details]
5:15 - 8:30pm
McCaw Hall at the Alumni Center [map]
Project Poster/Video due [instructions]