Note: In the 2023–24 academic year, CS224N will be taught in both Winter and Spring 2024.

Natural language processing (NLP) is a crucial part of artificial intelligence (AI), modeling how people share information. In recent years, deep learning approaches have obtained very high performance on many NLP tasks. In this course, students gain a thorough introduction to cutting-edge neural networks for NLP.

Instructors

Course Manager

John Cho

Logistics

Content

What is this course about?

Natural language processing (NLP) or computational linguistics is one of the most important technologies of the information age. Applications of NLP are everywhere because people communicate almost everything in language: web search, advertising, emails, customer service, language translation, virtual agents, medical reports, politics, etc. In the last decade, deep learning (or neural network) approaches have obtained very high performance across many different NLP tasks, using single end-to-end neural models that do not require traditional, task-specific feature engineering. In this course, students will gain a thorough introduction to cutting-edge research in Deep Learning for NLP. Through lectures, assignments and a final project, students will learn the necessary skills to design, implement, and understand their own neural network models, using the Pytorch framework.

“Take it. CS221 taught me algorithms. CS229 taught me math. CS224N taught me how to write machine learning models.” – A CS224N student on Carta

Previous offerings

Below you can find archived websites and student project reports from previous years. Disclaimer: assignments are subject to change - please do not assume that assignments will be unchanged from last year!

CS224n Websites: Winter 2023 / Winter 2022 / Winter 2021 / Winter 2020 / Winter 2019 / Winter 2018 / Winter 2017 / Autumn 2015 / Autumn 2014 / Autumn 2013 / Autumn 2012 / Autumn 2011 / Winter 2011 / Spring 2010 / Spring 2009 / Spring 2008 / Spring 2007 / Spring 2006 / Spring 2005 / Spring 2004 / Spring 2003 / Spring 2002 / Spring 2000
CS224n Lecture Videos: Winter 2021 / Winter 2019 / Winter 2017
CS224n Reports: Winter 2023 / Winter 2021 / Winter 2020 / Winter 2019 / Winter 2018 / Winter 2017 / Autumn 2015 and earlier
CS224d Reports: Spring 2016 / Spring 2015

Prerequisites

Reference Texts

The following texts are useful, but none are required. All of them can be read free online.

If you have no background in neural networks but would like to take the course anyway, you might well find one of these books helpful to give you more background:


Coursework

Assignments (54%)

There are five weekly assignments, which will improve both your theoretical understanding and your practical skills. All assignments contain both written questions and programming parts. In office hours, TAs may look at students’ code for assignments 1, 2 and 3 but not for assignments 4 and 5.

Final Project (43%)

The Final Project offers you the chance to apply your newly acquired skills towards an in-depth application. Students have two options: the Default Final Project (in which students tackle a predefined task, namely implementing a minimalist version of BERT) or a Custom Final Project (in which students choose their own project involving human language and deep learning). Examples of both can be seen on last year's website. Note: TAs may not look at students' code for either the default or custom final projects.

Important information

Practicalities

Participation (3%)

We appreciate everyone being actively involved in the class! There are several ways of earning participation credit, which is capped at 3%:

Late Days

Regrade Requests

If you feel you deserved a better grade on an assignment, you may submit a regrade request on Gradescope within 3 days after the grades are released. Your request should briefly summarize why you feel the original grade was unfair. Your TA will reevaluate your assignment as soon as possible, and then issue a decision. If you are still not happy, you can ask for your assignment to be regraded by an instructor.

Credit/No credit enrollment

If you take the class credit/no credit then you are graded in the same way as those registered for a letter grade. The only difference is that, providing you reach a C- standard in your work, it will simply be graded as CR.

All students welcome

We are committed to doing what we can to work for equity and to create an inclusive learning environment that actively values the diversity of backgrounds, identities, and experiences of everyone in CS224N. We also know that we will sometimes make missteps. If you notice some way that we could do better, we hope that you will let someone in the course staff know about it.

Well-Being and Mental Health

If you are experiencing personal, academic, or relationship problems and would like to talk to someone with training and experience, reach out to the Counseling and Psychological Services (CAPS) on campus. CAPS is the university’s counseling center dedicated to student mental health and wellbeing. Phone assessment appointments can be made at CAPS by calling 650-723-3785, or by accessing the VadenPatient portal through the Vaden website.

Auditing the course

In general we are happy to have auditors if they are a member of the Stanford community (registered student, official visitor, staff, or faculty). If you want to actually master the material of the class, we very strongly recommend that auditors do all the assignments. However, due to high enrollment, we cannot grade the work of any students who are not officially enrolled in the class.

Students with Documented Disabilities

We assume that all of us learn in different ways, and that the organization of the course must accommodate each student differently. We are committed to ensuring the full participation of all enrolled students in this class. If you need an academic accommodation based on a disability, you should initiate the request with the Office of Accessible Education (OAE). The OAE will evaluate the request, recommend accommodations, and prepare a letter for faculty. Students should contact the OAE as soon as possible and at any rate in advance of assignment deadlines, since timely notice is needed to coordinate accommodations. Students should also send your accommodation letter to either the staff mailing list (cs224n-win2324-staff@lists.stanford.edu) or make a private post on Ed, as soon as possible.

OAE accommodations for group projects: OAE accommodations will not be extended to collaborative assignments.

AI Tools Policy

Students are required to independently submit their solutions for CS224N homework assignments. Collaboration with generative AI tools such as Co-Pilot and ChatGPT is allowed, treating them as collaborators in the problem-solving process. However, the direct solicitation of answers or copying solutions, whether from peers or external sources, is strictly prohibited.

Employing AI tools to substantially complete assignments or exams will be considered a violation of the Honor Code. For additional details, please refer to the Generative AI Policy Guidance here.

Sexual violence

Academic accommodations are available for students who have experienced or are recovering from sexual violence. If you would like to talk to a confidential resource, you can schedule a meeting with the Confidential Support Team or call their 24/7 hotline at: 650-725-9955. Counseling and Psychological Services also offers confidential counseling services. Non-confidential resources include the Title IX Office, for investigation and accommodations, and the SARA Office, for healing programs. Students can also speak directly with the teaching staff to arrange accommodations. Note that university employees – including professors and TAs – are required to report what they know about incidents of sexual or relationship violence, stalking and sexual harassment to the Title IX Office. Students can learn more at https://vaden.stanford.edu/sexual-assault.


Schedule

Updated lecture slides will be posted here shortly before each lecture. (All CS 224N slides originally by Prof. Chris Manning, unless otherwise specified.) Other links contain last year's slides, which are mostly similar.

Lecture notes will be uploaded a few days after most lectures. The notes (which cover approximately the first half of the course content) give supplementary detail beyond the lectures.

Date Description Course Materials Events Deadlines
Week 1

Tue Jan 9
Word Vectors
[slides] [notes]
Suggested Readings:
  1. Efficient Estimation of Word Representations in Vector Space (original word2vec paper)
  2. Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)
Assignment 1 out
[code]
[preview]
Thu Jan 11 Word Vectors and Language Models
[slides] [notes]
Suggested Readings:
  1. GloVe: Global Vectors for Word Representation (original GloVe paper)
  2. Improving Distributional Similarity with Lessons Learned from Word Embeddings
  3. Evaluation methods for unsupervised word embeddings
Additional Readings:
  1. A Latent Variable Model Approach to PMI-based Word Embeddings
  2. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
  3. On the Dimensionality of Word Embedding
Fri Jan 12 Python Review Session
[slides] [colab]
4:30pm - 5:20pm
Skilling Auditorium
Week 2

Tue Jan 16
Backpropagation and Neural Network Basics
[slides] [notes]
Suggested Readings:
  1. matrix calculus notes
  2. Review of differential calculus
  3. CS231n notes on network architectures
  4. CS231n notes on backprop
  5. Derivatives, Backpropagation, and Vectorization
  6. Learning Representations by Backpropagating Errors (seminal Rumelhart et al. backpropagation paper)
Additional Readings:
  1. Yes you should understand backprop
  2. Natural Language Processing (Almost) from Scratch
Assignment 2 out
[code]
[handout]
[latex template]
Assignment 1 due
Thu Jan 18 Dependency Parsing
[slides] [notes]
Suggested Readings:
  1. Incrementality in Deterministic Dependency Parsing
  2. A Fast and Accurate Dependency Parser using Neural Networks
  3. Dependency Parsing
  4. Globally Normalized Transition-Based Neural Networks
  5. Universal Stanford Dependencies: A cross-linguistic typology
  6. Universal Dependencies website
  7. Jurafsky & Martin Chapter 18
Fri Jan 19 PyTorch Tutorial Session
[colab]
4:30pm - 5:20pm
Skilling Auditorium
Week 3

Tue Jan 23
Recurrent Neural Networks
[slides] [notes (lectures 5 and 6)]
Suggested Readings:
  1. N-gram Language Models (textbook chapter)
  2. The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview)
  3. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.1 and 10.2)
  4. On Chomsky and the Two Cultures of Statistical Learning
  5. Sequence Modeling: Recurrent and Recursive Neural Nets (Sections 10.3, 10.5, 10.7-10.12)
  6. Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
  7. On the difficulty of training Recurrent Neural Networks (proof of vanishing gradient problem)
  8. Vanishing Gradients Jupyter Notebook (demo for feedforward networks)
  9. Understanding LSTM Networks (blog post overview)
Assignment 3 out
[code]
[handout]
[latex template]
Assignment 2 due
Thu Jan 25 Sequence to Sequence Models and Machine Translation
[slides] [notes (lectures 5 and 6)]
Suggested Readings:
  1. Statistical Machine Translation slides, CS224n 2015 (lectures 2/3/4)
  2. Statistical Machine Translation (book by Philipp Koehn)
  3. BLEU (original paper)
  4. Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)
  5. Sequence Transduction with Recurrent Neural Networks (early seq2seq speech recognition paper)
  6. Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)
  7. Attention and Augmented Recurrent Neural Networks (blog post overview)
  8. Massive Exploration of Neural Machine Translation Architectures (practical advice for hyperparameter choices)
  9. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models
  10. Revisiting Character-Based Neural Machine Translation with Capacity and Compression
Week 4

Tue Jan 30
Final Projects
[slides]
Suggested Readings:
  1. Practical Methodology (Deep Learning book chapter)
Assignment 4 out
[code] [handout] [latex]
Assignment 3 due
Thu Feb 1 Transformers
[slides] [notes]
Suggested Readings:
  1. Attention Is All You Need
  2. The Illustrated Transformer
  3. Transformer (Google AI blog post)
  4. Layer Normalization
  5. Image Transformer
  6. Music Transformer: Generating music with long-term structure
  7. Jurafsky and Martin Chapter 10 (Transformers and Large Language Models)
Project Proposal out
[handout]

Default Final Project out
[handout]
Week 5

Tue Feb 6
Pretraining
[slides]
Suggested Readings:
  1. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  2. Contextual Word Representations: A Contextual Introduction
  3. The Illustrated BERT, ELMo, and co.
  4. Jurafsky and Martin Chapter 11 (Fine-Tuning and Masked Language Models)
Thu Feb 8 Post-training (RLHF, SFT)
[slides]
Suggested Readings:
  1. Aligning language models to follow instructions
  2. Scaling Instruction-Finetuned Language Models
  3. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
  4. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
  5. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Assignment 5 out
[code]
[handout]
[latex]
[colab run script]
Assignment 4 due
Fri Feb 9 Hugging Face Transformers Tutorial Session
[colab]
4:30pm - 5:20pm
Skilling Auditorium
Week 6

Tue Feb 13
Efficient Adaptation
[slides]
Suggested Readings:
  1. Language Models are Few-Shot Learners
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  3. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
  4. LoRA: Low-Rank Adaptation of Large Language Models
  5. Parameter-Efficient Transfer Learning for NLP
Project Proposal due
Thu Feb 15 Question Answering
[slides]
Suggested readings:
  1. SQuAD: 100,000+ Questions for Machine Comprehension of Text
  2. Dense Passage Retrieval for Open-Domain Question Answering
  3. Bidirectional Attention Flow for Machine Comprehension
  4. Reading Wikipedia to Answer Open-Domain Questions
  5. REALM: Retrieval-Augmented Language Model Pre-Training
  6. Lost in the Middle: How Language Models Use Long Contexts
Project Milestone out
[handout]
Sun Feb 18 Assignment 5 due
Week 7

Tue Feb 20
Security and Privacy (by Nicholas Carlini)
[slides]
Suggested readings:
  1. Extracting Training Data from Large Language Models
  2. Scalable Extraction of Training Data from (Production) Language Models
  3. Quantifying Memorization Across Neural Language Models
Final Project Proposals Returned
Thu Feb 22 Benchmarking and Evaluation
[slides]
Week 8

Tue Feb 27
Code Generation (by Pengcheng Yin)
[slides]
Suggested readings:
  1. Program Synthesis with Large Language Models
  2. Evaluating Large Language Models Trained on Code
  3. Code Llama: Open Foundation Models for Code
  4. Natural Language to Code Generation in Interactive Data Science Notebooks
  5. Teaching Large Language Models to Self-Debug
  6. Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
Thu Feb 29 Multimodal Language Models (by Armen Aghajanyan) Suggested readings:
  1. CM3: A Causal Masked Multimodal Model of the Internet
  2. Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
  3. Scaling Laws for Generative Mixed-Modal Language Models
  4. Small-scale proxies for large-scale Transformer training instabilities
Fri Mar 1
Course Withdrawal Deadline
Sat Mar 2
Final Project Milestone due
Week 9

Tue Mar 5
Human Centered NLP
[slides]
Suggested readings:
  1. Perspectives on Incorporating Expert Feedback into Model Updates
  2. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
  3. Multi-VALUE: A Framework for Cross-Dialectal English NLP
  4. Constitutional AI: Harmlessness from AI Feedback
  5. ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles
Final Project Report Instructions out
[Instructions]
Thu Mar 7 Deployment and Efficiency (by Tri Dao)
[slides]
Suggested readings:
  1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  2. Do we need Attention?
Final Project Milestones Returned
Week 10

Tue Mar 12
Open Questions in NLP 2024
[slides]
Thu Mar 14 Final Project Emergency Assistance (no lecture) Extra project office hours available during usual lecture time, see Ed. Final project due
Monday Mar 18 (7-10 PM) Final Project Poster Session Location: AOERC
On-campus students please attend in person!
[Printing guide]