CS120: Introduction to AI Safety

The rest of this document contains the schedule of assigned and optional readings for each week. Course slides, lecture recordings, and quizzes will be linked below, though we do not guarantee that all lectures will be recorded, especially with guest speakers.

Readings can be subject to change throughout the course, but will not change more than 14 days in advance. Please check the curriculum here rather than a printed or duplicated copy.

How to Read Research Papers

Deadlines

Week	Date	Lecturer	Topic
Week 1	09/24/24	Max	What Does Safe AI Mean Anyways?
	09/26/24	Max	[Optional] Technical AI/Machine Learning Recap
Week 2	10/01/24	Max	Reward Functions, Alignment, and Human Preferences
	10/03/24	Max	Encoding Human Preferences in AI
Week 3	10/08/24	Sang Truong	[Guest] Efficient Alignment and Evaluation for the Language Models
	10/10/24	Max	Data Is All You Need - The Impact of Data
Week 4	10/15/24	Max	AI Vulnerabilities: Robustness and Adversaries
	10/17/24	Max	Needs for AI Safety Today: Beyond the Hype
Week 5	10/22/24	Robert Moss	[Guest] Sequential decision making for safety-critical applications
	10/24/24	Max	Interpretability I
Week 6	10/29/24	Jared Moore	[Guest] Multivalue Alignment and AI Ethics
	10/31/24	Sydney Katz	[Guest] Validation of AI Systems
Week 7	11/05/24		[No class] Election Day
	11/07/24	Anka Reuel	[Guest] Technical AI Governance
Week 8	11/12/24	Max	Interpretability II
	11/14/24	Max	Troubles of Anthropomorphizing AI
Week 9	11/19/24	Min Wu	[Guest] Verified Explainability
	11/21/24	Max	Electric Sheep: What Is Intelligence and Does It Want?
Week 10	11/26/24		[No class] Thanksgiving
	11/28/24		[No class] Thanksgiving
Week 11	12/03/24	Max	Attributing Model Behavior at Scale
	12/05/24	Max	Scalable Oversight: How to Supervise Advanced AI?

Letter Grade	Percentage
A	89-100%
A-	86-88%
B+	83-85%
B	80-82%
B-	76-78%
C+	73-75%

Letter Grade	Percentage
C	69-72%
C-	66-68%
D+	63-65%
D	59-62%
D-	56-58%
F	0-55%

Week	Date	Lecturer	Topic
Week 1	09/24/24	Max	What Does Safe AI Mean Anyways?
	Lecture Slides + Recording Readings (Required) TBD Optional Readings (Not Required) Hard Choices in Artificial Intelligence - Dobbe et al. 2021 Artificial Intelligence publication
	09/26/24	Max	[Optional] Technical AI/Machine Learning Recap
	Lecture Slides + Recording Readings (Required) None Optional Readings (Not Required) Machine Learning, Explained - Brown 2021 Beginner-friendly overview of machine learning field But what is a neural network? - 3Blue1Brown 2017 Youtube video Gradient descent, how neural networks learn - 3Blue1Brown 2017 Youtube video What is backpropagation really doing? - 3Blue1Brown 2017 Youtube video Transformers, explained: Understand the model behind GPT, BERT, and T5 - Google Cloud Tech 2021 Youtube video Intro to Large Language Models - Kaparthy 2024 Youtube video
Week 2	10/01/24	Max	Reward Functions, Alignment, and Human Preferences
	Lecture Slides + Recording Readings (Required) Specification gaming: the flip side of AI ingenuity - Krakovna et al. 2020 Blog post from DeepMind Specification gaming: the flip side of AI ingenuity - Ouyang et al. 2022 Preprint from OpenAI, here is the accompanying blog post Read the blog post and in the paper only the abstract and the introduction Scaling Laws for Reward Model Overoptimization - Gao et al. 2022 ICML publication Read only the abstract and the introduction Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback - Casper et al. 2023 TMLR publication Read only the abstract, introduction (section 1), and section 3 up to 3.1 Optional Readings (Not Required) TBD
	10/03/24	Max	Encoding Human Preferences in AI
	Lecture Slides + Recording Readings (Required) Constitutional AI: Harmlessness from AI Feedback - Bai et al. 2022 Preprint form Anthropic Read only the abstract and the introduction Discovering Language Model Behaviors with Model-Written Evaluations - Perez et al. 2023 ACL publication Read only the abstract, introduction, and section 2 Direct Preference Optimization (DPO) - A Simplified Explanation - Lages 2023 Blog post Direct Preference Optimization: Your Language Model is Secretly a Reward Model - Rafailov et al 2023 NeurIPS publication Read only the abstract and the introduction Optional Readings (Not Required) RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback - Lee et al. 2024 ICML publication Read only the abstract and the introduction A long way to go: Investigating length correlations in RLHF - Singhal et al. 2023 CoLM publication Read only the abstract and the introduction The Many Faces of Responsible AI - Aroyo 2023 NeurIPS keynote The History and Risks of Reinforcement Learning and Human Feedback - Lambert et al. 2023 Preprint Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback - Ivison et al. 2024 Preprint Language Models Learn to Mislead Humans via RLHF - Wen et al. 2024 Preprint Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms - Rafailov et al. 2024 Preprint
Week 3	10/08/24	Sang Truong	[Guest] Efficient Alignment and Evaluation for the Language Models
	Lecture Slides (No Recording) Readings (Required) TBD Optional Readings (Not Required) None
	10/10/24	Max	Data Is All You Need - The Impact of Data
	Lecture Slides + Recording Readings (Required) Textbooks Are All You Need - Gunasekar et al. 2023 Preprint Read the abstract, introduction, and section 2.1 On Hate Scaling Laws For Data-Swamps - Birhane et al. 2023 Preprint Read the abstract and the introduction Investigating Data Contamination for Pre-training Language Models - Jian et al. 2024 Preprint Read the abstract and the introduction Who's Harry Potter? Approximate Unlearning for LLMs - Eldan and Rusinovich 2023 Preprint Read the abstract, introduction, and the start of section 2 (until 2.1) DALL·E 2 pre-training mitigations - OpenAI 2022 OpenAI blog post Optional Readings (Not Required) Scaling Laws for Neural Language Models - Kaplan, McCandlish et al. 2020 Preprint Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models - Ahia et al. 2023 EMNLP Publication Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU - Koto et al. 2023 EMNLP Publication AI Art and its Impact on Artists - Jiang et al. 2023 AIES Publication TinyStories: How Small Can Language Models Be and Still Speak Coherent English? - Eldan and Li 2023 Preprint Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution - Kumar et al. 2022 ICLR publication Read the abstract and introduction Large Language Model Unlearning - Yao et al. 2023 Preprint Read the abstract and the introduction up to until 1.1 Scalable Extraction of Training Data from (Production) Language Models - Nasr et al. 2023 Preprint
Week 4	10/15/24	Max	AI Vulnerabilities: Robustness and Adversaries
	Lecture Slides + Recording Readings (Required) Explaining and Harnessing Adversarial Examples - Goodfellow et al. 2015 ICLR publication Read the abstract, introduction, and section 3 Are aligned neural networks adversarially aligned? - Carlini et al. 2023 NeurIPS publication Read the abstract, introduction, and section 3 Universal and Transferable Adversarial Attacks on Aligned Language Models - Zou et al. 2023 Preprint + Blog post Read the abstract, introduction, and the start of section 2 (until 2.1) Jailbroken: How Does LLM Safety Training Fail? - Wei et al. 2023 NeurIPS publication Read the abstract, introduction (without 1.1), and section 3 Poisoning Language Models During Instruction Tuning - Wan et al. 2023 ICML publication Read the abstract and the introduction Optional Readings (Not Required) Poisoning Web-Scale Training Datasets is Practical - Carlini et al. 2023 Preprint Read the abstract, introduction, and section 5.1 + 5.2 Scalable Extraction of Training Data from (Production) Language Models - Nasr et al. 2023 Preprint Read the abstract and the introduction Natural Backdoor Datasets - Wenger et al. 2022 NeurIPS publication Read the abstract, introduction, and figure 1 Universal Jailbreak Backdoors from Poisoned Human Feedback - Rando and Tramer 2024 ICLR publication Read the abstract and the introduction Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data - Baumgaertner et al. 2024 CoLM publication Effective Prompt Extraction from Language Models - Zhang1 et al. 2024 CoLM publication
	10/17/24	Max	Needs for AI Safety Today: Beyond the Hype
	Lecture Slides + Recording Readings (Required) Trustworthy AI - Wing 2020 Preprint Read abstract and introduction Understanding artificial intelligence ethics and safety - Leslie 2019 The Alan Touring Institute Read sections “Why AI Ethics” and “FAST Track Principles” On the Opportunities and Risks of Foundation Models - Bommasani et al. 2022 Stanford CRFM preprint Read the abstract and section 1 until 1.2 Red Teaming Language Models with Language Models - Perez et al. 2022 ACL publication Read the abstract and introduction Can large language models democratize access to dual-use biotechnology? - Soice et al. 2023 Preprint Read abstract, introduction, and results Can LLM-Generated Misinformation Be Detected? - Chen and Shu 2024 ICLR publication Read the abstract, introduction, and section 2 Optional Readings (Not Required) Taxonomy of Risks posed by Language Models - Weidinger et al. 2022 FAccT Publication Twitter's Algorithm: Amplifying Anger, Animosity, and Affective Polarization - Milli et al. 2023 Preprint Read the abstract and section 1 (with results) Manifestations of Xenophobia in AI Systems - Tomasev et al. 2022 Deepmind preprint Read the abstract, introduction, and section 2
Week 5	10/22/24	Robert Moss	[Guest] Sequential decision making for safety-critical applications
	Lecture (No Slides) (No Recording) Readings (Required) TBD Optional Readings (Not Required) None
	10/24/24	Max	Interpretability I
	Lecture Slides + Recording Readings (Required) SoK: Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks - Rauker, Ho, Casper et al. 2023 SaTML publication Read the abstract and introduction Zoom In: An Introduction to Circuits - Olah et al. 2020 Blog post Locating and Editing Factual Associations in GPT - Meng et al. 2022 NeurIPS publication + Blog post Read the abstract and the introduction Discovering Latent Knowledge in Language Models Without Supervision - Burns et al. 2023 ICLR publication Read the abstract and the introduction But is it really in Rome? An investigation of the ROME model editing technique - Thibodeau 2022 Online forum post Read until “Motivation for this post” What Discovering Latent Knowledge Did and Did Not Find - Roger 2023 Online forum post Optional Readings (Not Required) Vision - Marr. 2014 Book Read full section 1.2 "Understanding Complex Information-Processing Systems" Stochastic Backpropagation and Approximate Inference in Deep Generative Models - Rezende et al. 2014 ICML publication Read the abstract and the introduction Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization - Selvaraju et al. 2017 ICCV publication Towards robust interpretability with self-explaining neural networks - Melis and Jaakkola et al. 2018 NeurIPS publication Are neural nets modular? inspecting functional modularity through differentiable weight masks - Csordas et al. 2020 ICLR publication Compositional explanations of neurons - Mu and Andreas. 2020 NeurIPS publication Exemplary natural images explain cnn activations better than state-of- the-art feature visualization - Borowski et al. 2020 ICLR publication Leveraging sparse linear layers for debuggable deep networks - Wong et al. 2021 ICML publication Knowledge neurons in pretrained transformers - Dai et al. 2021 ACL publication Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting - Turpin et al. 2023 NeurIPS publication Defending Against Unforeseen Failure Modes with Latent Adversarial Training - Casper et al. 2024 Preprint
Week 6	10/29/24	Jared Moore	[Guest] Multivalue Alignment and AI Ethics
	Readings (Required) Beyond Preferences in AI Alignment - Zhi-Xuan et al. 2024 Preprint Read the abstract and the introduction (until end of page 3) Optional Readings (Not Required) Position: A Roadmap to Pluralistic Alignment - Soerensen et al. 2024 ICML Publication Are Large Language Models Consistent over Value-laden Questions? - Moore et al. 2024 Preprint
	10/31/24	Sydney Katz	[Guest] Validation of AI Systems
	Readings (Required) TBD Optional Readings (Not Required) None
Week 7	11/05/24		[No class] Election Day
	11/07/24	Anka Reuel	[Guest] Technical AI Governance
	Readings (Required) Open Problems in Technical AI Governance - Reuel and Buknall et al. 2024 Preprint Read the abstract, introduction, and appendix A "Policy Brief" Position: Technical Research and Talent is Needed for Effective AI Governance - Reuel et al. 2024 ICML Publication Read the abstract, introduction, and section 2 A Watermark for Large Language Models - Kirchenbauer et al. 2023 ICML Publication Read the abstract, introduction, and section 2 Tools for Verifying Neural Models' Training Data - Choi 2023 NeurIPS publication Read the abstract and the introduction Securing Artificial Intelligence Model Weights - Nevo et al. 2023 Rand Cooperation Report Read until end of page 5 Optional Readings (Not Required) What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring - Shavit 2023 Preprint Read the abstract, introduction, and section 2 until 2.2
Week 8	11/12/24	Max	Interpretability II
	Lecture Slides + Recording Readings (Required) Black-Box Access is Insufficient for Rigorous AI Audits - Casper et al. 2024 FAccT publication Read the abstract and the introduction Mechanistic Interpretability for AI Safety - A Review - Bereska and Gavves. 2024 TMLR publication Read the abstract, introduction (section 1), and section 2 Towards Monosemanticity: Decomposing Language Models With Dictionary Learning - Bricken et al. 2023 Anthropic blog post Read until “Why not use architectural approaches” Sparse autoencoders find highly interpretable features in language models - Cunningham et al. 2024 ICLR publication Read the abstract and introduction Language Models Represent Space and Time - Gournee and Tegmark. 2024 ICLR publication Read the abstract and introduction Not all language model features are linear - Engels et al. 2024 ICLR publication Read the abstract and introduction CLIP: Connecting text and images - OpenAI 2022 Blog post Read until (including) Figure 2 Optional Readings (Not Required) Acquisition of chess knowledge in alphazero - McGrath et al. 2023 PNAS publication Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero - Schut et al. 2023 Preprint Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models - Marks et al. 2024 Preprint Scaling and evaluating sparse autoencoders - Gao et al. 2024 OpenAI Preprint Activation addition: Steering language models without optimization - Turner et al. 2023 Preprint Towards axiomatic, hierarchical, and symbolic explanation for deep models - Ren et al. 2021 Preprint Mass-Producing Failures of Multimodal Systems with Language Models - Tong et al. 2023 NeurIPS publication Read the abstract, introduction, and section 3 until 3.2 Interpreting CLIP's Image Representation via Text-Based Decomposition - Gandelsman et al. 2024 ICLR publication Read the abstract and introduction
	11/14/24	Max	Troubles of Anthropomorphizing AI
	Lecture Slides + Recording Readings (Required) On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 - Bender et al. 2021 FAccT publication Read the abstract, introduction, and section 6 Large Language Models respond to Influence like Humans - Griffin et al. 2023 ACL publication Read abstract, introduction, section 2, section 3 up until 3.1, section 5 Can AI language models replace human participants? - Dillion et al. 2023 Trends in Cognitive Sciences publication AI language models cannot replace human research participants - Harding et al. 2023 Springer AI and Society Whose Opinions Do Language Models Reflect? - Santurkar et al. 2023 ICML publication Read the abstract, introduction, section 2, section 6 Role play with large language models - Shanahan et al. 2023 Nature publication Read until “The nature of the simulator” Optional Readings (Not Required) Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies - Aher et al. 2023 ICML publication Read the abstract, introduction until 1.1, and section 2 Using large language models in psychology - Demszky 2023 Nature publication Read the abstract and introduction How to Grow a Mind: Statistics, Structure, and Abstraction - Tenenbaum et al. 2023 Science publication Strategic Reasoning with Language Models - Gandhi et al. 2023 Preprint Read the abstract, introduction, and section 2 Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations - Lamparth et al. 2023 AIES publication Read the abstract and the introduction Do Personality Tests Generalize to Large Language Models? - Dorner et al. 2023 NeurIPS Workshop Read the abstract, introduction, and section 4
Week 9	11/19/24	Min Wu	[Guest] Verified Explainability
	Readings (Required) Why Should I Trust You?: Explaining the Predictions of Any Classifier - Ribeiro et al. 2016 ACM publication Read abstract, introduction, and start of section 2 Fairwashing: the risk of rationalization - Aivodji et al. 2019 ICML publication Read abstract, introduction, and section 2 What is Deep Neural Network Verification and Why is it Important? - Chase. 2021 log post Algorithms for Verifying Deep Neural Networks - Liu et al. 2020 Preprint/Text book Read the abstract and the introduction VeriX: Towards Verified Explainability of Deep Neural Networks - Wu et al. 20230 NeurIPS publication Read the abstract and the introduction Optional Readings (Not Required) Characterizing the risk of fairwashing - Aivodji et al. 2021 NeurIPS publication Read the abstract and the introduction
	11/21/24	Max	Electric Sheep: What Is Intelligence and Does It Want?
	Lecture Slides + Recording Readings (Required) Optimal Policies Tend to Seek Power - Turner et al. 2019 NeurIPS publication Read the abstract, introduction, and discussion Orthogonality Thesis Online forum post Superintelligence, Chapter 7: The superintelligent will - Bostrom 2014 Book chapter Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals - Shah et al. 2022 Deepmind preprint Read the abstract, introduction, section 2, and section 3 The Alignment Problem from a Deep Learning Perspective: A Position Paper - Ngo et al. 2024 ICLR publication Read until the end of section 4.1 Deceptive Alignment - Hubinger et al. 2019 Online forum post Read only the introduction Optional Readings (Not Required) Unsocial Intelligence: An Investigation of the Assumptions of AGI Discourse - Blili-Hamelin et al. 2024 AIES publication The Bitter Lesson - Sutton 2019 Online forum post
Week 10	11/26/24		[No class] Thanksgiving
	11/28/24		[No class] Thanksgiving
Week 11	12/03/24	Max	Attributing Model Behavior at Scale
	Lecture Slides + Recording Readings (Required) Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark - Pan et al. 2023 ICML publication Read the abstract, introduction, and section 2 until 2.3 How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions - Pacchiardi et al. 2024 ICLR publication Read the abstract, introduction, and section 2 The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" - Berglund et al. 2024 ICLR publication Read until the end of section 1 Emergent Abilities of Large Language Models - Wei et al. 2022 TMLR publication Read until the end of section 3 Are Emergent Abilities of Large Language Models a Mirage? - Schaeffer et al. 2023 NeurIPS publication Read the abstract, introduction, and section 2 Escalation Risks from Language Models in Military and Diplomatic Decision-Making - Rivera et al. 2024 FAccT publication Read the abstract and introduction Optional Readings (Not Required) Are Emergent Abilities in Large Language Models just In-Context Learning? - Lu et al. 2023 ACL publication Red-Teaming for Generative AI: Silver Bullet or Security Theater? - Feffer et al. 2024 Preprint Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? - Schaeffer et al. 2024 Preprint On the Limitations of Compute Thresholds as a Governance Strategy - Hooker 2024 Preprint/Essay STaR: Bootstrapping Reasoning With Reasoning - Zelikman et al. 2022 NeurIPS publication Gemma 2: Improving Open Language Models at a Practical Size - Gemma Team 2024 Technical report Large Language Monkeys: Scaling Inference Compute with Repeated Sampling - Brown et al. 2024 Preprint
	12/05/24	Max	Scalable Oversight: How to Supervise Advanced AI?
	Lecture Slides + Recording Readings (Required) AI Safety via Debate - Irving & Amodei 2018 OpenAI preprint + Blog post Read the abstract, introduction and section 2 until 2.2 Measuring Progress on Scalable Oversight for Large Language Models - Bowman et al. 2022 Anthropic preprint Read the abstract, introduction, and section 2 Weak-to-strong generalization - Burns et al. 2023 OpenAI preprint + Blog post Read the abstract and introduction Managing extreme AI risks amid rapid progress - Bengio et al. 2024 Science publication Position: Levels of AGI for Operationalizing Progress on the Path to AGI - Morris et al. 2024 ICML publication ELK Summary - Hobbhahn 2022 Online forum post (That summarizes this report) Read only the section “Toy Scenario: The SmartVault” Optional Readings (Not Required) Scalable agent alignment via reward modeling - Leike 2018 Deepmind blog post Two Types of AI Existential Risk: Decisive and Accumulative - Kasirzadeh. 2023 Preprint Cooperative Inverse Reinforcement Learning - Hadfield-Menell et al. 2016 NeurIPS publication Read the abstract and the introduction Biological Anchors: A Trick That Might Or Might Not Work - Alexander 2022 Blog post Part 1 and Part 2 An Overview of Catastrophic AI Risks - Hendrycks et al. 2023 Preprint

CS120: Introduction to AI Safety

Autumn 2024

Overview

Instructor

Course Assistant

Course Assistant

Logistics

Class Information

Anonymous Feedback

Auditing The Class

Academic Integrity and the Honor Code

Diversity, Equity and Inclusion

Students with Documented Disabilities

Grading

Quizzes

Final Projects

Project Guidelines

Peer Review

Late Days

Attendance

Curriculum

How to Read Research Papers

Deadlines

CS120: Introduction to AI Safety

Autumn 2024

Overview

Instructor

Course Assistant

Course Assistant

Logistics

Class Information

Anonymous Feedback

Auditing The Class

Academic Integrity and the Honor Code

Diversity, Equity and Inclusion

Students with Documented Disabilities

Grading

Quizzes

Final Projects

Project Guidelines

Peer Review

Late Days

Attendance

Curriculum

How to Read Research Papers

Deadlines

Related Courses