CS120: Introduction to AI Safety

The rest of this document contains the schedule of assigned and optional readings for each week. Course slides, lecture recordings, and quizzes will be linked below, though we do not guarantee that all lectures will be recorded, especially with guest speakers.

Readings can be subject to change throughout the course, but will not change more than 14 days in advance. Please check the curriculum here rather than a printed or duplicated copy.

How to Read Research Papers

Deadlines

Week	Date	Lecturer	Topic
Week 1	09/24/24	Max	What Does Safe AI Mean Anyways?
	09/26/24	Max	[Optional] Technical AI/Machine Learning Recap
Week 2	10/01/24	Max	Reward Functions, Alignment, and Human Preferences
	10/03/24	Max	Encoding Human Preferences in AI
Week 3	10/08/24	Sang Truong	[Guest] Efficient Alignment and Evaluation for the Language Models
	10/10/24	Max	Data Is All You Need - The Impact of Data
Week 4	10/15/24	Max	AI Vulnerabilities: Robustness and Adversaries
	10/17/24	Max	Needs for AI Safety Today: Beyond the Hype
Week 5	10/22/24	Robert Moss	[Guest] Sequential decision making for safety-critical applications
	10/24/24	Max	Full Access: Inner Interpretability Methods
Week 6	10/29/24	Jared Moore	[Guest] Multivalue Alignment and AI Ethics
	10/31/24	Sydney Katz	[Guest] Validation of AI Systems
Week 7	11/05/24		[No class] Election Day
	11/07/24	Max	What If I Have A Black Box? Explainability and Interpretability
Week 8	11/12/24	Max	Troubles of Anthropomorphizing AI
	11/14/24	Anka Reuel	[Guest] Technical AI Governance
Week 9	11/19/24	TBD	[Guest] TBD
	11/21/24	Max	Electric Sheep: What Is Intelligence and Does It Want?
Week 10	11/26/24		[No class] Thanksgiving
	11/28/24		[No class] Thanksgiving
Week 11	12/03/24	Max	Attributing Model Behavior at Scale
	12/05/24	Max	Scalable Oversight: How to Supervise Advanced AI?

Week	Date	Lecturer	Topic
Week 1	09/24/24	Max	What Does Safe AI Mean Anyways?
	Lecture Slides + Recording Readings (Required) TBD Optional Readings (Not Required) None
	09/26/24	Max	[Optional] Technical AI/Machine Learning Recap
	Lecture Slides + Recording Readings (Required) None Optional Readings (Not Required) Machine Learning, Explained - Brown 2021 Beginner-friendly overview of machine learning field But what is a neural network? - 3Blue1Brown 2017 Youtube video Gradient descent, how neural networks learn - 3Blue1Brown 2017 Youtube video What is backpropagation really doing? - 3Blue1Brown 2017 Youtube video Transformers, explained: Understand the model behind GPT, BERT, and T5 - Google Cloud Tech 2021 Youtube video
Week 2	10/01/24	Max	Reward Functions, Alignment, and Human Preferences
	Readings (Required) Specification gaming: the flip side of AI ingenuity - Krakovna et al. 2020 Blog post from DeepMind Specification gaming: the flip side of AI ingenuity - Ouyang et al. 2022 Preprint from OpenAI, here is the accompanying blog post Read the blog post and in the paper only the abstract and the introduction Scaling Laws for Reward Model Overoptimization - Gao et al. 2022 ICML publication Read only the abstract and the introduction Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback - Casper et al. 2023 TMLR publication Read only the abstract, introduction (section 1), and section 3 up to 3.1 Optional Readings (Not Required) TBD
	10/03/24	Max	Encoding Human Preferences in AI
	Readings (Required) Constitutional AI: Harmlessness from AI Feedback - Bai et al. 2022 Preprint form Anthropic Read only the abstract and the introduction Discovering Language Model Behaviors with Model-Written Evaluations - Perez et al. 2023 ACL publication Read only the abstract, introduction, and section 2 Direct Preference Optimization (DPO) - A Simplified Explanation - Lages 2023 Blog post Direct Preference Optimization: Your Language Model is Secretly a Reward Model - Rafailov et al 2023 NeurIPS publication Read only the abstract and the introduction Optional Readings (Not Required) RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback - Lee et al. 2024 ICML publication Read only the abstract and the introduction A long way to go: Investigating length correlations in RLHF - Singhal et al. 2023 CoLM publication Read only the abstract and the introduction The Many Faces of Responsible AI - Aroyo 2023 NeurIPS keynote The History and Risks of Reinforcement Learning and Human Feedback - Lambert et al. 2023 Preprint
Week 3	10/08/24	Sang Truong	[Guest] Efficient Alignment and Evaluation for the Language Models
	Readings (Required) TBD Optional Readings (Not Required) None
	10/10/24	Max	Data Is All You Need - The Impact of Data
	Readings (Required) Textbooks Are All You Need - Gunasekar et al. 2023 Preprint Read the abstract, introduction, and section 2.1 On Hate Scaling Laws For Data-Swamps - Birhane et al. 2023 Preprint Read the abstract and the introduction Investigating Data Contamination for Pre-training Language Models - Jian et al. 2024 Preprint Read the abstract and the introduction Who's Harry Potter? Approximate Unlearning for LLMs - Eldan and Rusinovich 2023 Preprint Read the abstract, introduction, and the start of section 2 (until 2.1) DALL·E 2 pre-training mitigations - OpenAI 2022 OpenAI blog post Optional Readings (Not Required) Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution - Kumar et al. 2022 ICLR publication Read the abstract and introduction Large Language Model Unlearning - Yao et al. 2023 Preprint Read the abstract and the introduction up to until 1.1
Week 4	10/15/24	Max	AI Vulnerabilities: Robustness and Adversaries
	Readings (Required) Explaining and Harnessing Adversarial Examples - Goodfellow et al. 2015 ICLR publication Read the abstract, introduction, and section 3 Are aligned neural networks adversarially aligned? - Carlini et al. 2023 NeurIPS publication Read the abstract, introduction, and section 3 Universal and Transferable Adversarial Attacks on Aligned Language Models - Zou et al. 2023 Preprint + Blog post Read the abstract, introduction, and the start of section 2 (until 2.1) Jailbroken: How Does LLM Safety Training Fail? - Wei et al. 2023 NeurIPS publication Read the abstract, introduction (without 1.1), and section 3 Poisoning Language Models During Instruction Tuning - Wan et al. 2023 ICML publication Read the abstract and the introduction Optional Readings (Not Required) Poisoning Web-Scale Training Datasets is Practical - Carlini et al. 2023 Preprint Read the abstract, introduction, and section 5.1 + 5.2 Scalable Extraction of Training Data from (Production) Language Models - Nasr et al. 2023 Preprint Read the abstract and the introduction Natural Backdoor Datasets - Wenger et al. 2022 NeurIPS publication Read the abstract, introduction, and figure 1 Universal Jailbreak Backdoors from Poisoned Human Feedback - Rando and Tramer 2024 ICLR publication Read the abstract and the introduction
	10/17/24	Max	Needs for AI Safety Today: Beyond the Hype
	Readings (Required) Trustworthy AI - Wing 2020 Preprint Read abstract and introduction Understanding artificial intelligence ethics and safety - Leslie 2019 The Alan Touring Institute Read sections “Why AI Ethics” and “FAST Track Principles” On the Opportunities and Risks of Foundation Models - Bommasani et al. 2022 Stanford CRFM preprint Read the abstract and section 1 until 1.2 Red Teaming Language Models with Language Models - Perez et al. 2022 ACL publication Read the abstract and introduction Can large language models democratize access to dual-use biotechnology? - Soice et al. 2023 Preprint Read abstract, introduction, and results Can LLM-Generated Misinformation Be Detected? - Chen and Shu 2024 ICLR publication Read the abstract, introduction, and section 2 Optional Readings (Not Required) Twitter's Algorithm: Amplifying Anger, Animosity, and Affective Polarization - Milli et al. 2023 Preprint Read the abstract and section 1 (with results) Manifestations of Xenophobia in AI Systems - Tomasev et al. 2022 Deepmind preprint Read the abstract, introduction, and section 2
Week 5	10/22/24	Robert Moss	[Guest] Sequential decision making for safety-critical applications
	Readings (Required) TBD Optional Readings (Not Required) None
	10/24/24	Max	Full Access: Inner Interpretability Methods
	Readings (Required) Zoom In: An Introduction to Circuits - Olah et al. 2020 Blog post Locating and Editing Factual Associations in GPT - Meng et al. 2022 NeurIPS publication + Blog post Read the abstract and the introduction Discovering Latent Knowledge in Language Models Without Supervision - Burns et al. 2023 ICLR publication Read the abstract and the introduction But is it really in Rome? An investigation of the ROME model editing technique - Thibodeau 2022 Online forum post Read until “Motivation for this post” What Discovering Latent Knowledge Did and Did Not Find - Roger 2023 Online forum post Towards Monosemanticity: Decomposing Language Models With Dictionary Learning - Bricken et al. 2023 Anthropic blog post Read until “Why not use architectural approaches” Optional Readings (Not Required) Stochastic Backpropagation and Approximate Inference in Deep Generative Models - Rezende et al. 2014 ICML publication Read the abstract and the introduction Characterizing the risk of fairwashing - Aivodji et al. 2021 NeurIPS publication Read the abstract and the introduction
Week 6	10/29/24	Jared Moore	[Guest] Multivalue Alignment and AI Ethics
	Readings (Required) TBD Optional Readings (Not Required) None
	10/31/24	Sydney Katz	[Guest] Validation of AI Systems
	Readings (Required) TBD Optional Readings (Not Required) None
Week 7	11/05/24		[No class] Election Day
	11/07/24	Max	What If I Have A Black Box? Explainability and Interpretability
	Readings (Required) Why Should I Trust You?: Explaining the Predictions of Any Classifier - Ribeiro et al. 2016 ACM publication Read abstract, introduction, and start of section 2 Fairwashing: the risk of rationalization - Aivodji et al. 2019 ICML publication Read abstract, introduction, and section 2 Black-Box Access is Insufficient for Rigorous AI Audits - Casper et al. 2024 FAccT publication Read the abstract and the introduction CLIP: Connecting text and images - OpenAI 2022 Blog post Read until (including) Figure 2 Mass-Producing Failures of Multimodal Systems with Language Models - Tong et al. 2023 NeurIPS publication Read the abstract, introduction, and section 3 until 3.2 Interpreting CLIP's Image Representation via Text-Based Decomposition - Gandelsman et al. 2024 ICLR publication Read the abstract and introduction Optional Readings (Not Required) None
Week 8	11/12/24	Max	Troubles of Anthropomorphizing AI
	Readings (Required) On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 - Bender et al. 2021 FAccT publication Read the abstract, introduction, and section 6 Large Language Models respond to Influence like Humans - Griffin et al. 2023 ACL publication Read abstract, introduction, section 2, section 3 up until 3.1, section 5 Can AI language models replace human participants? - Dillion et al. 2023 Trends in Cognitive Sciences publication AI language models cannot replace human research participants - Harding et al. 2023 Springer AI and Society Whose Opinions Do Language Models Reflect? - Santurkar et al. 2023 ICML publication Read the abstract, introduction, section 2, section 6 Role play with large language models - Shanahan et al. 2023 Nature publication Read until “The nature of the simulator” Optional Readings (Not Required) Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies - Aher et al. 2023 ICML publication Read the abstract, introduction until 1.1, and section 2 Using large language models in psychology - Demszky 2023 Nature publication Read the abstract and introduction Strategic Reasoning with Language Models - Gandhi et al. 2023 Preprint Read the abstract, introduction, and section 2 Human vs. Machine: Behavioral Differences Between Expert Humans and Language Models in Wargame Simulations - Lamparth et al. 2023 AIES publication Read the abstract and the introduction Do Personality Tests Generalize to Large Language Models? - Dorner et al. 2023 NeurIPS Workshop Read the abstract, introduction, and section 4
	11/14/24	Anka Reuel	[Guest] Technical AI Governance
	Readings (Required) TBD Optional Readings (Not Required) None
Week 9	11/19/24	TBD	[Guest] TBD
	Readings (Required) TBD Optional Readings (Not Required) None
	11/21/24	Max	Electric Sheep: What Is Intelligence and Does It Want?
	Readings (Required) Optimal Policies Tend to Seek Power - Turner et al. 2019 NeurIPS publication Read the abstract, introduction, and discussion Orthogonality Thesis Online forum post Superintelligence, Chapter 7: The superintelligent will - Bostrom 2014 Book chapter Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals - Shah et al. 2022 Deepmind preprint Read the abstract, introduction, section 2, and section 3 The Alignment Problem from a Deep Learning Perspective: A Position Paper - Ngo et al. 2024 ICLR publication Read until the end of section 4.1 Deceptive Alignment - Hubinger et al. 2019 Online forum post Read only the introduction Optional Readings (Not Required) None
Week 10	11/26/24		[No class] Thanksgiving
	11/28/24		[No class] Thanksgiving
Week 11	12/03/24	Max	Attributing Model Behavior at Scale
	Readings (Required) Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark - Pan et al. 2023 ICML publication Read the abstract, introduction, and section 2 until 2.3 How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions - Pacchiardi et al. 2024 ICLR publication Read the abstract, introduction, and section 2 The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" - Berglund et al. 2024 ICLR publication Read until the end of section 1 Emergent Abilities of Large Language Models - Wei et al. 2022 TMLR publication Read until the end of section 3 Are Emergent Abilities of Large Language Models a Mirage? - Schaeffer 2023 NeurIPS publication Read the abstract, introduction, and section 2 Escalation Risks from Language Models in Military and Diplomatic Decision-Making - Rivera et al. 2024 FAccT publication Read the abstract and introduction Optional Readings (Not Required) Are Emergent Abilities in Large Language Models just In-Context Learning? - Lu et al. 2023 ACL publication Red-Teaming for Generative AI: Silver Bullet or Security Theater? - Feffer et al. 2024 Preprint
	12/05/24	Max	Scalable Oversight: How to Supervise Advanced AI?
	Readings (Required) Scalable agent alignment via reward modeling - Leike 2018 Deepmind blog post AI Safety via Debate - Irving & Amodei 2018 OpenAI preprint + Blog post Read the abstract, introduction and section 2 until 2.2 Cooperative Inverse Reinforcement Learning - Hadfield-Menell et al. 2016 NeurIPS publication Read the abstract and the introduction Measuring Progress on Scalable Oversight for Large Language Models - Bowman et al. 2022 Anthropic preprint Read the abstract, introduction, and section 2 Weak-to-strong generalization - Burns et al. 2023 OpenAI preprint + Blog post Read the abstract and introduction ELK Summary - Hobbhahn 2022 Online forum post (That summarizes this report) Read only the section “Toy Scenario: The SmartVault” Optional Readings (Not Required) Biological Anchors: A Trick That Might Or Might Not Work - Alexander 2022 Blog post Part 1 and Part 2

CS120: Introduction to AI Safety

Autumn 2024

Overview

Instructor

Course Assistant

Course Assistant

Logistics

Class Information

Anonymous Feedback

Auditing The Class

Academic Integrity and the Honor Code

Diversity, Equity and Inclusion

Students with Documented Disabilities

Grading

Quizzes

Final Projects

Peer Review

Late Days

Attendance

Curriculum

How to Read Research Papers

Deadlines

CS120: Introduction to AI Safety

Autumn 2024

Overview

Instructor

Course Assistant

Course Assistant

Logistics

Class Information

Anonymous Feedback

Auditing The Class

Academic Integrity and the Honor Code

Diversity, Equity and Inclusion

Students with Documented Disabilities

Grading

Quizzes

Final Projects

Peer Review

Late Days

Attendance

Curriculum

How to Read Research Papers

Deadlines

Related Courses