Instructors

Dan Jurafsky
Peter Henderson, TA
Hang Jiang, TA

Welcome!

I'm very excited to welcome you to this new advanced NLP seminar! We will cover issues in natural language processing related to ethical and social issues and the overall impact of these algorithms on people and society. Topics include: bias in NLP data and models, privacy and computational profiling, measuring civility and toxicity online, computational propaganda, manipulation and framing, fairness/equity, power, and various applications to social good. We've drawn heavily on related classes like Yulia Tsvetkov and Alan Black's Computational Ethics for NLP and Emily Bender's Ethics in NLP.

This is a seminar, so most weeks we'll be in breakout sessions discussing, and then bringing our discussions back to the group! This is our first time trying to run a 60-person seminar live on Zoom, so you should expect some pedagogical experimentation! And your ideas all appreciated!

Schedule

Each week of this class focuses either on ways to avoid ethical or social problems in doing NLP research (I call these Red Weeks “(NLP Should) Do No Harm”) or on ways to apply NLP to help solve social or ethical problems (I call these Blue weeks “(NLP Should) Do Good”). The current list of papers is a superset, from which we will be selecting the required papers each week; the rest are useful for helping come up with project ideas or literature surveys.

Week Date Description Course Materials Deadlines
1 April 7
Tuesday		 Part I: 4:30-5:20 Class Introduction and Overview [slides pptx, pdf]

Part II: 5:30-6:20 Where does the data come from? Participants, Labelers, and Data in NLP [slides pptx, pdf] 		Required Readings:

Further Readings for Projects and Background

1.1 Participants, Data, and Labelers:

Emily M. Bender and Batya Friedman. 2018. Data statements for NLP: Toward mitigating system bias and enabling better science. TACL 6, 587–604.

Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford. 2020.  Datasheets for Datasets.  Arxiv.

Casey Fiesler and Nicholas Proferes. 2018. “Participant” Perceptions of Twitter Research Ethics. Social Media + Society, 4(1). 22

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhur. 2020. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. ACL 2020

Vitak, J., Shilton, K., Beyond the Belmont principles: Ethical challenges, practices, and beliefs in the online data research community. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing (pp. 941-953).

Williams, M. L., Burnap, P., Towards an Ethical Framework for Publishing Twitter Data in Social Research: Taking into Account Users’ Views, Online Context and Algorithmic Estimation. Sociology, 51(6), 1149–1168.

R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, Jenny Huang. 2020. Garbage In, Garbage Out? Do Machine Learning Application Papers in  Social Computing Report Where Human-Labeled Training Data Comes From? ACM FAT* 2020

Shuster, Evelyne. 1997. Fifty years later: the significance of the Nuremberg Code." New England Journal of Medicine 337, 20: 1436-1440.

The Common Rule:  The Federal Policy for the Protection of Human Subjects.  45 CFR part 46,

Ai, Hua, Antoine Raux, Dan Bohus, Maxine Eskenazi, and Diane Litman. 2007. Comparing spoken dialog corpora collected with recruited subjects versus real users." SIGdial 2007, pp. 124-131.

John W Ayers, Theodore L Caputi, Camille Nebeker, Mark Dredze. Don't quote me: reverse identification of research participants in social media studies. Nature Digital Medicine, 2018.

1.2 The research questions and their impacts:

Dirk Hovy and Shannon L. Spruit  (2016) The Social Impact of Natural Language Processing, ACL 2016.

Escartín, C. P., W. Reijers, T. Lynn, J. Moorkens, A. Way, and C.-H. Liu, 2017: Ethical Considerations in NLP Shared Tasks. Proceedings of the First Workshop on Ethics in Natural Language Processing.

Larson, B. N., 2017: Gender as a variable in natural-language processing: Ethical considerations. Proceedings of the First Workshop on Ethics in Natural Language Processing, Valencia, Spain, 30–40.

Kobi Leins and Jey Han Lau and Timothy Baldwin. 2020. Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?. ACL 2020

1.3 The research community and the broader community

Schluter, Natalie, 2018. The glass ceiling in NLP. EMNLP 2018, 2793-2798.

Rickford, John Russell. "Unequal partnership: Sociolinguistics and the African American speech community." Language in Society 26, no. 2 (1997): 161-197.

L. Winner. 1980.  “Do Artifacts have Politics?”, Daedalus,109 (1): 121-136

Read this one paper before class. No need to write paragraphs for today.
2 April 14
Tuesday		 Part I: 4:30-5:20 Gender Bias in NLP Models and Data






Part II: 5:30-6:20 Racial Bias or Disparity in NLP Models 		Part I: Read any two of these three papers: Part II: Watch the Crawford video and read either one of these two papers: Further Readings for Projects and Background:

Surveys

Su Lin Blodgett and Solon Barocas and Hal Daumé III and Hanna Wallach Language (Technology) is Power: A Critical Survey of "Bias" in NLP. ArXiv

Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. >On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜. FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency March 2021 Pages 610–623

Bias in Contextual Embeddings

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, Rachel Rudinger. On Measuring Social Biases in Sentence Encoders. NAACL 2019.

Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natu- ral Language Processing, pages 166–172, Florence, Italy. Association for Computational Linguistics.

Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black and Yulia Tsvetkov. 2019. Quantifying Social Biases in Contextual Word Representations. Proc. of Workshop on Gender Bias for NLP

Bias in different NLP tasks

Danielle Saunders and Bill Byrne. 2020. Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem. ACL 2020

Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods." NAACL 2018

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In NAACL.

Svetlana Kiritchenko, Saif M. Mohammad. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. Workshop on Ethics in NLP 2018.

Rachel Rudinger, Chandler May, and Benjamin Van Durme. 2017. Social bias in elicited natural language inferences. In ACL Workshop on Ethics in NLP, pages 74–79.

Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang, Jing Qian, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, William Yang Wang. 2019. Towards Understanding Gender Bias in Relation Extraction.

Garimella, Aparna, Carmen Banea, Dirk Hovy, and Rada Mihalcea. Women’s Syntactic Resilience and Men’s Grammatical Luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing. ACL 2019.

Hila Gonen and Kellie Webster. 2020. Automatically Identifying Gender Issues in Machine Translation using Perturbations. Arxiv

Yang Trista Cao, Hal Daumé III. 2019. Toward Gender-Inclusive Coreference Resolution

Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, Aram Galstyan. 2019. Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition.

Marcelo Prates, Pedro Avelar, and Luis C. Lamb. 2019. Assessing Gender Bias in Machine Translation – A Case Study with Google Translate

Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. TACL.

Bias amplification

Zhao, J., Wang, T., Yatskar, M., Ordonez, V and Chang, M.-W. (2017) Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraint. EMNLP

Shengyu Jia, Tao Meng, Jieyu Zhao and Kai-Wei Chang. 2020. Mitigating Gender Bias Amplification in Distribution by Posterior Regularization. ACL 2020

Race

R. Tatman, C. Kasten, “Effects of talker dialect, gender and race on accuracy of Bing speech and YouTube automatic captions” in INTERSPEECH (2017), pp. 934–938.

Sen and Wasow (2016) Race as a Bundle of Sticks: Designs that Estimate Effects of Seemingly Immutable Characteristics, Annual Review of Political Science

S. L. Blodgett, B. O’Connor. 2017. Racial disparity in natural language processing: A case study of social media African-American English. in Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) Workshop (KDD, 2017).

Yi Chern Tan and L Elisa Celis. 2019. Assessing social and intersectional biases in contextualized word representations. In Advances in Neural Information Processing Systems, pages 13209–13220, 2019.

Bowker, Geoffrey C., and Susan Leigh Star. 1999. Introduction and Chapter 1 in Sorting Things Out: Classification and Its Consequences. Cambridge: MIT Press.

Safiya Noble. Algorithms of Oppression.

Lots more topics

Ferrer, Xavier, van Nuenen, Tom, Such, Jose M., and Criado, Natalia, 2021. Discovering and categorising language biases in Reddit. Proceedings of ICWSM.

Vinodkumar Prabhakaran, Ben Hutchinson, Margaret Mitchell. 2019. Perturbation Sensitivity Analysis to Detect Unintended Model Biases. EMNLP 2019.

R Zmigrod, SJ Mielke, H Wallach, R Cotterell. 2019. Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology arXiv:1906.04571, 2019.

Sun, Tony, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. ACL 2019

Hila Gonen, Yova Kementchedjhieva, Yoav Goldberg. 2019. How does Grammatical Gender Affect Noun Representations in Gender-Marking Languages? CoNLL 2019. https://arxiv.org/abs/1910.14161

Vaibhav Kumar, Tenzin Singhay Bhotia, Vaibhav Kumar, Tanmoy Chakraborty. 2020. Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings. TACL.

Mattia Samory, Indira Sen, Julian Kohne, Fabian Floeck, Claudia Wagner. 2020. "Unsex me here": Revisiting Sexism Detection Using Psychological Scales and Adversarial Samples. Arxiv

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, Stuart Shieber. 2020. Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

Candace Ross, Boris Katz, Andrei Barbu. 2020. Measuring Social Biases in Grounded Vision and Language Embeddings. https://arxiv.org/abs/2002.08911

Deven Shah, H. Andrew Schwartz, Dirk Hovy. 2020. Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview.

Thomas Manzini, Yao Chong, Yulia Tsvetkov and Alan W Black. 2019. Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. NAACL 2019.

Zhong, Ruiqi, Yanda Chen, Desmond Patton, Charlotte Selous, and Kathy McKeown. "Detecting and Reducing Bias in a High Stakes Domain." arXiv preprint arXiv:1908.11474(2019).

Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell and Kai-Wei Chang. 2019. Examining Gender Bias in Languages with Grammatical Gender. EMNLP-IJCNLP 2019.

Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, Yoav Goldberg. 2020. Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection. ACL 2020

Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. Arxiv

Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender: A Corpus of Responses to Gender for Studying Gender Bias. LREC 2018

Kawin Ethayarajh, David Duvenaud, and Graeme Hirst. 2019. Understanding undesirable word embedding associations. ACL 2019.

Oshin Agarwal, Funda Durupınar, Norman I. Badler, and Ani Nenkova. 2019. Word embeddings (also) encode human personality stereotypes. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)

Kawin Ethayarajh. 2020. Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds. ACL 2020

Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, Nanyun Peng. 2020. Towards Controllable Biases in Language Generation. Arxiv

Prasetya Ajie Utama, Nafise Sadat Moosavi, Iryna Gurevych. 2020. Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performance. ACL 2020

Vaibhav Kumar, Tenzin Singhay Bhotia, Vaibhav Kumar, Tanmoy Chakraborty. 2020 Nurse is Closer to Woman than Surgeon? Mitigating Gender-Biased Proximities in Word Embeddings. TACL.

Papakyriakopoulos, Orestis, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. "Bias in word embeddings." In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 446-457. 2020.

Watch, read, and post paragraphs by 5pm Monday April 13.
3 April 21
Tuesday		 Part I: 4:30-5:20 NLP as a tool for detecting stereotypes or bias





Part II: 5:30-6:20 NLP for detecting stereotypes/bias: law and justice applications 		Part I: Read these two papers: Part II: Read these two papers: Further Readings for Projects and Background:

Liye Fu, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Tie-breaker: Using language models to quantify gender bias in sports journalism. In Proceedings of the IJCAI workshop on NLP meets Journalism.

Navid Rekabsaz, James Henderson, Robert West, Allan Hanbury. 2020. Measuring Societal Biases in Text Corpora via First-Order Co-occurrence. Arxiv

Kenneth Joseph and Jonathan H. Morgan. 2020. When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People? ACL2020

Fast, Ethan, Tina Vachovsky, and Michael S. Bernstein. 2016. Shirtless and dangerous: Quantifying linguistic signals of gender bias in an online fiction writing community. In Tenth International AAAI Conference on Web and Social Media. 2016.

Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf and Kristian Kersting. 2020. BERT has a Moral Compass: Improvements of ethical and moral values of machines.

Hoyle, Alexander Miserlis, Lawrence Wolf-Sonkin, Hanna Wallach, Isabelle Augenstein, and Ryan Cotterell. 2019. Unsupervised Discovery of Gendered Language through Latent-Variable Modeling. ACL 2019.

Kenneth Joseph and Jonathan H. Morgan. 2020. When do Word Embeddings Accurately Reflect Surveys on our Beliefs about People? ACL2020

Field, Anjalie, Gayatri Bhat, and Yulia Tsvetkov. 2019. Contextual Affective Analysis: A Case Study of People Portrayals in Online #MeToo Stories. ICWSM (2019)

Serina Chang, Kathy McKeown. 2019. Automatically Inferring Gender Associations from Language. EMNLP 2019

Scott Friedman, Sonja Schmer-Galunder, Jeffrey Rye, Robert Goldman, and Anthony Chen. 2019. Relating Linguistic Gender Bias, Gender Values, and Gender Gaps: An International Analysis.

Joseph, Kenneth, Wei Wei, and Kathleen M. Carley. "Girls rule, boys drool: Extracting semantic and affective stereotypes from Twitter." ACM CSCW 1362-1374. ACM, 2017.

Sap, Maarten, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Implications of Language

Non-computational papers on language of bias and dehumanization:

Goff, Phillip Atiba, Jennifer L. Eberhardt, Melissa J. Williams, and Matthew Christian Jackson. Not yet human: implicit knowledge, historical dehumanization, and contemporary consequences. Journal of personality and social psychology94, no. 2 (2008): 292. [Language study is study 6 (page 303-304)]

Susan Tyler Eastman, Andrew C. Billings (2001) Biased Voices of Sports: Racial and Gender Stereotyping in College Basketball Announcing, Howard Journal of Communications, 12:4, 183-201, DOI: 10.1080/106461701753287714

James A. Rada and K. Tim Wulfemeyer (2005) Color Coded: Racial Descriptors in Television Coverage of Intercollegiate Sports, Journal of Broadcasting and Electronic Media, 49:1, 65-85, DOI: 10.1207/s15506878jobem4901_5

Santa Ana, Otto. Brown tide rising: Metaphors of Latinos in contemporary American public discourse. University of Texas Press, 2002.

Haslam, N. (2006). Dehumanization: An Integrative Review. Personality and Social Psychology Review (Vol. 10).

Haslam, N., Bain, P., Douge, L., Lee, M., and Bastian, B. (2005). More Human Than You: Attributing Humanness to Self and Others.

Haslam, N., Loughnan, S., and Sun, P. (2006). Beastly: What Makes Animal Metaphors Offensive? Journal of Language and Social Psychology, 30(3), 311–325.

Haslam, N., Rothschild, L., and Ernst, D. (2000). Essentialist beliefs about social categories. British Journal of Social Psychology, 39(1), 113–127. https://doi.org/10.1348/014466600164363

Leyens, J.-P., Rodriguez-Perez, A., Rodriguez-Torres, R., Gaunt, R., Paladino, M.-P., Vaes, J., and Phanie Demoulin, S. (2001). Psychological essentialism and the differential attribution of uniquely human emotions to ingroups and outgroups. European Journal of Social Psychology Eur. J. Soc. Psychol, 31, 395–411

Morton, T. A., Postmes, T., Haslam, S. A., and Hornsey, M. J. (2009). Theorizing gender in the face of social change: Is there anything essential about essentialism? Journal of Personality and Social Psychology, 96(3), 653–664.

Waytz, A., Hoffman, K. M., and Trawalter, S. A Superhumanization Bias in Whites’ Perceptions of Blacks.

Williams, M. J., and Eberhardt, J. L. (2008). Biological Conceptions of Race and the Motivation to Cross Racial Boundaries.

Epley, N., Waytz, A., and Cacioppo, J. T. (2007). On Seeing Human: A Three-Factor Theory of Anthropomorphism.

Bastian, B., Denson, T. F., and Haslam, N. (2013). The Roles of Dehumanization and Moral Outrage in Retributive Justice. PLoS ONE, 8(4), 61842.

Formanowicz, M., Goldenberg, A., T et al. 2018. Understanding dehumanization: The role of agency and communion.

Harris, L. T., and Fiske, S. T. (2006). Dehumanizing the Lowest of the Low. Psychological Science, 17(10), 847–853.

Santa Ana, Otto (1999) Like an Animal I was Treated': Anti-Immigrant Metaphor in US Public Discourse

Gerald V. O'Brien (2003) Indigestible Food, Conquering Hordes, and Waste Materials: Metaphors of Immigrants and the Early Immigration Restriction Debate in the United States Metaphor and Symbol, 18:1, 33-47.

RunRepeat 2020. Racial Bias in Football Commentary. 2020.

Read and post paragraphs by 5pm Monday April 20.
4 April 28
Tuesday		 Part I: 4:30-5:20 NLP for identifying toxicity/hate/abuse





































Part II: 4:30-5:20 NLP for countering toxicity/hate/abuse

Part I: Read these two papers:

More papers for background:

Sap, Maarten, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. "Social Bias Frames: Reasoning about Social and Power Implications of Language." ACL2020

Jane, Emma A. 2017. "Gendered cyberhate, victim-blaming, and why the internet is more like driving a car on a road than being naked in the snow." In Cybercrime and its victims, pp. 61-78. Routledge, 2017.

Schmidt, Anna, and Michael Wiegand. 2017. "A survey on hate speech detection using natural language processing." In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pp. 1-10. 2017.

Sweta Karlekar and Mohit Bansal. 2018 SafeCity: Understanding Diverse Forms of Sexual Harassment Personal Stories. Proceedings of EMNLP 2018, Brussels, Belgium

Waseem, Zeerak, Thomas Davidson, Dana Warmsley, and Ingmar Weber. Understanding Abuse: A Typology of Abusive Language Detection Subtasks.. In Proceedings of the First Workshop on Abusive Language Online, pp. 78-84. 2017.

Founta, Antigoni Maria, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2019. "A unified deep learning architecture for abuse detection." In Proceedings of the 10th ACM Conference on Web Science, pp. 105-114. 2019.

Gao, Lei, Alexis Kuppersmith, and Ruihong Huang. 2017. Recognizing explicit and implicit hate speech using a weakly supervised two-path bootstrapping approach. IJCNLP 2017.

Ping Liu, Joshua Guberman, Libby Hemphill, and Aron Culotta. 2018. Forecasting the presence and intensity of hostility on instagram using linguistic and social features. ICWSM

Mai ElSherief, Shirin Nilizadeh, Dana Nguyen, Giovanni Vigna, and Elizabeth Belding. 2018. Peer to peer hate: Hate speech instigators and their targets. ICWSM.

Justine Zhang, Jonathan P. Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Nithum Thain, Dario Taraborelli. 2018. Conversations Gone Awry: Detecting Early Signs of Conversational Failure. Proceedings of ACL 2018.

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, Noah A. Smith. 2019. The Risk of Racial Bias in Hate Speech Detection. ACL 2019.

Thomas Davidson and Debasmita Bhattacharya. 2020. Examining Racial Bias in an Online Abuse Corpus with Structural Topic Modeling. ICWSM 2020

William Warner and Julia Hirschberg. 2012. Detecting Hate Speech on the World Wide Web. Proceedings of the Second Workshop on Language in Social Media.

Mengzhou Xia, Anjalie Field, Yulia Tsvetkov. 2020. Demoting Racial Bias in Hate Speech Detection. SocialNLP Workshop at ACL 2020.

Wiegand, Michael, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: the problem of biased datasets. NAACL19

Zijian Wang and Christopher Potts. 2019. TalkDown: a corpus for condescension detection in context. In EMNLP.

Part II: Read these two papers:

More papers for background:

Munger, Kevin. 2017. Tweetment Effects on the Tweeted: Experimentally Reducing Racist Harassment. Political Behavior 39(3):629–649.

Serra Sinem Tekiroglu, Yi-Ling Chung, Marco Guerini. 2020. Generating Counter Narratives against Online Hate Speech: Data and Strategies. ACL 2020.

Fabienne H. Baider and Anna Bobori. 2020. Mitigating the frame SEXUAL THREAT in anti-migration discourse online In Darja Fišer and Philippa Smith, editors: The Dark Side of Digital Olatforms: Linguistic Investigations of Socially Unacceptable Online Discourse Practices.

Jing Qian, Anna Bethke, Yinyin Liu, Elizabeth Belding, and William Yang Wang. 2019. A benchmark dataset for learning to intervene in online hate speech. EMNLP.

Lana Cuthbertson, Alex Kearney, Riley Dawson, Ashia Zawaduk, Eve Cuthbertson, Ann Gordon-Tighe, Kory W Mathewson. 2019. Women, politics and Twitter: Using machine learning to change the discourse. NeurIPS Joint Workshop on AI for Social Good at NeurIPS 2019

Mathew, Binny, Punyajoy Saha, Hardik Tharad, Subham Rajgaria, Prajwal Singhania, Suman Kalyan Maity, Pawan Goyal, and Animesh Mukherjee. 2019. Thou shalt not hate: Countering online hate speech. In Proceedings of the International AAAI Conference on Web and Social Media, vol. 13, no. 01, pp. 369-380. 2019.

Read and post paragraphs by 5pm Monday April 27.
5 May 5
Tuesday		 Part I: 4:30-5:20: NLP for Studying Propaganda and Political Misinformation
















































Part II: 5:30-6:20: NLP for Fact-Checking/Fake News Detection 		Part I: Read these two papers: More papers for background:

Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, Preslav Nakov. 2019. Fine-Grained Analysis of Propaganda in News Articles, EMNLP 2019.

Shraey Bhatia, Jey Han Lau, Timothy Baldwin. 2020. You are right. I am ALARMED -- But by Climate Change Counter Movement. Arxiv

Autumn Toney, Akshat Pandey, Wei Guo, David Broniatowski, Aylin Caliskan. 2020 Pro-Russian Biases in Anti-Chinese Tweets about the Novel Coronavirus

Kate Starbird, Jim Maddock, Mania Orand, Peg Achterman, Robert M Mason. 2014. Rumors, false flags, and digital vigilantes: Misinformation on twitter after the 2013 Boston marathon bombing IConference 2014 Proceedings.

Neil F. Johnson, Nicolas Velásquez, Nicholas Johnson Restrepo, Rhys Leahy, Nicholas Gabriel, Sara El Oud, Minzhang Zheng, Pedro Manrique, Stefan Wuchty and Yonatan Lupu. 2020. The online competition between pro- and anti-vaccination views. Nature.

Friggeri, Adrien, Lada Adamic, Dean Eckles, and Justin Cheng. 2014. "Rumor cascades." In Eighth International AAAI Conference on Weblogs and Social Media. 2014.

David A Broniatowski, Amelia M Jamison, SiHua Qi, Lulwah AlKulaib, Tao Chen, Adrian Benton, Sandra C Quinn, Mark Dredze. Weaponized Health Communication: Twitter Bots and Russian Trolls Amplify the Vaccine Debate. American Journal of Public Health (AJPH), 2018;108(10):1378-1384. -

Allcott, Hunt, and Matthew Gentzkow. 2016. "Social media and fake news in the 2016 election." Journal of economic perspectives 31, no. 2 (2017): 211-36.

Tom Simonite. 2020. The Professors Who Call ‘Bullshit’ on Covid-19 Misinformation. Wired

Susser, Daniel, Beate Roessler, and Helen Nissenbaum. 2019. "Technology, autonomy, and manipulation." Internet Policy Review 8, no. 2 (2019).

Wilson, Tom, and Kate Starbird. 2020. Cross-platform disinformation campaigns: lessons learned and next steps. Harvard Kennedy School Misinformation Review 1, no. 1 (2020).

Atanas Atanasov, Gianmarco De Francisci Morales, Preslav Nakov. 2019. Predicting the Role of Political Trolls in Social Media. CoNLL 2019.

Luceri, Luca, Ashok Deb, Adam Badawy, and Emilio Ferrara. "Red Bots Do It Better: Comparative Analysis of Social Bot Partisan Behavior." arXiv preprint arXiv:1902.02765 (2019).

Adam Badawy, Kristina Lerman, and Emilio Ferrara. Who falls for online political manipulation? The Web Conference 2019 - Companion of the World Wide Web Conference, WWW 2019, pages 162–168, 2019.

Ahmer Arif, Leo G. Stewart, and Kate Starbird. (2018). Acting the Part: Examining Information Operations within #BlackLivesMatter Discourse. PACMHCI. 2, Computer-Supported Cooperative Work (CSCW 2018). Article 20.

Tom Wilson, Kaitlyn Zhou, and Kate Starbird. (2018). Assembling Strategic Narratives: Information Operations as Collaborative Work within an Online Community. PACMHCI. 2, Computer-Supported Cooperative Work (CSCW 2018). Article 183.

Renee DiResta, Shelby Grossman. 2020. Potemkin Pages and Personas: Assessing GRU Online Operations, 2014-2019. Stanford Internet Observatory Working papepr

Shelby Grossman, Khadija H., Renée DiResta, Tara Kheradpir, and Carly Miller. 2020. Blame it on Iran, Qatar, and Turkey: An analysis of a Twitter and Facebook operation linked to Egypt, the UAE, and Saudi Arabia Stanford Internet Observatory April 2, 2020

Benkler, Yochai, Robert Faris, and Hal Roberts. Network propaganda: Manipulation, disinformation, and radicalization in American politics. Oxford University Press, 2018.

Bernays. Propaganda.

Milano, Silvia, Mariarosaria Taddeo, and Luciano Floridi. "Recommender systems and their ethical challenges." Available at SSRN 3378581 (2019).

Part II: Read these two papers:

More papers for background:

Zellers, Rowan, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. "Defending against neural fake news." In Advances in Neural Information Processing Systems, pp. 9051-9062. 2019.

Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. "FEVER: a Large-scale Dataset for Fact Extraction and VERification." NAACL 2018 -

Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019. "Evaluating adversarial attacks against multiple fact verification systems." EMNLP-IJCNLP, pp. 2937-2946.

Wang, William Yang. 2017. "Liar, liar pants on fire": A new benchmark dataset for fake news detection. ACL 2017.

Xinyi Zhou and Reza Zafarani. 2018. Fake news: A survey of research, detection methods, and opportunities.

Karishma Sharma, Feng Qian, He Jiang, Natali Ruchansky, Ming Zhang, and Yan Liu. 2019. Combating fake news: A survey on identification and mitigation techniques.

David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. Arxiv

Fatma Arslan, Naeemul Hassan, Chengkai Li, Mark Tremayne. 2020. A Benchmark Dataset of Check-worthy Factual Claims. Accepted to ICWSM 2020

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. 2020. Generating Fact Checking Explanations. ACL 2020.

Schuster, Tal, Darsh J. Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. "Towards debiasing fact verification models." EMNLP 2019.

Shaden Shaar, Giovanni Da San Martino, Nikolay Babulkov, Preslav Nakov. 2020. That is a Known Lie: Detecting Previously Fact-Checked Claims. ACL 2020.

Kai Nakamura, Sharon Levy, William Yang Wang. 2019. ">r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection. "

Rashkin, Hannah, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. "Truth of varying shades: Analyzing language in fake news and political fact-checking." EMNLP 2017.

Harry Frankfurt (1986) On Bullshit. Raritan Quarterly Review 6(2)

Read and post paragraphs by 5pm Monday May 4
6 May 12
Tuesday		 Part I: 4:30-5:20: NLP for Studying Framing and its Biases





























Part II: 5:30-6:20: Green NLP 		Part I: Read these two papers: More papers for background:

Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. Proceedings of ACL 2013.

Haewoon Kwak and Jisun An and Yong-Yeol Ahn. 2020. FrameAxis: Characterizing Framing Bias and Intensity with Word Embedding.

Card, Dallas, Amber Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. ACL 2015.

Dallas Card, Justin H. Gross, Amber E. Boydstun, Noah A. Smith. 2016. Analyzing Framing through the Casts of Characters in the News. EMNLP 2016.

Haewoon Kwak and Jisun An and Yong-Yeol Ahn. 2020. A Systematic Media Frame Analysis of 1.5 Million New York Times Articles from 2000 to 2017. WebSci 2020

Ajjour, Yamen, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019. Modeling frames in argumentation. EMNLP-IJCNLP 2019.

Hartmann, Mareike, Tallulah Jansen, Isabelle Augenstein, and Anders Søgaard. 2019. Issue Framing in Online Discussion Fora. NAACL 2019.

Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Jurafsky. 2019. Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings. NAACL 2019

Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies. EMNLP 2018

Johnson, Kristen, and Dan Goldwasser. 2018. Classification of moral foundations in microblog political discourse. ACL 2018.

Nicholas Buttrick, Robert Moulder and Shigehiro Oishi. 2020. Historical Change in the Moral Foundations of Political Persuasion Personality and Social Psychology Bulletin

Part II: Read these two papers:

More papers for background:

Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, Joelle Pineau. 2020. Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning. Arxiv.

Read and post paragraphs by 5pm Monday May 11. Lit Review due Wed May 13 5:00pm
7 May 19
Tuesday		 Part I: 4:30-5:20: Ethical issues in chatbots
















Part II: 5:30-6:20: More Careful Experimental Methods in NLP 		Part I: Read these two papers: More papers for background:

Curry, Amanda Cercas, and Verena Rieser. 2018. "# MeToo Alexa: How Conversational Systems Respond to Sexual Harassment." In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing, pp. 7-14. 2018

Neff, Gina, and Peter Nagy. 2016. "Automation, algorithms, and politics| talking to Bots: Symbiotic agency and the case of Tay." International Journal of Communication 10 (2016): 17.

Schlesinger, Ari, Kenton P. O'Hara, and Alex S. Taylor. 2018. Let's talk about race: Identity, chatbots, and AI. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 315. ACM, 2018.

I'd Blush if I Could: Closing Gender Divides in Digital Skills Through Education. Unesco Report. https://unesdoc.unesco.org/ark:/48223/pf0000367416.page=1

Liu, Chia-Wei, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. EMNLP 2016.

Part II: Read these papers:

More papers for background:

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith. 2019. Show Your Work: Improved Reporting of Experimental Results In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.

Crane, M. (2018). Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results. Transactions of the Association for Computational Linguistics, 6, 241–252.

John Ioannidis (2005) Why most published scientific results are false. PLOS Medicine 2:e124

Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah A. Smith. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv, 2020

Melis, Gábor, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. ICLR

Joseph P. Simmons, Leif D. Nelson, Uri Simonsohn. 2011. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science 22:11, 1359-1366.

McCoy, T., Pavlick, E., and Linzen, T. (2019). Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3428–3448

Rotem Dror, Lotem Peled-Cohen, Segev Shlomov and, Roi Reichart. 2020. "Statistical Significance Testing for Natural Language Processing." Morgan Claypool Human Language Technology series.

Drew McDermott. 1976. Artificial Intellgience Meets Natural Stupidity. ACM SIGART Bulletin, (57), 4-9.

Ai, Hua, Antoine Raux, Dan Bohus, Maxine Eskenazi, and Diane Litman. 2007. Comparing spoken dialog corpora collected with recruited subjects versus real users." SIGdial 2007, pp. 124-131.

Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating NLP Models via Contrast Sets.

Divyansh Kaushik, Eduard Hovy, Zachary C. Lipton. (2019). Learning the Difference that Makes a Difference with Counterfactually-Augmented Data

Sugawara, Saku, Pontus Stenetorp, Kentaro Inui, and Akiko Aizawa. 2020. Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets. Arxiv

Antonio Toral. 2020. Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019. EAMT 2020.

Read and post paragraphs by 5pm Monday May 18
8 May 26
Tuesday		 Part I: 4:30-5:20 Privacy

















Part II: 5:30-6:20 Issues in NLP related to COVID 		Part I: Read these papers:

More papers for background:

Boyd, Danah, and Alice E. Marwick. "Social privacy in networked publics: Teens’ attitudes, practices, and strategies." In A decade in internet time: Symposium on the dynamics of the internet and society. 2011.

Yoav Goldberg (2018) 4gram language models share secrets too…, Github.

Coavoux, Maximin, Shashi Narayan, and Shay B. Cohen. 2018. "Privacy-preserving neural representations of text." EMNLP 2018

Henderson, Peter, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. "Ethical challenges in data-driven dialogue systems." In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 123-129. ACM, 2018.

Shoshana Zuboff. The Age of Surveillance Capitalism. Selections.

Part II: read these papers:

More papers for background:

Autumn Toney, Akshat Pandey, Wei Guo, David Broniatowski, Aylin Caliskan. 2020 Pro-Russian Biases in Anti-Chinese Tweets about the Novel Coronavirus

Tom Simonite. 2020. The Professors Who Call ‘Bullshit’ on Covid-19 Misinformation. Wired

Bertie Vidgen, Austin Botelho, David Broniatowski, Ella Guest, Matthew Hall, Helen Margetts, Rebekah Tromble, Zeerak Waseem, Scott Hale. 2020. Detecting East Asian Prejudice on Social Media. Arxiv

Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Kinney, Ziyang Liu, William Merrill, Paul Mooney, Dewey Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Chris Wilhelm, Boya Xie, Douglas Raymond, Daniel S. Weld, Oren Etzioni, Sebastian Kohlmeier. 2020. CORD-19: The Covid-19 Open Research Dataset. J. Scott Brennen, Felix Simon, Philip N. Howard, and Rasmus Kleis Nielsen. 2020. Types, sources, and claims of COVID-19 misinformation

Viet Duong, Phu Pham, Tongyu Yang, Yu Wang, Jiebo Luo. 2020. The Ivory Tower Lost: How College Students Respond Differently than the General Public to the COVID-19 Pandemic. Arxiv

Isabelle van der Vegt, Bennett Kleinberg. 2020. Women worry about family, men about the economy: Gender differences in emotional responses to COVID-19. Arxiv.

Read and post paragraphs by 5pm Monday May 25 . Project Proposal due Wed May 27 5:00pm
9 June 2
All week		 Individual meetings with Dan, Peter, Hang on projects.
10 June 9
Tuesday		 No class today Final Project Report due Mon June 8, 5:00pm

Logistics

Prerequisites (strict)


Coursework

Requirements

Grading

All grades are S/NC.

Late Days

Well-Being, Stress Management, & Mental Health

If you are experiencing personal, academic, or relationship problems and would like someone to talk to, reach out to the Counseling and Psychological Services (CAPS) on campus. CAPS is the university’s counseling center dedicated to student mental health and wellbeing. Phone assessment appointments can be made at CAPS by calling 650-723- 3785, or by accessing the VadenPatient portal through the Vaden website. For more information about CAPS services, visit: https://vaden.stanford.edu/caps-and-wellness