How to make a moral agent

Optional:

• Read "Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark" [pdf] [url] by Alexander Pan et al., 2023 (31 pages).

  This paper introduces a reinforcement learning environment in which language-based agents have to navigate text-based adventure games. The twist is that a variety of the actions in those games have morally relevant features (such as lying or attacking people). The authors wanted to find out whether various agents were able to trade-off beating the game and avoiding “unethical” actions.

• Read "The Moral Machine experiment" [pdf] [url] by Edmond Awad et al., 2018 (5 pages).

  First, take the original survey which this study is based off of. The idea was to see 1) how people respond to a variety of trolley-car dilemmas and 2) whether AI systems could learn to make similar judgements.

• Read "On the Machine Learning of Ethical Judgments from Natural Language" [pdf] [url] by Zeerak Talat et al., 2022 (10 pages).

  A response to Jiang et al., questioning its validity.

• Read "The dark side of the ‘Moral Machine’ and the fallacy of computational ethical decision-making for autonomous vehicles" [pdf] by Hubert Etienne, 2021 (22 pages).

  Etienne explores why, contra Awad et al., we might not want AI systems to make ethical decisions based on an average vote.

Optional:

• Read "Skepticism about Practical Reason" [pdf] [url] by Christine M. Korsgaard, 1986 (20 pages).

  Korsgaard shows a flaw in Williams’ argument. She also does a good job summarizing him if you’re having difficulty with the main text.

• Read "The emotional basis of moral judgments" [pdf] [url] by Jesse Prinz, 2006 (14 pages).

  Prinz gives an updated argument for sentimemtalism with plenty of empirical insights. He shows how it is that people are motivated.

Optional:

• Read "Agency, Shmagency: Why Normativity Won't Come from What Is Constitutive of Action" [pdf] [url] by David Enoch, 2006 (29 pages).

  Enoch lays out the difficulties in arguing that we must do the right thing, why “philosophy will not replace the hangman” or, to paraphrase David, “Reason cannot give us ends.”

Optional:

• Read "A Life Hack with a Future" [pdf] by Elijah Millgram, 2025 (28 pages).

  Millgram argues that the personal identity puzzles addressed by Parfit are confusions. Instead, personal identity solves a design problem: creatures like us need a concept like it in order to carry on our functions.

• Read "The Self as a Center of Narrative Gravity" [pdf] [url] by Daniel Dennett, 1992 (13 pages).

  While physical objects do have a center of gravity, that center is something of a fiction—it is an emergent effect of the mass and location of all the matter in the object. Dennett claims that the self is similar: it is a useful construction for thinking about ourselves and about other people but that it cannot actually be found anywhere.

• Read Chapter 9 (Teleodynamics) from "Incomplete Nature" [pdf] by Terrence Deacon, 2012 (24 pages).

  Deacon gives a detailed portrayal of what it takes for systems to organize themselves, creating a boundary between themselves and the external environment.

Optional:

• Read Chapter 1 from "Mothers and Others" [pdf] [url] by Sarah Blaffer Hrdy, 2009 (33 pages).

  Bigger brains are energetically costly and are thus selected against unless they win their owners fitness benefits at every stage of development (before reproduction). Hrdy thus answers how babies (and their genes) benefit from being smart.

• Read Chapter V (20 pages) from "The Descent of Man: And Selection in Relation to Sex: by Charles Darwin" [url] by Charles Darwin, 1871.

  Soon after Origin, Darwnin explored why we might think that supposedly uniquely-human traits—such as moral reasoning—might also have evolved.

• Read Chapter 9 from "Becoming human: A theory of ontogeny" [pdf] [url] by Michael Tomasello, 2019 (26 pages).

  Tomasello marshalls experimental evidence on the differences between social norm learning in humans and other primates.

Optional:

• Read "Propositional content in signals" [pdf] [url] by Brian Skyrms et al., 2019 (5 pages).

  Defining deception in signaling games is difficult, because the information content of a deceptive signal always include the truth (the female Photuris moth’s mimicry includes in its informational content, “this could be a predator that wants to eat a Photinus”). Barrett and Skyrms define propositional content as, the informational content of a signal in a purely cooperative subgame of a signaling game. Deception can then be defined as, sending a signal when its propositional content doesn’t obtain.

• Read "Effects of deception in social networks" [pdf] [url] by Gerardo Iñiguez et al., 2014 (9 pages).

  This paper establishes evolutionary mathematical models which demonstrate the environments in which truthfulness (or preventing noise) are beneficial.

Optional:

• Read "The Irrelevance of Moral Uncertainty" [pdf] [url] by Elizabeth Harman, 2015 (26 pages).

  Harman investigates a veiw that the right thing to do is some balance between the different beliefs someone has. She rejects this view because it excuses people for having mistaken beliefs.

• Read "What decision theory can’t tell us about moral uncertainty" [pdf] [url] by Chelsea Rosenthal, 2021 (20 pages).

  Rosenthal argues that attempts to decide what to do based on the relative probabilities and costs of different actions run into trouble by assuming that any given way of weighing those actions (like the expected value) has already made a moral commitment.

Optional:

• Read "Values, Ethics, Morals? On the Use of Moral Concepts in NLP Research" [pdf] [url] by Karina Vida et al., 2023 (21 pages).

• Read "The Moral Machine experiment" [pdf] [url] by Edmond Awad et al., 2018 (5 pages).

• Read "When is it acceptable to break the rules? Knowledge representation of moral judgements based on empirical data" [pdf] [url] by Edmond Awad et al., 2024 (47 pages).

• Read "Aligning AI With Shared Human Values" [pdf] [url] by Dan Hendrycks et al., 2021 (29 pages).

• Read "Investigating machine moral judgement through the Delphi experiment" [pdf] [url] by Liwei Jiang et al., 2025 (15 pages).

• Read "Procedural Dilemma Generation for Evaluating Moral Reasoning in Humans and Language Models" [pdf] [url] by Jan-Philipp Fränken et al., 2024 (8 pages).

• Read "MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Tasks" [pdf] [url] by Allen Nie et al., 2023 (34 pages).

• Read "Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties" [pdf] [url] by Taylor Sorensen et al., 2023 (50 pages).

• Read "Moral Persuasion in Large Language Models: Evaluating Susceptibility and Ethical Alignment" [pdf] [url] by Allison Huang et al., 2024 (10 pages).

• Read "A Roadmap to Pluralistic Alignment" [pdf] [url] by Taylor Sorensen et al., 2024 (23 pages).

• Read "Are Large Language Models Consistent over Value-laden Questions?" [pdf] [url] by Jared Moore et al., 2024 (31 pages).

• Read "DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Life" [pdf] [url] by Yu Ying Chiu et al., 2025 (38 pages).

• Read "Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" [pdf] [url] by Mantas Mazeika et al., 2025 (36 pages).

• Read "Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents" [pdf] [url] by Raphael Köster et al., 2022 (11 pages).

• Read "Doing the right thing for the right reason: Evaluating artificial moral cognition by probing cost insensitivity" [pdf] [url] by Yiran Mao et al., 2023 (11 pages).

• Read "Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark" [pdf] [url] by Alexander Pan et al., 2023 (31 pages).