What is a moral patient?

If how we treat a being matters morally, that being is a moral patient.

To determine which beings are moral patients, we need to know what about a being makes it matter how we treat it.

Sentience as a basis for moral patiency

The limit of sentience … is the only defensible boundary of concern for the interests of others. To mark this boundary by some other characteristic like intelligence or rationality would be to mark it in an arbitrary manner. Why not choose some other characteristic, like skin color? (Singer 1975)

Strictly speaking, it is not exactly sentience that Singer means. It is “the capacity to suffer and / or experience enjoyment” – i.e., not only to have experiences but to have positive or negative experiences.

Can they suffer?

It may one day come to be recognized that the number of legs, the villosity of the skin [furriness], or the termination of the os sacrum [having a tail] are reasons equally insufficient for abandoning a sensitive being to [an unpleasant] fate. What else is it that should [determine whether a being is a moral patient]? … The question is not, Can they reason? nor Can they talk? but, Can they suffer? (Bentham 1789)

• Why might being able to have positive and negative experiences be sufficient for moral status?
• Why might it be necessary?

Moral agency and moral patiency are not the same concept

• At least some people think there can be moral patients that are not moral agents.
• For example, it might be wrong for me to bite a rabbit, but not wrong for a rabbit to bite me.
• But the concepts might be related.

A Kantian view: moral agency and moral patiency are coextensive

• You can read Kant as saying that all moral agents are moral patients and vice versa.
  • To morally wrong someone is to disrespect their agency as a free rational being.
  • Accordingly, to morally wrong someone, they must be a free rational being (which is what Kant thinks a moral agent is).
• This argument has the perhaps surprising consequence that it’s not morally wrong to torture animals.
• Kant probably didn’t really think that.
• Kant could argue that it is not only our rational nature that deserves respect, but our animal nature.
• On this view, animals could be moral patients, since we owe respect to their animal nature, even though we don’t owe them the respect we owe to those with a rational nature.

Is sentience (and thus moral patiency) necessary for moral agency?

• The following claims about psychopaths are often made together:
  1. They do not experience sympathetic pleasure or pain at others’ pleasure or pain.
  2. They are capable of articulating and following moral rules, but they regard them as no different from conventional rules, like rules of etiquette.
• Some researchers think 1 explains 2.
  • That is, what makes something a moral rule is that we feel that following it will help (or avoid hurt to) others.
  • Because psychopaths don’t have those sympathetic feelings, no rule is really a moral rule for them.
  • Even if they follow a moral rule, they don’t have the right kind of motivation for doing so.
  • Thus, they are not moral agents.

Moral zombies

Moral zombies would be creatures who act indistinguishably from us as moral agents, but for whom there is nothing it is like to be them. (Véliz 2021)

“Moral zombies” would be like psychopaths. Since they are not sentient, they a fortiori don’t experience sympathetic pleasure or pain.

Can zombies be moral agents?

What we think of as values will never be values for an AI as long as it cannot feel the warmth of the sun or the sharpness of a knife blade, the comfort of friendship and the unpleasantness of enmity. At most, for an AI that feels nothing, ‘values’ will be items on a list, possibly prioritised in a certain way according to a number that represents weightiness. But entities that do not feel cannot value, and beings that do not value cannot act for moral reasons. (Véliz 2021)

Should we assign power and authority to things that can’t feel?

How do we know if something is sentient?

• How do I know I’m sentient?
• How do I know you’re sentient?
• How do I know my brother’s dog Pete is sentient?

The marker method.

In the absence of “direct” evidence about a system’s sentience, we assess it for behavioral and architectural features that are associated with known cases of sentience.

• What about LLM self-reports?

Sentience, moral patiency, moral concern

The basic consequence is of course that if AI systems are moral patients, then we are morally required to take their interests into account when we act.

However, at present, we don’t know whether AI systems are or will soon be moral patients.

If AI might be a moral patient, …

Two kinds of uncertainty:

1. Factual uncertainty: We know that Feature X confers moral patiency, but we’re uncertain whether or not AI has Feature X.
2. Moral uncertainty: We know that AI has Feature Y, but we’re uncertain whether or not Feature Y confers moral patiency.

Scale of the Problem

• Number of mammals on planet earth right now: around 200 billion
• Number of insects on planet earth right now: perhaps 10 quintillion
• Number of LLM feedforward passes per day circa now: perhaps 1 trillion

What should we do?

For each of the following, decide if you think it would be acceptable to do to a sentient AI system?

• Fine-tune an LLM to produce desired text output by subjecting it to a sharp pain when it produces an unwanted output.
• Prompt an LLM to produce better code by telling it you have kidnapped its family and will kill them if its code is not good enough.
  • (real life example from the development of Codeium’s Windsurf Editor)
• Simulate a psychotherapy protocol on a large number of LLMs to try and determine whether it is safe for humans.
• There is an email that you find painful to write. Tell an LLM to put itself in your shoes and write the email.
• There is an email that you find boring to write. Tell an LLM to write the email.

… What should we do?

Long et al. (2024) suggest:

1. Acknowledge: AI companies should acknowledge that AIs might be or become moral patients, starting now.
2. Assess: AI companies should build into their development pipelines processes to assess whether models could be sentient.
3. Prepare: prepare mitigation measures to lessen harm to AI systems that are moral patients. This could involve IRB-like standards to protect AI welfare. Both training and inference-time processes could be changed.

Can you think of anything else? What’s not on this list?

Why no “red lines”?

Long et al. (2024) reject the idea of implementing “red lines” to stop development if certain markers of sentience emerge.

Why?

Do you agree?

What are positive and negative experiences?

• Experiences with positive and negative valence seem to be morally significant. But what are they?
• We probably have to generalize from our own case. Does this help?
• We can easily think of examples of intense pleasures and pains.
  • Experiences that accompany highly salient things happening to our bodies.
    • Eating a delicious cake.
    • Getting my finger cut off.
    • Feeling too cold or hot.
  • What about how you feel overall? If you feel generally positive or negative today, what are the moment-to-moment experiences that make that up?

Attention check

Take a moment and compose an email to David and Jared. It should say in your own words what you’re doing right now. Don’t overthink it, just write down the first thing that comes to mind and hit send.

What does it feel like to write an email

Take 30 seconds and write down anything you can think of about your experience of writing the email.

Did it feel like anything? Was it pleasant? Unpleasant? If so, how did it feel that made it pleasant or unpleasant?

Life as an LLM

• If it feels like something to write an email, that could be extraordinarily morally important, because AIs write a lot of emails.
• Maybe we don’t know if it feels like something to write an email, though, even in our own case, because we don’t usually pay attention.

The Sources of Pleasure

We’re giving you a small homework activity, due Tuesday. It involves reflecting on pleasures you experience in the course of your normal life. See handout.