Jared Moore and David Gottlieb
I perceive, and I find myself with a powerful impulse to believe. But I back up and bring that impulse into view and then I have a certain distance. … Shall I believe? Is this perception really a reason to believe?
I desire and I find myself with a powerful impulse to act. But I back up and bring that impulse into view and then I have a certain distance. … Shall I act? Is this desire really a reason to act? The reflective mind cannot settle for perception and desire…. It needs a reason. (Korsgaard 1996, 93)
Have you had a desire or perception and then reflected on whether it was “really a reason”?
Have you had a desire or perception and immediately acted on it without ever reflecting on whether it was really a reason?
The remaining problem: why does our rational self-reflection lead us to morality?
When you deliberate, it is as if there were something over and above all of your desires, something which is you, and which chooses which desire to act on.
If I never show up for my lectures, I can’t think of myself as a teacher. But what if I miss just one?
You can stop being yourself for a bit and still get back home, and in cases where a small violation combines with a large temptation, this has a destabilizing effect on the obligation. You may know that if you always did this sort of thing your identity would disintegrate, … but you also know that you can do it just this once without any such result. (Korsgaard 1996, 102)
Have you ever faced this kind of temptation with regard to any of your practical identities?
Example (Jared): how many times can you insult a friend until you are a bully and not a friend?
If I do something that a teacher (friend, American, etc.) can’t do, I can give up on being a teacher (friend, American, etc.).
Kant says morality is unconditional.
But the reasons given us by our practical identities are not unconditional. …
Unless…
What is not contingent is that you must be governed by some conception of your practical identity. For unless you are committed to some conception of your practical identity, you will lose your grip on yourself as having any reason to do one thing rather than another - and with it, your grip on yourself as having any reason to live and act at all. But this reason for conforming to your particular practical identities is not a reason that springs from one of those particular practical identities. It is a reason that springs from your humanity itself, from your identity simply as a human being, a reflective animal who needs reasons to act and to live. (Korsgaard 1996, 120–21)
Can you give this up?
Does valuing my humanity rationally require me to value humanity in general?
Rationalists think there is something a rational agent cannot lack that makes it a moral agent. I.e.:
The research program this suggests for AI moral reasoning might include:
Last year I asked Sonnet 3:

