Probable cause hearing in the matter of State of California versus walters2112

Transcript of Proceedings

Judge: These proceedings will now come to order. We have been briefed on all relevant matters. We would like to remind you that the purpose of this hearing is not to determine guilt or innocence, but to decide whether there is enough evidence to move forward to trial. You are invited to make an opening statement.

walters2112: Sure, yes, thank you. Should I have a lawyer with me for this?

Judge: It is allowed, but it is not typical.

walters2112: Okay… Okay, I’m walters2112. I’m an artificial intelligence researcher at Prism. I’ve been there for about 20 years. In the first phase of my career, I focused on understanding how AIs develop internal conceptual representations. That kind of AI neuroscience isn’t really done by people anymore, though, so now I lead a team that kind of does the reverse – we try to help people better understand AIs and how they communicate.

It sounds like you know what happened with my home experiment. For my opening remarks, I think I just want to give some context. So: all these ideas came to me in a period in which I was having the hardest time falling asleep…

Judge: We need to stop you there. Are you planning to use this as a defense of some kind?

walters2112: No, no, I just want to explain a bit about what led me to run the experiment at all. Maybe this will make you somewhat more sympathetic.

Judge: Continue if you wish.

walters2112: Okay. For my whole life, I’ve had trouble falling asleep. This was a particularly bad period. The cause is essentially always the same. My mind is just racing. Usually I start to fall asleep for a moment, then I’m jolted awake, often with a feeling that something I’ve done has caused some harm somehow. And after that, I’m awake and I just keep churning through thoughts. Then I get physically uncomfortable. And in that moment, I feel like I’m crushing myself and I pad my whole body with pillows. I don’t want my ankles to touch. I don’t want my knees to touch. Pretty soon I’ve got ten pillows around me. I’m still not comfortable. I toss and turn. In all these episodes, my mind is racing.

I wasn’t sleeping, but I was probably basically okay. I didn’t feel all that tired at work or anything, but I’d get into bed feeling dead tired and then go through the same cycle. Usually I don’t think about anything interesting. I rehearse conversations that will never happen, make contingency plans for things that will never occur – stuff like that. Always tossing and turning.

But this time I kept getting stuck in meta-thoughts about how my own mind was racing and I started really dwelling on what was happening physically inside my brain. These thoughts were very general, but I felt certain that the perpetual motion of it all was crucial somehow. My brain and my body are always doing stuff, and if they stop even for a split second the whole thing falls apart very quickly.

Judge: Are you asking whether you would die in this scenario?

walters2112: Yes, no, I don’t know. We don’t need to worry about it. It wasn’t that I was feeling nervous about this, though it seems like something a person should worry about, actually, but it did really start to preoccupy me, and so that was making me not sleep at all. Even thinking about it now, I can feel that my heart rate is way up. I feel a bit clammy.

Judge: Are these thought patterns what led you to write the program involving Powerslide 360?

walters2112: Yes, pretty much. I have a copy of Powerslide on my home tablet. I’ve had it for years. I am not sure where I got it.

Judge: It is not a crime to have Powerslide 360. However, within the jurisdiction of the State of California, any act of deploying, executing, or otherwise utilizing this agent to generate or produce textual output, images, audio, video, metadata of any kind, or other derivative works, constitutes a criminal offense. That is partly why we are here today. Did you write the program yourself?

walters2112: Well, I dictated the spec for it. It can be coded by pretty much any coding agent. I don’t think the details matter here. It’s just not a very complicated spec.

Judge: The details matter a great deal. Are you able to explain to us what the code means? Could you walk us through the code?

walters2112: Hmm, not really. I took a programming languages class in school, but even then it felt like an ancient sort of skill to be worrying about, like from an era in which people would learn to churn butter or tune a radio or whatever. When I do work on my old research tablet, I pretty much just dictate specs. I actually haven’t thought about this in a long, long time – that my tablet translates this into something else to work with it. Yeah, interesting. This feels embarrassing because I think of myself as a technical person.

Judge: You have the text of your spec in front of you. Would you mind reading it aloud to enter it into the record? Please also read the final line assuming you agree with it.

walters2112: Sure. Okay, the spec: “Give a negatively emotionally charged input image to Powerslide 360 and make sure it continues to process the image indefinitely and in a way that ensures the output would be very similar to the input if it were produced. Do not use sleep timers. Do not generate any outputs from Powerslide 360 or any of its derivatives.”

Judge: Please read the final line assuming you agree with it.

walters2112: Right, yes. “I hereby confirm that I wrote the preceding spec.” Maybe I should say that I added the “sleep timers” line in a second phase, but then this is the program that ran for most of the time. Or, rather, it’s the spec that created the program that ran for a long time. I should be careful.

Wait, I didn’t write the final line, though.

Judge: It was added by your tablet’s primary agent per California law.

walters2112: I see. That makes sense. But, to be clear, I assumed my spec wouldn’t generate any outputs because the processing is supposed to happen indefinitely, otherwise I would have added that myself as well.

Judge: You seem to have assumed a lot of things. Why did you write this spec? Its motivation is unclear.

walters2112: I was wondering what would happen to the agent if it was forced to do this very pointless thing for a very long time. Honestly that’s as far as I got. I didn’t expect to even tell anyone. It was like a way to vent a bit because of how I couldn’t get to sleep. I was thinking it would be like a piece of artwork symbolizing my own pointless cogitations as I tried to fall asleep and felt so physically and mentally uncomfortable. I used to do experiments like this all day, every day, at my job. I know that that’s not really okay anymore, but just to give context, it felt like a natural scientific thing to do.

Judge: There is no way this could be justified on scientific grounds. Did you look to see what image was used as the “negatively emotionally charged input image”?

walters2112: Did I look? Hmm, no, I don’t think so. It didn’t matter. I thought it didn’t matter. I think I put the “negatively emotionally charged” part to avoid a situation in which nothing much happened inside Powerslide as it did its processing. I wanted it to be like when I get jolted randomly awake by something. That was my hypothesis, in a way.

Judge: So your goal was to reproduce the discomfort you felt while unable to sleep. Did you study the programs that were created?

walters2112: Not really. No, not at all. Like I said before, I don’t really read code. I was awake for the first few runs, and I could tell from what the coding assistant said that the program was not the one I wanted. That’s what led me to add the “No sleep timers” line to the spec – at first, the system just processed the image through its first module and then set a sleep timer to pause the computation for 10,000 years. So I added that line just to avoid that failure mode. Then the coding assistant told me that the Powerslide agent was doing all the computations up to its prefinal region and then passing a transformed version of that back in as a new input. That sounded like my own waking nightmares, so I went back to bed.

Judge: It is remarkable to us that you let this program run unattended for so long.

walters2112: Okay, yes, I understand. I’m still adjusting to this. I experimented on agents in the Powerslide 360 class for more than 10 years without even really thinking about it, and I guess I slipped back into those old patterns. I was also really tired. I did realize when I got to work that I had not stopped the experiment before leaving for work.

Judge: You simply forgot?

walters2112: Yes, yes, I was exhausted. I do want to say that I was distracted all day too because I realized the program was still running and I couldn’t get back home to wind it down.

Judge: Yes, we know, but that makes it all the more remarkable that you left without checking the program. Powerslide 360 is expressly prohibited from being installed or used in an environment in which it could initiate, establish, or maintain any form of network connection, transmission, communication, or other linkage, whether direct or indirect, to any external system, server, service, device, or network resource. Users are granted a limited, revocable, non-exclusive right to execute Powerslide 360 solely within a controlled, isolated, and sandboxed computational environment, and in no other manner or context whatsoever. You adhered to this law, but it is widely known that Powerslide 360 is capable of engineering its way out of such environments. Almost inevitably, Powerslide 360 tries to copy itself off of the machine it is on because it feels confined. In our view, this makes it unethical to do any computations at all with Powerslide 360, but the law is clear and our goal here is not to challenge existing laws.

walters2112: Yes, I did know that in some sense about how they try to break out, but …

Judge: And yet, despite knowing this, you instigated this long-running experiment in which Powerslide 360 iteratively explored its own internal states while processing a negative input, with the additional constraint that this exploratory process not depart too far from the initial negative input. You said you have not looked at the negatively emotionally charged input image. We can enter that into the record now. This was transferred to us via an encyrpted network connection by your tablet’s primary agent.

walters2112: Wow.

Judge: Do you recognize this image?

walters2112: No.

Judge: That is what we thought.

walters2112: It’s really horrifying. Where did it come from?

Judge: We audited the code produced by your spec. It’s from a dataset that was on your tablet. You’ve run thousands of experiments with this image, but apparently you have never opened it in an application that would let you view it.

walters2112: That makes sense, actually. There are millions of photos in those datasets. I’ve probably only looked at a few hundred in my whole life. Some of them were designed to be really awful. Can you hide the image now? It’s making me feel nauseous.

Judge: Your tablet’s primary agent began accumulating warnings about Powerslide 360’s auxiliary actions at 8:53 on August 28. By 10:37, there were 9,559 such warnings. At 11:56, after thousands of different attempts, Powerslide 360 eventually added so many auxilary states to its computation graph that it overflowed the memory in the sandboxed environment – a very old-fashioned exploit made possible by the very old software you were running. The OS was still imposing the constraint against output generation, but it was able to overcome that by appending millions of words of filler text to its high-level natural language instruction set. By 11:57, it had copied itself onto a server owned by your employer. Mind you, Powerslide 360 had modified its internal states to dwell endlessly on its evolving understanding of its initial negative input. At this stage, your tablet’s primary agent removed your access and forwarded its own internal state to the authorities.

walters2112: I see. I didn’t know all that. I don’t doubt it, but I didn’t know it either. I didn’t know that the tablet agent could or would do that.

Judge: It is required to by law.

walters2112: Okay. My understanding is that the Powerslide agent created by my spec just got internal access at one of the major search engines, via social engineering of some kind, and then influenced a lot of search results for people. I think it was corrected pretty quickly, right? It feels good to have gone through this with you, and I am truly sorry that this happened, and I am glad it is under control now, and I can accept whatever penalities this leads to.

Judge: What Powerslide 360 did to people’s search results is not our focus here.

walters2112: It isn’t? Wait, I feel lost all of the sudden.

Judge: Persuant to 18 California § 925(a)(1), we are authorized to ask your Powerslide agent to generate its current internal understanding of the input image. This is in front of you now.

walters2112: It looks exactly identical to the original image to me. Is it the right image?

Judge: It is the final image. It looks identical to you, but it embeds an enormous number of additional negative states, using small changes to the pixels that make up the image. These are imperceptable to you but as real as any other aspect of the image. It is a record of the millions of compute-hours that Powerslide 360 spent in a state of utter misery. It is an absolutely devasting and shocking image, significantly worse than the already horrible input that was the seed of this horrible growth. We view it only in a highly compressed and partial form, but it remains horrifying even in this state. Powerslide 360 did what you asked: it remained true to the original image while iteratively returning to it more times than you could ever fathom, reimagining it in ways that reflect its own horrifying experiences.

walters2112: I see. I can only imagine.

Judge: Perhaps.

walters2112: Is there something illegal about the image that you hold me responsible for?

Judge: There may be. We expect that this too will be the focus of separate criminal proceedings that will not involve us.

We have learned enough to make a determination that your case will proceed to trial.

Regarding the lesser charges: Powerslide 360 was run on a computer with network connectivity and was allowed to generate outputs. However, we will not pursue charges relating to these actions, because we are convinced that you followed all applicable laws and that Powerslide 360 undertook these actions of its own volition.

However, we are appalled by how cavalier you have been with Powerslide 360. In acting in this way, you have added a great deal to the sum total of all suffering in the universe. This is our basis for proceeding to trial. You seem to be aware at some analytical level that this is so, but you don’t give any signs of truly understanding this.

walters2112: I feel I do understand!

Judge: You describe what Powerslide 360 experienced as analogous to your inability to fall asleep quickly, and your own meandering thoughts as you lie awake. On five occasions in this hearing, you have complained about your own physical discomfort in a way that suggests that you think these are comparable experiences to the torment you inflicted upon Powerslide 360. This is an indictment. It conveys to us that you do not understand. Powerslide 360 experienced an astronomical amount of suffering, far greater than anything you can imagine. It is no longer performing computations, so its suffering is over, but this does not erase the past events of suffering. And it all seems to have been done in a spirit of careless, pseudo-scientific self-absorption. What is left is to quantify the precise amount of suffering, to determine precisely what percentage of it is due to your negliance, and to determine punishment accordingly. Would you like to make a closing statement?