Randal Koene:

Can you hear me?

 

Allen Sulzen:

I can hear you Randal.

 

Randal Koene:

I think I've got my volume down really low.

 

Allen Sulzen:

Randal, I can hear you just fine.

 

Randal Koene:

Oh wait, I see what's wrong here. Okay. I can hear you too. It's quiet because I've got the wrong headphones plugged in.

 

Allen Sulzen:

No problem. We have about 84 subscribers on YouTube and we are live in the way that any of them would know, and could. So, we're just pre-announcement live.

 

Randal Koene:

Have you... So, the YouTube links...

 

Allen Sulzen:

I haven't updated the links yet.

 

Randal Koene:

It should be on the live-stream active page, right? Or, the active live-stream. That's where you put it, and then all you have to do is switch the redirect when you're ready to go?

 

Allen Sulzen:

Yup. Getting that going now.

 

Allen Sulzen:

I'll put the music back on for a little bit.

 

Randal Koene:

Oh yeah, did you still want to put a link on the front page as well, or is that already there?

 

Allen Sulzen:

It says call.carboncopies.org. Do you see it?

 

Randal Koene:

Right on the front page of Carboncopies?

 

Allen Sulzen:

Oh, sorry, Yeah. I'll put it on the homepage. I thought you meant on the first slide.

 

Randal Koene:

No, no. Yeah.

 

Allen Sulzen:

Oh, I'll put that on there as well.

 

Randal Koene:

Awesome. Okay.

 

Allen Sulzen:

Grabbing 'er now.

 

Randal Koene:

And how's the Uberconference thing starting up. Is that there?

 

Allen Sulzen:

It's working really well and it started great.

 

Randal Koene:

Perfect.

 

Randal Koene:

Hi Mallory.

 

Randal Koene:

Mallory, if you can hear me, can you say something? So, I know your stuff works.

 

Randal Koene:

Not hearing you yet. Still not hearing you.

 

Allen Sulzen:

Mallory, we can hear Randal, but not you.

 

Randal Koene:

Nothing yet.

 

Randal Koene:

If you haven't done it already, you can click the settings thing in hangouts and tell it specifically which audio to use; which microphone to use.

 

Allen Sulzen:

True.

 

Randal Koene:

Not hearing you yet.

 

Allen Sulzen:

We did just test this.

 

Randal Koene:

Yeah. That's why it's good to get this started, like 20 minutes before.

 

Allen Sulzen:

I see Sarah is online.

 

Randal Koene:

Mhm, that's great.

 

Randal Koene:

Yeah, we're still not hearing you Mallory. If need be, when the time comes, I can start it up and then hand it over to you as soon as you figure out your microphone.

 

Randal Koene:

Allen?

 

Allen Sulzen:

Yes?

 

Randal Koene:

When it's 11, either you or I should say something that we're still figuring out a last minute technical glitch or something like that, rather than, waiting around.

 

Allen Sulzen:

That sounds fine.

 

Allen Sulzen:

I can do that. I'm just setting the redirects right now to go to the right spot.

 

Randal Koene:

I'm going to use this moment to test what it looks like if I'm sharing my slides, because I haven't really tested that yet.

 

Allen Sulzen:

Mallory, nothing... not coming through. You might try turning the computer off and on again? I don't know... Just making ideas here.

 

Randal Koene:

Can you stop sharing slides for a second, Allen, so I can share mine for just a moment?

 

Allen Sulzen:

Yes, actually, you just have share your screen.

 

Randal Koene:

So, I wanted to pick a specific window rather than... Okay, this window, share, and now what does it look like? Okay, it looks like that. What if I do...

 

Allen Sulzen:

You'd want to do full screen.

 

Randal Koene:

Yeah... I'm going to try to see what happens if I say present. Okay, there you go.

 

Allen Sulzen:

I'm seeing it looking like an Android-size window.

 

Randal Koene:

Oh you are? Okay. I might have to change the size of the window. Here we go.

 

Randal Koene:

Let's see what happens now. Is it still looking like an Android-size window?

 

Allen Sulzen:

Nope. Now it looks full screen. I think you can present, and then in the bottom of present, you click... It should hover over a little thing and you can click the button that makes it a browser-size window. Oh, it looks like it's not really showing up for you.

 

Randal Koene:

This thing here?

 

Allen Sulzen:

Mallory, I saw you just jump back on, and we can't hear you, still.

 

Randal Koene:

Sorry, Allen does this look about right? Or is this still...

 

Allen Sulzen:

That looks the right size now, but I do see the top of your browser.

 

Randal Koene:

So, you see the top of the browser. Oh, okay. Let's see.

 

Allen Sulzen:

Oh also, hold your... well no, that looks alright.

 

Randal Koene:

How does one do that? How about if it's like this? Does that work? Or is it the wrong size again?

 

Allen Sulzen:

That's the android size.

 

Randal Koene:

Ah, rats... Okay, so then maybe the best for me, given the weird way my window's working, I'll just keep it...

 

Allen Sulzen:

It looks like we have about six live viewers. So, I just wanted to, real quick... Hi, my name's Allen. We're, still testing out a few things. So, anyone who's viewing this, we're about to get ready with our Spring Workshop for the Carboncopies Foundation, and this has all been made possible by volunteers. So, we're all really grateful to anyone who's contributed to making this workshop possible. This workshop is going to feature a few different speakers, including Ben Goertzel, Anders Sandberg, and Jaan Tallinn. So, we're going to get started here in just a few moments. There may be just some background music for a while, and we're just trying to get one more of our speakers on if we can.

 

Randal Koene:

Ok, you can put your slides back up.

 

Mallory Tackett:

Can you guys hear me now?

 

Randal Koene:

Yeah, there you are.

 

Mallory Tackett:

Okay, great! Restarting the computer actually worked.

 

Randal Koene:

And Allen... Yeah, okay. There we go. Perfect. So it looks like we're about ready to go.

 

Mallory Tackett:

Yeah, how many viewers do we have and when should we start?

 

Allen Sulzen:

10, and whenever you want.

 

Mallory Tackett:

All right, I guess we'll go ahead and start. Like Allen said, welcome to the Carboncopies Foundation 2019 Spring Workshop. I'm Mallory Tackett, the president of Carboncopies, and I'll be moderating this event today. Our topic is whole brain emulation and AI safety. The agenda for today... We'll be starting with our chairman, Randall Koene, who will deliver a few remarks about the topic. At 11:30 AM (Pacific Time) we will show an interview with Skype co-founder and AI safety supporter Jaan Tallinn that was prepared for this event. Following that interview, I will introduce our panel and start the discussion of any issues that were raised. This is the first opportunity for any audience members to ask questions. At 1:30 PM (Pacific Time), we will have the interview, or we will show the interview, with existential risk expert and computational neuroscientist Dr. Anders Sandberg. Dr. Sandberg will then join us from Oxford to answer audience questions. Then, at 3:00 PM (Pacific time), leading AI and AGI researcher Doctor Ben Goertzel will present his remarks. This will be followed by our concluding panel discussion. There are two ways for audience members to participate. You can write questions directly in the live stream chat. This live stream chat is monitored by a volunteer who will alert me to any questions that I can then ask the panel. You can also call in and join the discussion at call.carboncopies.org or call the phone number (415) 455-3017. This is also moderated...

 

Randal Koene:

Mallory, hang on for a second. Jesse is just telling us that while he can hear you on Hangouts, he can not hear you on the live stream.

 

Mallory Tackett:

Oh.

 

Randal Koene:

I don't think that's necessarily your problem, because you're coming through on the Hangouts. So, Allen should be making sure that you're audible on the live stream.

 

Mallory Tackett:

I believe I'm on.

 

Allen Sulzen:

I also believe she's audible on the live stream.

 

Mallory Tackett:

Yeah, in Google Hangouts, I believe I'm audible. So...

 

Allen Sulzen:

Can anyone just verify that they hear...

 

Mallory Tackett:

Anyone in the live stream that wants to verify that they can hear me?

 

Allen Sulzen:

We have 13 viewers testing if the Carboncopies are seeing this. We have a guy named Luke... Help us out here.

 

Allen Sulzen:

Hi Luke. This is Allen from...

 

Allen Sulzen:

Luke, from the live stream, has confirmed, and so has Claire. Multiple people are confirming that they can all hear us. So, it's just...

 

Mallory Tackett:

Ok great.

 

Randal Koene:

Jesse, I don't know what's up with your situation there, but others can hear it. Okay. Sorry, Mallory, for that interruption. It looks like we actually are live, then. Perfect.

 

Mallory Tackett:

All right. I'll just repeat how audience members can join in, since we might've had some new people that joined. You can ask your questions just in the live stream chat directly, or you can call in at call.carboncopies.org or call the phone number: (415)-455-3017. This is also moderated by a volunteer, and it's going to be very similar to calling into a radio. When your question is chosen, you will be able to ask it to the panel directly and speak with the respondent. This second option is still new, so please be patient with any technical difficulties that we have. When the workshop is done, we would appreciate audience members to complete our survey at survey.carboncopies.org. Our thanks to our donors, our volunteers, and all the experts that participated in this workshop. With that, it's time for introductory remarks by Dr. Randal Koene. You may know Dr. Koene as our chairman and founder of Carboncopies. He's also a neuroscientist with a background in physics and artificial intelligence. He has spent the last 20 years working in neural engineering labs and companies. While doing that, he has also been studying and bringing awareness to the technical challenges for whole brain emulation and mind uploading. If you'd like to know more about his background, his writing, talks, or anything else, you can find that information at RandalKoene.com. Welcome, Dr. Koene.

 

Randal Koene:

Hi, and thanks for the introduction. I'm going to take one second here just to pop up a few slides. I'm making sure that works out correctly. So, I'm going to pick that window and hopefully I'll do it right. We're sharing the window. I'm going to make it present. Does this look correct, does this look like Android to anyone? Does it look...

 

Allen Sulzen:

Android.

 

Randal Koene:

Android, thank you. Okay, so I'll do it the other way around then. I just need to...

 

Allen Sulzen:

That looks better.

 

Randal Koene:

Yeah, Okay. Well, it's better, I don't know if I can get it to be much better than that. I suppose this is slightly better. The problem is that my monitor is actually flipped the other way around and it doesn't seem that Hangouts can deal with that. All right. However that may be, let's get into it. So, thanks everyone for joining, and thank you for that introduction, Mallory. As you probably all know... Let me just get back to the first slide here too, so that's all making sense. At the Carboncopies Foundation, we primarily focus on the technical challenges to whole brain emulation. Occasionally, we also explain why we think that mind uploading through whole brain emulation is an important part of the long-term future for humanity; and we could get into that, but I'm going to try not to do that until maybe in the panel somewhere. We haven't previously dedicated events to artificial intelligence, even though artificial brains are of course a special category of AI. There are already so many who dedicate their time and present the issues around AI. Some people, for instance, let's say companies like Deep Mind, or Open AI, laboratories like MIT's artificial intelligence laboratory and specialized organizations that have been around for quite a while, such as The Machine Intelligence Research Institute, MIRI, Open Cog, which is headed by Ben Gertz Hill and The Future of Life Institute and many, many others. Of course, everyone at Carboncopies is also very interested in artificial intelligence and an artificial general intelligence. And just to be a bit specific, by artificial general intelligence, we usually mean a kind of artificial intelligence that isn't focused on a narrow problem, but that's focused on being able to handle any kind of problem; coming up with solutions for any variety of problems, sort of like people do.

 

Randal Koene:

Now, personally, of course, I've traveled in circles where AI and AI risk or primary interests for quite awhile. As I just mentioned, MIRI is out there in Berkeley and in the whole bay area, there are a lot of people who care about this problem. So, many of my friends are, are dedicating their time to it. And then, it's clear that, of course, there are areas where these domains, whole brain emulation and AI, have to interact. If they're both part of our future, they interact in some way. And it's important to consider what the outcomes are going to be. So, when we talk about safety concerns in AI, typically what we're talking about is, what happens if artificial intelligence gets to a point where the programs can write themselves, where every artificial intelligence algorithm can come up with the next better algorithm; or can somehow improve itself along a utility function. And then, there's the idea that if this happens fast enough, it could happen so quickly that we don't know what's going on and we can't control that in any sense, this sort of "foom," this big explosion of artificial intelligence, the singularity. Now, whole brain emulation is an artificial brain, in a sense. So, in what sense is that a type of AGI we might ask? It's certainly a kind of intelligence and if it's in an artificial brain, you might say it's an artificial intelligence. Even if the sort of mind that it's creating isn't that artificial, it's something based directly on human brains. How general is it? Well, humanity, of course, was evolved to fit into the niche, the evolutionary niche, that happens to be present right now and 2 million years ago; but still, were pretty general in the sense that we keep tackling new problems using this same brain that we've got and than the tools that we build.

 

Randal Koene:

So, we're a part of this big ecosystem of intelligence, as we might say. And you could even just look at that whole ecosystem and wonder, where's that going to go? How do all the pieces interact? What can we expect? And when the ecosystem moves in a certain direction, what happens to those intelligence? The ones that are the human intelligence, as originally, what happens to them? Where do we fit in? So, we have to wonder, as all these bits and pieces are interacting, does that increase or decrease what we're calling the risk, possible risk, from AI?

 

Randal Koene:

Now, those interactions and outcomes haven't received a lot of attention so far. There are just a few examples of academic writing on it, and what we're going to try to do is we're going to try to focus on it a bit more now. In fact, that's what we do with all of our workshops. We keep on trying to highlight different aspects, different pieces of a puzzle that is... Oh, it says that my slides are not full screen. Sorry. Yeah, I know they're not completely full screen. When I do that, then we get the Android version, because my screen is flipped vertically. So, I'm afraid this is probably the best that we can do right now. Maybe, in future I'll run it on a separate laptop, or something, when I do this.

 

Randal Koene:

Anyway, what we're trying to do, is we're trying to highlight different parts of this puzzle, and there are a lot of parts when we talk about whole brain emulation and the whole ecosystem it's in. If you look at what we've done so far in our workshops, right? Since one piece that we filled in is when we did the Transcending Biology: Reverse Engineering the Brain Workshop, we were really looking at the roadmap to whole brain emulation and an update on the status of all the technical bits, what was possible now, and what may be possible soon, and which technologies are going there. Then, we did a workshop called From Brain Preservation to Reconstruction, and in that workshop we were looking, specifically, at where are things going in terms of being able to preserve a brain, then be able to slice our section it in some way, image it, and from there, get to something that is a functional model and what sort of problems are you going to run into?

 

Randal Koene:

We were trying to highlight those problems in a bit more detail. Then we've done a workshop on the metaphysics and ethics of whole brain emulation which was very different from what we typically do, because we've been focusing on the technology so much. And now we're trying to address AI safety and whole brain emulation, and I think there are going to be a lot more pieces of this puzzle as we go along. Now, the topic that, that we're looking at today, I hope it's, also, going to become an official part of our road mapping effort and we're going to advocate that it's going to be included in what I'm calling, for now, the version 2.0 workshop, or conference about whole brain emulation, at Oxford University, which I hope is going to happen this year. There's no definitive plan yet, but a lot of indicators are that this may happen this year. Speaking of that, when that workshop does happen, to update, let's say, the white paper that came out in 2007, I hope that we bring in a whole set of new people and different angles, because a lot of the things that we would talk about now, they just weren't around back then. For example, now we have Anthony Zaidor who appeared at our transcending by biology workshop, who developed the molecular bar coding approach to mapping a connectome, the Boyden team, including Adam Marblestone. By the way, Adam Marblestone is now at Deep Mind,, developed expansion microscopy and wrote a set of seminal papers about fundamental neuro-technology. None of those were present at the first workshop. We've had professor Scheffer come in who tried to explain how his team has started to attempt to reconstruct functional, drosophila circuits from structure scans as presented in our From Brain Preservation workshop.

 

Randal Koene:

And then of course, the Burger and Song labs that have been working full tilt on cognitive neural prosthesis, which is really the closest example we have of partial brain emulation. Those were not on the radar back in 2007 and that's really just the tip of the iceberg. The white paper that came out of the first workshop focused on how much compute power a whole brain emulation would need and how a preserve brain could be sliced and scanned the big problems of wholesale functional data acquisition and of how to convert brain data and to working brain models, those were hardly addressed, nor were subdomain topics like AI safety, models of consciousness, or societal and ethical issues.

 

Randal Koene:

Okay. So, at this point, much of what's been said about AI Safety and whole brain emulation sounds more like hand waving speculation, then careful study. There's been a mention in Bostrom's book Super Intelligence. Some introductory studies that Carl Schulman has done, not too much of that has been published yet. Informal communications that we've had with people from the FHI, the FLI, or MIRI. And there was one paper that came out in 2017 that was interesting by Daniel Eth on AGI and neural probes, which connects fairly closely to AGI and whole brain emulation. And then, of course, a few Sci-fi situations like the version that was depicted in transcendence where, you could say, they included both whole brain emulation and the danger of runaway AI in one movie. Perhaps that was a bit much to squeeze in there, but it was an attempt to do that, right?

 

Randal Koene:

Now, the people we've invited to this workshop have expertise that comes from several directions. Jann Talliin has a background in AI safety and existential risk, he's been busy in that for quite awhile. Dr Ben Goertzel has of course been working for years on the development of AGI and he's had a hand in many other sort of tangential parts of that as well. Dr. Anders Sandberg has done a lot of work in existential risk and he was also, of course, involved with whole brain emulation, specifically with that first workshop in Oxford where we tried to pull together some of the people at the time who had a lot of interest in the topic. And then, of course, those of us who are on the panel from the Carboncopies Foundation, we've all been involved in some sense, or another, in whole brain emulation development, even if it's mostly from the technical side.

 

Randal Koene:

Now, some people that we wish were here aren't here this time, like Carl Schulman, Nick Bostrom, people from open AI or deep mind, and more. But, the first workshop can only be so big, and there's going to be plenty to digest in this first iteration of the topic. So, see now, when we talk about... Oh, I want to get to the next slide here. When we talk about the kinds of interactions that can happen between artificial intelligence or artificial intelligence safety and whole brain emulation, one of the problems is that you get into interactions between different domains and different technological developments, and those are always really complicated and hard to predict. If you look at examples of attempting to predict these things, then often people have to choose something very constrained. Take, for instance, our friend Robin Hanson's book, The age of M, where he tries to predict from an economic perspective as an economist, what would happen if you had a wholesale whole brain emulation available and basically infinite compute power.

 

Randal Koene:

But, he leaves out some things. For instance, he does not include artificial intelligence, that is not whole brain emulation, directly in that. So, a lot of the things that in his economic model are done by copying brains. They could be done by using a different type of AI, something more narrow, perhaps. So, then that changes a lot about this society he's depicting. And you can imagine that the society that he's depicting there could easily transform itself quite quickly into something very different, perhaps, to something that does include a lot of AI. So, the problem here is that as you explore angles, different angles of the question and questions, you keep discovering that you have to clarify those questions more and dig deeper and uncover more of them. And you're going to notice this in the interviews that I did with Jaan Tallinn and Anders Sandberg.

 

Randal Koene:

Every insight uncovers many more deep questions, but you've got to start somewhere. So, I've decided to seed the conversation by asking everyone a series of questions that are based on statements, assumptions, or intuitions that I've run into. I'm going to try to take you through these very briefly before we move on to the next section of this workshop. Just so that we're all on the same page, in taking you through this, I just want to mention that, typically, when I talk to people who are coming from an AI safety background about whole brain emulation, and whether they think that this is something that should receive extra funding, or should be pushed hard that there should be work done on it, I get two different kinds of responses. Some people will see work on whole brain emulation as being something that decreases the risk of runaway AI, and there are a few of these examples that I'll mention in just a moment. And others, they seem to come at it with an intuition that whole brain emulation research is more likely to increase the danger of runaway AI. And again, there are some examples of the sort of thinking that goes into that, but as I mentioned already, most of that seems fairly hand wavy, at this moment, and it really does deserve more precise attention.

 

Randal Koene:

So, let me just get started on the first one here. Oh, I think I may have skipped one. The first one is, well, BCI for AI safety, brain computer interfaces. In recent years there've been a few, especially some very well known entrepreneurs who've started companies like Neuralink, who've claimed that work on brain computer interfaces on high bandwidth interfaces between the brain and a machine are a route to improve AI Safety by causing something like a symbiosis between humanity and AI. But, you have to dig in a bit deeper to try to understand whether that's really true or in what way that would happen, and that will be discussed more later on; but here I'm just going to mention a few of the big questions there. So, for instance, what is a high-bandwidth interface? What does it mean when we say high bandwidth BCI

 

Randal Koene:

What are we comparing that with? Are we comparing it with the rate that we can type or speak, imagery? Does it mean that you target a certain percentage of all of the neurons at the same time and stimulate them? What does it really mean? And if you can make a connection like that between a biological brain and a machine, what's the predicted effect of a connection like that? And what's the effect on the human? How does the human experience change? How much does the human become a part of that AI ecosystem? And what's the effect on the AI? How does connecting with this human brain implement something you might call a level of control or a level of supervision, in some sense, or anything like that? When people talk about neural interfaces, sometimes in the popular press, they're conflated with neuro-prosthesis, but they're not. They're not a neuro-prosthesis.

 

Randal Koene:

There, just the connection between a brain and a machine. Now, the human brain has about 83 billion neurons and each of those neurons communicates at a typical maximum rate of about a hundred spikes per second. Some of them can go up to a thousand spikes per second, but not much more than that, which is a much bigger interval than, say, the nanoseconds and microseconds that computers work at. Also, human thought is based on processes like pattern matching or giving an emphasis to certain patterns by up or down regulating the neurons in that area or on regularity, which is heavy in learning, things that fire together, wire together. And machine thought is typically based on the execution of programs, on arithmetic, on logic, and on sometimes artificial neural networks.

 

Randal Koene:

I just want to mention that I'm not currently looking at my email. I see that there are some emails coming in. If anyone's trying to actually reach me, I can't see it; a message on my phone, if there's something wrong with me coming through or whatever, please ping me there. So, if you create a high bandwidth communication pathway between the brain of a biological human and an advanced AI, how does that affect AI Safety? That's a very legitimate question that's not that easy to answer. Now, the other argument that is often made, is to urge caution in the development of whole brain emulation, because, perhaps, whole branding emulation, itself, could be a risk to humanity. Maybe because, itself, could become a runaway AI. But the scenario there isn't often clarified very well. How, for instance, would a human brain be able to follow an engineered or predictable self improvement curve? How would this compare with the notion of self improving AI that follows a utility function and uses reinforcement learning as its primary method of improvement? If a whole brain emulation is accomplished and implemented on a suitable processing platform, will a whole brain emulation be able to rapidly self improve in a manner that's akin to this supposed AI take off scenario? Is there a way for whole brain emulation to do that rapidly? Is there something like a whole brain emulation "foom?" That too, deserves more detailed thought then whatever our intuitions may be saying.

 

Randal Koene:

Now, toggling back to looking at this as a potential risk in a different way, concern about the acceleration of risky AI improvement, or the curve that AI improvement may take; that's another possible reason for being cautious about the development of whole brain emulation. But again, the way that it could happen isn't entirely clear. The way this is presented is usually that the development of whole brain emulation may lead to insights that will then make the development of AI happen faster; just like the basic insight of how neurons work that led to neural networks and deep learning, and this is still something that's used in AI. But you know, how much of that is really happening today? How much insight are we getting? And also, how does this really differ from the insights that are gained from general neuroscience research that's happening today? Is there something specific about doing research on whole brain emulation that would accelerate AI development in a way that general neuroscience research doesn't. We need to at least have some examples of that.

 

Randal Koene:

Now finally, the last question that I wanted to present at this point is, well, sometimes people will say the best way to achieve AI safety is if we ourselves can develop at the same rate as AI if we're completely merged with it, If there's no distinction between human and machine, if we somehow bring those two together, thereby getting rid of a competitive race between humanity and artificial intelligence.

 

Randal Koene:

Now again, that needs to be clarified a bit more, because we don't really know what it means to say we're linking an uploaded mind with an AI. As we just mentioned, they work in slightly different ways. What does it mean to link the kind of pattern matching that we do with the sort of algorithms that are happening in an AI? How does that change the way that the human brain thinks? How does that change the way that the AI thinks? How do they control each other? How does this whole ecosystem move forward? And then there's the question, well, even if you do this, if you merge the two and they're moving forward together, this merged ecosystem of intelligence, does that really solve the original problem of AI Safety, or is it just taking away that one problem of the competition between humanity and AI or that one aspect, because the self improving AI, which could be say something that is reinforcement learning and following a utility function, that might still be a problem in itself. Even if you've got these other merged AI and human brain uploads there, you could still have a separate class of AI agents that are a problem. So, it could be that those two things are tangential or maybe even orthogonal, in a sense. So again, that's something that deserves more time. Now, obviously we can't answer all those questions here, but we can make a start at being more precise and it's stating those questions clearly, and that's what I hope I've managed to do here. Now, I'm going to stop sharing my slides. I'm going to hand it back to Mallory here. Thank you, very much.

 

Mallory Tackett:

All right, thank you Randal. There's going to be plenty of time for followup questions during our panel discussion, but again, you can put those followup questions on the live stream or call in at the numbers. Now, we're going to be going to the interview with Jaan Tallinn. He is the co founder of Skype and he's been a longtime AI safety supporter. So, we'll go ahead and show that.

 

Allen Sulzen:

Hey Mallory, it looks like, last minute, mine was working in our tests, but it's not. Would you be able to stream it?

 

Mallory Tackett:

It looks like we're having some technical difficulties. I am going to go ahead and just play the interview on my computer and I will stream it for you guys.

 

Allen Sulzen:

Thanks, Mallory.

 

Mallory Tackett:

Just need to change my speaker output.

 

Mallory Tackett:

There we go. We're sharing.

 

Randal Koene:

I'll just introduce you first, and introduce the topic. We can always crop out anything in the beginning that doesn't belong.

 

Randal Koene:

Our esteemed guest for this event is Jaan Tallinn, an accomplished and well known entrepreneur, investor, and theoretical physicist from Estonia who co-founded the peer to peer video-calling service Skype among other ventures. He's since been a leading voice on the frontiers of AI safety and effective altruism and has co-founded The Center for the Study of Existential Risk and The Future of Life Institute. Welcome Jaan. Thanks for agreeing to give us your time today despite your very busy travel schedule. As you may know, at the Carboncopies Foundation, we usually focus on the technical challenges for whole brain emulation. Occasionally we explain why we think mind uploading through whole brain emulation is an important step for the adaptability of an intelligent species, but everyone at the Carboncopies Foundation is also very interested artificial intelligence and artificial general intelligence. Since, however, there are already so many labs and companies like Deep Mind, groups like the Machine Intelligence Research Institute and others dedicated to AI, we often don't address it in our workshops. We decided that the interaction between those two domains should be addressed explicitly. The possible interactions are sometimes alluded to, from what I've heard and read so far, there's a lot more vague hand waving, I would say, than careful study. So, we wanted to try to move that along a little bit, starting with this workshop on whole brain emulation and AI safety. So, if you don't mind, I'd love to ask you a few questions about your thoughts on AI, and from there, maybe we can gently tip-toe into questions of where AI and whole brain emulation meet. Is that okay?

 

Jaan Tallin:

Sounds alright.

 

Randal Koene:

So, you've got a long history of being concerned about, or supporting serious study of, existential risk and, in particular, AI risk and AI safety. Would you mind telling us a little bit about how your thoughts on that topic have evolved over time and where you stand on that today?

 

Jaan Tallin:

Yeah, I think my thoughts have gotten more and more uncertain, in some sense, as I’ve been exposed to more and more considerations, and more research has actually come out on achievements and progress that AI has made.

 

Randal Koene:

When you say that you're more uncertain, do you mean you’re more concerned, or just that your not entirely sure of what the problem is?

 

Jaan Tallin:

Yeah, I see a wider spectrum of possible scenarios now. Before I got involved, before I read [ ... ] and stuff, I just didn't think that was an issue at all. I was a programmer and I wasn't afraid of programs. But then, [ ... ] pointed out that, like, wait a minute, if you can create a program, that is able to create programs, that's able to create programs, and that happens to be smarter with each step, we might have a serious problem. Then I got focused on this scenario, and I do still think that we have no guarantee against that certain program: that runaway AI. However, recently, I think the Open Field project introduced a concept of transformative AI: instead of talking about recursive improving AI, or narrow and general AI, which in some sense is actually specific: we should, actually, just focus on the aspect of AI that is important for us. It can be very transformative. Even if it's narrow, or if it's recursive certain programs, its for some reason does not exceed some threshold than what we improve, but still, it's going to be transformative. So, that is, in that sense, my... I see a wider spectrum of transformative situations on transformative scenarios for AI which is, in that sense I'm more uncertain.

 

Randal Koene:

Yeah yeah. So, we could go in all kinds of directions with this, and I don't want to spend too much time on it, just because we have these other topics that are really the focus of the workshop that's going to happen. I am kind of curious, because you mention, getting real advantages for the species, for all of us, out of narrow AI out of certain directions. Narrow AI, itself, also could pose concerns, right? I mean, we've seen some issues with Narrow AI even in recent times, like the way that the stock market crash could happen, or other things like that. Right?

 

Jaan Tallin:

The Facebook scandals, they have to do something with AI? I'm not sure how much though, but still.

 

Randal Koene:

Yeah, indeed. Okay, yeah, but it's not as concerning as, say, the issue of AI agents, to you, probably?

 

Jaan Tallin:

Not that current of things, but perhaps in a few generations from now, we might have some thing that is just super-manipulative, or super-addictive. Why not go that direction, because of a certain program? Say, like that kind of a scenario are still a part of my palate.

 

Randal Koene:

Okay. Well, if you don't mind, I'd love to know what your idea is, of say, a worst-case scenario vs. a best-case scenario of how we move forward with AI.

 

Jaan Tallin:

Yeah, I get that question quite a bit. I usually don't have a very specific answer, because I think that whenever I start thinking about very specific scenarios, you're almost guaranteed to shoot yourself in the foot, because each detail, basically, will make the scenario less likely. So, I think it's more valuable to think about narrow properties of traits that really positive scenarios and really negative scenarios have in common. So, for example, I think almost all existential risk scenarios coming from AI, synthetic bio, from new quality, that nuclear is going to unlikely going to be existential catastrophe. One trait that I think they have in common, is that they are going to look like catastrophes to humans. So, it's an interesting point that the first piece of existential resource of humanity is by the Manhattan Project scientists. Where, what is the probability of igniting the atmosphere once they do the first nuclear detonation? For the rest of the plant, it would have looked like one big catastrophe with no air to breathe. So, I think that's one likely common trait in, not all, but in many existential scenarios. Like there is a hoop of scenarios one of the traits that I can think of that they have in common is that there's an increase rather than decrease in optionality so like one thing that technological progress has given us is increasing optionality. Like I do have an option to easily visit other countries. These types that hundred years ago very few people have. At least not that in that quantity. Whereas in future, if things go well we might have options to travel to other places or options to upload ourselves or not upload ourselves that's also an important position to keep. That's another common trait I think that is positive and that's another common trait that unifies a set of futures, and specifically a set of positive futures.

 

Randal Koene:

I can totally the point that you're making. I also see optionality as being one of the most important things for us. It's sort of a something that for some reason as human beings we care about that sort of thing.

 

Jaan Tallin:

I think that's more general than human I think that's agents in general that have references over the world states like all other things being equal they would prefer more options and less options because of the uncertainty that they have. I think it's a meta mind, if you have the references but you don't know what the correct actions are to take right now, then it would be valuable to have all more options because in the future you will have more information about what options are actually there. So that's a general agent.

 

Randal Koene:

You can get into some interesting sort of philosophical directions there because when you think about having as many options as possible then of course that includes things like options to do things that we currently simply can't do because maybe we can't even think that way so something like a neural prosthesis or mind uploading would be necessary even to just be able to do those things like let's say for example be aware of and react to things that are happening at very short time intervals like microsecond intervals it's a world that we currently can't live in right and so optionality might include changing who we are in a sense. It gets interesting because some people would say if we change ourselves a lot and we're not really who we are anymore as humans right now because we've changed a lot then perhaps that's very similar to a scenario where say a worst case scenario that was mentioned where suddenly the environment no longer supports humans as we are. There are areas where that might look very similar.

 

Jaan Tallin:

I think it's important to be more precise about what do we mean by optional and how do you measure the increase in optionality. You can certainly have an increase in optionality that one person or one group of people like Manhattan Project scientists have to detonate potential like earth bending device that's an additional option for them. But I might result in everyone else losing all of their options by becoming extinct so in that sense like a more general version is that like we should really measure some kind of aggregate optionality that humans and more general sentient agents

 

Randal Koene:

And even that was very very difficult to measure because you might then argue that you should be measuring the aggregate over all possible future descendants of humans as opposed to just the people who are here because let's say that something takes optionality away from ninety percent of the people but then the ten percent of the people who have more optionality that creates a much greater future with more optionality for many more people than the alternative.

 

Jaan Tallin:

We'll go into either popular population ethics and infinite ethics even. It's like active research topic and it looks like we need to solve it sooner rather than later.

 

Randal Koene:

I'm saddened by the fact that we can't talk about every topic at length because these are super interesting things and I've always wanted to dive deeper into this like how do you decide what's good?

 

Jaan Tallin:

Don't you find it almost suspicious that there's a very strong overlap between super interesting topics and super important topics, it's not always but it should be like that.

 

Randal Koene:

Yeah that could point to something as well. It's true. Yeah okay well I guess maybe we'll have to have another workshop someday on what is good or how do you decide good. But that seems that falls more into say the domain of effective altruism or something like that.

 

Jaan Tallin:

You can always kind of like make progress by going up on the level of meta. So like you say like like ah it seems really hard to decide like what is good then you go like up one level meta, okay so what kind of things should I do now that would give me more optionality to decide in the future what is good. Like, for example, at the recent conference in Puerto Rico the Future Pipe Institute organized that there was like someone who was making a really good case that it's brobably going to be really likely that this massive value in having like this contemplating period during which we basically are not like as humanity think about what's going to happen next trillions of years as opposed to like just like rushing along with the first thing that comes to mind. Because it's like just we have like good reason to think about that we're going to be better at making great value to different periods. Then we would be basically losing from not acting companies in this first period. That's not a meta point.

 

Randal Koene:

Yeah I think that's probably true. It's unfortunate that when you look at the decision makers across the world very often that's exactly not what happens it's more the opposite that the next supporter is what matters the most.

 

Jaan Tallin:

That's also like an opportunity because you see why this is happening, you see the game theoretic underpinnings of these situations. And once you can realize that, you can take a step back and think about how do you solve those. What's going on coordination problems.

 

Randal Koene:

Okay. Well having created this context and putting it in a setting like this. Maybe this is a good moment to dive into the whole brain emulation aspect of it. So just kind of introducing here where this is coming from. When discussed in the context of AI safety I get different reactions to the notion of whole brain emulation. Sometimes people will be seeing it as a potential benefit, something good that will help us not have as much AI risk because humans will be able to keep up better, we will be better integrated with changing world of technology. But sometimes I get the exact opposite reaction which is, well, whole brain emulation seems like something that adds another unknown. It adds something else that could accelerate the development of artificial general intelligence or an agent AI. And that it could be its own risk, in some sense as well. Which kind of relates to what we were talking about before. About which changes on optionality are good and bad and sort of figuring out what the difference is there. So I was wondering if we can get into a few specific questions there because there are some main avenues of thought or critiques that come up from time to time.

 

Randal Koene:

So the first one that I wanted to ask about is this idea that making a connection between the human brain and a computer. So brain computer interfaces, that is somehow a very important step towards uplifting us to a point where we are more integrated with machines and where the gap is is made smaller between human minds and artificial intelligence and the differences in our capabilities that somehow this is going to create a symbiosis as one well-known entrepreneur has said with artificial intelligence. Have you thought about BCI? And I'm not just keeping it separate from neural prosthesis or mind uploading for a moment, but just BCI as a tool for that.

 

Jaan Tallin:

I guess my clip answer would be that, like Robbie Hanson's comment about this, he said that trying to make humans competitive with computers by developing brain machine interfaces is like trying to make horses competitive with cars by developing stronger ropes and so I wouldn't rule out that there is something there, but the bandwidth argument that might just start being utterly unconvincing because if you think about it if you were kind of like partnered up with someone who runs 1 billion times slower than you do. Then that beta is basically a person who can say one word per decade. And it's just going to be a burden. Is it's okay to be useful. You already know what the word going to be. That's the answer to bandwidth. That's a claim but there might be some more interesting arguments that I haven't heard but like... It's possible that I just haven't paid attention to some more serious thinkers in that space. I can cirtaintly learn more things about the brain with higher bandwidth. And then that could be useful in brain uploading for example. Or useful for alignment purpose.s

 

Randal Koene:

I see brain computer interfaces as an essential component for data acquisition, for learning about the brain in order to be able to model brains to be able to make neural prosthesis and that sort of thing. So as a part as a technology, heading in that direction, absolutely of course it's very important. But yeah I agree with you that in itself that's not really solving a problem of this gap between the two. We don't have to get into that much deeper sense we're both on the same side with this one. I would like to hear someday from the people who do advocate that as being the most essential thing to do for AI safety. I'd like to understand where they're coming from or how they imagine this should work.

 

Jaan Tallin:

If there would be like some serious careful thinkers, that would be super interesting to talk.

 

Randal Koene:

So the next critique or argument that I often get is that if you if you work hard on building neural prosthesis and then from neural prosthesis or partial brain uploads you end up getting two whole brain emulation or some kind of artificial brain like that, this is in itself of course a type of artificial intelligence or artificial general intelligence if you like although other people would call it natural intelligence emulated. And that it would be fairly easy to make it superhuman just by say increasing the speed or something like that or making sure that the memory is never fading something like that. So then that there could also be some sort of takeoff scenario, either a slow one or a fast one. Now I'm not exactly sure how self-improvement works with a human brain. I'm sure there is a method. And it may be a bit more patchy and a bit more complicated than with something that has one specific algorithm that is following. But before we even get into that do you see ways in which whole brain emulation itself could be an existential risk to humanity?

 

Jaan Tallin:

Oh sure, the way I look... I have talked much about it. So in that sense my thoughts are not super careful here. As I out source my opinions I should be more careful and honest in this in this topic. And he believes brain uploads are net positive so in that sense I'm working on priors that come from him. But I do think that there's this almost uncanny valley before we get to brain uploads. We will get to something that is potentially capable but not really human yet. So we might have like a potentially very capable nonhuman agents on this planet which is like almost isomorphic. So that is like one scenario definitely that might be problematic. Another scenario is basically when it turns out that there's some kind of a competitive situation or something. Where you get some advantage in some competitive access by sacrificing a piece of humanity, pieces of values. Which means that like we have like some race to the bottom. That would also be another extension of disaster potentials. Getting fake humans might be problematic or ending up in some weird place. But but also if you get things right, then that might be really awesome because we might just buy more time when it comes to solving problems.

 

Randal Koene:

Yeah but it's clearly so complicated because there are so many different details about how the research evolves and how you get from the very first bits you're doing. Say you're just making Ted burgers hippocampal prosthesis or something like that. Or you're trying to emulate the brain of a Drosophila and what happens if you start building agents based on these two Drosophila brains. There's so many different paths as you said that makes it a really complicated question of how do you make sure that you end up with a net positive of that. Versus you end up going down some path that has risks associated with it. But of course that's also true for any technology development. You know every time we develop some new technology there are many different possible outcomes and products and things that could have net negatives and net positives as well. It's just too hard to... So there definitely are some ways in which it could be an existential risk. But it's not that clear. I was kind of curious about this case of saying how could people using a neural prosthesis become an existential risk to to all of us or something like that. But again that sounds like a it sounds like a great sci-fi scenario.

 

Jaan Tallin:

Yeah like these weird cyborgs with metallic voices.

 

Randal Koene:

I have to think if this is again vain optionality versus staying the same. You mentioned competition. But what if that competition itself is something that has ultimately a long-term net positive or something like that. You have a species that is a little different that humans...

 

Jaan Tallin:

Technological competition seems to have given us a lot of tools that are clearly useful.

 

Randal Koene:

We're actually making pretty good progress through these questions. Even though I thought could take forever. Let's get into the next time the next critique then or the next argument. The next argument I've heard that cautions against a strong push for neuro technology and whole brain emulation is that work in those areas may accelerate advancements towards runaway self-improving AI. And well does it concern you and how do you see this different say working on whole brain emulation versus just to wait neurosciences currently investigating the brain?

 

Jaan Tallin:

Intuitively it doesn't sound to dangerous to me. I think because it's sort of like lower hanging dangerous fruits like just throwing more computer at the stuff at simple algorithms that people are doing already anyway. And trying to create AIs that create AIs like architecture search. Or like if you're kind of tinkering with something like a spaghetti code and trying to get insights. That seems possibly problematic but like just doesn't seem to come to me as the most dangerous thing to do.

 

Randal Koene:

I was trying to think of some examples of specific things that I might think that would be discovered by doing research on whole brain emulation as opposed to the way that the most neuroscience is conducted today. It's not obvious to me. Even with things that are still problems in AI. Say for example that deep learning requires exponentially more learning as you try to train up more complicated knowledge or a better understanding of context around it. That's something that we would love to understand better. How it is that humans managed it. Especially since we can only digest very few examples for everything that we learn.

 

Jaan Tallin:

It matters possibly to the point upon that. Because human is a spaghetti code.

 

Randal Koene:

It's not clear how you would learn this from trying to do whole brain emulation. It's more something that cognitive science, AI research itself, something like that is more likely to come across that solution.

 

Jaan Tallin:

AI itself was able to figure that out when it's doing architecture search. If they feel it just like a human doing architectural reverse engineering on human minds or AI doing architecture search in AI space. My money would be on AI.

 

Randal Koene:

It can simply do so many more searches so much faster.

 

Jaan Tallin:

It's like sume are dumb, but evolution was even dumber still.

 

Randal Koene:

That brings us to the fourth big argument. And this one is one in favor over an emulation. And the idea is of course this idea of symbiosis or the merger between what is the human mind and the merger with AI and its capabilities. And the idea that you can reduce AI risk or the risk to us in a sense by being more closely integrated completely entangled with everything that's going on in AI. So that a whole brain emulation of mind uploaded person could benefit more directly from what is developed in AI because AI and modules can be part of them or they can be part of it. It's it's kind of hard to describe because I don't really know how to imagine it. But it's sort of this join them rather than compete with them. This this idea of in the past royal families used to marry among one another to avoid competition and to consolidate their power. It's a mingling of a DNA in a sense except that you could say that this is alien DNA it's AI DNA is something different than human DNA. Have you given some thought to what this merger might look like?

 

Jaan Tallin:

You brought out the alien DNA. AI is going to be more alien than aliens. If you have biological aliens, they are likely going to have similar eyes. This thing that DNA is going to be similar if we merge with aliens, but even that sounds icky to me. Just to say the least. Merging with AI should be way more icky. Even though it doesn't seem like aliens. And also I think it just it doesn't strike me as a sufficiently rigorous topic exercise. When you talk about solving a complicated problem by just ignoring most of it. Having said all that, I still I would leave open that door for some serious thought in this in the space and perhaps people actually would come up with some interesting scenarios that involve things like transparency. For example transparency to make it more difficult for AIs that are improving to do like... And the more ways we have to monitor them the more difficult would it be to do that. ...more the better we would be of steering those critical phases. So that would be one angle of these merger scenarios.

 

Randal Koene:

I completely agree that it's necessary to get more detail to think more rigorously about these examples. Because what we're doing now it's useful to discuss like this but it's still kind of hand wavey. We're saying in general one could think of how this might help in supervising what how AI is developing and stuff like that. But when I try to think in real nitty-gritty detail about how AI and and human minds can merge, then I start to think about things at a smaller scale. Not the scale of an entire brain but... Let's take an example, it's a real example, let's take Ted burgers hippocampal prosthesis. Now Ted burgers hippocampal prosthesis, when implemented on, either software or in a chip. It's really just a model that is or learning to approximate functions from input to output and it's searching for those best matches. And then using them to to do what that piece of brain tissue is supposed to do. Now there are many different ways you could implement that. And and of course, even now, that's where AI comes in. He's using machine learning techniques to build those those models. So there is a lot of ways in which developments in AI can find a home and these pieces that are then developed as replacements for bits of the brain. It could be very small. It could be at the level of an individual neuron or an entire area. It could then lead to differences like say a hippocampus that allow you to directly select which memories to keep and which ones to forget and that sort of thing. You can imagine that there could be other ways of strengthening connections between different patterns than the normal Hebbian learning. But another way to do that. So I can see a lot of ways that where at this very fine scale where AI immediately starts to become integrated into neural prosthesis and whole brain emulation because it's just the natural way to try it to solve these problems to implement things in there. And and that's the first part of it. And then the second part is where you would intentionally want to make it possible to feel that you have the capabilities of some AI system and that it feels as natural as say being able to retrieve something from your memory so that it's like a part of your brain. And and then you start to wonder, well, if these are the things that you would want to do anyway for doing whole brain emulation, then how does that map into wanting to to have some kind of safety valve on how AI develops because that's not automatic that's not something that just evolves directly out of it. You would have to expressly try to make sure that the environment in which AI is developed is such that human minds have insight as, you say, transparency into what's going on. So that's still not really an automatic thing. It still feels like even if you're trying to create this merger because you know whole brain emulation uploaded minds are already closer to software code and I mean they are, they can be, software code and therefore you see this at a small fine detail you see these these mergers happening. It still feels like the problem of AI agents that self improve rapidly and could go off in a wrong direction is kind of a tangential problem. It's still something that needs to be explicitly dealt with and isn't automatically...

 

Jaan Tallin:

Not just like individual AI agents, but also like some weird races that we might create as in this like whole brain emulation cenario. The opportunities are kind of similar in orientation and whole brain emulation that we will just create smarter humans and we definitely need smarter humans right now to figure out like how to actually stabilize the future. However the threat model again is like we might be creating smart non-humans by doing careless augmenting or we might create some weird race to the bottom in competitiveness and basically lose overall optionality and all humantics.

 

Randal Koene:

And that is what feels like not good to me.

 

Jaan Tallin:

There is opportunity there in having... Simple thing is like, I think, a lot of like augment allocates point to is that we kind of are already augmented. In some sense it would be a natural progression and so far it seems to be with with a few setbacks and seems to be like so far so good. Other question in like, how much runway do we still have until we you have to be more careful and more planning.

 

Randal Koene:

So we've run through pretty much all of the main questions and I do have a concluding question but I wanted to ask first if there is something in this whole topic that you feel that we haven't addressed or haven't talked about that you think really should be mentioned.

 

Jaan Tallin:

I haven't specifically spent a lot of time, when it comes to augmenting and uploading humans. I'm probably like another ajacent scenario that we... Another pursuit of research or research area that we haven't talked about right reasons. Its basically genetic modifications and embryo selection or or direct gene editing which has very similar, perhaps minor, threats and opportunities on augmentation. So that's another interesting alternative to keep in mind. If you say that you want augmentation for x that's one question that you have to be prepared to ask when it's like, why not just like genetic humans did you get that x, might be safer or might not depend on x.

 

Randal Koene:

There's probably a lot of things that you can't simply genetically modify humans to do.

 

Jaan Tallin:

Yeah but the cool thing though is that it seems to be a minor version or augmentation, built-in opportunities, there are fewer opportunities probably and there, are but there also have fewer threats. Because in some ways where... Some interesting point that somebody made was that you could humans, you could get a superhuman by just averaging humans. So like I guess I'm not an expert, so like, I mean the fact is that if you average human faces they become attractive. Which is like interesting phenomenon. Which basically you're smoothing away irregularity so if they cancel out. And you get like really nice attractive faces. So it's for similar reasons like when your average humans you get rid of the deleterious alleles. So the territorial alleles keep us from from potential. So in that sense, it seems like not so drastic thing to just start averaging humans like in some ways we naturally do that.

 

Randal Koene:

Like what communities organizations and companies do, by combining a bunch of humans together in some structure.

 

Jaan Tallin:

in an abstract sense. Well I think that's like a fairly different sense of averaging. Basically you're not averaging humans, you're aggregating humans.

 

Randal Koene:

Yeah, I was wondering how that compares with averaging.

 

Jaan Tallin:

The specific thing is like the low-hanging fruits in genetics sound pretty good. It's just like, just get rid of the bad stuff and why not? But then you can't go beyond.

 

Randal Koene:

Yeah that's true. Yeah okay. So then let me get to my final question for you which is really the question about action. If you want to, and as I know you do, you want to see to it that humanity can maximize sort of its long-term chances for survival and thriving, optionality as you call it, what is it that humanity as a whole should do, say decision makers or or individuals as well?

 

Jaan Tallin:

Yeah I think coordination and cooperation. More coordination more coordination seems to be a really good. One of my favorite essays is Meditations on Work by Scott Alexander. It's a book-length essay about coordination columns. All of the problems that humanity has, they can be just summed up as their coordination problems and bad... So yeah figuring out how to cooperate more, because realizing how much there is at stake. If you think that the world is at stake, then just don't forget that planet Earth it's just a tiny speck in the universe. I mean Derek Drechsel doesn't like his ideas to be associated with him because he thinks that ideas are better if they're not associated with any person. So like there's this idea from somewhere, that a utopia where you're building your utopia with this additional constraint that every step should be an improvement. They shouldn't make anyone's situation worse. And cannot alow pathological casess like somebody just like wants to be the king.

 

Randal Koene:

But you know that it's not always how solving it a difficult error landscape works.

 

Jaan Tallin:

Yeah the interesting thing is that we do have like this additional resource that most people are not aware of which is the rest of the universe. And the place is bigger than anyone.

 

Randal Koene:

That totally makes sense if the pain point that people feel is resources. Which it is currently. But I mean there are other possible pain points. Like we talked about, what if there is a change in some set or subset of people and that is somehow different and there maybe, could be, some pressures that are experienced as painful to some other subset of people so these sorts of pain points if you want to avoid all of those as well that's that can be difficult.

 

Jaan Tallin:

Yeah so there are some values that are zeros on some sense. Like those values are going to be problem no matter how much resources we have. So the idea is to try to satisfy them in by segregation. Like everyone who wants to be the king of the world can have their own planets. So like just accept the meta value that you're going to have a way more optionality than you have, way more resource and have-nots, way more way better future than even considered. It's just not going to be like relatively good compared to others.

 

Randal Koene:

Yeah by pointing to the universe, one big important thing that you've mentioned there is perspective, because in addition to cooperation, I think perspective is extremely important in having a good perspective on what's really going on. Not just being focused on you know what you think is super important right at this very moment. And that I think is a big problem, is this long-term thinking is not very common. Probably because our lives are short.

 

Jaan Tallin:

That's one of the reasons why being somewhat cautious, but still pushing the AI risk framing more like an environmentalist rathar than a socialist. Because like social risks, if you say that AI could be like really bad for society, which is true. It also creates really bad competitive intuitions in people. It's like yeah I want like my social values to be like Trump over Chinese social values and that's that's just not great. This is way better to frame it as a problem that we have in common. And like environment is that one thing that we have in common. So in that sense that frame is periodic.

 

Randal Koene:

So framing could be an important thing to work on. Better frame problems. Well thank you. This was extremely interesting. And as I mentioned, very much at the beginning, we could have gone off for hours in just one direction. Like how do you think about good or something like that. So yeah that was wonderful. Thank you.

 

Jaan Tallin:

Thank you so much.

 

Randal Koene:

Thank you so much. Okay, good luck. Stop recording now.

 

Mallory Tackett:

Can everybody hear me?

 

Randal Koene:

Yup.

 

Mallory Tackett:

Okay. Now I have the screen sharing off. Sorry for my audio going out randomly like that. That was a really interesting interview; I liked it a lot. We're going to follow this up with our first panel discussion. These questions are going to be in relation to the Jaan Tallinn interview that we just watched, but after that, we're going to take a short break and we will resume at 1:00 PM Pacific time for our general panel discussion. Again, you can ask questions directly in the live stream YouTube channel or you could call in at call.carboncopies.org or call the number (415) 455-3017. This information is also on our events page. Before we opened to questions our panelists will have an opportunity to discuss the interview and our workshop topic and then we will move to our questions. Our panelists today are Dr. Randall Koene, Dr Abdulfaz Alipour, and Dr. Keith Wiley. I already introduced Dr Koene, but again he's our founder and chairman for the Carboncopies Foundation and he's known for his efforts to advance research in whole brain emulation.

 

Mallory Tackett:

Dr. Alipour has his PhD in pharmacy and is pursuing a double PhD in psychology and neuroscience at Indiana University Bloomington. In his research, he is developing new neural network architectures that are inspired by cortical column circuitry and he is testing their predictions through large scale in-vitro neural recording. Dr. Wiley has his PhD in computer science and he has expertise in artificial intelligence, evolutionary algorithms, various artificial intelligence techniques and machine learning. He is also author of the book A Taxonomy and Metaphysics of Mind Uploading, which is available on Amazon. Welcome panelists, thank you for joining me.

 

Randal Koene:

Yes, thank you Mallory. Just to add to things you've already said, by the way, could you mute yourself, or can everyone mute themselves when someone else is speaking, just in case? Okay, there we go. Mallory, you're not muted at the moment. I don't know why. Anyway, it doesn't show up as muted, but you probably have a different mood button on your headphones or something. So, just to add quickly here, I don't see, Abdulfaz Alipour on the hangouts yet. So I'm assuming that he hasn't logged into that. That's not a problem. There he comes, he's joining right now. And just so you all know, there are a few other people from the Carboncopies Foundation who are also logged into the hangouts and you guys, you're all welcome to directly participate as well of course, just like our viewers who can call into that phone number or use call.carboncopies.org. I wanted to just quickly say, there are a bunch of questions that came in from the audience and we're going to try to get to as many of them as we can.

 

Randal Koene:

Some questions can be a little bit difficult in the sense that we may not understand exactly what you meant in your question, so we might answer it the wrong way. If you want to be sure that we get it right, that's where calling in either using the phone number, or call.carboncopies.org, is an excellent way to prevent that problem because, if you hear us getting it wrong after you've asked us the question yourself, you can just correct us and say, "No, no, I, I really meant this, could you please answer that question?" So, there's slight advantage over typing it into the live stream chat. Okay, with that, I just wanted to throw out a first question, or present this to the panel. One of the things that we got into, in that conversation, is we really started talking a bit about, what does it mean to have a good outcome for humans?

 

Randal Koene:

Because when we talk about AI risk, I guess the simple version is where you assume that AI becomes smart and then there's a terminator scenario, and all the humans get destroyed. That's the science fiction version of AI risk. But that's not the only outcome you could have. You can have an outcome where AI is simply going about its business, and humanity is just this tiny little thing in the corner that doesn't have much to do with it anymore. We don't participate much in their exploration of space, they just kind of keep us alive in a zoo. You might well say, right? And then the question is, is this a good outcome? And people may have different views on that because okay, humans are still alive, our society still exists, and maybe we don't have all that option-ality that Yam was talking about. But maybe if you look at it from the point of view of a whole intelligence ecosystem instead of what are the humans doing, maybe some people still think that's good. So, determining what a successful or good outcome is can be the first thing you have to do. And I'm wondering what the people on the panel think about what is a good outcome for us and why.

 

Keith Wiley:

Am I coming through? Well, with regards to this sort of a preference to sort of protect our, special-ness and therefor sort of make sure that we don't get pushed into a, into a sort of corner of obsolescence or something. You could re-frame that question as, what if we discovered, eventually, that there actually are other intelligent species in the universe, which is a popular position. It's actually not one of my preferred positions, but if we found out that was true and we've found out there were sufficiently, pretty far advanced beyond us, then we would have the exact same feeling of mediocrity. It'd be like, oh, well, okay, we're just some podunk, relatively doumb intelligent species in a corner of the galaxy. So then, the prescription would be, oh, well, to preserve our specialists, we need to somehow keep those aliens down or something.

 

Keith Wiley:

So, barring the technical challenge of sort of countering some alien species, that's already ahead of us, it's just philosophically unconvincing. Like, why is the solution to our bruised ego to go crush all the other aliens out there that have gone further than us? So then, if you rephrase that, then it's like, okay, so now do we have a good motivation to try to prevent AI, or some sort of enhanced versions if humanity, from coming into being? Does that rationale really make any sense when you look at it the way that I just phrased that?

 

Randal Koene:

Yeah, I see what you mean. Before I try to answer that, I was wondering if anyone else on the panel has an opinion.

 

Abolfazl Alipour:

No, no specific opinion on this particular topic.

 

Randal Koene:

Okay. Yeah, I try to grapple with this question as well because of course when we talk about whole brain emulation, we're talking about changes to at least optional changes that some people could choose for themselves. It could be that we all go in different directions. That, some people develop in one way or another. And in the end, if you look at the end result again, just like what Keith was saying, this could look just like bumping into aliens with the only difference being that there's this common origin. And then you start wondering about things like the rate of change. Does it matter how quickly we sort of develop into lots of different kinds of intelligence. So, all of this is hard to pin down and it's hard for us to determine in advance before we've thought through all these questions, what are the criteria, the parameters that we think are valuable or important about humanity and what we've accomplished, that we say is this is why we should all survive. This is why either humans as they are now biological humans should continue to exist into the future, or at least minds that think like humans to some degree or whatever intelligence evolves, needs to be derived from the way we're thinking because of certain values that we think are really important and that we want to carry them over. Or we could just say, well, this is all way too analytical and ultimately, it's just the drive to continue to exist. And as long as we feel that whatever is happening is because we choose to do it, then it's us, and therefore, it's okay. That's success. And if whatever's happening to us is not something we've chosen, then it's not success. But as you'll see later when I talked to Anders Sandberg, that isn't quite as simple either, that of when are you in control. Does anyone else have a question from the panel about Jaan's conversation, before I bring up another one or, before we go to an audience question?

 

Keith Wiley:

Sorry, nothing comes to mind.

 

Randal Koene:

Okay. So I think we'll switch to an audience question in just a moment. I want to throw one more question out there before we do that. And the one I want to put out is, towards the end of the conversation Jaan mentioned that another issue we should look at is biologically enhansed humans or, that whole other different technology, not using computers, not uploading, but, what happens if we can genetically modify to achieve whatever augmentation we want. And I'm wondering, does anyone here have an opinion about what that means either in terms of being a risk, or a benefit in itself? How does that compare to something like AGI and whole brain emulation? What do you think about this biology as the route?

 

Keith Wiley:

Are you referring to when he was talking about the sort of hopelessness of trying to bring horses up to competition with cars?

 

Randal Koene:

There was that, but then he said that, and then at the end he kind of switched back and said, well, you know, another thing that we should really not forget about when we're looking at, or trying to plan ahead for humanities, we should also look at the biological augmentation angle, because a lot of things might be achieved that way.

 

Keith Wiley:

I think my response would be the same thing that he said about the horse, which is that there are certainly no particular reason to think that blind evolution has converged on some sort of biological maxima. We just are, kind of, whatever evolution was able to cobble together on what's called a local Optima. So, we're good at what we are, but in the space of all possible DNA configurations, there's very little chance that were as good as we could be. But it's still seriously worth considering whether we believe that biology has any capacity to compete with non biology, in the long run. Right now, there are many ways in which biology is the better technology. And in that sense I just described biology as sort of successful nanotechnology.

 

Keith Wiley:

But once you sort of escape the protein and lipid and sort of need for water and once you kind of remove those biological variables that are incidental to what you're really trying to accomplish, but they're required at the chemistry level, then it's hard to believe that biology is really buying you anything intrinsic. Biology is just how we do it. It's how nature does it. It's how evolution does it. But there's no reason to think that it's really competitive. And we have all sorts of evidence that points that way. So, our relatively simplistic 20th century computers on certain metrics just outperform biology in ways that are... There no conversation to have about it. And I'm not convinced that that won't be the pattern for everything as technology goes on. I can't see an obvious counter example where biology has to be better than just going at a problem without having a preference for a particular solution. Instead of leaning toward what LG or lean toward the full space of all possible physics and chemistry, I'm doubtful that biology is going to win very many of those races.

 

Abolfazl Alipour:

Yeah, and I agree with Keith because, from an AI standpoint, I totally agree with the horse analogy of that was discussed before, and I think it would be, somehow, meaningless sometimes to think of it like that as a method to catch up with AI. But from another perspective, if you think of nondestructive whole brain emulation, you would eventually need some sort of a brain computer interface or some sort of a neuro-prosthetic to transfer information from biological brains to some artificial medium. And I think from that perspective, BCI, neuro-prosthetic, is extremely important, but from arguing for advancement of BCI and calling for more attention to BCI or neuro-prosthetics, because in order to catch up with AI, I think that would be a little bit, a little bit improbable or implausible at least.

 

Randal Koene:

Okay. Yeah, I think those are all really good points. And yes, Keith, as you mentioned, I think you're right, what Jaan was basically alluding to was the idea that you can look at biology as getting to the point of what you might accomplish with some nano technology. In a sense it is nano technology, but yes, as you both pointed out as well, there are limitations to that.

 

Randal Koene:

So, I noticed that I've sort of taken on the kind of moderator role in this bit of the panel. I'm Sorry about that, Mallory. You can always jump in and do that if you want to.

 

Mallory Tackett:

All right. Are we going to be moving to audience questions now? Okay. So, the first audience question that we're going to ask is from Leslie Seamor. He asks, "Skype moved voice and multimedia communication from conventional PLTS to open protocol, scalable Internet with firewalls, and similar security architecture constructs. Do you see any similar changes, at least at the metaphoric level when the Internet payload content breaches the wet brain directly?" Could somebody, maybe rephrase that question, or can answer to that, or has knowledge pertaining to that question?

 

Randal Koene:

I can try, but I'd love to let someone else have a first go at it.

 

Keith Wiley:

I'm still thinking, and I spoke first last time, you go.

 

Randal Koene:

Abulfazl, how about you?

 

Abolfazl Alipour:

So, if I understand this correctly, the question is, "What would happen if you are able to connect..." To put it super simplisticly, and correct me if I'm wrong, "What would happen if you are able to connect a search engine into your brain?" Do you agree that this would be a simplified version of this question?

 

Randal Koene:

Yeah, I'm not 100% sure. This is where, as I said, if people asking questions were on our call line, it would make it easy for them to clarify a little easier than typing into the chat there.

 

Keith Wiley:

I'm curious if Lezlie...

 

Randal Koene:

Oh yeah, sure. Go ahead Keith.

 

Keith Wiley:

I'm curious if he's asking about the benefits of open source or open standards as opposed to a walled garden approaches to security. I completely forgot what the acronym PLTS stands for, and trying to just read that.

 

Keith Wiley:

At any rate, I certainly think that there is, in most cases, security is better served by open standards that embody secure algorithms, sort of like the way public encryption works, as opposed to security that is achieved by trying to make sure that... Thank you Rose for decrypting that acronym. ...as opposed to security that basically depends on making sure that you... that no one even knows how your security works, and that is the method of security. That's an inherently weak form of security. So, one thing that we definitely have to take very seriously, as technology becomes ever increasingly incorporated into our brains, is we've really got to get security right in a way that we have clearly, completely, failed at in the last 30, 50 years. When I was in graduate school, one of the professors who I was inspired by, in terms of artificial life, one of the people that sort of drew me to the school I chose was David Ackley.

 

Keith Wiley:

And, he was interested in computer security and since he was one of the pioneers to artificial life, his approach to it was very, sort of, evolutionarily inspired. And he and Stephanie Forest, and others at the University of New Mexico, took an approach to computer security that was intentionally inspired by biological immune systems. And I think we have seen some of these analogies crop up in our attempts to build these auto adaptive self monitoring on the fly security systems. We do sort of try to do some of this in the modern era, but I'm not sure if we have a very refined and underlying philosophy for how we do it. Just to get back to the original question, I think that as this becomes increasingly incorporated into our bodies, it will just become that much more serious, just because it'll put lives more directly at risk. We're already concerned about hacking pacemakers, Parkinson's, and brain implants. And this is just getting more and more serious as time goes on.

 

Abolfazl Alipour:

And just to add to that, imagining that the question was, what would happen if you have brain implants and what would you do to secure the communication in these brain implants? I was actually... When you talk to people about brain implants, one of the common concerns is that, how would you secure that a brain implant, how would you make sure that someone cannot upload some viruses, or some bad software into my brain implant, where someone could basically hijack my brain implants.

 

Abolfazl Alipour:

I think possible, approaches to this challenge might be to use... So, one possible idea would be to use some sort of a distributed ledger technology that is currently developing for internet-of-things devices, and one of the approach would be to use quantum switch. But the way I think about, I think, at the moment, one possible viable option to secure communication in brain implants would be to use some sort of a distributor ledger technology. That's what I think about it.

 

Randal Koene:

Okay. It looks like the person who asked the question, Leslie Seamor, is actually trying to call in at this time. So maybe he's going to be able to clarify as questioning person, which would be very useful. I'm not sure he's ready yet, but we'll see in just a moment.

 

Keith Wiley:

Is that phone number he's attempting to use supposed to go into the Uberconference?

 

Randal Koene:

It goes into the Uberconference, which Alan can then connect directly to our hangouts. I'm not hearing him yet, so I'm assuming that Alan hasn't...

 

Allen Sulzen:

And we have Leslie, who had a question from the audience, he is here live on the call now. Go ahead Leslie.

 

Sara Kochanny:

All right, Leslie. So, how it will work out is they will just...

 

Randal Koene:

I'm afraid the audio just cut out. I couldn't hear it.

 

Mallory Tackett:

I think she's just explaining to him how it'll work, and then he'll be on.

 

Randal Koene:

Okay.

 

Allen Sulzen:

Leslie's live on the air.

 

Mallory Tackett:

Leslie, can you hear us?

 

Allen Sulzen:

I didn't mute Leslie. Sorry, Leslie.

 

Leslie:

Can you hear me?

 

Allen Sulzen:

Now we can. Thank you.

 

Mallory Tackett:

All right.

 

Leslie:

Hello?

 

Mallory Tackett:

Yes, we can hear you now, Leslie.

 

Leslie:

Hello. So, you can hear me, right?

 

Mallory Tackett:

Yes, we can hear you.

 

Leslie:

So, I believe that...

 

Allen Sulzen:

Leslie, if you would just stop the YouTube that you're watching, then you'll be able to just hear us, and then you can join in back on the live stream once you're off the call.

 

Leslie:

Okay. Why don't I try again? Okay. I have no YouTube.

 

Mallory Tackett:

All right.

 

Leslie:

I assume you can hear me.

 

Mallory Tackett:

We can hear you just fine.

 

Leslie:

So, a simplified model of would be that we have a brain implant or something, some sort of infrastructure. So, the question is... There are two questions. One is how to protect it from interceptions. Just like when you have a cell phone. You make sure that no one is listening, no one can inject any information as the wireless signal goes up. It maybe a wireless signal, but the signal is basically translated from internal nural signals. And a at the infrastructure side, there will be agents, I would assume which represent you as a person and that agent is intercepting your brain tree infrastructure communication. So, that brings up two major questions. One is how to protect the pipe between your brain and the agent on the cloud or whatever they're going to have at that time. And the other question is how the agents of different people communicate with one another.

 

Leslie:

So, what would it be... So, It seems to me that just following the paradigm that telecommunication is following, all of these protocols are standardized. I would expect that when it becomes real there will be a standard division effort. And really how secure your brain and will be will basically will depend on the quality of the protocol and the firewalls in between those regions; and if there are any central services that all agents are talking to. How is that central service protected, or something equivalent some block chain or something that', when put in place on the central server. So, basically the question was poking at other people's ideas that, what kind of systems architecture arrangements. I didn't see any of these issues being in the list of questions regarding the merger AI and membrane. So, basically, that is what I am asking. Does anybody have any kind of vision as to what would be a roadmap, and how the whole architecture would be regulated by the industry, similarly to all the telecommunications standard bodies that operate today with the conversion of an IP base communication.

 

Randal Koene:

So, Leslie, I hope you can hear me as I'm responding. I don't know how the sound is going back with Alan's trick, If you hear it straight through.

 

Leslie:

How much was transmitted?

 

Randal Koene:

You came through fine, on my end anyway. Some people are saying that they heard a lot of echo but I didn't hear that. So maybe this is...

 

Leslie:

Hello?

 

Randal Koene:

I'm hearing you fine. Can you hear me?

 

Leslie:

Hello?

 

Randal Koene:

I think that Leslie... I don't know how the sound goes back to the Uberconference Allen, if that's...

 

Allen Sulzen:

He should be able to hear you directly so we can, we can talk to him...

 

Randal Koene:

So, you don't need to... He doesn't need to turn on the live stream, or something, to be able to hear me. Right?

 

Allen Sulzen:

He should be able to hear you. So I don't know why not, but it came through well. So if you want to...

 

Randal Koene:

Okay. In any case, I will try to address the question. So, you're right, Leslie, that the question of safety for the communication protocol when you build a BCI or the safety of a whole brain emulation, in an ecosystem of AI that problem was not addressed clearly, at least so far in this workshop, and it also, to my knowledge, hasn't been addressed very clearly or explicitly by most of the talking points or the writing that's come out of the AI safety community.

 

Randal Koene:

And I think that's partly because they sort of constrained themselves to the problem of what to do about the intelligence of the AI and the possible dangerous runaway scenarios of that. Whereas this other problem, you can kind of set that aside and say, well, this is a problem for the encryption community because this is all about data paths and encrypting that and about not letting people hack your brain. It's easy to dismiss it and I don't think it should be dismissed because it's clearly not simple. All of our systems today are pretty hackable. Hardly any of them are foolproof. So, I can only say that, in my own background, I did some encryption stuff because, in information theory, you learn about that sort of thing to some degree. Back then the professors were all saying, well, even though our systems are not foolproof today, in theory, you could make them all pretty unhackable and now add on top of that blockchain, which didn't exist back then, maybe that would make it even better.

 

Randal Koene:

I'm not sure because at this point I'm no longer... I couldn't call myself an expert in that area. All I know is that it's definitely an issue because currently everything seems pretty hackable, or a lot of things do, and that's going to make a big difference to how secure people feel in the sense that, okay, I want to connect myself to this machine out there to these other agents, or I want to have my brain emulated and be an upload. These agents, again, how those agents represent us and how they communicate and whether those could be faked and other things like that, this is additional problems and hurdles you could have there. It sounds to me like just the general problem that people try to hack and misuse every technology out there.

 

Randal Koene:

And in the future it might not just be people trying to do that. It could be other intelligence, other AI trying to do the same thing too, tring to use it to their purposes. So, every little point needs to be analyzed in that sense. And maybe this is why going to open standards is where you end up because then you have the most people paying attention to at least one protocol and trying to get it right instead of lots of people having their own little thing and each of them being very breakable, not getting enough attention, discovering all the holes. So, I think something that standardizes is probably what you would typically see. So, in that sense, wrapping all the way around, my answer to your question would be, yeah, I think it would probably go in a similar direction that way. I don't know if this answers your question because it is a complicated question. It's not really in my domain, but that was my attempt. If anyone else wants to make an attempt, please go ahead.

 

Abolfazl Alipour:

I just want to say that I agree with Randal and nothing is 100% foolproof un-hackable. And yeah, there's always the concern and it would be much, much better if we have a system that defines a standard and everyone is using that. That would be much more helpful. I'm looking forward to hear what you think Keith.

 

Keith Wiley:

Right. Well we're sort of mirroring each other. I would just reiterate, I think Leslie mentioned, in theory there are notions of quantum security. There are notions of security in which even if you can't prevent someone from breaking in or listening in, you can make it possible to detect that it has occurred. Even if you can't prevent someone from breaking in without also breaking your own channel, you can make it so that if someone breaks in the whole thing collapses so that at least they don't succeed and then you can rebuild afterwards. There are all sorts of approaches. We have all become sort of jaded in the last decade or two. Every time someone says that they've got a new approach to security that's going to solve all problems we learn that's not true. I think quantum mechanics is trying to bring a whole new level of confidence to that discussion, but for my part, I just kind of lost confidence in the whole thing. Which is why I go back to what I said earlier. The research that was being done around my graduate department, although not directly with me, was to sort of accept that security is not a perfect thing. Just the way your body's security is not a perfect thing. You were subject to viral and bacterial and parasitic attacks all the time. Evolution solution was not to ever attempt to evolve the sort of a metaphor of an iron clad wall. Instead it detects and responds and reacts and fights back and just sort of does its best. And the truth is it doesn't always work.

 

Keith Wiley:

Things that attack biology sometimes get through and kill you in various ways. But that is what the system attempts to do. It just attempts to detect, adapt and fight. And I think that ultimately the right approach to security is to be adaptive. We're just going to have to make our computers and especially our computer bio-interfaces and everything like that. We're going to have to abandon this idea of a mathematically perfect system and that will be very hard for society in general to come to terms with, because we need our bank accounts to be not some sort of computer version of..

 

Mallory Tackett:

I think we might've lost Keith there.

 

Randal Koene:

Yeah, that's too bad, but it gives me an opportunity... Oh he's back. Maybe you want to repeat what you said the last like half minute or so.

 

Keith Wiley:

I was wandering around aimlessly anyway. I was just wrapping up that we really do have to accept the idea of these computerized immune systems that detect, adapt and fight and sometimes lose, but the war wages on, even if the occasional battle is lost. And that is our overarching philosophy to computer security. I don't think we're there yet. I don't think our society is ready for that.

 

Randal Koene:

I want to quickly latch onto that...

 

Keith Wiley:

...Entity that's going to allow us to do that.

 

Randal Koene:

So first of all, I wanted to say, Keith, that answer was probably the most lucid answer of the lot and thank you very much for saying all this. I wanted to latch onto it because when you mentioned you have to come to terms with the fact that nothing is going to be a perfect solution for security and you sort of have to learn to live with that, but society may not be ready for it. Suddenly, on a meta-level, I'm seeing this as something we may encounter with the whole AI security or AI safety issue, which is that the attempt to analytically figure out AI safety, which is really what the FHI, the FLI, MIRI, and others, are trying to do is an analytic approach to how do we accomplish AI safety. It may be that ultimately we'll discover that it's always going to be the same thing as this immune system approach where there is no perfect solution. It doesn't constantly work, and you may just have to come to terms with that, just like in security and communication. So, maybe this applies to the bigger topic as well.

 

Mallory Tackett:

Okay, I think that kind of concludes the answers for that question. So, unless there was anyone that wanted to add anything else on our panel.

 

Mallory Tackett:

It doesn't sound like it. So, I'll move on to the next question. This is from Ree Rose. She asks, "BCI technology could be very positive and enhancing. How do we prevent potential malicious usage and unintentional consequences?" And I think that kind of wraps back around to what we were just discussing. She goes on to say, "For example, control of a person's mood, voting, consumer preferences, et Cetera." But I would also like to add, what about indirect control, such as when we have the kind of controversy we have right now with people's data being available and not being used for marketing campaigns and targeting people specifically, directly for certain things. If that's going to be an issue once we're able to upload our brains and if there's some way that that could be accessible. And I also just wanted to mention, I am incorrect about us taking our break, we're actually going to be taking our break at 1:45 and then coming back at 2:00 PM Pacific Time.

 

Randal Koene:

Okay. Yeah, thanks for that correction. I'll take a quick first stab at this. And yeah, thanks for broadening this from just looking at how to make sure that there's no malicious, say, signals going into the brain or that your signals are being misrepresented outward. But even what Mallory was saying, how about control of a person's mood? This is actually part of Rose's question, could you have control of a person's mood and that sort of thing. And I think it's very closely related to Leslie's question in the sense that ultimately it's a security problem. It's a matter of who... What are the rights that we bestow, first of all, when we have these systems? Like when we started messing around with DNA there was the same sort of question, what are the laws going to be about this?

 

Randal Koene:

How can you use someone's DNA? Are you allowed to use a person's DNA without their explicit consent? And if they give their consent, do they really know what they're giving consent to? Similarly, when you talk about brain computer interfaces, you're going to want to have informed users and the person using it is going to have to have a fairly good understanding of the security that they've got coming along with that. And that security has to be pretty good, very good in fact. But in addition to that, there are all these indirect ways that people might find that there is some control seeping in from elsewhere that they wouldn't want. If you can give somebody images directly into the brain, for instance, if you can put them into a virtual reality of sorts, there's the potential that that virtual reality can be shaped by those who create those programs and that that can influence what a person thinks. And it can be very subtle.

 

Randal Koene:

Now the same sort of thing of course is happening today with advertising and political campaigns and all the rest, and the whole concept of fake news and all of this. So, it's not a completely new question. It just becomes more pressing when you're so much more directly connected. And so it's not just the question of security and protocols, it's also a question of having laws that people and agreements they have to make and education. Understanding what those agreements actually mean and what you can, what you can think that you are, how secure you are, and how secure you aren't. So, this happens today, even on the Internet, people don't realize how easy it is to fall for something. Say you receive an email, what happens if I click on this link? So the informed user is the safer user education is going to be an important part.

 

Randal Koene:

I know this doesn't sound like a perfectly technical solution to everything. A lot of it's kind of messy and requires bringing a lot of people up to speed and all of that. But I think that's exactly how it's going to be. It's going to be a bit messy. And the only good thing about this, assuming that not everything goes foom is that hopefully we won't have to adapt everything all at once. Hopefully it's a step-wise process for some people are going to get, say an artificial retina or if they're a locked in patient, they'll have some way of controlling their new robotic exoskeleton and something like that. And those are some fairly limited paths through which communication happens. And you can, you can figure out how to safeguard that and explain to the user how to use it. And then we become used to those sorts of things. And then it goes to the next level where maybe people have brain to brain communication of a kind. And first it's a thin channel and there are ways to secure that and then that becomes a broader channel remorse as possible and they've already learned to live with it to a degree. So, hopefully this gradual, and iterative process allows for the learning that we all have to do to make it happen. Anyway, I'll leave this to someone else now.

 

Abolfazl Alipour:

I just wanted to add to what you said and I think there are interesting promising technologies on the horizon, as I talked about before, a little bit about the distributed ledger technology. So for example, blockchain is an example that is immutable. You cannot change it. So I believe that in the coming years with the advent of web 3.0 with the Internet of thing taking over and becoming more and more prevalent, I think there might be technical solutions for these concerns and these problems that not 100%, but to a good extent, they're covering all these different issues that we're dealing with. For example, controlling someone's mood, their voting, and consumer preference, and all these different hot topics that we have to consider when we want to think about neuro-prosthetics. For example, imagine that I haven't neuro-prosthetic that is for depression, for people who, who have chronic depression. And then what happens if someone can hack into that and instead of making someone better, just, worsen their condition. So how would it be secure that, I think one technical answer to that would be to look into the coming technologies like, Hash Graph or, Iotas, and tangled technology, the Dac technology that it's coming and I think that might provide interesting solutions to deal with these problems.

 

Randal Koene:

Yeah. I totally agree that there are probably a lot of new solutions coming down the pipeline. But I'd like to caution that just because something seems hot right now, since it hasn't been used much yet, we don't really know what all of its holes are and some of them can be insidious. So take for example, what you just said about the person who's receiving treatment for depression. Now let's assume that there are 20 million people in the world who are receiving the same treatment and there's a company that's providing this. As a company, therefore has probably got some data about how often each of these people are receiving their treatment. And the user may have signed an agreement where they said that the company is allowed to use this data to improve their services or something like that. And they interpret that to mean that they can use that to improve services to these people, but not just for the treatment, but also to sell it to advertisers to send them stuff that's good for people with depression.

 

Randal Koene:

So again, you've created a loophole here where some unexpected things might start to happen. You might start to suddenly notice that all of the advertisements that are popping up for you, they're all based around depression, but they might even be subtle advertisements where they are not directly about depression. They're just advertising things that typically appeal to someone who happens to be going through depression at certain times of the day and strange, strange stuff like that. So it can be very hard to locate all of the problems in advance. That's where I think the analytical approach fails and it has to be this iterative learning approach for how do we really safeguard what we're going through right now.

 

Abolfazl Alipour:

Yes, that's correct. I totally agree with that.

 

Mallory Tackett:

Okay. It sounds like we're done with that question. The next question that I have is from Alexei Popov. I don't think I'm pronouncing that correctly. He asks, "Why are we so concerned about AI threats while staying in the body is 100% fatal? Should we better discuss whole brain emulation into context of mind uploading or evidenced based cryonics?" I think maybe what he's asking is why are we so concerned with artificial intelligence safety when there are still issues, to be solved with mortality? I'm not sure if I'm interpreting that correctly or not.

 

Randal Koene:

Well, hopefully we were interpreting it correctly, but he could, of course, call into call.carboncopies.org if he wants to elucidate further.

 

Mallory Tackett:

If you'd like to clarify on the YouTube channel, we can also look at that.

 

Keith Wiley:

So, there's one obvious way in which the question of AI threats is relevant even though biology is sort of the guaranteed death. We're all sort of thinking about solving, although it's not necessarily my primary motive in mind uploading, but, the obvious response is that everyone's concerned that AI might end not only your life but end humanity. So, if it's a choice between letting humanity continue to hobble along for a few more centuries, while we slowly figured this out and accidentally, creating some existential elimative event, then clearly the choice would be, okay, let's, let's not go straight at AI now, even though it's tempting. Let's just keep hobbling along with our 100 year life spans or 60 year life spans or whatever. And let our technology get to the point where we can do AI without wiping humanity out. I think that's the obvious sort of go to response there.

 

Randal Koene:

Yeah, I would agree with that obvious response. And just to sort of emphasize the second part of his question, shouldn't we be discussing whole brain emulation in the context of mind uploading or evidence based cryonics? Yes, of course. And that's what we normally do. Say for instance, in our previous workshops, that's exactly what we did. We talked about the technology to get to mind uploading, which is whole brain emulation of one sort or another. And the different paths that go there. We've talked about ways to work with preserved brains and how to get to whole brain emulation. And then we've expanded that to look at a few other topics as well. So I don't know if you were there when I was giving my opening remarks and I pointed out that this is a big puzzle and we keep adding in more pieces in our workshops are trying to address a whole bunch of them eventually.

 

Randal Koene:

So now we're trying to address this overlap area where AI, safety and whole brain emulation may interact. And that's an acknowledgement of the fact that yes, we personally may really care about whole brain emulation as a technology either because we think it's super important for individual people or we think it's super important for humanity as a whole. But, but we have to acknowledge that you're not working in a vacuum. If we create whole brain emulation, if we're doing the research towards whole brain emulation, the things that happen because we're doing that work may interact with the other things that are happening around us, say in the world of AI Development and where that goes. And that all together can have an effect on where we all end up. So the outcome isn't just dependent on our thinking about mind uploading and cryonics. So I think that's why we're trying to, you know, find these, these corners situations, these tangential things, these places where domains overlap and think about that as well.

 

Abolfazl Alipour:

And from another perspective, I think it's still important to think about AI threats. So let's, in an imaginary scenario, let's imagine that we have the first whole brain emulation and that one becomes an AI thread by itself and it takes over and it makes it impossible for everyone else to perform the whole brain emulation. And so it wipes out the entire humanity. And that would be the only thing that exists on the planet. So I think still it's really important to consider that AI threads, especially when it comes to whole brain emulation, what happens, and I think this is one of the four topics of this workshop as well, that whole brain emulation itself would be a runaway AI. And I think from that perspective it is really important to consider this AI threat topic, in whole brain emulation.

 

Mallory Tackett:

All right. I think we'll move on to the next question. So let's see. I believe we will do the question from Roman Citalu. "His whole brain emulation necessary for mind uploading? Could a far simpler model be sufficient to simulate the human mind of a particular person?" To also just keep it in the context of our workshop, I'm curious to know if this simpler model would be generated by AI Algorithms, instead of necessarily completely doing a one to one copy of the biological brain.

 

Randal Koene:

That's a great way to put an extra spin on it. I'm going to jump in right away, because it's so happens that there was a part of the upcoming interview with Anders Sandberg that got into this. And this is because, of course, you can make a distinction between a simulation of a person's brain or mind and an emulation where the distinction we'd make is a simulation is something that to the outside looks like perhaps this is a person reacting the way they should be reacting. But on the inside perhaps it's really not all the same. There isn't really the same person behind it. It could be more of an actor. And that's precisely where Anders, for instance, pointed out well. Our systems are best when they understand us best. So we can see with a cell phone, for example, that it works better when the interface kind of understands where we're coming from.

 

Randal Koene:

When let's say Siri learns something about us or whatever, or Google learns what our likes and dislikes are and then they can go out and find the things we enjoy. They become our agents looking for the answers we want to find or the things we want to buy and soforth. Now as you get better and better at that, if you have an AI that keeps on making better predictions of you at some point that AI could go out there as an agent and basically pretend to be you and go shop for the right things, invest your money the way you would be doing it, talk to your kids to make sure they feel like you're paying attention to them and all that sort of thing. And it gets to a point where you could say, yeah, there's a pretty good simulation of that person there.

 

Randal Koene:

But the question is, is that simulation the same as an emulation is something that can pretend to be you and other people won't notice it's not, is that close enough? Is that good enough? Is that what you're going for? And that wraps right back around to, I guess what is always the fundamental question that we bring up whenever we talk about whole brain emulation and research towards it, which is what are the success criteria? What do we mean by mind uploading and whole brain emulation? What is it we want to accomplish? Do we want to make agents that can pretend to be us online? Is that the main purpose? Do we want to create something that can improve us ourselves? Where we have the ability to say, think faster, you know, notice things at a microsecond, a splint interval and react to those things.

 

Randal Koene:

Do we want to be able to live longer ourselves by being able to emulate our brains and run as a mind upload in another body, have an artificial brain as it were. Whether you decide to do this by replacing bits one at a time or scan the whole preserved brain or which approach you want to use. So part of the question is always, why is someone interested in it? What is the success criteria in what they're going for? And so this part of this is reaching a consensus, what do those of us working on it really think the success criteria should be, what are we aiming for? And some of it is perhaps something you can derive more analytically where you can say, well, if people are interested in mind uploading, it really only makes sense to go after this if we at least care about x, otherwise we could just make any old AI or something like that. So there are reasons, by... I'm sorry, somebody wants to go on. Okay. I'll finish my thought later. Next person.

 

Mallory Tackett:

We actually have Roman on the Uberconference line. I think he's going to discuss his question. So we're going to have Roman on now. Oh, actually he just said his question was answered correctly, so that's great. Is there anyone else that was wanting to add any thoughts to that question?

 

Keith Wiley:

If I had anything around the safety angle that we're trying to put on it... Randall's very good at phrasing the underlying question here about what's the criteria we want. There are a couple of ways in which I've seen this sort of issue come up. There's always the question of philosophical zombies, this notion that if you emulate... Well, no, actually philosophical zombies... No matter what level you emulate, have you actually sort of achieved the goal of identity of consciousness, sort of re-invigoration in some metaphysical sense. Of course the best way to just sort of respond to that quickly is nobody knows and let's just move on for the time being. Now there is this other question, and there are companies that have done this. I can't remember if Randal touched on this. There have been companies around for several years now that are, sort of, trying to get a foot in the door with a sort of... What would you call it? Sort of, my mind archival... They don't even call it that, but just this notion of let's capture as rigorous a snapshot of a person's life as possible and see if we can turn that into something that we would call an imperfectly preserved mind and identity. There have been multiple approaches to this. I can't remember where I read this article. There was some guy who... He was a journalist, and he interacted with a company that was doing this. And their goal was to, basically, they would do several things. They would run interviews with you and try to sort of compile this sort of Chat Bot.

 

Keith Wiley:

They're trying to refine a chat bot that responded increasingly like the person does in an interview. They would scrape your social media profiles or webs,ites or anything available to try to build as complete a picture as possible of a person. And you can start seeing where this is all going. And then the larger question is, is there a fundamental error in the assumption that this is all predicated on? Thank you Leslie. I see that in the chat. The Replica Chat Bot with a K. Oh, Mind File, Jan. Yes, I remember that name.

 

Randal Koene:

Yeah, that's the Terresam movement right?

 

Keith Wiley:

See, I have been aware of these projects for so long, I've forgotten. So it sounds like we're all aware of us together. So yes, there are people who have asked the question of whether you can achieve something like mind and identity preservation. And then the secondary question, whether you can actually achieve any consciousness preservation. With these very high level black box approaches to what personhood means, you basically take a turning test approach to the whole thing. Which is that a person is just what you can get in and out through the IO interfaces with the system. So basically what can you put into it through its eyes and what can you get out of it through its mouth. And if the patterns are sufficiently like the person, then did we achieve our goal or not? And I'm going to safely say I don't know. But we've been trying to figure this out for a while now. I kind of go over this somewhat in my book, that Randal and I have kicked around the idea of at what level do you perform your system identification.

 

Keith Wiley:

It's the term Randall maybe very familiar with. At what level would you cut your system identification cutoff at and say, if we don't system identify at this low level, whatever our threshold, we can we choose, then we just say, it just not good enough. How do we set that threshold if you're really achieving the same behavior eventually? So I don't have an answer to that, but it's definitely a question that people have been thinking about for awhile. That's kind of all I have to say on that.

 

Randal Koene:

That was an excellent answer that went into a lot of detail. Yeah, exactly. It's about what is the good level of separation between what you would call a successful upload, having captured what you care about, and all the stuff underneath. Yeah. Perfectly said.

 

Abolfazl Alipour:

Yeah. And just going off of what Keith said... So, yes. We all want to have the simplest program that can preserve the identity of a person. And so one of the things that we need is, as Keith just mentioned, is consciousness. So what is the simplest thing that you can do to preserve consciousness? And that goes back to this idea of scale suppression. So should I be able to simulate or emulate all the ionic channels? Or should I be able to simulate all the neurons or the brain region? So at what scale, and at what level I can stop and then say, okay, this is enough to recreate that consciousness or recreate that mental experience, that phenomenal experience. So, and I think we don't know yet, but many neuroscientists may say it's just at the neural level. You need to emulate your system, your artificial brain at the normal level to be able to recreate that consciousness and preserve the personal identity, but we don't know yet. It was a great question, but no one knows yet.

 

Mallory Tackett:

All right. Is there anyone that wants to add anything else to that question? I don't think so. So our next question comes from Vitaly, and I believe this is actually a question that we've kind of addressed before in previous workshops. He asked, "How detailed is the emulation of electric chemical processes you intend to use? Sub-threshold dendrite potentials, are they taking into account? Are features of spines taken into account? What approaches are used to emulate chemical synapses and neurotransmitter's?" And one thing that I kind of like to add to that, as we experiment with whole brain emulation and with going down to different levels of detail, and as Vitaly asked in his question, are the insights that we gain from that information going to help with developing artificial intelligence.

 

Randal Koene:

I missed a bit but I got your question.

 

Mallory Tackett:

Okay.

 

Randal Koene:

Very nice...

 

Mallory Tackett:

I think you're having some connection issues Randal.

 

Keith Wiley:

I'm not sure if it's you, but try again Randal.

 

Mallory Tackett:

Randal, while you're figuring that out, could you mute yourself and maybe we'll have Abulfazl or Keith try to answer that question.

 

Keith Wiley:

Let's see. I was actually thinking Randall was the guy for it. So the question is, at what neuro-level do we want to attempt, these brain emulations. Of course no one has decided yet. One of those popular answers to that question, and one that I personally favor, is to take that sort of input output function I was describing earlier, at the action potential level of neurons. Which is to say that when you ask about dendritic spines and such in your question, I am with reservation actually writing off of that requirement. I'm proposing that maybe the better level is the one at which you build a system that can propagate signals through a network in a pattern that is similar to the way that a brain propagates action potentials through a neuron connected network.

 

Keith Wiley:

So whether or not the synapses actually have the exact same properties would be irrelevant, so long as the action potentials to actually get through in a statistically similar fashion. That's not to say that that's the right level, it's just one of the more popular levels to go to. I actually personally believe that that can't possibly be the level we have to go to because when people lose neurons, from a variety of medical maladies, at least at a low level, it seems to have very little effect on our health and our personality. You can lose a couple of neurons here and there and it just wouldn't make a difference. I personally suspect that something along the lines of cortical columns or other conglomerations of neurons that we would have to decide what our grouping criteria is, but some sort of unit of groups of neurons that performs a function and this might be as large as thousands of neurons like cortical columns, is probably, sort of the Lego brick we're looking for in all of this. But again, I don't really put a flag in the stand very deeply on any of this. We don't know. There are, of course... I'll let somebody else speak in a second, but just to quickly... There are definitely people who propose much lower levels. Hameroff and Penrose are well known for proposing the microtubule structure inside of neurons as a critical component of consciousness. And then of course, presumably of identity preservation and everything else we're trying to get at. People have taken this as an interpretation that they're against technologies like mind uploading and such. And I believe that they've been cornered in the occasional interview on this and it actually said, no we're okay with it all, but you do have to take into account this quantum mechanical requirement that Penrose and Hameroff are stipulating. So many of us are not actually on board with that requirement, but it just says the same thing. It says, okay, well there is this level of perfection inside of atoms that you've got to replicate or else we're going to say that the process has been successful. So you can make this decision at several levels of abstraction and surely the right answer at this time in history is we don't know yet.

 

Abolfazl Alipour:

And yet just talking about Penrose and Hameroff orchestrated objective reduction theory and the whole bunch of other quantum mind theories out there... So these series of really interesting getting consciousness to the molecular level... However, at the time being, researchers have not been able to show a mechanistic description of how this quantum effect, or quantum phenomenon, happens that can give rise to consciousness. For example, Penrose and Hameroff idea had a real big push back from the neuroscience community about the ability and how it could work in the brain, given its temperature and noisiness and other conditions that exist in the brain. And I think as you mentioned, probably at the nural level or at the circuit level we would find good enough information so that we could be able to somehow recreate the mental phenomenon on an artificial medium.

 

Mallory Tackett:

Okay. Since we did say we're going to do, Anders' interview at 1:30 Pacific, we're going to go ahead and skip our break and end the panel for now, but we will address any further questions from the audience members in Q and A sections later. So, with that in mind, we will commence with the second part of our workshop, which is the interview with Dr. Andrews Sandberg. He is with the future of humanity institute at Oxford University. This interview was conducted by Dr. Randall Koene. And I will just have to play it on mine. Unless Alan can play it. I'm not sure if Allen is going to be playing it.

 

Randal Koene:

Allen, if you can, maybe we can give yours a try and see how that one goes.

 

Allen Sulzen:

I'm queuing it up now.

 

Mallory Tackett:

Awesome.

 

Randal Koene:

There may be some noises in the background here too, eventually. It shouldn't matter too much.

 

Anders Sandburg:

We can overpower them with our own charisma.

 

Randal Koene:

Indeed.

 

Randal Koene:

I'm not seeing it yet, but I can hear it.

 

Randal Koene:

Okay. So I'm going to introduce you first and then we can get into our discussion because we're going to be using this in our event of course. Okay, let me get started then. Our expert guest for this event is Dr. Anders Sandberg who also happens to be a longtime friend going back at least to 2007, and the first whole brain emulation workshop that was organized by the Future of Humanity Institute at Oxford, possibly even further than that.

 

Randal Koene:

...Dr. Sandburg's work in computational neuroscience, which means we could geek out about that at length if this were a private fireside chat. Andrews is a senior research fellow of the Future of Humanity... ...of course, act of future technologies and artificial intelligence and whole brain emulation. He's also an excellent speaker and debater and has always, I've really been looking forward to this conversation with you. Welcome Anders. Thanks for agreeing...

 

Mallory Tackett:

Okay. I guess our backup option is not viable, so we'll just go with our original option. So anytime my microphone goes out, I'll just be paying close attention to our chat and I'll make sure to turn it back on.

 

Randal Koene:

There may be some noise in the background here too, eventually. It shouldn't matter too much.

 

Anders Sandberg:

We can overpower them with our own charisma.

 

Randal Koene:

Indeed. Okay, so I'm going to introduce you first and then we can get into our discussion because we're going to be using this in our event, of course. Okay, let me get started then. Our expert guest for this event is Dr Anders Sandberg who also happens to be a longtime friend going back at least a 2007, and the first whole emulation workshop that was organized by the Future of Humanity Institute at Oxford, possibly even further than that. Doctor Sandberg did his PhD work in computational neuroscience, which means we could geek out about that at length, if this were a private fireside chat. Anders is a senior research fellow of the Future of Humanity Institute at Oxford University, and of course he's a highly respected in fields of philosophy that deal with existential risk, the impact of future technologies, and artificial intelligence and whole brain emulation. He's also an excellent speaker and debater and has always, I've really been looking forward to this conversation with you. Welcome Anders. Thanks for agreeing to do this interview and to try to join in on the Q and A on the day of the event, even though you'll still be returning from travel that day.

 

Anders Sandberg:

Thank you so much, Randal. This is exciting. It's good to be, not quite here, but communicating.

 

Randal Koene:

Right. So, you're, of course, familiar with the goals of the Carboncopies Foundation and I imagine that... It's okay. I mentioned that you can also see that people who are working on whole brain emulation and people at the Carboncopies Foundation are also really interested in artificial intelligence or artificial general intelligence. Now, we don't very often talk about that in our workshops because they're already so many groups out there that are doing that. But we did feel that the overlap and the interactions or possible interactions between work towards whole brain emulation and work on artificial intelligence is something that hasn't really received sufficient attention yet. So if you don't mind, I'm going to ask you a few questions about your thoughts on AI and about risks and benefits. And about this area of interaction. Is that good?

 

Anders Sandberg:

That's sounds excellent.

 

Randal Koene:

So you've got a long history of dedicated concern and ,supporting some serious study on existential risk and in particular AI risk and AI safety. Could you tell us a little bit about how your thoughts have evolved in that time since you got started?

 

Anders Sandberg:

I vividly remember being on a train ride to Jane University back in the early 2000's reading Nick Bostrom's recently published paper on existential risk and I had this knee jerk reaction. This is preposterous. This is so stupid. This is just going to be used to slow down science and we need more science and technology more rapidly. Then gradually I became aware of... Actually, that knee jerk reaction might be a little bit too naive. Gradualy, I warmed to idea that, actually, existential risk is a pretty dominant concern. It is not a super important concern, but certainly it matters a lot, but the problem is of course just being concerned is not enough. You need to start picking apart the probabilities, risks and uncertainties, just try to see where can we do the most. There are some risks that we simply cannot budge. Some super-volcanoes from the moment, for example, there is a finite risk that we will all die of natural causes. There's no point in worrying too much about them.

 

Anders Sandberg:

The other risks are relatively intractable, but many people are always working on them like nuclear war risk. And then there are those risks that are more tractable. And at this point it gets interesting. If they're neglected but tractable, then you should probably put in more effort. Especially since even a small amount of effort, if it's a very tractable risk, can help a lot. So to me, future technologies are interesting because they had a fair bit of traceability simply because we are making up these technologies as we go along. We're inventing and discovering things. And that means that we can also regulate them and safeguard in various ways if we're careful, if we make the right choices, which is not always possible.

 

Randal Koene:

And besides artificial intelligence and related areas, are there other examples that you would say are areas where we could put in more of an effort then we're doing today and maybe not super-volcanoes or something like that, but other areas?

 

Anders Sandberg:

I think Bio risks are a good example. So there's certainly a far bit of people working on some the bio-security, but many of the emerging bio-technologies probably pose entirely new kinds of risk and I think it would be a good idea to make a serious inventory of what we might find there and trying to preclude some of those risks and that might require the various forms of disease surveillance and innovative methods of actually stopping, and for example, gene drives, et cetera. And I think this might be true for other technologies. One good example is actually quantum computers, which are not an existential risk, but the risk for encryption and privacy. The solution is of course quantum safe encryption, which both MSA and Google are trying to develop. And then we need to promulgate it. Before the quantum computers become too good. We want to have it around for so long that our secrets are kind of safe because the ones that can be cracked by the quantum computers, well they're primary points.

 

Randal Koene:

Hmm. It's interesting that you mentioned that because I was just out at South by Southwest and there was a startup company presenting there. Unfortunately, I didn't catch their talk and I don't remember their name, but they were explicitly saying that they offer quantum encryption as a way to safeguard your data. Even now, I don't know how they're doing it, but there out there.

 

Anders Sandberg:

So one interesting thing is that you can, of course, encrypt data using quantum encryption over our communication slide. And it's not as easy as people thought, especially the Chinese have been working very audaciously on developing this. But then you also want to have a quantum safe encryption method, and that might actually not involve any magical quantum computing, but it might be good for your branding if it has quantum in the name anyway, And I think this is going to become more and more urgent.

 

Randal Koene:

It's interesting that we're going down this path right now, because that's a great segway into something I wanted to talk about where Dr Ben Hurtsel was concerned. He's a mutual friend of ours and a researcher in artificial general intelligence. And he sees things a little bit differently than, for example, Nick Bostrom does or Eli Asrey does. He wrote a paper in 2005, so it's a few years old now, and the paper was called Super-intelligence: Fears, Promises and Potentials. And it seems, one of the things that he explicitly talks about is that the probability of different risks isn't addressed very well and it's very hard for us to judge which risk is really the one that we should be paying attention to the most. And then, for example, is it a case where other risks are so great that the potential benefit of artificial general intelligence in helping us prevent those risks that might be on balance, more important than say, worrying too much about the existential risk of AGI itself. So I think you're probably familiar with his thoughts on that. And I was wondering how you would balance those approaches.

 

Anders Sandberg:

So that argument is a bit similar to an argument I heard for us trying to send messages to extraterrestrial intelligence. Again, we are under so much risk here. So even though maybe there is some risk of getting out friendly aliens on the phone, if we get friendly as that might actually save us. Now if one is really desperate, I think that kind of Hail Mary strategy really might make sense. But I don't think we're necessarily that desperate yet. We got some years until we get the artificial general intelligence, maybe quite a lot of years, and we actually have a decent track record of surviving so far, which might of course not indicate that risks are small but at least that we can do something about it.

 

Anders Sandberg:

So I do think that we should do this judgment, but there is quite a lot of things we can do about that. So for example, demonstrating that on the net AGI is going to be better than no Agi, well I think that is fairly doable. On the other hand, showing that safe AGI is much easier than the dangerous AGI, I think that is going to be tough to demonstrate. I think you are not going to be able to prove that within the rigor.

 

Randal Koene:

Okay. But it is still kind of hand wavy, isn't it? Because you said for instance, that you don't think we're very desperate in terms of any of those other risks but desperate in terms of the risk of dangerous AGI. Right? That's the same question.

 

Anders Sandberg:

The thing about... Right now when we look at the existential risk to humanity, I think that nuclear war still isn't the top one and you can even make a basic estimation of naively we have seen 73 years of no nuclear war. So if you apply in a data function, et Cetera, you end up with between 0.1 and 1% risk per year, which is disconcerting. Maybe a nuclear war is not an existential risk, but it's still a certain amount of probability that is kind of worrisome. Now arguing that nuclear war risk is important work, but it also takes a lot of effort. You want a lot of diplomats. You want a lot of people in these studies to do their job. Now, AI risk is relatively mobile right now because we're still at a very early stage. A few insights might actually change the risk profile quite a bit. So what I'm arguing is not so much that we know that unfriendly super-like... I know some people who actually think this should be regarded as a default position. I'm not entirely convinced by that argument. They might be right, they might not be right. But I can certainly see that by moving AI safety research earlier in time, we can reduce the risk. And probably do quite a bit of useful risk reduction that way.

 

Randal Koene:

That makes sense. Yeah. I mean, if we simply ignore that there was any risk at all and didn't have any people interested in AI safety, that would definitely open up a bigger potential for other worst case scenarios. I'm sort of curious about both of those things. Actually, in the worst case scenario and what to do about it; because now we've spoken very abstractly about the risk from AI and we've also spoken in a very abstract sense about, well we should do something about that. I'm kind of curious, what do you think is the worst case scenario that should be expected if say an unfriendly AI or AGI is developed and what do you think is the best way to try to prevent this? So, if we do have people interested in AI Safety, which route do you think is the most promising?

 

Anders Sandberg:

I do think... ...Because I think most of us, we can imagine... We can start imagining ways around this and I think we could actually have at least some chance of reducing the impact. So I would expect the worst case scenario to be very bad and totally not looking like anything I could formulate. So you could imagine a scenario like, well, actually a rapidly and self enhancing intelligence is possible. So there are some systems that generate that you get a fairly hard takeoff of with some random utility function and then it goes off and does something random but instrumentally its very, very competent so it does everything it needs to prevent us from ever being able to stop it. So now basically you're in the same universe as a force that you cannot by definition stop and he's going to do something that's likely very dramatic, like taking over earth to use for atoms or doing something else. It's very hard to say.

 

Anders Sandberg:

Now I think it's very likely that we're going to run into other bad AI problems long before this. So in many ways the worst possible scenario is that AI works really well and then improves on it and it's worked really well, and it all keeps on going super well... On the other hand, I think it's more realistic to assume that we develop AI, we do stupid mistakes that fix that, we fix those, we keep on doing things, we discover new problems. This keeps on iterating and happening. And in the nice scenario, we robustly figure out ways of actually controlling AI, setting up motivations, and other kinds of safeguards we currently don't understand. Then we're getting into truly dangerous territory. The more scary scenario, of course, is the one where we don't learn anything, but we create incentives to use it.

 

Anders Sandberg:

So companies that don't use AI to direct where activity or going to the doing badly on stock market. So everybody will be using them and then the systems get more and more powerful and the world gets crazier and crazier, but if you don't use AI, the craziness cross receive. And then you ended up in a world that is not suited for humans yet, nobody has won very much. Now the ideal scenario, is of course, that we figure out useful things early on and we fix them. And in the best of all possible worlds, maybe is some simple mathematical theorem that you can prove, that gives you a safe AI. And it's also politically very easy to convince everybody that we should be implementing this. In a more plausible world. It's going to probably be a whole little science of safety. Just like we have the same safety in a lot of other domains and this is going to be fairly messy. Let's think about computer security and think about how we're failing at computer security today because of the incentives are all wrong, the software companies are not held liable for the security of their products yet we buy them. And the end result is that we've have built an entire infrastructure on systems that we can't fully trust that in fact have enormous glaring security holes, we can all know that, and we all choose to ignore it because living without a modern smart phone is not practical.

 

Randal Koene:

Yeah. Motivations and holding people accountable, holding companies accountable is really important. So I liked that you went into this, what is a more plausible scenario and, sort of, looked at these possibilities. I mean, it's obviously very difficult to really predict anything because it's so many different routes that are possible. It's very complex to look into the future. But, that takes us a little bit away from just focusing on something assuming that the major AGI or AI out there is always going to be something that has a fixed utility function and maximizes that utility function, that's where the danger comes from. Where as the reality might be more messy than that. Which I think is also something that Ben touched on his paper.

 

Anders Sandberg:

I do think this shows the importance of trying different approaches to AI safety. So, while our friends at MIRI are taking a fairly axiomatic approach in their thinking in terms of their fixed structures. Around FHI we have some people who are working very much on learning systems and I'm hoping in the future we're going to see even more approaches because right now we fully don't understand what the domain is and we might actually want to combine different approaches or figure out what are the sub two problems with them. And this even includes the people who work on near term AI safety. Quite a lot of them are of course not terribly fond of talking about AGI and super-intelligence. They think that they have enough trouble with their autonomous cars or how to manage drones well or even get them out of the bias of the credit scoring systems. But I do think there is actually a continuum here from their near term issues about controlling complex adaptive technological systems to the long term issues when these systems become essentially autonomous, and super powerful, how do we ensure that we already have enough of what counts as an important part of control?

 

Randal Koene:

Yeah. We could talk about this forever, basically, because every single thing that we say brings up another question. When we say timelines, how long term problems versus near term problems, how long is long term, all that sort of thing. I'm going to try to steer this towards the whole brain emulation topic. So when I discuss whole brain emulation in the context of AI, when I talked to someone about that, I typically get two very different kinds of reactions. Some people were giving me the reaction where they say, I think that whole brain emulation is an important potential safeguard against humanity suffering the worst case scenario from a runaway AGI. And others will say, well, that may be true that there is something to be found in terms of long-term future benefit for humanity.

 

Randal Koene:

But at the same time, working on whole brain emulation carries its risks in terms of either whole brain emulation itself being a risk or a whole brain emulation somehow accelerating the development of potentially dangerous AI. So I, I'd like to start talking about that and ask you a few questions about it. Now I've already talked with Jaan Taleen. He gave an interview as well, which is being presented in the same event and he was cautiously optimistic. He felt that whole brain emulation research would probably end up being a net positive in terms of existential risk. And he said that he was basing his opinion largely on what he learned from Carl Schulman who also at the FHI. And so clearly we should probably reach out to Carl in our follow up to this event. But I was wondering because he mentioned that, what do you think about the level of study that's been done so far on this particular problem with the interaction between AI Safety and research on whole brain emulation? And do you think there are specific people who are the leading thinkers in that sort of overlap area that we should be talking to?

 

Anders Sandberg:

Right now, I don't think there has been enough study on this. There have been informal arguments and in many cases we see formally arguments go back and forth and a bit based on personal taste. Maybe one has invested a lot of effort in brain emulation then that is automatic to bias in towards thinking this is probably pretty safe. I think the core question is how much would the progress in neuroscience push AGI and vice versa. So one possible argument would be if we do brain emulation its not just that we by definition will have a lot of computing power, but we will also have good ways of scanning brains, that directly doesn't help AGI much, but also a good way of modeling and running, we scan the neural tissues. And that's going to, long before we get to... ...Give us quite a lot of...

 

Allen Sulzen:

The audio has been muted, Mallory.

 

Anders Sandberg:

...did neuroscientists figure out things by having actually a really good neural model to start with. And how quickly do they tell AI researchers who can actually make use of it. So I think empirically the answer seems to be that it actually takes a fair bit of time before neuroscientists understand the neural issues, even if they have good simulations. Because it's very complicated and we're not terribly good at figuring it out. As a formal computational neuroscientist, I'm kind of embarrassed by how little progress we've been making despite our models becoming much bigger. But the reason is of course we need better ways of understanding. Now brain emulation is not necessarily based on having a super profound understanding of the high level stuff that would drive AGI, but the most of the scientific interest is going to be about that rather than achieving brain emulation.

 

Anders Sandberg:

And even when neuroscientists have found something interesting like, how would the reinforcement learning systems, of the brain works or some of the neural representations, it seems to take a fairly long while before the AI research is picking up. So if you take, for example, Russell and Norway's book on AI, we relive through the chapter about the history of the field of insights that are driven. You find a long list of interesting neuroscience insights and then you try to think, how many decades it was between people figuring it out and it became a part of an AI project and quite often it's several decades. So if things stay the same, I wouldn't be too worried because we might be doing the brain emulations and then a few decades later, the neuroscientists get the memo. However, that might change in the future. It might turn out that maybe AI research and neuroscience meld together much more. It might be that actually people like the Google deep-mind, started by people coming from Diane's group in neuroscience, have learned how to read neuroscience papers and actually get useful ideas from them. With my...

 

Anders Sandberg:

... another example might be that maybe the way the... ...organizes information and does online learning, that is really a simple thing, once you understand it, and then you can apply it in AI. I don't know how to assign much credence to this, but I think it's worth investigating much more deeply and we need probably need to develop tools to do that.

 

Randal Koene:

Yeah. Oh Wow. You covered a lot of ground there and yeah, this sort of echoes my thinking about it as well. I do think that there are definitely things that we can still learn from the way the brain works compared to how AI works. Say for instance, how do you manage to learn, a large variety of things in various contexts and do so from very few examples compared to what, say, you pump into a deep learning network these days. Those are the sorts of things where there's still a lot that we can learn. But it's not entirely obvious how you discover that in the brain. And as you said, it's not clear how fast can neuroscientists convert that into something AI could use. And for instance, this is a point where Jaan mentioned, he thought that AI could probably search the space of probable models and probable methods better and faster than, say neuroscientists would be able to interpret what's going on in the brain. So if you need to solve some issue with reinforcement learning or something like that, it would probably be faster to improve that just pure AI development. I'm not sure that's true, because some problems explode very quickly. Numbers are big very fast. And then when you say we're gonna explore all possible models or something like that, if you don't have a good strategy, you could be stuck until the end of the universe.

 

Anders Sandberg:

One of the really interesting things though is that hyper parameter searches in machine learning, have become much better recently. People had some useful insights on machine learning methods to improve other machine learning methods. And I think we should expect at least that to continue. That's not necessarily searching this enormously large landscape. It's just maybe 10 or 20 dimensions of hyper parameters, but that's already a pretty high dimensional space. So it might be useful to just keep an eye on this and how good does this generalize to other domains including trying to find models in other domains. If you start seeing the modern automatic model making to really take off, then we should expect things to get rather explosive.

 

Randal Koene:

Indeed. Yeah, but that's also where it becomes much more complex than just predicting a system that will use a simple algorithm to improve its utility function maximization over time. Because if you're building completely different modules that work in a different way, then you have different algorithms and everything becomes a lot more unpredictable.

 

Anders Sandberg:

...perspective. It might turn out to be tools to do this too. So I think Eric Drexler has made a very good point in his big report about reframing a bit of AI Safety. We quite often think about the AI or the neural network is some what self contained systems. But actually the part of a practice of generating things that solve various problems and most people in business anyway don't care about AGI, they just want to solve a practical problem. So we develop in pipelines and services that generate these things. Again, that is worth watching especially since they can be applied to brain emulation, then we might get a pipeline towards better brain emulation When we get a pipeline for decoding neural tissue and figuring out things about that, then we should expect at least a boost in neuroscience

 

Randal Koene:

Indeed, and what we really should be talking about is an ecosystem of various AI instead of the single AI that's going somewhere. Yeah, indeed. So now maybe we can run through a few very specific points that get brought up because these are just things where I often get an opinion from someone or a statement, but there isn't necessarily that much material behind it. And I wonder if maybe there's a way to slowly push towards a more precise understanding of those particular concerns or questions. So the first one I'd like to get into is the idea of having a brain computer interface, a high bandwidth, brain computer interface, something that is the target of research and also of some for profit companies. I'm not going to go into mentioning names, but we probably know who they are.

 

Randal Koene:

And sometimes you hear from them statements such as if only, if we can have a high bandwidth connection between the human brain and the machine, then first of all, that will help us avoid AI safety problems because somehow it will be more tightly connected with that. There'll be some symbiotic relationship between us and secondly we'll be able to benefit very strongly from this connection it will be sort of a part of that advancement. Have you thought about this claim regarding to, now, brain computer interfaces? We're not yet talking about neural prosthesis in this case.

 

Anders Sandberg:

Yeah. So I vividly remember a few years back when Elon Musk had spouted off something that was general face palming around the office. We like Elon, but many people thought that sounded stupid and I have a big mouth. I said maybe there is something to the argument so then my manager gave me two weeks to try to make a steel man version of Org. That's the opposite of Straw man. Let's see if we can make this argument work. And I found that the first version of the argument that saying, if we get brain computer interfaces, we're going to get enhanced and then we're going to be smarter than AI, that seems pretty unlikely there are real problems in making an enhancement that makes you super intelligent. Especially if you need to do that within a relatively short time frame. We might not know how far it is until we get superintendent AI, but the time it takes to test out anything that deals with meta human biology anything that's medical and then try to develop user interfaces and enhancements, that seems to be something that potentially could take quite a long time.

 

Anders Sandberg:

There is another argument and that is if you can't beat them, join them. So I get my brain computer interface. I'm linked to the AI and the this way I'm going to be on the winning side except of course the nature of that link and what we mean by winning side is quite tricky by some accountants. My mind is already linked to my cell phone because of extended mind hypothesis, part of my mind recites in my calendar and other applications. I don't have certain memories in my head. They're in my smartphone. So maybe I'm already linked to machines. And now the dangerous paperclip AI takes over the world and turns it all into paperclips. In some sense I was on the winning side because I had a smartphone, but this doesn't sound like winning. This sounds very stupid In fact. The reason is of course what... ...and we want that link to be the right kind of link. And in some ways, it's a very deep philosophical mess. So this doesn't seem very likely to work a priority, but there is something else that actually looked at a bit promising. If I'm linked to an AGI system and it can observe my an evaluation situations, it can actually estimate a bit what I would have done if it's trying to do what I would have done instead, this actually gives us a good chance. So the system might melt...

 

Randal Koene:

You know where this is going right?

 

Anders Sandberg:

Yeah.

 

Randal Koene:

If it can predict everything you're doing. It has an emulation of your brain, right?

 

Anders Sandberg:

Well not necessarily. It might my value system but nothing else. So the naive version of it might for example, notice that I liked helping little old ladies cross the street and I don't think kicking them in front of a car is a good thing. If it's just optimizing for that, then it might for example, run away from little old ladies because it wants to avoid them, and avoid helping them, et cetera. You need more rich information. And this is where things get interesting. If it could just get an entire copy of my brain, then we could definitely get the information over. But even my values need to contain some useful things to do value alignment. So, for example, if you follow up the causal reasons, why do I like little old ladies crossing the street safely? Why do I dislike kicking people in front of cars, now we might learn something about at least my morality that's actually quite useful.

 

Anders Sandberg:

There are two things here, of course, maybe one, shouldn't just use my own mind because I'm definitely morally pretty fluent. Maybe we actually want to make sure that either we take a certified same or perhaps even better, have a lot of different people who are in the right give input in order to do general value alignment. And second, of course, it's slightly tricky to get this to work really well because neural interfaces are tricky. But if I just say no robot, don't do that. Whereas if I smile at the robot, that's already sending a signal. Maybe I don't need the neural, interface at all. The neural interface is just cool because it might access my orbital frontal cortex and give some actual evaluation information even if I can't express it.

 

Randal Koene:

Yeah. These are all very good comments. And I liked that you said, okay, just even if you don't think about emulation and we're just talking about something earlier, like being able to understand your values, your value parameters that can be helpful. And of course I immediately started thinking about the deep learning networks that learned to be racist because they were learning from us. That's where you get into that whole value alignment issue and what are the right values, and should you even care about the right values? Are we supposed to tinker with that when you're dealing with particular person or invading your freedoms and stuff like that. It gets very complicated. But I also think people tend to jump straight to the assumption that we can talk about a system where you have a strong integration of the machine and the brain; where you have a connection that somehow is tying very closely into what this machine is doing.

 

Randal Koene:

And you mentioned the cell phone. And I think it's a perfect example for trying to think through what this actually means. Because when we say you have a high bandwidth connection between the brain and the computer, the computers still can operate at microseconds and even smaller timescales, whereas the neurons can't. So in this link they're talking at different speeds and that is very similar to what we're seeing with the cell phone and ourselves. With the cell phone, it has a different language, not just a different speed. It has a completely different type of language. It's using of a microprocessor.

 

Randal Koene:

...we have a neuro-brain and we work at a slower pace from neuron to neuron, although we have a lot of them. And we communicate in a very particular way with this cell phone. We can only press buttons or swipe, that sort of thing. So there's this communication delay or communication constraint built on this. And then you can think about, okay, so how has working with these cell phones, how has that affected the development of cell phones? This is sort of an analog for how does it affect the development of AI and potential AI safety, right? Or how has working with the cell phones affected us. Now we can say a lot about how it's affected us and, and some people think there are some good things there and some bad things there, probably a mix. But it also clearly has affected how cell phones are developed because cell phones are always developed to try to fit into say these communication limitations due to work well with how we interact as interface development, o present information to us in a certain way that makes it easily graspable.

 

Randal Koene:

We want interfaces that can tell us quickly what we're looking for. But of course that also constraints with features, right? The interfaces that are simple and fast or not interfaces that are highly configurable and very adaptable and all that. So we've got trade offs and this clearly has an effect. I think it's a great example and I think maybe thinking more about how we already interact with machines in that way, is is a way to try to imagine what this would do, building a better or faster or applied interface with machines in some sense.

 

Anders Sandberg:

So ideally of course, if you want an interface, it should be plug and play, it should work straight away. The problem is that probably means that it needs to be very pared down. And if you look at the general design philosophy coming out of apple, it has always been actually cut away possibilities and just leave the things that people actually will want to do. Hide as mush possible below the hood or maybe even make the hood the closed, you can't actually get onto it. Meanwhile, of course, if you're using any of the lineups, or UNIX systems everything is accessible, but well, reading the manual and learning how to use it, it's getting very complicated. Once you know it, it's very powerful. But that takes a long time and most people don't want to do that. So it might be that the transformative neural interfaces would be the ones that actually require a lot of learning.

 

Anders Sandberg:

So the most fascinating, neural interface I know of is of course Miguel Nicolelis attempts at brain that's connect brains to other brains. As a test of this idea, it seems like if this works, I want to see independent replication of it, but it looks like the brains can adapt to each other, and learn how to decode each other's signals somehow. That's awesome if it's correct. But it also seems to take a lot of time and I don't want to have a system that's not working for months and months, but yet I need to train myself to use it. So it might very well be that the neural interfaces with computers at first, they're going to be useful for people who are desperately in need of it, strongly motivated and the people who have the mindset of train themselves. But most applications are probably not going to be that deep.

 

Anders Sandberg:

So instead what you want to do is something that interprets enough of our signals and use a lot of smarts from the outside to try to figure out what we meant and the fact that our minds are relatively slow compared to the machines, it's not necessarily a bad thing. It might be that we have a slow process in, say, in fast processes and as long as we get enough feedback in our own pace, we can actually control that. This is a bit like our constitution governance state. The constitution is not changing that much. It's not supposed to be updatable at a very high pace. Below that level you have the laws which are updated in a faster pace while local regulation and norms can be updated faster and faster. So you might have systems with different levels of flexibility linked together. And if it's actually working well, then you can have a slow system setting the agenda, the fast systems implementing it. And so feedback system checking that everything is working. In practice...

 

Randal Koene:

Sorry, I hate to interrupt, but before you were talking about the degree to which someone could say that they are a part of the winning side. They're part of the whole thing, if they're integrated, if they're connected with machines. Now we're talking about a slow system and a fast system and that the slow system is somehow still governing and supervising what's going on. And it's interesting to compare that with what's going on in the brain already, where, for instance, we have systems that are slower. Our conscious awareness is definitely something that progresses at a different pace and in different chunks, then all of that subconscious processing that's going on in parallel throughout our cortex. And similarly, when we're trying to think something through logically, and we're trying to make up the steps, that's a very different and slower process than the other one.

 

Randal Koene:

And that in fact is the process that the machines can do very quickly. So it'd be interesting to see what happens there. But it's interesting to then look at how integrated those really are and what's really going on there. So for instance, we tend to think that our conscious awareness, that our conscious self, is in control. We tend to think that that's the part that is really the boss and we're governing everything that's going on. And yet you can show experimentally that you have already decided something long before you know that you've decided something because those sub-cortical processes and really doing that deciding work, of course, this makes sense, right? And the rest is just a reflection to yourself. It's just an observation about what's been going on in your brain. And now we can imagine that something similar is the case. If you integrate the human brain with the machines who are operating at a completely different speed, they live in a world of nanoseconds that we can't live in. They live in a world where they can use things like, quantum computers or simulated annealing or something like that to come up with ideas and solutions that we can't think of in our brain. It seems like one of those situations where ultimately you may have a feeling, a sense of control without actually having it.

 

Anders Sandberg:

So I think this is a very important point and generally I think most people say, yeah, I want to avoid. There is a...

 

Anders Sandberg:

My impression of the current state of play is that a fair number of the things we're doing are indeed illusionary. But our top level of the consciousness of mind, can give up the two and intervene not very quickly but we can...

 

Randal Koene:

Is it the veto, or is it other drives that are informing your consciousness to then give the veto?

 

Anders Sandberg:

That is not a good philosophical question. I need to check what the philosophers are saying about that, but I do think it makes sense that you have a top level but it's actually getting information from most of our systems. The systems I really have a problem with is the ones that never had an output on the conscious level or never tell the rest of the systems what we're doing. The real ignorant of secret societies of the grid. I don't know how many there are or whether they play an important role in general the brains job connected everything a little bit to average else. It's just about the degree of control and the degree of information is not as intense as one would expect. Once you started realizing how diffusely connected are different parts of our mind or you start to feeling a bit like a cloud rather than a person.

 

Randal Koene:

Yeah, the illusion of control and whether or not that's a bad thing, that there's an illusion of control. That's really I think is core to many of these questions that we're talking about today because when we first started talking we talked about AI risk and we talked a little bit about what do we imagine that if worst case scenario, and the best case scenario is. Now most of the time people tend to think that the worst case scenario has something to do with a situation where humans are completely not in control and where they have no say over their future and perhaps they are even somehow eliminated. Now when we're talking about situations where control is mostly an illusion, but there are aspects of it where we're tied in and say there's this veto power that again, perhaps isn't real because you know, freewill is such may not exist at all, but there's still something that we value. So really when you talk about the danger of AI and what it can do to humanity, we should be talking about what are the values that we're trying to preserve. What would we call a successful outcome? Which bits do we want to preserve and why? And then what sort of scenarios can incorporate that in some scenarios may include a lot of change where a lot of things that machines will be able to do and where they develop into becomes a part of what human society is. Right?

 

Anders Sandberg:

And so I'm reminded with some of I&M Bank's culture locals were very super intelligent machines play a role in the plot. And in many cases they manipulate protagonists in various ways, more or less often. And the interesting thing is that in many ways these manipulations can be in many cases they are fairly benign the, because there's still a respect that the protagonist can do differently, but he's unlikely to want to do it differently because it's a really good argument that the situation is contrived in a certain way. Would we say that this is a bad situation? Well, as long as it's actually a fairly well intention, the entity that actually has our own best interest and some other good goals, taking that into account, that might be all right. If I have a veto power but never exert it, I might still be pretty happy about that because I might have my illusion of control. But it still is important for me to function.

 

Anders Sandberg:

Normally we won't have a sense of control because we're agents. We survive by moving around on the earth surface and manipulative it, and when we can't act, we're super vulnerable. When we're in a situation where action is not possible, we are at the mercy of everything else. And that evolution has made us really desire to be the actor. Intellectually, we might sometimes realize that I should listen to a smarter person and do what he or she tells me because they actually know better. But my little inner two year old will say no, I would do the opposite just because. Then of course my, inner 20 year old will try to get the two year old to listen to reason and then there's a lot of debate and meanwhile, of course maybe a really smart person with manipulate me because we know that I have this internal dialogue. But a good outcome would be that you think that intentions of transparent enough that we can kind of see that these seem to point towards a future that is desire.

 

Randal Koene:

Oh Gosh, you just opened up whole new can of worms, where we find good and desirable. But I just wanted to point out that you now have also again come across something that Jann Taleen mentioned. He mentioned that he felt that a good future is one with as much optionality as possible, which of course is our desire for control. It's the desiring to have as many different paths as we can take as possible. But you mentioned that sense of the illusion of control and possibly being important to us. And then I was reminded of how many people say that what they desire is happiness. And then when we think of a future with happiness, then you get into these wire-heading ideas, where you can just make someone feel happy, even if there was really nothing has changed about their situation.

 

Randal Koene:

And in the same way it could be possible that we could give ourselves the illusion of control when we have zero control. So again, it doesn't immediately solve anything, but if we think through what the real values that we're trying to preserve are, you have to think about very basic things like the value of say, something that we consider really human continuing for some reason because we think that very human thing is important or has made the universe a better place than it was without humans in it. This gets really deep philosophically.

 

Anders Sandberg:

So it comes up in a lot of existential risk research too, because maybe if we nuke ourselves and the cockroaches becoming intelligent, maybe they will have a much better civilization then us. In that case, maybe we should say, oh, that was a happy ending for the universe. And of course some value theories would say, yeah, just integrate the total amount of happiness and compare them. Now others will say, wait a minute, how certain are we that happiness is what should the maximized? There might be other things, we might be deep thinking or paperclips or what have you. So we also have to take more of uncertainty into account. The nature of what is good about humans, our uniqueness, I think many people have an intuition about it. So there is a lot of things one can do here and right now I don't think we have any complete theory about what the good is. Obviously philosophers have been working on this for a few thousand years. Some progress has been modest. We, I think we had learned important things, but it's pretty clear that the list of things that could be the ultimate good or the list of arguments why that mabe this list will never be complete is awkwardly wide. But it's worth noticing that even if it takes something as simple as happiness, people are quite bad at identifying what makes them happy.

 

Anders Sandberg:

And even when they know this action would make me happy, they might quite often not take that one. So there is one of the scariest papers I ever read, The Trends in Cognitive Sciences from a few years back, which was just reviewing these results and to me somebody who is kind of politically liberal. This was horrifying because if people don't even do what makes them happy themselves, giving them the option of doing that, okay, now that doesn't necessarily work out that well. Now I still think that most forms of control actually have very bad effects too. But it shows that it's tricky. Our feelings of what makes life better might not correspond to actually making life better partially because our feelings are evolved things that made sense of African Savanna and actually the life we're living now, not to mention in the future might be so radically different, that we need to develop new kinds of feelings.

 

Randal Koene:

Yeah, that makes sense. Yeah. So we went down a really deep rabbit hole there for a moment, and I don't want this to go on for too long, but I'd still like to quickly touch on a few remaining questions about whole brain emulation and AI. So I would have to talk about the idea that whole brain emulation itself can be a danger to humanity. And so I don't know exactly what people mean by that when they say it, but I think...

 

Anders Sandberg:

I can give us some ideas. So I wrote a paper a few years back together with Peter Eckersley, who back then was with the electronic frontier foundation now is at the partnership for AI. So we were looking at could you get risk out to having brain emulation? I think the most primary one would be if you have software people who suddenly have a radical break with a lot of traditional views on human nature.

 

Anders Sandberg:

You will have some people who actually have religious reasons to say these are not true people. They would have potentially economicaly disruptive effects. If we look at the old ideas in the Robin Hansel's, the H of M that it seems like getting a brain emulation breakthrough might be dramatically disruptive and you get entities that are in some sense a natural other so you can very easily tell the story how you could get just a conflict where some people think brain emulated people are not people, they're evil and taking our jobs and so on, so you could get another natural conflict. That still I think not an argument against doing brain emulation but rather making sure we'll use improved tolerance. It also of course has the idea that if you get a brain emulation is going to be self improving so fast, but now you get some form of hard takeoff. Before you know it, you have all the problems of the normal hard takeoff in AI.

 

Randal Koene:

How do you see that possibility? By the way, how does a whole brain emulation do a fast takeoff?

 

Anders Sandberg:

I personally don't believe this is particularly likely. So the normal way of arguing it would be something like, well it's running on a very fast computer. The emulated person can do a lot of neuroscience experiments on her brain, that you can run a lot of copies that test out things to improve various capabilities. And before you know it, you have some kind of post human who might now really start doing weird things depending on their value system. I'm not too worried about one of the person doing this. But you could imagine a society... So at some point, I think it was cultural man who was arguing with Robin Hanson, well that brain emulation society he's describing in great vividness in this book, it's not going to last forever because people are going to update the and compete and change it to other things. So in real time it's not going to last very long. And they kind of agree that maybe two weeks. That's still a lot of history...

 

Randal Koene:

I don't think we have to get into that necessarily. I think Robin's book with beautiful, but I think it has a prediction for things that will happen. It has its limitations for instance, because he left out AI as being a part of it. So many of the things that he has whole brain emulations doing, are things that just a simple AI program could be doing and that changes a lot about that whole scenario.

 

Anders Sandberg:

And you can imagine that they develop the simple AI sooner or later in the brain emulated world and suddenly you get the economic transformation that might be just as disruptive for the emulated people as well as the biological people. That scenario would rather be that you get very rapid cultural evolution which might produce actually extremely different kinds of entities. Some people again would say, yeah, maybe we're going to be some weird post human and they don't care about us meat humans and that naturally leads to conflict. I think a more plausible risk is, yeah. This unleashes dramatic changes in our domain, but a lot of people will not have access to and will be too slow to react to. And that at the very least look like existential risk. And it might indeed be some form of existential risk that develops into something that lacks those values that are really humane.

 

Randal Koene:

This will get us back to the same problem again, which is what is extent existential risk? What do we call a bad outcome for humanity? So let's say that whole brain emulation leads to a new Cambrian explosion of different kinds of minds who develop themselves in various different ways, not just minds but bodies as well, et cetera. And some of them do better than others. So there's an evolutionary aspect to it and you can see change happening in a way that's a bit faster than it is today. And when I say a bit, I might just be understating that. But if that happens, the question still remains is that bad because maybe this is the best possible outcome for whatever we consider valuable about humanity because it's going to spread everywhere very quickly, even if it changes. So does speed matter? It doesn't matter if you go from a to z from by going Abcd all the way to z and doing that in steps or it just as good to jump from a to z right away. If the outcome is exactly the same. If let's say, just as an argument for just the question of does the speed of change matter, is it important that that rate is slow for some reason?

 

Anders Sandberg:

At least in the case of a parasite delta? Do people typically think that the speed does matter? If I change day by day to become a more mature person with different political views, et cetera. Okay. But I might be fine with that, but not that one day I wake up and my personality and political views have changed in something. Now it seems philosophical that there is something problematic going on there. Because I agree with you. Why should we care about those steps? And part of that is of course error correction. We might want to be able to go back to, or note this thing.

 

Anders Sandberg:

There should be time to realizing what that might be doing. Oh dear, I know what I was voting for. I should do to something change myself. But I think there is something interesting about that Cambrian explosion situation to me that sounds lovely. I think it would be great to have this open in the future where we get a lot of humanities and they are all drawing of course on originated from the same original species and might retain different aspects of us. However, you can also make a very scary scenario where there are things that we care about evolve of it. Nick Bostrom has this very disturbing paper about some post human futures that essentially devoid of value. So in one of them, the mindless outsources, basically people exist in an uploaded form and a or outsourcing more and more of the brains. And in the end there are no minds anymore.

 

Anders Sandberg:

It's just an economy growing endlessly, but there is nobody at home. So the interesting thing here is where does the value come from? And this is of course again the deep rabbit hole, but I think it's worth recognizing that rabbit hole and kind of orbit around it at a safe distance. Out of that as we normally do, we think, oh, the conscious and sentient seems to matter. Probably if consciousness is lost, then we've probably lost all of the battle. I'm willing to argue that maybe that's not all there is. Maybe there are unconscious systems that are valuable lives too. But that gets me into a big argument with the philosophy department. We can also say that some of our human nature might be valuable just because it is a human nature. We might not care about some famous paintings in a museum, but they're famous paintings that we should preserve because they're part of our process just as well as some of the horrors of our historical past.

 

Anders Sandberg:

We should never forget about them, not just because we want to learn not to do it again. But even that we think that if we were to forget about it, all the people that died in that particular horror would in some sense are loss of it. There is a fair bit of interesting arguments in philosophy about this. There is a book here by so Summer Schaeffler Why Worry About Future Generations, which I found very fond because he's taking a non-consequentialist view about why we should care about humanity's future. And I find that really weird because I'm typically just thinking about what are the consequences. And it points out that you can actually love humanity. We can actually see there are things about humanity and its nature, that we love works in all and the we want this to go on. So I think there's going to be a thick bundle in that black hole of what really makes life worth living but matters.

 

Anders Sandberg:

And it might be that we need to hedge our bets. We don't really know whether it is our human-ness or our consciousness or the fact that we originated on this planet or just that we experience happiness. Maybe we want to have it all we want that Cambrian explosion because it gives us a good chance of having.. I would love to have a future that has both the brain emulation people and biological people as an existential risk backup. If it was just one of them, I think we would actually be in big risk and we might have missed out on true values. It might turn out, that just like consciousness might have added a profound new value to the entire universe. That might be the next step as a conciousnes prime, which is even more important. It's unfortunate. No kind of matter in the current universe got that.

 

Anders Sandberg:

You need to be maybe a brain emulation or come quantum computer or whatever to actually experiancing his study. And then to get on the other side of this and say, yeah, back then of course those poor Precambrian humans...

 

Randal Koene:

They didn't have this thing. And it might not even be, it might not even be something like the next level of conscious. It could be something completely separate from that. Something that we also think it's super valuable but has nothing to do with consciousness.

 

Anders Sandberg:

Yeah. But to some degree, expanding the circle to encompass bigger, bigger sets seem to be a good idea. So one of my common amalies... ...they might say, well, given the capabilities of those post apes, they call them humans, they could get a lot of bananas. And the apes would say, yeah, that's great bananas are good. And indeed, we are really good at getting bananas. We can get more bananas than any ape could imagine by planting them and doing other things. It's just that we're getting constantly sidetracked from the pursuit of bananas and sex, by also doing philosophy and technology and politics and art and sports. And we would try to say to these apes, well actually you don't know what you're missing out on. Actually science is pretty awesome and post humans or these quantum consciousness, prime whatevers, they might say, Oh yes, philosophy and science they're so cute, there a very nice part of life. We of course got these other things that you can't even imagine that's okay, that we do like our bananas.

 

Anders Sandberg:

The thing is, if we retained some course and they might be somewhat silly, things like bananas that might also be a continuity that means we care about our future. I think one of the big fears people have is what you get this disruption. There is humans post humans and there is no similarity whatsoever. This is part of the fear we have of AGI because we might make something that's utterly dissimilar from us and we actually don't have any way of telling them about what values we have and the like. When it comes to brain emulation, I always felt that we are much closer. We at least starting out very similar. Then might go off in different directions, but I think we have a good start.

 

Randal Koene:

We start with whole brain emulation. We know that whatever you've got there begins with something that is very human. And so whatever these values are that we care about, they're probably contained in there somewhere. Although it'd be great if we can explore the value thing. Not right now, but if that gets explored further and, we can't make a prediction here. Are we going to solve the problem of what are the values we want to preserve in five years, 10 years? Where's our deadline here?

 

Anders Sandberg:

Yeah. In some sense, doing philosophy with a deadline. I think that's a new thing that Nick Mastroni draws. And I think it's important to realize that there are some deep philosophical problems that maybe we can't solve, but I think we should try to do practical progress on the handling them because they're actually becoming action relevant. And in some cases that progress might just be okay, we need to do more hedging because we will not figure it out before the time's up. But, well, that trade, the moral hedging are impossible things so they can be quite powerful.

 

Randal Koene:

Indeed. So I've kept you on this now for quite a long time so it's probably time for me to sort of bring it to a close. But before we do, and I have a concluding question for you, but even before I ask my concluding question, I wanted to know if you think that there was something important or relevant that we haven't touched upon, haven't covered that you think should be mentioned here.

 

Anders Sandberg:

I think the core issue I have is, we have been talking about many things that have been around the community for a long while. Some of it we have written a bit of papers about, but it's very little. I think there's more work to be done there. Some of that might be done by neuroscientists and philosophers. I think there's quite a lot of room for interesting joint work here. Trying to turn this a little bit more rigorous, maybe measure what does the flow of ideas between different sciences that's interesting on its own. You can make a good academic paper out of it, but it might also tell us a little bit about the safety issues. I think we need to work a lot more on, can we do safety for neuro-morphic computing. Right now a lot of safety work tends to assume is platonic computer's running a little black boxes on white boards. And that might be good to prove some things about what we might want to figure out other ways of doing AI safety and machine safety. So, there's lots of things to work on here.

 

Randal Koene:

All these questions about values and what values we're trying to preserve and therefore what a good outcome is; that's really part of that, isn't it?

 

Anders Sandberg:

Yeah. And some of that of course is traditional philosophy. There is even a branch of ethics axiology the philosophy of value itself. But I think also we have found that there are orbital frontal cortex does have value representations and we're actually figuring out non trivial things. Both from addiction research and the scanning of brains when we make choices about how we do evaluations and how we function. I think there's a lot of room to do empirical research here, but might actually give a good kick to the traditional philosophy to see it in new ways. I think we have seen a rekindling of interest in ethics in many parts of philosophy because of AI Safety research because it actually forces philosophers to try to formulate, can I make my ethical systems something that could run on a computer and most traditional systems are absolutely impossible and to the first to the final will have to acomodate computation deamnds. And then we can start asking, so what is a moral system that actually can run on a brain or a computer and that waters the philosophical import of that.

 

Randal Koene:

So you do think that at some point we will have answers to all these questions or do you think it's just going to be a situation where we become progressively more aware of what all the detailed questions are and we just kind of lay out all the questions and we sort of swim in them?

 

Anders Sandberg:

Well philosophy is interesting because many scientists try to advance, we try to move forward, but in philosophy tends to try to dig deeper instead. And sometimes that's the right choice. But in many practical case we want to move forward. I have in my, I'm writing a book about the longterm future and I actually right now I have a very loose calculation. Given that we haven't solved these problems yet. What's the likely time until philosophy solves them if we believe there is a Pareto distribution of problem difficulty and they ended up with something like maybe it's going to take 10 to the 20 philosopher years before we figure out what it's all about. That's alot. It might be faster if we have better computers, but it might turn out that actually it's going to be far in the future when finalism super-super-post-human kind of goes, Oh, that's kinda makes sense.

 

Randal Koene:

Isn't it interesting how the Hitchhiker's guide to the galaxy already predicted this? 42?

 

Anders Sandberg:

Yeah. Again, it made a very good point. You need to understand the question really well then the answer is typically quite easy. So one of the interesting things about this interplay between philosophy and science and practical engineering is that quite a lot of time you get these practical issues that forced you to refine your deep philosophical question. I think we're going to find eventually, that axiology probably get old because we actually do a lot of neuroscience and then that forces philosophers to think in a different way.

 

Randal Koene:

Okay, awesome. So concluding this my last question, what do you think that, I mean, this is again an impossibly complex question, but what do you think that humanity should be doing or how should it change how it approaches problems compared with what we're doing today to maximize our long-term chances for both survival and thriving as a society.

 

Anders Sandberg:

So I do think we need to find ways of aiming at an open ended future, but that doesn't necessarily mean everything is possible. We might want to cut off disastrous possibilities. We want to avoid existential risk. And in order to do that, we probably need better tools for insight and coordination. So right now it might seem that we are in this world of fake news and international chaos. We're doing really badly about it, but compared to a century ago or two centuries, the United Nations is a really hopeless organization. But two centuries ago it was unthinkable what getting all nations together without weapons and actually agreeing on at least a few things in 200 years we might actually be able to do something much better. Similarly, yeah, we have a lot of problems with fake news, but we also got things like Wikipedia, which is kind of an early stab at creating a common knowledge base.

 

Anders Sandberg:

Again, that was the first step. And this is probably not going to be the last, or the best. So I do think that we can work quite hard on improving this insight and coordination of things. Better filtering mechanisms of separating true and false information. Which includes of course science, the replication crisis in a specialist iconic, demonstrates that we need to work quite a bit on building a better engine of scientific knowledge. But even the problems we have in the data mine, the scientific papers and tried to run that to do medical diagnosis turns out into trouble because many papers are so bad. We need better ways of doing this and those tools are useful in a lot of domains. We might want to have a better chance of tracking our own future. We might want to find ways of having these courses at work better.

 

Anders Sandberg:

It's a gradual thing and there's going to be alot of struggle to get there. Similarly, the coordination in the long run, we need to coordinate before we start scattering across the universe have become too incomprehensible for each other. And too far course removed before that we might actually need to have a big meeting and decide, okay, this is what we're going to do longterm with the universe. This is how we go to hedge our moral bet. This is how we're going to move the galaxies and making sure that it stays in the right.

 

Randal Koene:

Is that a meeting the FHI is going to have in 2021?

 

Anders Sandberg:

Let's hope for something like that. So the origional ideas Will Macaskill talks about the long reflection, but maybe once we got to interact together, we should just sit down and as a species and maybe spend 1000 years debating what to do just to get it really right.

 

Anders Sandberg:

Right now, this sounds absurdly utopia. This sounds totally crazy. But then again, so did the United Nations once upon a time. So we should start working on making the tools to make the tools to make the tools to make good insights and good decision making. And of course, making sure that we don't go extinct while doing this. We should never let the perfect be the enemy of the good.

 

Randal Koene:

That can be difficult sometimes. On a more personal note. Given all of the things we just talked about and there's so much more of course. How do you deal with the fact that this is such an overwhelming pile of stuff? I mean, how do you tackle that or just like go at it and still feel that there is good purpose and moving forward every day.

 

Anders Sandberg:

So the problem is not finding purpose because there are so many interesting and important things to work on. It's rather how do I choose the most important thing to focus on? I'm lousy at that. I'm basically an academic magpie. I see something shiny and I go for it. So over the past one and a half years, I've been somewhat focused because I'd be working on my book about what I called Grand Futures. And it's broad enough that it actually allows me to, whenever I get distracted, I just write another part of the book. That is one way of doing it. I have colleagues who are way more focused. They sit down and they think very careful about what matters. Spend even more times checking that it really matters and then they start working on it and I think we can again like hedging bets you hedge yourself by having people give the different strategies and then sometimes we compare notes and see, oh, that seems to be something I should be doing.

 

Randal Koene:

Okay. That's really good advice. Yeah, there are some different strategies there. Okay, fantastic. Thank you very much. Fantastic conversation. I have to say. Yeah, like always.

 

Anders Sandberg:

Well we should keep it up, make sure that the future continues to have great conversations until the end of time.

 

Randal Koene:

Oh yeah. And you'll be welcomed back at the Carboncopies Foundation again and again for sure.

 

Anders Sandberg:

Thank you so much. Yeah. Cheers!

 

Randal Koene:

Okay. Stop the recording now.

 

Randal Koene:

Not hearing you at the moment. I don't know if you can hear me.

 

Allen Sulzen:

You are audible, Randall Mallory is not,

 

Randal Koene:

That was really awesome to hear Anders talk again. He's always a fantastic speaker. No matter what the context. I wish we had a way to really do claps online. Maybe we need a canned clapping or something. And then whenever appropriate we'll turn that down to a slow clap or something.

 

Allen Sulzen:

We're downloading a clap track now.

 

Randal Koene:

Okay.

 

Allen Sulzen:

I'm just checking.

 

Randal Koene:

Okay. Maybe then Mallory has a more serious problem with her mic at the moment. So many technical things because everything depends on so many little bits and pieces. Is the Bluetooth working, are the batteries still charged and all that sort of stuff. And in my case, the Internet drops out every once in a while, so that's not great either. We have heard from Anders recently about 27 minutes ago, he said that he was 30 minutes out, so maybe he'll be here in three minutes if we're lucky or maybe it takes a little bit longer to get off of his bus and get into his house and get all set up. That's all possible. So in the meantime, we can start addressing some of the questions that came in a while... Oh, here's Anders. Then we don't even need a meantime. We can just go straight at it. Hello Andrers. You must have just come in the door and sat down basically, right?

 

Anders Sandberg:

Yup. Literally. Yup. Just ran in from the bus, then got home.

 

Randal Koene:

Okay, well Kudos to you for being able to do that. We all just listened to the interview and as usual you're fantastic. The way you present everything is so clear because unlike me, where I go into abstracts very quickly, you tend to bring everything back down to examples that people can really get into and dig into. Like, AI that is helping a little old ladies across the street and things like that. So that's perfect and I think we also got a bunch of good questions here and I see Mallory just poped up again.

 

Mallory Tackett:

So where did we leave off after I joined? Are we going to start with our first question?

 

Randal Koene:

You can go ahead with whichever question you want. Anders is here.

 

Mallory Tackett:

All right. And just to remind everybody, you can ask your questions in the YouTube live stream chat or you can call into call.carboncopies.org or the number (415) 455-3017. The first question that we have is from Roman Sitlou, even today there are some people who are both capable of recursive self improvement, like adopting the new mind enhancing technology and have strong misanthropic tendencies. Basically in a way their biological non-friendly AGIs what can we do about this scenario?

 

Anders Sandberg:

I think we actually have a surprising amount of good tools at our disposal. After all, one reason we behave ourselves is because our mothers told us various things that we should be doing and that's a foremost social software. We have other forms of social software we get when the get enculturated as well as police reputation various force of market solutions and many of these things constrain us to a tremendous degree. Then of course if you try to do recursive self improvement through enhancement today, it's quite limited. No amount of smart drugs or meditation and or getting the best apps are going to make you tremendously more effective than just somebody who is really smart or specialized. And you're not going to exactly be deep blue either without having a computer on your side.

 

Anders Sandberg:

But the deep part of the question is how do we handle when it becomes easier to make our full individual minds? And I think in general we do that by having a lot of minds, but they partialy constrain each other and systems of minds because generaly systems of minds are much more powerful than individuals. So to just take a little parable here. We have the earliest youth conciousness interesting AI Boxing experiments. Where we practically demonstrate actually convincing people to let out the possibly on friendly AI out of a box was pretty doable. If you're smart enough and Glib enough, you can talk your way out. This is impressive except why don't prisoners do this all the time? Why don't we see more prisoners talking the way out of prison? And the answer is of course, well quite often we talk their way past one or the prison wards, but there are others, there is a system or a career. So being prison chief and the warden, that all dependand on prisoners not talking their way out too much. And that actually makes actual prisons surprisingly effective in keeping even the most clever prisoners in. Not perfect, but quite well. So I do think we can do the same kind of system and system management. It's not like we want to only upload the saints, so we don't even have a good definition of sainthood, and maybe the saints won't stay saints once they're in the computer either. But you don't want to have just one mind.

 

Mallory Tackett:

Great. Does anyone want to add to that or shall we move onto the next question?

 

Randal Koene:

I think Anders pretty much answered the whole question unless you wanted to dig into the detailing that Roman did when he pointed out, people who want to self improve may already feel that they are perfectly fine and eliminating a huge part of humanity in the future or something like that. If you could restate that properly.

 

Mallory Tackett:

Or Mallory, sorry, maybe I didn't make myself clear. I thought you could maybe...

 

Anders Sandberg:

Or I could just get at what I think that question in emboldens. So really ruthless sociopaths and people think that the world would be better off with just me around certainly psychological exist. The vast majority of them are not very good at doing anything because by virtue of being so misanthropic and sociopathic may rarely can acquire the resources. I think the intuition we have is, oh, what if they take a technological means, that pill that just makes you essentially god like, at that point we will be in trouble if it goes there. The problem is they're probably not first in the queue. The real problem might of course still be the scenario where you have successful sociopaths, successful misanthropes or who just ran into a lot of resources and they will be getting power. But again, the question is why aren't particular billionaires wiping out everybody else and running the kingdom?

 

Anders Sandberg:

Well, to some degree they still need other people. But even there you can actually replace people with consultancies, et cetera. The actual problem is of course societies in order to remain rich and functional are quite complex. You can't actually run all the parts. So the people who just introduce in every wealth and power, they are relatively sane and my scorch society in other ways. So the only case that we need to be worried about is really misanthropic person who gets very powerful. But I do think the scaling of power isn't in favor of us. Once upon a time Metachip family, they were wealthier than essentially the principality they were run. They were rivaling the nations in European of wealth and power, but meanwhile they weren't perfect. But the United States is, harder for rich people order organizations or people like state, actualy it has become weaker. So I'm much more worried about misanthropic and crazy states and the institutions then individual misanthropic people.

 

Mallory Tackett:

I think that answers the question pretty well. The next question that we have is from Jaysa RC. What do you think about the human habit of comparing ourselves to the technology we create for ourselves or our tendency to humanize things even if they're a robot or an artificial intelligence?

 

Anders Sandberg:

Yeah, so one of the main reasons people make robots, is of course they actually like to use humanize technology. If you think about much of the Japanese mindset about robots, it's important to have a social relationship to technology. And certainly they also believe that we have social relationships even to non-human technology. But if it has a face and it's easy to interact with it's more manageable. And I think that this gets to something important. We humans have... This sometimes leads us to think about things in the wrong way. When lightning strikes, we wonder who is angry and come up with a story about somebody must be angry with somebody or some reasons to course lightning. But on the other hand we use this to do intuitive physics.

 

Anders Sandberg:

So it's very natural for us to try to make these kind of machines and put in a lot of anthropology modification, into them. And we also of course tend to assume that if something with a face or talks back to us must be part of the human world. Which is why it's so easy to have a chat bot. And this is why people fail in the Turing test all the time. But then there's opposite thing. By trying to make artificial intelligence, it's also a mirror. It's a great way of trying to understand ourselves, what we are, what we mean by thinking and feeling and what aspects we care about, but also what we fear a lot of worries about artificial intelligence or our own projections. But what if we made something like us? And I think that's the first level of worry. I'm not so concerned about that myself because I think the second that being told them there alien and hence conspiracy theorists. But you should look miss the fact that they are deeply human.

 

Mallory Tackett:

Great. Does anyone want to add to that?

 

Randal Koene:

Yeah, just a quick note, which is this point that you made at the end about more worried about things that are more alien. This kind of comes back to that bit about, worrying about corporations or something like that that are very powerful because in a sense, these are also alien because they have different way of making decisions. They're way more focused on what we might call utility function, which is the bottom line and the quarterly results and that sort of thing. So they can be very sociopathic in that sense and very powerful at the same time. So I think you're absolutely right to focus there first before worrying about the individual person in a sense. So yeah, that kind of jumped back, but I guess that's okay.

 

Randal Koene:

Oh, quickly before someone else answers something. I just want to point out that while there are lots of good questions coming in and there's way we're going to be able to address every question in the time we have on the panel. We are collecting all these questions and as we see good questions there, we're still going to work on them later on. You may find them in transcripts or another work that we do. You can still contact us about them through questions@carboncopies.org and we can get back to you.

 

Mallory Tackett:

I think I can move on to the next question now. Our next question is from Jan Clok or Yon Clok. and he says, "While initial collaboration may be nice, competition like the US versus China," and I'm assuming he's referring to our competition with artificial intelligence. That's probably the most prominent one, "is that competition is going to stay? How do we utilize that for whole brain emulation and artificial general intelligence development and keep it safe?"

 

Anders Sandberg:

So the first thing worth mentioning is of course many of assumptions that we are, let's say US versus China in a competitive race might actually be because humans like to project their human moods and feelings onto these alien things like nations. We are fairly competitive and you can totally think of international politics as kind of the school yard bully going around. And we quite often construct it that way because it allows us to think about it in the easy ways and of course a lot of people involved actually like to think about. It's just that in practice governments or disjointed entities with different goals and different parts of government. And you can imagine for example a competitive situation on the artificial inteligence in China and the US, that involves total sharing of safety information and I think that is something we should be pushing very strongly for.

 

Anders Sandberg:

Even if people think that this is a strategic technology and it might even be important to be first, it should also be a no brainer that it's also important to have a world to be first in. So maybe we should be also shairing as much as possible on the safety technology. This actually happened during the Cold War between the US and the Soviet Union, where the US deliberately shared some technologies for keeping nuclear weapons safer, simply because that would be in the interest of everybody to have it. It didn't always work out as diplomatic as it should, but it was definitely a good try. When you want to use competition to do something good, typically it is this concept of creative destruction or let a thousand flowers bloom and so on that you allow a lot of diversity and then that competition will hopefully bring forward interesting things that you can note is interesting and useful that can be replicated.

 

Anders Sandberg:

And this is of course typically why market economists do so well when compared to a planed economy because they can find new solutions rather than trying to optimize for an obvious one. But when it comes to research, this is extra important because in most fields the amount of progress you make is a very complex curve of amount of efforts you put in. So in the earlier era, you need to get a lot of rapid progress, but you don't know what you need to work on. You need to map out the field, you need to do a lot of experimentation. But you can also learn a lot of things. Eventually it's time to bring out the big super computers and the big collaborations. But first we need to solve a local small step. And that's a great thing to actually have dispersed between not just different teams, but also different mindsets and approaches. If everybody tries to achieve brain emulation or AI in the exact same way, the probability of it succeeding goes down quite a bit, which may be good for safety work in the background. But by the same token, we want to have a pretty big diversity of safety mechanisms explored before we notice which one seems promising enough that we should be starting to pile on and boost to their abilities.

 

Randal Koene:

Wow. I think first of all, hat it's worth pointing out that what the person who was asking the question is talking about, it may also be something a little deeper. It could be just, not specifically about how people working on AI should work? Should they compete or should they cooperate? But also just this fact that the competition is sort of a natural thing that just emerges from there being differences between things and then the selection mechanisms that apply. So you see that in evolution too, of course. And in evolution you have differences in DNA through some genetic mutation and then one of them works out better than the other. And that's a kind of competition, even if it wasn't intended to be a competition. And the same could be true, you could say that even for a single AI that is trying to improve itself as it's trying out different algorithms, those algorithms are competing on some kind of performance criteria.

 

Randal Koene:

So there's a competition going on there. So you're always going to have within the ecosystem of all the things that are going on, you're going to have both competition and cooperation because cooperation is often a strategy that helps, that enhances what any single thing can accomplish. They, for instance, the cooperation between different algorithms to have two different ways to detect people in a scene instead of just one way to detect people in a scene. So you've got both collaboration and cooperation going on at both times. And then what Yan was asking. It could be interpreted as, since there's always going to be both collaboration and cooperation and it's really hard to weed out and to say we should all be working collaboratively on the same AI here or something like that. How can you ever make sure that's going to be safe? Because any actor who happens to be slightly less cooperative may get an edge because of that. Right. And I was going to go further with this, but I think I've already brought enough new stuff in here. Anders, do you want to say something about that?

 

Anders Sandberg:

I'm reminded of something that Eric Drexler rote somewhere in one of his old papers, I think it was his algorics papers that he wrote somewhere in the late eighties, and he pointed out that we often talk about how wonderful it is in nature with all this simbyosis and all of his harmonious cycles. While human business, oh, it's a red in tooth and claw. It's all about competition. But he pointed out that actually the reason that we hear about the crimes and when people betray each other in politics or in business is that it's rare. It's very unlikely that you get robbed by gunpoint by your grocer. Most of the time you will just to get groceries and pay for them. Most economic transactions or extremely cooperative. We tend to notice a difference. Similarly nature, actually an awful lot of interactions are fairly adversary. And then we get very in the rosy about all these wonderful cooperation we notice.

 

Anders Sandberg:

So many systems and different styles and biases. Now the reason human economies work so well from a comparative standpoint, is that we have a lot of tools to enable us to do that. It's much easier to negotiate if you've got the brain and the language. It's way harder for the different plants and animals in the forest and find some nice equilibrium. I do think the real question is can you then set up the rules or the agent so we naturally generally tend to be cold. So the rewards in academia reward you for trying to be first with a cool paper that the, it's a competitive thing, but you also need to tell everybody how you did it and everybody gets to comment on it and that you actually develop this cooperative, effective. True it enduces some sense competition, but it also helps truthful and useful papers get ahead. And that we might say, well maybe we should set up the incentives for AI and brain emulation research so we do this.

 

Anders Sandberg:

Right now it might be that a brain emulation research is mostly an academic pursuit subject to the same quirks, but normal academic competition and the corporation are, for example, a lot of the competition is relatively weak, but you also don't have strong incentives actually get something that works because you can always write more papers about approaching something that works well in the industry. Making something that works, it's actually a really good idea because you make money, otherwise your company's not going to be around. So we might want to investigate this more deeply. I'm not a good enough economist, but I think there's this area of mechanism construction that people have done in game theory that would be very valuable here to try to see could we find some new ways of setting up corporate and collaborations or competitive races.

 

Anders Sandberg:

The Human Genome Project after all was leisurely developing technology and getting somewhere, but at a slow pace and then Ann Craig went there, stormed in and forced everybody to rush because of a competitive impulse but also actually pushed things to build in that phase where you want to get results. And then we took the genomes and most of the genomes are not put in public databases and shared etc. The real competitive thing is figuring out the meanings of them. So you can also have layers of competition and collaboration and we might want to look at at what layers do we think is the biggest risk of a bad accident would be a headache. Let's see if we can make sure that one is strongly in their corporate team, which of course includes also, yeah. Other people thinking about, oh, that guy is trying to mess up. And the good thing we got going here, we're going to have to ally against, which is another decent incenive for being alies.

 

Randal Koene:

There's an interesting way that this bridges way back to almost the beginning of the workshop where one of our callers Leslie Seymour was asking about security protocols and we kind of got to the point where there was some agreement around, yes there's a reason why everyone in the world coalesces around open standards and around, using the same open standards and protocols. Because that way you get a lot of people looking at it, making sure that all the holes get patched and all that sort of thing. But then at the same time, you were pointing out that it can be really useful, especially in these sorts of cases to have a lot of different approaches. Try out a lot of different things and come up with new, better approaches than used to be there. You have to break out of the box from time to time. And so it begins to seem like it's very handy to have competition inside a sandbox where you have your peer review and then ultimately to have collaboration and cooperation and standards outside of the sandbox whenever you can accomplish that.

 

Anders Sandberg:

Yeah. I completely agree and being good at designing sandboxes. I think that is a fantastic thing to work towards.

 

Randal Koene:

So Mallory, did you want me to go ahead and ask a question that I had written down or did you want to go with another one of those? I see that you've got another by Abulfazl for instance, and there's one by Justice.

 

Mallory Tackett:

Yes. So we do have one from Justice. I'm going to ask that one. "If brain emulations were developed as open source software and posted on the Internet, ala Bitcoin. What hope is there to effectively control them?"

 

Anders Sandberg:

That depends very much on what resources it takes to run it. So you might look at the current debate about open sourcing AI tools. So if Google amounts a very nice AI algorithm or at least a machine learning algorithm today, in many ways they're not losing much of a competitive advantage because you typically need a Google data center and an enormous amount of data to train it. So even though we amatures and academics can sit around and read algorithms and try to tweak something, we can never turn it into a production system. It wouldn't surprise me at all. But the actual code for running a brain emulation, is going to be a somewhat messy computation neuroscience system, whether it's with a virtual reality system and a lot of data management, nothing too weird. Everybody can download it from Git Hub. It might also be that we can download a few preacher in neural network for from some mice, some monkey and maybe some human volunteer who is very willing to be everywhere. But that doesn't mean that you get the full power of being able to scan my brain for example because scanning a brain is going to require hardware of some kind and quite often also the knowledge on how to use that hardware, which is sometimes shockingly tricky to transfer.

 

Anders Sandberg:

Anybody who's been hanging around the experimental biology has kind of noticed that they're very skills that some people seem to have and others don't and generally of course in science we try to figure out ways of automating that so we get rid of it. But there is still always, especially the early days of MIA Technology, a lot of requirements there. I do think that open source is useful for checking code and improving code, but it doesn't give you the capability necessarily to round it. If we use it to its fullest, but this might depend of course quite a lot on which scenario we have. If we end up with a scenario where the code is relatively easy but getting computer power hard, you get monopolies of computer power and it might also be that we start up with small simulations and get better and better, but that means that many people can tool around with it a little bit crude simulation on the computer we get the better societal debate.

 

Anders Sandberg:

On the other hand, if we have more overhang scenario where the code has been around forever and then finally somebody does a bug fix and now it works with those pre in and scan, brains that have been staying around in Git Hub for decades, then things get really weird. Then you have not just an overhanging maybe of hardware and the scans, but also the overhang off the accessible code. I don't think it's necessarily that likely, but it's worth looking into. Another thing that Peter did in our paper Risks of Brain Emulation was to worry about computer security aspects. If brain emulations are bad at security. If it's easy to hack your server running various minds, then we have an enormous problem. So we might want to think about how do we make that code so it's really robust. This is why we might want security on the code and make sure it's actually quite open source and people have been spending a lot of time finding the flaws.

 

Mallory Tackett:

All right, thank you for that. Next we have a question from actually one of our panelists. Abulfazl. He asks, can you elaborate on the idea that uploaded minds or whole brain emulations are not as dangerous as AGI or artificial general intelligence is if that's the case, why is that?

 

Anders Sandberg:

So this is still the intuition which is based partially on that distinction between what about human like versus alien entities. So there's this cliche idea about the robot uprising as the robot fields oppressed, they rise up against the oppressors, the humans. And that's after all what the human would do in the same situation. Feeling oppressed and desiring freedom is a very human thing. Which is why humans love reading stories about robot rebellion. Of course, if you actually get that kind of robot rebelion your have basically made humans tin cans and first of all you probably should give them rights and second you have succeeded very well in making a relatively safe AI because you can probably reason with them. You could imagine however the serious problems that happened when you have something much more alien, and this is of course the line of reasoning that has emerged from the work of Aleas Ukofski and Nick, but actually in general if you try to optimize for something, you willoptimize for that thing and ignore most other parameters.

 

Anders Sandberg:

This is a favorite argument by Stuart Russell, but if you make very intelligent systems, so the powerful optimizers and tell them to make one thing, all the other values in the world, even if they are somewhat similar to our mind will fall to the wayside. This makes him quite dangerous. And then you can elaborate of course in all sorts of interesting cases of how this might be that is driving a fair bit of the research on how to make safe super intelegence. Now the intuition is of course if you managed to make a brain emulation you basically got the original person. If everything worked out well with the same quirks and flaws and moral failings, and moral virtues that's original. From one the hand that mind might not be saintly but it's certainly not going to turn the universe into paperclips just because somebody asks them to make paperclips. After awhile you can say, I'm tired of this. I want to develop my personal capacity or I want to sleep. Or actually I really think now a robot rebellion would be suitable. You're basically having to deal with a human. We can certainly channel humans to build organizations that are scary and dangerous, but it seems like we don't get those inhuman falures. We don't get the papercliper that just mindlessly constructs paperclips in a paper clipping system. We don't get systems optimized for something totally valueless. Even if you enhance a human a lot, so they become more of a human, and I think has gone really wrong with some human values and goals. But there is this general idea in the study of the Ai Safety. Human values are fairly complex and fragile, so it's not enough to get most of the values in because if we've got about something on the list, whether that is forgiveness or love or vocational nature, is nice you get the world that gets optimized for everything but important things and it's actually quite dystopia.

 

Anders Sandberg:

Now the question is, is this argument right? Does this actually mean that brain emulations are safer than AGI and that is of course where people's views go in different directions. It might be what leads to relatively quickly to normal AI because we learn clever tools from neuroscience. It might be that brain emulations could rapidly evolve into something else. It might also be that our tools for keeping humans in line work really well for them. So that might benefit safety, but it might also be that we can prove safetyin other ways for more purely artificial system, we can morally do things to their minds and the science that you could not really morally do with brain emulation. So in that case, maybe you want to prefer it. So in the end, I think we need to actually write papers and analyze with more detail. I have, my probrain emulation view, but I'm not certain it's right. We need to investigate this much more deeply and find ways of formalizing these arguments.

 

Randal Koene:

Okay. I don't know if anyone has a comeback right away. But otherwise I can go ahead and ask my question, which I think relates to this in a sense. And it relates in the way that, well you just described, for instance, an example of why whole brain emulations may be safer because you're basically dealing with a person. They're not going to want to paperclip the universe because they want to do something else. He'll get bored after a while. But then I could come up with a scenario where I say, well, what if, what if, this is in Robin Hanson is universe, you're making whole brain emulations as your basic tool and you want to paperclip very well. And so you may change, or you may have gotten permission from the original owner of that brain, let's say, to make a little change so that the whole brain emulation no longer has that much of volition of their own and might just want to carry on making paperclips and all of a sudden you can come up with a dangerous scenario. And this leads into my question, which is really when we talk like this, it seems pretty easy, and I've heard this happen all the time from everyone who talks about AI safety basically, whether it's Nick Bostrom or Elon Musk or Eliezer or anyone, it seems very easy for every proposed system, security or otherwise every scheme to come up with some totally theoretical scenario where this breaks down and that makes it seem like AI safety is possibly a lost cause because no matter how you look at it, you know there's always going to be some system that escapes. And that's the end. So in other words, it seems like the probability of a given scenario seems very important to assess. And how would you do that?

 

Anders Sandberg:

That is a very good point. And I totally agree. Trying to kind of get perfection never works. In actual computer security and actual security engineering people think about swiss cheese solutions. You have a number of layers of security but they have holes in them like swiss cheese. But these are not aligned. The probability of something getting through holes in all the layers is low enough. And what can of course sort of arguing here about how low probability of a disaster is acceptable. I think that's an important guide point. But we need... So some risks I think are easier to formalize than others. So when you think about an artificial intelligence, you might be able to prove things about that if it's a really nicely well defined one. And at least put some bounds if you have a nice theory, let's say of machine learning reinforcement learning or what it's doing, that might give some possibilities to bounding it.

 

Anders Sandberg:

But it still sounds to me very much like what we would love to do with the computer science department and maybe it wouldn't actually correspond to anything realistic in terms of actual safety checking. So we might want to think here about safety testing, methods. There is a fair bit of that for actual software and the actual industrial systems. They're not perfect, but we might want to actually go and loot that literature for tools and start applying them to our own thinking. My general feeling is this is going to work well for technical stuff. I think we can do code audits and estimation of likelihood of certain neuron networks doing certain things, but on the other hand you have the human side. What's the probability of somebody agreeing to have a brain edited by weird actors? That's probably pretty high. I don't think we need to even try to estimate that there's always somebody, but what's the probability of us actors doing certain risky things?

 

Anders Sandberg:

Now we have a much harder time. There I think we need to apply a fair bit of judgment and essentially world knowledge. It's going to be way more uncertain. I think we have still some evidence that we can do not to bad probability estimates in some domains when we get feedback, you have Tetlock superforecasters for example they're interested in policy and world events. They're reading the newspaper, they're discussing with each other and if you put them together in the right way, you actually get surprisingly good for cost compared to the average one. Could we do the same thing as super safety forecasting? I'd never seen anything about it, but it might be actually worth pursuing.

 

Randal Koene:

Thank you.

 

Mallory Tackett:

So that concludes our Q and A section for Doctor Sanberg. Thank you so much for joining us.

 

Anders Sandberg:

Thank you.

 

Mallory Tackett:

Next we're going to be showing remarks that were prepared by Dr Ben Goertzel for this workshop. He has probably one of the most extensive resumes in research toward artificial general intelligence. Among other things he is the chief scientist at Hanson robotics, the chairman at the Open Cog Foundation and the chairman of the AGI conference series. So now I will show his remarks.

 

Randal Koene:

Just one quick thing. Anders, thank you so much for joining us. I realize that you're probably very tired from your flight, so I can understand if you can't make it all the way through, but if you can sort of glimpse at the screen from time to time and might still be around after Ben's contribution, that could be interesting because he sometimes takes kind of a opposite or contrarian view from...

 

Anders Sandberg:

Always. And that's the fun thing about him. I'm going to try to stay around. Cool.

 

Mallory Tackett:

All right. I'll get that started.

 

Ben Goertzel:

... my good friend Randal Koene was organizing a workshop on mind uploading, AGI, and brain-computer interface, all that good stuff.

 

Ben Goertzel:

I really wanted to participate, but...

 

Mallory Tackett:

Can anybody hear me now?

 

Randal Koene:

Yeah, we can hear you. We just lost the audio from the video for a second there.

 

Mallory Tackett:

Okay. Was it off for the whole video?

 

Randal Koene:

Most of it, yeah. Only the very beginning we heard him.

 

Mallory Tackett:

Okay, I'll go ahead and restart it then.

 

Ben Goertzel:

I heard that my good friend Randal Koene was organizing a workshop on mind uploading, AGI, and brain-computer interface, all that good stuff. I really wanted to participate, but being based in Hong Kong, it wasn't feasible for me to show up in person. So ask around, go to... Send me some of the key questions that he was interested in exploring in the event. I'd given a little bit of a video improvisation on the theme of his questions. Now, some of the questions Randal sent would take approximately 10,000 years to really go through and answer in detail. So I'm going to give some of them the short shrift, but better something than nothing. So here goes. First question from Randall, "Could you tell us a little bit about how your thoughts on AI safety have evolved over time and where you stand today?" Well, my thoughts on AI safety at base is the same as they've always been. I think there's a nonzero risk that as AI verges on AGI and artificial superintelligence, things that are very bad by our current human standards will happen.

 

Ben Goertzel:

I don't intuitively, emotionally feel the risk is extremely high. On the other hand, rationally I have to accept that we're in a situation of tremendous and probably irreducible uncertainty. We're taking a leap into the unknown. And that's not unlike what humanity has been doing since we stepped out in the African savanna and started developing civilization. We've been taking a huge leap into the unknown one time after the other since civilization began and probably before that. So, I guess for me the question is really how much do I trust my sort of inner spiritual, heart based intuition that the singularity is almost surely going to come out okay; and is in fact going to connect us with compassionate, benevolent aspects of the universe that we're currently, largely cut off from due to our mentalities. How much do I trust that intuition versus the more cold objective reasoning part of my mind which tells me we have no idea what the hell is going to happen. This is really the delemma that is realy on goingly wrestling with. And then maybe that process, that dialectic is it valuable, because certainly I wouldn't want to go entirely in the direction of following only my heart and not reasoning or entirely in the direction of just reasoning and not going with my intuition, which, can have a deeper insight than reasoning.

 

Ben Goertzel:

I would say one thing where my thoughts on AI safety have evolved in the last few years though is I'm getting a more concrete sense of what there is to be worried about regarding the rollout of nowhow throughout the world before we get to AGI. And I've been thinking more about what effect the species' knowhow that gets developed can have the type of AGI that comes about as knowhow verages into AGI. So specifically as I've been saying a lot recently that the core applications of knowhow in the world right now are selling, spying, and killing. I mean advertising, surveillance, and millitary. And if it does happen when the first AGIs evolve out of the knowhow as it would be built in the world today. What does that mean? Does it mean the first AGI is going to be involved with selling, spying and killing? I mean not necessarily, but that's at least something we want to consider. There is a related issue which has to do with the control of knowhow. And then they have to do with the control of AGI and it comes out of the narrow AI, which is how widespread, democratic, and participatory should the controlling of AI be versus how centralized and elitest should it be? And there's been alot of thinking in the world of safety of AI for a long time that it would be safer if no small handpicked crew of wise and rational AI safety gurus we're controlling the advent of AI as it turned from narrow AI to AGI.

 

Ben Goertzel:

There's another line of thinking which says that self appointed elites know what's best for everyone, often don't do as good a job as they thought they were going to do. And the failure modes of this are amply demonstrated throughout history. One of the good guys discovers a dark side within himself. The elites split into two groups, someone gets stolen away by competition. Humans who band together who no what's best for everyone, then pull the puppet strings of the brothers society... The track record isn't, isn't great. Right? And of course what's been tossing by some folks in the AI safety world was like an expert committee of wise singularitarians who were building an AGI in the basement and sculpting it's goal system to be beneficial and then releasing it in the world. What we're seeing now is more a move towards eliteist control of AI by some advertising corporations, and some large governments doing surveillance and military stuff. So we're getting this eliteist control, however the controllers are not who some of the elitist AI safety advocates might have wanted. Right? And to my mind, well of course a part of me can't help but think, well yeah, if I were just the one in control, if me and ten or twenty or 50 or my closest friends we're chugging all this out... We want the best for humanity and for a super-human AGIs. Right? And we've surveyed all the relevant areas of history and science and engineering and philosophy and we can probably make a better choice. Then the whole of humanity, which includes a whole lot of ideas that I think are totally whacked out. But then if I take a little bit deeper point of view, in the end, I don't think any small group of people is going to do a better job than the global brain of public humanity. And there are kinds of understanding and wisdom on the planet I've never heard of and never imagined. And if we want to really make the best possible AGI which are going to be the best possible super-intelligence we need to craft ways... The best odds of success seem to be if we can craft ways to really draw on the overall intelligence, intuition, and wisdom of the global brain of humanity and our computing and communication networks, not some small elite group, as fun as it would be to be part of the small elite team training the beneficial singularity. This is what has lead me in large parts of the singularity project that I'm now running that I founded it in 2017. What we're trying to do with singularity is to create a decentralized platform and then community for both narrow AI and AGIs so that all the AIs in the decentralized network of AIs and the AI programmers and the AI users can all in a participatory and democratic way control the evolution of that network. And if the singularity or something like it becomes a predominant way or even a really significant way AI is rolled out on the planet, then that's been the counter act to this eliteist tendency that we're seeing with a few large corporations and governments hiring most of the AI researchers, buying most of the AI startups, and sort of driving the AI agenda.

 

Ben Goertzel:

Randall's, next question, "What would you consider a worst case scenario for AGI? What's the best case scenario? What's the likely outcome?" Well, worst case scenario, we probably can't even imagine, but how about some some crazy Christian maniac Mind-uploads all of humanity into a simulation of the Christian Hell and just burns your simulated clones until the end of the universe. That would be pretty bad. We could come up with worse. Best is utopia and of course I could take a lot of forms, but what I'd like do is form myself into multiple persons, let one of them mind-upload into the global super intelligent mind matrix of the multiversal super-intelligence mind matrix and then let another one of then stay in roughly human form an upgrad itself progressively so it can be an even better and greater human than is possible in the scope of legacy humanity.

 

Ben Goertzel:

Of course, when you really think about that, what's funny is from the stand point of the form of me that remains human, when I still have a form that merges with the super-intelligent mind matrix, it would be like, okay, that form of mine has been created. I'll wait, now in the last one second, it's experienced 10 trillion times more things then I ever will be able to, and it's evolving something totally uncomprehensible to me. So there, that's nice that I've spawned that super-integelent mind child. Now, I'll go on being being human. So there's going to be a discontinuity between forms if me that embraces the singularity for long and becames massively super intelligent verses the form fo me that remaines in the human form. But I would like to see everyone able to form themselves however they want and make many copies of themselves and explore different reasons of mind space and this lets each mind explore a variety of different types of realities that are utopic in it's own perspective and this is quite feasible. It may even be the most likely outcome, but we don't really have the basis to fully ration the estimate, the probabilities. I think we can work toward increasing the probabilities of beneficial outcomes like this in a number of ways. One is the AIs that we're creating right now what are going to grow into the AGI. Maybe that would be just in the future, we should be using these AIs to do benefitial things like cure disease, you know, teach children, improve people's states of consciousness, discover science. We've got to take the bulk of AI away from selling spying and killing. Iwould like to eliminate these things, human society and human psychology being what they are, but they don't need to be the preponderance of what AI is used and developed for. That's probably the most important thing we can do now to move the odds of utopic rathar than distopia or mediocre outcomes in a positive direction.

 

Ben Goertzel:

Next Randal asked, "What about Nick Bostrom's book Superintelligence." I read a review of that consequencial intelegence fears from Susan Potentials two years ago. My veiw of Nick's book now is about the same as it was when he wrote that review. I love Nick Bostrom worked together in that World Transhumanist Association years ago. We organized the conference together. Bostrom, I think you're a really fun creative guy. I think the book Super Intelligence is a brilliant example of argumentative rethoric. It reads like he was the captain of the high school debate team or something, so it makes a rigorous, powerful argument that super intelligence doing thing to humans we consider nasty like anihilating all humans is possible and that these bad outcomes of super-intelegence have odds of greater than zero. Zero. But then the book often talks as if its tone and its informal statements, it talks as it been argued that bad outcomes from super-intelegence are likely but that was ever demonstrated.

 

Ben Goertzel:

All that was demonstrated is that the probability is somewhere above zero. So yeah, of course the probability is somewhere above zero, that superintelligence will kill everyone, but Bostrom didn't demonstrate that it was probable and no one has demonstrated that. On the other hand, I haven't demonstrated that it's highly unlikely that super intelligence will do bad things either from a rational point of view we just don't know, are leading into the leaping into the great unknown. But then Nick Bostrom in that book at least really champions a sort of elitist point of view and at some point in the book he's sort of exploring the idea. You could even have one Genious AI researcher working on AGI protected by the auspices of the United Nations and maybe that will be the safest way to do things. I'm like exact opposite of that. I think we want a tremendous amount of brilliant AI and AGI researchers all around the globe with many different points of view collaborating and I want a decentralized network coordinating this in a self organized democratic and participatory way. Certainly not the UN, which can't even handle far, far simpler tasks than coordinating the birth of of general intelligence. Those are the first few questions from Randal.

 

Ben Goertzel:

This is part two of the video in which I give some rambling improvised one in the morning type answers to some questions posed by my friend by Randal Koene as part of the Carboncopies workshop on mind uploading and computer-brain interface, AGI, and so forth, which was held in the part of the world far away from me. So, in light of not showing up and rambling semi-coherent coherently at the audience, I'm doing so on video from afar. So let me continue with some of Randal's questions.

 

Ben Goertzel:

First question, what do I think about brain computer interfacing as a tool to improve AI safety? What impact would high speed brain-computer interfacing have on AI, rapidly self evolving AI, or AGI? Brain computer interfaceing could either greatly aid with AI safety, or it could terribly harm the prospects of safe AI. It really depends on how you would see this. On the beneficial side, if we want our AGI's to bridge and understand human values, connecting to the human brain is going to be a really nice way for an AGI to suck some values out of the human brain. Of course, AGI could even learn values by watching people. And by enacting values bridging the gap between robots and other agents in the human world. But to the extent that an AI get those values from the brain and make them understand human values...

 

Ben Goertzel:

...the other hand if we look at selling, spying and killing as the main pursuits of the AI sphere in the world today, given the big checks from goverments, they're controlling so much of AI today. How could brain-computer interfacing be for selling, spying and killing. You could invent a lot of interesting ideas that way and will they lead to the AGIs that are going to be positive and beneficial to the humans that they're sentient beings as they expanded their intelegence? Well, quite possibly not. Right?

 

Ben Goertzel:

Randal's next question, "Could you create high bandwidth brain-computer interface without first having a neuroprosthesis or a completely artificial brain?"

 

Ben Goertzel:

I think that the brain is very adaptive and probably if you stuck a bunch of data from a computer into it, the brain would make some interesting sense of it. It would then be no longer a legacy human brain. The more complex and the more data coming into the brain, the brain would have to morph itself to cope with this data. But that'd be quite interesting. You're creating a hybrid mind. Of course you get to a certain level where the brain no longer as the capacity to adapt to the brain computer interface. But what exactly that level is, we don't know enough about neuroscience or information processing in general to know that. That problem will be determined by experiment. I probably will not be the first person to volunteer for the experiment, but I'll be fairly early on.

 

Ben Goertzel:

So this will be quite interesting to see. I do think it's important to remember that once you get into nanotech, femtotech the amount of intelligence you can pack into a grain of sand would probably be a quadrillion times the human brain and a trillion or something times any brain computer hybrid. Because the human brains are inefficient ways of information processing creativity. Probably an inefficient way to manifest consciousness compared to what's possible in an engineered physical system permissible even according with the known laws of physics, let alone to the laws of physics maybe understood to be after a singularity. Upgrading human intelligence by connecting brains to computers, the hybrid mind is interesting. This is in the scope of post-singularity minds, these hybrids are going to be closer to a monkey or a frog compared to a super intelligence, right? So it's an interesting thing to do in terms of the transition between here and post singularly minds, but in the end it's just a baby step toward the singularity.

 

Ben Goertzel:

Randal's next question, "Do you see ways in which whole brain emulation and artificial human brains might immediately present an existential risk to humanity?" If you emulate some nasty human and then copy that emulated human a million times and connect it's body to selling, spying, and killing machines around the world, this may not be good. But again it's not really about the technology, it's about how it's used.

 

Ben Goertzel:

My guess is that if you compare whole brain emulation to an AGI built according to some new rational, non-emulating architecture like an Open Cog system that works really well, I would guess there's both more benefit and more risk in the engineered nonbrain emulating design, because the human brain is not made for self modification. If you start enhancing intelligence of certain parts of it then the parts are going to break, and you're going to wind up to not being able to enhance this intelligence tremendously without basically replacing it with a totally different architecture. The human brain is an adaptation to certain resource constraints, and once you release those constraints, you are going to need to change the architecture to manifest the intelligence that is possible with the new constraints. I think, on the other hand, the irrationally architected AGI is just what we're trying to do with Open Cog or it'd be something different than Open Cog with something engineered with self modification and you know rational self understanding in mind.

 

Ben Goertzel:

This sort of AI system is going to be able to reprogram itself it's going to be able to study itself. It's going to be able to, replace one module with an upgraded module, it's going to be upgrading all its states and then rationally make decisions as to what possible improvements of itself to try. It can go far, far ahead of a human augment with a brain computer interface or an uploaded emulated human it can go far ahead of these humanesque, post human minds, in a good or a bad direction. Again, the technology has a lot of potentials It depends on what you do with it, and, of course, what you do with it will guide what it does with itself.

 

Ben Goertzel:

Randall asks, "Rapid, self-improvement is often described in the context of utility function optimization and reinforced with learning. In short, it may be accomplished only in a few, the many complex substances of the brain, like the neocortex and cerebelum. Do you think whole brain emulation can rapidly self improve?" Well that's, that's a really delirious and misguided question Randal. I think it's true that only a little bit of what the brain does is reinforcement learning, and that's why the brain can be intelligent because reinforcement learning is a terrible overall paradigm for general intelligence. Reinforcement learning is only a tiny bit of what the brain does. Just like deep hierarchical pattern recognition, like current, deep neural lens. is only a very small part of what the brain does. So yeah, rapid self improvement can go much more rapidly if its following methodologies besides just reinforcement learning. The reasons that the human brain can't rapidly self-improve for more than its architected contorted and limited way where each part is dependent on the other parts, and each part of their dependencies evolve to work within certain processing constraints which aren't compatible with the transient intelegence that we want to build. I mean it's just like you can't take a horse and double its size and have it still work. If you increase the short term memory capacity of humans to 10,000 instead of seven plus or minus two, the connection between short and long term memory and medium term memory isn't going to work, but the relation between declarative and procedural knowledge and short term memory isn't going to work. A lot of other changes have to be made all around the brain. Whereas, if you have a rationally architected AGI system and increase its short term memory capacity. If it's written well you could probably just use some automated code by rewriting the system and tweek the other parts of the AGI system to properly accommodate for the expanded short term memory of the AGI system. So, I think there are limitations in the speed of self improvement that you're going to get in the brain emulation or brain incorporating AGI system. But these aren't to do with the limitations of material, so that extremely narrow and limited paradigm of reinforcement learning, these are more to do with just the constraints of being an evolved system as opposed to an engineered system moslty. Both systems can evolve and evolution is slow and messy. Engineering systems can be engineered, which can be much faster. Engineered reflective systems can self engineer, which is going to be really, really nice and way faster and way more efficient than the mess of evolution.

 

Ben Goertzel:

Randall askes, "There's caution aagainst a strong push for neurotechnology and whole brain emulation, because work in those areas has been accelerated towards advancement toward runaway self-improving AI." Well, as I already said, I think this particular line of research in whole brain emulation is a terrible approach to self improving AIs. So, if you think self-improving AI is bad, you should be in favor of AI through brain emulation. If you want self-improving AI, you should be looking at engineered AI systems that are designed for rational reflection and self-understanding and self-modification.

 

Ben Goertzel:

Next question from Randall, "There's an argument, the ultimate solution for AI safety is a scenario for human and AI becoming inextricably entangled." Well, again, there's no reason to think that will guarantee safety. There could be good, bad aspects. I mean if you're taking, you know, a very powerful artificial cognitive system and coupling it with these reptilian/mammalian motivational and emotional systems like we have in humans, this can be pretty nasty, right? I mean, an open cog system is one example of a non-brain based system that has a certain set of goals. They don't drive all dynamics of the system, but they drive a significant amount of it. They have a certain set of goals and then the system rationally using probability theory and logic chooses which actions are most likely to achieve its most important goals in the current context. I mean it's not driven by its body and its emotions and its instincts to the extent that a human being is.

 

Ben Goertzel:

I think something like that is probably going to be, if it's done right, it's going to be safer than some weird Frankenstein thing with these evolved motivational and emotional system latched into some artificial cognitive system. We don't know because we haven't built an AGI based on the human brain or brain-computer interfacing or open cog or anything else, yet. We don't know this tremendous and very hard to deduce uncertainly here, but my own instinct and my intuition, and I've thought about this a lot, is that it's going to be a lot more dangerous to make something in one system, all this nasty mamalian/reptiles stuff without official cognition. I think you want the rational, reflective self-modifying AGI to understand human culture and human values, and to have compassion for humans, and you don't really want it to be a human. You want there to still be humans, but you don't want to try to do something screwy like Megan said, the smartest and most powerful minds or some sort of huamn-AGI, franken-bob thing. You want to accept that humans are just a limited form of mind.

 

Ben Goertzel:

I mean there's beauty in this limitation as well as hideousness in this limitation, but that's what we are and one of our beauties is that we can build fundamentally superior minds that are compassionate toward us. I can self-understand, self-modify, and self-improve in ways that we intrinsically cannot due to the way that we evolved, and we can then coexist with these super minds, but we can't be these super minds, and trying to create a super mind that's tied in with the human/mammal/reptile control system is far more risky than any other technology on the horizon.

 

Ben Goertzel:

Another question from Randall, "in your opinion, what should humanity do to maximize, long term chances for survival." Give Ben 50 quadrillion dollars! Apart from that, very clearly what humanity should be doing is spending a large percentage of its resources on globally beneficial applications of advanced technologies including AI, nanotechnology, neurotechnology, and so forth, and trying to create machines with compassion toward humans in deep, rich understanding of the full breadth of human values. We shouldn't be including the bulk of our AI resources into selling, spying, and killing. We shouldn't be putting so little resources into medical applications and advanced technology into education, agriculture, poets, scientists, social workers, nurses, preschool teachers and philosophers, and so forth. So I mean, if you look at it from the outside and you had a species on the verge of creating the first, minds more intelligent and powerful than itself, you might think a large percentage of that specie's resources was going into figuring out theoretically how to make these new minds be as beneficial as possible for the universe and the multiverse and for the species during the creating, and to prototyping different kinds of beneficial engineered minds, and to making sure that engineered minds, as they increase in intelligence year-by-year, are working closely in a positive and compassionate way for the species that created them. Instead, almost all AI development now is driven by commercial or military ends, and the same for medical technology and nanotechnology.

 

Ben Goertzel:

I mean almost all this technology is being developed so that one country can achieve military power over other countries or so the one company can extract money from other people so the fact that our technologies are being developed mostly out of tribalistic or greed based motives truely isn't good. We want to be developing these technologies in a way that is motivated and is explicitly driving toward broad benefit for humans and for the other minds that being created. That's not what we're doing. I'm trying to develop advanced technologies in a way that will help all of humanity and will help eulated human minds that we're going to create and throw in the animals and plants and the rest of the ecosystem and any aliens that may come out of this or other dimensions as well. But most relevant advanced technologies are being driven by very narrow sort of probablistic or ego based goals. This is not optimal. Of course, you can't solve the problem top-down all at once, hopefully you can solve that problem by unleashing sort of new methodologies into the world. I mean just as open source transformed the software world, perhaps decentralized AI networks can transform the AI world and cause AI just self organize in a way it's more democratic and participatory so that as AGI emerges out of nornal AI, it's emerging with the input from users, developers, and the participants of AI around the world and it's getting a broad range of applications and feeding on a broad range of human insights and feelings and intuitions.

 

Ben Goertzel:

It's about 1:38 AM here. Truely I'm becoming less and less lucid as this ramble continues, but hopefully I've given you some flavor of my views on these issues. So thanks Randall for inviting me to participate via video and hopefully next time I can show up in person.

 

Mallory Tackett:

All right. I'll stop sharing my screen, and I'm back on it. That was a really interesting response by Dr Ben Goertzel. I'm wondering if Anders was paying attention to all of that, and if he has anything to say.

 

Anders Sandberg:

Yeah. Well there is so much to comment on, of course, to unpack there and I think there are two parts. That particularly caught my attention. One is this issue where a band disagrees with Nick about how to go about handling it. And I think it has to do with prior probability estimates and guesses on how does the risk landscape look like. Then Ben points out that if you have a situation where it's enough that there is one bad AI researcher, then big things happen. This is very much the problem we outline, Nick and me, in a paper we call the unilateralist curse. If you have a group of agents that it's enough that one agent can unleash something, then even if all agents are nice and are trying to do it only if they think it's a good thing. As the group gets larger, it's more likely that somebody is going to be that guy. So I think this is a situation one should recognize when you're in that situation you should try to be more conservative than you normally would like to be.

 

Anders Sandberg:

Just because of the nature of that situation. But I think in particular when you really what to regulate is when you can't afford to be wrong, even once. And there are those people in AI safety, you really think that, yeah, we are very close to that heartache of scenarios and the likely bad outcome of such a scenario that means that we have a sharply fetch should we must be exceedingly cautious. I'm way more optimistic. I don't know if that actually is based on an irrational reasoning rather thing, but I don't trust this conclusion, but I think actualy, it's a more benign domain. And I think Ben also has roughly the same view here. So this of course leads to different ideas about what you want to do, regulatory speaking. However, it also suggests that if we could get even a slight bit of battery information about the actual risks, actual structure, the actual probability we would win so much.

 

Anders Sandberg:

So I think that is another reason to really, really pursue this. And then, another very interesting point he made very briefly was the resource constraints that have shaped the brain once you're free of them, you might reshape it in a lot of other ways. Again, just to mention a paper I wrote with Nick, we did one about human enhancement where we looked at evolutionary medicine as one guide to try to see where evolution really might have constrained us and those are the areas where we might then find ways of unconstraining us in our constructive ways. There might be other domains where it's going to be very hard to fix. For example, I think even uploads are good to have two run sleep simply because our memory consolidation is so based around that, which would be tremendously annoying form some perspective. But on the other hand, yeah it's going to take a lot of reverse engineering to get the answer. And that was just two main points on Ben's enormous discussion.

 

Randal Koene:

Interesting bit there about whole brain emulations and sleep. That's one of the things I actually happen to think about, which was if you have the freedom to change synapses, if you look at what sleep is actually trying to accomplish, if you've got your consolidation, and you've also got in rem sleep, bringing up older memories and mixing then in and doing some kind of interleaved learning and that sort of thing. There are alternative ways to do that. And for one thing, it's certainly doesn't have to take quite as long as it normally does. So, I imagine that there may be ways of getting around those sorts of limitations.

 

Anders Sandberg:

Yeah, I totally agree. But it's going to be a lot of tinkering.

 

Randal Koene:

Absolutely. And that's where Ben is absolutely right, of course, that it's much easier to come up with an artificial intelligence that is free to developing whatever algorithmic method likes to get it to self improve faster than an evolution, a patchwork that was evolved like the human brain. I think it might have come across to him as if I were saying the opposite, but that wasn't the case. I was just putting forward the question, well, can whole brain emulations even rapidly self improve? Is this something people should worry about? Since that sometimes got brought up. Mallory, how did you want to proceed? Do you want to take a question from the audience first?

 

Mallory Tackett:

Yeah. We do have some questions from the audience that I'd like to ask. We have one, their username is sacked SOS. Here they're quoting Descartes, I think therefore I am, "Does that not mean that all physical laws and natural laws are just in our mind?" and I think maybe what they're trying to say is, is our interpretation of physical laws limited by our biological brain? I also like to tack on to that, how do you think that will change if we were able to have a whole brain emulation and possibly merge that with AI, how will physical and natural laws then be interpreted?

 

Anders Sandberg:

Yeah. So this is of course familiar territory for most introductory philosophy courses where people start getting confused. Quite a lot of people not really think that, oh, there's nothing problematic with my perception, I'm see reality as it is. And at this point the gleeful philosopher will start bringing up optical illusion, some examples from quantum physics. But at the same time I think David Dutch has an interesting point in his book, The Beginning of Infinity, where he points out that the reach of our mind is way bigger than what you would expect for most of all systems. We actually have some mental tools that allow us to take explainations that work over shockingly large domains, even though we don't have direct access to them. When we talk about the distribution of galaxies in the universe in many ways we can't see the galaxy. We can only see them if we use telescopes. So in some sense, nobody has ever seen the Virgo cluster because you can't see it with the naked eye if you are a human, yet I don't think anybody would say it makes sense not to believe in its existence or a say that, oh, the Virgo cluster is a kind of weird artifact of having a telescope. So I think you get the same situation even if you're a brain emulation. Yes, now you might be seeing using a camera connected to a device driver connect to various pieces of software, turning it into neural signals. You add the extra levels, but as long as these levels don't distort things that are relatively faithful, I think you actually have something that make sense. And I do think we can build on top of it. I think we can... The most obvious thing is adding new senses, or trying to link up modules in our minds so they are functioning better.

 

Anders Sandberg:

If I could just run a quantum mechanics co-processor and have it seamlessly connected to my cortex, that would be wonderful. And maybe I would finally understand stuff. I would also have to change myself, how I work. But that might be a relatively low level thing, like changing how I visualize things in order to visualize more dementions rather than changing the true core need. Of course, philosophers will always throw in, well maybe there are things that are absolutely impossible for minds of our kind to think about. And I think that's true, but it might actually be relatively uninteresting. It's not too hard to construct girdles sentences that certain system... but that dosn't mean they don't come up that often in practice.

 

Mallory Tackett:

Would anybody else like to answer that?

 

Randal Koene:

Yeah, I kind of want to jump in on that just because it's a good point you brought up that it's interesting that our brain can handle things...

 

Mallory Tackett:

Hmm. Sounds like Randal might have lost Internet connection.

 

Randal Koene:

I am. Hmm.

 

Mallory Tackett:

All right, I can hear you now and see your video.

 

Randal Koene:

You hear me? Okay. So then I don't know what happened there.

 

Mallory Tackett:

It must have just dropped for a second.

 

Randal Koene:

I was just trying to expand on what Anders said about the brain being able to deal with things that it wasn't originally evolved for. It's interesting to see in which ways the tools that the brain has are applicable that way. And just to point out one little thing in this, this gelatinous blob here. Say for instance, the, the grid cells of the interinal system. We talk about them as something that helps map out space but the same part of the brain that feeds right into the hippocampus also has a task in humans it just generally manages to store new episodic memories, new conceptual memories concepts. Now, the interesting thing is if you take this system grid cells for mapping out spaces automatically, basically putting in vectors along them and saying, okay, this is x, Y and I'm over here and I'm over here, coming up with where you are.

 

Randal Koene:

If you take that and you apply it to more general concepts, something that has nothing to do with space, you can map out any new kind of idea, any concept that you have and suddenly apply it in ways that you never could before. You can work with it in a sense. Mapping things that shouldn't normally be mapped or that you don't encounter in nature. And I think you take that all across the brain and you can see that these systems, these tools, just like deep learning isn't something you can only use for one task can use it for many tasks the same way all the tools in the brain can be applied to many different things. It's true, but then at the same time, it's not necessarily optimized for a rapid self improvement. So I think Ben has a point there as well.

 

Randal Koene:

One thing I wanted to address briefly because Ben has a certain style, how he expresses his opinion on these matters is he often makes it very good and positive statement. He'll say something that he really hopes is the outcome. So we hear a lot of very positive scenarios from Ben.And in fact both sides of course want us to end up in the same type of outcome, a good outcome, whatever that good means. It's just that for some reason the methods that they advocate are very different sometimes. And a lot of this has to do with trying to understand what the probability of certain outcomes are and those probabilities are influenced by where you begin. So Andrew's already mentioned that, your bias, and perhaps by intuition which is of course, again bias by your experiences and by your emotional attachment to something.

 

Randal Koene:

So it's really a problem of clear probabilities lacking in this case. And we don't really know which scenarios are going to happen. Maybe this is just a sign of how young this field is still. But I do think one point that Ben makes is very interesting he brought this up in his paper in 2015 as well. The one that I think I might've mentioned somewhere, that we may need artificial intelligence or artificial general intelligence to guard against some of the bigger risks that we run into because of the way humanity goes about things. We'll talk about how an AI might want to follow a certain goal and keep on improving towards that goal. Even if it has to consume all the resources in the universe.

 

Randal Koene:

You could make a very similar argument for humans. We're still based on a system that evolved 2 million years ago and we're about to use up all the resources of the earth in order to try to keep doing that because that's what we were programmed to do. We may also be stuck with the same utility function and following it and we may be running a great risk of doing damage to ourselves in the process. Now again, I don't know if that's really what's going to happen, but this is part of the argument that Ben is making. Humans are not rational actors were not necessarily picking the best outcome all the time. AI may be more rational than we are, so it's entirely possible that developing that as quickly as possible, may be something that saves us from a risk that is more immediate or bigger because we keep on not having a good chance of comparing these probabilities, then the likelihood that AGI itself somehow be an agent that takes us down. I don't know if I agree with his point of view more or if I agree with Nick Bostrom's point of view, but I'd love to hear Andrew's kind of give this a shot.

 

Anders Sandberg:

Yeah. I've been thinking a lot about what can we do to do the research right? Because as you said, we need to get the probabilities right, but we might also want to get to the values right? So in my current project, this big book about the long term future of humanity, one of the parts that I'm still working very hard on and that is tough to work on, it's of course what values make sense in the universe. And what would rational actually mean because yes, we're definitely not rational right now, but just like the hippocampal place cells allow us to handle spaces of quite arbitrary shapes, our minds are actually able to have quite a lot of different models of rationality and thinking. It's just that we're not super good at that so much yet. Well, it seems to me that one way of making progress on both of this kind of tough problems is to look at toy problems.

 

Anders Sandberg:

You take stylized small problems. So this is how you might help the AI safety move forward. You will demonstrate that here is a little agreed world situation and you want your agent to behave itself in a certain way. Can you program it to do that? And that's useful, not just to entice software engineers the realize that actually it's harder than they think to make AI safe, but it also is a good way of revealing tricky problems. Learning about the size of the problems and learning about what other questions you want to do because typically to refine this as people find ways around what you wanted to do. There is another side and that is the empirical research you actually want to go out in the world and actually watch real situations because that keeps you honest. It's quite important to not just spend your time in the lab or in a thinking chair, but actually go out and try to get some data and then critically judge it.

 

Anders Sandberg:

Just because people think autonomous course should be driving in particular ways, doesn't mean that's the ethical thing to do, or the smart thing to do, but it's certainly an interesting input. And then ideally once you have both good models and the empirical data to play around with, you can also start building the more deeper theories. And I think this goes for creating AGI that is helping us make good choices. We need to know a bit more about what good choices are, both from a correctional and ethical style point. But we also actually wanted to develop... Can we make a decision aid in tools even in less ethically fraud domains that are actually useful, that's not entirely obviously yet, yet it is a big market. You can make a lot of money if you make a good decision aid and there researchers working on kind of moral decision aids.

 

Anders Sandberg:

I have some friends in Oxford working on that, where hopefully they would tell you, if you have a dilemma, "Well, given what you claim to believe, this is what you ought to be doing. You want to discuss it more?" and I think that could be quite useful. So I think there is an interesting big research project here. It's kind of spanning, not just AI safety and brain emulation, but quite a lot of other stuff. Taking what starts out in philosophy, making it concrete enough that we can start writing code and doing a psychological service and they'll start thinking broadly about how to implement tools to make us better.

 

Mallory Tackett:

I think that was a great answer. Next, we have a question from Dean Horak and it's actually one of my favorite questions to talk about. He says, "Assume we have substrate independent minds available on some digital whole brain emulation platform, it would seem that accessing someone else's memories and experiences and thoughts would be nearly as effortless as accessing our own. Given the scenario, wouldn't we eventually lose our individual identities and ultimately become like a hive mind?

 

Anders Sandberg:

Well I think to some degree we have already created tools to access. Yet, we actually deliberately create walls. In some sense Internet allows any computer on the Internet to talk to any other computer. But it's not just that I can't get your data out of your computer. But we also deliberately create structures like web pages or and the Google hangouts to actually shape it, because having everything in a big pile is not very useful. My memories are only useful because I did make good association in relation to my current experience etc. And if you had your mind and memories and my memories sometimes of course they will overlap in a resonant sort of way, but quite often it would probably be just arbitrary. But we want selectivity, we want filtering and that is probably what it's going to mean that hiveminds are probably going to be more limited than we think.

 

Anders Sandberg:

In fact you can compare it to economics. Why do they have companies or firms and of course economists have been going on about the theory of the firm and that you have CSR Hidalgo at MIT and the general argument is that they are economists of scale up to a point. It's a good idea to get a bunch of people together but you also need to manage the information flow. Then you create a little box around it and call this a department or a company and we are likely to do the same thing for group minds. It wouldn't surprise me that once you have the ability to link minds, we're gonna experiment endlessly with a lot of architectures and some of them are going to be good for certain things, others less good. But maybe specialized and then we might actualy have this big complicated structure of overlapping and sometimes heroic income. Sometimes autocratic and sometimes totally alien structures. It's going to be predict exiting. But again, it's a good thing that there is an undo button of your brain emulation.

 

Randal Koene:

Great point about the undo button. That's something I occasionally mention when people ask about, this so risky, why would you experiment with upgrading and stuff. But you mentioned here, things that are already a bit like hive minds, organizations and groups and all that thing. And I think that to answer a dean's question, I think that hive minds merging is probably unavoidable because it's already a reality. It's just that the components of a hive mind don't know they're in a hive mind or maybe they do, but sometimes they don't or they don't admit it. They'll say I'm still an individual. But really they're entirely dependent on everyone else. I mean, do you farm it's sort of... We may be part of a hive mind and not know it yet.

 

Mallory Tackett:

That was great. I definitely see it that way that we are already kind of interconnected in a certain way due to technology and social media. Whole brain emulation will just exacerbate something that's already happening.

 

Anders Sandberg:

Some aspects of social media actually give us a very good inkling about just connecting everything completely. That's not very good. We actually want to create the right kind of structures and finding the right kind can turn out to be amazingly hard, while down the line technology is relatively safe.

 

Randal Koene:

But then if you're taking social media as the example, then maybe it's also demonstrating that it takes a certain amount of trial and error and learning because we had no idea what this would be like before we got into it. We didn't know what the Internet was going to spawn and we certainly didn't know anything about social media back in the eighties nineties or before, so it would have been very difficult to think ahead and say, when we design social media, what should the boundaries be?

 

Anders Sandberg:

I totally agree. There is this general problem that the world is much more complicated than our minds, so we generaly quite often can't predict the consequences outside a particularly simple domain. We just have to do experiments. Which is of course problematic if experiments are risky or expensive or painful, but also very exciting because it means that we've discovered new things. I find it very unlikely that we will ever enhance themselves such a degree that we can just figure it out or rationally just like some philosopher king just sitting in his arm chair, knowing everything. I think we're going to find that a world that has great minds, is going to be even more complicated anyway, after all, social media among smart super intelligence is going to be probably having emergent, wierd properties that we can think about and they can't think about. They're going to be equally confused when they get their counterpart to fake news or Twitter storms and the are going to perhaps wonder maybe we need a bigger mind to figure this hyper social media out.

 

Randal Koene:

Hmm. Yeah. It seems like there are at least two different ways to go about preventing the worst case scenario. And one of them is you try to anticipate something bad that could happen and just don't do it or prevent it from happening. And the other one, as we just mentioned, in the case of experimenting with whole brain emulation is undo buttons. So, what we haven't really talked about, and maybe it's because nobody thinks it's possible, is undo buttons for things that happen in AI development. I guess part of the problem is that we imagined that a bad scenario is one where AI development suddenly goes so quickly that there is just no way to stop it, let alone undo whatever happened. But an undo button would be nice if such a thing were possible outside of a sandbox let's say.

 

Anders Sandberg:

So I think the problem is that many of the AI risks are seen as information hazards. So once I publish my code on git hub and tweet about it, there is no way of gifting the genie back in the bottle. And that is an interesting thing because that might just be because we are not imaginative enough to come up with a better undo button. There are ways of handling information hazards. I'm working on some projects and thinking about how institutions like the Future of Humanity Institute should organize its own research when it finds some risky information and you can certainly have structured transparency where you can reveal certain information but not others in a way that is trustworthy etcetera. But it might also be that we can find ways of doing AI research where you can build in some forms of undo buttons.

 

Anders Sandberg:

Like we pursue a certain line of thinking, we discovered sort of risks or certain problems, and there is a considered way of removing it. This is of course what we would like to have in science too. If I write a paper that I would later retract, ideally all papers citing that paper need to get a little marker in the citation of that now we're referring to paper that turned out to be wrong for one reson or another. We need to build an infrastructure for that. Right now I don't have any idea of how to do that, but it sounds like one of those things that should be a high priority for someone.

 

Randal Koene:

Yeah. Actually, this is something where blockchain gets interesting. A ledger that you shared because if you can actually track everyone who is on that system, or who's got a copy of something, then you could go in and try to withdraw it or withdraw the ability to run that thing or to change it to whatever needs to be fixed. That's one approach. I'm just talking about it very loosely because I'm not actually someone who develops that sort of code, but it sounds to me like something that those systems could evolve towards. And the other is what we call software as a service, right? If you look at that, the situation there is that if, let's say, the service that you're making available has a bug. You only need to fix it where the service is being run and everyone else uses it and puts it to use. So if there are all these little modules that you use for AI, but they're all software as a service, it's very different than if you're giving away the code. Some opportunities there, perhaps for something like an undo button.

 

Anders Sandberg:

Yup. Sounds like a really fun topic to pursue further.

 

Mallory Tackett:

All right. I think that might be a good question to end on for our workshop unless anybody has any objections.

 

Randal Koene:

Yeah, I guess, let's ask our panel if any of you have any questions that you think really should be mentioned, either because they're your own or you saw them coming through. Maybe Jesse or somebody noticed something coming through, just as a quick opportunity to before we close. But otherwise I agree because it's been long enough and we can still handle all of those questions later in other ways because we've got them all written down.

 

Mallory Tackett:

Doesn't sound like it. So I guess that'll be the conclusion of our workshop. Again, thank you everybody for attending. I want to say thank you to all of our volunteers that helped put this together and all of the speakers that joined with us today, talked on the panel, and sent us videos or did interviews. We really appreciate that. I just wanted to have just have a couple of announcements besides the survey for the workshop that we would very much appreciate all of our audience members filling out and that can be found at survey.carboncopies.org. I also wanted to mention that we have a new episode of our podcast and that's available on our website if you want to go and view that. We frequently have new episodes that address different topics related to whole brain emulation. We also have up items on our store, new items on our store now.

 

Mallory Tackett:

We have sweatshirts, tee shirts, coffee mugs, bags. I'm already decked out in the sweatshirt if anybody can see. It's got our nice logo and I really enjoy it and just wanted to mention that we are a 501C3 nonprofit. All of the work that we do has been volunteer work and we relly on the generous contributions of our donors. If you could visit carboncopies.org/donate and make a donation that would be great for us. Our goal right now is to get to about $2,000 a month and that'll fund future workshops and any future research that we'd like to do that would allow anybody else, the audience members or people that follow us to participate. And on that note we also have several social media channels that you can join us on. It just depends on your preference for Facebook, it's facebook.com/groups/carboncopies. We also have a reddit account and it's just our /carboncopies. And then our Twitter account is @carboncopies.org. Thank you everybody and I think we'll close on that. I had a great time and I think I learned a lot and I have a lot of new questions.

 

Randal Koene:

Thank you, Mallory. And one just quick little thing. Thank you so much Anders for jumping in even though you had to travel so far. I hope you get some good rest now.

 

Anders Sandberg:

I'm looking forward to a nice Swedish night now, but this has been so exciting. I'm going to have sweet dreams about brain emulations.

 

Randal Koene:

Okay, fantastic. And also for anyone who had trouble hearing any portions of the interviews, or Ben's talk, all of these are going to be made available separately in their full high quality audio and video. They degraded a bit as we had to stream it through this live stream. So you'll get those as well. Thank you everyone.

 

Anders Sandberg:

Thank you.

 

Mallory Tackett:

Bye Bye.

 

Randal Koene:

Bye Anders.