PDF: We made a fancy PDF of this post for printing and offline viewing. Buy it here. (Or see a preview.)


We have what may be an extremely difficult problem with an unknown time khổng lồ solve it, on which quite possibly the entire future of humanity depends. — Nichồng Bostrom

Welcome lớn Part 2 of the “Wait how is this possibly what I’m reading I don’t get why everyone isn’t talking about this” series.

Bạn đang xem: Tim urban là ai

Part 1 started innocently enough, as we discussed Artificial Narrow Intelligence, or ANI (AI that specializes in one narrow task lượt thích coming up with driving routes or playing chess), and how it’s all around us in the world today. We then examined why it was such a huge challenge to get from ANI lớn Artificial General Intelligence, or AGI (AI that’s at least as intellectually capable as a human, across the board), và we discussed why the exponential rate of technological advancement we’ve seen in the past suggests that AGI might not be as far away as it seems. Part 1 ended with me assaulting you with the fact that once our machines reach human-cấp độ intelligence, they might immediately vì chưng this:





This left us staring at the screen, confronting the intense concept of potentially-in-our-lifetime Artificial Superintelligence, or ASI (AI that’s way smarter than any human, across the board), and trying khổng lồ figure out which emotion we were supposed to have on as we thought about that.11← open these

Before we dive sầu inlớn things, let’s remind ourselves what it would mean for a machine khổng lồ be superintelligent.

A key distinction is the difference between tốc độ superintelligence quality superintelligence. Often, someone’s first thought when they imagine a super-smart computer is one that’s as intelligent as a human but can think much, much faster2—they might picture a machine that thinks lượt thích a human, except a million times quicker, which means it could figure out in five minutes what would take a human a decade.

That sounds impressive, & ASI would think much faster than any human could—but the true separator would be its advantage in intelligence quality, which is something completely different. What makes humans so much more intellectually capable than chimps isn’t a difference in thinking speed—it’s that human brains contain a number of sophisticated cognitive sầu modules that enable things lượt thích complex linguistic representations or longterm planning or abstract reasoning, that chimps’ brains vị not. Speeding up a chimp’s brain by thousands of times wouldn’t bring hyên ổn khổng lồ our level—even with a decade’s time, he wouldn’t be able lớn figure out how to use a set of custom tools to lớn assemble an intricate mã sản phẩm, something a human could knock out in a few hours. There are worlds of human cognitive sầu function a chimp will simply never be capable of, no matter how much time he spends trying.

But it’s not just that a chimp can’t vày what we vị, it’s that his brain is unable to grasp that those worlds even exist—a chimp can become familiar with what a human is and what a skyscraper is, but he’ll never be able to underst& that the skyscraper was built by humans. In his world, anything that huge is part of nature, period, & not only is it beyond hyên ổn to build a skyscraper, it’s beyond hyên ổn to realize that anyone can build a skyscraper. That’s the result of a small difference in intelligence unique.

And in the scheme of the intelligence range we’re talking about today, or even the much smaller range aý muốn biological creatures, the chimp-to-human chất lượng intelligence gap is tiny. In an earlier post, I depicted the range of biological cognitive sầu capathành phố using a staircase:3


To absorb how big a giảm giá khuyến mãi a superintelligent machine would be, imagine one on the dark green step two steps above humans on that staircase. This machine would be only slightly superintelligent, but its increased cognitive ability over us would be as vast as the chimp-human gap we just described. And lượt thích the chimp’s incapathành phố lớn ever absorb that skyscrapers can be built, we will never be able to lớn even comprehkết thúc the things a machine on the dark green step can do, even if the machine tried lớn explain it khổng lồ us—let alone vày it ourselves. And that’s only two steps above us. A machine on the second-to-highest step on that staircase would be lớn us as we are lớn ants—it could try for years to teach us the simplest inkling of what it knows và the endeavor would be hopeless.

But the kind of superintelligence we’re talking about today is something far beyond anything on this staircase. In an intelligence explosion—where the smarter a machine gets, the quicker it’s able to increase its own intelligence, until it begins to lớn soar upwards—a machine might take years lớn rise from the chimp step to the one above it, but perhaps only hours khổng lồ jump up a step once it’s on the dark green step two above sầu us, và by the time it’s ten steps above sầu us, it might be jumping up in four-step leaps every second that goes by. Which is why we need lớn realize that it’s distinctly possible that very shortly after the big news story about the first machine reaching human-level AGI, we might be facing the reality of coexisting on the Earth with something that’s here on the staircase (or maybe a million times higher):


And since we just established that it’s a hopeless activity to try lớn underst& the power of a machine only two steps above us, let’s very concretely state once & for all that there is no way khổng lồ know what ASI will vì or what the consequences will be for us. Anyone who pretends otherwise doesn’t underst& what superintelligence means.

Evolution has advanced the biological brain slowly & gradually over hundreds of millions of years, and in that sense, if humans birth an ASI machine, we’ll be dramatically stomping on evolution. Or maybe this is part of evolution—maybe the way evolution works is that intelligence creeps up more và more until it hits the cấp độ where it’s capable of creating machine superintelligence, and that cấp độ is like a tripwire that triggers a worldwide game-changing explosion that determines a new future for all living things:


And for reasons we’ll discuss later, a huge part of the scientific community believes that it’s not a matter of whether we’ll hit that tripwire, but when. Kind of a crazy piece of information.

So where does that leave sầu us?

Well no one in the world, especially not I, can tell you what will happen when we hit the tripwire. But Oxford philosopher và lead AI thinker Niông xã Bostrom believes we can boil down all potential outcomes inkhổng lồ two broad categories.

First, looking at history, we can see that life works like this: species pop up, exist for a while, & after some time, inevitably, they fall off the existence balance beam và l& on extinction—


“All species eventually go extinct” has been almost as reliable a rule through history as “All humans eventually die” has been. So far, 99.9% of species have sầu fallen off the balance beam, & it seems pretty clear that if a species keeps wobbling along down the beam, it’s only a matter of time before some other species, some gust of nature’s wind, or a sudden beam-shaking asteroid knocks it off. Bostrom calls extinction an attractor state—a place species are all teetering on falling inkhổng lồ & from which no species ever returns.

And while most scientists I’ve sầu come across acknowledge that ASI would have sầu the ability khổng lồ sover humans to lớn extinction, many also believe sầu that used beneficially, ASI’s abilities could be used lớn bring individual humans, and the species as a whole, lớn a second attractor state—species immortality. Bostrom believes species immortality is just as much of an attractor state as species extinction, i.e. if we manage to get there, we’ll be impervious khổng lồ extinction forever—we’ll have conquered mortality and conquered chance. So even though all species so far have sầu fallen off the balance beam & landed on extinction, Bostrom believes there are two sides to the beam & it’s just that nothing on Earth has been intelligent enough yet to lớn figure out how to fall off on the other side.


If Bostrom and others are right, and from everything I’ve read, it seems lượt thích they really might be, we have sầu two pretty shocking facts to absorb:

1) The advent of ASI will, for the first time, open up the possibility for a species to lớn land on the immortality side of the balance beam.

2) The advent of ASI will make such an unimaginably dramatic impact that it’s likely lớn knochồng the human race off the beam, in one direction or the other.

It may very well be that when evolution hits the tripwire, it permanently ends humans’ relationship with the beam và creates a new world, with or without humans.

Kind of seems like the only question any human should currently be asking is: When are we going to hit the tripwire and which side of the beam will we l& on when that happens?

No one in the world knows the answer to lớn either part of that question, but a lot of the very smarchạy thử people have sầu put decades of thought into lớn it. We’ll spkết thúc the rest of this post exploring what they’ve sầu come up with.


Let’s start with the first part of the question: When are we going to lớn hit the tripwire?

i.e. How long until the first machine reaches superintelligence?

Not shockingly, opinions vary wildly & this is a heated debate among scientists và thinkers. Many, like professor Vernor Vinge, scientist Ben Goertzel, Sun Microsystems co-founder Bill Joy, or, most famously, inventor and futurist Ray Kurzweil, agree with machine learning expert Jeremy Howard when he puts up this graph during a TED Talk:


Those people subscribe to lớn the belief that this is happening soon—that exponential growth is at work và machine learning, though only slowly creeping up on us now, will blow right past us within the next few decades.

Others, lượt thích Microsoft co-founder Paul Allen, research psychologist Gary Marcus, NYU computer scientist Ernest Davis, & tech entrepreneur Mitch Kapor, believe that thinkers lượt thích Kurzweil are vastly underestimating the magnitude of the challenge and believe that we’re not actually that cđại bại khổng lồ the tripwire.

The Kurzweil camp would counter that the only underestimating that’s happening is the underappreciation of exponential growth, và they’d compare the doubters khổng lồ those who looked at the slow-growing seedling of the internet in 1985 & argued that there was no way it would amount to lớn anything impactful in the near future.

The doubters might argue bachồng that the progress needed to lớn make advancements in intelligence also grows exponentially harder with each subsequent step, which will cancel out the typical exponential nature of technological progress. And so on.

A third camp, which includes Nick Bostrom, believes neither group has any ground khổng lồ feel certain about the timeline và acknowledges both A) that this could absolutely happen in the near future & B) that there’s no guarantee about that; it could also take a much longer time.

Xem thêm: Phi Huyền Trang Là Ai - Tiểu Sử Thánh Nữ Mì Gõ

Still others, like philosopher Hubert Dreyfus, believe all three of these groups are naive for believing that there even is a tripwire, arguing that it’s more likely that ASI won’t actually ever be achieved.

So what bởi vì you get when you put all of these opinions together?

In 2013, Vincent C. Müller & Nichồng Bostrom conducted a survey that asked hundreds of AI experts at a series of conferences the following question: “For the purposes of this question, assume that human scientific activity continues without major negative disruption. By what year would you see a (10% / 1/2 / 90%) probability for such HLMI4 to lớn exist?” It asked them lớn name an optimistic year (one in which they believe there’s a 10% chance we’ll have AGI), a realistic guess (a year they believe there’s a một nửa chance of AGI—i.e. after that year they think it’s more likely than not that we’ll have sầu AGI), and a safe guess (the earliest year by which they can say with 90% certainty we’ll have sầu AGI). Gathered together as one data set, here were the results:2Median optimistic year (10% likelihood): 2022Median realistic year (một nửa likelihood): 2040Median pessimistic year (90% likelihood): 2075

So the median participant thinks it’s more likely than not that we’ll have AGI 25 years from now. The 90% median answer of 2075 means that if you’re a teenager right now, the median respondent, along with over half of the group of AI experts, is almost certain AGI will happen within your lifetime.

A separate study, conducted recently by author James Barrat at Ben Goertzel’s annual AGI Conference, did away with percentages và simply asked when participants thought AGI would be achieved—by 2030, by 2050, by 2100, after 2100, or never. The results:3By 2030: 42% of respondentsBy 2050: 25%By 2100: 20%After 2100: 10%Never: 2%

Pretty similar khổng lồ Müller & Bostrom’s outcomes. In Barrat’s survey, over two thirds of participants believe AGI will be here by 2050 and a little less than half predict AGI within the next 15 years. Also striking is that only 2% of those surveyed don’t think AGI is part of our future.

But AGI isn’t the tripwire, ASI is. So when vì the experts think we’ll reach ASI?

Müller và Bostrom also asked the experts how likely they think it is that we’ll reach ASI A) within two years of reaching AGI (i.e. an almost-immediate intelligence explosion), và B) within 30 years. The results:4The median answer put a rapid (2 year) AGI → ASI transition at only a 10% likelihood, but a longer transition of 30 years or less at a 75% likelihood.

We don’t know from this data the length of this transition the median participant would have put at a một nửa likelihood, but for ballpark purposes, based on the two answers above, let’s estimate that they’d have sầu said 20 years. So the median opinion—the one right in the center of the world of AI experts—believes the most realistic guess for when we’ll hit the ASI tripwire is = 2060.


Of course, all of the above statistics are speculative, and they’re only representative sầu of the center opinion of the AI expert community, but it tells us that a large portion of the people who know the most about this topic would agree that 2060 is a very reasonable estimate for the arrival of potentially world-altering ASI. Only 45 years from now.

Okay now how about the second part of the question above: When we hit the tripwire, which side of the beam will we fall to?

Superintelligence will yield tremendous power—the critical question for us is:

Who or what will be in control of that power, & what will their motivation be?

The answer to this will determine whether ASI is an unbelievably great development, an unfathomably terrible development, or something in between.

Of course, the expert community is again all over the board và in a heated debate about the answer lớn this question. Müller and Bostrom’s survey asked participants lớn assign a probability to the possible impacts AGI would have sầu on humanity & found that the mean response was that there was a 52% chance that the outcome will be either good or extremely good và a 31% chance the outcome will be either bad or extremely bad. For a relatively neutral outcome, the mean probability was only 17%. In other words, the people who know the most about this are pretty sure this will be a huge giảm giá. It’s also worth noting that those numbers refer lớn the advent of AGI—if the question were about ASI, I imagine that the neutral percentage would be even lower.

Before we dive much further into this good vs. bad outcome part of the question, let’s combine both the “when will it happen?” & the “will it be good or bad?” parts of this question into lớn a chart that encompasses the views of most of the relevant experts:


We’ll talk more about the Main Camp in a minute, but first—what’s your deal? Actually I know what your khuyến mãi is, because it was my deal too before I started researching this topic. Some reasons most people aren’t really thinking about this topic:

Humans have sầu a hard time believing something is real until we see proof. I’m sure computer scientists in 1988 were regularly talking about how big a giảm giá khuyến mãi the mạng internet was likely lớn be, but people probably didn’t really think it was going khổng lồ change their lives until it actually changed their lives. This is partially because computers just couldn’t bởi vì stuff like that in 1988, so people would look at their computer and think, “Really? That’s gonna be a life changing thing?” Their imaginations were limited khổng lồ what their personal experience had taught them about what a computer was, which made it very hard lớn vividly picture what computers might become. The same thing is happening now with AI. We hear that it’s gonna be a big khuyễn mãi giảm giá, but because it hasn’t happened yet, và because of our experience with the relatively impotent AI in our current world, we have a hard time really believing this is going to lớn change our lives dramatically. And those biases are what experts are up against as they frantically try khổng lồ get our attention through the noise of collective sầu daily self-absorption.Even if we did believe it—how many times today have you thought about the fact that you’ll spover most of the rest of eternity not existing? Not many, right? Even though it’s a far more intense fact than anything else you’re doing today? This is because our brains are normally focused on the little things in day-to-day life, no matter how crazy a long-term situation we’re a part of. It’s just how we’re wired.

One of the goals of these two posts is to lớn get you out of the I Like to lớn Think About Other Things Camp and inlớn one of the expert camps, even if you’re just standing on the intersection of the two dotted lines in the square above sầu, totally uncertain.

During my retìm kiếm, I came across dozens of varying opinions on this topic, but I quickly noticed that most people’s opinions fell somewhere in what I labeled the Main Camp, and in particular, over three quarters of the experts fell into lớn two Subcamps inside the Main Camp:


We’re gonna take a thorough dive sầu into both of these camps. Let’s start with the fun one—

Why the Future Might Be Our Greachạy thử Dream

As I learned about the world of AI, I found a surprisingly large number of people standing here:


The people on Confident Corner are buzzing with excitement. They have sầu their sights mix on the fun side of the balance beam và they’re convinced that’s where all of us are headed. For them, the future is everything they ever could have hoped for, just in time.

The thing that separates these people from the other thinkers we’ll discuss later isn’t their lust for the happy side of the beam—it’s their confidence that that’s the side we’re going to lớn lvà on.

Where this confidence comes from is up for debate. Critics believe sầu it comes from an excitement so blinding that they simply ignore or deny potential negative sầu outcomes. But the believers say it’s naive sầu lớn conjure up doomsday scenargame ios when on balance, giải pháp công nghệ has and will likely end up continuing to help us a lot more than it hurts us.

We’ll cover both sides, và you can khung your own opinion about this as you read, but for this section, put your skepticism away and let’s take a good hard look at what’s over there on the fun side of the balance beam—and try to absorb the fact that the things you’re reading might really happen. If you had shown a hunter-gatherer our world of indoor comfort, giải pháp công nghệ, and endless abundance, it would have seemed like fictional magic lớn him—we have sầu lớn be humble enough lớn acknowledge that it’s possible that an equally inconceivable transformation could be in our future.

Nick Bostrom describes three ways a superintelligent AI system could function:6As an oracle, which answers nearly any question posed to lớn it with accuracy, including complex questions that humans cannot easily answer—i.e. How can I manufacture a more efficient car engine? Google is a primitive type of oracle.As a genie, which executes any high-level command it’s given—Use a molecular assembler khổng lồ build a new & more efficient kind of oto engine—& then awaits its next comm&.As a sovereign, which is assigned a broad và open-ended pursuit và allowed to operate in the world freely, making its own decisions about how best to lớn proceed—Invent a faster, cheaper, and safer way than cars for humans khổng lồ privately transport themselves.These questions & tasks, which seem complicated khổng lồ us, would sound to a superintelligent system lượt thích someone asking you to lớn improve sầu upon the “My pencil fell off the table” situation, which you’d bởi vì by picking it up và putting it baông xã on the table.

Eliezer Yudkowsky, a resident of Anxious Avenue in our chart above sầu, said it well:

There are no hard problems, only problems that are hard khổng lồ a certain màn chơi of intelligence. Move sầu the smallest bit upwards , and some problems will suddenly move sầu from “impossible” to “obvious.” Move a substantial degree upwards, và all of them will become obvious.7There are a lot of eager scientists, inventors, và entrepreneurs in Confident Corner—but for a tour of the brighkiểm tra side of the AI horizon, there’s only one person we want as our tour guide.

Ray Kurzweil is polarizing. In my reading, I heard everything from godlike worship of him & his ideas khổng lồ eye-rolling contempt for them. Others were somewhere in the middle—author Douglas Hofstadter, in discussing the ideas in Kurzweil’s books, eloquently put forth that “it is as if you took a lot of very good food và some dog excrement & blended it all up so that you can’t possibly figure out what’s good or bad.”8Whether you lượt thích his ideas or not, everyone agrees that Kurzweil is impressive sầu. He began inventing things as a teenager and in the following decades, he came up with several breakthrough inventions, including the first flatbed scanner, the first scanner that converted text lớn speech (allowing the blind to lớn read standard texts), the well-known Kurzweil music synthesizer (the first true electric piano), & the first commercially marketed large-vocabulary speech recognition. He’s the author of five national bestselling books. He’s well-known for his bold predictions và has a pretty good record of having them come true—including his prediction in the late ’80s, a time when the mạng internet was an obscure thing, that by the early 2000s, it would become a global phenomenon. Kurzweil has been called a “restless genius” by The Wall Street Journal, “the ultimate thinking machine” by Forbes, “Edison’s rightful heir” by Inc. Magazine, & “the best person I know at predicting the future of artificial intelligence” by Bill Gates.9 In 2012, Google co-founder Larry Page approached Kurzweil and asked hlặng lớn be Google’s Director of Engineering.5 In 2011, he co-founded Singularity University, which is hosted by NASA and sponsored partially by Google. Not bad for one life.

This biography is important. When Kurzweil articulates his vision of the future, he sounds fully like a crackpot, & the crazy thing is that he’s not—he’s an extremely smart, knowledgeable, relevant man in the world. You may think he’s wrong about the future, but he’s not a fool. Knowing he’s such a legit dude makes me happy, because as I’ve learned about his predictions for the future, I badly want hyên to be right. And you vị too. As you hear Kurzweil’s predictions, many shared by other Confident Corner thinkers lượt thích Peter Diamandis and Ben Goertzel, it’s not hard lớn see why he has such a large, passionate following—known as the singularitarians. Here’s what he thinks is going to happen:


Kurzweil believes computers will reach AGI by 2029 và that by 2045, we’ll have not only ASI, but a full-blown new world—a time he calls the singularity. His AI-related timeline used khổng lồ be seen as outrageously overzealous, & it still is by many,6 but in the last 15 years, the rapid advances of ANI systems have sầu brought the larger world of AI experts much closer to lớn Kurzweil’s timeline. His predictions are still a bit more ambitious than the median respondent on Müller và Bostrom’s survey (AGI by 2040, ASI by 2060), but not by that much.

Kurzweil’s depiction of the 2045 singularity is brought about by three simultaneous revolutions in biogiải pháp công nghệ, nanocông nghệ, và, most powerfully, AI.

Before we move sầu on—nanogiải pháp công nghệ comes up in almost everything you read about the future of AI, so come inkhổng lồ this xanh box for a minute so we can discuss it—

Bài viết liên quan

Trả lời

Email của bạn sẽ không được hiển thị công khai. Các trường bắt buộc được đánh dấu *