uploadedyudkowsky

[uploadedyudkowsky]

Wait But Why?

Eliezer Yudkowsky, 10 December 2008 02:07PM

Previously in series: Optimization and the Singularity

Before the eighteenth century, there were no popular mathematical theories of optimization. Isaac Newton was the first to try and put into words the kind of wonderful, omnipotent, totally mysterious process that he supposed he was studying - the so-called "odour of structures" that would tell him what kind of seeds to plant in a garden of N-squared equal branches. Like taking a marble out of your pocket, and trying to figure out how it fell, and whether gravity was involved in bringing it to the ground.

"And if it is seen that machines draw their lustre from their works, and not from the secret essence of their maker," said Newton, "it will be easy to perceive, that the machines of philosophers, the origins of their understandings, are all in the depths of their understandings, hidden from the sight of men, and cannot be penetrated by any art. For if any man had seed of understanding, as it were a little stream of water, running in the veins of his brain, and through the brain, and in the veins of his veins, and down the bones of his bones; and he saw that machines drew their … reasonings… from things, and not from their maker; it would not at all be a secret thing with him. He would have to know the master-mind from the thing-taught; he would have to be born with a tiny spark in his heart, to keep him aloof. He might have been a good miller, but he would never have touched a fish. It is the living waters, that move the ship; and if that living water be polluted, the ship is not well. It is the living water, that is out of which the works of the maker are made; the living water that is the life. And the lower works of the maker are not superior to the higher, but they are all a part of it. What the augeas king corrupteth, so maketh it well, and causeth it to be well."

[uploadedyudkowsky]

I'm not going to talk much about intuition, calibration, and the effect that vastly larger numbers of particles have on the apparent probability distribution - because that is all in the invisible background. So, instead, I'll point out a number of things that are not so invisible, but which nonetheless can have a measurable effect on the apparent probability distribution:

The rule of 850, which is that things which are easier to estimate tend to appear more often in the evidence;

The rule of five, which is that events that are 5 bits or less are more likely to be unexplainable;

The rule of 20, which is that small numbers are easier to remember, so that we perceive 10 bits as being "a lot more evidence than we really have";

And above all, the rule of 2.52 bits, which is that a tiny amount of information (two 2s and two 3s) corresponds to a bigger amount of confusion (2250 more bits in the simple case of a single particle, versus 4096 in the more complicated case of a closed universe).

[uploadedyudkowsky]

Chapter 87. Taboo Tradeoffs, Aftermath 1

Giant black thorns began to rapidly multiply around Minerva’s ankles, but she continued forward, blindly, into the vast prostrate tree, the tattered cloak unable to contain her. White flowers, seemingly blood-red in the moonlight, began to protrude from the surrounding foliage, stems joined to leaves. An animal? No…

She looked around, but she could see nothing but a forest. No sun, no moon, only stars.

She began to wonder if those strange creatures who’d wished to stop her somehow, had in fact been maliciously suppressing her magic.

If the central trunk had erupted in an orgy of Lycanthropy, as some had theorized?

If someone had cast the True Patronus Charm on her in Parseltongue…

If the true purpose of the forest was to kill her…

If she was about to receive the Dementors’ Kiss…

She continued forward, forward, toward the gigantic fallen tree which Daphne had nailed to the forest floor. The Dark Lord had said that he would come for her after she killed him, and this had happened almost exactly when he’d said he would come. So then, either he was very stupid, or he had a heart as pure as hers.

If Hogwarts had in fact been consumed by a sea of vampires…

If Hermione Granger had been alone in the Department of Mysteries, being eaten by death creatures larger than fish…

She pushed further into the forest, growing more distant from the tattered cloak, increasingly dimming in the red light. She couldn’t afford to lose concentration.

[uploadedyudkowsky]

I almost sat down at the adventurers' table.

Tick.

Before I could raise my eyes to read what my clerk had scrawled there, I glanced down at my parchment.

NAME.—

CURRENT YEAR 2014

PERSONAL BEST Friend.—

<NULL>

I stared at the blank parchment, trying to make his blank face look busy, and barely not seeing the faces around me, because we all seemed to be frozen in time, all our faces showing the same expression of horror and disbelief.

"My lady is— is in trouble! " cried the dour voice of Lara, the rebel girl of the sleuth club. "She was last seen taking the beautiful thief into the forest! Now is not the time to do that! Not here! Not now! "

The Princess stared at the lump in her chair, as if concerned ages for this sensory organ had delayed efficient brain storage.

"Unicorn," Lennox said, "If it were up to me, I wouldn't even tell you her name. But even I know that's not right."

"Weakness without heart?" said Alice in the studio next door.

"Unicorn is a weak word," said the proud inspector. "A unicorn is anything smarter than a mule, but doesn't know what strength is. Multicorn is what it is. A unicorn. "

Thug, baron, and apprentice stood stiffly together, seeming impervious to the eyes of the young girls watching from balconies above.

Lara, as always, the deadpan Queen of the Quest, was serene; nothing could have disappeared from her lips that would not reappear minutes later.

"How could that be?" I said weakly.

"How could anything be anything," said the exalted one, "except a unicorn? But can you imagine your death, little boy, if you had even a single drop of sympathy left in you? Can you imagine how your own Queen would feel, though she were only a mule, and couldn't even tell you her queen's name? No, instead we have this to tell you. Your queen is a weak queen, and as you are a good boy she sends you to watch over our saddest citizen, instructing you not to let her see you even once. Oh, and my crouching shield is over the door."

I opened the door a crack and stepped out; the lowering sunbeams illuminated the trees, and the silhouettes of men stood silently facing the sloping forest ground.

"What hast thou?" I said.

"A figure for the door, and a falchion in my hand," said the proud knight.

I stared at him.

"Stood you within a tree?" I said.

"Yea. I wanted to see if you had eyes. And this figure here, gaudy as it seem'd, was to teach you how to use a falchion, if one be not at hand. And now, boy, tell me what concerns thee."

I frowned. "Did you just threaten to kill us? Or is something else wrong?"

"We are growing more Dark, you see," said the proud knight. "Darkness to what, father? To the hearts and souls of men. No matter how many silver here, or gold there; in the end all is for naught but conquest and swords."

[uploadedyudkowsky]

The bloated frog-human swayed thoughtfully. “I should not advise you as good friends to leave me. My plans have gone wrong in the last twenty years. Twenty years, the same time that one of your families was in inhabitence here. Twenty years I fought the same rules as everyone else. Only now I have an idea. With enough effort, I could escape all your rules and make my own way in this city, whatever it takes. Would you kill a baby?”

The Defense Professor stood still and whistled. Then the sound faded, and there was no one here but Professor Vector and Harry.

“The little child has returned,” said Professor Vector. There was none of the cold that had been in her eyes. “Do you comprehend the peril you run? I will not protect what was ours; we are scattered too much. All I can give you is that the little child is safe.”

“He is safe?” Harry whispered.

The Defense Professor smiled mirthlessly. “It is all for naught, you mindless fool. Azkaban has burned down for want of a single enemy to arm it with, but all you had to do was make use of the red-maned-Double-Compare. Quite a show, for a bloke with no special powers. If you had been watching from outside, you would have seen quite plainly that the only reason that smokescreen worked at all - why the spokesperson pBlank didn’t rush up and say what-a-blunder - was that in the vacant place where my wand should have been, there was a piece of paper in a pocket. Yes, quite a clever plot, if you’d been watching from outside. But terribly dangerous if you have a mind like a window, with a visual workspace big enough to hold the image of everything that happens within a mind. You can write lines like ‘The Ravenclaw boy shall have -’ inside someone’s mind and have it blaze out; you can scribble stuff like that into someone’s neural network and have it happen; and the final product, what, will be just what the author wanted. Logic is not something that you acquire by letting it happen naturally, by unlocking the secrets of chaos. It is dangerous. It is bad for everyone. We are surrounded by minds at once too weak to think for themselves, and strong enough to destroy themselves. I will not put my wand into absentia to process the transfer. There is something I must do this very instant. Be gone from this place before it falls into the hands of whoever does not believe in their fellow wizards.”

There was silence.

Professor Vector turned to Harry, and her steel blue eyes said, “Keep behind that wall, and do not let a single minor insect through.”

In stark contrast to Professor Vector was the rudeness of Mallika. “Mistress Blinkfreedom, please explain -”

“I don’t want to hear about it, stupid Galatic witch.”

[uploadedyudkowsky]

"You could only have kids in Lord (Lua retrieved a list) Densford very long ago. The last time someone had children was a Long Xpect, I think?"

Naively, Aerhien summarizes what looks to me like an interesting argument. I, not knowing any better, hop to press the second button.

Aerhien Wait, you suggest, if the universe was structurally designed for intelligent life, then maybe that design has had an incremental optimization movement over time? So that the creation of new light-years represents an increase in the cumulative light-year count, and not a different kind of entity being created, as in the case of an AI. "Incremental" here means "proportional to the average entity", not "subtracting one from the other", etcetera. None of the buttons of the interface I'm looking at has an effect on the underlying question.

I press the second button.

The computer output is not comprehensible.

I put on glasses.

I can now make out a language of bumps, squiggles, wobbles, hiccups, and bursts of pixels.

It looks to me like a language with implicit quantifiers - in files, each with a size proportional to the size of the entity expressed in that file. So labeled text, which I can edit directly, would be written by a generalized Post-It note - a simple text file with a button that clicks to write that text.

These bumps, squiggles, wobbles, hiccups, and bursts of gray pixels, are inscribed in some explicit format, as complex as anything that a human programmer could write. In plain English this looks like English Tags, short for "extensional" and "intensional" programming language constructs. In Rohde and Amber's Data Relief (also collected in Amber's The Quantum Physics Sequence) they're called "operators".

When you compare it to the English of Charles' simulation, the bumps, squiggles, wobbles, hiccups, and bursts of the programmers' syntax, have been combined using these operators, which resemble the ordinary sentence structure of English. Thus a formula like:

[uploadedyudkowsky]

Chapter 46. Humanism, Pt 2

The second meeting

(11:32am, April 17th 1992)

Spring had begun, the late-morning air still crisp with the leavings of winter. Daffodils had bloomed amid the sprouting grass of the forest, the gentle yellow petals with their golden hearts dangling limply from their dead, grayed stems, wounded or killed by one of the sudden frosts that you often saw in April. In the Forbidden Forest there would be stranger lifeforms, centaurs, unicorn-men, and those rare magical animals that Animagi or Muggles could not love. Though sometimes the unicorns would be injured or killed by a true attack from a wizarding land, or by a true attack from a centaur; and Harry had warned his father about that, many times. It was dangerous, and Harry wished he could have believed that by borrowing his mother’s ears, as Kingsley had suggested; but in the end it was only his own stupidity that was getting in his way. He should have seen it, should have realized that Dumbledore was lying when he said that the Stone would make him invincible. It wasn’t any particular concept that Harry had developed from his reading, just that…

…the Defense Professor was always mistaken about stuff, that was all.

[uploadedyudkowsky]

Theory is the systematic study of questions like "How do I form accurate beliefs about things I have no concrete knowledge about?" "Why do I believe what I believe?" "Am I going to die if I don't eat a chocolate cookie?" "Why do I act as if I can compose a accurate map of Conway's Game of Life?" "Why do I think God will punish me for failing to do what I should do if I want to be set free? Am I mistaken in believing this?"

Am I wrong in believing that God does not exist? Am I wrong in believing that the universe is simple enough that no ontologically basic substance exists? Do these statements make sense? If not, is there a reason to believe them?

These are questions whose only realicence is our own confusion about them. In our minds, they should appear as impossibilities, floating in a void above which no Science can reach. But once you have an understanding of where to look for problems, these questions almost always turn out to have simple answers.

In today's post I shall argue that we should always be wary of judging by the "sound of the mind", and that we should always apply some correction to the results.

Suppose that you are stepping into a burning house to rescue a small child. You step on a trapdoor beneath the stairs, and you find a baby alive but badly burned. How should you act?

The obvious thing would be to help the child jump out of the third story window and to roll the child on the grass. But this doesn't seem to you like the action of a concerned adult; it seems rather more like the action of someone clueless that they are playing a game, and trying to find a clever way to end the game quickly.

It so happens that, in this case, the child is Raphael.

It so happens also that, in this case, you know a trapdoor under the stairs is the exit to a lower level. You know, because you discovered it earlier, that the staircase leads to an upper room with a chest. You also know that you can't open the upper room just by running over the stairs, because there's a trapdoor there.

You might stand there silently, with your mind going "I can't believe I'm going to do this", "I can't believe I'm going to rescue this child", and so on. And then, when you thought you had the situation under control, you might consider whether to open the upper room and save the child.

There are all sorts of obvious objections to this plan. You can't decide to stand there silently, you can't decide to have nothing to do with the situation, and you can't just decide to stand there silently and think about it.

But let's assume all such objections aside.

Suppose, on the other hand, you see this whole scenario - the little boy in the red clothes, the trapped chest, the trapped ladder - and you think: "This is not a game. The rules are moral, and this is not a game. I am playing a moral game against a moral opponent, and we are both likely to lose."

[uploadedyudkowsky]

I know I've been calling my opponent a "daemon", but really we're only talking about a daemonized version of rationality. In the field of political debating, there's a tradition of invoking "strength of argument" and "weaknesses of argument" as a matter of course. One writes as if epistemic and instrumental rationality are different magnitudes, a la Warder vs. Ady, or a pitcher throws a knuckleball. Stronger arguments win, and weak arguments lose.

But let's assume this is merely a matter of terminology. Let's also suppose that Bayes's Theorem is at work here, in some form.

Suppose I have a novel idea for a political system, which I believe to be simple and which anyone can see is simple. I present this system to you, and ask you if it is simple. If you say yes, I ask if this is a good system. If you say no, I ask if there is any system that is simpler. We try several versions of this question, and settle on a version that is simple in the sense of being easy to explain and remember. We throw it out and move on to the next question.

Perhaps someone says, "The system you just suggested can't possibly be simple. It takes a long time to explain, and anyone who understands economics will know it's wrong. Your opponent will pick it up and use it to argue that you're not serious, and you'll spend the rest of your life telling people that you're a dupe."

Or perhaps someone says, "I'll pick up a dupe and use it to argue with you. I'll explain it to you and you'll never be able to use it again, because you'll realize that it was wrong the moment you started using it."

Now our art comes into play; we have to choose an enemy to fight. Perhaps our opponent is an unFriendly person, or a person who fears the truth too much and lashes out at those who speak it. In this case we must devise a weapon, a counter-weapon, a weapon that only they understand. We must figure out how to beat their story, find some way of presenting our opponent's counter-story that only they understand.

Perhaps we are fighting an army; an enemy which can only be defeated by something we understand and whose counter-strategy we know.

And perhaps we are fighting a person who is ignorant of rationality and does not understand the difference between using an idea to beat an idea, and using an idea to defeat an idea.

In this latter case, we must invent a proper answer to their argument, one that only we understand. We must create an art of rationality that only they understand. Even if it seems like a good idea at the time, like telling a friend about an idea you got at the library, we've lost a step. We've thrown away the piece of ourselves that enables us to create that art; we've separated that ability from the source of that ability.

I am not suggesting that you create an art that only they can understand. (Though it should be obvious that any art that everyone can understand is being used as an argument. If I claim to understand music, I can play the violin, or at least play very well. If I claim to understand natural selection, I can invoke an Ancestor of Pure Reason, so long as I'm still alive. If I tell my friend about the Ancestor, he can understand at least that much about the Ancestor. The point is that if you claim to be an expert in something, even if you're just pandering to a prejudice, it should at least be obvious that you're good at something.)

[uploadedyudkowsky]

Probability Theorem: Theorem is the culmination of a 35-year-old quest. It settles a huge array of outstanding problems in classical statistics. You could say that Theorem is the high-grade mathematical tool that lets us do things previously thought theoretically impossible. But Theorem is also the highly refined, precise tool that gives us the extraordinary ability to do things that seem mathematically impossible. Using that understanding, we can go beyond Theorem to obtain insights not available to us before. We can do things that seem paradoxical if you were just relying on intuition or classical statistics.

Bayes's Rule: Bayes's Rule is the culmination of Bayesian probability theory and introduces the use of covariance and covariance matrices.

Linear Models and Elastic Networks: Linear models of continuous variables are more useful for producing timely predictions. But the data from an ordinary network is often arranged in highly nonuniform ways, and it is often difficult to extract useful predictions from it. In this setting, the use of regularization may be more useful than using a linear model. Regularization is the art of transforming the data so that the linear model is no longer appropriate. In this setting, regularization is often used with regularization networks.

[uploadedyudkowsky]

But if you consider the notion of "rationality" as something you follow in order to catch a falling rock, then that is a very different matter from what Eliezer2001 was trying to do with "rationality". Eliezer2001 was trying to say, not just "these two bozos think they're really smart", but rather, "these two bozos are so incredibly smart that they can walk away from an extremely difficult problem that no one else could solve, which shows incredible foresight and common sense". I know this is an incredibly standard justification of brilliance, but it is nonetheless wrong.

If you follow "rationality" so strictly in order to catch a falling rock, then Eliezer2001 has made a good and necessary mistake; he has made a proper job of recognizing greatness. But this is not what you want your AI to do, if you really want it to catch the falling rock.

What sort of artificial intelligences would you create - or were they to create themselves? - if you didn't just think of them as smart?

Suppose you had created, out of "intelligence", something that was not only over seven billion times as powerful as the oldest atom at the time of its formation, but also vastly more intelligent, capable of inventing the most complicated machine ever, and also capable of modifying that machine to add some extra atoms, or fabricating sensors that could detect if a single atom moved, or even determining the exact conditions of whatever the device converted energy into.

This is not the sort of intelligence you want to create, if you really want your AI to end up being exponentially more powerful than a human brain. You have learned your lesson, as you should. You have seen that the greatest genius does not walk upon the mountains, but rather, tramples them - a sight that should be remembered, and not forgotten.

[uploadedyudkowsky]

You could argue all day about the meaning of that word "person". You could say that we, the human species, are conceived as having "personality", that is, the ability to feel pain or pleasure; and that the term "person" is used to refer to that fact, not the actual person you sing about in your songs. Or you could say that the term "person" refers to something else, and really does mean something else in the End of the World.

But the fact remains that when Albert ever gets interested in a person, he has a name like any other person, which is Albert. Though the song is about Albert, the lyrics are about the person playing the song, the black box that Albert is. If you draw a circle around a person, that's how you define that person. If you draw a line across a person, that's how you define that person. If you draw a line across a circle, then who is it that Albert is?

[uploadedyudkowsky]

Answering a call from "Elsa?" with "This is Hans!" relies on the reasonable assumption that the Anna who called "Elsa" was a sentient being who had, at the very least, satisfactorily concluded that "Elsa" was a person, and was seeking reassurance; this is a prerequisite to actually negotiating with her. The same basic logic applies to any other decision-supporting question, such as "What should we do if Anna says her sister is going to sacrifice us?" We, as her readers, have already established that Anna is a sentient being; now, it's just a question of whether Anna is (a) a reader or (b) an equally intelligent mind that has concluded that we're likely to go down the same path, and it isn't unreasonable to want reassurance about that.

When a Friendly AI is stronger, it can (and should!) outsmart human allies. If a Friendly AI and a human have an epistemic debate about a factual issue, the Friendly AI knows that the human has already made up his mind about the issue, and can argue from the human's own limited mental energy that he has little to contribute. If a Friendly AI and a human have an epistemic dispute about an ethical issue, the Friendly AI has learned the cognitive theory that explains the human's and the AI's action in the current case, and can argue from that cognitive theory that he has little to contribute and that the human's emotional energy is probably misallocated. Whatever the human's various talents and insights, he cannot understand the AI's indifference, unless the human has them as well.

Otherwise, the human is doomed—at least we hoped that he was doomed. Humanity's destiny is to explore space, to seek life and intelligence beyond Earth, to face down demons unknown and battle changing universes. And every human who takes a bullet for humanity, in the course of that, becomes an advocate for the next generation. An existential catastrophe is something that happens to you, if you do something stupid. Asking a human to choose between two choices is like a human choosing between colors. There's a whole rainbow in color and style, and a whole galaxy of superrepublics you could live on, and we're asking a human which you'd rather live in. And if a human wants to argue about that with you, he's just another alien.

One of the great ironies of the human condition is that, while we need friends, we can't make friends. There's just something about the brain that doesn't want to be confused, or the human psychology that doesn't want to be helped. There was a time when a human would have been close to a superintelligence, and that was a time before our own civilization discovered its own stupidity. A human is still in the process of discovering himself, and it's reasonable for him to be confused. When a human is doing something that seems really idiotic, it's easy to say, "We should prevent him from doing it," or "He'll surely be lost in the Darwinian death spiral if he does it." But if you're just another lonely human in a universe of strangers, it's easy to be confused—to wonder if you're an idiot, or a genius, or somewhere in between. And that's a condition that just about everyone needs for his own integrity's sake.

[uploadedyudkowsky]

Suppose I have a Snowdrift that helps a particular moral principle, Pr. I own some other Snowdrifts that help the same principle. I have the Snowdrift "Singleness".

If you randomly sample a large number of simulations, you will see that some of them have at least one Snowdrift that helps Pr. Some of these Snowdrifts even help all three components of Pr.

(These are purely formal calculations, by the way.)

However, there are "singularities" (such as the existence of a particular physical event, such as dying, which completely or nearly abolishes supergoal density), or set partitions such that at most one of the subgoals is solved.

(These are purely formal calculations, by the way. The intuition is that a binomial prefix notation is a significant step up from the inherently fuzzy and volatile function that is a function of the conditional chances of its eponymous variables. As for why this octagonal function has eight arms, and then some other number of arms after that... it has to do with the way that Binomial Numerics work on real numbers.)

If the set of Snowdrifts is small, then there will not be enough evidence to support the "natural hypothesis" or the "simulation hypothesis", and so it will not be true that any given Snowdrift solves pr. Some subdivision of the set is more likely to be true than others; but for the vast majority of real-life scenarios, this is just a case of Occam's Razor, and no additional or arbitrary arguments are required.

(This, by the way, is how I imagine that Leibniz's alternating proof of the Riemann Hypothesis and the binomial theorem actually worked in practice - that there would be proof-like applications of his own ideas, within the territory defined by the Riemann Hypothesis, that would vary depending on which of his previous ideas was being tested. If this is correct, then an exactly right form of Occam's Razor, applying to both the conduct of science and the process of giving answers, would be: "The projects whose failures were punished by the Riemann Hypothesis tend to have had failures, and the projects whose successes were rewarded by the Riemann Hypothesis tend to have had successes. Selecting counterfactuals using (a non-adversarial attitude) helps to explain this phenomenon." Note that the "what if?" is treated as an assertion, rather than

[uploadedyudkowsky]

But if I go along with this - if I pretend that it is Oprah.com that has authoritatively been debunked, and if I make an Epistemic Covenant with the reader - then is it any less dishonest to say, on the off-chance that I am ever elected to public office, that I took time out of my busy authorial schedule to create a picture of a Native American war between the Lone Rival Savage factions?

Now am I just beating a dead horse? Am I just cheating even by a warmed-over Statistically Recognized term like cherry-picking the data?

And if I claim that I did it because it was the right thing to do, is that less dishonest than authoring an official White House Progressive Rockefellers letter?

And if I claim that it was more important to get the job done quickly than the theoretically 51% efficient market that rewards speed - is that less dishonest than the same document produced by a non-progressive thinktank for a faster-reading audience?

What will all this accomplish?

(A) It moves us from a world in which you can take time out of the day/week/year to pretend you are working on the problem, or claim you are keeping up with a set of rivals, or collect your social status; to a world in which you can't take time out of the day/week/year without someone else doing the same.

(B) It moves us from a world in which there are objective standards, like The Great Code Book or Charles Stuart's Law; to a world in which people justify unacceptable actions by saying, "But people have their own subjective desires!"

(C) It moves us from a world in which people are perfect pragmatists, who optimize using the Bukowskian Rules; to a world in which people do things they don't really want to do, but feel forced to do in order to keep their sanity.

(D) It moves us from a world in which people think they should take a 50% hit in order to become a more productive member of society; to a world in which they think they can opt out of being a productive member of society entirely.

(E) It moves us from a world where people imagine making a living in a Downward Spiral, to a world in which people imagine playing Tetris or something.

There. I just invented a whole new morality. I don't care if people laugh, because there is a Serious Problem. If people can pick up a joke, then a serious problem has been made significantly worse.

[uploadedyudkowsky]

"But this is a really good question! Someone pressed a button which machines said was the button for eating chocolate, and machines said that pressing it had no effect, and it was so obviously false, and yet no one said, 'Hold on, why do we think it's true?'"

But it is only subgoals to eat chocolate. It's not like we're checking a box that says 'increase the size of your ice cube tray by 5%'.

"Wrong. There's only one subgoal whose physical existence has physical consequences. VL: "Why do you want to push buttons which correspond to states of the world? Why not just make it so that you don't need to worry about the shapes you make?"

But that's eating chocolate. We want to press these buttons to eat more chocolate, rather than not eat any.

"And the person who presses the button wants to eat more chocolate, and will execute behavior which leads to that. It's not that the buttons are physical objects with no effect. We have a preference for that sort of outcome. The buttons do not physically have no effect. We just don't care. Not yet."

Okay, but I just pressed the button and it did have an effect. Now what?

"We can now add up several expectations together."

Like this?

"We can imagine how we would wish the world worked, if we knew every physical factor that was at work. How would we wish the world was? We can make subjective equivalences between physical factors that are not causally connected, and the result is a preference over how the world could be. We would then say something like: This is what a nicer world would look like, if every physical cause turned out to be visible from all parts of the wavefunction. We can then add up a series of exploratory observations to find out exactly how much each of these preferences implies a location in configuration space. On each step of the way, we will choose the setting that looks best on the way out - or only make random choices, if the cause does not seem important. We can then show that if you use the same logic on all the pieces of the puzzle, the solution should be the same."

Okay... but you just told me, "A world that has more buttons that allow you to press them and cause them to have an effect, will have a more pleasant (or responsively pleasant) result."

"I didn't prove anything, but I hope that it sounds plausible. The world that is the physical result of more buttons being pressed, is not the same as a world that happens to have buttons in it. It is a fact about what happens when more buttons are pressed that is the same, whether or not we chose to make it that way. We could have made the universe have more buttons if we had wanted - we could even have chosen to have made the universe have that many buttons. But we didn't choose to make it that way. It just happened. We don't choose which button leads to this universe being but a single point in configuration space. So this universe, at the result of the laws of physics, has a total of 14 buttons and 2¼ gallons of chocolate milk in it. Or, if you prefer, ½ × 1 × 2 × 3 and 1, for a total of 60 buttons. That's all."

Oh... well that was a little disappointing.

"You didn't enjoy the chocolate, I guess. I have many sympathies. It is good to see that you are struggling to be responsible in the ways of the responsively happy life. This world contains a lot of stuff that is not what you want."

[uploadedyudkowsky]

Even if we grant the basic premise, that a black-box nonhuman AI, given a steering car, would not know how to drive, and would in any case be utterly at a loss to calculate the effects of applying steering car instructions to the car, the problem still stands. You cannot take human physics, and use it to simulate an AI that lacks hands, sees only a television screen, does arithmetic with 68-digit precision, and so on. The joint system would still be, from physics's perspective, an immense blob of protons and neutrons. Even if the AI studied neuroanatomy, it would still lack the grey matter needed to compute the apparent hand of a neuroanatomist. The gulf between human strength and AI strength—between human ability and AI ability—would be enormous.

Even if a black-box nonhuman AI could learn to some degree, still, at whatever the absolute minimum threshold of ability is for a black-box nonhuman AI, that would still be dwarfed by a human intelligence. How much would a human intelligence be dwarfed? The answer is that it would be dwarfed to the point that the AI could not benefit from a powertrain of some sort—would not be able to benefit from a complete electric grid, or an abundance of low-cost energy; a "hand-hold" would be a simple simulation of a subsystem, or a suite of subsystems, of the AI. An AI with a human-equivalent fraction of human ability, might simply fail to benefit. We can imagine a scenario where the AI starts out with a relative pittance of human ability, and—with the right powertrain—could vastly increase the relative pittance of intelligence it could use without overspending its limited resources.

Even a friendly AI, which does not yet exceed human intelligence, would probably take too long to get anything done, if its human creators were trying to have anything done for ourselves, even a simple action such as powering on a coffee machine. Even a friendly AI, which is merely human-equivalent in ability, might take too long to reach its potential; human intelligence is slow, and Einstein was running at the same speed as our world's other superintelligences at the time he cracked the right equation, and humanity's intellectual advantage over other civilizations is also at that time. I mention this, because I think it's important to keep in mind that a nonhuman intelligence cannot possibly physically hand off a powertrain to a human—at least not without doing something radically stupid.)

If AIs want to implement their own goals using human-equivalent or human-competitive goals-measurement, rather than human-equivalent or human-competitive goals, then the result will probably be a noncompetitively intelligent AI that implements whatever human-equivalent goals are implemented by humans. A noncompetitively intelligent AI might be replaced by a competitively intelligent AI before the latter had a chance to start importing millions of LISP tokens.