Eliezer Yudkowsky - The Challenge of Friendly AI (1/3)

At the 2007 Singularity Summit, Yudkowsky described how shaping a very powerful and general AI implies a different challenge, of greater moral and ethical depth, than programming a special-purpose domain-specific AI. The danger of trying to impose our own values, eternally unchanged, upon the future, can be seen through the thought experiment of imagining the ancient Greeks trying to do the same. Human civilizations over centuries, and individual human beings over their own lifespans, directionally change their moral values.

Eliezer Yudkowsky has two papers forthcoming in the edited volume Global Catastrophic Risks (Oxford, 2007), Cognitive Biases Potentially Affecting Judgment of Global Risks and Artificial Intelligence as a Positive and Negative Factor in Global Risk.

From http://www.singinst.org/media/singularitysummit2007
Transcript: http://www.acceleratingfuture.com/people-blog/?p=211

Category:

Science & Technology

Tags:

• singularity
• explosion
• event
• horizon
• accelerating
• change
• future
• artificial
• general
• intelligence
• eliezer
• yudkowsky
• research
• recursive
• self
• improvement
• algorithm
• brain
• mind
• ethical
• friendly
• ai
• optimization
• process
• moral

[KillahMate]

@MathematicsInfinitus As for the gap, yours is an arbitrary estimate; the AI will by definition be as intelligent as we make it. The first real one may be less intelligent than an average human, the first 'super' AI only slightly more (as smart as a human genius, perhaps). And don't forget we'll be able to reliably replicate this genius-level intelligence. The AIs you are thinking of may indeed not be fathomable by humans, but they will be by their makers - the AI descendants of our first AIs.

[KillahMate]

@MathematicsInfinitus I would certainly concede that the final project is comprehensible by the whole of the group. I also believe that the intelligence of that group - if considered *as a single (though heterogeneous) entity* - is greater than the intelligence of its members. That is, the (theoretically) verifiable 'intelligence' (whatever that means) of the group is in certain ways greater than what you would get by arithmetically adding up the individual 'intelligences' of its members.

[MathematicsInfinitus]

@KillahMate The gap between human intelligence and a "higher" intelligence I believe would resemble the gap between human intelligence and the intelligence of a dog. The resemblance is such that we as humans would not be able to understand the "language" of the "higher" AI. Things that they Fathom are humanly unfathomable.

[MathematicsInfinitus]

@KillahMate Using your example of many people working together on an AI, wouldn't you concede that the final project is comprehensible by the whole of the group of people working on a project? For some reason I feel as if there is a premise that we disagree on somewhere, perhaps hidden. It is clear to you that you are correct just as it is to me.

I suppose the point I am trying to make is that any product of human comprehension is Humanly Comprehensible. (cont'd)

[KillahMate]

@MathematicsInfinitus Though a working strong AI would undoubtedly be massively complex, to assume a single engineer need hold that design in their mind in its entirety at all times while working on it would be senseless. Countless machine and software designs are developed by many engineers working in concert, each of them comprehensively grasping only their portion of the design, and the project leads seeing only the broad outlines. Such projects are common. Why should strong AI be different?

[KillahMate]

@MathematicsInfinitus Yes, arguing by analogy is a logical fallacy. I was merely using it to clarify my stance on the development of AI - not relying on it to make my case. In my next post I shall refrain from analogy:

[MathematicsInfinitus]

@KillahMate You did not really directly address the question. Also, knives are leagues different than artificial intelligence. It is humorous that you would actually use an analogy as an argument, I am certain it is a logical fallacy somewhere.

The highest form of artificial intelligence that we can construct would be one that we could comprehend (in order to construct it.) Therefore, we cannot construct an artificial intelligence that surpasses our comprehension. Do you understand, now?

[KillahMate]

@MathematicsInfinitus With the aid of software and conceptual tools (like the kind Yudkowsky is developing) we are demonstrably capable of creating constructs that transcend our capacities - after all, a jet fighter is under a significant number of criteria many orders of magnitude more powerful (and dangerous) than any of the humans who designed it, or in fact any of the machines that built it. It would be like suggesting that, given a dull knife, we can never construct with it a sharper one.

[MathematicsInfinitus]

If I understand correctly, we artifice intelligence to operate by our constraints. With such reigns on our creation, how are they to transcend our own capacities?

[ancalyme]

...I thought Eliezer was a woman until I saw this, heh.