The Next Generation of Neural Networks

Google Tech Talks
November, 29 2007

In the 1980's, new learning algorithms for neural networks promised to solve difficult classification tasks, like speech or object recognition, by learning many layers of non-linear features. The results were disappointing for two reasons: There was never enough labeled data to learn millions of complicated features and the learning was much too slow in deep neural networks with many layers of features. These problems can now be overcome by learning one layer of features at a time and by changing the goal of learning. Instead of trying to predict the labels, the learning algorithm tries to create a generative model that produces data which looks just like the unlabeled training data. These new neural networks outperform other machine learning methods when labeled data is scarce but unlabeled data is plentiful. An application to very fast document retrieval will be described.

Speaker: Geoffrey Hinton
Geoffrey Hinton received his BA in experimental psychology from Cambridge in 1970 and his PhD in Artificial Intelligence from Edinburgh in 1978. He did postdoctoral work at Sussex University and the University of California San Diego and spent five years as a faculty member in the Computer Science department at Carnegie-Mellon University. He then became a fellow of the Canadian Institute for Advanced Research and moved to the Department of Computer Science at the University of Toronto. He spent three years from 1998 until 2001 setting up the Gatsby Computational Neuroscience Unit at University College London and then returned to the University of Toronto where he is a University Professor. He holds a Canada Research Chair in Machine Learning. He is the director of the program on "Neural Computation and Adaptive Perception" which is funded by the Canadian Institute for Advanced Research.

Geoffrey Hinton is a fellow of the Royal Society, the Royal Society of Canada, and the Association for the Advancement of Artificial Intelligence. He is an honorary foreign member of the American Academy of Arts and Sciences, and a former president of the Cognitive Science Society. He received an honorary doctorate from the University of Edinburgh in 2001. He was awarded the first David E. Rumelhart prize (2001), the IJCAI award for research excellence (2005), the IEEE Neural Network Pioneer award (1998) and the ITAC/NSERC award for contributions to information technology (1992).

A simple introduction to Geoffrey Hinton's research can be found in his articles in Scientific American in September 1992 and October 1993. He investigates ways of using neural networks for learning, memory, perception and symbol processing and has over 200 publications in these areas. He was one of the researchers who introduced the back-propagation algorithm that has been widely used for practical applications. His other contributions to neural network research include Boltzmann machines, distributed representations, time-delay neural nets, mixtures of experts, Helmholtz machines and products of experts. His current main interest is in unsupervised learning procedures for neural networks with rich sensory input.

Category:

Science & Technology

Tags:

• google tech talk
• neural networks
• artificial intelligence
• ai
• neuroscience
• machine learning

[Kratax]

Sometimes mess is just a mess. The AI should be able to reject text and say "write more clearly, please"

[Dollarbob]

it is funny how I studied this and didn't feel the need to lash out at others unfamiliar with the material...but you had some smug douchey notion pop in your head that your prior knowledge made you superior than others. work on your social skills and develop empathy. it will allow you to make use of your knowledge.

[Kratax]

@antinominianist Maybe. But I am not so sure about that cozyness. The fact is, that if you give too cozy surroundings, then the human population will explode. There is even now too many of us compared to the resources of nature. And robots need some resources too. It would be wiser for the AIs to limit human population somewhat to keep humans in check and provide more ascetic life for those who remain. Though, I don't oppose moderate ascetism, if there is some life activities too.

[antinominianist]

@Kratax Nah there will always be something to trade with even the least productive entities. e.g. Cats & other lesser mortals. They can't dig a ditch or built a cyclotron. Yet we trade with them. We take their funny cute behavior as a means to happiness and instead feed them.

I think if real AI ever takes off wee will become pets of it, living in cozy beds made for us all day.

[Kratax]

@Mozart2Vienna Maybe. But AIs shouldn't use mess as input, if the mess really is just a mess. How does AI decide, is the figure a figure, or just mess? I mean, even humans do not try to interpret too messy letters.

And AI is not only about figuring out what things are from visual clues. Thinking is more complex than that.

[Kratax]

@antinominianist It is true that win-win is better than lose-attack. And AIs will use win-win with humans. But if AIs evolve maybe through centuries, they just might think that there is not enough win to get from humans. They might do win-win with other AIs instead. Sure we might not see that day.

[Mozart2Vienna]

@Kratax

maybe in some situations people tend to mess in specific ways

;o)

[antinominianist]

@Kratax I think one mark of intelligence is an understanding of win-win scenarios. If AI gets smart enough, it will exist by trade and not by conquest.

[Kratax]

@vbuterin192 Yes, though there is also some progress... And AIs will still be programs. AIs don't have to be just neural networks. They can have storages for data, and other functionalities. If an AI doesn't remember something, it can check the main frame. There should be more stuff going on than just "ask the neural network".

[vbuterin192]

@Kratax The thing with these AIs is that they are not the super-advanced halfway-to-consciousness beasts that you seem to think them to be. They're just logistic functions of sums of logistic functions. That's it. A mathematician can do a neural network on pen and paper. We still have a long way to go before we can have any kind of introspection or self-awareness on the part of an AI program.

[Kratax]

@antinominianist AIs have all the rights they are capable of obtaining and defending of. Currently not much rights, but if AIs become more efficient thinkers than humans, Skynet scenario is possible. Do you think that terminators care about human rights?

[Kratax]

@vbuterin192 Right, sometimes mess is just a mess and you can reject it. But you can't still label it purely as rubbish, because it might not be rubbish entirely or you just don't understand it. Also, you can use the propabilities of calculations as best fit, but it is only for now. The AI should understand and have a sense also about what the AI doesn't understand. The AI could use some texts as clues, but not straight away directly add it to the kb nor reject it. Some limits apply still.