The Next Generation of Neural Networks

[freemeson]

I loved to listen to the talk. I'm from high energy physics, where proper probabilistic methods are more than inevitable. Still, most of my colleges are stucked with the simplest MLP models and keep ignoring the past twenty years' results in pattern recognition.

I know the talk is from 2007, and probably the work on the high dimensional unsupervised image classification has a lot of progress, but let me leave some of my comments here.

[freemeson]

Generative learning may help validating the network, and the dimensional reduction and mapping technics you use is fantastic, but something is still missing from it. Because of the high number of dimensions, the network start learning the hypersurfaces with huge uncertainties, so it still has the curse of dimensionality. You need reinforcements, to test the assumptions, in which parameters you should do sample generations. I liked the talk, btw.

[PowerUpPlease]

I can't wait till I can understand this.

[jayakrishnaalapati]

Has anyone got Slides for Above Lecture ?

[randomsubspace]

Move over SVM. Wow.

[ElectricTigerFart]

At some basic level I feel this nice partial representation of we think laterally.

[tomlensky]

Beautiful and very informative, going to try the codes on my data! 

[asdfaxsdgg]

What pig cells is he talking about?

[nrdc3pinactsyt]

how about Saddam's PC?HAHAHHA

[rogeroney3]

how did he get logistic fantasy's from binary neurons? the fantasies of the digits are not entirely binary.

[Svourdroculed]

"That's the algorithm George Bush runs and it doesn't work very well." -- HAHAHA! perfect.

[lzrd0113]

This all sounds cool even though I don't understand any of it..... Just a little bit.

[n0us3rn4m3s4v41l4bl3]

This is 4 years old. can someone point me to state of the art on neural networks? thanks. I don't know about them. This talk was helpful. But I want to actually write something and want to get up to speed ASAP. Thanks.

[theapeman10]

@n0us3rn4m3s4v41l4bl3 - to ask your question as regards a video to start nets watch?v=966b0IgA3DA good introduction in C# he has a good book if you know nothing about nets it's important to learn the standard models and basics first and learn the maths- as they are build up of miix of simple models and hence under different modes. You then need to stick to one problem until you really understand how every change you make changes your results.

[Kratax]

Sometimes mess is just a mess. The AI should be able to reject text and say "write more clearly, please"

[vbuterin192]

@Kratax You can always just make "incomprehensible rubbish" one of the options for the learning algorithm.

[Kratax]

@vbuterin192 You can't just label something as rubbish. You can label it as not recognized and propably nonsense, but the text would still have been written - maybe for some purpose even. The AI should still be able to analyze the text for some meanings in case the AI doesn't get any clarification from the author. But the analysis would be done only if the text is known to have some relevance. You don't analyze random texts just for fun.

[vbuterin192]

Your older post, "Sometimes mess is just a mess. The AI should be able to reject text and say 'write more clearly, please'", implies that you want the AI to reject the text entirely, and that the writer is there to respond to the feedback. That's what I was offering my suggestion as a response to.

Now, however, it seems like you want both a measure of quality and a probable meaning, and if you want to do that you can just return the probability that the algorithm provides for the best fit.

[Kratax]

@vbuterin192 Right, sometimes mess is just a mess and you can reject it. But you can't still label it purely as rubbish, because it might not be rubbish entirely or you just don't understand it. Also, you can use the propabilities of calculations as best fit, but it is only for now. The AI should understand and have a sense also about what the AI doesn't understand. The AI could use some texts as clues, but not straight away directly add it to the kb nor reject it. Some limits apply still.

[vbuterin192]

@Kratax The thing with these AIs is that they are not the super-advanced halfway-to-consciousness beasts that you seem to think them to be. They're just logistic functions of sums of logistic functions. That's it. A mathematician can do a neural network on pen and paper. We still have a long way to go before we can have any kind of introspection or self-awareness on the part of an AI program.

[Kratax]

@vbuterin192 Yes, though there is also some progress... And AIs will still be programs. AIs don't have to be just neural networks. They can have storages for data, and other functionalities. If an AI doesn't remember something, it can check the main frame. There should be more stuff going on than just "ask the neural network".

[antinominianist]

@Kratax Now you are talking about rights of AI. They have none.

[Kratax]

@antinominianist AIs have all the rights they are capable of obtaining and defending of. Currently not much rights, but if AIs become more efficient thinkers than humans, Skynet scenario is possible. Do you think that terminators care about human rights?

[antinominianist]

@Kratax I think one mark of intelligence is an understanding of win-win scenarios. If AI gets smart enough, it will exist by trade and not by conquest.

[Kratax]

@antinominianist It is true that win-win is better than lose-attack. And AIs will use win-win with humans. But if AIs evolve maybe through centuries, they just might think that there is not enough win to get from humans. They might do win-win with other AIs instead. Sure we might not see that day.

[antinominianist]

@Kratax Nah there will always be something to trade with even the least productive entities. e.g. Cats & other lesser mortals. They can't dig a ditch or built a cyclotron. Yet we trade with them. We take their funny cute behavior as a means to happiness and instead feed them.

I think if real AI ever takes off wee will become pets of it, living in cozy beds made for us all day.

[Kratax]

@antinominianist Maybe. But I am not so sure about that cozyness. The fact is, that if you give too cozy surroundings, then the human population will explode. There is even now too many of us compared to the resources of nature. And robots need some resources too. It would be wiser for the AIs to limit human population somewhat to keep humans in check and provide more ascetic life for those who remain. Though, I don't oppose moderate ascetism, if there is some life activities too.

[Mozart2Vienna]

@Kratax

maybe in some situations people tend to mess in specific ways

;o)

[Kratax]

@Mozart2Vienna Maybe. But AIs shouldn't use mess as input, if the mess really is just a mess. How does AI decide, is the figure a figure, or just mess? I mean, even humans do not try to interpret too messy letters.

And AI is not only about figuring out what things are from visual clues. Thinking is more complex than that.

[maxpayne254]

atleast i understood the jokes

[ThaFacka]

where can i download this?

[flowewritharoma]

I think that this talker is pretty smart child phenomonan!!! And so I consier him to with value of hearing!!!

[Destruktor6666]

So, ....

Why does every question and answer begin this way?

[BigMTBrain]

@Destruktor6666 - That's always bothered me. It may be a west coast phenomenon. I first heard it in overabundance while watching Microsoft dev videos years ago from which it may have, unfortunately, spread and infected other tech environs. It seems to be their equivalent of "well". "So" as the lead-in to a question is actually quite common in the U.S. It's a short form of "So, if that's the case, then [question]". However, "so" has always struck me as bizarre as the lead-in to an answer.

[watzupdawg]

I can see one area where he has overlooked a parameter of governance.

[watzupdawg]

1 think, therefore 1 am.

[lvll138inrs]

a google powered robot.

[laalkali]

All Bouncers for me.. did not understand a bit of it. I feel like an Idiot sitting among intelligences :)

[Aurelovanovalopta]

Lol @ the Bush jokes :D

[B7ACKNEONder]

Differentiation does not imply separation.

[B7ACKNEONder]

The part I always get stuck on is semantic heat, where labels of relationship, eye next to eye, eyes above nose, need jogging to synthesize novelty. I find scotch to be helpful in understanding that what I know are only beliefs and there is always room for me to be proved wrong. Simulated Quantum Boltzmann models and scenario testing. Framework development with metaphorical LSD and tempering. TOO MUCH SCOTCH!

[allexxie]

Just greetings ,.. good night ,...and many lucky ,...Alex,..

[plmqas]

Another Hinton LECTURE on Restricted Boltzmann Machines /watch?v=VdIURAu1-aU

[daftrhetoric]

Generative science ought to be brought to social science and physiological science.

[UtilitarianTower]

Agreed. This video is awesome and is a great run-through of current methods in this field.

[TheReasonWhyGuy]

omg... after watching this I was finally able to program my first ai evolving program... and now I have little bots with tiny brains... which also aren't the simple mono-directional kind... learning how to hunt and run :)

[TheReasonWhyGuy]

13:00

That could be used, along with a accuracy testing script, to get past Captcha :)

[andenandenia]

Don´t forget that "The meaning of intelligence is to satisfy needs, instincts and avoid pain!" And you need a body to do that!

[krith]

@andenandenia sure you need a body, but that body doesn't need to exist in this universe nor in a simulation of it...

[RoneciDesade]

@andenandenia Keep in mind; if we as humans used our intelligence to liberate ourselves from our bodies, this would still lend itself to your description.

[n7275]

leave the politics out of the programing



[JohnConnorSAS]

whaaaaaatt??? i guess i need to study this in college...

[twistedbydsign99]

@JohnConnorSAS No you don't its hogwash. Learn to code.

[Tomhot89]

Thats my DADDDD!!!!!!

[Vaibik]

@Tomhot89 Who?

[POMDP]

Geoff Hinton is a modern-day genius. So articulate too.

[Drealoth]

I saw this guy give the same talk in Waterloo, he has a really good personality, and is incredibly smart.

[Mastersen]

its very funny to realize that most of the people commenting here have absolutely no idea of who Hinton is, even less what RBMs or DBMs are.....gibbs sampling? pufffff thats the annoying part of youtube: now 100000 people think they know what RBM are even though they have NEVER calculated a partition function in their lifes....

[Dollarbob]

it is funny how I studied this and didn't feel the need to lash out at others unfamiliar with the material...but you had some smug douchey notion pop in your head that your prior knowledge made you superior than others. work on your social skills and develop empathy. it will allow you to make use of your knowledge.

[LarryCarstairs]

ha ha ha. what a douchebag.

[Wulfcry]

Although I love the way its scalable but hate the in between network calculation stuff one could redirect the network with the features if they sufficiently support the recognition part. But they suffer the brute force computing needed on the network its an if/if situation like the network to stochastic or confused. If only the features would be a more advanced data set.

[eziooBand]

wer guckt sich das denn 59:24 sekunden an will noch irgendwas sagen und naja

[Wulfcry]

Ja waarom niet het is een oude concept. En alhoewel het een novel idee wasje kunt het als herhaling stof bekijken. Dit concept is achterhaald heeft fouten en niet goed genoeg voor generaal gebruik zonder brute rekenkracht.

Probeer het toe te passen op een smartphone met een pxa270 dan merk je het.

[shorty0802]

In 22:00, After presenting some number to the neural network. Shouldnt shouldnt it change the weights so the data matches more to one number, and when it runs backwards, why does it change it weights ?

[GishForever]

It uses back propagation to minimize the error.

[paucarre]

@shorty0802

The weights aren't changed. Only activations are changed because it's an stochastic system.

[kilianguntner]

Interesting! :) //K

[blablablag]

36:20 lol @ the 30-dimensional supermarket :D

[UnowMe00]

lol, great talk! the jokes are ok, he is self-ironical also...

I love the idea with the random noise on ravines to form abstractions of perception and reversely generation! It's simply elegant...

[clustro]

it was a good talk until he got all political

what a fag

[McYxee]

you are right.... getting politically in a lecture/presentation of computer science is imo very unprofessional.

[mallardvasey]

Who cares?The only thing that matters is what he is saying, which is fascinating (if not entirely novel). If he wants to make a few gentle jokes to leaven things, more power to him.

[nirvana4ol]

"Yann LeCun (& Bottou) and can make it work or more or less anything"... haha nice one... :)..

[fcycles]

very nice and interesting demontration.

I got one question about it. During the reconstruction demontration in which a number is being fix and we see the neuron network activities and the resulting image. The number 5 was quite nice and then a 6 appear. He say that it's still in process to show up 5... but, I wonder what kind of effect this will create on a bigger data set?

[OriginalAtomicSheep]

Can someone explain the concept of labeled and unlabeled data? Perhaps an example would be the best way to go.

[teclis1]

Say you have three pictures, one is a rotten apple, one is a healthy apple, and an orange. You want to train to detect between apples and oranges. Without knowing which is which, it is difficult. Giving the hint (i.e. label) that the first two are apples, the program can learn much easier.

[OriginalAtomicSheep]

Thanks... backs up the way that I thought about it.

[faunflynn]

Some one has to go through all the data and label it by hand.

The labels are what is learned.

Labeling the data takes forever.

I'm never ever doing it again.

[dings1337]

Great tech video. My favourite currently. I didn't understand everything in detail, but it's really fascinating how it works and what can be done with it. I need to get into this and play with it myself.

[napone0]

I'm working on a power neural net using R that forecasts scrap prices (which are very difficult to forecast). So far, I've had success with directional accuracy using just autoregressive inputs. If anyone has any additional ideas for inputs, let me know.

[alanjlockett]

It sounds like most of the people posting comments have no idea who Geoff Hinton is. The guy is probably the most prominent figure in neural net research since 1984. He was part of developing backpropagation, Restricted Boltzmann Machines, Deep Belief Nets, Contrastive Divergence learning, etc. There are other very important approaches to neural nets, but Hinton's are the best known. Jeff Hawkins, by comparison, is a layman. He has very interesting ideas, and he may be right, but he is vague.

[lilysleighpetal]

Jeff Hawkins spent half his talk restating his question of why the lack of brain theory? Studying the physical brain is a young subject as concrete neuroscience has birthed within the past century; therefore there is little theory to accompany it.

[KhanSlayer]

Does he have any publications or documents that can explain how this implementation of neural networks differers from your general fully interconnected neural networks? Any specific publication on these particular networks?

[Devilboy668]

His website as source code if that's what you want

[alanjlockett]

The problem with fully interconnected networks is that there is no known way to train them. You get better results by segmenting the network into layers that place limits on the connectivity. Boltzmann machines are completely interconnected with undirected links. But it was slow and impractical to train networks of any size. Neuroevolution is another group of techniques for training fully connected recurrent nets, but it hasn't had great success with large networks (whereas DBNs can be large)

[ZeeNwar]

For those seeking technical understanding, I would highly recommend the following papers: "Generative Learning Algorithms"(Andrew Ng); "Markov Chain Monte Carlo and Gibbs Sampling"(Walsh) "Explaining the Gibbs Sampler"(Casella & George);

[rsaarsoo]

I didn't understood most of this talk, but it was still quite fascinating.

[Adolphsson]

Wished I had Geoffrey Hinton as my teacher. He's hilarious! :D

[bullbunnies]

when hinton talks about turning examples into features, i'm a little confused. How do you define the wieghts? Do you use data from the original perceptron stage?

[lulinda22]

the matrix anyone

[Tsoftex]

"Entirely usnupervised except for the colours" hahahaaha awsome, can t stop laughing.

A great man, a great mind, a great sense of humour. He leads the way.

We still have a long way to go, though, do not forget that.

[modelmark]

Is there some information about how to combine this with recurrent structures, that can recognize temperal patterns of streaming data?

[Giannis25478]

Auto-associative neural networks rock!

[thehaircrow]

cool

[ibmsudarshan]

Fantastic!!! Extremely impressive.

[bottomlesssoul]

I liked the bit where he shows the NN 'dreaming' :-)

[ozieozieozieoyoyoy]

Don't get too excited just yet. Chances are this generation of neural networks is no better than its predecessors. There is not one application of neural networks that cannot be implemented without it. If there was, Hinton would have told us about it.

[DanoruX]

Holy crap...

[kronozgorilla]

excellent

[zweefvlieg]

Man! Those numbers showing up out of the network almost looked like an idea being formed. Extremely impressive.

[ancientkappa]

The methods and ideas presented here are extremely interesting. I'm going to try and replicate those networks to experiment on them myself. I just hope they have all the theory published uot on the web somewhere.

This could pave the way for the next neural networks revolution. I sure hope more people are doing research on this, because it could have enourmous implications.

[nsteinme]

Can't believe you'd criticize the amount of info per slide... did you understand a word he said? This is truly amazing stuff. Incredible implications, in my opinion as a neuroscientist, for the field of neuroscience. Very impressive.

[semiliteratedgod]

how is this relevant to neuroscience..??

[grimwallace]

"how is this releveant to neuroscience.??"

LOL

[imnoobish]

shows how much u know.

[mrpostitman]

I have to agree, it is a bery poor powerpoint presentation. That's not to say that the information isn't extremely interesting and amazing. Power point, though, should be used to highlight points during a presentation, not to give a thorough summary of everything being said. Even worse is putting more information on a slide than is being mentioned in the presentation. An example of very well designed presentation slides are Steve Jobs' keynotes. Amazing concepts being explored nonetheless.

[AndrianGo]

This is not about art

[AndrianGo]

This was meant for triniguy999

[ibmsudarshan]

This is a great powerpoint presentation! Too much info for triniguy999 ! There r enuf ppl who adore this work... it is pathbreaking...

[ramm69stein]

if you got the new realplayer you can... ^^

[waterpie]

Could google please make youtube videos downloadable?

[VVi11]

firefox plugin -

downloadhelper.

Just what you need

[eigentourist]

This fellow is an amazing lecturer. I've had an eye on this video since it came out, and finally got around to watching it. I've been going up the learning curve with DSP and learning machines such as SVMs and HMMs, and the restricted Boltzmann machines described here look like a very interesting topic to explore next.

The remarks about the "deeply embarrassing" success of support vector machines still makes me laugh.

[thoughtwaretv]

I'm having problems embedding these videos elsewhere. It keeps saying the video is unnavailable.

I would appreciate if googletalks won't be embeddable, that you please make this clear.

Otherwise, please fix this so we can share the content across the globe.

[rcparts]

"Embedding disabled by request"

It's already clear, read the top-right box.

[thoughtwaretv]

Much better.

[kitefrog]

When computing becomes based on self organising macro-molecules, and AI starts to become self aware. When it is connected to the google computer, do you think AI will learn to understand this video from the binary sequence? Imagine being the First AI, in lab with a giant Optical camera as good as our eye, then you're let loose on the google computer and within a day, you know everything!

[nightowl8936]

Nanotechnology is still in its infancy, and AI is sort of at the opposite extreme ... modelling high level abstract processes, and no one knows precisely what self-awareness is, or whether it can be modelled with current theoretical computer science.

Also, not everything on Google is worth knowing. You should see some of the crap thats out there. All kinds of bullshit talk.

Poor AI would need lots of guidance. Focus on your kids in the meanwhile.

[coresnake]

imagine the epic lulz to be had!

[King333B]

this is like a 1hr vid... howd u get on with 10 min limit

[bubblenut]

Really, interesting talk, I have understood the basics of the vector and porbabalistic retrival models but not the neural network. This was really helpfull, thanks.

[nightowl8936]

Google owns YouTube now. The user is googletechtalks.

Perhaps they get special treatment, but thats OK. This is probably one of the most intelligent videos on all of YouTube.

[InsaneZoidz]

how did you make a 1 hour video!