Deep Learning RNNaissance with Dr. Juergen Schmidhuber





Rating is available when the video has been rented.
This feature is not available right now. Please try again later.
Published on Dec 6, 2014

Deep Learning RNNaissance

Machine learning and pattern recognition are currently being revolutionised by "Deep Learning" (DL) Neural Networks (NNs). This is of commercial interest (for example, Google spent over 400m on start-up "deepmind" co-founded by our student). I summarise work on DL since the 1960s, and our own work since 1991. Our recurrent NNs (RNNs) were the first to win official international competitions in pattern recognition and machine learning; our team has won more such contests than any other research group or company. Our Long Short-Term Memory (LSTM) RNNs helped to improve connected handwriting recognition, speech recognition, machine translation, optical character recognition, image caption generation, and other fields. Our Deep Learners also were the first to win object detection and image segmentation contests, and achieved the world's first superhuman visual classification results. We also built the first reinforcement learning RNN-based agent that learns from scratch complex video game control based on high-dimensional vision. Time permitting, I'll also address curious/creative machines and theoretically optimal, universal, self-modifying artificial intelligences.

Talk slides:

Outline of slides:

- First Deep Learning (DL) (Ivakhnenko, 1965)
- History of backpropagation: Bryson, Kelley, Dreyfus (early 1960s), Linnainmaa (1970), Speelpenning (1980), Werbos (1981), Rumelhart et al (1986), others
- Recurrent neural networks (RNNs) - the deepest of all NNs - search in general program space!
- 1991: Fundamental DL problem (FDLP) of gradient-based NNs (Hochreiter, my 1st student, now prof)
- 1991: Our deep unsupervised stack of recurrent NNs (RNNs) overcomes the FDLP: the Neural History Compressor or Hierarchical Temporal Memory / related to autoencoder stacks (Ballard, 1987) and Deep Belief Nets (Hinton et al, 2006)
- Our purely supervised deep Long Short-Term Memory (LSTM) RNN overcomes the FDLP without any unsupervised pre-training (1990s, 2001, 2003, 2006-, with Hochreiter, Gers, Graves, Fernandez, Wierstra, Gomez, others)
- How LSTM became the first RNN to win controlled contests (2009), and set standards in connected handwriting and speech recognition
- Industrial breakthroughs of 2014: Google / Microsoft / IBM used LSTM to improve machine translation, image caption generation, speech recognition / text-to-speech synthesis / prosody detection
- 2010: How our deep GPU-based NNs trained by backprop (3-5 decades old) + training pattern deformations (2 decades old) broke the MNIST record
- History of feedforward max-pooling (MP) convolutional NNs (MPCNNs, Fukushima 1979-, Weng 1992, LeCun et al 1989-2000s, others)
- How our ensembles of GPU-based MPCNNs (Ciresan et al, 2011) became the first DL systems to achieve superhuman visual pattern recognition (traffic signs), and to win contests in image segmentation (brain images, 2012) and visual object detection (cancer cells, 2012, 2013) / fast MPCNN image scans (Masci et al, 2013)
- Why it's all about data compression
- 2014: 20 year anniversary of self-driving cars in highway traffic (Dickmanns, 1994)
- Reinforcement Learning (RL): How NN-based planning robots won the RoboCup in the fast league (Foerster et al, 2004)
- Our deep RL through Compressed NN Search applied to huge RNN video game controllers that learn to process raw video input (Koutnik et al, 2013)
- Formal theory of fun and creativity

More in the invited DL Survey (88 pages, 888 references):
Also published online in Neural Networks (2014):
Hardcopy to appear in Volume 61, January 2015, Pages 85–117


When autoplay is enabled, a suggested video will automatically play next.

Up next

to add this to Watch Later

Add to

Loading playlists...