 Welcome to week nine. Week nine, we're going to cover recurrent neural nets. And this really is part of a two-week sequence. Week 10 will cover natural language processing, which is the one of the big uses of recurrent neural nets. But this week we'll talk more generally about recurrent neural nets for time series, sequences of measurements. Now, it should be obvious, but the world we live in is mostly time series. Things happen over time. Now, up till now, we've looked at neural nets that take in a fixed vector, like an image, and produce a label. But most things are sequences of measurements. Some things are things we call time series if you want to predict how much energy a power plant will use tomorrow, the day after, the day after that. If you want to know how much inventory you should keep on hand in your warehouse, if you want to know what the bandwidth or disk usage will be, network utilization for computer centers, all of these are time series, usually multivariate time series, at each time, every hour, every day. There's some measurement. And given a sequence of measurements, you often want to predict what the next measurement will be tomorrow or the day after. How much stuff will I need next week? Robots, drones, sequences of measurements, images, geolocations over time. We will later do reinforcement learning to control these. Videos are sequences, are time series. One has a sequence of sounds and images. Often people these days try to, for example, recognize who's speaking when in a video. But of course, lots of information one could extract. All the commercial speech recognition systems take in a sequence of sounds from your microphone, and produce a sequence of characters. All of them use deep learning methods that we will cover this week. Translation, Google translates your favorite translation system, whichever it is, takes in a sequence, for example, of English characters, and produces a sequence of, take Chinese characters or vice versa. All of these also use deep learning in methods that we'll cover this week and next week. Question answering or holding a conversation. I type a question to the computer, it types an answer back to me. Maybe it's a sequence of those. Question, answer, question, answer, question, answer. These are methods we'll cover also next week. Again, sequences of characters or sequences of words in, sequences of characters or sequences of words out. All of these are dealing with time series. Now, neural nets, all of the advances in the last 20 years, I think pretty much, take advantage of invariances in the world. Things which are true about the nature of the world. Convolutional neural nets, which we saw before, assume that local pixels are correlated so that we can take a local kernel and average them together to do feature detection and critically assume translational invariance, the same feature detector, the same kernel works here and here and here and here as you translate across the image, the same feature detector is used everywhere. They're translationally invariant. This week, we'll cover recurrent neural nets, which deal with time series. Now, time is one-dimensional, unlike a two-dimensional image or a three-dimensional brain image, time is one-dimensional, and we will assume that it is also translationally independent, invariant, or as we call it stationary, which is to say that the same model that occurs at one time or one piece in a sentence will also occur at another time or another sequence of the sentence. So RNNs critically rely on the stationarity of time series. So this week, what are we gonna do? We're gonna talk about a number of different ways to represent time series as inputs to deep learning. We will do a brief digression to talk about embeddings, how to map words and images, other things to vectors. We'll cover these in more detail next week. We'll then introduce the core architecture for recurrent neural nets. Talk very briefly about estimating them using back propagation. It's all just gradient descent. We will talk about three main uses of recurrent neural nets for generation, for language models, for tagging, and for sequence-to-sequence models. All these will be explained. We'll talk about more advanced methods like gated recurrent units and long short-term memory LSTM neural nets that try and deal with the fact that recurrent neural nets tend to forget stuff too quickly. Next week, we'll have fancier methods like attention for dealing with these. And then at the end of the week, we'll look at some societal issues and there'll be a brief introduction to the yes, the final project, which will be lots of fun. Welcome to week nine.