 Okay, I know we're running behind time, so I will try and go as quickly as possible. I'm just curious, how many people are using machine learning already in the room? Okay, a few. How many of you are using deep learning already? How many are interested in getting into deep learning? All right, a lot more hands, cool. Okay, so some information about me. I'm a Google developer expert for machine learning and deep learning. I'm co-organiser of the TensorFlow Meetup in Singapore, based in Singapore most of the time. I spend pretty much all my time working on stuff to do with deep learning, a lot of it to do with language and dialogue, and some things to do with voice. Is there audio coming out of this, no? There is, okay, cool, that's right. I've basically done the whole startup thing. I'm currently in a startup. We started this last year called Red Dragon AI, and we do a number of different things. One of the things we're doing at the moment is a lot of deep learning, consulting and prototyping for people. We also do education and training in Singapore and around the region. So if you're interested in learning, taking courses and stuff like that, we currently do some stuff with SG Innovate, so I'll tell you a bit about that later on. No? Okay, but also the most reason we're doing those things is because we're working on products at the moment and research costs a lot of money in this field. And the things we're working on is stuff to do with conversational computing, natural voice generation, and knowledge-based interaction and reasoning. And so actually, my co-founder and I had a paper at NIPS conference, which is a big machine learning conference last year about this, and we're currently working a lot on natural voices. So this is where we'll see if it works. I am very pleased to be at Fust Asia with you today. I hope you will enjoy the talk from one of my creators this afternoon and that you will start your journey in deep learning. So this is a voice that we're currently prototyping that's built all entirely with neural networks. We're basically a neural network. It uses deep learning for the whole thing. And it's built on training data and then you can basically, you can just read English and then speak it straight away. Very different than the sort of, probably what you've heard of, like the phoneme systems and stuff like that. Okay, so let me get started. I meet a lot of people who ask me questions. The questions I get a lot are things like, you know, what is the best way to learn deep learning? What's the fastest? What's the easiest? What's the most efficient? Do I need to know math? Can I just write it in the current language, the programming language that I already know, that kind of stuff? And I get these questions really regularly. So my response ends up being questions back. And the questions I ask back, I was, okay, what do you want to use deep learning for? And what do you want to do with it? How deep do you want to go? How much do you want to learn? What do you already know? And then also sort of the way you plan to use these skills. So I'm in the lucky position now that I've seen a lot of people learn deep learning through teaching them and also just through, you know, people, friends learning and stuff like that. I've kind of seen what works, what doesn't. I, so I want to go through some of those things today and then maybe even get you started by teaching you some of the sort of key concepts to understand this stuff. I'll say that it's not hard, right? Everyone sort of looks at people in deep learning and stuff like that as, oh, wow, you know, you've got this amazing ability. But for the majority of things, it's gonna be quite simple for you to get started and probably to build projects. So goals, you know, I think if you're gonna learn something like deep learning, you need to have some core goals for yourself. You need to have something that you want to build. So everyone in here is a coder, is that correct? Yeah? Anyone not a coder? Okay, a few people, mostly coders in the room. Okay, so the deep learning field itself is so big that I could just talk for a whole day and not even finish all the different types of models that are out there. So you've got to be very specific when you're gonna go into this about what it is that you want to achieve, what is it that you want to learn. The other thing I want to sort of get across today is that don't be stopped by sort of hitting roadblocks of where you don't understand certain things. Deep learning education is still extremely new. That said, in the last year, there's a hell of a lot of new things that have come out and it's getting easier every day. There's lots more tutorials, that kind of thing. So my goal for the talk is to basically give you a 10,000 foot view and then go into some of the key concepts and show you some key resources for this kind of thing. When I talk about sort of learning, deep learning, I think of deep learning as sort of three levels of proficiency. The first being sort of just, and this is even a sort of step above what the IOT speaker Maku was talking about. He's talking about using pre-made models. Any developer should be able to do that quite easily. It's just peeing in API. But you can very quickly also build your own sort of very simple models without customizing them that much to make your own APIs. And I think that's one of the first levels that you want to be able to get to. The second is being able to customize things. The first level is extremely easy. The second is basically customizing things now where you understand a little bit of how deep learning works and you're trying to make it fit your particular situation. This is, yes, more challenging, but it is definitely still very easy to do. The third level is basically sort of where you're creating techniques and methods for new kinds of data and problems. This is very, very difficult to do. This is what I spend hours sort of working out, trying to get voices sounding more natural or get certain algorithms to work better. So first off, what is deep learning? Well, the media thinks that deep learning is this. They seem to think that it's sort of like Terminator meets some sort of oppression system that's going to take over the world. I find that most people, when you talk to them, think that deep learning is some sort of icy board with lots of dots all over it. And you see these in logos a lot for AI companies. But really deep learning is just a subfield of machine learning. It's actually neural networks is a subfield of machine learning and then deep learning is a subfield of neural networks. Neural networks have been around for a long time. This is not new stuff. People were working on this stuff back in the 1980s and even before then for certain models. But certainly it's really only become viable in the last eight years or so because of, one, the amount of data that we've got that these things require to learn. And also, two, more probably more importantly, the amount of compute that they need to learn. So as Chris was showing you in the last talk, to do some of these big models, you need an extreme amount of compute. Now you shouldn't let that discourage you when you're starting out because nowadays, actually, just with your laptop, you can build models that were impossible to build 10, 15, 20 years ago. So it's all relative is my point in that. The other thing that's probably making deep learning so famous is there's pretty much every breakthrough that you're seeing or certainly the majority of breakthroughs you're seeing in AI and AI products, really it's deep learning at the back end of that. Whether you're talking about something like AlphaGo, whether you're talking about self-driving, whether you're talking, all these things are actually using mostly deep learning to do the stuff. So when we talk about deep, why do we use the word deep? So here's a model called Inception. This is Inception version three. So this is quite a famous model. It won the ImageNet competition a few years back. Certainly it's sort of like one of the stock standard models that you will find around. Google to this day uses this for a lot of things. In fact, I was suspected a lot of those pre-made APIs, not auto ML, but the pre-made ones are using Inception v3. Now, when we talk about deep, we're talking about the number of layers this way, in this case. So on the left here we've got the inputs. On the right we've got the outputs. And you can see we're doing each of those little sort of modules that you're seeing there is doing a bunch of calculations. And to do this, you really have to run it on something like GPUs. To give you an example, I've gotten, I think this is like 2014, 2015, this model came out. When it first came out, it took Google probably around about, it would cost them about $30,000 to train the model for just the amount of compute and power and stuff like that to do it. Now, we're seeing such an acceleration in GPUs and stuff like that. If you've got a decent GPU card, you could probably train this at home over a period of a few days to a week or so. So what can deep learning do? I sort of at you that you should pick a project and go for the project. So let's look at some of the things you can do. So you can do classification. And if you look at this picture, some of these are chihuahuas, some of these are muffins, right? Now, it turns out that actually this is a very simple classification task, right? For any sort of computer vision algorithm can spot this quite simply. But it does sort of show to you that, often we find that using these things for classification of images is a big factor. In fact, this is probably the most common thing that you hear people talk about. If you think, for those of you who in here for the IoT talk before, a lot of what he was doing was just classification. They're all sort of stock standard classification things. But you can go so much further. And this is where, and I would say, so if you've heard some of the talks over the past few days about AutoML, Google's new product, AutoML, in many ways that is replacing sort of a lot of the need for you writing your own classification models. Because it could just do it so much better and so much quicker. But there are so many other models that are really important. So segmentation models, these are used a lot for things like self-driving. And really what we're doing in these kind of models is we're doing classification, but at a pixel level. We're classifying each pixel as is this a car? Is this a person? Is this a traffic light? And more and more, these are starting to be used for a whole bunch of different sorts of products in industry and stuff like that. We've got image captioning models. So where basically you can train up an algorithm to just look at a picture and then write text about that picture and write up a description of that picture or write up something like that. And you'll see that a lot of these things are, this is an example of where the model has basically worked out that even though we can't see that the kite is connected to the human, the model is working out that the probability is high that this human is flying this kite so it can therefore basically give it a caption. This kind of technology is also being used in industry to basically talk about spots on LCD screens and describing problems in products and stuff like that. We also have things like generative models and this is where my passion lies a lot. So these two things that you're seeing here are things that are models that I built early last year. The top one is what's called a super resolution model. It basically is an 8x super resolution model. The picture on the left is the input in the middle there is the predicted output and the picture on the right is the real world output. So you can see that it learns pretty quickly to get it pretty damn accurate. The one at the bottom is a colorization model. Again, the picture on the left is the input. The picture in the middle is the prediction. The picture on the right is the output. In this case, it's kind of interesting because you can see it doesn't mess up with all the bad lighting effects. You can see in the one on the right the lighting is very hot and that's why you've got that sort of yellow tinge but it's able to work out that, okay, this is what it thinks it should look like. We've also got sequence models for doing things like text and NLP and doing time series analysis. In fact, we've got so many different types of models that I could just keep talking about all the different models and keep going on. We've got neural machine translation, object detection, speech recognition, Q&A systems, a whole bunch of these different things. In fact, inside Google, this is an old thing but pretty much sort of the middle of last year, they were up to over 6,000 directories where Google stores its repos for its code, contain TensorFlow code or code that was being used for machine learning and deep learning. It's pretty much being used in the majority of Google products, now if not almost all of the products. So we talk about models and this is something that you heard Chris talk about, you heard the previous talk before Chris talk about. So everyone should be wondering what the hell is this model thing? So I wanna go through just some of the basics of sort of what the building blocks of models and get you to sort of think about knowing what they are so that then you start to know what sort of questions to ask when you're actually starting to learn this stuff yourself. So here's a very simple model and what you can see we've got some inputs which are going into a number of hidden layers and then we've got an output and then we're basically taking what's called a loss function we're comparing the prediction to the loss deciding okay how accurate is our model and then we're using an optimizer to basically update the weights in those hidden layers to then get better at learning and this is where we talk about the learning concepts going on. Basically what's going on is that we've got all these variables that are weights and biases and we're adjusting them so that as we pass inputs through the model gets to learn something about them and is able to make predictions. So when we're talking about the building blocks the really the key things to think about are your inputs and outputs, your architectures which I'll come back and I'll talk about what you think of are those as like the design patterns or the different types of connectivity inside models, different types of losses and loss functions and then optimizers. These are the four things that you will find in pretty much every deep learning model. So once you know that these four things exist you then just starting to think about okay let me learn the different types of those things. So I'll sort of just go through that with another graphic quickly. If you see here we're passing in the data and we've got labels that we've at the start already because we're training. We pass the data through the whole network it goes through the hidden layers. You see that on the left there we've got weights and biases that are basically being used for the mathematical equations. We then take the loss of the network and work out okay how accurate is this network and then we basically use an optimizer to update it. Now that loss we basically compare what's coming out of the network with the labels to see that okay are we getting things right or wrong. So let's look at some of the types of inputs that you can have. People tend to think of things like pictures or something like that but you can actually have whole varieties of different inputs. So hopefully what I'm trying to get through to you today is if you're coding an app or if you're doing some sort of web app even or something like that you can start thinking about okay what sort of inputs could you then use to make some sort of prediction that you want as an output. So we can have lots of vectorized data, continuous data, categorical data. Obviously images and if you think about it images really are just sort of three channels maybe four if we have an alpha channel in what we call RGB format or BGR format. We can also process audio by converting it to an image by making it into a spectra and then using that spectra as a picture and running a model on that picture. We can do the same with 3D medical scans of being able to take mammography scans or something like that and use the exact same sort of thing. We can do sequences. So sequences we'll see a lot for things like text models where we're basically looking at sequences of either words or characters, speech. Even sort of images again can be sequences if we're talking about video or something like that. Then we come to the actual model architectures. And this is where you hear a lot of the buzzwords sort of talked about because what people are talking about is what kind of model is it? And really currently they fit into sort of three kinds of models. You have your fully connected models. You have what's called convolution models which basically pass a filter across an image or something like that and then basically sort of do a dimensionality reduction and play with it and do a number of different things like that. Then the third type we have what's called recurrent networks. So basically networks where they feed into themselves and they're useful for things like time series stuff like audio processing, like all your sort of things like your automatic speech recognition things are using things like that. So the first type, the fully connected one, a great example of this is the TensorFlow Playground. So I encourage you just go along to the TensorFlow Playground, open it up and start playing with it. You can't break it, it's very simple. You can add layers to make it to see, so the hidden layers is where we're defining how deep the network is. And you can basically play around with this for hours trying to work out like, okay, what's the right system to be able to predict the different patterns? And they give you a number of different datasets to play with. Losses I kind of explained already, I think you're basically just comparing your predicted outputs with your real outputs. Optimizers are basically, I don't want to go into too much detail, but we're trying to find what's called a local minima for the weights. So the optimal point for those weights, so we therefore use optimization functions and a variety of them, which basically try to look around and search that space to find the right point or the right value for each of those weights in the actual network. And then eventually you get something like this, which is a convolutional neural network that can predict the cat or a dog. And you can see that you should be able to look at this model now and understand this, right? That okay, the input layer is the image, right? Each one of the pixels that is going in there, the output layer is predicting either cat or dog. So that's why we've only got two neurons that are right at the end. And the other ones are the hidden layers as we go through them. So one of the big things that I get a lot of people ask me about is all the math that you need to learn for this stuff. So yes, if you are gonna sort of go deeper into it and start to get more involved, you're gonna have to learn some math, right? That said, you can already get away with so many things without knowing the math. So don't be intimidated. It's more like, I would say for most people, it's better to just go along without worrying too much about the math, learn how to build stuff, and then at some point you're gonna start wondering, gee, I wonder how this particular part works, and then you'll wanna learn a little bit of the math to understand what's actually going on in the code there, or that kind of thing. So I put down some of the things there that you can do. I've also put a link back to a video that will explain back propagation quite easily. I don't have time to go there today. Programming languages, a lot of people ask me about programming languages. Okay, yes, you can build neural networks in any programming language, but really there's only one language that counts nowadays, right, which is Python. So if you're gonna be building something for the cloud or something like that, you definitely wanna do it in Python. Even if you're building a model that you're gonna then deploy to something like TensorFlow Lite or to Core ML for Apple's mobile neural network models and stuff like that, you're gonna build those models in Python, and then you're gonna export out the model and it will get converted into the local format. So you definitely wanna learn Python. The cool thing is Python is a great language and it's very simple to learn. Do we have Python programmers in the room? Yes, okay, agreeing, it's very simple. The syntax is very easy to read. So much so that when we do our developer course for the deep learning, for the people who don't know Python but are already coders, we literally just give them this one video. It's a one-hour video that teaches you Python in one hour. And if you can code another language reasonably well, you can go through that entire video and pretty much walk out understanding all you need to know about Python for this kind of thing at least. The other thing is, okay, frameworks. A lot of people ask me about frameworks and there are a lot of different frameworks. And this has been one of the challenging things for sure. You've got, certainly TensorFlow is one of the biggest frameworks today. You've got PyTorch is coming on very strong. You've got MXNet from AWS. You've got CNTK from Microsoft. The cool thing is this one in the middle is Keras. And Keras basically is a higher level abstraction for deep learning, which can run on pretty much all of these frameworks. So I would basically say to anyone who wants to start out, start out with Keras. It's very simple for you to get started. Like I said, it is a separate version that you can pick the back end or you can actually use Keras that is built into TensorFlow. The cool thing with Keras is that you have access to a lot of pre-built and pre-trained models. So you can just download those very simply. I, like Chris was sort of showing you in some of his code, I think you were using VGG16 or something like that, or one of the models, something you were doing a pre-built model. It's very simple to basically load those models and you're then basically getting all the work that other people did in building those models. And you can use it straight out of the box. You basically just change it for whatever inputs you want to use, whatever outputs, and then go for it. The other cool reason I think Keras is really worth checking out is that pretty much now, because Keras has been around for a few years, there's almost examples of any type of model that you want in Keras on GitHub. So if you're not sure how to build something, you can basically just go on to GitHub, do a search for Keras plus image object detection or something like that, and you'll find something that someone has probably built already, often in the actual Keras repo itself under examples that you can then sort of use. So just quickly, I'm gonna show you a very quick example of Keras. Okay, so for those of you who don't know, what I'm using here is Jupyter Notebooks. Jupyter Notebooks is a great way to run Python code. It's very simple. It allows you to see the output of each cell as you run it. And what I'm doing here, I've just got a bunch of imports to basically import some libraries and stuff like that. I'm using Keras, and I'm using TensorFlow as the back end in this case. I actually, let me go to the one I want to show you guys. And here I'm basically importing a dataset very simply. And what we're gonna predict is if you can't see it, this is basically an image of a handwritten digit. It's quite famous dataset called MNIST. Here's some examples of actually plotting them out. So you can see these are basically just handwritten images. There's 70,000 of them in this dataset. And we're gonna try and classify them between zero to nine. So I put a little bit of sort of theory in here to explain some of the things. What is a tensor? Tensors are just the different shapes. For those of you who have never come across this before, think of them as sort of multi-dimensional arrays is one of the ways that you can sort of think of them. And then you can see that in this case, basically our images are 28 by 28 pixels. And it's only one channel because it's just black and white. So we've got a grayscale image there. But then we have like batches of them. We actually have 70,000 of them. So we've got 70,000 deep. So then we wanna basically build, so here I'm importing them. You can basically say I'm setting them up to basically, this is what's called one-hot encoding, where I'm telling the model that, okay, this is how I want you to predict each of these digits. So show me by showing me 10 numbers and then put a one on the one that relates to this number. Okay, the key thing is that I wanted to show you was this. So this is the entire code for building this model. It's literally with one line, we define that we're building a sequential model. We then add in one hidden layer, just like we would in TensorFlow Program where we press plus and add in a hidden layer. We tell it basically how many, so I've sort of set these up before, how many neurons we wanna have in each of these hidden layers. We go through, we add it, so we've got two hidden layers, then we've got an output layer. Cool thing with Keras, very simply, we can check out what our model looks like just by doing dot summary. And we can see that this is the shape of our model. This is the number of parameters that we're using. So what Keras has done is worked out based on the amount of neurons that we're using, how many weights and how many biases do we need for each of these layers? And you haven't have to write the code for any of this. This is already done for you. Then we basically compile the model by telling it what kind of loss function. Remember I talked about loss functions are one of the four things that's really important. And the other thing I talked about is the optimizers, what kind of optimizer is really important. So we're basically telling it that, okay, we wanna use a loss function called categorical cross entropy. And we're gonna use an optimizer called stochastic gradient descent in this case. And we're gonna basically get it to work on accuracy. So we're basically optimizing for it to get the highest score possible of predicting these numbers. We can then basically just train it by doing this. So we basically just fit the model. So we're basically passing in both at the start. We're passing in those images. We're passing in the labels for what those images are. And we've got a few things there like epochs, which is the number of times through. We wanna do this. We're gonna do two times through. And then we're gonna validate. So I didn't have time to explain the whole concept in machine learning that you never validate on the material that you actually train on. You keep a separate data set for validation. So we've got a separate data set here that we're gonna validate on. And you see, we'll run the model and very quickly we're up to 94% accuracy with just a very simple model. And probably 20 years ago, this would have been state of the art. We can then even sort of take it to the next step and make it even further by making it a convolutional network. By making it a convolutional network. So the one I just showed you there was what we call a fully connected network. By making it a convolutional network, again, the model is just slightly a little bit more complicated but you can see it's not many lines of code at all. Again, showing you what the model actually looks like, compiling the model and then training the model. And here, with not that long, we're up to 99% accuracy on this data set. And this is training from scratch. If we were using something like a pre-made model, we could actually train it much quicker and we could do much more advanced images and stuff like that. And it would still be able to do it really quickly. This one I deliberately ran through on my laptop before to show you, it's not gonna take forever on a laptop. It's taking about two minutes or something on a CPU. On a GPU, this will take like, yeah, two, three seconds per epoch. It's much, much quicker. We can see this is, we can get some accuracy and sort of see what's going on in our model. Okay, I will go back to finish up. So if you are interested in learning more of this sort of stuff, these are some books that you should definitely check out. I would say the one on the left is probably the most difficult. So start with either of the other two first. In fact, the book on the right there by François Chalet, François Chalet is the creator of Keras. So that book actually goes through everything, using Keras and explaining to you how to do it. And this is something you can just sort of do as a passion project in a month. You can get through that book and learn all the key concepts for deep learning. If you're looking for courses, definitely, it's worth checking out Andrew Ung's course. Jeremy Howard has a good course at fast.ai. Jeffrey Hinton has a course which is somewhat out of date now but is still worth going through because he's pretty much one of the key founders of deep learning and neural networks. And also Stanford has numerous courses online. If you want to come in person and do a course, this is a course we're going to be doing in Singapore. I think it's three weeks from today, the 14th of April. So if it's something that you're interested, come along. The whole idea with this course is that we want people to bring projects along and we're going to start going through and the whole idea with this course is that you should be able to build a basic project by the end of the three nights. We did this once before where we just did everything in one day and realized the problem with that was that people went home, tried a lot of stuff out, more stuff out and realized they had all these questions. So we've now added evening sessions to basically answer questions and go through and help people who are not understanding certain parts of it. Okay, just to sort of wrap up, I would say have a clear application of what you want to build. Realize that this field is too big to know everything. Even for people who are professional researchers in it, I can tell you that having been at the Had a Paper at NIPS conference last year and talking to a lot of professionals, people specialize, right? So don't try to learn everything yourself because you're going to get totally overwhelmed. Well, another thing I would say is if you don't understand one person's explanation for something, find someone else's. This material can often be very difficult to explain and often people tend to click with one explanation versus another explanation, both of which might be quite valid ways to explain something. Again, go back to the basics. It's all about just the IO, the connectivity patterns, your losses, your optimizers. And inclusion, I'd say everyone who succeeds learns deep learning in a different way, but everyone who fails fails in the same way, they give up. So it's not that difficult. If you're prepared to put the time in and do it, it's something that you can certainly do and I encourage you to do it. Thank you. Interest of time, no questions.