 Good morning. How's everyone? Yeah, excellent when I looked yesterday into my swag bag. I found three pants. I thought wow how unusual I love pants and it's good to see that you can you know still get them as a conference giveaways and Yeah, these days I guess most people would use dialysis and things like that But just out of interest because this talk will hook into handwriting and things like that Who's actually taking notes on pen and paper? Oh Wow, this is actually impressive. This is like quarter of the room Well, you will have an advantage Because you actually why your brain to read handwriting faster and to actually decipher it this obviously is Somehow sorted have a look through it and as a human you're really good in identifying. Hey, these are Actually all the same in a row or not but this is actually what a Computer put together and you see for example if you go down here The four in the ninth Yeah, computer things. It's the same, but you can definitely tell it's not the same So we're going to look a little bit into this whole how do you actually? Train a computer to recognize handwritten digits Use Python for that and a little module called or a little project called brainstorm and I give you a little bit of the background kind of what you need to know for it. I give you a little bit of The stuff we're going to run so you're going to see a little bit of code and there's also That's for the impatient so you can go and download it now the actual example Which is very light-white and I hope easy to follow because this is really meant as an introductory talk and It's literally step on to three you run through that the rest is more or less the craft you need around it and a few test images if you don't want to scan and prepare your own months and This is not a talk about Artificial intelligence deep learning and all that stuff you read in the press and where you actually think okay Yeah, you need to be Google or Facebook or one of these this is more like the Kiwi shed version of Machine learning a really well-prepared example Really the basics if you want to extend from there there are many routes to explore. That's really a way to get into it At some point you won't get around the math But right now we won't need very much you see a little bit, but it's only to illustrate what's going on so if you think like oh Doing math Won't be that hard you don't have to leave the room yet That's my uncle shed by the way As you can see He's outgrown this hobby so the whole idea comes from biology and especially brain cells and If you look at it you have the extra the brain the axon Signals transmitted and whatnot and scientists thought okay, we can replicate that and we create these nodes and we connect them all up This is almost also where it stops your brain is way more complicated then kind of what you can Built in a computer kind of easily and on that level and there are other mechanisms at play But it's a nice analogy if you think kind of okay. How does this work because it all Works in parallel so it's not that you have like one pass through here and then one pass through there and so on It's literally that you take kind of the different nodes or neurons We talk a little bit about how they behave in the moment They're all connected up and they all interact at the same time. They all fire at the same time Going back to the idea of a signal and say okay if I have this little node I have to determine when it actually Fires kind of to the next one. So you see one node can have many inputs. Yes three it does something and fires an output How strong these connections are you can determine they're not all equal some are stronger links some are weaker links and Think of it as a white. How big is your pipe? How much can go through here? Can't be different from this one here and in here if you think okay everything all what's coming in is accumulated in the node at some point It'll tip think of it a bucket filling up and then it pours out which means it fires kind of to the next one or Well, there isn't enough input. It doesn't fire. It doesn't actually go through and Nothing happens The makeup of these is always the same if you think of Python writing a class. It's pretty easy to write a little neuron class and then Get objects from that and you connect them all up There are plenty of examples of that on the net if you really want to go that basic But it's a lot of work in the end kind of to buy it up and get it right. So what we're going to use is also framework and What you have to do when you create kind of these big nets of nodes is actually optimize Because Somehow they all connected come with all their signals and you want to find at the end these points here kind of at the bottom Here you have only kind of two inputs parameters Think of that and Later on You can say okay. We have three and each node contributes one parameter. It's right now That would be good enough for three. We will have in our little example, which is very very small in the end 2.4 million so unless you're Living in another dimension similar to a check has got to the galaxy, you know and dimensional beings It's really hard for us to visualize kind of more dimensions than 3d. It's maybe time but Just think kind of hey the whole principle holds you can do it with many many dimensions and calculating that stuff is pretty hard and luckily Other people thought about that the libraries for that stuff around and can use that you don't have to think about it It's really standing on the shoulder of giants and just plugging it together And then tweaking it and making it work Going back to our 3d example. What we're actually trying to do is also find a minimum for us later on when we kind of train and Think of a skier in the mountains. You want to get downhill. It's foggy. You actually can't see where you're going You only see kind of the vicinity here. Okay. Yeah, this direction is going downhill You go a bit further you go a bit further and bit by bit You zigzag around you go into the valley and deeper and deeper until you can't go further and you've found a minimum could be local could be global As kind of like illustrated here because you actually don't know because he can't look too far am I on the Here in the minimum in the valley or am I here in the valley? Also if you take small steps it takes a long time you want to take big steps But if you take big steps, you can go in the wrong direction and to balance this all out Needs a bit of experimenting a bit of experiencing So we coming back to that in a moment and hopefully it will get clear then So in our case we are looking at handwritten digits Because it's actually pretty universal has been stable for Yeah, a couple of centuries are not so much like handwriting. This is actually 130 years ago and use what Keywords will be familiar with briskos kind of the shop. This is actually the original briskos and I mean and You still can read these numbers you recognize them and not too far. You'll have a much harder time reading this So we make our life really easy. We say, okay, we're only going to look at handwritten digits they're 10 of them 0 to 9 and We're going to analyze that For that we're using brainstorm. It's a library written By a Swiss AI lab and the open source that It's in Python and it has a few dependencies. I'm going to talk about that in a moment It's running on multiple platforms I got it to work on Ubuntu and Debian a couple of tips later on on that Windows much harder if you have a choice Go with Ubuntu or Debian much much easier because If you try to spin up the compiler for the mess stuff on your Windows box You have to do a bit of plumbing and it's not fun, but it works This is more or less What you have to run in the beginning Python dev headers. A lot of you will have seen that That's for building the Python extensions lip HDF 5 that's the library for File system that actually stores Atresies and numbers in a very compact and very efficient format because we're going to store a lot of numbers and so we're going to use that and then The last bit the lip open real is Yes That is the bit that does the heavy lifting in terms of math and the calculations Luckily at this point we don't have to understand that we just have to install it Which as I said kind of on Windows is not too easy I had good success with win Python Unfortunately, that's Windows Python 2.7 Not under 3 and under conda. I got it to work under 2.7 and 3.5 These packages the links are in the slides. They actually Help you kind of with the pre-compiled stuff and you simply have to put it in So with the last up with the basic linear algebra sub-programs that does the Let's call them calculations and can do a lot. It's also optimized for your hardware and We couldn't run all these calculations on kind of the Laptops and desktops you have at home and that way without that stuff It comes back to this whole idea QB shut. I mean if you have the facilities and In terms of cloud computing or the bigger ones you can do way more but here you actually can go far out to Calculating a lot of tricky stuff, especially if you have powerful graphics cards and brainstorm Can enable that you can use the GPU? GPUs on the graphics card are really good with parallel processing You can leverage that if you really really want to optimize it Alright once that's over line That's actually after you pull down The example I put up on github Literally install NumPyPillow and brainstorm I Walk you through these in a moment And I found on Debian you also sometimes have to set Environment here. It's kind of one of these things kind of install it. It should work. It should be easy and kaboom It blows up and then turns out Environment variable missing you find it somewhere on stake overflow or things like that, but yeah Here is kind of like maybe I save you an hour Or maybe your Google foods better than mine, but that's kind of like soft that it's kind of often these little things so NumPy That is on top of the BIS that I mentioned and does The metrics calculations and really stores nicely a race that you can handle slice and dice Just as an aside, I put this together if you never touched on it. You really should have a look Here NumPy give me a range 10 which is 0 to 9 and spits that out and you see kind of it has that It's an umpy object. So it gives you kind of the array and it also tells you with an empty array What it's meant to be is it a float is it integer and things like that kind of these little details You can Slice them you can make them multi-dimensional as I mentioned before kind of 2d to 3d to 2.4 million D This can handle it for you And you can still navigate it You can do things like reshaping which will come handy in a moment When we bring it together because we're going to read in images and you have kind of the row of pixels and you actually want to Arrange them things like that. So you cannot do operations like much more Really worthwhile if you haven't touched before and not having a look pillow This is the more or less newish Python image library. This is kind of what we're using to actually read in our little handwritten sample and Somewhere far far away. We have this little image Samples are quite small and I tell you in a moment where I got them from they're only 28 by 28 pixels and they're prepared and Roughly look like this background is black and Just as a simple JPEG and if you want to get this in so we can use it for the calculations simply go Open Reshape it and load it. So that's numpy. That's pillow Now we need actually the data This is often with machine learning the hard bit to get good training data Um There is tons of stuff out on the internet in all forms. This is actually one of the most common ones because it's so nicely structured and It has been used over and over again think of it as the hello world machine learning That's kind of what it's used for. It's 60,000 grayscale images originally they were Written by public servant and yes Institute and high school students, I think and They were separate and they would be actually originally used them for it's like hey The public servants are the training data and then we test the high school students didn't work very well because size and writing was quite different So other researchers actually mixed it up and made this data set out of it and There are another ten thousand A verification on top of that so actually We don't know you get seventy thousand twenty by twenty eight pixels each and They're normalized which means kind of from the bigger samples from the bigger scans to talk They actually made sure because it might have been blue pen. It might be red pen whatever background always black But just white. It's a grayscale. So apart from gray values, you have nothing in there. It's 28 by 28 pixels and The actual digit is kind of like centered or waited around So it sits consistently Kind of in this 28 by 28 square a lot of work to prepare So if you do that stuff with your own data or it might make your own experiments, you'll discover You're getting sick of Photoshop or image magic pretty soon But that is kind of a really really important step to get this to work Right step one now. We are starting to go through a bit through the code. It should be as easy as you run it It says please be patient goes out to the internet downloads the packed data set for you and it creates the HD a file which I said kind of is a nice way of storing it Because at the end we kind of want to read the pixel information directly. So it actually Packs up for you in the proper file format doesn't take too long and That gives us Validation data where we actually can test so our Little program born to use this to actually learn what numbers look like there's never seen it before so use that and We also know what it is So we not only have the image with that comes a label little description. Hey, this is three This is a four and this has been verified and tested and then the actual training data 60,000 images which also have the labels Which we pumped through the network and then see hey so right or not and Then correct it accordingly That You don't do with sixty thousand and one go so we can chop it into little batches In my example, it's a hundred at a time. You can choose whatever you like So how does that now look when you wire it up? This is your prepared image Ignore that it's blue here. I just thought it's prettier. It is actually like in white and each pixel You connect to one neuron 28 by 28 really quickly 784 Magic, okay, so we have 784 neurons one per pixel and each neuron has some value zero to one depending on if it's black or gray and That'll be our input layer. That's how we get actually the stuff into the network now The next layer we use way more neurons gives us more flexibility and We connect them all up together So first neuron from our input layer Goes to all of the 1200 neurons and the 1200 is just I Decided 1200 I could have made 1600 That is again kind of where it comes down to experience how do you design that stuff? That's where you have to dive a bit deeper and don't have what works in what scenarios because it'll be at the end very specific to our use case And some stuff works well and some stuff doesn't work so well an additional optimization which I'll talk about later is that we actually Reset 20% to zero said but now It's on the face of it. We actually mess with our network. So when this is kind of going through say, okay I get this information blacks your note on fire don't fire and so on 20% of the information gets dropped This is the input layer. This is our first layer after it Which will be actually not our output layer because what we want at the end is a classification You say hey is this as three is this a four how many classifications can we have ten? so at the end we're going to have Ten neurons so it goes from our input seven eighty four because that's how many pixels we have Here we have also two dimensions. You can also think of hey, what's with the video? well, suddenly you have time sequence of images and then you get this as well which obviously Makes your input layer so much bigger already again This is a simple example But even in this network if you count all the connections here and up of 2.4 million if you're diving into Deep learning things like that. They are many many of these hidden layers in here Hundred hundred twenty and you run into also problems where they don't behave as you would expect This is simple enough that you don't have to do anything special in terms of training You can use always the more less same type of neurons things like that how they react in More advanced more complicated example to decide actually per layer how they react and how you train them Maybe you only train part of the network You preset it differently and so on But effectively for us Only thing to know is okay input layer output layer two in between and Here do 50% dropout We kill that stuff. That's all design. How does it look now encode in brainstorm? Well, not too bad Here you create your layers 28 by 28 because that's how big our pictures and One deep so for example if you have here 24 frames of video, maybe that would be 10 24 10 out and The projection is the layer before kind of the output classification. You can give it a name and The naming just makes it easier later on to actually reference it when you actually investigate your network There's tools and other stuff in there So instead of remembering some note number and things like that You give it pretty names Then we create the network layers input layer is our dropout layer with a 20% Then fully connected. That's one type of layer There are many more different types in there and you learn as you go about them But the fully connected means really from if I go back to the previous one Each learn from the previous layer gets to all of the next one As you can imagine that gets pretty heavy pretty fast, but here it's okay. So we're using that Give it a name hit one for hidden and in two and More dropout layer and that's kind of we're put out Network defined. All right time for training And my boys, I thought I give you a break at this point It is a lot of input But yeah training I mean data acquisition has one part and as I said kind of we just pulling down the prepared data A lot of work has been done for us So that's really good. Usually You would be very very busy to get that to that point already. Okay, um import brainstorm In the examples if you actually look at their web page the import SBS I thought no, it's not really good. Not in an English-speaking country. This might be okay with it So I went with that and you initialize it and you just give it God means kind of noise you just Set the It's everything randomly because you don't know where to start They just they pick something and then they initialize the trainer and some trainer class and For training you have different choice again. We are using a momentum stepper and if you remember back to the 3d valley slide, you're the skier who wants to go downhill that Tells you how big your steps are the bigger the learning rates bigger the jumps are you jumping more into the fog and The momentum is like how much can you actually change direction? It's not something. Oh, actually, you know, I'll go uphill again And turn around how much Can you let this happen? So you can influence how fast or how long your training is because if you want to very precise very small steps It can take very long because love resources, so it's just trade-off And then I only mentioned here one Once you have the train object you add hooks. They're different hooks in here That's a pretty neat one and you'll see it in the next slide gives you progress bar So it actually tells you when it's running through training what is happening and you get so a little bit of a sense for That stuff. All right. We're running step two I'm a little example. So it starts to run and then says hey before training and And how accurate are we? Okay, randomly 0.11% Well, you have ten things to choose from So, yeah, okay pretty much random. It's what you would expect And we have something in there a hook it saves actually the network into this file That's locally and that's again in the gf5 file format Makes it very efficient. All right. We leave it running and every run think of it as an epoch and In there you see 1 2 3 we have these mini batches always a hundred samples Goes through comes to the networks and says okay my example I know it from classification is three Statistically what comes out at the end is okay the likelihood for zero is ten percent For two is eight percent and so on says how well did that go and then something is called Back propagation happens. We say okay. I actually know what the result is I know what the outcome is I have to adjust it and it goes back and adjust the waiting in the network It gets better and better and you're starting to fine tune the waiting in your whole network And you run next epoch goes through Until I stopped it here after ten Just to see how well that goes and You see the accuracy is now at 95% Not bad for ten runs and you see each run took about 30 seconds. That's not with a GPU That's just running on this old notebook so That's half decent to get to a good result around the 98 99% you need around 100 epochs I let it for 500 and I put it up into the repo So even don't have to run the training you can download it from GitHub if you want you can go straight to step three By the way, the best one not with brainstorm, but there's actually shootout because This data set is so popular. It's currently 99.77 If you can do better you should publish Um The the loss in the rolling training Which is up here is With the dropouts and the stuff that didn't get into the adjustment That's rolling kind of it's around kind of the Every batch And then you do it's kind of the validation the same and that leads directly come to the next slide Much because you want to actually compare what happens when you train Yes, your accuracy So the higher the better and you see down here the epochs kind of we're running through. Oh, it's pretty bad It's getting better. It's getting better and then starts to flatten out until all more one day Well, you never will reach one but getting close to it But the validation isn't actually getting any better. It's actually with the unknown images it doesn't hit it as well and that's where the dropout because the network learns so well the exact examples and tries to fit that exactly that it's not actually getting good at kind of the unknowns we want to train the unknowns and so By actually dropping some of the information make it not That exactly you're getting a much better result. You see kind of the validation actually as Trumping the training which overall, you know, we didn't train it as much here The curve is flatter. It takes longer to get better, but the validation is much better and That's where the dropout layers become useful And this noise we think like it doesn't make sense comes through 20 percent or 50 percent of your information away But that actually works and I've also read that story that and Kaspers was playing IVM That I am only actually one Deep blue one because of her programming error and Kaspers didn't expect the move So actually noise is in this scenario It's not bad Which amazed me when I came across it With all this putting it together adjusting it how many neurons and layer how many layers The weighting things like that training data preparing and so on The tools are there and you'll find kind of there many also very big frameworks up there I'm very proposed full stuff but still your key in terms of how you put it together how it works for you and To actually be useful for your scenario There's no universal architecture. I mean, that's kind of the holy grail It won't find something that kind of learnt anything and also where you want to get away from the training We're saying oh all this training prep Can't it be self-learning and generic algorithms and things like that kind of I looked into in terms of like, okay This could actually help So it's not human-assisted on the other hand. There is so much information around that is already classified and that can be used They can base it on so you might have to find the case there. So it's really down to you to find The right stuff, right, that's the last one and That's the whole code for actually classifying it. Let me step behind here You see we're just reading in network information from the pre-created file and you reuse that over and over and over again Also, the reason why it works nicely on your smartphone because all the hard work for our speech recognition stuff like that has been done and you only Read the trained information back in get your test image Be a bit of reshaping Run it through and said hey, what's coming out of it? Why is it not? Oh, sorry guys. I was only shifting it here. I can't actually see that sorry, I have Duplicate it. No, I understand the blank looks All right, no, I hit that on the presenter screen actually all the way no touching okay, and You run it through and then you say, okay let's what's the highest value in my classification and I would be interested and actually see okay. What did you think for in this case? Where is it my number three my three image? What was the likelihood the percentage she gave for the other numbers? And that's what the output looks like So recognized yep number three With point nine nine nine nine nine nine nine almost one. So that's I'm super sure Zero and don't ignore the minus ten which means like here. There's actually ten zeros So your point there is is is is is is is is it but so it's very very small as it's algorithm is very sure it's a three And it's actually right and the repo there are a couple of other ones Well, I have to thank my family who Was okay with me taking over the playroom with stuff where I constantly had to say don't touch. It's not for you my employer sent me here ministry for primary industries and For you yeah, you guys for showing up and showing interest and you can catch me at the conference and You can chat a bit more. Thank you very much All right, we've got time for just maybe like one or two quick questions Thank you for your talk. I was just wondering there's a lot of neural network artificial intelligence frameworks available for Python what Or when would you recommend using brainstorm in particular or what's the best thing about it compared to other Things that are available The best thing is it's actually not too big I mean you can't roll through the code base in the weekend and get familiar how it's set up There's a lot of tooling in there It It's not necessary kind of the one thing you want to put into production and the documentation isn't great I mean, there isn't too much on that. It's really literally read the code in many Many examples But it's not so overwhelming as other big frameworks And if you actually look at some where I saw a list of how popular is what? It was like pretty much down at the bottom but for me it had the cute factor and You know enough tooling enough stuff to actually experiment with that what happens actually kind of if you don't use dropout layer What happens if I use in my hidden layers only 50 neurons and how does it explode and wrecked? it has a little tool with it that actually also creates and designs networks for you and Some of the tips are kind of like in there so you can still follow along kind of what's happening And what they thought kind of when they put it in there Goes into that That's the tool I mentioned the create network from spec We come up with a specification with them. Oh, yeah, I'm gonna try that and actually does it for you And so it's not so error prone You can obviously do it yourself and you can even create your kind of like on layer types and whatnot. I mean you can come completely crazy You're on architecture and That's not recommended don't do it by hand You you get it's insane All right. Thank you. I lied. We only had time for one question turns out So, can we just all thank Holger for his great talk