 Welcome back everyone. This is Brian and we're talking about an introduction to multi processing and this is very different than threading It's multiple processes running the same script. So far every time I've clicked this little run button We've run one process. See ta-da. We've run it one time What we're gonna do is run the same script through multiple processes Now, what are we talking about here a process has its own memory space when I say process think program Has its own memory space and its own threads Technically the operating system owns the threads and lets the process borrow them But processes typically don't swap threads back and forth. Let's take a look at an example here So let's just say this is your computer. This is kind of a combination of CPU and RAM I'm gonna go ahead and let's make a process. I don't like that yellow for a process turn this to say like a green And this is our process every time I hit that green arrow. We're making our own little program Now the thing is as We're doing that other processes are also out in the operating system and they can be sucking up different amounts of CPU and RAM some can be big some can be small like for example Chrome will just takes an ungodly amount of RAM or something like, you know The little text pad will take a very small amount, you know, things like that so what we're really talking about here is We have our process and we want to take the same script Let's just see if I can type here. Yeah, here we go script.py And we are now going to make a copy of that out in memory and run that Let's put a little arrow here So we're now making a copy and running it and we can do this a lot of different times We can have an entire pool of these things. It's just ridiculous. So let's go. Whoops Let's grab this CV. So you can make a bunch of these and just have them all over in memory And then you can have All of these pointing back to your master right here and everything just works the way you think it would and life goes on The tricky bit though is understanding how this works under the hood and how Python actually handles all this and because each process is An independent island itself and each process now has its own threads inside of it. Let me actually make Like a little yellow thread pool here So like this one would have its threads And this one would have its threads and so on and so forth. I mean this is highly advanced computing you go to do this in like Some other language like C++ and you're talking about a six-week course and theory before you write any code whatsoever, but Python we can do this in minutes as you're about to see this is Beyond powerful. This is ridiculous. How much power you have at your fingertips? All right, normally I blast right through this next part, but we're gonna talk about imports real quick And I'm gonna slow way way down. So we're going to import Logging and we're gonna introduce a subtle bug here and I want to point that out right in the very beginning The bug is going to be how we configure logging We're also going to import Multiprocessing in case you're wondering why I'm intentionally making a bug It's to prove a point on how multiprocessing is different than multi-threading and then from Multiprocessing I'm just gonna make my life a little bit easier. Some people don't like it when I do this Say import process that way. I don't have to type multiprocessing dot process I can just say process and then of course we are going to simulate some activity We're gonna do more complex stuff in the future But right now we just want to simulate like we're doing something. So I'm going to put the current thread to sleep That's right the current thread. So each process is going to have a main thread and we're gonna put that main thread to sleep Later on we may dive into multiple processes with multiple threads. Let's see how all that interacts But just understand why we have these imports first thing we need to do we need a Starting function for our process think of this like the main function, but it's for each process We're gonna spawn off. So I'm gonna say run and we don't have to call it run You can call it whatever you want and give it any number of parameters you want to give it It's really that simple. We're just making a function I'm gonna say name equals and this is where people get really confused because the API is very similar to what we learned in Threading we're gonna say process dot current process name So each process now has its own name. So don't worry about naming your processes. It's all done automatically You can definitely feel free to pass information back and forth as you need to Here's the subtle bug that I was talking about earlier with logging. We're going to say logging info Right off the bat. What is the big problem here? Well, this is the starting function. We have not configured logging, but I don't want to reconfigure logging every single time I'll show you how to get around that here in a little bit, but just understand that it has to be configured before it gets here All right. So now that logging is in here, we're going to say Running and we want to know the name of the process And we want to know how python ran this. So I want to say name Remember if it's underscore underscore main underscore underscore, then that means python's running the script correctly Otherwise it will be named something else. So I want to know how we're actually being run as And then I'm gonna say finished Don't really need to know the name for that And then we're just going to simulate some type of work here. So I'm gonna say Time dot sleep I'm gonna take whatever number they give us in factor that by To why not you can feel free to change that to whatever you want however long you want to wait I don't want to waste a lot of our time here in video land. So I'm just going to say two And that's it That it's really that challenging. It's just a normal python function and we can run whatever code we want to run So let's look at how we actually start the process Let's take a look at basic process usage. And this is not at all hard I was a little intimidated the first time I did this years and years ago But python makes it Simple, I mean ridiculously simple in other languages. This is this puts fear into senior developers But python you can do this in minutes So we're going to say main Now unfortunately with that simplicity comes a degree of complexity as well If we're about to talk about so in our main function here Normally what we would do is some sort of configuration. So for example, I'm just going to take this code plop it in You see logging and if I go ahead and run this nothing's logged Well, simple. We just have to configure the logger So I'm going to paste that code in there. We've done all these logging configuration Of course, I'm not going to really explain it watch my previous videos But now everything will work. We're now actually logging Notice the name underscore underscore main Some boxes and some versions of python are different. So for example As we run this on this linux machine every process is going to have a name of underscore underscore main But python's using the process name To determine if it's the main process or a sub process On my macbook, which is literally sitting a foot and a half from me as a different Oh as different version of python It acts totally differently and what I mean by that is as we run processes It will start here at this function It effectively never runs this main function. So logging never gets configured. So we never see the logs for our processes Oh, that is frustrating. So To negate that whole problem What I like to do is not configure logging in main, but instead I'll configure it out on the global scope That way all Of the sub processes regardless of python version are going to have the same logging level and configuration and everything So I need to just magically work in the background so All right, we ran that we got no bugs. Let's go ahead and clear this out and now let's actually work with processes And this I always try to make this harder than it needs to be. So forgive me if I start slipping up I'm going to say processes And we're just going to make a list of processes for x in range Let's just say we want five processes Now we're going to say p equal multi processing Process we want the class make sure that's an uppercase p and then the api is exactly like threading, which is why people get very confused Say target and we want our run function args I always love that word args I'm going to say x so I'm going to give it a list of arguments here Now we want to set this to daemon equal true meaning Every time we start a process We want to make sure not only is that process running, but if we kill our main process We want all processes. We've started to also stop with it. We don't want a lot of hung processes sitting there chewing up memory So we want to say processes Append And we're just going to append that process and then p dot start We're going to start each process up so We're going to have to wait for those processes, but just look at this code right here We're just saying start a bunch of processes and then our program is actually finished or say our main process is finished So watch what happens here You can see running main and then we have process dash one Finish process dash one starting process two, but our main process finished We never saw finish two and process three is kicked up, but we never see finish three meaning we killed all those processes While they're in mid running that could have corrupted data. It could have done some very bad things. So we are going to wait For the processes And this sounds super complex, but it's really the simple 4p in processes Join and it's going to do exactly what you think it's going to do It's going to sit there and block execution of our main process until all of these other processes are done It's going to join it back into the main process of execution if you will so let's go ahead and clear this out And rerun Now we can see we are spawning up all these other processes And then we're just going to wait for them to finish every single one of them is the number times two So by the time we get to five we have waited 10 whole seconds This is extremely powerful. I mean I cannot even begin Especially if you're a complete newbie and you're just brand new to multi-threading and even multi-processing This is why python is used in data science and artificial intelligence and deep fakes and all that other high end technology because you can Really utilize the full power of the machine very quickly without knowing a lot about computer science So just to recap here Multi-processing is using the same script across multiple processes There is a slight little I don't even want to call it a bug But maybe version issue where if you configure logging in your main you may not see it in your child processes So you're going to want to do that out on the global And you may notice you should not depend on this name This process name don't like try to hard code that or anything because it is determined by the operating system And even the version of python itself And when in doubt always set your processes to daemon equal true that way if your main process dies Or you shut it off or the user kills it all your sub processes stop with it I hope you enjoyed this video You can find the source code out on github.com if you need additional help Myself and thousands of other developers are hanging out in the void realms facebook group This is a large group with lots of developers and we talk about everything technology related Not just the technology that you just watched and if you want official training I do develop courses out on udemy.com. This is official classroom style training If you go out there and the course you're looking for is just simply not there drop me a note I'm either working on it or I will actually develop it I will put a link down below for all three of those and as always Help me help you smash that like and subscribe button The more popular these videos become the more I'll create and publish out on youtube. Thank you for watching