 Welcome back, everyone. This is Brian. We're going to talk about the multi process pool. And this solves the problem we had in the last video, where we want to create a worker, have it do some work, get the result back, and also wait for it to complete. So we're talking about pools of processes and getting their results. The end result is going to look something like this, where we're going to start our main process. And then we're going to have some worker processes. And we're going to be able to see when they finish in, we want to actually get the result of each one of those back. When all of our process of finishes, we want to automatically finish our main process. Let's dive in and take a look. First step in any application, as you guessed, is our imports. And I'm going to just do a little bit of copy and pasting. We're going to use logging, multi processing. And then from the multi processing context, we want to use process, we're going to stop time, and we're going to use a random number generator. This example is going to be eerily similar from the last example. So if you watch the last video, it's going to look virtually identical, but we're going to do this very, very differently under the hood. And we're going to have much better results. Let's take a look. All right, now that we've got our imports, we can go ahead with our worker process function. And this is going to be virtually identical to the last video. I want to show you that you can do basically the same thing. It all happens under the hood the way you would expect it. So there's really no gotchas. Just in case you really missed the last video, I am going to actually type this out. Don't want to waste your time. So you can feel free to skip ahead if you watch the last video. But so I'm going to say define work item. Count. So we're just going to have an item or some sort of name or message, and then we're going to have the number of times that we're going to count. And on each iteration of that count, we're going to go to sleep. So I'm going to say name. And I want to do multi processing dot current process. I'm going to go ahead and get the name of this, maybe if I can actually type it out. Yeah, my keyboard is just not being super helpful today. And then we're going to do logging. And again, we're just going to assume that by the time it hits this point in the process that it's actually, you know, logging is configured and everything's good to go. I'm going to just format that out. And I'm going to say we want the name, along with started. And I want to know what we actually started here. So I'm going to give it the item, which is this parameter right here. So we're going to say the process name started whatever item we're starting. That way we can start having different blocks of work that we can segment off to different processes here. I'm going to just for the sake of time, copy that so I have it. And then for x in range. And we're going to do it for the count. Again, very, very similar to the last video, we're going to say logging info name and then I want item equals x. It really doesn't matter what we put out on the screen just as long as we can show, hey, we are doing something, the process isn't hung up. And then we're going to go ahead and say, let's go ahead and put this process to sleep or specifically this thread in this process to sleep. If we had other threads running in that process, they would be happily turning away in the background. And then we're going to go ahead and say logging info name followed by finished. Now we're going to do something a little bit different here than the last video. I want to actually return a value and this is why I wanted to type all this out. So return, remember the last video didn't return anything, but this one's going to. So we're going to say item plus, and then just a string. I don't really care what we put is finished. And you could put some sort of number or whatever you wanted to return here could be an into bull string couple doesn't really matter. You can just return whatever you want. So looking at this, it's, well, it's boring. It's plain Python, but that's the point. This is so powerful. You can give it very simple code and even very complex code and it's just going to work on a multi process system. Okay, major takeaway from the last section just in case you did skip ahead is our function is almost identical except for we are returning a value. It doesn't really matter what we return. We're just returning some type of value because we're going to get that return value back. We're going to do that in the main process. So I'm going to just say main. And let's go ahead and define our main function. That little keyboard happy there. Now I'm going to go up here and I'm going to define what's going to be called a callback. You're not familiar with a callback. It's a process or a thread or even another function is going to call back or basically call our function and we don't know when that's going to happen. So I'm going to say proc result and I'll explain this a little bit here in just a second. But right now just kind of take it on a leap of faith that something is going to call this. And we're going to say result from here. We're just going to say logging dot info. Go ahead and format that out. I want the result equals and then whatever the actual result is. Now in case you're wondering what is this result going to be. Well it's actually going to be the return value here from our worker process. That's the beautiful part about what I'm about to explain is you can actually get the return from your worker process. Assuming it actually had an exit code of zero and it worked as expected you can get that. If it was terminated it's never going to call this proc return. At least I don't think it will. I haven't fully tested that but I don't think it would. I may actually test that in a future video. Anyways moving along we're going to go ahead and do our logging basic config and we're going to flesh out this main function. I'm going to do just a smidge of copy and paste here. So I'm going to say started max equals five. So we now need to grab a pool because we're talking about pools of processes. We've talked about thread pools before and it's virtually the same thing. It's just a group of reusable processes. So I'm going to say pool equal multiprocessing dot pool and that's with a capital P max. And that's going to create an instance of that pool object for us. I'm going to say results equals list because we're going to get a list back of these results. Now when we talk about the results this is actually different than the process result. We're going to be working with asynchronous future values. I'm going to say for X in range max and I want to make an item here. This is just going to be an argument we're going to pass to that. So I'm going to say item plus the string representation of X our count is going to be and yes you may have already picked up what we're doing here is we're making this stuff right here. So we're just going to make the values for our parameters. So we're going to say count equals and here I'm just going to do a random. Say random dot random range and we want to do say from one to five you can put whatever number you want but I'm trying to keep this video a little bit short. Notice right off the bat we don't know how long this process is going to run for because each count is going to make this go to sleep for one second. So we're going to have to figure that out and that's what we're really working with these results with. So I'm going to say our equals and we want to take our pool object and I'm going to apply. Now you notice there's an async. Apply is going to do this synchronously meaning it's going to do it one at a time where apply async will do it all at once. Whenever you see asynchronous immediately think of everything all at once. Now spoiler alert there is also a map and map async and that will take a function call and map it to an iterable list or an iterable container of some kind. That gets a little murky when you have to fill in all these values and stuff and that's why we're doing the apply async. It's a little cleaner looking on the screen basically they both accomplish the same thing though. And then I want the item and the count. Now that we've got that in there what we need is the callback. In case you're wondering I could actually be really nice about it. Add the target, add the args that way it's a little clearer about what's actually going on there. And now we want our callback which is our rock result. This guy right here. So basically what we're saying is cool, apply asynchronously meaning go out and do this all at once. I want to do five of these. All of them should start with that function work. Here are the arguments we're going to pass that function and if you have a result I want you to call back to this function here. This is a rock result which is this guy. Now if we were to terminate this process that's running we're probably not going to get a rock result and if we did I wouldn't actually trust it anyways because it would probably be some sort of corrupt data. Now I'm going to say results.append. We're going to add that asynchronous result in there. Alright so a little bit confusing about what's going on but now we have all these processes running. We need to wait for the results. So I'm going to say four are in results and there is better ways of doing this. We're still in newbie land so I'm taking kind of the easy path here and we're just going to wait. And yes you may have guessed this is a little bit dangerous and you should put some sort of optional time out in there. We're not going to do it for this specific video but you can put a time out in there if you don't want to sit there and wait for a process forever because you could very easily hang up your program if one of those processes just doesn't play nice. Alright once we've gotten all those asynchronous responses back we're going to say pool.close and notice how I have this in a comment because you can do that for pool.terminate. Now close is a very gentle way of doing it where you're just simply closing the door and terminate is like slamming the door in somebody's face and you don't want to do that because it will send that sick term out to those processes and just obliterate them. So let's say pool.close we're going to do it the friendly way. And now we want to take all of that great stuff all that machinery in the background that's running and say go ahead and join all that back to our main process. And then from there I'm going to say logging.info and I want to know that our main process is now finished. Alright a lot going on there but let's take a look at this real quick before I run it assuming I didn't have any typos. We have our main function which is going to go in and we're going to get a multiprocessing pool which is a pool of processes we can use. We're going to get a list of results and then we're going to say 4x in whatever maximum we put in this case 5. Go ahead and let's make our parameters here or actually our arguments so we're going to have a string and then we're going to have some sort of integer of a random number. And then we're going to go ahead and say pool go ahead and apply async which means it's going to go out to our pool and say create all of these at once. Don't wait on us just go do your thing. And as soon as those are created those are going to start working even as the pools creating processes it's actually really cool. Then we're going to have a target of this work function up here. We're going to pass those parameters and then we're going to have a callback. So we're going to say hey process if you've done your work you haven't crashed in why haven't terminated you go ahead return your value and return it to this function here. And then we're going to spit out the result from there we're not done. We're going to wait on those processes so we're going to say for each asynchronous result we're going to wait which basically puts our main process in kind of a holding pattern until each result is done. We're going to try to gracefully close that pool and then we're going to join it back into the main process and finally the glorious moment where we are finished. Let's go ahead and fingers crossed that I actually typed all of that out and I did not we have been betrayed. Okay so where are we at here line 50 let's go down a line 50 here wrong one line 34. All right what do we got here. Apply sent got an unexpected keyword argument of target. Oh yeah I put those in there for example and we don't need them. That's why there we go. So we're going to do positional work and then our variables now it should work just as expected. There we go. So you see we have started up and now we have finished so we have gotten items are starting and you can see they're already working as the pool is starting other processes. Then they're going to churn away and work in the background. We can actually see the individual workers finish and then at the very end we get all of our results back once all the results are in we are now terminating our main process. Let's go ahead and kick this out just a little bit I want to do a longer version so I'm going to say 10 seconds. Quick look at our code unlike the last example we don't have a hard time out we are going to wait on all of these processes to finish. So I can kick this out as long as I need to although you could and probably should put some sort of time out right here. So you could say there and it will raise some issues but for this example we're just going to wait indefinitely until these are done and let's see it in action. So this is the multi process pool running five worker processes for 10 seconds. There we go. Working as expected. Major takeaway from this is this is highly simple and highly efficient and you can get a callback meaning you can get the end result back from those processes. One thing to take into consideration is waiting for a process could have bad results and you may have to work out how to actually wait for it and how you want your program to behave. And then when you're done you have to decide do you want to gracefully shut the door or say you know what I'm done waiting on you and I'm going to just slam the door right in your face and that will terminate the processes. And then we're going to join everything back into the main process. I hope you enjoyed this video. You can find the source code out on github.com. If you need additional help myself and thousands of other developers are hanging out in the void realms Facebook group. This is a large group with lots of developers and we talk about everything technology related not just the technology that you just watched. And if you want official training I do develop courses out on you to me dot com. This is official classroom style training. If you go out there and the course you're looking for is just simply not there. Drop me a note I'm either working on it or I will actually develop it. I will put a link down below for all three of those and as always help me help you smash that like and subscribe button. The more popular these videos become the more I'll create and publish out on YouTube. Thank you for watching.