 All right, hello midterm survivors, so you're kind of surviving your week, I guess so no well Hopefully this will help you with your lab because we're essentially going to be doing lab for today So yay, so this is in some ways is going to be harder than lab four and other ways. It's easier So let's just get through it So we left off yesterday talking about deadlocks when we have multiple locks and spoiler alert, that's what we'll have today So remember all of our conditions for deadlocking. There's only two we really care about There is hold and wait So you have to have a lock and trying to acquire another while holding it and then the other one is no preemption You can't take locks away and then circular weight So you're waiting for a lock held by someone else. So this would be an example of a deadlock so again thread one locks lock one then lock two and Thread two locks lock two and then lock one Well, we might get in a situation where one thread is waiting for another and That thread is waiting for that thread and neither can continue neither can make progress So example of a deadlock Thread one gets lock one thread two gets locked to and now Thread two tries to get lock one held by lock or thread one and then thread one tries to get locked to held by thread two We're screwed so It's two ways to prevent a deadlock one is to always acquire the locks in the same order So if we have this function and we have locks Creatively named L one and L two well if I always get L one first and then L two I have no issues here We're all good. So this doesn't deadlock. I only get L two if I get L one So there's some type of implication there So I don't try and get L one if I have L two so I have no problems So I don't have that hold and wait problem The other one way to prevent it is using try locks So we've seen that a few times and we're like, oh, why is that useful? So it's actually useful in this scenario. So instead of always ensuring the order sometimes that's not possible Well, sometimes I might just try and acquire a lock and then in this while loop I do a try lock and try lock will return zero if it successfully grabs a lock and If it's not zero, it means it did not grab the lock So if it did not grab the lock it goes into this loop where it would unlock It would give up the lock it already acquired So it got L one if it made it to this point and then it would wait yield do something Let another thread do whatever it needs to do to hopefully give up that lock And then before it tries to get L two again, it would acquire L one So now when it goes back up and tries to get locked to it already has L one So it tries again if it doesn't get it gives up and Does that whole thing again? Otherwise it'll fall out of the while loop and your protected code will execute while you have L one and L two Then after that, of course you unlock So again, we can't deadlock here because we actually give up our lock if we can't get both of them So any questions on that? All right So that's what we saw another tool to ensure order when we were talking about condition variables for more complex things but more important for lab four and Practically like you're locking granularity. How much you lock something you want to Prevent data races, but you also don't want to be massive or else you turn your program into just this just a serial version that's even slower and Sometimes you might want multiple locks in which case you have to prevent deadlocks. So we're ready to run a bank. All right, great So we're going to run a bank We have been entrusted to do 10,000 or 10 million transfers And we're going to simulate actually running this bank So we're going to be able to simulate however many a variable number of accounts at our bank and each account is going to be Given a unique identifier with it so just one two three four Etc. And each account is going to start with the thousand dollars just so we have something to play with and we're just going to generate random transfers between accounts and Whatever you there's a transfer you take ten percent of the funds from one account and give it to the other So at the end of the day, you're just transferring money We're just moving money back and forth. So this should be a zero-sum game where the bank doesn't actually lose money It's just moving around between accounts someone goes broke who cares. It's not the bank's fault, right? It's like real life. So Yeah, and as part of this code It must securely connect a bank before starting the transfer. That's just to simulate wasting some time And that would be independent So we're ready to do this our goal is to not have any data races not have any deadlocks Make it go as fast as possible sound similar to lab four and we want to make it better than 11 seconds So it's in that examples directory and let's go ahead and play big bank So let's read the code So can everyone see that hopefully more or less All right, no yelling. So that's good so I'll make it one bigger just in case So All we do is get the number of accounts from a command line argument. We malloc some memory who cares We initialize the starting balance. So that's all done in serial So we're not trying to speed that up. What we're trying to paralyze is this code here So this is our for loop. So for 10 million transfers, we're going to just generate a random number To send it from and then another random number to send it to and then transfer between the account So I give an account Which is just a struct account that I'm transferring from another one then transferring to so let's go ahead and look What is in that struct account and within that struct account? It's not terribly It's not terribly complex. There's an unique identifier called ID and there's a balance which will start at A thousand and that's it and then aside from that We have our transfer function and all of our transfer function does is securely connect to the bank take Take 10% of the froms account and then subtracts it from their balance and then gives it to the to account and Adds to their balance. So again zero sum game here And then at the very end of it All we do is do a quick little sandy check We calculate the total balance held by the bank Between all the accounts just by adding every accounts balance while initially we know how much we have right? It's the number of accounts times a thousand. So hopefully at the end it all agrees So let's go ahead and execute that so Let's start with the cool million dollars So here's our bank. Oh wait, I should time this So cool million dollars. Let's see how long it takes to do What was it it's 10 million transfers of a million dollars should go back and forth Some people go broke some people are rich who cares not our problem again. We're the bank We ideally we make like, you know a few percent commission on all these and they actually don't have money And we get most of it, but that's the bank So yeah, this takes like about 11 seconds and at the end of it We have the same amount money that we didn't lose any money as a bank Okay, so Ten seconds is slow. I'm a big bank. I need to scale up So what should I do to make this go faster? Yep Create more threads perfect. So create more threads. Do I want to create? So if I want to run, you know, I have like six cores on this or whatever odd number of cores I have so if I want things to run faster Well, I can make threads or I can make processes too So between the two would I rather make a thread or a process? I hear a thread Anyone reason why? Yeah Yeah, so here If I split this off into multiple threads I'm randomly picking bank accounts and I need to keep that Consistent across everything. So if they're in different processes, they would all be independent But every time I make a transfer, I'm gonna have to communicate between processes which have to go through the kernel Which is gonna be really really slow because I'm gonna have to transfer data every time and then make it all synchronized between eight processes It's gonna be terrible. So I'll choose threads here because they're cheaper and everything's in the same address space anyways So I get everything shared essentially for free. So threads would be the good thing to have here It's not like Firefox or whatever where I don't do that much communication between tabs or something like that. So For this I made eight threads already So There's a define up here that has the number of threads conveniently set to eight So let's go ahead and start using that. So I already made some code here that creates a thread and We want to paralyze this loop All right, so now we make a bunch of threads So we make eight threads and then have a loop that initializes all of our eight threads So here we're going to malloc an ID so we can tell each thread its ID again We can't like pass it on we can't just use a local variable here and give it the address because Be giving away essentially an address on the main thread stack and it's gonna stay the same So if we want each thread to have its own independent ID We have to malloc it and then this is mostly just to show what it looks like when we're passing arguments So I'd malloc an integer make sure that I have memory for it and then set the Set the value of the integer and then I can actually pass it as a pointer Using p threads create. Oh geez p thread creates last parameter here, which would be the arg So I just give it a pointer and then I can actually use in the run function and Of course these threads are all joinable because I didn't give any attributes or anything like that So if I want to make sure they all finish before I check all my bank stuff Well, of course, I'm gonna have to join them So any questions about making all these threads or anything like that? Yep. I've I haven't made a lock yet So just high level if I want if I can do this in parallel if I have like a thousand accounts And I have eight separate threads that are all doing transfers Well, if I transfer between account one and two and then two and four I could do them at the same time, right? Like if you two are transferring money and then these two are transferring money those two are transferring money those two Right, I can make lots of pairs here and I could do them all at the same time if I wanted to okay A deal just happened back there. Yeah. Well, in fact, we'll just We want paralyze this loop. So let's do the easy thing. So we want to paralyze this loop now We created eight threads. So let's do everyone's favorite programming technique copy paste All right, everyone loves a good copy paste. So everything is running this run now. So I'll just paste it here Am I good to go? All right, so I'm good to go eight times faster now Yeah Before I even added the locks. So am I still doing 10 million transfers here? Yeah Yeah, so in fact here I created eight threads Yeah, so now instead of doing 10 million transfers, how many transfers am I doing? 80 million So if I want to do things in parallel, I shouldn't make everyone do all the work eight times. That's silly So what should I do if I want to divide up the work between each thread? So Yeah divide by eight So every thread doesn't have to do all 10 million transfers. So I'll just divide by the number of threads Okay, that's better. So I split up the work every thread is doing an eighth of the work Everything's in parallel. Everything's good. I see some unease but Start from what different indices? Oh, okay, so that's Yeah, so the comment is oh, well Um Each of them would just do the first Eighth of the transfers over and over again, but that's a good point But for this one the transfers aren't ordered at all So they just pick random numbers. So this index i never gets used So it's not important and it would be local to each thread It's just to count how many times the loop goes through. So that's a good point But in this case, I don't need to use it. So that is something you in general want to watch out for Okay, so other than this this looks good each thread now doing an eighth of the work So we have a group So I mean this is a great bank Compiles fine. So now if we time it should be like eight times faster, right? So we got a million dollars going it took 11 seconds last time Hey, it was pretty fast Uh-oh We lost some money That's not good. So it's faster, but it's not quite there yet. So What about if we were like, I don't know What's one of the like US banks that was too big to fail wells fargo or something? So hey, let's be wells fargo or some other crappy bank We got a billion dollars now And going through oh, we lost a little bit of money. We're a bit bigger So we got 10 billion dollars. We're using a lot of memory now So in this case, hey No problem. We're too big to fail Right. So why do I see if I have more accounts that it's I have all my money still anyone have an idea Yeah Yeah, so in this case well Why I'm getting the wrong numbers, of course, I have a data race. So what's my data race on? What data is being written to and read from from multiple threads at the same time? Well the balances and all the accounts right because I'm subtracting and I'm subtracting and adding to it and it's available in every single thread. So it's all shared And I didn't do anything to protect against the data race So in this case, I just got lucky that I had so many accounts I only did 10 million transfers that they just didn't happen to have a data race But you know, if I'm a small bank say I'm I don't know Some credit union or something say I only have like 20 accounts. So 20 grand I'm managing Do do do and now I have 71,000 Like uh, well, that might be good, but Might not be good. Well, let's try again. Let's see. Might be 1,000. What if we only have like I don't know say we only have three accounts To do 27 bucks not good. Well, both we only have two accounts Point 2,000 to do 18 bucks Yeah, not not good if you're a small smaller bank because it's more likely you have data races 18 again The really funny one. Wow, we're consistent three There we go Anyone want to explain why that happened? So did I get rich? Is this a good thing to happen? Yeah Yeah, well something under flowed so The balance if I look at the type of the balance. Well, it's a unsigned int So it can't go negative. So If because of a data race I like queue up an amount. I say I'm whoa, we'll e-crap transfer I queue up an amount. I say a it's going to be I'm going to take like 100 bucks out or something like that And then another thread takes money out and actually add or subtracts it And I only have like 90 bucks left now and then this resumes and tries to subtract 100 bucks from 90 bucks. Well because it's unsigned integer It'll under flow wrap around to the highest value and now it's suddenly gigantic So not only does our data race look really bad It can actually under flow and be like some gigantic number that doesn't make sense and we will go bankrupt So we talked about data races. So how am I going to fix my data race? So what did we what synchronization things have we learned about? We got some mutexes. We got some center fours. We got some condition variables What do we want to use to prevent some data race? uh-oh center four So could we use something even easier than the center four? So one idea is we could just use a center four for the amount two because it's just a number Maybe we could do that Maybe we don't want to but yep Yeah, let's just use a mutex use a lock. Is that what you're going to say to the condition variables? So you could put everything on the queue and pop it off as you need it, but what would you be popping off? So if I put everything on the queue and then pull it off one by one I'm doing everything just one by one. So no because this thread just puts itself to sleep and adds itself to the queue So you could that would be a lot of rewriting and you probably wouldn't want to use a condition variable So we also have another slight problem that Hey Even my data racy code isn't that fast. So it took 11 seconds and then I gave it eight threads So hopefully it would take you know about a second and it takes like four So I don't have you know, you'd hopefully expect that If I have eight threads and they're all doing an eighth of the work Well, if it took 11 seconds, it should take an eighth of the amount of time it takes Takes it doing one So one of our problems that is beyond the scope but of this but something you will have to look out for is This rand so this is something we call And there could be a data race involving rand too, right? We don't even know if our I mean If there's a data race involving random numbers, maybe that's not bad because then they're more random but Even for these they will argue about data races. So For most of the functions if you read the man pages for them So like here's rand and then it gives us rand r or whatever that is an s rand If you read the documentation Again, this is over scope, but it will tell you stuff. So it will actually tell you if they're thread safe So it says rand r and r s rand are all thread safe Which means Well, there's some global state involved with getting random numbers And if they're thread safe that means there's no data races, which means there's probably locking if we have Eight threads all calling it So if we read this and like kind of deduce that oh rand's making us slow because probably only one thread can call rand at a time And that's taking a little bit of time. Well, I can read these and say that oh um So rand uses a hidden state blah blah blah. I don't care, but this says rand r Essentially rand r is independent. It uses that seed as an initial variable. So it doesn't use any global state So if I want things to be faster, I shouldn't use rand for all of my threads. I should actually use rand r So that will be our first little optimization that won't help our data races, but it will speed up things So I can change rand to rand r And again beyond the scope of this, but let's do it anyways. So I'll set my seed just to some random uh, some random id that's unique to the thread And I'll give it the seed So now I replace rand with rand r and hopefully if I do that I make my data races happen even quicker, which is just fantastic So it was like uh, 3.9 seconds before if I do it again Hopefully it'll be faster. So now three seconds. So it got a little bit faster didn't get that much faster, but you know, we tried So that was a little aside that This thread safety stuff when you actually use it You'll probably want to read documentation For any function you call from multiple threads to see if you can even do that and get what you expect But we're going through we have a data race on the balance So the easiest thing to do to prevent mutual or to prevent a data race Is to just have a simple mutex lock that only allows one thread at a time So we kind of argued that our Data race was involving the balance of the accounts because That's the only thing the variables in the loop They're independent to each thread doesn't matter that the two and from indexes independent to each thread So that doesn't matter and then the securely connect to bank. We're going to assume that's specific to each thread So there's no data race involving that and it's all of the balance of the accounts So Let's go up and just create a mutex p3 So easiest thing to do which cough cough looks a lot like lab four is to create a single mutex So if you can create a single mutex and kind of do like the java monitor thing You can guarantee that there's no data races So I create a mutex and I will initialize it So now I have a mutex. So where should I place my lock and unlock? So I'll just start typing and someone can scream at me. All right. Is that good? No, why not? Yeah, so this is going to make this transfer only happen with one thread at a time Yeah, all right. The suggestion was like, uh, I'll connect to the bank first and then I'll lock So If I do that my lock is smaller and do I have a data race anymore? well, I mean If I did this I definitely didn't have a data race because the whole function Just runs with one thread at a time because only one thread can get the lock So if I have this I won't have any data races So let's go ahead and check Let's do our initial thousand. So hopefully my bank is Going to have all of its money at the end so In this case, it seems to be a bit slower, which isn't great Doesn't seem to be very quick at all does it? so 15 seconds Or it's closer to 16. So it was slower than if we had threads at all But we have all our money. So that's good, but We have eight cores and using them all made our thing slower. So yeah, the first suggestion was Well, let's move this lock down here Because each thread can independently connect to the bank, right? I don't care if multiple threads all connect to the bank at the same time So even if I just move that And then run it again Well, a lot of the time is spent connecting to the bank as you can see now It's much quicker and we also don't have a data race. Yep So why is it slower if I do Then the first case of not having threads at all Yeah, when I didn't have threads versus this case, right? So When I didn't have threads, it just went through it just steamrolled through it did everything in this case essentially it is Doing the same thing but like the same amount of work only one thread can do anything at one particular time Because I essentially made the whole thing serial And in addition to that there's a bunch of overhead Because I had to create all the threads and now they switch between threads and they all try and get the lock and they might switch and uh, so This is like One of the crucial things that you might try and might drive you insane if you like care about performance or doing threading stuff Some stuff if you throw threads at it, you might make the problem worse. You might might make it slower. Yeah Yeah Yes, because adding numbers computers are really good at doing that So I had to like artificially slow it down by the connect to bank thing just to make it Actually, so I can actually illustrate this point otherwise It would be even worse like the threading overhead would be way way bigger because I'm just like adding a few numbers Which takes what like a few nano seconds Yeah, so if we move the lock like this That means that all eight threads can do its own connecting to the bank all at the same time So eight threads can all do that the thing I'm making parallel is This little bit of code. So just transferring between two accounts so Yeah Yeah, this is the thing about making the lock as small as possible because the smaller it is the less things you make serial So this hopefully really illustrates that So if I move it here, I have to I can move the lock here if I can argue that I don't have a data race in securely connect to bank And for the purposes of this, I'm just telling you that you don't but In the general case if you call library functions or look up things you would have to Read the documentation for that function and hopefully they have taken this course and know about threads in it and thread safety and all that And could actually do it All right, so this oh, yeah Yeah, so I guess your point is like this works. There's no data races, but I could still do a bit better because this locks any transfers. So if You two are transferring and then you two are transferring Well, I could do them in parallel, but with this code It would lock wait for you two to go and then unlock then lock wait for you to why I could do them in parallel so I want to be able to do those two things in parallel because this is a bit slow and we can do better and your comment was We want a lock or we want a lock to make sure that we Just that they're independent or something like that Something like that. So what about so Where could I add some more locks or how many more locks do I need? Yeah Yeah, that sounds good. So everyone agree with that. Let's add a lock per account So we only have a data race with one account. Yep Yeah Yeah, if you use center for you would have one for each account if you wanted So in this case, let's uh, let's make a mutex per account Yeah So each thread or sorry each account now has a mutex that should protect the balance because that's where our data race was So let's make a mutex per account in this case. So I can't use p thread initializer because it's not a global anymore So I have to go to wherever I allocated memory and then initialized everything. So I have to use p thread mutex and knit Uh and give it accounts I Not mutex And then give it a big ol null So now I made a mutex per account Following good programming practices if I have a knits before my program dies, I should probably Uh free everything or in the mutex case. I should actually destroy everything. So Just to set a good example, I will actually go ahead and destroy everything So destroy whoops if I spelled destroy right So right before I exit right before I properly free my memory. No data leaks everyone Everyone's valgrine has always been perfect, right? Hopefully So I'll go ahead destroy set a good example. So now all my accounts have a mutex so Now let's go to our problem. So now our problem was here So I have two mutexes. So I don't just have a mutex. So I have a from mutex And I have a Well, I have a from mutex. All right. Am I good data race is solved Yeah Yeah, so I should also lock two because It's also modifying the two's balance. So I have two accounts. I have two locks. I should probably lock both of them So let's Do that All right, this looks like some something that we saw literally in the first five seconds of the lecture so What problem might I be encountering now? Yeah, I have a deadlock, but They're in the same order, right every thread gets from and then to Yeah Yeah, well, it's random. So I could have different orders. So I could have Where I could transfer I could transfer a to b and then I could also transfer B to a So this looks like it's in a set order, but My parameters can change because they're pointers. So in this case. Well, what would it be locking? It would be locking a whoops. Let's put it in comment And then it would be locking B and then what would this function be doing? What would it lock first? So what's the from What's the from B Wow, how'd you get that? All right lock So this is what they would actually do. So this looks exactly like we saw at the beginning So our deadlock could happen because this thread could lock a Then we switch over this thread locks B and now we're at that deadlock situation where thread two wants a but it has a Duh, duh, duh, duh, duh. So we still have our deadlock situation. So Any bright ideas how to fix this? Oops Not a center for so Let's go back so This is our insurer example. We said We couldn't well I tried to argue, but you told me that that wasn't in an order. Could we ensure some type of order between these? So I always acquire one lock before the other Yep Yeah, everyone likes that idea So his suggestion was that well, I told you each account has a unique id the unique part's probably important So if every account has a unique id if I always lock the one with the lowest id first, then I have some type of order Everyone agree with that? Seems plausible. So that sounds like a good idea. I like it. So let's do that so In that case, we'll just do p-thread So we're going to be super creative with names. I'm sure you have made better names before So we're going to create mutex m1 and m2 So what we want to do is check whoever has the lowest id. So if From id is less than two's id Well, then m1 can be uh From mutex And m2 can be two mutex Oops Otherwise we flip them, right? So this is our notorious copy paste Except we will flip them like that and then of course We shouldn't lock Uh two and from anymore We should always What the hell did I just do? We should always lock them in the same order So m1 and then m2 and we always lock the lowest first because we set m1 as being the lowest And then of course we have to unlock m1 Or m2 and m1 in this case. I could flip them the order Doesn't matter what order I unlock them in but generally just to have they're kind of like parentheses It makes my ocd feel better if they all match like that. So let's give that a try So everyone think that's good deadlock free Seems possible if I didn't lie to you but you know Sometimes I do so this should be faster than three seconds one two three four Let's check My poor cores aren't really doing anything in this case. I only have four cores anyways. That's why things aren't speeding up super well Uh, so yeah, I pro I can probably guess I'm deadlocked because no cpu core is actually doing anything useful And this is still sitting here dying so Why did I deadlock I acquired threads in the same order and It seems fine. What what really bad happened yet? yeah, so we accidentally Reaccidentally doubly deadlocked ourselves, especially if we got super unlucky. So there's nothing to prevent from and to being the same account So I'm transferring money to myself And if I do that well, both the mutexes are the same one So I lock it and then I try and lock it again Which doesn't work So the easiest thing to do and the fastest thing to do which will speed up our program is hey Well, everything Has a unique pointer. So if from is equal to two aka they're the same account Well, I don't have to do a transfer at all because it's the same account. So why even bother? So I will just return and essentially abort the transfer because nothing's going to happen So that will also help speed us up too So now if I do it Yay, it's fast. It's not actually that much faster because I only have four cores Uh, but It works. There's no data races and it's quite quick, especially if I do. Hey, let's say the worst case was when I had two accounts before in this case It's really fast mostly because The likelihood of transferring to the same account to and from the same accounts probably pretty high So it doesn't even have to do anything All right, that looks good. Any questions about that? Yep So why don't we use center for us because they will probably be a lot slower and aren't super involved like We could use a center for and essentially just Remember we made like an equivalent Mutex out of a center for so we could use them to do that And then it's just more lines of code because we have to initialize them as a value Or you could make Like you could make the balance the center for if you really wanted to But in order to transfer money, I'm going they only let you decrement an increment So I'd have to decrement a lot just one by one by one, which is going to be like really really slow Okay So we prevented the deadlock. We fixed all our data races. We're all good. What about What about if we didn't like that solution? So we saw this where we ensured order What about if I didn't give you a idea that's unique? So how would we we could do this? So Let's try and do that. So what would this look like in code? So I would keep that check just to make sure I don't deadlock myself And then it would go back to doing the from mutex into mutex Whoops So let's go back to this situation. So this had a deadlock. So if I didn't have an ID to If I didn't have an ID to order them well I would have to if I acquire the from lock. I should just try and get the two locks So I shouldn't do that hold and wait thing. So if I were to do that, it would be like while p thread p thread mutex try lock And then I would try to get The two side And so if I make it out of the while loop I acquired the lock we're all good And I have them both. So now I don't have to lock it. Otherwise if I don't have the lock Well, whoops if I didn't oh Jesus If I didn't get the two lock well that means I should probably unlock from And then in that example, it was like do something yield do something who knows And then it was p thread. Well just copy and paste And then we try and get the from lock again So if this was like lab three, what would I put in this question mark thing? if I so If I had the lock and then try to get it, but I couldn't get it So I can't make any progress. So what should I do to be nice if someone else can run? Yeah Yield so p threads quite seem pretty advanced. So yeah, let's p thread yield So you would think this would exist doesn't So The lives kernel because they're kernel threads and processes run a kernel threads It doesn't there's no process yield and no thread yield and said they try and be a bit more general And it's actually called schedule So Why don't ask me why it's called scheduled. So in this case, we would just yield And that's just a yield so another thread could pick up and then that's it So if we do that We should be able to do that and also have no data races Yay, so we saw two techniques to do it Don't know which one you like better, but we saw that um Oh, yeah last thing so valuable tool that I will give you as a reward for coming and maybe watching Is Wow Identifying all these data races and stuff and deadlocks sure does suck Uh, I wish something could help me other than me just reading the code and luckily something can help you So let's say Let's go back in time Where we just had So This is our back in time when we still had a data race, right? So that was data race fill And when we ran it, we didn't get any warnings no nothing so There are some very nice tools that are newer that you have access to now and they are called sanitizer tools and they are fantastic so When you build you can give it this flag for setup desanitized thread And there's a few more that might make your life useful that probably should have told you about earlier So there's like an undefined one. There's an address sanitizer There's all sorts. They're fantastic In this case this one is called sanitized thread and what its job is is to tell you It's not guaranteed if it tells you or sorry It is guaranteed that if it tells you there's a problem. There's probably a problem But it might not detect all of your problems So if it gives you a warning for sure you have that problem If it doesn't give you a warning, you're still not in the clear but it gives you a bit more confidence So if we go ahead and run this now, so let's go back So this is our version with our data race. Whoops Stop it So this is our version with a data race Whoops We have to compile because we nuked everything So if I do that hey tells us there's a data race In fact, it even gives us a line. It says it's on line 59 Which is that? So that gives you a very good hint where you could have a data race and in fact Not only can it help do data races. So if we have this This didn't have a data race anymore, right? But it did Potentially have a deadlock. Well if we run it with threads sanitizer, it gets very angry because If you see it says mutex acquired here while holding this mutex So it actually gives each mutex a number so you can tell what mutex you have and essentially tries to identify any hold and wait So if I grab another mutex while holding another It will definitely yell at me and as you can see yells at me a lot and it says warning A lock order inversion potential deadlock So I didn't acquire them in the same order and it will throw up at you if you do that So that's another great tool. So it even tells you the graph So you can see that this is a fairly involved graph. It's just not a and b This is a very long cycle of you know Seven yeah 70 the nine blah, blah, blah, blah All right, anyways, we're out of time so But just remember just because this is clear doesn't mean you don't you still have to argue that you don't have data races Or deadlocks, but this will definitely help you find them. So just remember for you