 Thank you for coming. I hope that you're not too sleepy. I'm gonna tell you a story. It's a story about how I found out that I was wrong and how I found out the scheduler. It was not a problem I was looking for. So who am I? Martin Peuchot, a French developer from Paris. I'm really happy to have a lot of BSD enthusiasts and developers around here at my hometown. I joined the open BSD crew around six years ago and since then I've mainly been improving the kernel, working on some parallelization stuff in the network stack, improving the USB stack, adding some support for drivers. I don't know, a driver's driver, profiling. And lately I got a problem with Firefox. When I'm not working on open BSD on my day job, I try to work on open BSD. So I go for a consulting and I say, well, I believe in open source. I believe in free software and I'm sure a lot of you, a lot of company have problems, not necessarily with free software, but they have problem with software. And when it comes to resolving a problem, I was discussing that with a lot of you. We have a lot of discussion about Dtrace or any performance tool and we have a great keynote tomorrow. But the question is, how do people can think? How do people can use those tools? And what's the step? Because we are all making assumptions for here about the scheduler. So I go on consulting and what I hear generally, oh, I have to go there. Yes. What I hear generally is UOS, meaning open BSD, when they talk to me, it does not work. It's your fault. It doesn't scale. The scheduler sucks. I will switch to whatever Linux free BSD windows. Okay, so I just say, okay, I go back home. Sorry. Thanks for coming in. Now, my work and my passion is to try to solve problems. So how do we solve problems when somebody come and say, well, it sucks? I had a problem with Firefox, right? So let's take this example. First, I will expose the regression or at least, well, how it came to me. Then I will dive into my own mistakes. So what I call the first little hacks on how to solve this problem. Based on that mistake, how I learned about it, I found a real solution with some interesting challenge technically. I will explain that before the conclusion of this talk. So first, my Firefox problem. So I like to watch a movie on internet, especially an HD and well, if it's a kitten, why not? At the time in August 2015, I was a really, and I'm still a big fan of Firefox and there really is like a new version. The number really doesn't matter. At that time, it was Firefox 40. And I could not watch my movies anymore. So, well, what did I do? I start looking at the change load. I didn't see anything obvious. I had the feeling like I was coming back 10 years ago when we had like this flash plugin which was not working and we had to download stuff manually and it's like, no. So I just switched to the long-term support version of Firefox, which still worked at that point. And when it did not, because it got upgraded, I switched to Chrome. So problem was fixed. I could still watch my movies. But the problem wasn't fixed from the point of view of an OS developer. You still have something that happened. You don't know why. So what can we do at that point? What we all do at this point? Why do we start making guess? Well, it's because the moon at that time and we start blaming people. So I'm not sure if Landry is here today. Landry? No. Landry is maintaining Firefox on open business, doing amazing work. He will talk about that I think tomorrow. And so Landry at some point start to say, please stop telling me that Firefox doesn't work. I know it does not work. I know we had a problem but nobody is doing anything about it. So my thing is, okay, if you complain, that's fine. But if you really want to make things better, you have to explain or to say what's not working. And that's not trivial because we all use the computer as a consumer to watch our Keating video and we don't know what to look at. So generally when it comes to open source and free software, people tend to focus on the source code. And this week, for example, last week, a couple of weeks ago, I introduced a regression in the kernel and all I got was like, oh, this commit broke, blah, which is great because people have access to the source code so they can tell you, well, this commit is wrong. But looking at the source code or at history of a program does not always work. In the case of Firefox, you have this huge program that even if it's open source, it for me really complicated to deal with. And open business is not a supported platform in the sense that they don't check every change on it. So it's really hard to backtrack. So what can we do? The same applies to closing through software with your own software developed by your co-worker that you don't know. Now what I said and what I did is I took the black box approach. So the black box approach is basically, well, Firefox, I don't look at what he's doing. I will just look at the metrics. What are the metrics? Well, if you take the metaphor of the car, generally if your car, something does not go well in your car, like a small light start blinking, oh, what is it? What is it? And in all our voices, we have a lot of tools that give us those information. We just don't know how to use it. We don't know which one of them we have to look. So I list a lot of them that I use regularly. Just, well, just run it. And since in this case we had a software ESR version, so the long-term support of Firefox, I could grab my video of Kitten, run one statistic tool, see how the metrics are, and then I grabbed the newer version, the one that I could not see the video with. And look at the statistic. What, is there a difference? So you consider your software as a black box and you poke at all the registers or all the statistics your operating system is giving. I did that. I did that and before answering to the mailing list, I gathered the relevant information. In the case of Firefox, in the case of this regression, had two really interesting metrics. So this kind of interpretation now, even if you're not a developer, you're aware of saying, oh, this is different. So you can suddenly email, hey, I found that it's different. I don't know what it means, but it's different. So what I got, I got VM stat, the minus I option on OpenBSD, but on other operating system as well, reports the interoperate, so how many interrupts you got in a period of time. On this case, with the new version of Firefox, I had an incredible interoperate for inter-process communication, the IPI. And I was like, the old version of Firefox are almost none. And on this version, I like 30,000 or more. Well, that's something. We don't know what it is, but that's something. The other interesting metrics that when I run top, sometimes I run tops. Well, I just see that basically Firefox was playing ping-pong with my CPU. So one process was going on CPU one, then it was going to CPU two, then coming back CPU one, and then going to CPU three. And it was like, oh, what are you doing there? My movie is not playing, but you're like... Anyway, that's something. Maybe I cannot interpret it, but okay, let's dig a bit in this direction. So what do we do next? First, we have got the metric, something is different. Well, we use a really useful tool that we have in OpenBSD, and we have this saying of care trace or it did not happen. So care trace, you might not know it, but it's a tool that register basically all the syscall. So they function that enter the kernel and exit the kernel with doing some action and tells you if it succeeded or not, what's return value at which time it stands. And the upper left value on the slide is the PID, so the idea of the process. And after the slash, you have an idea of the threads. So on the first line, you see that a thread with some ID called syscall, which is schedule. Well, we don't know what it is, but it called that. And then on a tool line, it returned from it, then it called it again. And then another thread from the same process. So now we know, well, that's these two threads, they call the same syscall. Well, cool. And then another one, which different number also called this syscall, whoa, there are a lot of threads, well, that might explain the ping-pong game, right? And they're all calling the same syscall. So now we are curious, we can look at the manual page, well, man syscall, man schedule, what does schedule does? Maybe you already know. Well, schedule is say, oh, sorry, I'm there, I don't know, I cannot do anything, just please pick somebody else. So it's like communicating with scheduler, I say, well, for any reason, I don't want to continue doing something. So there I got, okay, something is related to scheduler. So first guess, I explained why it was wrong, I said, well, scheduler is a problem there, because somehow we are connected there. So let's dig a bit, what happened after that? Well, well, I looked at the difference between the catchphrase dump that I got from the RS ESR version, so the long-term version of Firefox, and the nightly version that I built from source to be sure, well, something is happening. And I could isolate the problem, but this year call that I saw were really different. So maybe they're related to the automatric I've seen. So far, I've seen a ping-pong, I've seen a huge number of IPI, and in terms of codes, a lot of schedules, really interesting. What next? What could I do with that? Well, this is directly an entry point to the code, so I could look at the code. There is many tools on internet, oh, could look at the code, well, look at the code. There's many tools on internet that allow you to search directly. So in this case, I was looking for Firefox and for OpenBSD. So I put two web interface, bxr.su, that I use every day for developing OpenBSD, which is a website like it's on OpenGrowth. Maybe you know that, it allows you to search in the source code of all the BSDs, so very useful to check for differences, to know where this call is really used. And you have the same for Mozilla projects, which is digsr.mozilla.org. So my entry point is this is called, and I found basically two different places where it was called. Either by Firefox directly, when I said directly, there's a lot of layers of abstraction, but it's directly in the binary. And through a library that got linked to the binary, which is the leap-thread version of OpenBSD in the second point, and in the name of our pthread libraries, leap-earthread for real thread. So I say, what else can we do with that? Is it coming from there? Is it coming from here? Or both, right? So there I say, we have another two. We have L-trace. So L-trace is basically to see which function are originating from a library. It's based on OpenBSD on the K-trace, so use it exactly the same way. However, Firefox is a huge process, you don't learn anything when I tell you that, and you cannot directly, well, I would say running L-trace from the beginning and all the startup process of Firefox meant it really complicated to see what's happening because you have a lot of noise. So it's like a general concept when I'm trying to analyze a problem that, oh, you know something is wrong, but when do I have to look at it? And how do I make sure that what I'm looking at is effectively what's corresponding to the problem? So in this case, I try to generate the simpler test case with Firefox possible, and for me, it was starting the browser, waiting for it to settle, and on the blank page, so nothing to render, nothing, just move the curse of my mouse like a little bit, like you're just touching it. And that triggered all the crazy stuff I just described before. So there, I start Firefox generally, and I started L-trace to see when I moved the mouse, what's happened? Two second of sampling, that's the sleep tube, and then stop the tracing. And what did I see? Well, I see that it come indeed from the lib-air thread, because I think that a lot of the skill deal were coming from a function, a function which is called spin lock. Oh, interesting. And I matched the number of the skill deal that I had in my K-trace with a number of spin lock that I had in this dump, and it was exactly almost the same. So I kind of going in the right direction, right? Well, I think something must be wrong with the scheduler. I don't know why. It's like scheduler, it's a funny subject even for me. So then you have those guests. I learned last week, I learned last week I was consulting and that our brain works in two different fations. So you have like this suggestion that your brains make, which are not always interesting. So the guy asked me a question. So if I buy a cell phone and a case to protect it for a total of 110 euros, well, and on top of that, if I tell you that the cell phone costs 100 euro more than the case, how much cost the case? It's an interesting question. And the answer is it's not 10, right? And exactly that was happening in this situation. I got this intuition and I it's one. And that happened to everybody. What was more complicated is that I said, okay, the problem is scheduler, so let's hack the scheduler. Well, what did I do? I started to say, well, on OpenBSD, we have like a modified version of the original be a scheduler. And every time a task want to be executed or thread in our case. We will say, well, first select a CPU, put on the queue for this CPU. And if there is somebody already being executed, you will be picked later. So I said, ah, the problem I said, I guess in another time, well, that should be certainly that because they're playing ping-pong between each other. So maybe I didn't know. I really didn't know. So I start by ripping out all those per queue CPU, per CPU queues. So just say, well, as soon as the CPU has nothing to do, it takes from a global list, what I represented here on the right, the next thread of task that needs to be executed. And it worked. Why? Well, that would be interesting to explain. But it's more interesting to explain why it was wrong. So is it the problem really there? I make a guess. I write a div. It works. People are happy. People are running with it. And I said, no, I cannot commit that because I don't understand it. What did I do next? Well, I got to do a deeper inspection of what was really happening into the library. I wanted to use GDB, but I'm crying for help till today. We have a real problem there. There's no debug symbol in port. So I've been whining for years, but it's still not happening on OpenBSD. And it's really hard to debug threaded programs on OpenBSD with GDB. Basically, as soon as you stop debugging it, your programs stay in stop at state and you cannot do anything with it. So we don't have dynamic tracer. We don't have anything else. So let's go for print of debugging. Why not? And with my nice print, if I got with those information on the slide, what does it say? That you have the address on the code where the yield function, the scale yield is called and has been executed and how many times. So it's kind of rate limited. That's really interesting here. What we say is that multiple times, single thread has been calling the same scale yield. So it's like in a loop, right? It's trying to call this this call. The other really interesting thing that we see is that they're calling them from two different functions on the right. Mutex log and con for condition variable time in the way. They're both related to Mutex in some way, if you know P thread libraries. What did I learn there? Well, I started to fear that the problem was more in the P thread library. But more interesting, that other OpenBSD developer was saying, ah, the problem is coming from contention on malloc. But if the problem was really only on contention of malloc, I'm not saying it's not. It was also. We should not see there so many yield on the condition time wait. So we also have another problem that we still don't know what it is. So what it is? Well, at some point you try to read some code and make a guess about what you read and see if it's matched the information you're getting. So I read some code. At that time, another guy on our mailing list started to post a new scheduler without really explaining anything. And I started looking at this different. I found it really interesting. What I found really interesting, it could be completely different talk is how the priority works in an NBSD scheduler. What interesting is that at that time when the code has been written, you didn't have threads, right? What does that mean? That mean that when we started supporting threads, we didn't change the scheduler and we didn't even say, oh, is there a problem there? And indeed there is. The problem that I found by reading the code is that if a thread has a high priority, which is true if it has not been running for some time, it will be selected before other thread of the same process. That's how the scheduler works. And its priority decrease proportionally to the time it has been executing. So as long as its priority is high, it will still be picked by the scheduler. So it can call schedule as much as it wants. It will still be selected. So generally, when you call schedule, say, hey, I cannot do anything. I don't want to be executed. Please pick somebody else. But in this case, the scheduler was picking the same thread again and again. So you could in a loop those 900,000 times or even more the same function. Now, one could say, well, then fix the scheduler. Well, somehow that's what I did. And I had a hack to the function to the schedule function. Well, actually, I suggested a hack and K tennis had it. And is it when a multi-threaded program, a thread from a multi-threaded program asks to be stopped and he doesn't want to do anything, he calls schedule, then you change its priority to make sure that if another thread was blocking it because he was holding a data structure in your text that the one wanted, then you want the one, the thread which is not being executed with a lower priority to run. So the way to achieve that is to decrease the priority of the thread calling schedule to at least the priority of its siblings. So that works somehow. It kind of improve Firefox's edge. It improved a lot of libthread ports, actually, because most, if not all the ports use some pthread function now. And it even improved all the third-party code like Java that rely on on schedule. But was it really the problem I was looking for? Because if you followed my talk to this point, I said that you call scale yield and it doesn't work as expected. So maybe a scheduler is wrong and to some extent it is. But maybe we should not call scale yield in the first place. So that's how I came up to the real solution. I said that I start looking in libthread and how does that work? Why? Because when I say I don't want to run anymore, please pick somebody else. How do I guarantee that somebody else will make progress and that I will continue my work after that? There's no such thing. That's why we had this priority problem, which helped but did not solve it. So if you had like an, I would say an older machine, you could still not use Firefox to play your HD video. So now I hope you're not sleeping. No, it's fine. We're going into the technical stuff. Right? Fine. So I guess all of you have heard of mutexes, lock and shower stories. I'm very famous for the shower and locks but it's not for today. For today I'm not going to talk about how do you use a lock but how you implement a lock. That's really interesting. Because generally you say, oh, yeah, well, I need to protect this data structure. I go to the toilet so I close the door and turn the lock and nobody can enter. Fine. But how do you implement the door? Did you look at it? Well, in software, there's many ways to do it. How was it in 6.1? How was it when we had that bug on Firefox, this regression? Well, that's what I tried to explain. So if you don't understand, sorry, don't hesitate to raise your hand. Yes, Henning? Except Henning. Okay. So what I'm describing here is the pthread mutex lock. So the simple idea to say, hey, I'm a thread, I want you on this lock is to say, well, to write my name, Martin, in the lock. So everybody else say, oh, it's Martin. It's basically what you do when you have a party and everybody has a glass and you write your name on the glass and people know it's your glass. And unlocking it would just be washing your name from the glass and then somebody else can use it. It's fine. So that's what we were doing. But in order to make sure, in order to make sure that I'm the only one writing my name on the glass, right? Well, I would take the glass, go to the toilet, close the door, I have a lock, write my name, then I can open the door and I come back. And I'm sure I'm the only one because otherwise I will fight with Henning with same glass and trying to write my name on it. Doesn't work. So the first function we enter and that I represented in the graph is this lock. So, okay, I grab my glass, I grab a lock and look, oh, is there a name written on this glass on this mutex, by the way? If there is one, because I want still to use this glass, another one, well, I will wait. Waiting means calling another. She's called that you have on the right side, which is thread slip. And I will hope that somebody will wake me up. When somebody wakes me up, I try to acquire the lock again because the function thread slip is designed to release the lock, then go to sleep. And then I will start again. Is there a name written on this mutex? Well, there isn't. Oh, well, then I write my name. I release the lock and now it's my name. So that sounds really familiar. But what's not really clear on this graphic there is that on your code, your critical section, actually what you want to protect with a mutex, start after that. So that's the function you call printed mutex lock. You got a lock. Then you do something and then you unlock it. So what's the problem with that? How did I manage to understand that the problem was to call scaled yield? Is that first we have a spin lock, right? The spin lock is basically try to grab the lock. No, you do not grab it. Try to grab the lock. And every time you fail, you go to the channel and say, well, please, I cannot make progress because I cannot grab this lock. So pick somebody else. Time can make sense. But in this case, we even don't know if somebody has the new mutex or not. So you're waiting before even knowing if you have to wait or not. That might be confusing. You want to protect data structure. So you want to grab a lock. And now you need to grab a lock to grab a lock. Oh, it's confusing. But the first primitive that we have to build is a spin lock. And we don't know if the mutex is hold or not. And we kept some time checking, oh, can I check? Can I check? Can I check? And actually, Mozilla was using a lot of try lock, which is basically the same. You want to look at the mutex. Oh, is somebody holding it? And if there is, you say, well, it's, it's busy. Somebody has the lock. So basically you were busy waiting just to check if somebody has the lock. So you're creating extra contention of the contention. Now, the problem of that is what I've written down, which we are, what we are interested in, that if you have a contention, I mean, if you don't have contention, you don't have any problem at all because you don't have contention. If you don't have a simple system, you don't have a lot of problems. Now, in the content, in contented case, you might end up doing as many schedule and or as many atomic operation as while you, you've been executed or as the scheduler say, what, try again, try again, try again. So what can we do with that? Well, maybe you've studied that. What you want is they say, well, if I cannot grab the lock, well, just go to sleep and tell me later what I want. There is many solution for that. And I say, well, I'm not going to reinvent mine because I want to get the job done quickly. And I want other people to contribute. So I went for a well-known solution and went for a solution based on Suretex. So Suretex, you might have heard about it. Some say it's coming from BOS. It's a different name. It's well documented and there's many options and many subsets. But basically, in the implementation we have on OpenBSD right now, the way we use it, it's the diagram of the graph on the right still on this. So the difference is that you rely on an atomic operation. And if the atomic operation fails, you know that somebody is owning the Miutex. So you don't need to spin loop, spin loop. You just do one atomic operation and no scared you. And then, oh, somebody has a Miutex. Somebody wrote his name on the glass, right? So please tell me, you tell this guy with the Suretex syscall, please wake me up when you finish with the glass. When it wakes you up, you still have to do another atomic operation to make sure nobody stole the glass between you and you really check it. And if nobody wrote his name, well, nobody took the glass and it's for you. And you have it now. So the really interesting here is that, well, you really reduce the number of syscalls and atomic operation in the contented case, which was the problem we had, or I think it's a problem I experienced with Firefox and a problem that created like a snowball effect with the scheduler we had. And this new solution, which is not new actually, it's just a new implementation, solve a lot of latency problems for all the parts. So why did I go for that? Well, I kind of explained, but there's a little of documentation. My time is limited. I want to solve the issue that I want to watch a video. I don't want to reinvent something that already exists. And it's already complicated if you have to reimplement the syscalls, if you have to rewrite the library bits without a real debugger working. So if I can rely on existing tests, well, it helps. That's free software. I like it. Now, of course, there's much more left to do. The solution is better. Right now, this solution that is described with the mutex is only enabled on I-36. I'm 64 and mutex 64 architecture. I think it's a matter of testing and maybe finding the remaining bits. And there is a challenge for some architecture that do not provide the atomic operation, this solution relies on. But other operating systems solve that, so there's possible options. Now, I concentrate myself on the most critical part of the lib pthread library, the one that we're really contented and used by most of the program. We can continue and clean that. And I can come back to the scheduler because now that the bottleneck, the real bottleneck has been fixed in user land, we are exposing a new bottleneck. Software is never finished. You always find bottlenecks. So what did I learn with that? It's kind of my story. And I wanted to share it because I think it's more important to explain how I do stuff. And it's not magical stuff. And it's not over complicated stuff. But let's go step to step. I do mistake. And I have to live with that. So you always will always have a problem. And no matter who you are talking, it's not good enough. And they're always to change it. But actually what people want, and when I go to some consulting, they want the problem to be solved, no matter how you solve it, right? I like to fix free software. So I do that. Gathering data is like black box analysis. What I was saying is I think a method which is not very well practiced, either in a professional IT world in general. I was discussing that with a lot of people today. And it's really nice that we have all those people talking about tracing tool, performance tool, because when your car does not work, you don't get by a new car. Look at what's not working in your car. Or you bring it to somebody. And we have to do that. We might not know how to do it properly, but there are many tools that are available since 30 years, and they already fulfill a lot. Even if you don't have Dtrace, even if you don't have a real debugger, you can't find most of the stuff. Now, I make a lot of guess. I try not to make them, but I still make them. And it's really complicated. When you make a guess, you make a diff that sounds great, and it hides the problem. It's a problem because it might solve your problem now, but another developer will come in five years and will hate me because I'm hiding the real problem. And now he has to undo my work and redo something else. And I can tell a lot about that. So finding the bottleneck is generally the hard part about it. So it's great to have great tools, but even simpler tools can help with that. I love to fix them, and not everybody can fix or like to fix them. If I could just spend my time fixing the problem, I would be even more happier because I don't have to find the problem, right? So I hope that you can learn from my experience and also find the problem and then tell me, oh, I have this problem. Can you fix it for me? And, well, I think it's kind of another buzzword now. Yes, we need a dynamic tracer. I encourage you to come to the conference of Jasper tomorrow because we are working on that with OpenBSD and we are proud of what we're doing and we like it. So come see what we have to say. Thank you for your attention. Oh, I went too fast. I'm not going to exit. Well, so I'm keeping a blog. I'm trying to write articles and explaining how I do stuff, how we do stuff on OpenBSD, and you might be interested on a particular subject. If I can make that more accessible to you, well, come talk to me. Now, if you have any questions, don't hesitate. Did you implement like visual, like, ways to search, ways to solve the text or also like priority characters or was it in the previous way? I just implemented. But recueing as well. Now, recueing is kind of tricky to use correctly. And I had a work in progress version for the condition variable, but since it's not yet the bottleneck, I don't want to spend too much time on that. Why not? Sure. Any other question? Well, nothing prevents it, but I find bisecting Firefox code really complicated. So for me, if I have to debug a Firefox or proprietary application, it's the same. It's like too big. And I don't know if I thought at that time like Ted wrote an article and I got contacted by Firefox developer would tell me, well, maybe we might help, but I don't know where we have to look at. And somebody has to bisect and compile Firefox. And you know how much time it takes every time you compile it. And the point of the approach and the talk I was giving is exactly, yes, you can always bisect, but bisecting only gives you part of the solution. So I can give you another example. I was talking, we had like those security bugs found some weeks ago by a bunch. And as a measure, we decided to remove a similar code that we found suspect. And I'd remove some of those code in a USB driver. And suddenly I introduced a regression. And people did bisect my channel and say, your change is wrong, but I still could not figure out, okay, now I back out the change, but I still, what's the problem? And we managed with K trace and by looking at what's happening to see, oh, it's blocking. It should not block there. So when you run talk, you say, it's blocking there. It should not block there. And bisecting does not give you this information. So I think we should start or continue to learn this tool, what's happening in the system and not just focus on the code or more questions. I was really sad. Not user facing problems might be even easier. Let me explain you one. What you need basically is a way, well, the approach that I was describing and I'm trying to apply is the black box approach. You consider your software or your appliance on whatever as a black box and you say, well, when I do that, I have a problem. So what do you mean when I do that? That's the problem we have to answer. So if you, for example, measuring a firewall application and you say, ah, look, I configured my system, I send that traffic and one that's in more than that many packet per second, it start doing some crazy stuff. So now you have like a reproducible scenario. That's a really important part. So you try to build a scenario that you can repeat and that you can monitor. And when you build this scenario, you have to make sure you don't introduce too much noise in it. In the case of Firefox, I try to build a really small scenario just moving the mouse. But when you're like forwarding packets, maybe you want to make sure you forward packet for enough time so that you have like a stable flow that triggers the problem so you don't have interference. And once you have your scenario and you can reproduce it, then you start watching the metrics. It's not an assumption. It's the beginning of the problem, right? So I think we should always look for it. So when you work for performance, it's in my opinion, and the way I work with that, it's an every week or every day work. So you have to allocate some time and to start gathering data to have more knowledge of what's happening. What does my application do? Which syscall do they do? How much time do they spend? And if you say, oh, look, I have some regression tests, regression performance tests, and this week it's much better than last week. And you have data and you start to why? Oh, this week is worse than last week, right? So I would say you should always look for it. Not maybe the full week, but take one hour or two. What does my program do? And if you do that in your development process, then you will create awareness. And that's the most complicated thing, because you cannot address performance by just one day. Oh, now I want 100% like generally manager wants. Well, this month we'll do performance. Oh, great. No, you have to build awareness around the time to look all the time what you're doing. Thank you very much. Thank you.