 My name is Malton Rasmussen. I work for ARM. I suggested this powerware scheduling boss session hoping to meet some people that are interested in this topic and have some useful discussions. I hope this will be very interactive so and because it's all recorded please use the microphone when you ask questions. Actually it would be better if you set closer up here so we could have some some interaction. So please move forward if if you want to participate in the discussion. I'm not sure what what the audience here today is really. I have some slides I can talk about what we are what we have been working on why we're trying to push this powerware scheduling idea give you an overview of what we've been working on what have been discussed at a workshop at Colonel Summit on Wednesday. But really I want to hear what your interest is in this topic as well and and how you deal with power management on your systems. If any of you are developers working on specific applications on specific systems what what challenges are you facing. How do you figure out when is my system tuned well enough or what's important for you. Is it tuning CPU Frank is a tuning CPU idle because we we are discussing various ways of actually changing these things. So it would be very useful to have your feedback for for these changes. So powerware scheduling is not just about the schedule. It's about the schedule and all the power management frameworks that we have in the Colonel today. Most permanently we have CPU Frank and CPU idle. These two frameworks both have drivers and governors that sort of try to implement some some power management policy pick the best C state pick the base best P state. But as it is right now they're actually completely uncoordinated and they don't get any information or they don't give any information back to the scheduler. So we sort of have the scheduler doing something places task running on one CPU on other CPU. And then depending on what the scheduler did CPU idle tries to do the best thing with whatever the scheduler decided just before. This is not ideal. We could do much better if the scheduler actually knew about what are what are our idle states what's actually the cost of waking up this CPU compared to this CPU. Because if you have on our systems we have typically various couple piece that is sorry C states and it might be very expensive to pick one CPU over the other even mean powering up an extra set of CPUs. So there is a good potential there for actually improving things by by making the CPU idle CPU Frank and the frameworks talk to the scheduler. So this is really what what power scheduling is all about. I tried to depict here roughly how the the the different power management policies are playing together today. We have the the scheduler it decides the load balancing it decides decides which CPU to wake up if we need another one. But it doesn't really know anything about C state or P states. That's something that happened sort of by the side and we can actually have some some really bad side effects by not having this coordinated. For example if your CPU Frank governor is too slow to react to the load of the CPU the scheduler might actually see that a CPU is becoming overloaded and start moving tasks away. But the reason why it's being overloaded is because it's running at a too low frequency and CPU Frank just haven't ramped up yet. So this is a clearly a problem. So we need to fix that this we need to coordinate things. And this is what this work is all about. So yeah some of the issues that we're seeing right now is this scheduler CPU Frank scheduler feedback loop which is not there at the moment. Which means that if CPU Frank is too slow the scheduler might do some suboptimal choices. Other things that we need to save power is another thing called task packing. At the moment what we're seeing is if you have sort of a lighter load scenario where you have we see it on Android for example where we have things waking up in the background if you're just doing nothing real just look at your home screen or something. As it is right now the scheduler doesn't really care where things are waking up. So you might wake up different to be used at different times which is clearly not ideal from from an idol perspective because it keeps CPUs in more shallow sea states than are really necessary. So what we're working on is actually trying to figure out when can we pack all these tasks to actually run on just one CPU and that will be the one waking up all the time and then leaving the rest of them in deeper sea states and therefore save power. These are just some of the things we need to just add topology awareness into the scheduler so it makes better choices and that was really what was discussed on Wednesday in the workshop at Kernel Summit. Another thing which is really important for ARM is we need some support in there for our big little architecture which adds another dimension of complexity to this problem. But it's all a part of the same solution. We can't really take out one problem and just work on that. We need to consider the whole thing. So there are some things that we wish would be there in the scheduler. I don't know if you're familiar with all how things have been developing in the scheduler recently but in 311 there was added a different way of estimating how tasks, the weight of tasks inside the scheduler. In the past they were just assigned a static weight. So no matter how much or how little a task was running it had sort of the same weight in the load balance calculations. That was changed. So now the load of a task is actually based on the history of execution of that task. That is good but that history is affected by CPU frag because the way the history is constructed is that it's based on the time that the task was actually running in the past. And if you lower your frequency it means that your task is running more and it will appear to be bigger. So here you have this feedback thing again that needs to be sorted out. So clearly there needs to be some integration between the scheduler and CPU frag. Yeah. And in general we need to apology awareness and there are other things we want to address as well. There are lots of things we want to integrate into the solution. That's the thermal and power budget awareness. On some systems we see that we have a fixed power budget and we can't just run everything at the same time. And that's not just CPUs we're talking about here. It can also be GPUs or the subsystems. Things that I'm not taking care of at all at the moment. But things we want to address going forward. So yeah I've tried to illustrate here what we have in the scheduler at the moment and some blue boxes showing what we would like to have in there. So we want to extend the SPOF to be more aware of theほしずつ witch. And when we get to that point, we need to do it. And we need extra features and load packing algorithms. We need scale-in variants. We don't have the have a patch set yet that does all of this. There's been proposed a number of different solutions over the last year or so, packing, integrating different things, but they don't really fit together as it is right now. So we need to work together and figure out how is this all going to work to create the solution. So the outcome of the Kernel Summit workshop that we had on Wednesday is that there was a broad agreement that we need to move the power awareness into the scheduler. We can't let it sit outside because it will be very hard to define. If we were to let CBU Frank and CBU Alexist as they are today, we would need to define interfaces between the CBU Frank governor and the scheduler and it will be very hard to define the proper interface there. So the proposal is move that functionality into the scheduler. And actually one of the first steps that we might look at is actually moving CBU idle into the scheduler and maybe completely get rid of the governor concept because if we're in the scheduler, we'll have much better information about what's going to happen and that may completely eliminate reason for having governors. If the scheduler is just the best place to have it, then just let that drive idle state selection. So that's one of the things that will be worked on. Pretty much everyone agreed that CBU Frank is not fit for its purpose if we want to drive it from the scheduler. One of the problems we have with the CBU Frank framework as it is today is that drivers are actually allowed to sleep. And if you try to call that driver from a scheduling context, you actually get a recursive call into the scheduler. So it doesn't work as it is. We need something else. And on some platforms, changing frequency is actually not that complicated. If you take Intel, for example, it's just writing a register and that's it. That's easy to do from a scheduler context. And if we do it from the scheduler, we can change the frequency based on what we know is being scheduled. We don't have to wait until the load goes up and the CBU Frank driver recognizes that, oh, there's something running now. I should ramp up the frequency. If we do it from the scheduler, we can ask for frequency change at the same moment as we're scheduling a task, which we know based on its load history is a big task that will need the frequency to go up. To make energy aware decisions, we need to put more information into the scheduler about how does this platform actually work? What sort of energy can I save if I wake up this CPU compared to this CPU? So that was also one of the outcomes of the workshop that was we need some way of telling this scheduler what is the energy trade-off of ramping up the frequency on this CPU compared to waking up another one and putting tasks on that. Because, for example, the task packing I was talking about before is not so much about just packing tasks until the CPU is full because does that mean you should pack it until it reaches its maximum frequency and then when you have sort of overloaded the CPU at that frequency, then you should spell on to the next one. That might not be the most power efficient way of doing things. It might be going maybe 50% or 80% up because then the last frequencies you ramp up your voltage so much that's getting too expensive in terms of energy. So if you have a number of tasks running at the same deep CPU, you might be better off by just splashing them on to the next one so you can run two CPUs but at a lower frequency. Those things, the scheduler doesn't know anything about today and we need some weight or some information in there to let the scheduler make better decisions. Another thing that was discussed a lot was benchmarks, power benchmarks. How do we actually evaluate whether the changes we're proposing are actually doing anything good? The patches that have been proposed so far, some of them have come with a few numbers and a few simple workloads on one specific platform and some improvement but really we need to figure out how to actually measure that the patches we're proposing are actually doing something good in sort of the broader sense. We don't have as it is today a benchmark suite of power benchmarks where we can say okay we can run all of these and we see improvements across most of them and therefore we should have these patches in. That is completely missing right now. It's not an easy task because what is the power benchmark? How do we measure if we actually get what we want? It's fairly easy for performance. You just run it and you measure the execution time, your throughput, whatever but for power it's not just about using the least possible power because it's fairly easy to save a lot of power on your phone for example. You can just turn it off then you save a lot of power. It's really a trade-off. We need to find the right spot between performance and power. Maybe you're fine with giving away 10% of your performance if you can save 25% of your power but clearly you're not willing to give away 90% of your performance to save a lot of power. It's not even that. For some workloads we don't really care about performance. Well, we do. For example, video playback. As long as we get every frame, no frames that drop, we're actually fine. So maybe the metric for a power benchmark is actually user experience in some cases but that's really, really hard to measure and it's really hard to come up with a user experience score that will apply to all use cases that you can think of. That's actually one of the things where I think it would be nice to get some feedback from you. How do you actually tune your systems? If you have an application, when do you decide that now I have saved enough power and when do I get enough performance? Are you just monitoring your frame rate for video playback or how does it actually work for you? Is there anyone? Okay, so you're saying that your way of doing is that you have a power budget and you just do whatever it takes to meet that power budget but how did you come up with the power budget to start with? Okay, maybe we should pause around. This is working? Yes. Maybe we should pause around the microphone. Yes, so my point was that when I used to work for TI, the methodology was to define power targets or power budget and thermal budget as well in the early phase of the chip development and then with this model that would produce a power target and a power budget and then once we got the software running, we would compare the measurements with those budgets and try to reach the budgets and iterate or eventually reassess the model because usually that was based on assumptions so some of the assumptions was not fully agreed and so this iteration was used to converge towards some supposed optimized power consumption for a given use case. Okay. And so the other part of the answer was that we had no specific methodology to say that, to tell us that yes, you didn't degrade the user experience. That was the next issue, not issue but unresolved completely issue like what can we monitor to say that those power optimization, usually when you do power optimization, you somehow degrade the performance by entering more low power states, running CPU lower, et cetera, et cetera, introducing more latencies in the system. And so we wanted to make sure that the user, the final user, so we were working on mobile phones, so we wanted to make sure that the experience wasn't degraded. So in the case of video playback, we were making sure the framework was fine. We were making sure that there was no buffer on the floor, for instance. So we were monitoring the video buffers and making sure there was no underflow. Without any underflow, we could say, okay, there is no degradation. The jitter also, we would measure it, make sure there is no jittering on the screen, things like this. For audio playback, for instance, we were making sure there were no glitches. You couldn't hear any glitch. But there is no formal yet. I agree with you. There is no formal methodology to a power benchmark. It's very nice to say it this way. There was no power benchmark saying, yes, this one, you're good. No, okay. You said, okay, we will give you a discount on our technology for one sensor, a mobile phone. You install our power waste schedule, you give back information by the end user that uses your mobile phone. And then we know that your scheduling is doing something better. So it's very nice. So using the mobile phone on the market to measure the, because it's at the end, I know the problematic area. But then the user experience of the data collected by the real devices, because they use, they can play video, sometimes they can listen to the music, they can do everything. So I think that user experience, so coordination between manufacture and user mobile phone and arm or Intel company will be the best to prove to give a proof of concept or your scheduling algorithm for power aware, because, and this is very fast, because you can collect every information you want from video sub system to audio sub system to everything. And you know exactly what the task do, what the standard use case on the end user, because yeah, right now you don't know. So basically silicon vendor or end user vendor, like mobile phone vendor, can you give you an idea what is the use case, the typical use case. And in this use case, you can balance your, you can have an idea what is the good benchmark to test your algorithmics. So there are some of what you had the script like balancing is nothing to prove in that one. If the frequency scaling change the balance of the task, these need to be fixed because this is a problem scene. So it's not related to power aware, it's related that it's wrong, the calculation is wrong. So you need to account the right, you need to give the right way to the task. But for me, because I work a lot on this and what I think is, and only the end user can give a proof of concept of what you are doing. And now the only way for me it's let the end user to test the algorithmics. And if you are in the position to say, okay, we can give you the possibility to experiment our algorithmics in some way. So mobile phone is a good way to do that. In that way, you you really prove that you save power and you not give a bad user experience to the end user. I see a point, but if we need to give away our code or distribute it on device before we can actually get feedback about how it works, it's sort of a bit latest that you okay, but you will never know exactly how it's running. So this is a good way to balance in real time your algorithm and have more and more informations. So if there is one million dot alpha activation a day of Android and a potentially 800 million than the user, you know, it's better than do they write your algorithmics without having any feedback. So you need at least collected information about the use case by by the end user. So not give not only put the algorithm in the mobile phone, but at least have an idea of what is the power consumption, how they use the mobile phone to simulate know the algorithm using a real use case. Yes. Well, one of the things that came up at the current summit discussions was that they basically wanted a set of benchmarks that we can run when we have the patch to see if it actually does good or bad things. And waiting until it's in the product and then getting the feedback is this maybe well, it's definitely useful to have, but we need a filter before that to pick which patches are the best ones. I believe. Yeah, there's question behind you. I think it's simpler than that. And in the case of say 3D video, there's really two targets, which I think have any real worth. One is if you're playing something like video or just a user interface that's drawing things on the screen. You want it to hit a certain frame rate so it looks smooth. And that's all that really matters. So just the minimum number of cycles to hit that frame rate is all you need to account for. The other target would be latency. So if you're playing a game, latency is king. And all you need to do is drive for lowest latency, which is going to use probably more power. As long as the benchmark is taking to account both cases, and you have a figure of both, it's really up to the user to decide which particular, you know, option they want to choose to play a game. They probably want to switch into low latency mode. But of course, right now, video is not done a lot by the CPU. So I understand that the video could be a good, but 10 years ago now, if you do a video, the 70% of the encoding of the video is done by other parts of the COC system. So even in the video, the problem is that in the real case, I show you an example, very simple. Suppose that you optimize the power consumption of the system by phone, and the user is very happy. Then suddenly arrive a mobile phone call, then the CPU needs to wake up in some way, but you need very fast feedback to the end user because for the end user, it's very important that the ring come and not the weight or delay by the running video. So this is the reason that I said that the use case, user experience is very different for everything that you think in your lab or in your room. So this is the typical example where collecting data from the device on the field is very important to understand what the real use scenario. So I think that the power we're scheduling need to be very simple. So you can collect every data, very nice things, and do the better calculation inside the scheduler. But this is a very basic calculation because you end up to spend power to understand how to not spend power. So if you need to take the decision in a slack of 10 milliseconds and you spend 80 milliseconds to take this decision, you lose the slack. So my approach is very more do it simple, like what you described like calculate the right balance of the task. This can be done in a simple way. So do a lot of things simple and let the high level frameworks to decide some power, high level power scheduling decision, and not let all to the kernel. So this is my idea on what would be power aware. So doing the decision but quick, even not optimal but quick, as quick as possible, to not spend time in the decision. Yes. There's a question behind you. I think I agree in that point that it needs to do simple and quick decisions. And maybe just very primitive support for power benchmark is needed in the kernel. Because if you look at power consumption, for example, for a smartphone and CPU, it takes very tiny part of it. And you need to look at the screen, Bluetooth, and so many other peripherals as well. In order to make that kind of decision, I don't know if it's suitable to do that in the kernel, actually. So to start with, like the last bullet there, to define use cases, that's very good to start with and to have some primitive support first. Yes, I fully agree. I think only the kernel really has the knowledge of what power domains are where. I don't think anywhere else really has the information needed to do scheduling on that sort of basis. It would be really nice if the kernel could sort of wake up three or four tasks in a row that all want to access the same peripheral so that it can wake up, do three or four tasks, and then just go to sleep again rather than be woken up constantly. There's another question. I just have an idea very personally, because as we mentioned, the power consumptions. But first I think the user case is very important for this scheduling. Suppose if somebody just uses a mobile phone, it could just recognize such like if they have some scenarios to check the new email periodically. If that case is very, very surely, that means it would use some very lower consumption cores in design phase, it's OK. And if someone is just to do in the very high-experienced games, so it's just a user very high performance cores. So I think if we want to just do the dynamic power wearing scheduling, the application should have the characteristic like that. Some part is just a very heavy load, and some part is maybe some light load. So that is very dynamically. So that means if we want to do these things, maybe I could provide some tracing, or as you mentioned, some payload detecting mechanism. That part could be unique, provided to some open source developer peoples, such like cross sourcing. Like cross sourcing, that means everybody just running the same benchmark or same trace tools on their own device. And they could feedback to you, what about their consumptions and use about their applications. It's intensive computing, or maybe less intensive computing. So I think if you gather all of these feedbacks, maybe you could just calculate the real scenario is exist or not. And then from hardware side, you could calculate the energy and the power consumption. It's suitable for a little core, or maybe it's just suitable for dynamic change between big core and little core. Because currently, Huawei's high sitcoms is also manufactured with arms, big little cores. But currently, it's just these problems, challenges. No sure how about the scenarios should be placed by that. Because currently, from the software side, we do not aware about consumption. Because if I program the applications, I do not know how about the frequency about the cores and how about the CPU is I do or not. Because in my program, I do not invoke any system call or any interface to determine the core is busy or not. I do not know. So that means if we want to do the power awareness scheduling, that should give some hint from the hardware side. My program has a characteristic like this or not. Suppose like somebody inside a room, if I always inside a room, you should not power off the light. And sometimes, many hours, I just inside the room. And sometimes, outside the room, you could turn off the light when I outside. But this should be very, very, very cooperatively tightly. That means when I just go inside the room, the light should be power on. So that is very, very dynamically inside the operation. So my suggestion is first, we focused on the use case and maybe I'll provide some tools to find these characters. If it's truly in the scenarios used, we could just to find out and to summarize these characters to provide some power awareness scheduler. That's all. OK, thank you. That's interesting. The problem we have right now when we talk about power scheduling is that inside the kernel, we don't really know about anything that happens in user spaces as it is at the moment. What we can see is just we'll have some tasks, some run a lot, some don't run a lot. But we have no idea which task is actually an important task that actually directly affects your user experience and we don't know what is actually. Yeah, that's very challenging, I think, in the background. And that information can only come from user space because we simply don't know inside the kernel. Because from user space, so many applications is run very dynamically. So I think maybe we could not know how about to recognize it's suitable or not suitable. So my suggestion is maybe we could just do this like crowd sourcing. That means provide some basic tools, such like we provide some basic method. Maybe it's based on Perf or maybe based on Ftrace or maybe LTTNG, yeah, to let the different user to identify how about their application it's have this character or not via the open source community. Yeah, maybe very personally, very personally. So just share. I'm not 100% sure what the tool would do. That tool should, that means suppose I am software development developer, I could just find maybe my application have this character or not because I know my application. Take the microphone, please. I maybe understand, so it's possible to characterize the application, understand which section of the code are CPU intensive, which section are not CPU intensive, which section are important for application point of view. Having this information, what this suggests I think is I have some application interface that is able to give this information to the DT kernel. So the kernel can make some decision or they, I prefer that the CPU made some decision. So suppose that you can inject instruction that he said from now end up to this section of the code it will be a CPU intensive, it's important task. So the CPU knows the prefetch instructions, the CPU prefetch knows that from this, this distraction to the end, it not go down in power or not switch offs unit like I don't know, FPU or because you know that you need. So the idea is to give information to the CPU when the application needs more power. In that case, it's not the kernel that she knows, the kernel just send the task and the CPU has the knowledge about what is going to execute in this CPU or in this core or in this unit. So if we have, for example, for a HALU unit and five FPU unit and the task said, okay, I not an important application, I can use just one adder and just one FPU unit and not multiply instruction so not pipeline a lot and prefer to switch off this unit and let the good application a little bit slow but reduce the power consumption because you have characterized your application. Okay, yeah. This is bypass the problem so understand from the kernel what the application, if the application is power or not because it's the application parameters that inform that he needs more power or he needs more better services and that's it. This is the same if you do, for example, encoding algorithmic, suppose that you have a section that you want to reduce dynamically the algorithm because you would like to consume less power so you can inject instruction and you can say, okay, I need less power basically or I need less services. So it's a way to dynamically inform the CPU what is going on at the user lab because the kernel doesn't know exactly what the application, what are the application requirements? I think you could actually get away with just telling the kernel about it because the kernel will do the scheduling and if it knows it can trigger some Ufriq or something. But it's not fast decision, you need to raise instant with a section of the code, correct? Yeah. If a programming user space needs to run then it will be runnable and the kernel will schedule it at some point and if it continues to be runnable repeatedly then it will get scheduled more. I don't see this as a problem. Well, because it's runnable more doesn't mean that it's something really important. Yeah, but something that's really important will probably try to run more often on average. I don't know, you can have. Yeah, I'll have it written as well. The important that does not mean that it matters a lot. No, you can also have some setups where you have a multi-thread scenario where you have a lot of shorter tasks that sort of depend on each other. So they actually add up to become quite a lot of work. So if you don't run all of them fast then you will hurt your user experience. So it's, but just looking at the size of the tasks it actually doesn't tell you that much about how important it is. Well, you can use it as the first choice. Well, if it is a big task then it might be important but you're not guaranteed that it's important. Yeah. And the same thing the other way around. I have a question down at the back. Well, I think a lot of the things I was thinking of was already talked about. So let me just literally summarize it from my point of view. Well, for me the problem is some kind of three-layered. It's one that the application layer which actually shouldn't know about how the kernel is achieving any goals. And the kernel has certain means how to achieve some goals. I think the key would be to find parameters, how you define really from an application point of view what this application has and demands. That might be an interface which the application when it comes up tells the kernel or some layer beneath it what it wants to do. Or it might even be an interface which is periodically somehow triggered so that something can be derived from that. That would be for video playback on these things. I would think about maybe these periodical stuff. And then I would think there should be kind of a layer in between which does kind of a translation to the actual capabilities of the kernel so that the application really can define that in application words. So the application program I shouldn't know about how the kernel does it at the moment or on that machine or whatever. Our goal should be to have kind of generic applications. And on one machine it's done like that and the other machine is done like that. We should try to find terms which an application programmer can really, in these terms, tell about his needs. Then we would have some kind of a translation layer and beneath what the kernel can do. Of course, these things are not independent from each other. Yes, I fully agree that if we are to introduce extra syscalls or extra C groups or something to identify which tasks are important or not. It needs to be very, very high level because there's no way that the application program should deal with the complexity of what's happening inside the kernel. And I think we can actually get quite far by just having the application programmer tell the kernel if this task is an important one or not, which is a very high level thing and I think it should be possible for the application programmer to identify that. And even the application should not tell one application how the kernel should behave all in all. Because usually there's something more going on than one application. Of course, maybe one application has the focus and because of that has certain demands, of course, but there's something going on in the background and other applications. So these layers somehow would have to sum it up and optimize it all in... Well, try to meet all the requirements. Yes. So it cannot be just one application controlling the kernel, that's not the thing. Yes? Yeah, yeah, I fully agree with my colleague. I don't think the application should drive anything because that's part of the definition we have user space, we have kernel space and when we write the application we are not supposed to write it for a given platform except a dedicated tool. So a tool, sorry, an application is written and it may run on a mobile phone, on a tablet, on a PC, whatever. We have written a programming language that completely abstracted and that's the purpose of writing application and developing frameworks is that the application doesn't know the platform it's running on. So we cannot trust the application also because, well, I don't know a developer that would like to write an application that would voluntarily degrade its maximum performance just to save power. I mean, if we would provide the capability to an application to tell the kernel, well, give me the lowest priority. I mean, if we say that, the application would by default say give him max performance because today when you download an application or a 3D application or whatever, you wanna make sure you have the highest frame rate possible that you don't want to have any lags so no one would like to do that to say, okay, I want to minimize the power consumption because you know that you may degrade your performance and your experience. So I would rather state that we would need some sort of system policy or something that understands what's happening on the platform at the moment, not only on the CPU but also on the GPU, on the hardware accelerators, on the peripheral buses, some things that understand actually what's the use case running on the platform and that would provide hints to the scheduler to whatever other processing units because we talked about a lot of CPU, CPU centric and the CPU kernel but actually there is also some sort of kernel running on a GPU, running on a DSPU and anything else. So there has to be some, I would say, system power management framework that would gather information from all different parts and would feedback this information in the right format to each specific unit. The Linux kernel should have some APIs provided so that we could drive the decision. We could have just a small cursor, for instance, that you would say, well, here at the moment I want to be more power oriented than performance oriented and so based on that hint, the scheduler would decide whether or not to pack to do any sort of decision. I mean, we have to understand what's happening on the platform if you have a display that is turned on, if you have the GPU that is also running, it's not only about the CPU today. No, but how do we know from inside the kernel what's important with this load? You have many of them. Yes, but it's not the scheduler that has to know that. It is some sort of system policy that would understand the use case or the use cases that are currently running on the platform and given that knowledge and the thermal and power budget it knows about would give hints to the scheduler to be more or less aggressive in terms of power management. The scheduler would still do the scheduler thing and what you proposed before is very nice to provide some feedback to the scheduler from CPU idle and CPU fraud because you're right. They are corrupting each other and so today this is a big problem but then there is also something else on top of the scheduler to help him also take better decisions based on the other peripheral on the system. Yeah? About the feedback from the application developer... No, not from the application, from something else, some intermediate layer has been mentioned. Coming back to your point about the application developer would always ask for the highest possible performance. I have the same concern but at the same time I think if you do that and you use too much power you get pretty bad reviews on Google Play or whatever. So I think the application developer should actually start worrying about how they actually write their applications. You are right but at the same time look at Android application I would say because they are everywhere. You would say that you have to write a different application depending on the platform it would run on. No, it doesn't have to be platform independent. I think just a hint like saying that this threat I've created here that actually renders my user interface so that is important but I have this other threat that indexes my emails or something in the background. That is not important. It's simple things like that that can make it easier for the kernel to actually do the right decisions. Yes, but Android can know that. The application doesn't need to tell Android that it's using the display. Excuse me? Yes, but you may put some middleware that would drive and provide the information through the kernel based on... I mean today you know if your Wi-Fi is on, you know if your Bluetooth is running, you know if you are doing a lot of data sharing, if your display is on, you know how the external world is by the temperature sensor. So you have to make your system more smart, more aware of its environment and more system aware so that you can understand the use case. Today we have a lot of limitations because the only thing we know is what the scheduler is scheduling and we care very few about everything else. If you had this knowledge and you could concentrate this knowledge into something that knows the platform, that's exactly the point you actually raised about knowing the platform, then you can take the right decision. If you know that you are listening to MP3 and your screen is off, you may understand that well, you are in the train, in the plane, whatever, and you just listen to music, so then you can be very power-aggressive. But at the same time, if you know that you have your 3G connection, you know that you are doing web browsing, whatever, you know that here what's important is the user experience, so then you can relax a little bit the power. You can also know, for instance, you could monitor the battery level and starts getting more and more aggressive with regards to power. The, as long as, when the battery gets low, all things like this. I think it's up to us to more understand the system and the system has to know how it is being used and then take the right decision. And we shouldn't let that go to application because today the application developer, they don't care much and they don't want to. That's the point of developing application and developing kernel drivers. Okay, it's running too much. We'll reduce the importance of the applications and then rebalance again. I agree that there is, we need some middleware. We need not to do everything in the kernel. We need some middleware to decide the importance of the application. But in the end, if I want to watch a video and I like to consume a lot of power, I don't want that the scheduler takes some decision and then reduce my feeling of, for example, listen to the music. I listen to the music. I have better responsibility for my application from other applications that run in the system. I can decide by myself but to increase the importance of the application at user space level. So in that case, it's... I think that my voice is so loud that... I think we're actually running out of time. So I mean that it's end user decision if the application is running or not running so fast. So give the possibility to the developer to decide the importance of the applications or give the user to rise the importance of the application. Give some information at the kernel level in the end that you need to give more budget. Let me say this one. This is more share of the CPU to this application. Or just because they don't want... You want to consume more power but because you want to listen to the music or watch a video, I don't see any problems on this. So the scheduler need to do a quick decision based on the load but it's the end user that can decide to consume less power. So I want something simple in the kernel and give the infrastructure at the user level and the middleware to give some help to end user to have a better feeling on the application side. Oh, let's system DD side. Yeah. Okay, I think we need to stop here. We're already over time but I'll hang around so if you want to continue that I'm here. Okay, thank you everyone for coming.