 Machu Ghar from Red Hat, he's speaking about enterprise power management features, so Machu. As Dave said, my name's Machu Gharis, I work for Red Hat on the Enterprise Kernel hardware team. So most of my job is spent working on power management on server level hardware these days, although there's a fair amount of also looking at desktop hardware. I'm going to be concentrating on enterprise level power management for change, because mostly when we think about power management we're still mostly thinking about laptops. Obviously something that's close to all our hearts, people really like having increased battery life. But in terms of the overall power consumption of computers, laptops are not really the biggest problem. I have to say that OpenOffice has suddenly got much better because it's giving me my slides on screen, and the next slide is actually not the next slide rather an empty box, but now my current slide is an empty box. So you'll excuse me if I spend some time staring at the screen rather than you. Data center's power sink, incredible power sink, something like 3% of the United States total electricity consumption is devoted to data centers. Now in terms of, well, 3% doesn't sound like that much. The US, however, does tend to consume rather a lot of electricity. Something like 15, or a constant 15 terawatts averaged at. That's approximately 15 average sized nuclear power stations devoted to keeping the internet running in America. That's quite a lot of nuclear power stations. That's quite a lot of electricity. And reducing the amount of energy we use in data centers has a measurable direct impact upon the total energy consumption of society. A lot of this is overhead. Average data center in the US presently consumes approximately 1 watt of electrical overhead for every 1 watt of power consumed by the computers themselves. But some of that is stuff like network infrastructure. Some of it is just power supply loss, but the majority of that right now is spent on cooling infrastructure. Something in an average data center, something like 0.8 watts of power is used for every 1 watt just to keep the temperature stable. So every watt you can slice off the power consumption of a machine is less cooling you need to do. Because, well, we would love to think of computers as devices for turning electricity into awesome. Computers are in fact primarily devices for turning electricity into heat, which is fine if you're in Iceland. It's less convenient if you're in Florida. Why people put data centers in Miami? I don't really know, but apparently they do. So we can improve this a lot merely by ignoring to begin with power management of computers themselves, but looking at how we can reduce the overhead. And the two big things in terms of reducing the overhead are reducing the amount we spend on cooling and reducing power supply loss. There are several initiatives in the US and I assume elsewhere in the world at the moment to improve this. Data centers in the US, if they're in the bottom 25% of power consumption per machine, will get a special award which then is supposed to be good for marketing purposes. There are now energy star compliance programs for power supplies. We're seeing power supplies move from around 75% efficiency to 90, 95% efficiency, so that's a big difference as well. But in terms of overall impact here, the main difference is people working on redesigning their data centers. People are doing things like relying on ambient cooling in order to reduce the, rather than relying on air conditioning, if you can use more environmental air, if you can just use air from outside, if you can run the data center at a higher ambient temperature, the only people who are likely to be upset are all people who lock in their computers. Nobody else is likely to notice. And this is improving things quite a lot. We're seeing high quality data centers in the US now get down to about 0.2 watts of overhead per watt of system power consumption. So if that can be repeated everywhere, we've already pretty much halved the power consumption of data centers. And we haven't had to think about software at all, which makes my life a lot easier, because I don't do hardware, so, fine. Well, that's kind of a low-hanging fruit. And guessing that sort of figure requires this to be a consideration from the moment you start building a data center. Companies like Google are now doing things like using groundwater as a cooling mechanism rather than using air conditioning. You just pump the water around and then, if you've got a good enough water supply, and you can just put it back in the ground and it'll cool down itself, the earth is a wonderful heat sink. I mean, if you've got an entire planet to dump waste heat into, then life's a lot easier. The things are people starting to put data centers in cooler locations to start with, people using ambient heat as allowing the ambient temperature. And one other is beginning to be implemented in some places. We can, in principle, reduce the overhead to less than the power consumption machine. In principle, it's possible to get your overall efficiency down to about, well, you're spending maybe 0.1 of a watt on cooling and everything. If you can then use the residual waste heat from the machines to heat something that you want to be warm, then you can get your overall data center efficiency to the point where you're actually getting energy out of the energy that you're putting into the computers. Again, that's not really something that's common at the moment. We're probably going to have to see increased infrastructure for sharing heat in that way. But in the long run, probably not even the long run, in the next few years, we're probably going to see data centers that are capable of heating the offices that they're attached to. It's kind of unfortunate that generally you don't want to heat offices in summer, when you want to call the data center the most, how, you know, just pipe all that air from Australia to Boston right now. That would suit me fine. Even if we assume a very efficient data center, we still have trouble in terms of the machines taking lost power themselves. A 48 core system is currently not really a representative piece of hardware for the majority of people, even in the server market. But Intel are now providing platforms that will give you 48 cores with just four packages on a single board. It's not going to be too long until we see people running. But using these as a virtualization host or using them for high-end web servers, idle one of these machines takes something like 350 watts. Kind of a lot of power. A rack of these you're looking at several kilowatts. A row of racks you're potentially looking at somewhere in the order of half a megawatt just for powering those machines. Energy density is getting higher, and this is inevitably going to result in this not being getting bigger. A power machine where we're now able to draw more power when idle than we have been in the past, even allowing the fact that we've improved power management. Software, we have primarily been looking at CPU power management. And on current systems, the processor is probably biggest dynamic power draw in terms of the difference between idle and load. At idle, a modern X86 will consume literally something like zero watts. It's terrifying how efficient these are now. Intel has taken their mobile technology and applied it to server markets primarily so that they can get this number as low as possible. When idle, these chips will unclock themselves. They will power down the cache. They will power down the memory controller. They'll put the memory into self-refresh, and they will run at something like 1 volts. There is an incredible amount of dynamic power saving involved. Of course, idle isn't necessarily the case that people talk to my sport, but I'll guess that later. If we have the core full power, that's more like 20 watts. So 48 core system, per package, that's then 12 cores. You can look at getting on for 200 watts or so of power dissipation through a single CPU package. If you then take all the cores in the system and run them all at full load, then you're potentially looking at almost a kilowatt just from the CPUs. That's a large number. That's basically an electric space heater. I suppose you can argue that the computer may be doing something useful, but chances are the computer is only producing pictures of laughing cats, whereas the space heater does actually keep you warm. It would like that number to be lower. Really, we're talking about processes that are designed to do work. If you're running them at full load, that implies that you do have a workflow that requires that. So we're not going to be able to instantaneously change the laws of physics in order to reduce that power consumption. And really, our transistor technology doesn't allow us to reduce the power consumption at full load much more than we currently have. Really, what we have to focus on is getting the processors to be idle more of a time. As I said, an idle processor consumes about zero watts. But as I also said, modern processors have the memory controller built onto the CPU package. So your memory rather than going deep next to a Northbridge is connected to the CPU directly. On most modern processors, if every core on the package is idle, the package goes into its own sleep state. And when the package goes into deep sleep state, the package idles the memory controller. And at that point, you stop running the bus between the memory and the CPU. An idle memory bus is still clocked at about... See, it's DDR2 for... It's somewhere on the order of 600 to 800 megahertz, I think, for an average DDR2 and DDR3 system. That clock is still running even if you're not putting any bits over the bus because you have to keep refreshing the memory. Otherwise, if you don't refresh the memory, the DRAM goes into a state where the bits just kind of fall out, which is not conducive to continuing to run your software afterwards. DRAM itself is completely capable of handling that itself. You can tell the memory to go into a self-refresh state where it will look after refreshing itself just often enough to keep the bits where they're meant to be. So memory in self-refresh is great because you're not running this high-speed clock and also you're refreshing the memory less often. And that can take you down from something like 5 watts per stick to not being 1 watts. And 5 watts doesn't sound like a great win. When we're talking about large servers, we're talking about systems that potentially have over 26 of memory. We're looking, in that case, at a difference of overall somewhere in the region of 100 watts just by being able to get the memory idle. So that's great. On the other hand, it's kind of difficult to write software that never uses memory. It's also writing software that does nothing. But writing software that does nothing is very easy. It's difficult to write useful software that does nothing. That's not something that people have necessarily thought a great deal about. Of course, with these multi-package systems where you have the memory attached to specific packages, if you're touching memory that is attached to a package other than the one that you're executing codes on, then you'll already take performance hits because you happen to speak to that CPU which has to speak to the memory and then start that comeback. The question now was, if the memory doesn't have to be refreshed at the full clock rate in order to retain its contents, then can we produce the clock itself? The answer to that is, if the clock is full, it is taking power. You've got these lines that are then having to go low and high at a clock rate. It's preferable to be able to put the memory to self-refresh. The only downside to self-refresh is that there is latency in loading the memory back out of self-refresh. But there was also latency involved in adjusting the clock. So it's not obviously a win to run the memory at a lower clock just in order to avoid the refresh stuff. But that's actually something that I'm going to get onto in a bit. We would ideally like programmers to an algorithm where, for the most part, you hit main memory in bursts, get your cache populated, and then work out the CPU cache because that way we can keep the main memory idle as much as possible. If you're working sets in cache, your code's going to go faster and you're going to touch main memory less. So that's a win for power, it's a win for performance. And in a lot of cases it turns out that optimizing your code for performance is also optimizing for power consumption. What we're getting now from vendors is a decreasing amount of interest in the process of power management because for the most part we've solved that problem. Linux is very good at taking advantage of CPU low power states. We're actually much better than any other operating system at getting x86 into runtime low power states. That's huge. Like I said, it's the difference between 200W per package and 0W per package in a worst case scenario. So we can now, on an idle system, see deep C state residents of approximately 90%. So for 90% of the time in the system's idle, the process can seem 0W. And for the other 10% we're generally in a low power state at the top twice as low so we're talking about maybe 5W. So right now vendors can't differentiate themselves based on CPU power management anymore. So most of our work with vendors is down to vendors trying to come up with ways that they can advertise how their machines work much better with Red Hat than anybody else's machines, that kind of thing. We're starting to see vendors become significantly more interested in memory power management. And like I said, ideally we would have the memory be idle all the time or as much of the time as possible. But there are times when you access memory in a thirsty manner and the performance hit of going in and out of self-refresh may be too great to justify using it. In that case, as long as latency is not too much a problem and as long as you're transferring small amounts of memory then running the memory at a lower clock rate may actually be a viable alternative. So as I said, we're seeing people look into that. We're also seeing people become increasingly interested in using virtualization as a means for power management. And of course, like I said, the CPU has a lot of power. The CPU is the thing we've generally been thinking of in terms of power management. Memory is an obvious other big hit, but looking at the smaller system components each of which may only take a couple of watts that's still something that over the size of an entire data set adds up and that's still something that then is requested. So how does virtualization potentially result in power savings? And the three-fourth thing is generally speaking, computers spend a lot of time doing nothing and while it's great that we can get a machine down to 350 watts when doing nothing, when it would take a kilowatt to do something ideally we'd like to get that down to zero. And turning a computer off is an easy way of getting the power down to zero. The problem is that people generally don't like it when their web server stops working because the machine went idle and turned itself off. Virtualization is a straightforward response to this. If you assume that you have hardware to satisfy your worst case scenario in terms of load, if each of those machines is running a virtual guest rather than running on a bare metal you can migrate those guests to a smaller number of physical systems and suspend or power down the hosts that no longer run the guest. So that's great. We can turn a computer off or suspend it and this power consumption goes down to approximately zero which is quite a bit less than 350. In fact it's 350 less than 350. So we like this approach. The downside is obviously you need to be able to migrate those guests rapidly. You don't want there to be significant downtime while the guest is being moved from one host to another host. There's still work to do on that front in terms of making this particularly practical in a turnkey environment. Right now you'd really need some infrastructure work to your server software calls to make this work well. But customers are interested in this. It's very likely that we will see these sensors where the majority of machines spend most of their time switched off and then light back up for peak demand periods. So, so far it's been this nice story about how we have all these wonderful power management technologies where the operating system looks after you. The devices go to sleep when you're not using them and they come back when you are using them. And so everything's wonderful because you don't need to care about power management. It just works for you. It turns out that there are some cases where this kind of free power management where there's no drawback to using the power management have drawbacks. Enterprise customers have enterprise demands. If you want then you could probably think about this by replacing the word enterprise with completely insane. There are a lot of people running servers who care about every single bit of performance and are happy to burn as many polar bears as necessary to power their systems. So an example of this. If you want to know the time accurately and fast then the easiest way to do this on XA6 is to read the time stamp counter because that's a single instruction. You call readTSC and then suddenly one of your registers contains an integer that runs at a constant clock rate. So if you've calibrated yourself in advance you know how much real time has passed by taking one number subtracting another number from it and then devising appropriately. The problem with the time stamp counter is that traditionally XA6 is guaranteed to be a monotonically increasing. It is not guaranteed to always run at the same speed. In fact it's not guaranteed to run at all. What we saw on older chips was what we called a P-state variant. That is if your processor ran at 2 GHz and then you used speed steps to slow it down to 1 GHz it would now run half as fast as it did previously. So at that point if you want to be able to reliably calculate time you need to also know what speed the processor is running at for the entirety of the time between the two time points. You can't actually do that without a huge amount of work which can defeat the point of the power management to begin with. Since about 2008 that hasn't been a problem. The majority of chips since then have had P-state invariance that is it always runs at the same clock no matter what speed the CPU is running at. Wonderful. This turned out to not be particularly useful. As far as power management goes changing the P-state that is the speed the processor runs at isn't a big win because if you can get into deep C-states the chip unclocks itself completely and again that whole zero is better than another larger than zero thing comes into play. If you went into a deep C-state the time stamp count would stop completely. We can still get away with calling it monotonic because it would never go backwards and because if you executed 3 TSCs twice then the chip would have to be awakened both of those so it was increased by at least one tick. But in principle if the system worked completely idle you could wait five years and discover that the TSCs increased by one. This again didn't make these enterprise customers particularly happy. As of about a year ago the majority of chips now have time stamp counters that run at the same speed regardless of what C-state you're in. Is Jeff a question? Right. Migrating VM guests if you use 3 TSC in a virtualized TSC I'm sorry. TSC goes through to the host CPU rather than emulating a TSC if you migrate between two hosts that don't have synchronized TSCs and of course they don't have synchronized TSCs because that would be insane. It's difficult enough to synchronize TSCs when they're both on the same motherboard where you've got a chunk of Ethernet between them. This just isn't happening. I do not want to think about synchronized Ethernet. But anyway, right, so if you migrate from one to another and then it's entirely possible the time stamp counter will go backwards. The alternative involves when you migrate something over you suddenly bump the TSC by some level and then suddenly anything that was running on TSC for calibration purposes thinks that five years have passed and again that's not good. This is a difficult problem I'm sure will come up with some kind of solution for it or rather I'm sure that somebody more intelligent than me who's paid to care about this kind of thing more than I am will come up with a solution for it and with luck it will be elegant and beautiful. Anyway, so as long as you're running on bare metal now you can rely on the TSC even with C-states enabled which is wonderful because we no longer have to tell customers that if they want the TSC to work they have to disable C-states. Latency, again kind of a problem. When we're changing C-states you're powering down chunks of hardware that are clocked. When you're powering them back up you're going to have to do things like wait for voltage to stabilize you're going to have to wait for clocks to re-synchronize you don't want to end up at a point where you're running one core at a speed that's slightly different from another core where they're both on the same speed and you really, really want everything to be completely locked in phase from C-6 or C-7, the D to C-states or mod processors back to C-0 can take on the order of a millisecond which sounds like a completely insignificant amount of time but so say you're a large company on Wall Street and say your entire competitive advantage consists of you being able to make trades in response to market conditions half a millisecond before somebody else does then suddenly a millisecond sounds like a big number. Obviously I can't comment on who our customers are or what sort of requirements they have or why they have these requirements but let's just say that latency is a concern for some of them and even a millisecond of added latency in terms of going from not executing instructions to executing instructions can be too much. So in terms of power management the three-fold solution for them has always been just to disable C-states which is a shame because that then takes you from 350 what's on one of these big systems up to about 600 even for an idle system. We have some misogation techniques of this we have something called PFQOS which allows user space to indicate to the kernel what kind of quality of service it requires you can open a device node dev CPU latency QOS or something like that and it will then you write the maximum latency you can tolerate and then the system will not enter any C-states that have a higher latency than this. If you know when you require low latency and if you know when you can get away with more latency this is fine. It lets you say I'm in a low latency mode at the moment and it lets you say I still need to be awake I may still need to respond to incoming events but I'm not concerned about high latency in responding to it. This is possibly the best we can really do at the moment but it's not straightforward for software to predict the future so it's not possible for us to switch to low latency just before the event comes in because obviously that event hasn't arrived yet. Further, when we talk about C-states in terms of performance then it's obvious that C-states cost you performance because they increase latency. This turns out not to always be true but it's very workload dependent. Intel processors and AMD processors now both have the ability to overclock themselves dynamically so you say run at your state speed but also enable turbo mode and then if a core is loaded and if doing so will not reach thermal end flow will not reach the power end flow That core can run at in some cases up to about 500 MHz faster than its rated speed In order to be able to do that you need the CPU to otherwise be running in a power efficient mode you need every other core to be running in a low power state So by having good idle power management by getting all your other cores down to C-states then the single core or in some cases the two cores that are running can run much faster than they would otherwise run So if you have a multi-threaded machine but most of your workloads are single-threaded you want good power management because doing so means that your system will run faster Another side effect of this is that if you're benchmarking these machines you get much better results if you put the machine out in the snow because if it's colder the processor runs faster I don't think we started using this for our benchmarking yet Right now it's really cold in Boston it's not obviously a bad idea Conflict with the air conditioning Right, so what I was saying earlier was that we could run data sensors at a higher ambient temperature Yes, there's a conflict because some people will want their machines to be colder so they'll get better performance out of them Those people should probably just buy faster machines to stock Yeah, the difference in cost to the air con would probably pay for a better processor Instead we can turn more devices off Right now we have mostly been thinking about the CPU and to some extent the memory when it comes to power management There's a lot of hardware on a typical machine So even a server will usually have a bewildering array of USB ports Those USB ports are attached to USB hub controllers and those host controllers because USB was designed by braven monkeys rather than anything useful Up until USB 1 and USB 2 it's effectively impossible to build a host controller that doesn't wake up the processor several times a second even if there's nothing plugged in This is less than ideal USB 3 I believe is better at this but only if you plug USB 3 devices into it if you have a USB 2 device plugged in you still have some overhead This is especially offensive on servers because typically there's nothing plugged into the USB ports So you have this chip that is waking up the processor 10 times a second for sockets that in some cases will even have been popsied in to prevent people from sticking USB keys in and copying data off the servers My viewpoint is that this is kind of pointless In fact it turns out that the designers of the hardware also thought this was kind of pointless USB controllers all have the ability to indicate a wake-up event if a device is unplugged if a device is plugged in or if an external USB device generates a wake-up event If we put the host controller into a D3 suspended state then it can tell us when something happens and while it's in the suspended state it doesn't wake us up That's great, it turns out that saving this saves us about half a watt per machine so it's not awesome but it's worthwhile What does the host controller tell the operating system that this has happened? And the answer to this is actually partly in the USB specification partly in the ACPI specification partly in the chipset specification partly written down in documents that were placed in a filing cabinet that was then launched into space aimed at the sun and everybody involved was killed before they could divulge the information So PCI devices all have PCI I think is 2.1 or 2.2 and all PCIe have something called PME the power management event On PCI there is a separate physical line that represents PME The idea here is that you hook the device into a suspended state you enable PME generation and then so in the USB case you plug something into the port the chip has enough it raises the PME line then nothing happens because it's not respect how the PCI PME line interacts with the operating system PCs this is typically handled by the PME line being all joined up to a pin on the south bridge and the south bridge then generates an ACPI general purpose event Implementing the code for this in Linux turned out to I started working on this two and a half years ago it mostly works now we found a bunch of cases where the code in Linux was just not written to deal with this case at all we found a huge number of bugs everything's pretty much sorted out now in general this works you plug a device in PME goes high the chip generates a general purpose event a notification routine is run or alternatively we work out the mapping between a general purpose event and a physical hardware device by magic it's not actually magic but it involves too much ACPI because it really wants to tell you about it now that's awesome this is actually now enabled in Fedora raw height if anybody wants to test it's currently kind of broken on some AMD chipsets and it's completely broken on the IH chipsets and it seems to be broken on one Intel chipset but I'm looking at that one for PCIe you've got two choices rather than there being a separate line for PCIe there's a PCIe special packet that goes across the bus and then either that will be intercepted by the chipset and it will give us an ACPIGP again or alternatively it generates an interrupt on the root port and then we receive that interrupt in the interrupt handle that we then scan downstream devices kind of works most of the time there's still some cases where it doesn't it seems that certain chipsets don't work if you try to deliver a PME over MSI because who would ever do a thing like that MSI message signal interrupt is a more efficient way of passing interrupts from PCIe devices to the operating system in terms of how reliable the hardware is there's a lot of variance the machine I have here is an HP 2530P the previous one I have was 2510 the difference between them is pretty small the difference is 20 but ignoring that the difference between them is relatively small they're pretty much identical except the fact that 2510 had a 965 chipset this has a GM45 chipset so you would think that everything would be wired up equivalently it turns out that Windows never uses PCIe PME on the 2530 they decided not to bother wiring up the PME line between the Firewire and your digital controller and the chipset so you put a card in and the chipset oh I should generate a wake up event and it waggles this little line and it's doing this and no to compare it because there's nothing connecting that pin to anything else so what we do now is if it's a device that isn't past the chipset and it is PCIe we poll it once a second on the assumption that you're going to have at least one wake up second anyway and so this doesn't really add too much to your power consumption and you're probably going to save more than it costs if you can poll these twice as down. We only do this if the device is suspended and we actually have support for if we do ever see a wake up event we will stop polling it's the least bad thing we could come up with. Other things the AMD USB controllers as I mentioned if you plug in a USB device it generates a wake up event we rescan it and then the host says that there has been no status change so we go back to sleep this makes people unhappy turns out that they do set the flag if you read the individual port status registers rather than the overall host status register so now we just do some more register reads so that's also okay this is mostly user transparent obviously there's a small amount of latency but really if you care about the extra and of latency introduced in putting a USB device in your machine that's probably insignificant compared to the length of time you spent trying to push it upside down before you realise you're trying to push it upside down so we can get away with that that's not too much of a problem USB this works pretty well there is code now for SD slots that should pretty much work if there's no cards in there as they're powered down but there's no traffic to it we can power it down this isn't entirely good stuff yet but that should be landing soon firewire we can do the same if there's nothing plugged in we can suspend the firewire controller completely I need to rebase that code so it doesn't crash the system when we suspend it but that's a simple matter of coding audio in principle we can power down the right now we already have support for powering down the audio codec when there's nothing playing we could also potentially even further in all your concerns about this when there's not something plugged a pair of headphones in we can then get a wake up event from that that's probably less useful, there's not so much wind for that but it may be worth it in terms of less enterprise things the other big hit is GPUs, thankfully people very rarely put NVIDIA GPUs on server boards because for the most part we're not very good at GPU power management yet Intel we have support for something called IPS on R&L systems that's Intel Intelligent Power Sharing the idea here is that you've got the the GPU and CPU are both on the same package so when I was talking about turbo mode earlier you've got a power budget for the package so if the GPU is idle then you can downclock it and that actually means that your CPU can run faster so merging this driver should have meant that you could be faster on x86 than previously it turns out it doesn't work very well for a bunch of reasons, like if you ever manage to let the GPU go idle for even a millisecond it clocks down the GPU and then the CPU is waiting for the GPU so you never actually get the GPU busy enough for it to clock up again apparently that's the fix last week Sandy Bridge has something similar except you don't have to have a separate drive for it so woo Intel Radeon we have some support for automatically changing the power state of the GPU it doesn't work so well in the right now we tend to we have to try to do this during the vertical blank interval because otherwise we reclock your memory while you're trying to scan out the memory and that's really bad but we do that if we hit the V blank then we clock up quickly we may miss the V blank several times that's kind of a problem also the implementation is relatively naive if more than three commands are currently in the queue waiting to be executed then we suspect you probably need to be running faster this is most obvious if you run, say, Unreal Tournament in quick arena in benchmark mode everything starts very slowly and then gradually gets faster and faster and faster it's not ideal so there is work being done on improving that new though we have support for changing the power management state on some chips right now you have to do that statically I don't have any hot demand switching so how are we doing? and the answer is we're doing pretty well GPUs are pretty much the only case where we have worse power management than Windows at this point which is way better than we were before in a lot of cases like USB, SD we're doing way better than Windows they don't even implement the lossless functionality which is why vendors do things like don't attach wire one to pin one because that way we can save the cost of one strand of copper we have fewer wake ups Windows currently does not have ticklers kernel and idle Windows system will still wake up at least 60 times a second if you're running Outlook it will wake up 60 times a second because for some reason Outlook requires high resolution timers we're better at c-state management on Intel systems right now we can ignore what the BIOS says about c-state support and we can use every c-state that the CPU supports even if the BIOS vendors didn't bother adding support for that Windows can't do that so we're better at using c-states we're better at getting into c-states anyway so we're awesome you should all run out of Linux and not Windows in terms of future work there's some further tweaking of CPU Freq it would be nice to if you're using CPU Freq then we still have cases where we don't ramp up as quickly as we should do that's performance here that makes people unhappy we could do better than that to go to the tweets it would be awesome if we were better at working out whether we would be better off consolidating tasks on a single package or stepping across possible packages running on one package means the other package can go to sleep that's awesome and drivers support we still need to carry on adding support for power management features of drivers there's a lot of hardware to function on so that we're not using chances are nobody else's either which means chances are it doesn't work because the one thing I've learned and one thing I will tell you is the secrets of hardware enablement hardware vendors lie about everything the majority of data sheets you get if they say we have this wonderful feature and you think that's a wonderful feature nobody else is doing that yes if we do that we can be awesome the reason nobody else is doing this is because it doesn't work anyway that's everything I want to talk about I guess we may have time for one or two quick questions so anyway thank you