 I think so, yes. OK, good afternoon, everyone. My name is Hans de Gouda, or Hans de Goet in English, because people have trouble pronouncing my last name. These are the topics I want to talk to you about today. First a slide spoiler, just a quick summary. Then answer to the question, why not just use Powertop or TLP, which is a bunch of shell scripts to sort of do the same thing I'm doing. And then a bunch of the features I've been working on, four of them, which is not that much, actually. So for starters, what's this all about? It's mostly not about anything really spectacular or new. A lot of people who are already played with Powertop, with the tunables in Powertop, may know most of this. It's more about the out-of-the-box experience for laptop users. So let's say we have a beginning Linux user. They install Fedora or some other distro. And then usually, the power consumption story is not all that good. We have a number of tunables in the kernel, which are disabled by default, because they cause problems on some devices. And if you actually enable those, you can usually save, well, in this particular example, the laptop I've been doing most testing with around 2 watts. On 8 watts. This is all idle usage, right? If you start compiling a kernel or playing a video or doing a browser with 60 tops open, then there's not much which can save your battery life. It's just going to drain. This is about when you're just doing some relaxing stuff behind your laptop, maybe typing something in VI or whatever. So as I said before, this is the spoiler part. If you're a power user, you're a more experienced user, you know, Powertop-autotune, and you're already using it or using TLP, you already have most of these benefits. I might still be able to give you an extra 0.5 watts of power saving on idle again, but not as much. So, well, if power users already... One question, how does this number compare to Windows? Do you know this? I don't know. I have only been looking at Linux power consumption. So if power users already get these advantages anyway, why do the work? Well, at least the way I see things, I want Linux to be usable for everyone, right? We want to displace Windows still, or nowadays it's, I guess, more about displacing Mac OS X on laptops. So the out-of-the-box experience matters a lot. And this is something not necessarily draining the battery as something which should just work for all Linux users. Also, there is an actual reason these tunables are turned off by default. One of the big wins is the satana link power management stuff, I'm going to talk about it a bit more lately. I met you, Garrett, already found that about three years ago that saves about 1 to 1.5 watts, and then I'm being conservative. I've also seen machines where it saves two watts, which is huge, but it has the small downside that on some machines it corrupts all the data on your disk. So, yeah, which is also a reason why PowerTop, Desktop Auto Tune, or TLP are not necessarily good. A PowerTop is also well known for making USB keyboards and mice not work. TLP avoids that by recognizing it's a keyboard or a mouse and not enabling USB auto-suspend. So the first thing I did is enable auto-suspend for USB Bluetooth controllers. USB auto-suspend is interesting. If you play with USB auto-suspend and you make one of your devices auto-suspend and you look at your power consumption, almost nothing changes. It's within the noise level of the measurement of the power management I see in your battery, which is the only way we have to somewhat accurately measure power usage in a laptop. But it becomes interesting if you manage to cause the last device to auto-suspend because then the controller also auto-suspends. And then you actually get a win of about 0.4 watts on a household generation hardware. Might be slightly less on newer hardware because slightly more energy efficient, but we get a win. On a lot of laptops, the only device which doesn't auto-suspend is actually the Bluetooth controller. I mean, your average laptop has a webcam. The webcam driver already does the right thing. It has a fingerprint reader. If your fingerprint is supported by LibF print, there are Udev rules which calls it to do the right thing if you have LibF print installed. And then there's the Bluetooth controller or the Bluetooth host interface. So if we enable it on the last part on the Bluetooth host interface, we get this 0.4 watts saving. So I wrote a kernel patch to actually allow distros to flip a bit in the kernel config and enable auto-suspend by default. After I did that, it turned out it broke on a bunch of real tech and a TIROS devices. That's actually was already known. There already was a quirk in the driver to do some special handling on normal suspend resume. I just needed to rip that quirk out and replace it with the standard quirk mechanism which is shared by all USB devices which also disables runtime suspend on these devices then. So I actually removed code to fix the auto-sum real tech devices. And the next thing is a slightly more tricky. Another part which consumes power is your audio chip or the audio part of your chipset and the separate analog audio codec which contains the headphone amplifiers and speaker amplifiers and stuff. That actually already has a Kconfig option to enable it. I've been talking to some of the upstream audio people and Susie already has this set. So this is only a win in Fedora and maybe other distros. So what I've been doing is I've been turning this on again and receiving bug reports. Unfortunately, the analog part of this is really complicated. There are many, many ways, different ways to hook up an external speaker amplifier and stuff. And if you don't do the power sequence, power up and down sequence in exactly the right order, you get a nasty plopping or hissing noise. So I created a blacklist for now. Every single entry in that blacklist is a bug in the Linux kernel. But we don't have enough people to chase these down and fix these. So there is, I don't know if you've ever looked at the hardware database part of system D or HWDB. Every file in there which has a set of rules has a header which explains how to submit new rules. I put a rule in there that upstream system D will only accept new entries if they have a kernel bugzilla. So you must file a bug before your blacklist, your specific laptop for this so that we can hopefully fix it one day. But for now, the only reasonable way to work with this is unfortunately a blacklist. Unlike the Bluetooth stuff where I could just blacklist two vendors and then be done with it. This is different, unfortunately. I will submit the UDF and HWDB stuff upstream soon-ish. I was actually hoping to have done that already, but so this is the interesting one. This is the big one, actually. This is also the one which I expected to cause my mailbox to explode. And so far it hasn't, which is somewhat nice and somewhat scary. So as I was saying in the intro, without advanced link power management, the CPU package, if you look at a modern CPU, it's actually more of a system on chip. It has a lot of peripherals inside. So the CPU package as they call it uncore, anything which is not the actual CPU cores itself cannot reach lower power states or deeper sleep states. So the PC number, the higher the C state, the better the power savings and the higher the latency to get back from it. So without SATA power management activated on a machine which has SATA devices, you cannot get higher than PC2 and it goes up to PC9. Although I've never seen higher than PC7, actually. Well, as soon as you enable this, you get like 50, 60% of the time PC7. Which gives this nice one 1.5 watt saving. So the existing setting, before I started working on this, min power, enabled three things or more even, but three I know from the top of my head, you have depth slumber, which is a special mode which puts the disk in an even deeper sleep state. It's not really that interesting. It saves about 70 milliwatts. You also have the link power management and the link power management can be device initiated or host initiated. And Matthew Garrett, when he was working on this three years ago, he found a document from Intel which describes their default setting for their IRST, Intel Rapid Storage Technology, Windows Drivers. And they only enable DIPM, or Device Initiated Power Management. The problem was that the Linux min power setting was enabling HIPM, host initiated, which probably meant that the power saving kicked in in a moment which was not really useful for the firmware of the SSD. And the firmware responded badly to that and just hang or whatever. So, yeah, that was a problem. So I added a new power level which unfortunately, that Linux designed for this, the kernel Sisyphus interface has chosen to use levels which are described by a string. While there are multiple bits you can toggle, so it would have been better maybe if you had just a separate Sisyphus attribute for each bit. But I introduced a new level called medium power with DIPM, which gives us just enough Device Power Management to power down the link and get this huge shape. So that's really nice. We may even also need like a min power without HIPM. So we could also try Dev Slumber for the last 70 milliwatts. But yeah, 70 milliwatts compared to 1.1.5 watts is not really a big win. So I started with the low hanging fruit. This has actually all gone upstream. I was amazed when they took the patch adding the new power level. I was even more amazed when they also upstream accepted the patch where you can, in Kconfig, set a different default level for mobile chipsets. If you look, sorry AMD users, this is all Intel. Maybe someday we can also look at the AMD side of this. If you look at Intel, how they do PCI-IDs for their SATA controllers. They have different PCI-IDs for their server hardware, their desktop hardware, and their mobile chipset hardware. So I have now a new kernel contact option which allows you to enable matte power with DIPM by default for mobile chipsets only so you don't get the latency penalty on servers. Actually, if there are any people here working for say Google or Amazon, I hope you're already using this because if you multiply the saving, probably maybe some of it per disk for a data center, it will also definitely be interesting. So that's something else. Asset, it has been enabled in raw height. Kernel patches are upstream. And since January 1st, and so far no one has been yelling at me that I ruined their data, so fingers crossed. This was actually the one which I was surprising to be the most troublesome but it looks like the audio codec thingy is slightly harder. So given the sort of relative success of the above tree, I decided to fill my plate further with a four thing which is panel self-refresh. If most modern laptops have a display port panel, an EDP embedded display port panel, and that has a feature where you no longer need to send the data to the display each frame. An LCD panel really is a lot like DRAM. You need to refresh each pixel. And if you don't, the pixel becomes transparent so it becomes white. You see the backlight. So you cannot stop refreshing the screen even if the content doesn't change. But some screens have enough memory on board that they can go into, they can buffer an entire frame and they can go into self-refresh mode. And that actually saves a nice chunk of power that saves around another half watt of power in my measurements. But this is a total clusterfuck. Sorry, but. It looks nice. Saves half a watt. It defaults to off because it's known to cause issues on my first version of the slide set some devices. This version of the slide says a lot of devices. Before Fastem last Thursday, I think I wrote a blog post about this and said like, look, people, I need you to test and see how well this works on different models. I was hoping we could maybe enable it by default and have a blacklist with 10, 20 entries for some bad hardware. That's not going to happen. Most of the hardware out there is bad when it comes to panel self-refresh. There's a whole range of different issues. All very good questions and honestly, I don't know, I don't know, I don't know. I started examining this last Thursday. Sort of, I had it online to-do list and since the rest was going well, I added this, like, let's throw this in the mix before Fastem. Yeah, so after the blog post, I got about 100 reports. My mailbox totally exploded. Really weird, I did a similar blog post for CEDA and I had, like, a big disclaimer. Please test this for me, by the way, it may eat your data. And I got, like, 20 responses. And then I did this blog post and I got a lot. So to go into the question which was asked by Thorsten, one of the users pointed out to me that he recently got a BIOS update from Lenovo which actually sets a fixed flickering panel by disabling panel self-refresh. So, I mean, vendors are issuing BIOS fixes which increase power consumption and disabling this. So it's just, it's, the only way I see to reasonably deal with this because it does work on some devices and it is a work while power saving, while it works. And again, going back to my earlier slides, I want stuff to work out of the box for the end users is to do a whitelist. The plan for that whitelist is to match both on laptop model and panel ID because some laptop models have different options, right? Low-rescreen and high-rescreen. Some models even use different panels in different production batches. So we definitely need to also check the panel. And if the panel and laptop are a known working combination we can try to enable this out of the box. Don't worry, kernel developers. I'm going to put this in HWDB again and I'll put it in user space because I know you don't want these quirks in the kernel. Yeah, so that is pretty much the story on panel self-refresh, which is not pretty unfortunately but hopefully we can, if you look around, also if you look at the hundred reports I get most Linux users who take the trouble to actually file a report for this are either using the Lenovo laptop or Dell XPS or maybe a Dell Latitude. Yeah, go ahead. Flickering. Flickering often combined with a way higher power consumption like one to much higher because it goes into and out of self-refresh all the time and it actually consumes power going in and out of it all the time. Huge input lag. So it probably goes into panel self-refresh but when the contents actually changes it doesn't come out in time. That might actually be a driver issue especially the one, the people who are reporting mouse cursor lag. I guess that when only the hardware mouse cursor has updated then the driver isn't smart enough to come out of panel self-refresh. So that might be one for the Intel folks to look at. Okay. It has to redo the link training again. The link training and that's causing the flicker issues but yeah, we're working on it. Ouch. Yeah. Okay. So yeah, long story short, this is complicated and being worked on. So I promised it would be boring. I hope it wasn't really boring but at least I tried to keep it short so we would have plenty of time for questions. So questions anyone? Now Torsten, oh, you're welcome. Any further questions? 416. Some of them. The medium power with Dippem level is supported in 415. The Kconfek option to enable that by default on mobile chipsets is in 416 or will be in 416 when the pull request gets merged. Now this is all hot from the press basically. Any more questions? Sounds like a kernel bug. Or an SELinux bug or something like that or app or armor or that's weird. Any more questions? Yeah. No, not really, but Christian over there is trying to make user space better or was trying before he got dragged into Thunderbolt. So once you have all this figured out, I mean one thing we could look into is PCI ASPM which is powering down the PCI links which is pretty much the same story as the SATA link, right? You need to have all those links powered down for the encore to go in a higher power saving level. But that seems to be working pretty well recently, ASPM stuff, might be worth revisiting. But no, not directly actually. This was sort of a side tour for me. We'll see what the future, what I did realize which is interesting in a way is when I was going to FOSTA, I usually make a list in my head which people do I want to talk to and I wanted to talk to a colleague about a certain customer and I wanted to, don't remember, to talk to someone else about something. I was thinking what am I working, oh I wanted to talk to Michael Thayer from VirtualBox about VirtualBox because I've been working on VirtualBox support, at least the guest side of VirtualBox support. And then I was thinking and I'm currently working on power management, who can I talk to about power management? And I drew a blank. And not just because the person who I could talk to isn't here at FOSTA, there doesn't currently seem to be anyone in the Linux community on the kernel side who sort of is responsible for power management and taking the lead there. And I think we have a whole there as a community. We need someone who's sort of becoming the go-to person for power management issues and having the overview and doing coordination and things like that. And no, I'm not volunteering. Thanks for the delay. Could this be applied to Sandy Bridge? Yes, actually it's interesting. The question for the people in the street. Sure, sure. So the question was, is this work bound to certain platforms or CPUs? Or certain generations of CPUs? Well it is bound currently mostly to Intel CPUs. NX86 in general, but theoretically the CEDA power management stuff is only interesting for Haswell and newer because with Sandy Bridge the CEDA controller is actually in a separate chip set, so in a PCH. But testing shows actually that it's still useful because it allows the PCH to power down. So I'm also seeing the same savings on Sandy Bridge in my own testing. Although theoretically it doesn't help the CPU to save power if the separate chip set on the motherboard goes down, it also helps. And the hard disk itself goes in a lower order, the SSD goes into a lower. So yeah, at least for Sandy Bridge I can say it also applies there, it helps. The CEDA stuff, the HDA codec stuff goes back even further, that it should help a bit. Same for the USB auto-suspense stuff. So it varies a bit feature by feature. For example, panel self-refresh is certainly not available on Sandy Bridge. Those usually still use LVDS panels. A panel self-refresh also doesn't apply to most Bay Trill, Cherry Trill and Apollo Lake devices. Those could in theory be using EDP, but in practice most of them are using DSI because they're using cheap panels, which is a different interface. So yeah, it varies a bit. And it would definitely be interesting now that AMD is getting back into the game of doing mobile chip sets, right? Hopefully their next APIs will be decent-ish and we'll have some competition again to also look at the AMD side of things. Really, really primitive. Unplug the machine so it's running on battery, run PowerTop and look at what discharge rate PowerTop is reporting. I had a discussion before about Wi-Fi and power management with someone here at FOSDEM. The issue is that currently, especially the last three or four months, Intel has been pushing some Wi-Fi firmware updates and we're seeing on a lot of client machines that Wi-Fi is becoming unstable because of firmware issues, so I don't want to start playing around with Wi-Fi power management until at least in its current state it's like rock solid or rock solid-ish. We're already seeing issues now and I throw another factor into the mix. It's not going to help. I mean, causing wake-ups. Or users don't have a browser open with 60 tops. It would be one, I guess. Maybe even the biggest one. I mean, we shouldn't be blaming this on users. Again, stuff should work out of the box. The problem with a browser with 60 tops is that there is a bunch of JavaScript that somehow wants to pull the server once a second or whatever. I know Grom is already getting better at delaying these timers to maybe once a minute if it's a tab which is not visible. I'm not sure what Firefox is doing. Browser vendors could do a lot of stuff here to say, like, this top isn't visible or the user hasn't looked at it for the last minute. Let's just freeze the top until the user clicks on it again. Then you're only using a bunch of memory instead of also eating your CPU with the tops. I guess one which is probably obvious to most in this room is if you want to save power, install an ad blocker. Anything else, Christian? Yeah. Well, one win, actually, which we still have to do, which is tricky because software patents is we really need to get better in Linux, both at the browser level but also as distributions on supporting hardware-accelerated video decoding. Because that would also, if you were watching a movie on Netflix and you could hardware-accelerate all the decoding, that would save a ton of power. They support it under Windows but not under Linux. Yes. But I think the question was, you showed, like, six watts are still used on your system even if you only have a terminal running and you are talking to them. So what are the six watts? You have the display and 50% of brightness? Yeah, I have the display at 50%. In my experience with some other machine where I did some more testing, which is an Apollo Lake thingy, which actually goes down to, like, three watts or two watts even, is that display is a big part of it, because this was with panel self-refreshed already. But am I running out of time? Yes. Okay, well, then this will be the last question. So one thing which you can do which helps if you're behind your laptop and you're not going to use it for five minutes is lock your screen or do whatever causes the screen to go blank. And actually into DPMS, that saves a lot. And yeah, that's the most obvious one, really. And where the rest is going, I haven't been doing any advanced power management. My boss doesn't want me to open up a laptop and start cutting PCB traces and adding resistors so I can measure current and stuff like that. Apparently, the laptops are too expensive for that or something. Thank you very much. You're welcome.