 Everyone who successfully arrived at the conference did so thanks to k itinerary, which is not the topic of this talk So we're going to talk about energy consumption instead Fulker So yes Following a bit on the previous session How can we actually measure the energy consumption of the stuff we built To get started why do we care about this? Yes, multiple reasons There is for example The battery use on battery power devices so the mobile case that we discussed earlier, right? So if you use less energy there the battery lasts longer clear tangible benefit for the user Another thing is Performance right all the electrical energy we put into a device comes out as heat again If it doesn't come out quickly enough the device what holds down So if you put less energy in that happens less often we get more performance out and then of course the the climate impact of of energy use On that subject I can recommend you to watch some of Joseph talks If you haven't done so already Here's some actual numbers in there. I think the IT impact is in the same order of magnitude as Transportation and aviation combined. So there is a lot And worse, it's rapidly growing right. So there's there's definitely something we need to do about How does software impact the the actual energy use so How can we change this? There's of course the obvious thing right every instruction the software sends to hardware requires energy to be executed on any kind of hardware But there's also less obvious things beyond that For example power management settings, right? So if there is hardware that you currently don't use That can be switched off a lot of this is Transparent to to all level of software right that happens somewhere on the kernel or driver level But in some cases that also benefits massively from more context Say for example, you can switch off the display backlight much more aggressively on the lock screen While Compared to when you're watching a video right then you probably shouldn't do it at all so we can also Provide useful impact on on that level Somewhat related to that the idle behavior of software, right? So what does the software actually do when it's supposed to do nothing? More often than not it's actually not doing nothing, but it's continuously looking if there's more stuff to do And that means the hardware actually wakes up checks if there's stuff to do And that means we don't get to the highest Power save states and again waste energy there I mean realistically we are going to talk about very small quantities here. So Is this even worse the effort? But even small quantities matter here, right? You reduce the idle consumption by half a watt on a modern laptop that can translate to ten percent more battery, right? Which might be an hour extra. So that matters and There's more effects that that support this The some of the effects in there aren't aren't linear right so especially towards the extreme in the Bosnian like high performance part as well as in the the idle area Even smaller changes can have a disproportionate impact And then there's the the scaling effects right that's again something you might know from Joseph's talks here some actual numbers in there Because this is not a one-time local optimization, right? It multiplies over time and it multiplies by number of users and I'm not sure if I remember the numbers correctly, but I think within like reasonable like our level of Distribution and scale I think you've got up to like the energy use of a small city within a year, right? So that is that is actually a sizable quantity and that is that is our level of Scale right if we think about browser engines or something like that. That is a completely different level The flip side of that however is if we make mistakes, right? They are also going to be quite costly So, okay. This is important. This matters. We can make a difference. How are we going to do? Or how are we actually going to to optimize our software? And like with any other optimization as well the first step is we actually need to understand what what the software does Because what the software is supposed to be doing what we think it does and what it actually does More often than not are three different things. So we actually need to have a look on what's going on On Linux our go-to tool for that is the perf profiler And so we run perf record with the application that we want to measure Perform some usual operations in that We get the profile file out open that import spot and look at the details We're going to see a few screenshots in a minute That works for looking at the specific application for the idle behavior We want to see the full system view. So there we need a specialized tool Something like power talk that gives us the overview for that And this is the result we get in this case, right? So this is a flame graph view on the horizontal axis we have cost or in this case like time spent on CPU and Virtually we have the the call graph so you can dive into various parts of the application and see what's going on there Now seeing stuff in there doesn't automate automatically imply that it's it's bad, right? This is something that you look for Things that shouldn't be there things that are Surprisingly more expensive than what you would expect Right, so that is the interesting things to look at Another useful view in there is the the timeline view. So each row in there is one one thread Horizontally is the time right and every orange marker in there is activity on that thread over time and as we would expect right during startup you see quite some activity and then towards the The end the application sits idle and we would expect to to not see any activity But there's at least two threads where there's still something going on and again, that would be Something to look at because we expect the applications to not do anything at that point in time So what's going on there in hotspot? You can basically select that area of interest and filter on that and then you get the flame graph for just that particular area and then again Maybe it turns out that is necessary or it turns out that this is something that shouldn't be there This is the the output of power top that is just a table you get on the terminal List of processes sorted by the amount of wakeups they cause per second Same thing here, right? This will never be empty and zero. There's always activity going on It's about finding that activity that is avoidable unnecessarily or could could be done cheaper right, so you will see Quinn updating the screen because Power top print the output you are looking at so that required a screen output By one example that we had recently in there was Kate background window showing up with 30 wakeups per second that turned out to be an Animation timer that wasn't switched off while being in the background, right? So that is the stuff to look for So once we have Identified things that That we can optimize How are we going to do that? If you're lucky, right, you find something where you can just do less Right that is always strictly better But those cases tend to be rare right usually it's more about Making a different type of trade-off So let's take an example like you have Some expensive computation You do this twice and only use the result of the of the second run right and you just don't do the first one easy So let's assume you use the result in both times Then one possible optimization would be you just store the result somewhere, right? So if if the result is an 8-byte number That is most likely always going to be a good trade-off If storing the result means a hundred megabyte of IO right then Yeah, I'm not so sure anymore. So we need a way to To qualify those those trade-offs right, so it's that really better And this is somewhat complicated because it can also quite easily depend on on the specific hardware So if you optimize for making the software faster, right? We usually do this by measuring the time it takes until it's done So if we optimize for energy, that's the point where we then actually need to To really measure the energy use That is that is really the only viable approach there a lot of this stuff as I said earlier doesn't really behave linearly so Don't really trust your intuition Measure this and then decide which one is better So how do we actually measure this then? I'll focus here on stuff that you can practically do at home So Anything that requires special equipment or equipment with price on request price text rate that is out of scope So if you are coming from an electrical engineering background some of this might look weird, but That's the The stuff we found for for home use So in terms of where can we actually measure The first option is basically the AC side So the power plug That has the advantage right that you really capture everything that goes in You don't miss anything It is a bit of a problem for battery powered devices Wait, so when we talk about measuring a mobile because you might also see the battery charging or the battery might hide some of the the use by buffering that away So, yeah, either you need ideally you just managed to remove the battery entirely Yeah, and then there's the small problem with safety rates of AC power kind of is deadly so We can avoid that. Let's look for other options The secondary side so basically the other end of your power supply With the rise of USB-C unfortunately that is Usually not practical anymore because you need to specialize the equipment to measure on USB-C and on your workstation It's even worse, right? You know this giant power cable that comes out of the power supply In theory possible, but not practical or at least not practical for for what we are looking at here Which leaves then the the third option and that is built-in sensors Those are particularly widespread on I think standard Intel hardware and some of the AMD hardware as well So pretty much anything sold in the last decade will have some built-in sensors on Battery management systems you usually also have them But they might be tricky to get Get access to right if you have them they might actually be a very viable option to measure on on mobile Because the The big downside of those sensors is They only capture a partial view right so if you have a Sensor only measuring the CPU consumption anything else happening on the system will be missed For the battery management sensors that is less of a problem because that really captures most of what's what's going on Those sensors also tend to be extremely precise to the point of being a like a viable side channel attack factor and Well on the other hand you're kind of measuring your measurement with this right so If you have like a compute heavy workload the measurement might point their ground and the cost But if you're measuring an idle scenario right the measurement itself might be the most Costly thing running at this point in time. So that is something to keep in mind So I think that part we have to skip because That's not turned out to be only 30 minutes instead of 40 minutes So let's get to some Practical devices you can use for this We start with stuff. I wouldn't recommend you get but stuff you might find around and that Is still better than nothing maybe so the thing on the left is Usually something you find in a hardware store sold for as an energy monitor or something like that for relatively cheap Those those devices aren't machine readable which means they give you an update rate of more towards a minute and a second Which is very very way too long But if you manage to stretch your measurement Sufficiently long right that might still be good enough to say Verify the power management setting actually have an effect the other type of device is Is Like server management power strips with remote switchable sockets Those sometimes also have power management sensors and they are internet connected. So machine readable Update once a second. So that's already much better Just the price point is usually unreasonable for what you can get out of those devices Get if you have that like it gets a drop down More attractive is something like this this is like a very cheap Wi-Fi power plug sold in a Smart home context Those also have some power sensor built in And there is a free software firmware called as motor That works on those devices So you can get rid of all the cloud nonsense and get High resolution access to the sensors It's however not really built for our purpose But we managed to get like five samples a second all of this with some MQTT polling hacks If there was somebody interested in like see microcontroller programming I'm pretty sure the hardware is capable of much more and we could get much better results If we wouldn't have to abuse some MQTT protocol stuff for this but for the steady stream with proper timestamps of the of the sensor values So cheap and heckable very compact that is like does Probably as good as it gets Slightly more expensive but actually built for the various the purpose We actually want to do here. So really just for measurement is something like this the outcome powers by two That's something the Mozilla people pointed us to One downside of that thing is it comes with an ancient Windows software And we have only reverse engineered half of the protocol so far so for life measurements It's we can use it, but We don't get to all of the other features of that yet And yeah significantly more expensive than than the other one Something also interesting we got to work with during one of the sprints Is the development board for the actual sensor chips the sensor chips itself are super cheap a few dollars Single-piece price. So if you are into electrical engineering and building boards around that This is great right and this had like a one kilohertz something rate over USB so Much much more towards what we actually need and want However, this is basically an open PCB running AC power for it. So Okay ish in a lap if you know what you're doing Certainly not something you want at home on your desk with small kids around So and this is something we could get out of I think this was done with one of those So it's basically a plot over time for the the energy use You can see that this is rather noisy That's not the fault of the sensor The reason for that is in the slides. I had to skip It's massive under sampling on the high frequency of a switch mode power supply That's why I'm so obsessed with higher sampling rates But you still see some some Changes in that signal right so in the in the middle section between 100 and 150 You see that the signal overall is slightly elevated and then there's of course the clear peak in the middle And what was done there is that in that middle part the most cousin was moved and During that peak it hovered over the taskbar and show the two check That is not meant to say that most moving is bad But it it shows how sensitive those sensors are so like we really see operations on that level in there Like which is extremely useful if we want to verify if Our changes had any any effect then for the built-in sensors As I mentioned the the most common one is I think the one from Intel Rappel Running average power something The way to access them on Linux is also via perf This perf list you can check which one of those are available on your system if you have an interlaptop You most likely will find a few of those This usually cause which is the the compute part of the CPU package, which is the whole CPU carrier with L3 caches and the IO controllers and what Whatever else is on there. Sometimes you'll also find memory and GPU with separate sensors Those can be read out with first start or with the pinpoint tool I like the letter Better because it gives a continuous dump on on the measurements Which then can feed into the laptop or anything else to to visualize this And those tools however make it kind of look like you are measuring the energy consumption of a specific Apple workload That is not exactly what's happening there those sensors always measure the entire Energy use of the system right so you can put in Sleep 1,000 as the workload and do something else and you will measure that something else Which is a great you just need to to know about that, right? and This is Barely visible This is then basically the result piped into lab plot On the lower half on the upper half is the CPU load You can see that this is Somewhat correlating right so if the CPU load goes up the energy consumption goes up so as we would expect But you can also see like the peak in the middle where the energy consumption goes up Much more than the CPU consumption goes up if you see some Some peaks ready Like the energy used by the memory goes up this proportionally And so that is of course then the interesting bits because there's something that That behaves other work contrary to expectations or intuition right so Again, this is mainly meant to show you the the level of detail we can get all of this with the sensors But then if you have access to neither of those right we can also measure in the cloud the The sum of code project from current job that Joseph had spoken about in more detail earlier So the basic idea is the CI builds you the Flat pack of your software right you have a test driver a Selenium test driver for this We run this wire get lab on the Blue Angel certification setup. It's one there several times to counter to get you to counter the The noise and other stuff interfering and then it applies some statistical magic to get bit of this rate and then it produces a report With the results and plots and whatnot so quite fancy This takes too long to run it after every commit, but it's something you would run after major changes or Before release to verify that we don't have the questions, right? Yeah, so how do we put all of this together right so the The first step is just doing general profiling and optimization that is always a good idea You'd be surprised by what you find in there it helps with making the software faster for the user and it helps like usually at the same time also is Reducing the energy use If you are looking specifically for the energy use the biggest impact of course is on long-running processes And there's two things in there that are interesting right so there is Continuous or frequent workloads So for example Media playback probably has a higher impact than say opening the configuration dialog in cake right just because one usually runs Over hours and the other one you do a few times a month And then the idle behavior is the other thing to to look at System-wide because every single wake-up counts right so whatever we can get rid of there is is overall healthy And the other thing you probably saw from all this repurposed random other devices and weird command line hexes The tooling is extremely basic This is nowhere near what we are used to from for profiling for other goals, right? there is no But you basically end up Doing some manual pulling on on that device and piping that in a CSV file and opening that in as a life-source in lab plot and then Trying to put things together There is there is a huge amount of stuff that could be Could be done better and could be just also become more accessible for For us as the the users of it By not having to figure out each time how you you put this again together Yes, Joseph mentioned we have a pop on this on Monday 10 o'clock Yeah, anything related to the sustainability goal, I would say but I have some of the measurement devices with me so we can Yeah, J1 just those or you can try them if you want Yeah, that's that's it. Thank you. We have about five minutes for questions. So Oh questions Answers is this more of a comment than a question? It's a question In power top. Do you happen to know what that tick? underscores skid underscore timer entry that's always at the top actually means I Think that's some kernel thing, but I Don't know more detail either There there are things that you take kind of for granted if you're developing legs a moving things from CPU to GPU is always a good thing because It's good for Yeah, good better feeling. Is it true still true for? power That's good or bad. That's that's a very good question, right? I Think it's on the slide, but I didn't talk about it Solving the same problem with a different kind of hardware That is exactly the point where you need to measure because the There is no documented best practice. It very much depends on the hardware usually and It's often contrary to the to the introduction, right? There is some documented research from from the HPC people use for them they this works at a completely different scale and I think their GPU usually outperforms CPU But For our use cases, I think it's always measured Usually it's simply not known What is better for the for very specific work? Okay So I have a question do you have any ideas how to do real-time measurements like for example You have an Android you can see the power consumption of different applications and Do you know what do they do and how you can use this maybe on desktop? That is a good question. I don't know exactly how they do that The I mean this whole topic of attributing energy used to specific processes I know that there's some research, right? We have been at some meetings with people looking into that a lot of that is Just proportional use of the CPU and then more or less guessing, right? So there is to my knowledge no way of measuring this with any kind of certainty, right? So it's just on average probably it correlates with amount of time spent on the CPU and I Somewhat guess that this is what what I'm going to write it. It's probably close enough Just because of the big differences in there and it doesn't really matter if it's a few percent off But like proper strict attribution, I don't think there is the the means for that I have a care Here two questions from our remote participants One is our cheap wall outlet meters without certifications safe to not burn down your house I mean those are sold as consumer devices with all the necessary Certification So they are generally safe. I would say and we don't modify them, right? those all are I mean the reason why also the the more professional device has a Bluetooth interface rather than a cable interface is because it makes the safety easier, right? If you don't have a low voltage signal cable coming out of something that runs AC power, right? Then you solve the whole lot of safety problems So yeah, I mean all all of those have the CV certificates or whatever you need in your your country I don't want to take any guarantees on this is 100 cents safe, right? But this is like off-the-shelf standard hardware made for home use. So It should be as safe as it gets Thanks and the second one Do we have a general guide with Katie within Katie to know how to do a general profiling? That is a good question. I don't think we have an like like single place go-to thing for Like a profiling tutorial But I think there is a lot of material out there conference talks and Other tutorials on on that general subject, right? It might help if we collect that and employing people there At least that part is missing. Yes And the last question I think we'll come from Heinz Hi, so This was a lot of talk about measuring usage on the CPU Is there anything related for usage on the GPU? from the built-in sensors if you have an An input CPU with GPU built-in that will also show up as a as a sensor. I think the one of the lines in here the The blue one is actually GPU I'm not sure if you get the same level For dedicated graphics cards, right? So that depends on Whether they give you access to the sensors they have I'm not sure if we have that but for the built-in Embedded into GPUs we get that. Yes All right. Thank you Volker Thank you people asking questions. Thanks for the online participants for the real-life participants here It is a lunch time You can talk to the cute stand They're hiring and you can talk to the sandwich people they're feeding