 Hi, it is wonderful to be here again. For a change, I'm going to be talking about making laptops work with Linux. I am not going to talk about suspend and resume at all. Because really, I'm very bored of it. If you've heard me talk before, you're probably very bored of this as well. And also, it really just makes me want to stab people. But laptops, laptops are wonderful, incredible devices that allow us to carry around a machine that's capable of running calculations more quickly than was conceivable even a short number of years ago. Of course, when it comes to writing software, laptops are incredibly annoying, dreadful things. In this presentation, I'm going to be attempting to pass on some, well, a good amount of what I've learned over the past seven years or so I've spent working on these things. As a result, I may have to go quite quickly because I don't have seven years. Every time I ask the organizers for that sort of length of time, they tell me that that would cause problems. Anyway, what is a laptop? It's very easy, straightforward, to think that a laptop is merely a PC in a small box with a screen attached and keyboard. And they're kind of like that. You can typically boot an unmodified operating system on a laptop unless it's an Apple. So obviously, we have cases where that's not the truth, not. There are some fundamental key differences to tops and PCs. And that is what I'm going to be talking about today. What some of those differences are, how those impact your ability to use the full functionality of the laptop, and how you can actually deal with the fact that there's a moderate chance that right now that stuff just doesn't work for you. The main thing that distinguishes laptops from normal PCs is something called system management mode. We also see this on servers, but it's typically most annoyingly prevalent on laptops. System management mode is this wonderful feature of PCs, of all x66 chips pretty much since the 3x6 SLC in the late 80s, which was the first mobile x86 part. What happens in system management mode is that the processor stops running your operating system and starts running some code that is hidden by the hardware and inaccessible to the operating system, meaning that it is actually impossible for you to know what it's doing. This can cause some problems. Well, of course, it causes problems. It's software. And software that you can't see is typically worse than software that you can see. So system management mode works fairly well, as long as you behave identically to Windows. An example of a fascinating case that we had, and I say fascinating because everything else I could say would be against the conference abuse policy. One thing that often triggers system management mode access is, for instance, closing the lid at the laptop. Because obviously, back in the old days, before we had operating systems that could do things like make your graphics work, so 1983 or so, that was a joke. I'm sorry. You close the lid and then it executes some code. On HP laptops, if you close the lid, it would execute some code. If you open the lid, it would then execute some more code. If you happen to be running on a dual core system and if you happen to be running on core 1 rather than core 0, when it came back, it would trash the register state because it restored the core 1's register state on core 0. This results in hilarious things like clearly impossible backtraces because that's not where the instruction pointer was. So we eventually figured this out. And when I say we eventually figured this out, I screamed at people at HP for a while and eventually one of them took pity on me rather than calling the police. And we discovered, right, we need to run this on CPU 0. And now, as a result, the ACPI interpreter in Linux always runs events on CPU 0, hooray for progress. But how did we get into this state where system management mode is a problem for us? And we have to go back to a period in the early to mid-90s where there was this wonderful thing called APM. Those of you who know anything about APM, you probably don't want to listen to this because it'll just bring back bad memories. APM was advanced power management. By advanced, they mean it was better than having an on-off switch. It was completely pretty much handled by the BIOS. The operating system had very little control. If you hit the sleep button, then the specification stated that the operating system could do whatever it wanted, provided that after it's done whatever it wanted, it said, OK, you can go to sleep now. So really, we had this awesome position where policy existed in the BIOS. And we all know that BIOS authors have wonderful taste. Again, that was a joke. Advanced power management, as he says, was mostly handled by the BIOS. Eventually, people decided that firstly, the advanced power management specification was a problem because it meant that the operating system had no control. The other reason that advanced power management was a problem was that the specification was short and readable. ACPI fixes both of these problems. ACPI is a dreadful, awful, horrible specification as are all specifications, so that's all right then. But it's a mechanism where we provide, where the firmware, rather than doing things itself, gives a standardized interface to the operating system and policy is left in the hands of the operating system. So whereas previously, things like fan control were entirely handled by the BIOS, ACPI means that the fan control interface is standardized to the operating system and knows how to turn the fans on and it knows how to turn the fans off again. Backlight control. Backlights were for a long time controlled in a wide variety of different ways and you had to have driver specific hardware to make them work. ACPI version two introduced an appendix which provides for standardized backlight control. So wonderful. Windows 7 finally mandates that people provide this on their machines, otherwise they're doing this Windows 7 sticker. So it actually even works now. It's awesome. Batteries, back in the APM days, you were able to say how much time is left in my battery and what percentage is the battery at. That turns out to not be a particularly useful interface because how on earth did anybody think the BIOS authors were the best people to work out how fast your batteries go to discharge. This is some crazy mixed up universe where BIOS authors are useful people. IPMI devices, how many of you know what IPMI is? I'm sorry, the rest of you keep it that way. A lot of, so basically beyond fans, backlights and batteries, pretty much everything that ACPI standardizes is completely irrelevant to laptops. ACPI has turned from a wonderful brief 500 page specification into a 900 page specification because it is now at least 15 times more enterprise than it was 10 years ago. So there's a lot of things that laptops have and that vendors want, which are not standardized by ACPI. Things like hotkeys because it's nice to be able to, actually that's not quite right. The sleep key is standardized by ACPI. Lots of vendors completely ignore that anyway because why bother with a specification when you can have two standards. Anyway, hotkeys, so things like web browser keys, hibernate keys are not standardized by the ACPI specification. Controlling the radio control here, I mean controlling the radios as opposed to being able to drive your laptop around from a distance. And various small miscellaneous features on devices which have trans-flective screens, for instance, there's often an ACPI mechanism for changing the screen from backlit into trans-flective mode. There's opportunities for, oh yeah, they've protection systems. So if you detect that the laptop has fallen off a cliff, then at least we'll spin the hard drive down before it plummets into the sea at about 150 miles per hour. People have put a lot of this into ACPI and they've all done it differently because laptop vendors are only marginally more competent than bios authors. So if you have a laptop and if you have hotkeys or RF kill controls or any of this functionality and you want to work out how they're controlled if they don't currently work, there's two common mechanisms. As I said, first is ACPI, the second is WMI and I'm going to get to both of these in due course. ACPI obviously since none of this is standardized and even then the only thing that ACPI standardizes is the interface to the operating system. The actual backend implementation of all of these can be done in a variety of ways. There's typically two main things. The first is that an ACPI access can somehow trigger system management mode and the other is that the ACPI access can trigger embedded control access. So let's go through both of these. This is what an ACPI method that triggers an SMI looks kind of like. This is from a Toshiba laptop. This is the method used to attach a Bluetooth device. When I say attach, I mean it's in terms of laptop. You need to call this for it to actually appear on the USB bus. As you can see, this does nothing except call the SMBR function. SMBR is not a standardized ACPI function so let's go through the ACPI tables and try to find out what SMBR does and the answer is SMBR does this. Those of you familiar with X86 will realize that this means that it is copying the first four arguments into sections of memory that have names that look awfully like X86 register names. So that's your first warning that something very bad is going to happen now. And then the next thing it does is write to this TRP4 which is actually bizarrely, it's a temperature receptor protein in Drosophila. But that's not strictly relevant right now about painful memories. In this case, what that means is it's a trap, a write to that trigger system management mode and the system management codes dispatcher in Toshiba. At this point, I'm now guessing because I can't see the system management mode code because thank you Intel. So you write there and then suddenly the operating system stops. If you're lucky, at some point later the operating system starts again but that's not really guaranteed. So it traps into system management mode. The system management code then reads these things out of memory, puts those into registers and then jumps to a function based on what these arguments are. So at that point, presumably the system management code then does something like attach the USB device internally but for all we know, it's also simultaneously DMAing. I don't know, DMAing a copy of the 1812 overture over your file system. Perfectly valid thing for it to do under the ACPI specification. If you see this kind of thing, then you have problems. Obviously, as I said, there's no way to really reverse engineer system management code except with some old buggy chipsets where you could flip a register bit and then suddenly instead of where your graphics memory was the system management code was. But unfortunately Intel realized that this was a bug and they told BIOS authors how to fix it. So now you can't do this. Thank you, Intel. So this is basically impossible to go any further here. The only way you can work out what this does is from context. As I said, this is a function called SMDR but it's being called from the AUSB method. In this case, the AUSB method is part of an ACPI device called TOS6205. If you put TOS6205 into Google, then it tells you that this is a Bluetooth enabler driver for Windows. So AUSB associated with Bluetooth, this probably makes your Bluetooth appear. So wonderful. You've worked out what it does without having to care about the system management code. Vendors are thankfully moving away from using this kind of system management code. Instead, we're moving generally towards using the embedded controller. This is wonderful because in system management code, the entire machine is stopped. Every processor, if you've got a 48-way system, one of those cores, in fact, core zero for reasons explained earlier, is executing system management codes and the others do nothing. If you use the embedded controller, then those other 47 cores can be doing something useful like polling the embedded controller, waiting for it to start talking to us again. But what is an embedded controller? And right. An embedded controller is an AC51-style microprocessor. It runs its own proprietary closed firmware. You do not have access to it. Currently, there is a partially complete, open implementation of an embedded controller firmware. You cannot replace the embedded controller firmware in your laptop with a generic one because it's laptop-specific. The embedded controller can do approximately anything in response to anything. The one thing the embedded controller can't do is DMA the 1812 Overture over your file system. Joy to the world on the other hand, it can. No. Right. So when you're looking at embedded controller access, as I said, the embedded controller is handled by the operating system rather than system management codes. So there has to be some way for the firmware to tell the operating system which embedded controller registers to write to. So we're looking at a method and then, right, so we're looking at this TOS 6205 device again in Toshiba. And this time I'm talking about a more recent Toshiba that doesn't use system management code because thank you. Binance authors occasionally do learn. It just takes a few years of shooting them first. If you've got a method name called BTON, and it's associated with a Bluetooth device, then maybe that has something to do with turning your Bluetooth on. By the way, in terms of method names, ACPI method names and object names cannot be longer than four characters. Some vendors, such as HP, decided that this was utterly ridiculous and they would use proper, useful names so that they would know what they were writing. The downsides to this approach is that they then can't put those proper, useful names in the actual firmware because the specification doesn't allow it and the interpreter's just collapsing dimiserably. So they post-process their ACPI source code and every method name is replaced with C and then a hexadecimal identifier which starts at zero and just increments. The wonderful thing about this is that if there's a BIOS update that adds a method, all the methods you approve to the user get renamed. So the take-home message there is when you're making ACPI calls, you should attempt to identify the proper entry point rather than merely finding a method that works and calling that. If that's called from somewhere else, then you probably should be calling that instead. Anyway, the embedded controller may have a register called BTPW and well, BT, that's probably something to do with Bluetooth on the assumption that your firmware probably doesn't implement BitTorrent. So BTPW, maybe something to do with Bluetooth? Right, so we had previously, remember the AUSP method that had the system management call. This is an equivalent where rather than calling system management mode, it says store one and then it stores one to that BLTH register in the embedded controller. When that happens, we go back to the kernel, the kernel talks to the embedded controller, writes one to this, the embedded controller then presumably brings up some sort of GPIO line and suddenly power goes to your Bluetooth device. Awesome, that's a pretty straightforward, easy example. If anybody's interested in, that's in more detail, I wrote an article on rising a driver for this device that's in LWN a year and a half or so ago. It's not always as straightforward as that. So Andrew Morton used to have this Sony laptop that was a wonderful device because every time anybody changed anything, including code that couldn't possibly even run on it, it would stop working. Sony and exploiters, I think, pretty much every way to build something that is nominally a PC and yet runs Windows fine and then just collapses into a gibbering heap the moment we try to do anything else. Okay, Jynian suggests that maybe this is best than Apple and the answer is yeah, probably, but being stabbed in the groin is best than being stabbed in the eye. I wouldn't suggest either of them. Not on the Sony, Hockey's work on older Sony's. Then they released a new range around the end of 2008 to start of 2009 and the Hockey's didn't work. Irreshacing because it's useful to be able to do things like control the backlight. So first thing I did was look through ACPI tables looking for notify methods. Notifies are the means for the firmware to tell the operating system that something has happened. I looked through a bunch of them. Most of them were things like notify, EHCI, blah, which is sending a notification to the operating system saying that something just happened to one of the USB host controllers. So in general, that's a power management thing. If the USB controller were powered down, you plug something in, we get the notification through ACPI to wake it up. But there was one that was not something like a USB device. And this was this SNC thing. SNC we knew from previous work is a function, sorry, is the device that provides all the Sony specific functionality in the firmware. This notifying method was in, this notify call was in a method called SNNE. So the next thing to do was look for references to SNNE elsewhere in the ACPI tables. And well, wonderful. This is nice and straightforward and promising. Q underscore Q, underscore Q indicates that this is an embedded controller interrupt event. When the embedded controller weighs its hands and says, hey, something just happened, then this is the method that would be called. And it did, if HKEV, where HKEV was a bullet register. So, okay, at this point, we're going to make the possibly dangerous assumption that these firmware authors are not horrendous awful people. And maybe HKEV means hot key event. Actually turns out to be right in this case, which was awesome. So if there's been a hot key event, we store the contents of HKCD in this ECR variable. And then we call SNNE. Awesome. So we now know how the notify method gets called. The problem is that the notify, as you'll see, is just 0x90 plus a number, and that it doesn't actually tell us in the notification which key was pressed. So we need to work out how to get hold of HKCD. As I said, HKCD is an embedded controller register. Awesome. So you'd think that we should just be able to resets the embedded controller, which would be absolutely fine provided that SNNE never, ever changed the location of that register. Though of course, SNNE repeatedly changed the location of that register. Which means we can't read it directly. We need to find out how to get access to it through the firmware. So remember, we stored HKCD in ECR. So we need to find something that takes ECR and hands it to the operating system. And there's a function called F100, catchy name. So we need to call F100, but the problem is that F100, again, sounds like it's one of those things that you probably can't guarantee will always be the method you want to call. So we wanted to check where F100 gets called. At this point, I should emphasize that I was trying to write this driver without actually having one of these laptops. It gets called by a method called SNCF. This is not the French train company, but if underscore T underscore zero, and again, we're all over these catchy ACPI variable names, is 100. So score T underscore zero depends on the argument to SNCF. Index CFGIA at local zero of X20. And then it copies it into T zero. So what that's actually saying is local zero, X20 to it. Go into an array, go to that entry, and then copy that entry of this array into SNCF is called from SN07, which we'll get to shortly. As you can see, we have OX0100 there from that. Briefly remember, I said that we wanted to call SNNE with OX100, so we know now that we need to be getting entry zero from this array. In order to have OX100 copied to that variable and then from there. So the CFGIA thing is copied into a buffer that is OX20 bytes larger than it is at the end. So that's why we have that OX20 offset. So the offset of OX100 is there. So what we now know is that we need to call SN00 with because OX20 plus zero will then give us the argument that gives us OX100. You probably don't actually care about this. I'm really sorry. So how do we get the offset of OX100 in the first place? Well, the answer to that is we call SN00 and we call SN00 with arguments going from OX20 to OX30 until it tells us OX100, and that means that we then know the offset that gives us OX100. And finally, we know key register, not particularly significant. This is purely an example, but the main reason the details aren't important is because the driver is there. And if you care about how a driver works on hardware designed by Sony, then just find something else to do with your life, please. ACPI functions also write we know how to do all of that now. You have now gained all the knowledge required to work out or does radio control or anything like that. And you can write your own driver. So get on that, except there's a problem at some point in their past decided to design a specification called WMI or Windows Management, it's not interface instrumentation, which was a straightforward API where you could hook various device specific monitoring code into operating system infrastructure and then have it all set up so that it would automatically contact you if one of your servers got too hot or something. At some later point, they decided that they could integrate this WMI information into the firmware rather than having to add extra codes to Windows. And then at some later point, Vendor decided, oh, well, if we hook up in through WMI, all into ACPI that way, and that's far more straightforward than actually making ACPI calls ourselves. The main problem with WMI is that because there is this extra indirection through the WMI, the details of WMI are unpleasant, horrific, miserable, bad, awful, dire. An article on LWM, which tells you how to, you should read that because if I start talking about it, I'm going to cry. So the problem's more fundamental than that. Most hardware is only tested with Windows, so there is the potential that instead of functionality not working, things will just behave a bit oddly. For instance, maybe you try to reboot your computer and rather than reboot, it hangs. Apple did manage, and most hardware is only tested with Windows, so if it works on Windows and doesn't work on Linux, then that's typically because we're doing something different to Windows. Generally speaking, we try to make the kernel as compatible with Windows as possible when it comes to hardware interaction. This is not always as easy to sound because it turns out that Windows isn't particularly well documented in terms of its hardware interaction. So given that disassembling Windows is something that would probably cause awesome lawsuits, if I, as a Red Hat employee, were to do it and then use what I'd learned to improve Linux, we need to find another way. And that other way is QMU. QMU is an 866 emulator. You can run it over KVM and then have virtualization rather than emulating a full processor, but CBIOS is the open source BIOS image that it runs. You can edit the ACPI tables in CBIOS. You can then hack QMU to dump useful information and then you can run Windows within QMU and see what Windows is doing to the hardware. And when I say profit, unfortunately so far, this has not resulted in any significant profit. Me developing a taste for fine whiskey, the life of an open source developer. For instance, if we have here in ACPI terms, what this is saying is that there is a system IOPORT at OXB080. This is four bytes wide. And within that region, there is something called DBGL as a register, which is 32 bits wide. So if you write to DBGL in the ACPI tables, the contents of that write will go to system IOPORT OXB080. In QMU, if you then call register IOPORT write OXB080 for one of those forces with the other four is the number after three, then you have a function that will be called if anything writes to that port. So if we then have in the ACPI table, if you want to know the contents of a variable at a specific point, you can store that variable in DBGL and then that will be written. QMU will then call your magic function and your magic function probably does something useful like print that to the console so that you know what the value was. And see if they're doing the same thing. This has been a wonderfully useful tool in identifying certain ACPI bugs we had, like the one where ACPI provides a method for saying, I am an operating system and I support these features. Please stop doing this yourself and give me control of the features. We were calling that once per PCIe root port. Windows called that once per boot. Some systems were doing things like creating objects in that routine. So if you called it multiple times, they tried to create an object which already existed which is a violation of the ACPI specification. So you got errors and all your PCIe ports stopped working. Turns out the customers complain about that kind of thing. So you boot Windows, you boot Linux. If they don't do the same thing, you change Linux so that it does what Windows does. And then you post a page and then somebody says, oh, but this is different to what we're already doing. It might break something. And the answer to that is yes, it might break something but the only machines that will break are ones which don't boot Windows and those machines basically don't exist. So rather than optimizing for machines that don't exist, maybe we should optimize for machines that do exist. And then your patch sits there for another three months and then you repost it. You have the same arguments again and eventually it might get applied. To recap, if you have something like a hotkey that doesn't work, look through your ACPI tables, attempt to find a notify method. If it's then associated with an ACPI device, write a driver. Pretty straightforward, read my tutorial on that. If it's on a WI device, read the other tutorial that's on LWN. And yeah, I've written several WMI drivers now. I don't enjoy it. That's all there. If things don't work, try to identify where the Windows and Linux are doing the same thing. If they're not doing the same thing, make Linux do what Windows does. So that is the end of my presentation and I've even managed to do it almost precisely on time. So we have a few minutes for questions. I would prefer questions to have some at least tenuous relationship to my presentation. But, you know, we'll see what we can manage. Oh, thanks. Samber got a bit of traction in dealing with Microsoft with specs and stuff. Have you guys got any traction there with Microsoft in getting hardware manufacturers to deal with both of you or something? Right. As far as hardware events are concerned right now, Linux is something that you put on big servers which have the word enterprise somewhere in their name that you sell to large customers who probably have, I don't know, Goldman Sachs in their name. And they generally don't have built-in radios or hotkeys. In terms of laptop support, most of what we've been able to deal with so far has been with vendors who do actually sell Linux on devices. So companies like Wave. It's one of Acer and... I think it's Acer's have actually been relatively useful in that Novell have been collaborating with a couple of network vendors and we've been getting good driver codes out of Novell for that. In general, vendors believe that the secret of how you turn the Wi-Fi cards in their laptop on and off is so amazingly awesome that if they were to release a specification, somebody else might steal their mechanism, which would be awesome for me because then we could write one driver instead of 15. LAUGHTER But by and large, no, we haven't had a great deal of joy getting stuff out of vendors. The other thing is, in the law of Microsoft, really, most of these drivers are provided by the hardware manufacturer Windows ownership with them. That's why if you install stock Windows rather than the recovery desk on a laptop, a lot of stuff just won't work. So, Derek from Intel says that this is changing. I would love to believe that. I hope that we will see wonderful, huge standards and drivers. And we will merge those drivers and life will be perfect and then I can stop having to maintain code. Oh, God. LAUGHTER Anyway, next question. Yes, if people have laptops where the WYSIBits don't work on the Linux, how can they get help? How can they get help if the laptop WYSIBits don't work on the Linux? Well, first things first, you should obviously teach yourself how to write drivers and then you should write a driver. LAUGHTER If that doesn't work, you should file a bug in the kernel bugzilla and you should email the platform driver x86 at vega.kernel.org, mailing list. It's a wonderfully catchy name. I came up with it myself. Email that, say, this doesn't work and then we'll look at your ACPI tables and we'll try to work out how to write a driver. What will happen to your final whiskey addiction once UEFI starts being deployed more widely? Ben asks whether UEFI changes this at all. The answer is no, UEFI doesn't change this at all. All the magic stuff is still generally done through ACPI, rather than through the UEFI firmware. UEFI doesn't provide anything particularly useful. Was originally going to put more words after that and now I realise that it just didn't actually change the meaning. Matthew, you seem to indicate that most of the sort of firmware level, the symbolic names anyway, left in the firmware is pretty inane. For what reason do they leave the symbolic names and not strip them in that case? The majority of vendors leave the symbolic names there, I assume because they wrote the codes with the symbolic names to begin with and they could post-process them out, but that would involve BIOS writers doing extra work. My general baseline assumption is that BIOS authors don't want to do any work at all, so that just wouldn't happen. By the way, is anybody in here a BIOS writer? I should probably have asked that earlier. Yeah. So Matthew, how do you suggest that your average Linux loving hacker goes and buys a laptop today? How do you just make the decision? How do you get the information about what works? What's completely broken? Which vendors are being helpful? Which vendors are being recalcitrant? Practically speaking, just by think, Matt. As far as I can tell, Lenovo, when they bought the consumer brand stuff, kept IBM's firmware authors and the firmware interface hasn't changed a great deal. Every so often, a new Lenovo comes out with some sort of wonderful feature like when you unplug the power cable, it decides that instead of reporting in milliamp hours, it'll report in milliwatt hours and it'll do that by taking the milliamp power value and multiplying it by 10 for an 11.8-volt battery. But there is obviously there's vendor insanity there, but usually ThinkPad stuff doesn't change a great deal. It tends to work. I would have said that HP's sourced relatively well, except I keep hearing the occasional reports that new HPs that I don't have when you try to disable the Wi-Fi instead make your embedded controller stop working. I consider a bug. In terms of working out what's the problem there is I just haven't had the opportunity to reverse engineer it. Somebody was in fact writing code to fix this and that was great, so I didn't do it because they were doing it and then they emailed me to say that actually they've been told that they were not allowed by legal to release this code. So at some point I'll fix that until then just by ThinkPad, it's really going to be easier. In terms of cheap stuff, Dell actually works reasonably well. We have occasional trouble again with the radio control but I think most of that's been worked out now. If you've got ThinkPad, go to ThinkWiki.org where you will be told various things which may or may not be true or helpful. Or obsolete or, well, to be fair, really every time I, a lot of what ThinkWiki provides is working around for Linux bugs and deficiencies. I keep fixing some of those bugs and deficiencies. When I do that, I should really go to ThinkWiki and indicate that this is now fixed. But I don't do that because like BIOS Rises, I am lazy. Yeah, yes. Which kind of version of that? Right. So one thing that might, I don't think you made really that clear yet is actually how you get the ACPI tables from the BIOS so you can actually handle it. How do you get the ACPI tables from BIOS? You install a package called PMTools. This is not to be confused with PMUtils. And then you run the ACPIDump command as root. That then copies the tables, dumps them into a file. You then run ACPI extract on that file and it gives you each individual ACPI table. And then you install the IASL Intel ACPI Source Language compiler and then IASL minus D and then the file name that will decompile the ACPI table back into something that resembles the original except with all the comments removed, obviously. This will be our last question. So you've covered off most of the kind of traditional Intel-based platforms running Windows as you alluded to earlier, Apple does things differently. So what's additional steps you have to take when you're trying to make Linux behave well on Apple's hardware? Right. Apple, I've actually been working on improving Apple support the past couple of weeks. Apparently because I hate life. Apple, the most recent bug that I hit is when you start EFI, if you boot by EFI, EFI has this wonderful functionality where rather than trapping into some special modes to call EFI functions, you can in fact just get a function pointer from the EFI tables and then you just call that. On X86, you have to call it using Windows Calling Convention. The EFI specification helpfully calls this the C language calling convention because Windows, I don't know. So you call the set virtual mode, which then means that EFI uses the operating systems page tables rather than assuming linear mapping. And on the Apple implementation, at this point, it jumps to some code that is no longer meant to be there and we have marked that page as non executable and then you fault and then you die. So far, I haven't come up with a good way of dealing with Apple stuff other than staring at it very, very hard and crying and then doing the same for another five days. Far as things in Apple stuff are still broken, I'm working on it. Unfortunately, blending company property is probably not conducive to long term employment on my behalf. Anyway. Yeah. That's all we have time for now. As a sign of LCA's appreciation towards Matthew, I have this little bowl made from macadamia nuts. Thank you. Thank you.