 Hello, good afternoon, and it's the last session. It's almost over. Goodbye So my name is Mark Zangier. I'm Apparently with arm for been for a while. I tend to look after things like KVM and also things like RQ domains and That's the that's the subject for today. So I've tried to have the longest possible title I'm not sure I want so that means Or how dealing with modern in top architecture can affect your sanity and Yeah, that's another statement So what are we going to talk about? Quickly go over what's an interrupt wait, I won't last very long interrupt controllers What sort of data structures? Linux uses to represent Intrupts and interrupt domain Chain interrupt controllers Intrupt controls your keys MSI's and whatever and the whatever is Splitting that sometimes Please interrupt me If there's one thing I hate to do is to just talk on my own that usually doesn't last very long and I fall asleep Especially as I had very little sleep lately, so please interrupt me if that's something you don't understand something Where you think I'm completely wrong, which is likely Please yeah, raise your hands cream shout for something at me. Just do it Late advertisement So I listen to the really really really good talk on on Tuesday Please have a look. It's probably will be uploaded somewhere at some point So it's it's really orthogonal to the to this discussion today. It's really about the the dynamics of Interest handling When things get preempted and then and threaded interrupts, it's really good. So please have a look What's an interrupt? So pretty simple. It's just it's just a wire. Well at least it's been for a very long time just a wire That the device shakes repeatedly sometimes Towards the CPU and Sometimes things happen. So what the purpose of that line is just to indicate a condition on the device something has happened What has happened is completely device dependent, but yeah Something the device was expecting has miraculously happened and it's telling you please go and have a look So as you've seen here, I have just one interrupt one device one line to a CPU Of course, we that's that's a bit weak. We want something slightly bigger So we like to multiplex interrupts and for that we get the help of an interrupt controller. So As we have you know tons of them It's nice to have a dedicated piece of IP for that. So it's very often architecture specific though it has a number of Let's say Features that we we can find very often such as masking interrupts Setting the SMP affinity things like waking up a CPU when it's in deep sleep mode and Also things Linux doesn't use like priorities Features so That's what an intro controller looks like from a very high level view That's my my friend the geek with which I spend way way way too much time Very fine thing So yeah, it's it's say if we go back that that yeah that box is actually a much bigger thing in real life And it deals with per CPU interrupts dealt with priorities being being able to to Respond to both software and hardware interaction at the same time in supposedly a race-free way. So it's a It's an interesting piece of kit and most interrupt controllers are you know, eventually quite complicated the closer They are to the core the the more complicated they are so general stuff there's there's I'd say two classes of an interrupt as the on my left level triggered interrupts on my right the edge trigger interrupts And yeah, they're not quite the same. They don't indicate the same thing So level triggered interrupts could be either low or high depending on on what the device and the control agree on That indicate a persistent condition if you receive a packet on your Ethernet controller and the internet control tells you I have received a packet It could say I've received a packet and I won't I will keep a set that line until you come and service that interrupt Because I don't have a buffer. I just I can just hold one packet and as long as you haven't Service that interrupt drain the packet out of the Ethernet Buffer, I won't be able to receive another one. So it's really persistent. Yeah, you have to do something and Then there's the opposite which is an extra good interrupt, which is something has happened and that's it so you can that's just a pulse and Most of the time it doesn't prevent The system from carrying on working and if we keep the analogy of a offer an Ethernet controller it's because that Ethernet controller is capable of DMA into your Into memory directly. So it doesn't really matter if you have to come and service it immediately Has all other interesting features Such as if you get two of them well in Quick succession you can't really distinguish them. It's really quite likely that you will only see one They will just get a coalesced at the interrupt control level Also some some system just ignore the notion of of a trigger type could be that Substracted it's it's virtualized power virtualized or it's just a glorified exception handler Like some architectures to sort of Let's not talk about that And now for something slightly different and then we get into the meat of the thing So Linux has a number of data structures to represent interrupts. So one of the most Famous one the RQ chip. That's a bunch of methods functions function pointers To drive the hardware, so you'll find things like mask and mask and interrupt EOI and interrupt if you if you Your interrupt control is capable of dealing with a with fast EOI Things like setting the trigger type so that's really Configuring and interacting with the hardware at a very low level But that's that's the first thing you're going to implement or I need to to drive that piece of hardware I'm just going to add these functions that that deal with directly RQ domain is A much more abstracted thing so first it's Usually tied to a firmware description. So if you're if you have a device tree The it will be tied to them to the OF node that correspond to that Intrub controller, so we tend to tie an IQ domain to an interrupt controller. They really go hand in hand and That that's reusable because we can represent here our keys like we can say this device points to this RQ controller in DT you have an interrupt parent property somewhere And that gets represented as well in them in the kernel So we know that for example that that interrupt domain the RQ domain correspond to that RQ chip Also, we have a way to convert Let's say for example an interrupt specifier in DT to something that the kernel actually understands which is an ID an Intrub controller specific IDs we've called hardware IQ Which is really the if you look at the data sheet of an interrupt controller will say oh, yeah pin number two is Intrub number two so that in your DT you'll put a two somewhere and that's your Hardware IQ that's really tied to that Particular in truck controller and that's only valid at that level that has no global meaning We also get things like your trigger type from DT or a CPI or whatever firmware description you could use Well, Linux has only these two so far Also, and that's The most important thing it can from this hardware and RQ this local Value can give you the corresponding Linux View of an interrupt which is an RQ desk or either an IQ desk or an interrupt number These are actually the same thing So it has nothing to do with this local number something global that the kernel Understand and deal with it when you do request RQ That's that's the this virtual interrupt number that's this global space we're using and We get onto RQ desk So RQ desk is that global view of an interrupt Very big data structure. That's all kind of states, which we're not going to even Well into it. It's really really big Contents all the core stuff all the states That's where Linux is going to keep Really at any point in time. Okay, which state that interrupt is in is it in it's a the threaded interrupt Is the handler running have we disabled something is it masked? As it fired too many times counters everything will be there And as this one-to-one mapping with the interrupt number RQ data Until very recently was only embedded into RQ desk and then That's the link between Your RQ chip and the RQ desk it contains both this global interrupt number and the local one It really allows you to to go from them from the virtual number the global number to the local one You have to go through RQ domain to go to to the inverse translation So these are the basic for data structures We're dealing with No question nobody's lost I Hear someone snoring so when we Start taking an interrupt. Let's say. Wow was the pointer. Yeah, it's not that come back So let's say this device generates an interrupt so The CPU is going to access the the interrupt controller Read a hardware register here of some sort. It's going to say oh Hardware interrupt number two has fired What am I going to do with that? so We We From the hardware interrupt number. We asked the RQ domain. Well, what was it? We get an RQ desk actually the virtual interrupt number, but that's that's that's a one-to-one relationship and with that the the core kernel is just going to Tackle the the interrupt just Executing the your back ends are the RQ chip level. It's really really really simple There's there's no magic the the real magic happens here when you are able to convert the RQ chip the The RQ chip local view of an interrupt into a global one And that that is populated through device tree for example or ACP I so quite often we we need more than a Handful of interrupts your interrupt controller can do me you can for in this Chromebook here The gig on on on this machine has I think 256 interrupts the the system is using 300 and something obviously you need to to do something so we tend to do these Chained interrupt controllers This machine that's two of them for an extra 40 or so interrupts So how does it work? Well, it's quite easy. You dedicate one input line of your primary interrupt controller And you dump another a job controller on it and like that you must you multiply things That's yeah, we've been doing that for ages the Every x86 system has one even if they're not using much of it these days So that requires a two-level handling you you need to act At the first at the primary control level and then go up to the secondary level there's a whole set of API to deal with that But please never treat that as a normal interrupt handler this chain handler is really really specific It's not even a real handler. It's actually a flow handler So that has a real meaning in the in the RQ subsystem. It's really not a normal Device driver handler And that's how it's handled. It's a bit. Wow overwhelming. That's only the beginning. I'm afraid So what happens say same scenario the your primary interrupt controller fires hell was the hard work you number That's the one RQ2. I'm going to resolve that to my Intrub RQ domain get the RQ desk at that point I'll kick a a chain handler here That brings me to the the secondary domain corresponding to that interrupt controller From that interrupt controller I resolve its own hardware RQ Resolve it into the top level RQ desk and only at that point I'm able to start actually executing the handler. So it's quite convoluted It involves reading two hardware registers. It's not going to be fast and if you were at Linus valet's talk a few minutes ago It will have explained that yeah, you can also have these over an I2C bus And yeah, that's really really not going to be fast But that's useful meaning you don't need to have you know fast interrupts all the time Sometimes you can just take your time So as most people are using device tree, I hope any any a CPI fanboy in the room No, oh, that's one. Thank you. We did it one So how does it work so we have this device here That's connected to that interrupt controller here, which we find here so itself it has an interrupt and Where is that interrupt going well to find out because there's no Intrub parent property here. We have to have a look at the default. That's up there So that's the gig So we find that this interrupt It's plugged in this interrupt controller, which is cascaded into this one and don't be fooled by this interrupt here, which is a Private interrupt to the gig which itself can interrupt generate an interrupt That's for for virtualization should have removed it from the from the examples with confusing if you ignore that that city interrupt here you find the cascade with just Which just shown before So that gives you the the full interrupt path in DT's pretty pretty easy to read and If you haven't done anything silly The interrupt won't the chain interrupt won't appear in proc interrupts If you if you ever implement the cascaded RQ chip and if you find the cascade Interrupt in proc interrupts. You're doing something wrong You're using the wrong kind of handler and your system is broken. Do not do that I know it's everybody likes to have pretty numbers in in their proc files And they all have to like to hack CPU info and but no don't do that If it if it appears there is strong It's because you've you've told the kernel that it was a normal interrupt and it's likely that you can generate a deadlock with that now Multiplexing is not everything There's a number of cases and actually more and more cases where We end up having Interrupt controllers or Things that process interrupts that I don't have an end to one relationship But instead of one to one relationship So they have as many input as they have outputs and yet they have an impact on the on the interrupt signaling So they can be Things like intra pruters, you know, you can you can imagine This thing has been some kind of crossbar and being able to have some internal routing is completely programmable or that can be a Wake-up system wake up controller when that it if you suspend your system and that interrupt controller, which is the primary one Goes off, you know, it's being poured off. You need something to wake up your CPU So can imagine that here on the receiving an interrupt you have some lines in parallel coming here and In normal when you normally drive the system, this is completely transparent You really don't see it, but you might just intercept things and stash them somewhere for for let's use And you have things like programmable line inverters that that sounds completely silly. But yeah, people do that. So if you have a Why am I gonna get doesn't work. Well, never mind under on the on the far right if your interrupt is only capable of let's say level high and all your devices are level low Well, you want to be able to To invert the signal so you could just put an inverter Directly on the board people seem to think that no, it's better to have it programmable Just more chance to screw it up. It's always good thing. Hey Don't want to miss an opportunity really So we have we have this this kind of thing and yeah, there exists the programmable if you don't program them They will don't they will do the wrong thing So they're not necessarily really interrupt controllers It's just that it is extremely convenient to represent them as such We don't really want to invent a new class of you know Interrupt rewriters or something like that. So yeah, we call them, you know, our q-chips just like any other thing And yeah, we call these your key call or stack configuration So even more diagrams. I have worse. Don't worry Really, you won't be laughing for longer So how does it work? But it's fairly similar to to what we've seen so far. It's just that we've we've had to tweak things a tiny bit at the beta structure level so First one thing Remember with a cascaded Scheme we had two RQ desk one per intrap controller because they were effectively two different introps Here because we have this one-to-one relationship We want to make sure that the RQ desk is constant for the whole signal path across All the intrap controllers. That's really important. We don't want to have to Recompute the the Linux in crop number. We don't want to have to allocate another RQ desk It's pretty expensive. There's tons of off state in that that be silly so what we've done is We've taken that RQ data which is Normally embedded in RQ desk and yeah, we made a standalone version. It's the same data structure. I just we can also allocate it separately Another thing we've done is that we've introduced this is parent pointers pointing towards the towards the CPU side and That gives us the representation of a chain An interesting thing to see is if you follow the the pointer We we process the interrupt in in the signal order whilst with a cascaded case We're effectively processing the the interrupt in reverse order. That's a much much nicer way of of doing it So what happens When the interrupt fires here at the top level in truck controller same thing we Find out the interrupt numbers till interrupt number two Resolve it with the RQ domain and Wow, we end up at the top level here at the RQ desk level and from there we're able to walk the chain and for each RQ data we we find we call the corresponding RQ chip and Either we we do some local processing or not and we pass The the control to the next one following the the parent Pointer in the RQ data structure or we can even cut it short if we know that yeah There's no further processing to to be done And that's that's a pretty powerful construct. I'm going to see what we can do with that Yeah, so that's the other the endler the endler is the is in the RQ. It's not what I wanted to do Come back The end up is in the RQ desk That's where it's stored So at that level we are so for example if you want if you're going to mask your your interrupt You've taken them the interrupt you want to mask it You're going to iterate over the RQ data's calling mask at every level if necessary You may you may cut it short and say oh once I've masked it here. I know it can't happen Below there. I'm going to you know return immediately and there's no need for further processing It could with flags it can usually it has the knowledge what it's plugged on so all these Secondary or you know stacked interrupt controllers that they know perfectly what they are stacked upon So they know that for example, I know that this inverter is stacked on top of a gig and Is it will it will? carry on sending them in processing them the The mask operation at the following level so that that's really a way to chain elementary operations at the at the various levels of your interrupt chain So that that that won't impact the way you you actually Perform the handling of of your interrupt that that will still be a call of the the handler that stored over there so yes, so we walk that chain in in in signal Delivery order and and we have this option of carrying on The chain or just cutting it short so in DT That looks almost similar So here we have this This you are for example that has a single interrupt and So we don't have an interrupt parent here, but yeah, we have this one here So we use this seesaw queue, which is the the infamous Inverter so we go here that which this one has as the intro parent the gig Funnily enough if you want to Terminate the the chain of interrupt controllers You have to be a parent of yourself So the gig is its own parents a bit weird. That's how DT works Couple of things Here we don't have any any interrupt specifier which is By design we don't want to use interrupt specifiers for this kind of configuration because Effectively the interrupts that are but the interrupt lines that are used are used by the device itself and not by the Intermediate pieces of IP and using an interruptifier would say oh here. I am using this This interrupt myself so in that case Implicitly this interrupt controller cover covers all the lines Provided by the gig if you have restrictions you can have an adult property and describe a range or discrete lines and Anywhere you see fit That's about it so yes Here we we use the the same interrupt specifier as for the gig that's a That's a feature of them of this interrupt control which has this as decided to use the same The same interrupt specifier could be different That's completely up to whoever rise the DT and the code so there's there's a bit more than Then wired interrupts. Thank God There are MSIs MSIs are really really interesting so An MSI for those of you who have never Did deal with it. It's just a 30 to be tried from a device To another device so from a device that generates an interrupt to a device that is usually in an interrupt controller an MSI controller So the the location where you you write that message It's called the doorbell so It's a device again. It's Usually have a close relationship between the data you write and and The intrap number or the the interrupt vector that is generated on the other side of the of the interrupt controller so you can't it can be For example, if you write if you write 20 as a value you could generate intrap 20 It could generate intrap to 2,000 depending on the on the intrap controller you have whether it's something really simple or really complicated By definition, it is edge-triggered. You can't as if you write something It's it's a one-off heaven. There's there's no notion of holding a line. So it's really similar to an edge-triggered configuration There's I've seen intrap controllers doing level triggers level-triggered MSI's but they end up having a Right to set high and a right to set low. So that's a bit a bit weird So why why MSI's? That's to that's two really good reasons one is When you design an SOC If you have tons of devices on it like people like to have you end up with tons of wires Going all the way from the center to the periphery and routing those signals is a pain You get this this spider web of of intraplanes that you know cross over the buses and it's pain to route So if you if you have Already a database that is present Make some sense to route the intrap data on top of it as well. You already have you know 64 bit worth of wires That might as well use them More importantly even as if you think of DMA your your devices writing a law of memory all of Data to memory and then raises an interrupt If if you don't use the same transport You may end up in a situation where you raise your interrupt whilst the data is still in transit You know in some buffer somewhere and by the time you CPU handles the interrupt the data Still hasn't made it there and you will you read stale data By using the same conduit You effectively guarantee or you should guarantee that By the time you get the interrupt the data has made it to memory It really it guarantees that it pushes the data. That's in front of it Sometimes that's wishful thinking but that's that's what an MSI guarantees But as always we have bugs So historically that's been Tally tied to PCI if you look at the kernel you'll find that Mostly PCI implements MSI in the jail very recently. We didn't have any any support for anything with PCI But he's been present on on all kind of buses including things that are not buses So any device these days? Can can implement Something that looks like an MSI So we've decided at some point. I think it was something like 18 months ago that Would be nice to support MSI in a really really generic way and there's been a lot of of discussion between Thomas Gleisner Jungley who was at Intel at the time and myself To find out how we would structure things And we wanted to support a wide area of what range of of really Sometimes a bit bizarre hardware. Well, that's that's the Intel DMAR which remaps memory but also remaps interrupt The what concerns me the most which is the the gig v3 ITS which is can be seen as a page tables for interrupts Don't laugh really It's really good. It's refund That's something like a free-scale MC bus where they have discrete functions that they can assemble into a Virtual device that pops up on the bus automatically and generates them well things that looked like MSIs Some platform devices USB3 can have MSIs even implemented as a platform device and we have this thing called a mbi-jen Which is a yeah, we'll see later interesting so also one of the Design goals were to be able to Nicely fit with the existing existing a PCI MSI implementation Really don't want the ecstasy's guys to come at me with it with an axe. I'm not prepared for that It turns out that the the hierarchical domains are really really nice for that We managed to implement most of it in core code Not architecture specific not bus specific really really nicely I think And with a just a few front ends that are bus specific And we're going to see why so How does it work? So we've added two method to our Qtip One is method that composes a message. So By composing a message I what I mean is put in that structure and address and And the piece of data. So the address is the doorbell address The data is the what you want what the device needs to write in order to generate a given interrupt So that is completely bus agnostic. Whatever Burst your own You always write this data to that address and we have a second function that It's going to write a Configure a device With that address and that data and that is extremely bus specific I mean if you if you have a PCI device you always write into that MSI bar and nowhere else That's where the thing goes. You have an MSR MSI X bar and you're going to to put that into the MSI X table So that that has to be bus specific and also we have an MSI domain info which we use to Describe things so If you remember indeed with DT we had a way to you know establish links MSRs are completely programmatic. There's no way to describe an MSI in DT because there's they're simply It's a pure software construct. You just say write this piece of data somewhere So we need a way to to express that In code really so what we have here I can find my pointer. Yeah So that's for a PCI setup so we have another Qt chip here that implements MSI and As the right message, we have this standard function that PCI MSI domain write message Spread your views what it does at least I hope And we have this structure MSI domain info Which has a number of of flags this one says yeah, use the default operations for for the domain use the default operation for the for the RQ chip and Yeah, we support MSI X and not only just plain MSI so we can have up to 2k in trucks instead of just 32 And yeah the pointer to that RQ chip And then what we do is let's create that MSI domain here so based on Based on on my hardware domain a hardware domain. I hope I get a Virtual domain that is specialized for MSI. So if your intrap controller is capable of Doing something that looks like MSI you can on top of that be the pure software construct that deals with PCI and On top of the same hardware You can build something that is going to To do platform MSI something that we were not able to do before same thing it's even simpler actually because most of the most of the data store can be given as empty and use the default ops to to fill them in the the framework will Will cater for that and We have the same the same kind of function where on top of the the RQ domain that implements our our core MSI functionality we specialize it so That looks like this Well, that's very similar to what we've seen before except that here We have this glaring hole Because where we had before A stacked intrap controller. Well, we just have a Virtual RQ domain that represents a view of specialization of your MSI hardware for a given bus But otherwise it's exactly the same functionality You will take your intrap go to the RQ desk and for any of the RQ chip operation Do the same dance going through the the various data structure And that makes it really easy to if you build a new bus like that The free-scale bus we've talked about before it's It's about an 80 lines of code to support an entire bus and Provide an MSI functionality as long as you have that that core functionality of being able to Generate an MSI though platform MSI It's a bit bizarre in a way a platform devices I mean, there's no such thing as a standard platform device. They are all different. They like to be different. They're all special we're all special and so there's no real easy way to To write that Function that that is going to program your device to to actually put the doorbell and the data into it As no standardized bar like a like on PCI So the work around for that is that When you when you do create your Your platform your platform device when you attach your your platform device to your MSI controller And you allocate Your MSI is here Why are you going to pass that function? Which you've written Especially for your little device so that for the arm smm uv3 It's just an extract of it. I've removed tons of things. So you you allocate a bunch of MSIs and Tell the kernel. Well, by the way when you when you program the when you when you've allocated everything, please call that function So that I can write the The MSI configuration Into the device and you can say you can see what we compute the doorbell from the message But it's 232 bit words computer 64 bit one And we write the data into some Some configuration register here and then we can hear iterate and And find out what the or Q was was and do the request RQ and all that it's pretty neat But yeah, that really allows for About anything and any any bizarre device you could have You can implement something with that and yeah, and that's that's where she hits the fan really Because the interrupt strikes back and yeah, we thought that yeah now that we have MSIs for everyone Everybody's going to be happy and we can have you know at the periphery of our SOC We can have everything talking MSI Except that now people want to put wired devices there again And we're gone full circle So that's the case of the high-silicon and bi-gen and and a few others apparently so I've heard and So how does it work this? So let me find it Yeah, that's that's this wired MSI bridge. So it takes Wired interrupts as an input and for each of these input it can generate a separate MSI to the interrupt controller Okay, so how do we deal with that? so This this interrupt controller this MSI bridge is is a device a platform device Just like the SMM you have shown before So one thing we can do is Instead of when we allocate these these MSIs instead of you know giving them as a as discrete interrupts What we can do is pack them into an IQ domain way So we can stack that our Q domain on top of the one we already have and And It turns out that that's pretty easy. I mean it's a it's for your hundred lines of code Took us some time to figure it out But once you get the the idea it really falls into places pretty pretty nicely And It just works just like the previous example you you take that interrupt you go the way you pass your your parent data just Yeah, I know what to say. I don't even know why I wrote that slide. It's so easy So the important thing is here you need to have something that is going to have and I and I did the mapping between the wired interrupts and The MSIs are going to be generated and so the code is for the the MBI gen is a Bit convoluted but not that much actually I have to read it from the bottom as usual so We create That this platform MSI create device domain So it creates a domain for this device this number of pins Pass that function that programs the MBI gen for MSIs Takes a number of operations Which are pretty generic which I haven't even bothered copying and Some some platform data Just that that's going to be passed here if you look at the The alloc so we have this this allocation structure this allocation function. Sorry Which translate the The the firmware spec that comes from the device tree because all of a sudden we're back into Device tree land We thought oh, yeah, we now that we have MSI we completely abstracted from firmware But all of a sudden these devices that are connected to with a physical wire To the to the MBI gen need to be described in the device tree. So here we go We translate them we allocate the The the MSIs for that do a bit of processing and you know treat that as a as a normal interrupt we wire it with this The data that's going to be used the MBI RQ chip and Yeah, that's a normal in chop controller. It's really a bit of a schizophrenic Piece of kit on one side. It's a It's an MSI client on the other side. It's a it's an interrupt controller There are some drawbacks as well because as something that still doesn't work properly on Linux, which is Dependencies probing order for this for this kind of thing It's likely that you'll have to to do some probe differ Absolutely everywhere if you want to guarantee that things are going to be probed In the right order But yeah, one problem at a time as I suppose Yep, so the the message So the the message is actually passed to that function here So which I haven't copied here, but if we go back to this As the equivalent of that function which take a message as a parameter So when you've requested your your MSIs you've asked them The core code to allocate a number of MSIs For each of those you allocate MVAC MSIs for each of those you're going to call that function here and program the The door the doorbell and the data into your device Yes, so we are we have so there's another Data structure, which I haven't described here, which is MSI desk Which is which is associated to an RQ data which RQ data and Contains the that message as well That creates the link between the the Linux RQ number the device and the message itself But yeah, if I couldn't put all that on the single on a single slide so As I'm about to run out of time I wanted to show you why it looks like this kind of thing on an M64 so We usually start with a on a modern system. We have a gig V3 Which is on its own pretty simple And then usually we have the ITS so all that are just RQ domains. I haven't bothered Showing the devices themselves All the various linkages. So that's the one that does page tables for interrupts and We have both platform MSI's and PCI MSI's and Yeah, it's not uncommon to see this this MSI Wired to MSI bridges as well And now we if we start adding devices. So we have a bunch of devices Wired some platform MSI stuff. We have a PCI root complex that does either legacy interrupts or MSI's and The worst part is this is that you can replicate it as much as you want You know, it's not uncommon to have several instances of all that, you know, that's a That's that could be a socket or actually a An IO subsystem in a socket Which is not uncommon at all and that's where I stopped because you could carry on like this for a long time, but yeah That's about it. Any question? Yeah Thank you. Yeah. Yeah, that's so that's all in mainline if it wasn't mainline. I wouldn't be talking about it Oh, you were pretty quick Yes, so The trouble is one of the issues is that actually RQ domain is a is a pretty old subsystem He's been written by Ben Hansch mit about 10 years ago for power PC And it has ever of the lot One of the issues that short of rewriting the whole of the power PC code Which nobody wanted to tackle so far we've carried on a compatibility interface Which has some drawbacks doesn't allow us to express everything. So what we have is some sort of compatibility layer and What we now consider as being the new core RQ domain, but yes It's it's a complicated mess and I didn't want to go into how it's used Internally because that's quite complicated But really the goal of that talk was to highlight How we can compose things I agree with you the there's a big effort to be made on documentation and and probably Get rid of the old stuff So if anybody is power PC savey and once you have a go, please come and talk to us I'd be much appreciated Anything else? No, well in this case. Oh, thank you very much