 Thank you a lot. Oh, thanks a lot. I guess I'll start with why I came up with the idea for this talk. I'm kind of since a few years, the DRM I915 into graphics, Cernelimentena. And graphics is a bit special because every driver, every GPU has their own private interface. dearlight caching, submission of GPU commands to the whole thing and all these things. So we don't just have one generic interface, we have one interface for each driver. So we have lots and lots of cheats. What happened? I'll pull that switch up. Yeah, doesn't it good for my quality serie of graphics. Anyway, yeah, don't, like, take pictures of that. Oh, so we have lots and lots of IOC, all written by kind of driver maintainers like me and people with clue. So, and the other thing is also cheaper, it's like change all the time. Every two years everything changes completely and we need to throw out stuff, make new stuff. Of course the problem is with user space interfaces, you're never going to get rid of them. So, like, if you put it up, you're going to live with these mistakes for like 10 years at least, something like that. And I've also spent a lot of time kind of cleaning up ISHL code and legacy code and trying to fix all mistakes. So, that's essentially what I'm coming from and I noticed when people talk about IOC, they're often discussing things as you absolutely need to do that. The IOC just not even really look at the pageant as I read you checked it. So, figured, we seem to have quite a good, oh no, that wasn't me. Oh, we seem to have quite strong opinions and best practices and how to do user space interfaces simply because we have so much experience in getting it wrong. So, that's what my talk is going to be about. Is it going to, did it just phrase? That's kind of awkward. Well, that's better. So, produce all the things I'm going to talk to you about. Now, we don't screw it up and kind of have the bone marks to prove it. So, it's going to be three points. The first is the basics. Like, if you don't do this, if you get that wrong, you might as well forget it and never use the shiny new IOC number that you've assigned again because it's just hopeless. Second point is about technicalities. That's usually what people bring up when they talk about getting IOC tools, right? That's stuff like padding your structures so that you don't end up writing compact layers and things like that, which is a bit embarrassing. And at the end, I'm going to talk about a bunch of special topics. I expect that's kind of in the really opinionated stuff is going to come in. So, I mean, this is primarily about experiences from graphics drivers and maintainers like me and Dave, much more. But if you have, like, other war stories that kind of contradict things, I'm kind of trying to show, hey, are very much welcome to input. We can have a bit of discussion. Oh, if you don't agree with things, or anything, please raise the questions. So, let's get started. The basics first is, like, do you even need an IOC tool? Because, yeah, I just kind of the easiest choice. You just have a device drive and you do that, and mostly you can sneak it under the eyes of kind of the senior kernel hackers and trying to get something in that no one really reviewed. I mean, one example is, hey, the KD boost stuff, the review is like, why should you just do a proper system core? So, that's kind of one question. The other is, often with device drivers, if you do an IOC tool, why don't you just, like, use read write and pull and all that stuff on the file descriptor? So, I mean, it's all there. You can all use that. And very often these operations make a lot of sense. I'm going to talk about file descriptors and stuff some later on in the special case topics. There's also, like, lots and lots of other interfaces that the kernel has, which I know, like, system calls or anything like that. This, yeah, this is a first configuration, especially a debug interface. If you have something that only your task suite needs that no one else needs, maybe put it there to kind of hide it better so that people with production systems never ever see it, which is nice because that means they can also never, ever complain to you about the regression or breakage. And the other thing is these lots and lots of subsystems that kind of doing, well, not maybe not a perfect job, but these lots of encoded experience about lessons learned and things, so reuse them. For example, in the GPUs, we're currently working on the intergraphics driver to to expose performance counters to user space. And of course, we could do an IOC tool and reinvent everyone, bring both and do all these stupid things and learn the lessons again. Or we could just reuse both. Of course, that means we get to deal with Thomas Glecksner a bit, but, yeah, long term that might be better. So that's kind of the first thing you really should be asking, like, what's the right interface type for the thing you want to expose to user space? The other really basic thing you absolutely need to have is a real-world user space implementation, a user of your interface. A demo is just not going to cut it. And I mean by real-world, I mean, I need to be tested. I need to be reviewed by whatever project you want to use it. For example, if you have some great new idea that's going to make databases faster, have a fully tested reviewed and ready for merging patch for postcressor. Oh, for graphics, obviously. Or the requirement is that the entire stack is implemented. So we don't just want the kernel side, we want the interface shim and lib DRM. We require the OpenGL feature, I'm sorry, the OpenGL feature implementation in the Mesa library. And we require all the piglet OpenGL test cases to really make sure your interface actually survives the real world and doesn't like fall apart once you have to deal with all the corner cases, with the error handling. I think a great example that Mike Kerrysg always brings up when he's discussing interfaces is all the file-notify APIs, which all can have been designed with demos. And then people try to actually use them and implement them. This is where you can recover in any other way by throwing away re-scanning the entire file system, which is not so great. So, really, a Mesa v production code, you need to have it. That's the only way to remotely make sure that the semantics of your interface are correct. I mean, if you don't get this right, you can throw away your article. If you get one of the technicalities wrong that I'm going to talk about later on, there's no problem. You just make a version two and then implement version one on top of version two, which is doable as long as the semantics are kind of the same. And that's why you need to have this. But this is a very big thing. Always merge the kernel patches first. Because there's a good chance that some random reviewer will spot some minor data and your interface is going to slightly change. If you have shipping user space out there that's using your interface, the old version of the interface, it can develop your script. Because you can just throw away that IOC and never use it again because the existing user space that's shipped out there will not work with the version that's actually merged into the kernel. So, always merge the kernel patch first and then merge the user space. Of course. That's actually such a hard and fast rule that they fairly broke compilation of a bunch of user space components when people violated that rule. So, that's really... We're not going to discuss about that in the graphics world. Now is the button. The next basic thing is task cases. So, you have your real-world user space thing and it kind of does at least beyond reasonable, they'll prove that your interface is not totally broken. But it's not going to test all the other things. And all the other things are usually all those bits and pieces that are going to score your CPU or some other disaster. So, you really need to have task cases. You need to have lots of test cases for all the corner cases and the things that people use to break into your kernel and all the things you can test with a functional level. That's pretty much going to be a running gag in my task cases. So, this... I'm going to bring up lots of examples where I think you really need to test this and that. So, we're going to be more specific about this later on. So, that's kind of the basics. They're going to make, hopefully, sure that you use a shiny new system called a Ryan's SQL or whatever, or the new interface is solvable. But these are technicalities and it's kind of nice to get them right because if you don't get them right, you end up writing a compact layer and that's just boring, busy work and it's also somewhat hilarious for everyone else. They're going to watch you do that. So, the rules there is, if you do a struct that's part of your rabie line, only use the four special type types that the kernel has, the double underscore S32, U32, S64 and U64. I mean, the reason for these special rules is that the kernel is kind of older than C99, so are the U and underscore T kind of types in no go. I mean, nowadays C99 is kind of old, so you're not actually going to be all that successful in actually breaking user space and making it fail to compile if you'd use the C99. Yeah, that's a nice to, especially if you kind of screwed up and you want to know where the holes are that you actually put in. PA hole is really nice too. Oh, thanks. So, but the thing is, it's still really nice and useful to use these types because if you're kind of running around the C-scope in your kernel files, at least me, and I usually don't notice that I'm in a user space header file, but if I see these types, I realize, okay, this is ABI, I'd better be very careful when I touch this. So it's still, I think it's still really useful to just use these types. Then, like Dave just said, this is an anti-topic with padding and alignment, so if you just use 32-bit integers, you're going to be fine, but if you use a 64-bit one, align everything to 64-bit and pad manually, and then you can use PA hole or something like that to check whether you didn't actually accidentally put a hole in there. So, both align all the members and the size of the structure because eventually someone is going to do an array of these and things like that are going to use size of, for example, a size of in the iOS CTL, which the size of the structure is encoded in the iOS CTL number, so that's going to slightly change. And the next one is pointys or 64-bit on-site, shocking to know. And finally, if you did screw up, just use attribute packed or manually pad it up to whatever kind of alignment you do have and then take out all the humility you have and write your compact layer. So, with that, I mean that's kind of the boring stuff. You can pretty much, if you google for iOS CTL design, there's parts and parts of explanations of all that. The next technicality is input validation, and there's kind of two reasons why you need to be really, really careful about this. The first thing is if you allow users space to supply random stack which are registered garbage, that's going to break your extendability, because essentially that field or that register or whatever this is not useful, because at least on all kernels, if you didn't check for garbage in these values, anything was valid input in there. So, you can't decide whether the application accidentally wanted to have the new feature or whether it just accidentally set this value. So, that's one thing, always check all your input, because otherwise it can't extend your interface. The other thing I saw, if you don't check all your input, some people will try to figure out whether they can put something evil inside these values that kind of pipe passes you checks. And then, like I said, it scores you a nice CD, which is again not too great. So, what you actually should be checking in addition is all kinds of overflows, like array sizes, like adding two numbers, and all these stupid things that see an emotional influence upon us. So, any time you multiply out or do generally anything with values supplied by user space, you have to double, to check that you don't know anything else like that. And the other kind of important thing is, and that's especially important if you have like a big complicated interface that has grown over time and has grown lots of different modes and combinations, check all the invalid combinations of flags and values. So, if you don't set that flag and then the ratio be zero size, check for that. Because if you're unlucky, you can't assume in your code that no one is going to give you input data that doesn't make sense. So, your code implicitly assumes that if that is like, that flag is not set, the ratio is zero and suddenly you have like yet another explorer at hand. So, especially take good care with all combinations and of course, half test cases for everything. Because at least in my experience, it's generally really hard to review code that does input validation because you don't see all the checks that are missing. And those are the importance. Whereas, if you write the test case, you have a different mindset. You look at the entire input space and just go through all the combinations and make sure, okay, this should be rejected. This one should be accepted and resolved and blown or something else. And so you can go through all of them and make sure that you have a test for all of them and make sure that it either gets rejected with the error code you can expect or it does what you expect it to do. So, that's one of the major reasons why you need to have a special test suite on top of your real world user space thing that hopefully also has a test suite to make sure that the functionality at the high level is correct. So, that's... Now, this is a special case with flags, with input validation. I think, nowadays, it's pretty much common knowledge that you should have a flags parameter. Even the kernel is full of version 2, 3, 4 ISATLs just to add the flag parameter and system calls and things like that. But the thing is, so you added that flag parameter, you're really proud that you didn't screw up and unfortunately you forgot to check that it's zero and reject that with the email. Someone is going to give you stack error which in top flags parameter is totally useless. So, really make sure that you do check for flags equals zero and reject that with the email because, yeah, otherwise you're just going to have version 2 of your interface. That is not set by the user space. Yeah, exactly. Check for flags equals equals zero. Exactly. Because, yeah, then you have user space that gives you garbage. Someone will get that wrong. It just happened. And, yeah, then the flags parameter is totally useless because you can't differentiate between garbage and intentional garbage. And have a test case for it because you're going to forget about this. And the other thing with flags and that's kind of ties back to the thing about combinations and invalid combinations on the previous slide. Really check for all the invalid flag parameters because it's super easy to squeeze something in there that you can assume didn't work. For example, you have a bunch of enums in your flags and you just have five but three bits kind of gives you eight values and unfortunately the three other values cause problems and things like that. So, really check for all the flags. It's super easy to squeeze our interesting books into flag parameters if you don't properly validate them. That's kind of another technicality. I mean, the entire point of flags parameters is to have some means to be able to extend the interface without breaking old user space. And that's something you need to think about anyway is compatibility in both ways so that old user space and new kernels and new user space in all corners both kind of work. So, one thing is, like I said, merge the kernel stuff before you merge your user space stuff. Unfortunately, because otherwise you're going to end up with interfaces that are kind of not compatible. I told you already last before you really started. So, one trick that they use in the DRM system quite often is hide a big new interface between a module option that kind of tains the kernel and needs to be explicitly set and kind of looks dangerous. The thing is, you can kind of stretch your window by one to two kernel releases because by that time user space is all over the place and essentially already using your interface, it can't change it anymore. So, for actual, what once you kind of have your interface and it's frozen down and locked down, for actual backwards and forwards compatibility it's like, yeah, have a flags parameter every day for everything. The other thing is have, there's kind of some other, I mean, sometimes the flags parameter is kind of not good enough because by the time user space wants to do your system call or IRCTL it's kind of too light already. For example, with GPUs you you have to construct this massive command stream and you kind of need to decide why you construct that command stream which features you can use. So, by the time you do the actual command submission to the kernel it's already way too light if the kernel then tells you, oh sorry, I'm a too old version, I can't do this for you. So, you kind of need driver capability flags, that's what the I915 driver does. The RAID and driver just has like interface revision that they increment every time they add something. I mean, there's kind of, both approaches have their pros and cons. The driver capability thing does lead you to maybe believe that you could disable something without the other. But, fact, there's kind of that user space just assumes that when one capability A has been released before capability B then when they check for B, A is always going to be erect. So, it's a bit dangerous. One thing we also do a lot in DRM is user space opt in flags and the user space has like, okay, I'm new, I understand these new concepts. One example is a recent introduction of universal planes. I mean, to the display side of DRM had like the concept of the primary plane which is just like your desktop and the cursor moving around. And then people added additional overlay planes for video and compositing and data and for example. And then kind of realised, well the primary plane and the cursor, but that's just normal planes. So, why don't we expose, just expose all the planes as plane objects. But of course, old user space within C, the primary plane and the primary plane object and think there's two different things. So, for that case, we have an opt in flag. The user space says, I understand this. Please give me the full list of objects and I'm not going to use like, both the legacy primary plane and the new plane primary plane object because I know it's actually the same thing. And so there's lots of cases where user space has to explicitly opt into the new behaviour. So, that's another way. One thing and that's probably a bit legalist, lawyer trick is, I mean the point here is to avoid regressions and all that. And it's only a regression if this is an actual book report from a user. So, if you can somehow trick your user into not realising that you just broken this system, you can get away with it. And we've actually done that. In the 1915 driver, we have the big break is kind of kernel mode setting and user space mode setting. And all the code for user space mode setting is horrible. That was written a long time ago. It's full of security holes and horrible things and things are wasted for user space to crash. So, we kind of want to get rid of that. Unfortunately, we've only switched to kernel mode setting like five years ago, which is way too new for just throwing the thing out. But the X server transparently falls back to the Vesa driver if the real driver doesn't work. So, what we do? We load the module. We do not load the driver. That's why users don't realise that the driver didn't load because Alice Mart store shows I915. And then the X tries to load the Intel driver. That doesn't load because the driver, the kernel interface isn't there, falls back to Vesa. And we get all the nice features of that. And the only real regression is that it's no longer accelerated. But, in fact, these chips, these old chips where people kind of stuck on these old versions are so slow that, well, CPU rendering is actually forced. So, we got away with breaking user space as long as no one notices. I mean, other examples is like just ship new versions of your user space tooling that kind of used a new interface, then wait for a few years for the distributions to upgrade to the new toolings. And then maybe you get lucky and don't break all the scripts. Of course, there's interfaces where this is impossible, like system calls used by Lipsy. Essentially, you have to keep them around until the heat depth of the universe. But still, it's only a regression and the only broken compatibility if someone actually notices. And that does help quite a few times. One last thing, that's engine-ness. It's terrible. I mean, in the GP of it, some people do actually care about that because raven and ditch ship and power PC machines, I brace all the time. So, the hardware is really nice. It has all the bits where you can tell it, well, my command stream isn't big Indian, but the texture is over there in little. And then you have all these bits to fiddle. And so, it should be. But the matter of fact is the build is little Indian. And so, there's like two users on power PC or something like that who care about this. And occasionally, like every few years, report box and then someone tries to fix it up. So, and the same thing is kind of network is little Indian and all that stuff. So, in almost all cases, you can just say we're in a little Indian world and mostly get away with it. But if anything that you pass around between kernel and user space kind of goes over the network or lands on disk or comes from this, like on graphics where you have textures or video streams and stuff that you load from desk, it might be very to think like the two seconds about this. So, that's with the technicalities. So, let's look at a few other topics. One is resources, all kinds like allocating memory or synchronisation objects. And the thing is attach everything to your struct file because when the process dies, all the files get closed and you can release all the things and not leak stuff. The other thing is do you consider standardised file types? So, if you want to share or expose a memory object and maybe want to allow user space to pass that to some other driver, use dmabelf. We don't reinvent the world and there's also fences which are generally just Android only for synchronisation. Another nice new thing is, for example, MFD, that's just basing around memory blocks. So, that's the other thing. The really important bit is every time you have an interface that opens a file descriptor, like allocate a dmabelf object or something, have a flack parameter for close and exec semantics because, I mean, honestly, I didn't really understand for a long time why you would need this. It kind of didn't make sense. The reason for that is in multi-threaded applications, there might be one thread using your shiny iOS ATO, while the other thread isn't some random library which does fork and exec some helper script tooling. So, if that thing opens, if your library opens a file descriptor and not have to close an exec call and the other thread does a fork which means your file descriptor gets duplicated and then an exec, you leak dot file descriptor into that binary and creative people have used that to exploit things. Because maybe that file is running in a different security context, well, that executable, or things like that. So, you really want to have a oclo exec flag to close that gap. The next thing is about sharing resources, topics like that. I mean, file descriptor is all nice because you can just pass around them on Unix, the main sarchins, KD bus will support passing file descriptors around. It's all really great. Unfortunately, there's some things where you have so many objects that file descriptors just kind of don't scale up, at least not by default. One example is user space GPU buffer objects where some desktop environments easily have a few thousands of them. So, for these special cases, it's kind of okay to have an ideal lookup table on your device file or something like that. But yeah, like I said, don't do it like DRM and reinvent your own resource sharing and passing because it's kind of annoying. You're most likely not going to think about security all that much, so you make it there. Everyone can get at the shared object kind of thing, which is not so great. Another thing is if you actually have a use case of passing around objects, think about uniqueness requirements. Does your user space on the other side need to know whether the thing it gets is the same thing it might have gotten from the other side? I know this is important for GPUs because the command stream validation does not allow back references. So, if you give it the same buffer twice, it's going to tell you, no, I can't check this and validate this. So, user space needs to know whether it got the same buffer twice. But something like the Wayland RX server, I can't really trust it's client all that much. The client might say, I'll have you buy your V-Buffer here with three planes, but I'm going to give you the same buffer object for all three planes. So, it needs to know on input time that it's the same buffer object. Kind of the nice way to do that would be the F-stat. Unfortunately, that means you get to implement the full virtual file system, which is a bit much work. So, what do we actually do in the DRM world? That's also a different problem. I mean that the sealed memory thing is so that the receiver can be assured that you either don't change the memory anymore, which simplifies the input validation thing, because then it doesn't need to copy. It could just check in place without the other side kind of being sneaky and changing things. And the other thing is you can also see like just the size of your shared memory block so that if you memory map it, you can have an assurance that the other side doesn't just shrink the file and you then get the sick buzz. So, different problem. But, yeah, what we in DRM is just when you input and it's the same object, we give you back the same integer ID in your local namespace and that way user space can figure out that it's the same thing. The other thing is about revoke if you have kind of any global resources that are not shareable, like keep your memory or things like that, think about revoke so that you can have like multi-user setups. DRM has got that because of some historical accidents with multi-user, which for example loginD can execute but not your X server. And if you switch from one X server to the other X server, the first X server does not want to give up the display hardware, loginD just takes it away. So, that is a very special case. The other thing is also if you actually have shareable objects, like you cheap you memory address space, think about properly isolating different clients. The thing there is the memory mapping support in DRM is broken like that because you don't want to do a memory map operation from an ISCTL because then background won't understand it, which is a bit annoying. So, you want to do a memory map unlike the device node file, but we've forgotten to restrict memory maps from all the clients for boffers it doesn't there. So, be careful there. The next thing is slightly hilarious, it's signals. I mean everyone hates them, at least I do. Unfortunately this is your next and there's no way to avoid them. Eight and no one is going to use your system called a RIO CTL or some will use them and just lost signals. So, you do need to have an answer. Unfortunately, the Mampage is completely useless, it says something about slow devices can return E-inter and the question is what's a slow device? Essentially, it's the other way around. If you ever see an E-inter, you have a slow device. So, the Mampage is totally useless. Yeah, it's like and the thing is like maybe the next scale version suddenly started the maintainer decided well maybe we should make processes at least killable and suddenly your device changed from fast to slow. The Mampage says it can sleep forever. Sleeping is kind of loud and forever is hard to prove and not prove. People are usually not like. One solution is just require that you use the space handles E-inter correctly in all cases. That's just not going to work. The other one is just don't ever break your blocking sleeps anyway. I mean, both cases are solutions of the kind of I'd like to have a pony. Because users are going to really be pissed if they can't interrupt and stop your applications. So, that's not going to work. The next one is killable weights, which is kind of nice because you only need to care about when recovering from an interrupt when your process actually dies. But if you do it now, you'll see that it's a false script and someone else might have a tube from that false script. So, if the process dies, the thing actually doesn't disappear. And that duping thing is really common. For example, log-n-d has it for the revoke. The x-server has it. Sorry. Oh, this was fine. I thought that was it. Anyway, the x-server has it to share it with clients. And all these things. So, that's kind of not that great a solution. And the other thing is also really hard to test. Because you only can test all that recovery code by killing a process. It's really hard to figure out that anything worked correctly. The driver will fall over the next time around you use it. So, the solution we pretty much use in 9-on-15 is to stop varying and blow off signals. The thing is, once you decide that you have to be able to restart all your things, it makes testing your error recovery code really easy. Because you have all these functional test cases, all these test cases for corner cases. And the way you check error handling code is you just add new copy-pasted sub-tests of all these test cases. The second process bombards the first process that does the actual testing all the time. And if that's not enough test coverage, you just add more killable, weightable sleep, interruptable sleeps, or manually inject interrupt points. And yeah, the nice thing is that e-injury is an error code, but in definition it's a recoverable one. So we still have all the correctness guarantees. You can run any kind of application constantly, interrupt them, and if those things work, it gives you reasonable assurance that your error code is not totally screwed up. Which is nice, because that's one of the same evil people like the exploit. So summary, my recommendation is pre-opinionated support fold restart. Just do it. Ah, the other thing is, because the application of writers will still get it, all wrong, have for your subsystem driver whatever one function subsystem IOCTO, which does the restarting for you unconventionally. So that no one ever gets an e-injury and just falls over with white segmentation violation or something else. Because the thing is, the book reposer will come to you because it's uLibrary, not the other programmes, yeah, uLibrary functions are in the back of the trace. Then exploit e-interhandling for testing, because it's really nice. Or just don't do any like sleep and weights and do it all with fallable file descriptors. That's kind of one way to do stability. So that's the thing on signals. Do you have any questions? I'm running a bit late overall, otherwise I guess I can throw in a few more things. What is a bad time? The thing is, time is relative. And you realise that really quickly a senior start to grab time stamps from various hardware blocks because they're just not synchronised. So, firstly, make it really explicit to user space and then you can interface thing. Which kind of time source you're using? If you do anything on the CPU, just grab a clock monotonic because that's what DRM also and video for Linux are using. So you have a good chance to actually match with the other subsystems and drivers. If you're sampling or harvey clocks, give user space some way to synchronously grab a time stamp so that its own tracing and blocking can use the same clock and you don't have like the effect that the CPU size is kind of not synchronised with what the GPU does or your device driver does. We started rendering here and then completed here, but we kind of received the bytes here before the rendering actually. That doesn't make sense and kind of confuses people. The other thing is, well, use 64-bit seconds and 64-bit nanoseconds because that thing called 32-bit overflow that happens in 2038, it's kind of a thing. And again, be really paranoid about input validation to make sure that no one likes the stupid tricks with overflowing one side and then you thought you had a positive time out, but actually kind of a bit go around and produce seconds or something insane thing like that. So that's kind of on some, do you have a question? The other thing is waiting. Like I said, just use polar polarities. There's some people who've thought really hard about how to make that scale and work well. The other thing is support absolute timeouts because if you have this restarting behaviour and you need your own weight aioctol and the thing is every time you restart, if you have a relative timeout, it starts to skew. So just do absolute timeouts and if you get a relative timeout because you screwed up your interface and extended it later on with a flux, just convert everything in an absolute timeout. There's some fundamentalist apparently who think this is really horrible because you're supposed to only have the timeout right around the weight, but the thing is user space is not going to be able to tell the difference. I mean again, if no one reports the bucket, it's not one. So if you do the absolute timeout, it kind of includes all the setup and tear down and whatever CPU time you waste around, it doesn't matter because you're still going to sleep as long as, at least as long as kind of your absolute timeout. I'm over the time actually, a little bit at least. So documentation that's kind of an off the thought, I really think you should have test cases because they give you an executable specification and that's so much better than anything written because people just disagree about what English means, where CPUs are pretty strict. One thing is if you have a really generic IOC tool or maybe even a system call to consider to write a man page, you're going to at least make Michael Kerrysg's life a bit easier. The other thing is there's tons and tons of documentation, something ABI and honestly, I don't think anyone really uses it that much. I tend to completely ignore it. There is some subsystem cases. The maintainers insist that you update the documentation in there, but I mean to submit the patch, get yelled at and fix it up because yeah, people tend to ignore documentation. So summary, only two minutes over the time, have a real-world user because otherwise no one is going to check whether your semantics are actually correct. Then have test cases for everything, for all your Conan cases, for all your error handling using interrupts and IOC TL restarting and all these things. Don't screw up the technologies too bad. On the other hand, it's not a disaster if you do so. In this case, people will laugh at you and you get to write a compact layer. Finally, think a bit about documentation and maybe decide not to. So, that's pretty much the lessons learned from writing graphics driver and totally getting it wrong. Thank you for listening. I guess we don't have