 Well, I start okay great. Yeah. Hello everybody. Good afternoon. This is Adam gear. I'm Fiona Traja We're from Intel. We work in Shannon on the west coast of Ireland and we write a device driver for a hardware device an accelerator device for the Linux kernel So we're gonna talk a little bit about what we learned as we've done that So I'm gonna give you a bit of background on the quickest is just device itself That's the hardware device that we write the driver for Then I'm going to talk a little bit about entry versus out-of-tree driver And our journey in getting from one to the other And then some things we did in terms of reviewing how we design stuff how we analyze implement and validate and Some threat modeling we did And then negative testing Okay, so just maybe a little bit of background first on the whole concept of acceleration So if you've got a computer intensive workload, and it's running on your CPU on the left-hand side here It might be fully utilized in your CPU 100% no cycles available for anything else And if you design some special hardware as we've got in the quickest device to do compression for example or cryptography These things are quite compute intensive Then you can offload the process from the CPU and that frees up the cycles on your CPU So your CPU then can do other things So it'll allow you to either complete your workload faster or to have your workload taken less of the cycles of the CPU during the The time it's running and that makes it available for something else. So that's the point I guess is that for acceleration you've got some compute intensive Workload and by design and the silicon and the firmware it can be optimized and how that's done So the quickest device itself that we have as I said it optimizes cryptography and Data compression and for cryptography. We're talking about two different types. There's symmetric crypto We usually call that bulk crypto Cyphers like AES and and Shaz and so we do the authentication and and Cyphering and then there's public key. So asymmetric cryptography There's a bunch of engines in the hardware and they do digital signature There's elliptic curve. There's some of the basic primitives like point multiply point Inverse inverse and multiply sorry for that in the name of that one So there again, they're compute intensive on the CPU and then the third area is data compression compression decompression and there's some Check sums that go with those as well So so far we've talked about the hardware and this has just shown what software modules we need to do to enable that hardware The focus today is on the the green linux kernel module, which is over here Yeah, but if we start at the bottom of the diagram just to give you the bigger picture We have our device at the bottom and the device has Firmware that we need to download to the device Then there's a bunch of stuff in the linux kernel that we use to enable the device may need the kernel driver itself It plugs then into the linux kernel crypto framework and through that crypto framework There's symmetric and a symmetric crypto can be accessed by either other processes in the kernel itself or by Processes in user space over here to the left that are kind of grayed out that they can come through actually They might be not the right ones. There's ones that can come through. Yeah, and into the AFL APIs into the linux kernel framework and then into the device driver and the device driver will mediate The descriptors and build the descriptors to send the data off to the hardware to do the work we have a second path so on the the the kernel driver Enables SRIV in the device at well SRV is built into the device, but it enables SRIV and that enables virtual functions so the physical function is what the kernel driver deals with itself and and Offloads the data to that physical function interface but in the user space side of things we want to enable virtual functions and they're smaller or more and It allows you to scale the device up so that you're going to have more processes in user space each having an independent Seeing it as an independent device so so that that's enabled up through some through vfio For security and then the vfio enables the CSRs from the device up into user space and in user space These we've got a set of these blue libraries. These are a set of libraries be exposed in github Called quad lib and then above quad lib There's different applications that can use The APIs that we provide the quickest APIs and they can do compression or crypto And just to complete the picture in this diagram some of those user space applications that we have like quad engine and quad zip They can also upload to a software Software libraries that run the same type of workload on the chip itself on the on the processor in that case It wouldn't be offloading to the hardware But it's just an alternative path if you want to run the same workload on your software that on your CPU Okay, so we've talked a little bit about the whole stack So now just the purpose of this talk is just to talk about the the kernel driver itself So this is a more detailed picture of the kernel driver and the kernel driver has a bunch of different work it needs to do and and different Components that it interacts with so when it wants to Those areas are on the left-hand side here. We've got some ccfs interfaces This is where we do some basic configuration allow the user space Process or person human person to read what's how the device is configured and also to reconfigure the device We also have an interface to the PCIe Framework because it's a PCI device and to handle errors that the PCIe framework would detect We register some callbacks and those callbacks get called if a PCIe error is discovered Then another job of the kernel driver is to download firmware to the device And so there's some CSRs there in the the bar That it uses to download that firmware Then this path is quite an interesting one It goes all the way from the quickest driver in the kernel to Whatever is used in the VF and that could be something up here in user space using the VF The virtual function or it could be a kernel function as well that could use the VFs at the moment We don't expose it We have some kernel drivers for VFs, but we don't do them for our latest one But mostly it's user space access that wants to use that VF path Then we have the data path. So on the data path here The the data path is is In that case what we have is a lot of data usually because if you're going to encrypt data or compress it You often have a large amount of data So that's not passed by the kernel driver directly to the device What happens is it's put into a place in memory in DRAM and then the chip will you will DMA it in from that point in memory and There's two different types of things that are DMA'd in and out of the device one is the data itself that needs to be worked on and the second is some descriptors and the descriptors are built by the Driver to tell the chip what it needs to do So both of those are in DRAM and both of them are DMA'd into the device and the way the kernel driver Tells the chip that it needs to do work is is this doorbell, which is effectively just a right to an into a CSR So you write your register and then the chip will Read in the descriptor over DMA and then we'll read out any associated data work on it and put it back out again with the response into DRAM And so this data path I guess what's interesting about that is So it's used in the case of any process that's coming through LKCF and wanting to encrypt or decrypt data The quickest driver itself will build the descriptors and and send them down to the hardware in the case of user space This isn't the highest performing path So there's a if you've got a user space process and it wants to do this kind of encryption And it hands over to the kernel driver and the kernel driver has to pass the data on you've got a change of context So in the user space processes, we have a separate path which is exposed via VFI OPCI And that's that provides a direct path to the hardware So the VFI OPCI enables the CSRs that are in the kernel driver here sorry in the device itself in a VF to be visible in user space and And once they're visible then the process in current in user space can actually directly access it on the data path So there's no there's nothing in between to mediate the the data or to build the descriptors It's the user space so the user space process itself that builds the descriptors We'll come back to that data because it's an important point. So just the name of the path we have So that's that sorry, that's that's the data path and I've covered the other paths if you have comms and the other paths Okay, so we've talked so far about the hardware the stack itself and then What the kernel driver needs to do? So just a bit of background on the driver itself So we've always had kernel drivers and this family of devices has been here for about 10 years Maybe 11 or 12 actually since the first one since generation one But around 2014 we put the first driver into the Linux kernel And at that point we had what we call our out-of-tree package and that out-of-tree package is something which has firmware kernel driver user space driver sample code and Installation scripts all wrapped up in one package and you have to go to an Intel website to download it That's great for a lot of our customers. That's what they wanted in the past and it was much more targeting comms customers Building networking devices often appliances who have full control of their system But we recognize the value of entry of being in the Linux kernel And so we also started to move into the entry Into the kernel in around 2014 and then with the second generation devices 2015 And we base that off our out-of-tree driver and we read you know change We had to change coding standards because they're obviously different in the Linux kernel Then we would have necessarily had and there was some other things we changed like configuration But basically it was kind of similar to our out-of-tree driver Then in 2020 we have the next generation device, which is going to be released later this year It's not released yet, but we are getting ahead of it with the software and getting the software into the Linux kernel And that Driver that device is targeting a broader market. So it's a more cloud-ready device. It's targeting So that the the previous devices were targeting trusted environment So we expected that whichever process built the descriptors be that in the kernel driver or a user space process That those would need to be trusted. They would they would need to Send not not send malicious data to the device because the device wasn't designed to handle that use case But when we you know from about 2020 onwards and in our next generation We're targeting a cloud-ready device and that cloud-ready device means that you can't trust What's the process running in the VF because it could be anyone You know, you don't know who's going to get your your device in the cloud. So the silicon and the firmware were redesigned to to cope with untrusted applications and So we then had to take a take a step back and look at our entry driver and say, okay Is that also suitable for this tout ready? Marketplace and so we kind of re-reviewed how we did things we didn't change everything we did But we certainly made some changes and we just with that hat on of thinking. Okay. This is now Potentially untrusted process is the new space that are accessed in the device. What do we need to change in the driver? So I think I've probably covered all that here Yeah mentioned so in kernel space of the aval users or that the kernel driver mediates the To the device so it actually builds the descriptors whereas in user space It's the user space process that we now can't necessarily trust is building them So we we looked at our our whole stack design really not just the kernel driver But from usability extensibility robustness and then security Okay, so some of those things you might think well They're still valid for an out-of-treed driver. So, you know, are the customers have the same end requirement They want to be secure and robust So what why why is it something different? That's that's kind of the question that it took us a few years to come around to this to realize that but I think the subtle difference is because when we have an out-of-treed package and an out-of-treed driver and Kernel driver, we we have a relationship with customers. So our customers talk to us They typically have a PA or a field the platform engineer or a field engineer that talks to them They read our documentation and actually we've even designed the driver for them often So they would have told us their use cases. We'll have built for those use cases and we'll also validate those use cases and In that we may have missed other cases We also have in some cases Had to make trade-offs. So for example, if you take the out-of-treed driver It uses UIO to expose the device up to user space and UIO is in turn inherently not as secure as the FIO But if it gives more scalability so it allows us to break the device into actually smaller Smaller granularity than just vf. They can actually break a vf into multiple channels and access the hardware Directly with multiple channels. You could have multiple processes using the same vf. So You know when you can have that conversation with your customer and they're aware of how to use a device and you've done all your testing around their use cases You know, that's that's one model once we were in tree. We kind of realized after a while Well, you know, we really don't know all our customers We know some of them and obviously we talked to the big ones But anyone could download the device anyone could handle the driver and they could run it and they may end up running it in For example a norm trust environment even though they're not supposed to and even though they didn't read the documentation But then you know who reads documentation. So we needed to kind of think okay How do we make the driver so that they can't even accidentally use it wrongly, you know make things programmable Programmably clear that they can't actually accidentally use something in a way. It wasn't designed to be used So they were some of the you know the things that we kind of thought about and with that hat on then we went to Did some reviews Actually go back to that one Yeah, so just to point out as I've already said a lot of these things that I'm going to mention aren't things that only apply to our Intro driver there are things that we do anyway with our outro driver But we kind of thought about them again with a different hat on so Just keep that in mind as we go through the next few slides Okay, so in terms of design I guess those three different areas we looked at first of all We looked at some design areas some implementation details and some validation so in terms of design We did some threat modeling just identify where the vulnerabilities might be or if there are vulnerabilities We simplified configuration again We know that our users of the device now once we're in tree might not know You know might now want to either read complicated things or also what to understand really how the device works so for our outer tree we have these Configuration files which have a bunch of parameters in them you need to understand how to use them and you can change them to make the device to break up the device in different ways have different services like data compression on one vf and Symmetric Group on different vf So we knew that customers don't necessarily want that level of complexity. So we got rid of config files and We used CISFS So instead of so we expose a file and CISFS that explains that expose it to the user how is the device configured and we allowed some very Simplified actually at the moment we don't have what we're going to push patches in To allow us to do some very simple configuration there But we're keeping that to an absolute minimum and spend a lot of time trying to think what's the most simplified interface We can expose to customers that they can easily use and again in that configuration. It said we're using CISFS rather than Ioctodes so we have Ioctodes where you can do some of that stuff in our other divide in our other driver in the Auschwitz driver But Ioctodes again mean that the user space and kernel are more tightly coupled. They're sharing the same header files You don't necessarily want you know an Ioctodes They can break I guess it's hard to debug as well. So we've tried to Separate out our kernel new space components because again, and this is maybe a slightly thing that took us a while to cop on Well, we have this out of three package. I mentioned it has all the parts together But that means they've all been tested at the same time and there's like one version of kernel goes with one version of the user space Library now once we started to get into the Linux kernel We're shipping our kernel driver through patches that we send up when we have no Control over the time frame that released, you know, they get in whenever they get in We're separately shipping with the completely different release process a user space library and those use base libraries have to work with Whichever kernel they happen to be loaded on a system with so we we tried to decouple the user space process from the kernel space process as much as we can and keep that a very clean interface with very clear a bi requirements between them and in CIS of s for example, there's documentation going with CIS of s In the Linux kernel in the a bi documentation and that says how stable your a bi's are how stable your CIS of s is And you can say it's experimental if you want to but once you say it's stable Then you should keep it stable and not break it forever more for a very long time anyway So yeah, so we tried to decouple them a little bit We also moved to this capability query approach and again we actually we are doing this in our out of three drivers as well But we're more kind of This is a journey. We're not even fully there with the entry driver, but previously we have code in our user space that's saying Make an assumption so, you know, I know I'm a gen 2 device Therefore, I assume I'll be able to do data compression with the deflate algorithm or I know it's a gen 1 device So I know it you know can only do 2k pke keys instead of 4k or whatever So we wanted to remove that type of code We want to make it very much based on a query coming from the user space to the kernel driver to say okay Tell me what the device can do What does this BF do does it support this algorithm of that algorithm and to do it on a functionality based query instead of assumptions based on the hardware and That also allows us to deprecate algorithms because things like you know sha one You know becoming less less well at an MD5 say are becoming less secure, you know as they get broken We don't want them exposed anymore But our library is out there and it might say well I found it I found a device and I assume that that device can do sha one So I'm just gonna try to get it to do that So we have no mechanism for deprecating devices when we don't do a capability query approach Or sorry for deprecating algorithms Okay, and the last thing here is is to make this trust environment requirement more explicit So I mentioned already that the previous devices gen 1 and gen 2 these are Built for a trusted environment So what we did was we added in the vfio PCI so not in the kernel driver itself not in the entry kernel Sorry, not in the quickest this driver, but we added in the vfio PCI driver a new feature Which was the denialist and that denialist allows you to specify a set of hardware modules of PCI modules that shouldn't be exposed if they're found by default and and we put our Devices into that list so the PCI device IDs for the gen 1 and gen 2 devices are in that list And so by default in this kernel you won't get access to those devices and so So a customer can't accidentally now find those devices and just you know offload crypto to them They don't register automatically with that case yet But somebody can use them if they want to but they have to kind of program actually go in They'll have had to read the documentation to realize that that's possible and then they can Remove the denialist and take that or take our devices out of it And our gen 4 devices so our next generation devices They don't need to be in that list because they are okay in a trusted environment Sorry in a non-trust environment. Okay. I think I've probably covered everything there Okay, so that was the design areas that we looked at and then we also looked at implementation So, you know our actual code what what can we do differently there? Code of use we were doing all over the place anyway for all of our modules or all of our components But in this case we first review within the team, but you know, we're a relatively small team We're growing we don't have a whole lot of expertise We have you know some some great expertise in the group in the Lillis kernel, but we have lots of new people join the team so One way to get the experiences that we have a code review team a PDL in Intel of people who write other kernel drivers And have expertise in the kernel, you know come back for years So before we send our patches out to the mailing list we send them to this internal mailing list and we get feedback there from to help us And then once we've done that obviously we send them to the external mailing list Our goal generally is that when we send them to the external mailing list We don't actually want feedback. We just want to know accept it. That's it We don't want to be the maintainer to have to be you know telling us to Change things do things differently. So generally that's our goal I guess is to get all the expertise ourselves into the team and with the rest of the Intel organization wider organization and to To get things accepted for easy to be kind of some in good quality code And the other area we do code views are secure code of use So we have like a checklist of things that we check for a secure code of use to make sure that there's nothing there That can expose any vulnerabilities And that's part of the the SDL review So we have an SDL process within Intel which is secure development lifecycle and that That has a whole series of of checks and we run through that maybe every quarter there abouts for our our entry code And it has a list of things that we have to you know double check Are we making sure that we're not depending on third parties that we don't that were if we are have we checked Their security status do they have CVs open? There's a whole whole bunch of other things. Do we invalidate all the input? Do we code review that we validate? You know in the code views have we checked it and do we also actually test for that? So there's a bunch of things we go through when we do that periodically walk through it as a team to make sure that we're covering security Yeah, also in the lens kernel. There are a bunch of managed device Resource API's so probably you're aware of these they we used to have with a dev underscore now with this DevM and PCI M and DMA M and they're all managed resources and the main Advantage of those is that it cleans up on a failure path. So as soon as you if you fail for any reason It's going to clean up the memory that you've used So we've changed to use those as much as possible and there's also some functions We're aware of and again this comes from our code of your our secure code checklist, but the one example is string copy We don't want buffer over on so we use sdl copy or maybe sdl and copy sometimes mostly sdl copy We make sure we've got parameter checks on the input paths and we refactor the code This last one is maybe it doesn't sound maybe a lot to do with robustness, but at the end of the day When you've got a bunch of devices and in our case we say with our gen2. There's three different devices in our gen2 stage of the devices a phase of these devices and And so we found we were duplicating code we had code per device in our in the repo So we looked at what's common and we took the stuff that was common and tried to put it together So we created a generation structure. So some code that's common to every device in the generation. We put that together You know that helps to make the code more maintainable I suppose we're constantly trying to make it as maintainable as possible Because if you got code duplication, then you end up fixing a bug in one path You don't fix it another one. They diverge from each other a little bit. It's it's generally not as as robust Okay, and the validation review then we had already done a threat model So we use the threat model to identify some possible attack vectors and and to locate how do we watch we focus our validation efforts on and We made sure we had some Robust error handling on the for invalid input and Adam's gonna talk a little bit about this in a minute And also some error simulations that we did. There's some tools out there. We used for that and Then we run static analysis tools on all of our code And we build a different compiler. So we build with GCC. We build with C lang We make sure that we use the warning minus w equals one on for both of those to pick up any warnings and address the warnings We run check patch With the minus minus strict option to make sure that we pick up anything there So we basically try and find out what tools are available and use those tools Yeah, and the one last thing we have on this internal mailing list that I mentioned We have some server set up in Intel so that as soon as you send a patch to that mailing list It automatically goes to server farm and does a regression build on it and lets us know if there's any regression issues in that Okay So just a little bit about threat modeling I can't share the threat model with you that wouldn't be something we would normally do But just in principle just the idea of doing a threat model is that we you know look at what Assets are there that need protection? That could be the device itself and we make a list of what those are so the device itself The DMA the memory that's used that we DM in in out from the firmware that within the device So we look at what are all those assets and then we you know We have an Excel spreadsheet where we go down to each one of those and then say okay So what's the attack surface free to those is it possible to get at those somehow and look at where then the trust boundaries are crossed so You know is it on the data path or or from CISFS? Is there someplace where you've got an untrusted? Process that has access to these assets that need protection and then some of those You know the answer is actually no there isn't really and it's impossible to get to them But we you know we look at where they are and then what's the potential impact if you could get to attack those those assets And and how likely is it and and what mitigation is going to put in place? So that kind of informed in how we how we validate and some other things that we changed Okay, so Adam's going to talk about a couple of things that areas that we looked at so these were some of the areas where we We wanted to focus some efforts. So one was CISFS So we're exposing some files We just want to make sure that nobody could manage to cause a denial of service attack or anything by by writing bad data in there Another interface we wanted to check was this PCIE when it calls a callback to make sure that we would handle the callback correctly and That the device would recover as expected And a third area just we want to share is is how we tested this PFEF comms interface here Which is where the user space is trying to send a capability query for example to the kernel driver Okay, so I'm gonna hand over to Adam. I think he's going to talk a little bit about some of those some of those testing tools that we used Thank you Fiona So I guess you don't mention CISFS already, but I'm just going to do a little bit more detail of what it is So CISFS is essentially a pseudo file system that can allow kernel subsystems or a Device driver to expose information configuration or functionality to user space and it does it by using a file That can be accessed by either a user process or a user So for this what we need to do is we used a an open-source tool called redamza We're damsel that us give it an input and it'll give us a fuzzed output You can see some examples of that there on the top Some of these are going to be things that you might not think of Entering into CISFS yourself and it's a handy way of getting generated inputs for that And because redamza is a CLI What we can do is we can write a batch script to just constantly send a text over and over and over again until maybe it goes down Maybe it doesn't but hopefully it doesn't So we're using this tool what we're able to do is is validate a range of boundary conditions And ensure that our driver or CISFS interface was robust enough to be upstreamed And as well as that we learned that we need to tighten up the validation over input strings And as well as that because we don't exactly know who's going to be access in our driver. We should probably have pretty usable So pretty usable Inputs so for example, we have The state of the machine which is exposed or the state of the device which is exposed And this can be either up or down very simple and for configuration of crypto. We have Symmetric and asymmetric or both So quite simple easy to understand for anyone that doesn't read documentation like me So, yeah, so moving on to PCI the PCI interface So what we're going to be doing here is looking at the AER Advanced error reporting which is supported natively by quick as this so to kind of get at this what we need to do is have a machine that supports they are natively and We can do this by enabling in grub and In the kernel as well as self We also need to have a driver that support they are as well and quick as this does We can use an open source tool called a or inject And what we can do is we can inject certain errors into PCI A or framework And we can see we can observe how our driver responds to those errors So just kind of two Categories of errors that you can inject One being fatal errors and one being non fatal errors Um for the fatal errors what you might want to see from your driver as a function level reset and that driver returns to a usable state And that you can you can test using functionality tests or integration tests to just make sure that everything returns back to a usability usable space and for the non fatal errors Essentially, you should just return to normal kind of the same as the fatal errors but you might not even see any sort of pick up at all and um Yeah, the driver should return to a completely normal state Using this tool what you do is you type a or inject and then you pass in a file And you actually make the file yourself And you can pass in the pc id of the device and as well as that The error that you want to inject there's a bunch of these And you can kind of pick if it's correctable or non correctable or fatal or non fatal and yeah So what do we learn from this what we learned that we need to reset or change the reset flow of the device And that's for one as well that this tool a or inject was quite easy to use I think I set it up in five minutes And I was able to just test if the device was responding correctly to these a or uh errors And um, yeah, it's pretty handy for any developer that wants to check this Yeah, and I think I'll pass you back to Fiona. Okay. Thanks Adam Okay, so yeah, so the last area we want to talk about was um the pfv of comms. That's the physical function to virtual function interface um So I might just jump back to this Somewhere here the diagram. Okay So this is the interfaces that already that I've talked about between the kernel driver and something in user space that's using the virtual function and I guess well the first thing so we've got this already in our gen one and or in our gen two Yeah, and gen one Drivers is that we realized early on that if a user space process Just pass correct data in On this interface, which means it would write into a csr here. That's just a 32 bit csr And as soon as it writes triplicular beta and that it triggers an interrupt over here in the kernel driver um So one of the things a used space Device could have done or driver could have done in the past is it could have Just created lots of perfectly valid requests But had a flood then of interrupts on the kernel driver and caused a denial service attack by doing that Um, so we've got some talking that's in the kernel driver here So it won't allow that to happen so it prevents a flood of interrupts being handled um from any particular vf And then the other thing we wanted to do was to make sure that what we wanted to this time was to make sure That when you write into this csr Random data rubbish, whatever the no matter what you write into that the kernel driver is going to behave itself And not get not have to reboot Either the platform itself sometimes you can actually in the past we've been able to pull down the platform with this But um, we've made sure that that's not possible anymore So let me go back a little bit to the ways we thought about doing this Okay, yeah, so said we've already done functional testing and to do the functional testing of that interface There's a protocol with kind of timing across that interface So we had a homegrown test tool that we were using for that And that test tool focused on passing in correct messages like get the version or get the capability query The testing we wanted to do was to extend that testing and to focus on corrupt messages invalid messages putting rubbish into the different Fields in the csr. And so we had to trade off between using the fuzzing tools Or using the the in-house tool that we had and this in-house tool we had As I said it it was a quick ramp for developers. We already know how to use it and We found it faster to get there because the the other tools like afl or or kfl. So afl is a A kernel version of afl, which is american fuzzy lop, which is a fuzzing tool um, and afl both of them what they do is they Generate data something like radamsa does they can generate a data set and they can fine tune that data set And then they can also add code coverage in whatever you're testing as well So inside and the kernel driver can put in code coverage So when it sends in a particular piece of data it can then see the path that takes to the code And it can fine tune that data to find that to make it go through all the different paths in the code and try and get full coverage um The tool that we used we we had we had a tradeoff between you know, which do we ramp on this and use it and because of the complexities of our protocol and because The the data would be injected in user space Whereas it's the kernel space driver reading from a csr's where we're actually trying to trying to break essentially They were two different Domains, so we found it quicker to get up and running with our own tool. So that's what we did And and it was effective It was effective, but I guess we can see also the limitations of it So what we've done a lot of fuzzing using that tool. We are also looking at using the actual fuzzing Off-the-shelf tools as well because I think they're quite useful and so the advantage of them is that they do I said it's a more comprehensive set of data and and just to give a kind of an example from Looking at what they can do if if you as I mentioned you can get code coverage by using these tools um if you took An example of say you're passing in the number 10 and that's the parameter that's going to be passed into the to the to the process of testing And if that goes to a path, which is saying, you know, it's it's a it's an error It's a greater than whatever value you're supposed to have Then if you pass in 11 it goes to the same path and you pass in 12 and it goes to the same path And you pass 13 it goes to the same path. You're not really learning anything new So the algorithms in afl and in the the tools that are available are actually quite They're able to tune for that so they can you know They can recognize that you're you're regenerating the same kind of data going to the same paths And so they throw away those data sets and and generate other you know using the fuzzy logic generate other data paths And I can get a condensed set of data eventually that you can use that that fully tech checks all your paths to the code So we use the homegrown tool the advantages that got us to results quicker It's a bit more limited maybe in in how your your reporting is done And it's not and and the data set that you've got is quite random in the end We just randomized all the fields So the recommendation we would have is to use the fuzzing tools rather than homegrown tools You know Use one to get there faster with your homegrown tools But fast track the learning curve by getting help from others in your organization So we have people in it who have used these fuzzing tools before So we're getting some support from them to get up and running and re running the test using fuzzing tools Okay, so I think that's pretty much all I have I guess the main points I want to get across is just that you know Code in the Lillis kernel driver code isn't exactly the same as out pre code You do tend to have to kind of have a mind shift if you're if you're in the kernel and your your audience is the world Then you have to maybe think differently in how you validate How you design things how you implement things to make sure that you're keep making Everything as programmatically secure as possible that you can't accidentally use something in the wrong way And that's you know a good learning from that we've had and that there's plenty of open source tools available to help You to do any kind of validation you need to do Any questions Ready, we do have some online questions. Okay Um, forgive me if I do not pronounce this correctly as well Is qat infrastructure ready to be used by kernel file system will be efficient to use qat for delta encoding compression or deduplication by kernel space file system to accelerate file system operations Um Yes, not deduplication though. Sorry the other ones were data compression So delta encoding compression or deduplication by kernel space, right? So The first one i'm not sure about the delta And not the deduplication but the compression in kernel space. We haven't upstream jet. It's going to be upstreamed Have we sent patches? I don't know if this week we're not there this week. So In the it's imminent We have working versions in-house and they will be upstreamed very soon Okay, okay, and then one more is qat device is ready to be sorry Is the qat device ready to be offloaded? computation I mean access to storage device directly and to execute decompression for example not to use cpu resources So I'm so sorry. Let me see. Yeah. No. I mean it it is so for It sounds like the same question or just a variant of the last question. So is it available? It's our talking about in kernel. Okay, then the entry driver then As again, it you know, we're going to plug in so compression the kernel comes through the linux kernel crypto framework And so far we've plugged into the crypto parts of that But compression also comes under lkcf and we it's on it's It's a compass the api and The algorithm deflate is the one that we're going to expose very soon In the next few weeks, hopefully So yeah, so that that that is available. Um, it is available already in our user space Libraries, so you can access the device if you've got a user space process and use qatlib, which is on github Then that you can access it, but if they're talking about in the kernel then two lkcf patches are coming in the next two weeks Okay, wonderful. That's it from the virtual attendees Okay Oh, okay. We do have time for one question What sort of static analysis tools you guys use and what do you what are the common things you normally find when you run a Static analysis tool on your drivers. Okay, so I think there's coaching l and Sparse was sparse and I haven't used that one, but I think you did or one of the other guys in the team did Spatch And I think there's a third one we use we've kind of pulled down whatever's there and try them Some of the other people on the team the common things we find um You know uninitialized pointers or at least ones that say they can I mean to be honest a lot of times It's not that they're false negatives um, sorry false positives, but that You know you can look at the code sometimes and know that actually you probably can never get to that point But the tool will throw up something saying yeah, I can get you know that there's a false positive here um So yeah, they tend to be fairly quick to fix I mean and they're they're generally that kind of level of pointers uninitialized are Paths not followed through the code or paths followed through the code with some data not initialized before it was passed into it Okay, sorry got to stop now, but happy to take questions afterwards for anyone. Okay. Thanks everybody