 Cool, so I wanted to share Quick prototype of something that Ted pitched to me yesterday I see Ted is in the midst of a deep conversation, but Ted pitched to me using dragon to Validate kernel data structures because apparently you see four has a bunch of of a kernel code that it has to Validate that a bunch of EXT for specific stuff is It's saying so I Figure or he mentioned that maybe dragon could do this so I hacked something together that would work for that So for those of you who aren't already familiar with dragon. Here's like a quick Quick introduction basically Let's you look at kernel variables So like here's the Initial file system Structure you can look at members in it you can Since last time I presented this which was like 2019 you can do stack traces and You can look at variables inside of strict stack traces. I'm cheating here. I looked at the source code earlier but Epoly has a max events thing and you can just get the the local variable out of it and then like do stuff with it But like I said what what Ted pitched to me was using dragon to validate data structures So I wrote a ugly little kernel module that does some artificial bugs first one is link list that are I'm pretending here that to live to list ads raced to add Their own entries to the link list So you end up with a link list that doesn't make sense the next and previous pointers aren't consistent with each other Then I have another example here where Have a red black tree should be sorted on this key field Have like the classic red black tree boilerplate and then have code here that set the key without deleting and reinserting it in the red black tree so Have some dragon code which is based off of the the dragon helpers that I already have that know how to walk through list Link lists and red black trees, but they do a little bit of extra validation on top of that So it's it's straightforward. What do you expect? we Get when you're walking the list you get the next field and the next previous field should be your current nodes thing if not Just raise an error For a red black tree we can do a little bit fancier We check a few things we check that the the red black tree nodes parent is the same thing as The as what you'd expect and you can check that the ordering is all right. So Does it compare it to what you think it should and whether your red black tree allows Like two elements that compare equal or not So I stitched those up with a thing that will just print out if that fails Same thing have a key comparator thing here that kind of checks that that invariant And again just cause a validation thing but most of the validation things also return to you The actual items so you can do further validation on them So I Here's something that's that does the that checks the vmap areas Which are in both a linked list and a red black tree that are both sort should be in the same order So it runs through both of those at the same time checks them and then also checks that the the start fields and end fields are That the start is before the end So we can just Run it I don't remember if I already inserted this I did not So that was the the buggy kernel module I can run drag in Since this is not installed I have to tell it to where to find the kernel module to get the debugging symbols for it and then I Can just run that script. I just showed you so just tells you Next field should have been this but it's I got this instead The red black tree is out of order and the Vmap areas thankfully are correct on my prod kernel. So Yeah, this is I thought I really like Ted's idea So I just prototyped a very basic implementation of just two of the most common data structures But be cool to see where else we can go with this. So I'll probably be adding These two to the dragon repo if anyone has anyone else has ideas of what else you'd like to validate The building blocks are really nice because we all have data structures built on top of these things But they're sure to be like a butterfest specific thing like we want to check that our our red black tree of like Range locks make sense and stuff like that. So That that's all I had If you want to find out more about dragon you can literally Google dragon debugger and I guess it's gotten SEO'd enough It'll show up cool. Hey everyone so kind of Tacking on to what Omar was talking about here What right now Dragon does is based on dwarf and uses the dwarf debug info and if you're one of those lucky people who happens to have debug info Packages installed on all your systems. That's wonderful for you. I'm so happy for for you guys But not everyone's that way and sometimes, you know, you've built a kernel and you already lost the debug info It's stripped away But something that we happen to have right now In the kernel is a few things btf and that includes function type definitions for all functions ones exported ones that are not And per CPU variables as of now K all sims you depending on your configuration It has all exported symbols or more likely if you like stack traces Then you have it for all symbols, not just the exported ones and then we have other cool tools like orc that can let you do a stack traversal without using frame pointers and Those those all on their own sure they're not as powerful as the you know The the call frame information that dwarf can give you the the source code mapping things that dwarf can give you But if you're just looking at a crashed system and you don't know anything and you don't know You don't have the debug on debug info. This could power a debugger and I don't need to Rag on dwarf. It's great if you have it So what would we need to do to use that? It turns out a couple things. So first we have btf we'd need to add debug we'd need to add information for global variables right now We don't have that for some reason. It's just per CPU variables, but it's like 2.5 megabytes of data You know not every kernel wants to have that in loaded and not able to swap out So it's a it's a it's a it's a blocker for some people But some people in a server might not mind using that 2.5 megabytes so that their VM core is always debuggable In the kernel side of things you'd need a way for a debugger to be able to find Chaos sims there are a few symbols that point you in the direction of chaos sims and then from there You can decompress it all and look at addresses look at names and everything and then last thing that you'd need is Support and dragon to be able to read btf and do k all sims and as it turns out I've I've done all three things in some very very hacky fashion So just just as a demo here, and I guess I should have this over here We have well first of all this is a VM core. It's kind of small and Yeah 130 megabytes I'd say and If we just tried it with regular dragon the same one that that Omar was using we wouldn't be able to look at these variables like Let's see in it FS. Yeah, it can't find anything because there's no dwarf information But if we were to do the same thing With this custom thing that I've that I've got I've got some stuff in there and you know this little line It was able to load k all sims in btf data. So to be clear There's no there's no not even on the file system. Is there a dwarf debug info for this program? But or for this VM core, but if we do it Turns out we've got all this data as almost as you'd expect, you know, how about the stack trace? Should tell it which PID but there we go. We've got stack traces These are using frame pointers not the built-in orc data, but it's all there and Yeah, I just as an as a final example. I think Omar had some good Example right at the beginning and we can actually just paste this right in just to demo that all these features are pretty much working as expected These are on this particular Kernel all the modules that we're running over with over 10 references That's really all I have. It's just a neat starting point to say that Our kernels as almost as they are today have enough to power a small debugger, you know live We could do the same things live We could do these things on a VM core. So There's work going on. I'm starting to To check this out on the BPF mailing list. So if you want talk to me after but yeah, that's what I have Just wanted to quickly go through How BPF CI works and this is a follow-up to the talks that Joseph did yesterday about How CI systems are useful for the maintainers? So we're using patchwork As a source of all the patches Basically the way it is set up every patch that is sent to BPF at VGR it will appear on the net def BPF list and So patchwork will show status for each patch after we tested it It knows how to collapse page series into one and how to identify it and It can be enabled for any Linux subsystem. In fact, I know that it's already enabled for better fast and for many of them so This is the screenshot the list of patches for BPF and you can see the statuses some of them are passing with warnings some of them are failing This is the example page It shows that it is part of the series and it shows the status of tests for this page and This is like just a small amount of subsystems that already on boarded to the patchwork We use Internally implemented kernel patches demon It's Python service. It identifies new patches on the patchwork It creates a pull request to the github kernel patches BPF repository it merges github actions on top of it and Github will magically execute tests for us and then KPD takes results from the github and post it to the patchwork So for example, this is the example Pull request originally this patch series it has four patches, but we also add github actions on top of it as a separate commit and This is a rep over github actions is stored and all the magic is in this github.github directory So everything that should be done is defined in there. It's just a script so we have these two repos and Kernel patches BPF is just a mirror of BPF and BPF dot next git repositories from the Linux kernel And how we actually execute this so we have cell hosted github runners We currently support Two architectures one is x86. We also support IBM as 390 x To test different types of indians, right? But you Your test support cross compilation you can run it on any platform that QM supports basically So this is how runners look from the github perspective Like you you can see that for x86. It's basically a ws instance that we are spinning up to run all the tests and What as we are running we are running BPF self-test and This is how result looks for the successful run This is how result looks for the failed run and we also highlight what was the issue with the test and Yeah, we have pretty rich road map in front of us. So we want to add Sanitizers we want to add more platforms And we actually really want on board more Linux subsystems to it and some of those things is like better FS RCU That have some tests already available We'll be happy if someone will want to contribute or to discuss it offline Yeah, thanks all guys