 Okay, I think we have most of the interesting part in the room for VM test Yeah, let's get started next then we talk about a VM test Hello, so this is about VM test testing Running tests inside a virtual machine not testing virtual machines So this is the motivation but before I actually begin. I just want to Make a note that none of the ideas here really that new in fact. I just stole most of the ideas I Just looked at all the other projects around and I took all the ideas I liked and I combined them and in particular I even stole the name But everyone was doing that already so I I don't feel too bad about that But uh, yeah, so the motivation So the reason I started this project was that you know, there's a few reasons one like tests are good It's pretty hard to maintain Complex software without good automated testing. Otherwise, you're just thinking really hard when you're looking at code. I'm not very good at that BPF is also really fast-moving target. I do a lot of BPF related development And so if you have like a non-trivial application You have a lot of feature gates and fallback code and especially in like the new k-funk world There's no like UAP I guarantee so it's It would be good to like test your code against a bunch of different kernels to validate your assumptions But that being said This is a pretty generic problem. It's not really limited to the BPF world. So The solution is hopefully applicable across a variety of domains But more in BPF so BPF program. I think is actually really powerful It's one of the really cool things that BPF powered applications can do Essentially, you can run a program in freestanding mode You provide the input context and you can like the return value. You don't need to attach it to anything But it's inherently dependent on the kernel you run on because it still runs in the kernel It doesn't run in user space. So you have to wrangle the right kernel The last point is that VM based testing is actually pretty popular a lot of bigger well more Resource to projects have support for that, but it's like it's all very bespoke stuff Very sometimes kind of hacky and definitely not reusable. You can't pull it off the shelf So the main thing I want to do is I want to create a solution They could pull off the shelf and it's really easy to use and in fact, I want to make it so it's silly not to use it so the goals in order of Kind of importance the first is reusable infrastructure I want people to be able to pull it off the shelf whenever they need they can instinctively reach for it The second I want to make it run inside CI. That's really important because I don't it's much easier to tell contributors Hey, just make the CI test pass versus they run these 10 steps on your machine and do it the way I do it And hopefully that works and the final goal is just to make it simple easy to use I don't really think this problem space is that complicated. So I think you can hide it behind a fairly simple interface That's about all the talking points. I had the rest of this presentation is mostly just demos I just it's easier to explain if I just showed demo There's two demos I'm gonna do one is as a development tool So you can use VM test the binary as part of for example, you know iterating iterating on the kernel changes You can just run it run some tests really quick. The second is a VM test action, which is a github actions wrapper I think that's just actually just called the github action in the singular form And you can it demonstrates how to drop it into your existing workflows really easily Oh and also note that there's two parts to this thing right VM test is the binary contains all the key move orchestration logic and some Terminal interface things and the second thing is Just a little bit of JavaScript that installs all the dependencies and configures VM test to work in github So, yeah, consider this config file. It's VM test that Tom will just Tom will format This is the local development example. There's two targets Essentially what we're trying to do is we're checking that the kernel version is what we expect The first one checks if the running kernel is 62 the last one checks if it's 61 Notice how we're running the same 62 kernel for both. So only one of these will pass. I put a gif and a PDF It doesn't work. So I also put a link Yeah, so pretty much. Yeah, you type VM test and looks in the current directory for the config file And then it'll start running things. I actually went through a great deal of trouble to make this interface Dynamic it also does the correct thing if it's you know If you're not attached to a terminal, so just prints it out straight to a log file and works as you expect It does some nifty things like it'll collapse Output if a step succeeds, but if a step fails, it'll leave it expanded So it's a little easier to debug just the UX thing the first time I did this I actually didn't make it like streaming output, but you get really antsy if there's no progress on the screen and despite it taking the same amount of time Runtime it feels much faster this way fancy, huh? The second example is the CI example So I create it like a demo application. It's really simple. This is main.cpp pretty much a dynamically links against JSON cpp I Runs you name and it prints out the kernel version in a JSON object for such a binary you might have a See I configure like this if you're not familiar with get-of-actions. This is Really just three steps the first step is you install some build time dependencies and maybe some runtime dependencies as well second step is you build You build your binary Configure slash make and then third one is you run your test runner So now consider because your binary depends on the kernel version you want to say Support everything 6.0 plus and you want to have a matrix of jobs that test that so this is how you would use VM test to drop into your CI workflow So the first part of the diff is You have a matrix try testing strategy. This tells GitHub to spawn three parallel jobs That will you know run 606162 and the URL is just linked to some dummy asset Where I dummy release where I host a bunch of test assets in this case. It's just the base. It's the standard VZ image The bottom part of the diff is how you drop it in so you replace the test runner thing With the kernel and the command and it just works And so note that the dependency installation and the build steps happen outside the VM And you want that because it's probably faster than being in the VM in the final step We run as little as we can inside the VM and we just only run the pre-built binary And so this is a good example because it does dynamic linking this binary. So it proves that The host user space is the same as the guest user space and we do that through Clever use of the 9p file system as we mount the root The host root of s as the guest root of s This is what it looks like the output for the 6-2 job at the very end. You can sort of see it print 6-2 Don't believe me. I have a link To the job and I can show off some of the other things So yeah, here's the 6-2 and then we do the run and Notice how the output is not collapsed. It's just all there and you're booting your kernel and the very end Runs the command and there it is So the implementation there's really no new ideas here You can squint at this later if you want, but if there was one novel thing I did it was I used the programmable interfaces for QEMU and QMU guest agent So a lot of the existing things they kind of just shell out and they have like triply escaped commands that they shell out with Which is which isn't necessarily a bad thing But I get really nervous when I try to extend that kind of stuff because you can't really rely on the type system or programming interfaces Cuma guest agent actually is really neat. I don't know if anyone's tried that yet for at least this use case But it's a it's a very clean out-of-band mechanism to do things inside the guest So you don't have to bring up networking or SSAH for wrangle SSH keys Pretty much is just a binary that runs inside the guest VM. It listens on this bird IO serial Port and they have some kind of RPC communication. That's statically typed So yeah future stuff the core functionality is mostly there for the use cases. I have in mind in the projects I'm going to add this to But if there's other feature requests we can discuss that and I can add more stuff One limitation right now is that the get-up action wrapper builds VM test from source That's because it's not packaged anywhere So a binary distribution would be good because it would cut down the CI time and short see it times are good There's a lot of ways to do that in the Rust ecosystem. So I'll just do it at some point. I Was chatting with some people yesterday One-liner interface would be nice. So if you could do like Really quickly just run things under different kernels With a couple words that'd be neat Tying into that idea if you build a registry of See I made we could build a registry of distro like kernels. It's actually not that hard I have these scripts that use Docker to repeatably Do kernel builds which is different than reproducibly which is much harder So you pretty much you take a distro K config you flip the configs that The M test needs pretty much you need to build nine PFS in as equals. Yes. That's the only thing you really need You can have a registry and can upload it to all the the Yeah, the fake asset. This is an idea. I stole from Omar. It's a very neat idea It's a way to get around paying for Amazon s3 You can just use github to host binaries Yeah, if anyone needs stuff you can open a ticket email me discuss it But I think it's pretty useful and it's really easy way to just do VM testing in your projects So Thanks, that's really cool The It was really impressive like how fast the test was in the give or achieve I don't know So we have some infrastructure for this in Cillium. It's called little VM helper. I would be surprised if you heard of it We we do something similar. So We use it to test Cillium that are gone and pack it. Where are you? And the basic idea is that we have a Program that looks a bit like this, but it also can build root images And it can also be used to build kernels So we need to build root images because for some of them We need to have kind inside so that we can do like Kubernetes tests and for others We just need to do like something. I don't know like run a bunch of code tests and there's another repository that has a Configuration of how to build these images and what it does is basically whenever you do a PR on it There's a GitHub action that will compile the kernel, build the root images and push them into OCI repositories So we already have like, I don't know five or six kernels that we test against in these repositories And also root images that one could use again like our use cases are kind and no kind But yeah, this looks really cool. I Just wanted to be because there seems to be like a big overlap So if you feel like something of for the from little VM helper would be useful would be great to like bring the two together Does that's so the thing you're describing it builds images But it's not Doing like the testing stuff right because I think the Cillium stuff uses vert me like a wraparound So Cillium's last CBPF use vert me Cillium uses this thing little VM helper so In addition to build images you can do LVH run Which is basically like VM test here. So But like this is I am pretty sure like it's pretty similar to what you do What you do is probably better because better because it's much faster Yeah, well one other thing I think I forgot to mention was because all the programmable interface stuff is there I it's actually really solid foundations in my opinion. You can I Feel pretty confident adding a lot more stuff. It's not very hacky at least in my opinion Cornelius just said that putting things into container images work well, right? So I Wrote the the vert me integration of use for the ebpf library And I think the m test for me is attractive because or why like vert me is essentially like a precursor to this like Less featureful I would say and why this is as nice as like this idea of having the nine pfs Kind of the host file system mounted Into that VM is for me really powerful because it means that I can very easily Test stuff that I have cooking on my local machine without kind of having going to go through like I build an image and all this other stuff And I actually went and created some issues this morning for you I think one one thing that would be really interesting and where where I think this Idea of having The host file system mounted is really powerful is We could add a flag to the m test that actually starts the gdb server in quimu And basically then the only thing you have to do is like you do VM test this thing and You can then connect to You can you can then connect to kind of the the current that's running your whatever your test is really easily and kind of Have all the tools that you're disposal Inside of that VM This also supports VM images. I didn't show any of that off, but it can boot up like a q-cat to image And it also does the mounting stuff with 9p So 9p is read only and I would like to use this for like better of us Brog's which is gonna have to write that's something Is it? Yeah, so it like it creates images and like corrupts them and then runs FS check against it to make sure that it does the thing So like I could probably make it run in tempfs inside the VM But like I would like to have the option of like I've used for me for like my quick tests And that lets me add disks. So like I don't know if you want to deal with all that bullshit That would be useful for me my excuse Okay, I need to look into that. I thought 9p was read writeable. I don't know I thought it's well, maybe then but it's just the way I use it as always read only so I could just be dumb Oh, you can read right. Yeah I think like in some of the VM tests other project VM test projects They mounted the 9p root of fs's RO Command line. Yeah, kDevOps wants to read only and I've only ever used it read only so I'm just I'm an idiot If it's read write then we're good. Yeah, I think it has to be right if it was used for the plan 9 OS stuff Yeah, I don't like again. I've only ever seen it as like give me access to like my get tree or whatever and like But yeah, you can add more disks. It already wrangles stuff like that and there's internal scaffolding for it. Cool. Yeah. Thank you everyone