 I'll get started with my talk today, which is called QMU, emulate your way to success, give a quick introduction to myself and what I do with my work, and then we'll go through some basic case studies, information about QMU. So welcome everybody. I hope you won't have a nap after lunch. It's great being back in Japan. I was here back in 2019. So I'm a senior engineer at a company called Codephink. We're an open source consultancy based in Manchester, UK. I myself have done a number of contributions to things like QMU, Linux, and a few other open source projects. I really like QMU. It's been really useful, and we'll show some of that. Legals wise, I believe that I've been OK to put this out as CC by SA licensed V4. It is not V1.0 because I have done a couple of minor edits from when this was last given. So the introduction. So we'll aim to show some things that you can do with QMU. We'll go a little bit into what QMU can do. We won't be going into any deep technical dives because QMU is a very big project. It's not kernel big, but it's certainly big enough that it warrants its own confidence, I would say. But hey, we don't have that yet. I won't profess to being a QMU expert. Not all of the work that I'm going to be talking about today is my own. We have used QMU extensively throughout our company in a number of different projects. So we'll start with a quick introduction. What is QMU? So it's flexible open source system virtualizer to emulator. It can basically act as something more like a container to a completely different system running on your own computer. It supports many different architectures. Some of these are very familiar. Some of these are now lost at the history of computing. So, for instance, x86, both 32 and 64, risk 5, 32 and 64, ARM again 32 and 64 bit, and a number of other things. Importantly, it does not need any kernel drivers to allow it to work. It can be run as a purely user process. Although there are enhancements that features like KVM can give it, they are not necessary for it to run. I'm fairly sure the main source is GPLV2. There are some bundled components that are not, but those are mainly extra bits you can use. It runs under most Unix systems under Apple or Windows, so you can pretty much do the same thing that you can do in your Linux under Windows. And pretty much most major Linux distributions ship QMU as standard. We'll do a quick bit of history. This is not very well researched. The first release tag I could find was 0.10 back in 23rd of March 2003. The current release, and this is current, the v7.10 release may well become v7.2 soon. That I believe there's a release candidate. By the time I finish this talk, it might have moved on. There's also a v8 starting in development, but that will not be around for a bit. It's got over 2,000 contributors. I'm not going to say it's accurate. This is me using Git and some scripts to work out. Some contributors are a few patches to at least one with over 9,000. I'm not going to do a lot of talk about the first way you can run QMU. This is an API emulation. If you want to run ARM code on your x86 laptop, this is the way you're going to do it. It allows you to run foreign code under your host system. It translates that code into sort of native format. It also will translate the system call information signals, that sort of useful stuff to allow a process to run as if it was a standard process on your system. This is very useful if you're doing, say, a churru into a foreign system. You can use that to set up a churru or set up a bin format extension. There's documentation for that, and often your distribution will do that for you. It's probably not very easy to read this one, but that was just me running a terminal starting in x86 land. I use schurrut to go to a Debian ARM64 that I have on the laptop. My work laptop, I left that in the UK, so I haven't got any examples here. Then I can go in the same shell. I'm now an ARM64 system. This is very useful if you just want to build small binaries, et cetera. I'm not going to go too much further into this because most of my talk is going to be more around the system level emulation. This is where QMU can be used to emulate your complete system. More like your virtual box or something like that. It emulates everything from a CPU cores, a bus setup, devices. You can even change the memory system emulation. If you're running on a standard laptop, you can make your emulated system run as a non-universal memory architecture. Another useful feature that you may want to use is to change the boot flow of the system. You might not want to start with a BIOS. This is useful if you're testing a full boot system. It can suspend and it can migrate your virtual machines. Again, that's something I'm not going to talk about. There's also a huge area where it has its own little code generator for speed. Again, that is an absolute minefield of stuff and wants its own talks having played about with this on something else. Again, TCG may not support all architectures and all use cases. It's there if it can do it, it will. So, what does QMU as an emulation system emulation provide? Well, pretty much all the standard things you'd expect from a machine. You can have input, display, things like character and serial devices, audio, block, networking. You can even do trusted platform modules if you need to do secret management. Some of these have a single connection option. If there's things like display, you can provide all sorts of different output routes from local display to BNC. Again, you have a number of different buses you can emulate from standard memory mapped IO through pretty much everything you would expect from a system. QMU will provide a lot of standard configuration options for systems and, as we may see later, you can configure to an extent what devices your emulated system can see. This is not going to be an exhaustive list. It's more of an example. Again, with the devices, there are a number of device models that it provides. Again, there are many of these. A machine model may provide you a standard set of devices that you would expect. So, if you're just using a standard x86 PC type emulation, you might find a standard input output PCAE, etc. It's an interesting use case for actually emulating PC within PC, and we'll talk about at least one of those later. Caveats for this device models, whilst they are good, may not be 100% accurate. You can actually also add your own, and we will talk a little bit about controlling them later in the talk. Some of this you can do via command line, and hopefully, if anybody can read this, this was a quick example. I went up a while ago using QMU to emulate an ARCH 64 system. So, the command line here is for a virtual machine. We don't want any graphics. I'm going to give it 2 gigabytes of its own memory. It has two SMP cores. The rest of the command line is saying, this is the kernel I want. This is the Nishardee. The arguments I'm going to give to the kernel that I'm running, which is the append. And then what I'm going to add is I'm going to start a USB system. Give it a QMU XHSI emulation so that it has something that the kernel can see as a way of actual device. Give it a drive for that. So, the drive, I'm going to say, is called USB stick. It's a raw file that I've created somewhere. And the final line of this, I can say to the USB storage device model, here's your USB stick. That's what you can do. The device will start up. You'll have an emulated USB stick, which you can use for storage. I'm not going to go for a demo here because time and demos very rarely work properly. So, I talked a little bit about the whole emulation setup. Usually there were just a set of standard machine models. These will define how your system works. Standard devices, standard boot method, and you should get a system running. This generally good enough for quite a lot of stuff. The interesting problems happen when you actually want to build your own machines because that can run into some issues. Quick talk about networking. It provides networking. Usually this is done via a virtual network device that can either be backed by a real device. So, something like a tap or tunnel. Or it can emulate a system and just use the host's network socket calls to actually make the network connections out. It does usually IPv4, IPv6, you can do some raw ethernet. Can slightly different, but still you can use a virtual can socket connected to the machine. We've had systems where you can create virtual can networks between a number of different QMU hosts running on the same machine to emulate, say a vehicle with a can network. Again, there's a lot of mileage there. You can read up on exactly how complex you can make systems as a post-talk exercise if you really feel that way. So, boot flow. Your system needs to start somehow. You can go from anything from just giving QMU a blob of code and get it to do execute it. We've done that for some very interesting testing of bare metal cases for customers. QMU will just give you a basic entry, call the code and get it to run. You can also use higher level start systems. So, imagine things like on your PC that would be a BIOS, QMU comes with at least a standard open source BIOS implementation you can use. Or you can give it something else. You can use trust zone, which will then call down into another level, which you will need to also have loaded. OpenSBI is the risk 5 similar machine level services block. These codes can then install what they need to do before they call into a kernel of a binary block. You can do all sorts of boot flow emulations if you like, so you could provide U-boot initial loader and then allow that to load code from an emulated flash device. It's very useful if you want to test your modifications to U-boot. You can load U-boot main, grub or whatever. It's very flexible. So, if you've got a running QMU, you can control the state of the system via a system called QMP, which is the QMU machine protocol. It's very JSON based and you just have a socket you can connect to either locally or via network interface, which you can use then to control the state of your system. So, if you would like to inject or hot plug devices or anything else you've done, there's a D-bus interface that allows you to get at some of these interfaces as well. And if you're just running it on local command line, there is an interface you can use just to break out of your console and then issue very much the same command methods. This is useful if you want to automate some testing. Again, not only can you debug QMU itself, you can also debug into your emulation. QMU has a GDB server that will serve the state of your running emulator system. It also has its own internal tracing you can use if you need to, say, do some development work. So, for instance, we've done work adding machine models to QMU. Unfortunately, legal issues have stopped us actually submitting that there, which apparently they've been working out while I'm here. So, for internal tracing you can use the command line to enable tracing. So, you can say trace M25 star, which will trace all the M2580 flash driver operations. Very useful to see how you'll say if you've got a system that isn't booting properly you can trace what the flash was doing. You can trace maybe what the SPI driver was doing. If you get really desperate you can trace all the memory operations, but you better have a big log file ready for those because it produces a lot of output. So, that was a very much just general overview of what QMU can do and how you can use it. So, go go through some projects that we've done. So, first one won't go into too much detail, but this was the use case of using an x86 machine with an x86. We needed to do some work on the kernel entry code. We could have done this by setting up some machines, but if you don't know, PCs take a long time to boot and are a pain in the... Well, let's say they were a pain to manage. So, we use QMU on our desitops to allow us to get a better privilege without having to actually do anything that would crash our own machine. We can use it to control the process of features that our emulation can see so that we can say, OK, our very much latest laptop can now look like an Intel Atom. And then we can use that to run and debug our kernel entry because we can also stop it, use GDB. I also found that a lot of even the embedded x86 boards are not very helpful when it comes to debug. Yeah, so using the QMU here made the development cycle very much faster. We could automate our testing and see how our changes worked and do various testing there. So next thing, this is probably much more interesting to people producing systems. We can use it to integrate into your CI pipeline. So testing your real software without actually having a piece of hardware or allowing it to spin up multiple different hardware to test. We've done this in a number of projects ourselves, but this is one of the first points where there are good real world examples. So SUSEI use QMU with OpenQA. They can test each release they produce out of their CI pipeline without putting it onto a real machine. They can do completely boot flow from as if you were installing it from a USB or CD device from that boot to actually getting to a desktop and even doing things like, can we launch a media player because that would be a good test that we actually didn't install what we thought we did. Hopefully there's the URL there you can go and look and it will show you a thing. So we've done very much similar with QMU with both OpenQA and also Lava for doing various kernel CI type stuff and for project work. So going on to more real world examples of this. So a lot of our work involves real hardware. Customers come along saying we have this all this software we want to integrate it or we're having problems. They ship us a massive thing you put on your desk and you go that's very nice, but I now don't have a desk to work on. And also you're going well we're hand testing this. How do you test this? And the reply is well we have 20 people sat in a room somewhere running through scripts. So they can only generally test a release or to a day. So that's not like you commit a branch and have it tested. So that's not great. So. As part of improving the testing. We introduced a virtualized version of these rigs. It allows us to do not quite, but almost commit level testing. It's easier to scale. When you ask a client can I have another one of these very big things that fits my desk they go. Yes you can have one of those in another two months. And then you just hope that they can build them within those two months and say it doesn't get lost in Holland. Because who doesn't love couriers. Couriers get very upset when you say that they you just lost $20,000 worth of stuff and they suddenly find it. Anyway. Part of we found the of doing these automated tests improves the testing. And we'll talk about a bit about that in this side of the next. So we have these automated test systems that have virtual instances. So we can now use open QA and to replace the testing that manually was happening. The client goes. That was great. We can now do commit testing before we even merge this to our system. One of the big things that we found it does remove human error from this. Whilst humans are very good generally at doing things. They're not always correct. So. Part of this also means you also have logged out from your testing. So you know what happened for each test run. So. Unfortunately and we'll we'll there are some downsides you cannot. Get 100% accurate 100% testing the various reasons and we'll talk about next. You can do things like insert a USB stick into the system and with a known set of media on it and check what happens. Now we one of the issues we found with human testing is that. Say if your humans are mostly European. They won't generally know a lot of the non European script. If science slightly changes and say your sort order on your media. They may not notice it looks like Japanese to them. It may not now be in the right sort order. But that was sort of one of the things that we have picked up as part of this testing. Somebody changed the code that sorted their media. Not picked up. You can also feed in simulated hardware events. So for example if your your rig happens to be a car. You can feed in. Say a known set of test drive information and just check that all the dash happens. Or you can look for very simple issues where maybe you want a warning light to happen. So from experience we have improvements in the way you can scale the testing up. However, as I say we cannot test everything. So. You may not get a completely accurate hardware emulation. But you're usually getting something that's close enough to work. As I said there may not be. 100% accurate device emulation. I mean QM you cannot give you a case where say you plug a phone in. Or that you can do some emulation of that. Well probably not going to be perfect. Certainly. A complete car is probably out of the range of what you're able to do for your environment. There's a trade off between effort of emulation. And what you want to do. So. We're saying that. You can get close to what you need for real life, but you don't have to get it perfect. A lot of the time say. If you're just doing this testing. You can control your startup environment was one of the big things we found. If something goes wrong. It's much easier to pick out what happens in your emulated environment. Then try and work out why you've just bricked. Free rigs that you have to then plug into a windows laptop to reflash. Because somebody managed to break their system these startup scripts. So. Again. Performance is not going to be 100% the same as a real system. You may not have exactly the acceleration hardware, but you know. Things like graphics. You can get fairly good results with that IO GPU and GL pass through. They're efficient and they're well supported by pretty much all the software you would expect to run even on an embedded system. So the hardware support is generally good, but CPU can be a bit slow. Again. It's more about giving a quicker feedback for developer experience. Without having to give the developer a complete system. So I'll go quickly through one of a weird edge case that we found. Sometimes multi monitor systems don't work at boot, which is very annoying. If you're saying working on a system that has three or four different monitors on it and they all need to start up at once. There was some issues around how QMU deals with its graphics state. The documentation in this case was not exactly helpful. And the source is even worse. It's confusing. It uses terms like my monitor display head. Very interchangeably. So. Actually, we just ban. We just really quickly had to fix together. Which was to start our virtual machine paused and then just issue. The D bus controls to connect the monitors as we needed. This is something we would like to fix later, but we really can't. See charging the client for it because. We're working on client time. And whilst the client likes what we've been doing with this is going. That's great because we now have a much better CI pipeline. And we don't have to pay a people like for another 20 people in Germany or something like that. Because, you know. Paying people to do stuff and then still not getting as good flow. So in all in all that that sort of CI. Process has shown to be a generally a win. There are much more. Good results out of it than bad. So. Next. Project that I've. I this is much more personal project for. This is a project I have been working on. So. Client comes to say. We're going to make this new hardware. Can we have some Linux for it, please? And we're going. That's great. We can provide you some Linux. When you'd like to start, well. Start as soon as possible. Okay. Well, can we have some hardware then? No, because the hardware we're making next year. So. Okay. Now how do we test this software? Well. There is a single FPGA system somewhere. So. So. There is a single FPGA system somewhere. That we don't know about buried in some secret lab we get access to. Which is great for developers and one FPGA. So. We've been looking at this and when we're actually. The best way of getting started. Would be to use QMU. Now our first problem that we've got is that. Not all the. Silicon IP may be in QMU yet. Either it's new or somebody hasn't used it. QMU does not. Generally support every single variety of hardware out there. It tries to go more for a virtual machine and then. There are some actual machine emulations. But the. Device models are smaller than say a Linux. System where there would be a lot more devices. Being. Available. So. Our first. Problem was what do we do do we add some new. You do add new device models. Do we replace something. Do we ignore it. So. As in replace we could say okay we'll not use that piece will use this similar enough. Also hope that the IP that we've got includes the CPU. Because. That would be even worse trying to update the CPU cause. So. As part of this project. We decided to start adding our own machine model. So that describes how the entire system fits together. And we run this up and. We had to start debugging because now we have. Not only the real FPGA silicon to try and debug but we also have QMU to try and debug because. Adding things we might find issues. So. Advantage. We've got lot less access problem and we have our full developers doing stuff. Without having to worry who's got the login on this very expensive FPGA. By very expensive I believe it costs more than a. Average family car board. Another advantage is the client goes that was a good idea. We should probably release this at some point. Disadvantages. On this. Again accuracy. We're not going to get a completely 100% accurate model. Is more work. It's thankful that we had some delays in the project where we could work on this otherwise. We probably be having to build a client more money and. Generally unless. They're very understanding they don't want to pay you more money because. Well budget. One of the interesting problems we came across is actually upstreaming. So from both a QMU developer point and a client point. QMU would not doesn't want to accept driver models for machines that don't exist. But the client doesn't want to tell people this machine exists until they got some next year. So we can't say here's your machine owned by the way we wrote five new driver models for this. It also turns out that the client forgot to take. To talk to their legal. People and go. We're commissioning these people to do Linux. All that documentation we gave them shouldn't we have cleared this for release before say now. It's like. Hopefully that's now all sorted out. So the experience. Working both ends of the emulation. From making a device. And then to using your device may not always be great. Especially a small team mistakes can get duplicated. You may read a data sheet one way. You may not have a correct data sheet. There's a saying there's a lies down lies and data sheets. Now something else but you know might be statistics. Your QMU core emulation. May not cover exactly all the use cases. Either it hasn't been released yet because maybe it's not available. Or that the client hasn't got round to doing that part of the work either and then employees somebody else to do it. So. In our case the actual core features were not a problem so we just pick the closest available. For instance. This is another interesting issue we found early on in the boot system. Risk five atomic memory operations. So the operations you do to say make sure a core is synchronized with everybody else. Do not work on all memory types. Which is annoying when you go to your real system and find out that the actual core features. Which is annoying when you go to your real system and find out that the boot block you're using. Is not covered. By atomic memory operations and what the first thing you do. Is an atomic memory operation. You go. Well that way that crashes it works under QMU because QMU. Makes no check as long as it's memory atomic memory operations will work there. Again we're not going to be timing accurate so. Things like say SPI bus timings not going to matter. If you miss a cycle. It may not care. Internal hardware states maybe they're not modeled either. But. So okay doing well for time. So. Thank you. Some of the conclusions so. As I say. QMU may not be totally accurate your vendor may provide something that is more accurate. But it isn't going to be open source. Even though it may not be totally accurate it's probably going to be close enough. In most cases we found that. It gets enough. Coverage to say. We'll catch about 80% of the mistakes. We're having to involve a real human. Again it's also it's much easier to scale something like QMU. I believe in a couple of other presentations they've also shown systems like. Using AWS instances. But again this is going to probably be cheaper than making a lot of your own hardware. Especially at the moment when things are not easy to get. The. Scriptability. Is also very good. Allows you to inject errors control the system. Get state out of it. So. Thank you very much. Hopefully I've just realized that I have not uploaded this. Slide set to the talk to the. Conference set. However is pretty much the same as the one I gave OSS Dublin. Hope this gives people some. Thought for how. QMU could be used to improve. Their workflow or. Used to emulate other systems. Again. QMU is a big enough project. That you could get yourself lost in there very. Easily. I think we've got a little time for questions. Don't know. I haven't. So there was. Is there any fundamental difference to say. Linode. I wouldn't. I unfortunately I don't know. I don't. That's not the. Some some. Linode is not something I've used. So. Again. I'm not saying. QMU is the. Absolute best is just. We found it pretty useful for what we've done. So. I think that's pretty much. Well. Thank you for coming. I hope that that was something interesting or useful. And. Hopefully that you can. Find something. To take your interest and carry on. So. Lovely being here.