 I am pleased to introduce our next speaker, Eduardo Hapskost from Red Hat, who will be speaking about QMU. Hello. Thank you for coming. My name is Eduardo. I work mostly on QMU at the virtualization team on Red Hat. I'm going to talk about internal abstractions and APIs inside QMU. These are the contents of this talk. I first try to explain what QMU does and has to do. I won't try to explain everything about it, but just as an introduction. Then I'll give you another view about some of the internal APIs inside QMU used to perform its job. And after that, I'll try to talk about how those abstractions work or don't work together. That's what's not included on this talk. I'll try to not present solutions or try to say what's the right way to do stuff. I'll just try to describe how things work. It's like an introduction talk, an introductory talk. Believe it can generate interesting discussions later. And also keep in mind that this is not going to cover everything, every single API inside QMU, but just the ones that I think is most relevant. So let's see some context first. What QMU does, what it needs to do to do its job. So how many of you know what QMU is and what it does? How many of you have worked on QMU code? Okay, that's good. So I'll try to make this part quick. So what QMU does? It's described as its website as a generic open source machine emulator and a virtualizer. And it's also a very important component when you're using KVM and virtual machines. And let's see what QMU, and QMU has more interface to interact with the outside world. The main ones are the command line, config files. It has the human monitor protocol. That's a command line interface to control QMU after it has started. And QMP, that's the monitor protocol. That's a machine friendly protocol to communicate with QMU based on JSON. These are the external interface it has. So after this very quick overview of what QMU does from the outside, let's see what it has to do on the inside. Basically, on this slide we see an incomplete list of what QMU needs to handle internally. Among other things, it has to keep track of configuration options, handle monitor commands, keep track of device configuration, device state and backend configuration, and other stuff. But each of the APIs we'll talk about will be used to solve one or more of those problems. Let's go inside QMU then after I try to make the introduction very quick. The first API I'll talk about is QMU Ops. QMU Ops is a very old API introduced in 2009 to handle QMU command line options and config files. It has very few basic data types and it has basically a flat data model. QMU Ops is basically used to parse many of the command line options inside QMU. If we look at what I think is the most relevant one, it's used to handle the most of them. Also, even when a command line option is not parsed using QMU Ops, it's used as a style system for command line options. So we could use the same infrastructure to handle config files and command line options at the same time. So let's see how it looks like. This is one example, a simple example of QMU Ops in action. We can see here QMU command line and the declarations of the code that make it possible. What's most important here is the desk session. I forgot to say, if something is unreadable here, you can see the slides on the Fossum website. These slides are published there. So it describes the options accepted by the dash memory option. It's very simple. The next API I'll talk about is QDev. QDev is the best and the vice-tree hierarchy system inside QMU. It allows us to provide a generic interface, external and internal interface, to create, configure and plug devices in QMU instead of having different APIs or different command line options to each device type. And it has a property system that allows configuration and introspection of devices. One thing that's important to keep in mind is that QDev was introduced in 2009, but when we introduce QOM that I'm going to talk about later, the QDev abstractions will basically reboot on top of QMU, but I'll talk about that later. So QDev is quite successful inside QMU. It's used internally to create and configure basically every device inside QMU. And in addition to its internal usage, it provides generic command line and monitor interface to handle devices, plug devices, and its hierarchy and property system also allows users to introspect and pick what's inside a virtual machine, for example, using the InfoQtree device. So let's see QDev in action. QDev does lots of stuff, so I won't explain every detail here because the focus of this talk is to see how stuff works together. So this slide is just a small sample of the property system in QMU. We can see here how the properties of a device type like E1000 are declared inside code. You have boilerplate code to define device types, define how they do it. It's just a sample of how it is used, how things are declared in the code. The relevant parts here, RGD, for example, the MAC option is defined inside an array. It's declared here. It has many data types and many helper macros depending on the data types of properties. This is something also provided by QDev. So QDev allows you to see the whole device tree. InfoQtree is going to show all the buses and devices inside QMU, and this is all handled by QDev. Next API, QAPI. So QAPI is a system for defining QMU external interfaces. It uses a JSON-like language to define data structures and interfaces like QMP comments. That's the monitor protocol I talked about. And it provides a visitor API to implement data input, output, conversion, and so on. And it's based on a QAPI schema. It generates lots of codes for declaring C types corresponding to those data structures. Civilization, visitors, visitor functions, QMP comments dispatching, interfaces inspection, and documentation. QAPI is very successfully used inside QMU to define and dispatch all QMP comments and to define and parse a few command line options. So every single data structure used in QMP is heavily documented on the QAPI schema. So it's a great system, and we have been trying to use QAPI more inside QMU when defining external interfaces. So let's see QAPI in action. This is a QMP comment. Let's try that app. It has a few arguments. One argument is ID, and another one is backend. These are the actual QMP messages being exchanged. The response is empty because there are no extra data being returned. This is how it's described in the QAPI schema. Basically, I removed all the documentation comments. But basically, data structures are all defined. You have declarations to declare how QMP comments look like, what they get in an argument, what they return. This comment uses a feature that's called a union, so you can have different attributes depending on the type of option you are using. And this is the function that's going to implement the comment. So the QAPI system is going to take care of serializing everything, calling the right function using a simple C script, and then returning the data later. So let's, but not least, we have QoM. QoM is the QM object model. Don't confuse that with QObject. That's something else I'm not going to cover in this talk. So QoM started as a generalization of Qdev. It has lots of interesting features. I won't try to explain everything about it. It would cover a whole talk. So basically, it has a type hierarchy of classes. It has a property system. And basically, all the distractions in Qdev will be built on top of QoM. So many of the Qdev codes are just wrappers around QoM. And let's see QoM in action. These are just a few examples. So I won't try to explain how QoM works inside, but I'll just see where it is used. It's used when you're using Qdev. When you're plugging a device, Qdev is built on top of QoM. It's used when you're creating a backend object file, that can represent some entity that's used to implement VM. It's used by the machine type system in QoM. It's used by the accelerator system. It's used by the CPU model configuration system. And it is also used internally. For example, we have data types for IRQs, memory regions. So it's used at lots of places. We benefit from the property system when we are dealing with options on the common line, when we are dealing with CPU options, or Qdev options. And probably we have lots of other places where QoM is used that I'm not even aware of. So it is everywhere. I see that. Now that we have seen an introduction to each of those abstractions individually, let's see how they work together. That's where things get interesting. First example we have inside QoM is the dash-numer-comment-line option. It's one case where we mix QoM-Opts and QAPI together. Let's see how it does that. This is how the dash-numer-comment-line option is defined on the code. Note the empty desk field here. That's where the accepted options for the dash-numer would be defined, and it's simply empty. Instead of defining the options here, you use the QAPI schema to define that. So all the options on dash-numer, let's go back here. These options here are defined on the QAPI schema. So basically you have a declaration saying that a special name and numer-node options can have those fields, and those fields can be specified on the command line. And we have some blue code to convert the QoM-Opts data to QAPI data, so they can be used elsewhere on the code. That's used by these three here. So we have the visitor API provided by QAPI, and it can be used to convert QoM-Options to QAPI data structures and validate them. So let's summarize this case. Numer is fortunately in one case where things are working as they should because all the options are specified on the QAPI schema, and there's no duplication. There are two options that are on the QoM-Opts declarations and the QAPI schema, and this is made possible by the ops visitor helper. All the command line options that are very similar are dash-net, dash-net-dev, dash-ACPIA table, and dash machine. So this is basically an example where things are working. We can make sure that QoM-Opts and QAPI work together. Let's see another example. Let's see one case where QAPI and POM are used together. This is the object add, QMP command. It's used to create a backend object, basically. It's a generic system to create objects inside QoM-Opt, and there's a QMP command to do that. It has to take a few arguments. It takes the type of the object and the additional properties for the object. The properties accepted by the object add are defined basically by the object itself, the object class itself. So the fireman arguments on the command here are declared not on the QAPI schema, but declared at the QoM code by the RNG random class, because we are using the QoM property system to declare which properties the backend object accepts. And the QAPI schema looks like this. So we have a command called object add. It has those two arguments, and also an additional argument that can take any data type. So basically it means that the QAPI schema says you can put anything here. What happens here? So in case of object add, the QAPI schema is incomplete. You don't have the QoM data appearing on the QAPI schema, so you don't benefit from it. You don't have the type checks provided by QAPI. You don't have the generated code that you could use to implement the interface. And we have a very similar problem with device add. It's even a bit more complex on device add because of the way the properties are defined on the schema that I want to get into that. So in this case, the abstractions don't work perfectly together. We have some limitations here. Let's see another example that's based on QoM. That's dash CPU. So the CPU model sees an inside QoM uses QoM internally because basically CPUs are devices. Devices are Qdev devices, and Qdev is implemented using QoM. And we have command line options that are going to control the CPU devices, basically. So let's see how it works. This is an example of the options accepted by dash CPU or how they are specified. In the case of CPUs, each architecture registers their own QoM properties the way they need. So in the case of x86, we have a few static properties using the Qdev property system. And we also have some properties declared registered at runtime because we have lots of CPU features configured at least on x86. I don't know about the details on other architectures. And this is the group code we have. The group code is going to take the command line options. It's not using QoM ops. It's not using its own parser, but that's something we want to change. And using the Qdev global property system, that's something provided by Qdev that ensures that every device of a given type is going to have a few properties set. So basically when you use dash CPU, the CPU features declared on the command line are going to be declared as Qdev options. So the summary for dash CPU is we don't even have a QAPI schema for dash CPU. It's simply not very well documented. So you don't have any information required to know which command line options are supported by dash CPU. If you want to know that, there's no interface to do that. And it has an additional problem that it does not even use QoM ops to do that. But that's something, as I said, that we need to change. So this is one example where we see the friction between those interfaces. The QoM-based stuff being used from the command line is not very well integrated. Let's see another example, even more complex, related to dash CPU options. That's query CPU model expansion QMP command. So this is one case where we have a QoM-based implementation and a QAPI-based interface. I won't try to explain what the command does here, but basically you get a CPU model name and you're going to get information about it on the props attribute. About all the features inside the CPU, all the properties for a CPU model. And this is the QAPI schema for that command. Like on the object ad case, we have to use this hack here that basically says the props field can have anything. So the semantics of this props field is documented on the documentation for the command, but the schema doesn't tell us which properties can be there, what's the data type, what you can expect there. You need to read the documentation to do that. So summarizing this case, in the case of query CPU model expansion, the QAPI schema is also incomplete. It doesn't have all the information we want. And it has an additional problem that the group code that will make sure the QoM implementation can report QAPI-based data is specific for each architecture of CPUs. So we need to clean up that to make sure we use common code to see how we can get common infrastructure to that. There's a pattern here that can be summarized here, is that when they're using QoM, you'll normally have trouble when using something QAPI-based because QoM classes and properties are registered at runtime, especially QoM properties. They are not a static file in the code. They are added at runtime when you're initializing QoM. But the QAPI schema is a static file. So you simply can't represent any runtime information on the QAPI schema. So that's why we have these issues with the form being used with QAPI. So I think I was faster than I expected. So concluding that, my first recommendation here is that lots of practices inside QoM are not documented anywhere or not very well documented. So this talk is an attempt to write down some of the stuff that we have learned in practice, but you have any question, don't hesitate to ask on the QoM development in this. Because sometimes we are still discovering where we want to go, what practices we want to follow. So it's good to have these things explicitly discussed on the mailing list. I think I'll try to, just to show a summary as we have a few times. Okay. So this is something I had as bonus material if we had time. Just to summarize how the data types represented on each API looks like. So most of the APIs, all of them involve some kind of data representation. And another issue we have is that they don't support the same data model. QoM is the most complex, complete one. It supports all the range of data types I enumerated here. But QoM ops and QBF are more limited. QoM is also a bit more limited, but it has some limited support to more complex types. So this is one source of the friction we have when we are trying to make one abstraction talk to another one. Now something else I tried to summarize. Okay, this is very hard to read, but basically on all those abstractions here we have some equivalent data structures. We have often some type or option type, some option type defined. We normally have a property system say which attributes can be set for each option type, device type or schema data structure. On most of these cases we can have mechanisms to define defaults for them. And then we have the runtime data structures. They are basically the options themselves, the device themselves, or the actual C-struct runtime data. One difference on this case that makes it difficult that I tried to explain before is that some of the data that is static, for example, for QAPI or that is static for QBF is actually not static for the other system. So QoM properties can be registered at one time and you can easily translate to the static data on QBF. The defaults for QoM ops and QoM are also defined at one time so you can't just convert to the other data structures later. So this is another source of friction. Questions? Comments? You can ask. So Eduardo, is there any plan to get rid of some of these layers to do something about all this? Because there's a lot of duplication and it's also causing bugs and it causes a lot of effort during development. You don't know what you should do. Sometimes the overlap causes problems as well. That's a good question because the difficult part of this question one is that sometimes we are not sure or we are not on agreement of where to go. But I can say what I know or my impressions. My impressions are that instead of removing those layers we can try to make them thinner. For example, QoM ops today have a way to define those options and we have a way to define common line options using QoM or QAPI. If we could move all the common line options to be defined using QAPI or QoM then we could remove the option definitions from QoM ops and have only parcel. On the other case of QAPI and QoM I think they are going to leave there and we just need to make them work together. I think we won't get rid of QoM or we won't get rid of the runtime registration of properties so we just need to make sure they work together. There is some work to describe common line options in QAPI, in a language like the QAPI schema but I don't know the details of how it's going to be done but we are trying to do that. So I don't think on the case of the abstractions I showed I don't think any of them is going to be removed in the near term but we can make them be rebuilt on top of the others and make some of the code inside them less necessary. Qdev is one example. So Qdev was basically rebuilt on top of QoM but it still has some distractions that are not inside QoM but we can work to make Qdev reuse QoM code instead of duplicating. So I don't think we can remove the layers but we can reduce the duplication between them by reusing the semi-frostructure. One example are the parsers. When looking at the code for the talk I found out that we have at least three different parsers three different parsers for interval numbers for strings so you have a number on the string you need to parse it. There are three different implementations. Each one with different bugs and with different limitations so I think we can reduce duplication not by removing layers because it's going to be a huge effort but by making them benefit from each other and reusing code from each other. Anybody else? So thank you. Do you have any recommendations on how other projects could avoid this mess basically on the long term? Sorry, I couldn't hear everything. You've got multiple APIs with overlapping functionality and whatnot. Do you have any insights for other developers working on other projects and how to avoid depending on that? I don't have any advice but I can say what I think went wrong. For example, I think some of the new abstractions were added without a clear plan to replace other ones. So when we introduced QOM and tried to convert all the existing devices to use QOM or QDAV there was no clear plan to say, okay, we are going to convert these devices and these ones and we are going to keep these and these abstractions and remove that one. There was no clear plan to do that. When we started using QAPI for some of the command line options there was no clear plan to say, okay, this is the process we are going to follow. We want to convert the old ones to the new QI so I think that's what didn't work on QOM. There was no clear plan. Yes? Adding another answer is having test cases ready from the start makes any conversions absolutely easy. That was one of the issues that we were facing. Anybody else? Okay, thank you. So the slides are available there and if you don't want to write down the link you can just get these slides and follow them. There's a blog post I wrote about with an introduction to QOM APIs with links to other ones and I have been trying to write a summary of which one so if you want to follow it. Thank you very much.