 Hello, my name is Aurel Esteve and I'm an engineer at Red Hat. For the last eight months or so I have been in the automotive division, so private product. One of the things that I have been working on has been developing a new build-a-yo backend device for Kyomo, specifically build-a-yo video. To be completely honest with you, I did not know a lot about build-a-yo devices at the beginning, so I had to read the documentation and teach myself a few things and I wanted to share in this presentation a little bit of that learning experience with you, specifically focusing in the build-a-yo data flow. So this is the agenda for today, but it's a lighting task so you will be there soon enough so I will jump to the map because I do think that every good book has to start with a map and this is our map. This is the system architecture that I will be assuming for the rest of this presentation and you can see that we have QMU plus KBM running in a Linux-based system and then we have a guest there as with a Linux-based system and the goal here is to have this hardware device that you see at the bottom exposed in the guest. And to do that we will be using build-a-yo devices. Build-a-yo usually is a detector in three main parts. The first part being so build-a-yo here is the red boxes that you see here and the first part is the build-a-yo driver that is in the guest kernel and the fact that the build-a-yo driver is in the guest kernel is the reason why usually build-a-yo is referred to as a para-virtualized solution because as you can see the operating system running in the guest needs to be aware that it is running in a virtualized environment and the driver will basically take the communication that received from the guest user space and then communicate with the hardware device. But it will not communicate with the hardware device directly. For that we have the next piece of the build-a-yo detector which is the backend and we are using for the backend because user device because for our own use case we do care about performance and this is the most performant model in the BTO backend and then the BTO backend what we do is to receive the communication from the driver and pass on the communication to the hardware device directly. And then the last part of the BTO architecture is the transport layer so what we have in between the front end and the backend and it is fit in two parts. The first part being the control layer that goes through a unique socket and then the next part is the data plane that goes usually through a shared memory between the guest and the backend and in the end they are composed of what we call build-a-yo rings. And with that we can go to just launch the QEMO simulation and while it starts, while it boots the system we will see how build-a-yo devices are imitialized which is usually performed through the control plane and the first thing that we will do is exposing the device in the guest machine which is typically done through PCI or PCI express parts using the PCI autodiscovering mechanism so that when they are acting during the boot process build-a-yo devices will identify themselves with the PCI vendor or device IDs and then the guest kernel will use these identifiers to select the correct driver in the kernel which will be the build-a-yo driver that matches the device and with that we will expose the device in the guest. There are a few other alternatives but by far the most used is the PCI or PCI express boss alternative and that's what we use in our use case. Then the next step, once we have that, is the feature negotiation so different backends have many different features and QEMO needs to be aware of the features that are supported by the backend side to know how to talk to them so for example we have the config feature and if the backend doesn't have it QEMO cannot send a get config message to retrieve the backend configuration so it needs to know which features are supported by the backend so they need this feature negotiation which is initiated by QEMO side and QEMO will send the features message to get the bits that are supported by the backend the backend will fill the bit array with the features that it himself understands and then answer to the QEMO message and then once QEMO receives the response it will match the features with the ones that he can support I put here a few examples for bit.io features and because user protocol features take into account that there are some generic and some specific ones I put a few here specific about bit.io GPU and then this backend sender that allows backends to send file descriptors back to QEMO but there are many many features both generic and specific the next step in the initialization is the bit.io configuration so this is for the data plane, it's a shared memory so what QEMO will do is allocate memory regions as shared huge spaces as it boots and then it will pass these shared huge spaces as file descriptors to the backend which will map into their own address space and then after that so they can both see the same memory regions the bit.io driver will reserve part of this memory for bit.io so bit.io's are basically like logical queues where the sender can write their message the command that one the reader the receiver to read and they usually have different logical meanings so that you know what kind of messages are you retrieving from specific queues and again same with features that are some generic and some specific like we have here the control queue or the event queue that are generic for most of the devices but not all but most of the devices but then we have some very specific bit queues like the cursor queue, tx and erics that are for GPU and socket respectively and with that we can log in and we should provide some available so our QEMO can start for the data so in this part of the presentation I'm going to focus in my experience with bit.io video and as I said we have already the device accessible from the guest and we will have an application running in the guest user space that want to interact with that device right in this case we can assume that it's just streamer and let's say it wants to start at the coding stream and then it will create some buffers and then it wants to queue them so for doing that it will send this BDSC queue buff IOCTL filling with some data that is required by the driver and then the driver will handle the system called the IOCTL taking into account that some operations that depends completely on the implementation of the driver are handled by the driver or completely ignored in this case this queue buffer will actually trigger an interaction with the hardware and when that happens what the driver will do is take the information that has been through in the IOCTL and write a command in the bit queue that is specific for this type of communication and for the queue buffer command it will send this command receives queue will write it in the shared memory specific bit queue and then the backend will be able to read it taking into account that the format is defined in the specification document it varies for each bit queue device so this command resource queue is only for bit queue video devices different will have a different set of commands and the format is agnostic to the driver or device implementation by definition so that you can have multiple implementations coexisting between them as long as they are there to the bit.io specification they should be able to understand each other and then we left it where the driver brought the message in the bit queue and then it will send a notification and the backend will be able to read it from the bit queue and then it will reconstruct the bit queue buffer in the command the information should be connected in the bit queue that the gstreamer sent in the guest user space and then the device driver in the kernel will handle the request and interact with the hardware it will receive the ioctl and then we have this final interaction here and we have arrived to the hardware device and that would be in most cases but in some cases we need a response that is defined by the bit.io specs not all commands do, in this case it does these responses can be synchronous or asynchronous again in this case specifically it is asynchronous so what we are doing is queuing this buffer in the hardware device and once we start streaming the hardware device will fill the buffers and once they are filled it will launch an event upwards and once the event it will pick it up and send the response back to the first bit and then to the guest user space so it's exactly the same path but backwards and that's the end of our journey I think we have a few minutes so I will give you a few hints for debugging completely based on my experience so the problem with this is that it's something fails or the communication misbehaves it's not easy to debug because the problem might be in the guest user space in the communication between the front end and the back end or the communication with the back end and the hardware device so it's not really that easy to debug so what I would say here are a few points first get from the back end even if you are implementing the back end or focusing on the back end it's good to know what the driver is doing then also that depends on the back end but in QMU most back ends come with a dash dash debug option use them because they have a lot of extra information that are useful then also these two commands strace strace I assume most of you are familiar with them but if not they are very useful to figure out what's going on strace is for signals and strace is for inspecting the function flow in the kernel trace is for Ftrace so they can be very useful to know what is going both in the guest or the host so use them as much as you can then you can also run what I said we run in the guest user space you can run it in the host to and trace it to know that you are doing the same thing or the right thing in the back end then another good tip is use QMU monitor and GDP debugger this is this blog post written by Stefan and he gives very good hints to use these both tools if you are built that your back end gets stuck somewhere you don't know where you can check this blog post and try to figure it out and not just patience and good luck and just to wrap it up these are a few links that helped me when I started with it and they are all good reads including the built IO specification the last one that has been obtained which is 1.2 I think 1.3 is due for this next summer and the last link is the page that I posted in QMU for the built IO video so if you are curious check it out please do and that's it from my side any questions? yeah, so the question was that whether the let me see if I can phrase it correctly if the built IO driver is a kernel module that interacts with the hardware device in the host so yes and no I mean it does interact with the hardware device but it does it through the back end so it needs another built IO piece which is the back end so you can consider it the front end and then it can send commands to the back end through the share memory and the back end will interact with the hardware directly yes, so the question is about the benefits of externalizing the back end into a different process that's a difficult question I mean not difficult but it can be long to explain but long story short it's about context switches like if you have for example the back end embedded in the QMU process every time you do a system call you need to do a context switch to the kernel and then back again for the data and if you are like the device is writing data into your buffer you're gonna have to wake up QMU and it's a tough beast so that's why it's better to externalize the back end to avoid these switches but which Qs are you referring to? sorry so the question is whether the Q used in the back end is somehow limiting or could be a different option I think it's not limiting like you don't write that many the communication is not that fluent between the front end and the back end you just write a bunch of messages there and then you have a ring and you are looping through the ring with the messages so I think it works pretty well for what it has to do I guess I never gave that much thought into why it was done that way