 This talk about two of USB device controller UDC in the Linux system. So just a few words about me. I am an embedded Linux engineer at Butlin. I work on the Linux kernel and also on butloaders, such as Reboot. And I contribute to the re-nizaziz N1 USBF UDC controller in the kernel. I live in Toulouse, south of France. So I will start talking a little bit about the USB standard. And I will just focus on the USB 2.0 standard, not the USB 3 or even the OTG part, the really core USB 2.0. So the USB 2.0 standard is quite old standard. It was released 23 years ago. And it defined everything about the USB from the mechanical part to the electrical part and also the communication protocols. It is publicly available at usb.org. And it supports high speed, full speed, and low speed devices. So the USB bus, we have a host with its root web. And we can have multiple devices using some apps. Devices are a pluggable device and discoverable device on the USB. Each device has a uniquely assigned address in order to communicate with. So the communicative flow or USB transfer are initiated by the host. Most of bus transactions involve up to three packets. The first one, the token packet, the first packet in the transaction is here to identify the transaction and the reception of the transaction using the USB device address and the endpoint number. You can have some data packets, carry on the data related to the transaction. And we can have some empty data packet and they are called the Verwallance packet. We have also the upper limit, the max packet size. In fact, it is the limit of the data packet. And it is attached on each endpoint. Each endpoint has a max packet size. And the third packet involved in the transaction is the uncheck packet. Hack, NAC, install in order to indicate whether the transfer was successful or not. So let's have a look at the standardized transfer. The transfer defined in the USB, the first one, the bulk transfer. So this transfer, the system guarantees the delivery of this transfer. But there is no guarantee in terms of bandwidth or latency. They are present in both directions, in and out. This direction in or out is from the host's point of view. So in is from device to host. And the other way out from host to the device. So let's have a quick look on the pattern here. So we have the in token, the in token. Then the device can answer with the star if the endpoint is added or if an error is detected. And NAC is more to indicate that we are not ready to set the data and so please retry later. When we have the data, we answer the in token with the data packet. And in return, the host acknowledge the transaction. The same from the other direction output with just the host sending the data packet. Another kind of transfer, the interrupt transfer. So in this case, there is a maximum service period guaranteed. And on any failure detected, the transaction will be retried on the next period. In fact, interrupt is just a periodic pulling from the host. And you can see the pattern involved exactly the same as the bulk transfers. Isochronous transfers, there is in this case, in this case, guarantee in bandwidth and in the data rate. In fact, isochronous transfers are the periodic transfers in the USB world. And also, you have to know that there is no retry in case of failure. And even there is no unchecked packet in an isochronous transfer. The last one, the most complicated one, and the mandatory one is the control transfer. This transfer is used for any configuration command, some status on the device. And it is made of three stages. The first one, the setup stage, made of the setup token and the related data, is here to identify the kind of request asked by the host. Then you have a status stage. For instance, a control wheel, the status stage, the host requests the data, then the device set the data and on the control right, the opposite side, the host sends the data. The last stage, the status stage, is here in order to acknowledge all the previous stages, so the setup and the data. And we have also some control transfer without any data stage. So we have only the setup stage and the acknowledge, in order to acknowledge the status stage, in order to acknowledge the setup stage. So I use the word in point, it's now time to define what is an in point. You lie, okay? No? Is that better? Yes. Ah, cool. So the end point is simply the terminus of communication between the host and the device. It is uniquely referenced in the system using the device address, the end point number and its direction. We have one mandatory specific end point, which is the end point zero, and it is used to control the device. It is mandatory because there is a lot of requests defined in the USB standards that are through this end point. We have the notion of interface, interface is simply a group of end points providing a function. We can receive our interface is available at the same time on a device. For instance, a multi-function device, a printer scanner device, we have one interface for the printer part and another interface for the scanner part. The configuration, it's much more the device capabilities. We have the power budget, the remote wake-up support, the number of interfaces present in this configuration. So one or more interfaces can be present in the configuration, but only one configuration can be activated. Whoops. So I told you the control transfer and through end point zero and the USB defines some of this transfer. In fact, on the left side, I extract the setup definition, the setup token associated with the data for the setup stage and the data exactly mapped this definition. So to identify the request, we have the request type and the request. We have a possible value, we have an index, and a length of the data stage, the data present in the data stage. So if the length is zero, there is no data stage. On the right side, I extract from the standard some standard requests such as script feature, get configuration, descriptor, get status, set address, quite important, set configuration, important too, and some orders. The USB defines also some device states. So we have a quick look at them. A device can be in an attached state. In this case, it's just plug-on bus, but there is nothing more than just the connector plugged. The bus is not powered, so the V-bus on the bus is not present. Another state when the V-bus is set or asked to be set to the hub by the host on the bus, the device is now in powered state. In this state, it's not allowed to answer any transaction. It has to wait the USB reset to start unlinked transaction. When it receives this reset, it switches to its default state. In its state, it can only answer to the default USB address, the address zero, and only on its endpoint zero. The host has to assign the unique USB address. To do that, he used the set address request, and when the device receives this request, it switches to the address state. In this state, the unique assigned address can only reply to a transaction using this address, and only on endpoint zero. The other state, the configured state, the device receives the set configuration request. In this state, all functions are available. The function presented in the interface is available in the configuration chosen by the set configuration. All of these functions are available. The device answers all transactions on the endpoint and using the previously assigned USB address. To spend its state, just a few words, the device ends up in suspended states when it detects no bus activities during a specific period of time. And it automatically releases the suspended states when it detects some bus activities. So a quite important step at the beginning of a live device on the USB is the bus enumeration. The goal of this process is to assign an address and configure the device. So all of this enumeration is done through standard request. And this sequence is just the device is plugged. The bus is detected. The device connects the pull-up data line resistors. This connection is detected by the app and informed to the host that there is a new device on the bus. The host asks for the app to enable the bus. The port, the device is plugged on. And to reset that port following the reset, the device is in default state. It answers to the default address. The host assigns the unique address, the set address request. The device uses this address. The host gets more descriptor and quite some more stuff. And at the end, send the set configuration request, switching the device to this functional state, I would say. So that said, let's have a look at the UDC part. First, the UDC is part of the USB gadget. But what is the USB gadget? USB gadget is simply Linux at the USB device. So it is composed of several parts, the gadget core and all the common stuff. The upper layer providing the function. I will not talk about these upper layers. Their goal is to handle what is function-specific. There are also some requests on endpoint zero. They are connected to all the sub-system in the kernel, such as serial, sound net, and so on, to provide the function. Here in this talk, we're going to look at the lower layer. The UDC part is, in fact, the hardware abstraction. The UDC drivers are to manage the hardware, of course, under some set-up request on endpoint zero and perform all the transfer asked by the upper layer in order to have a functional USB. So it's true, we have to include gadget.h, which in turn includes ch9.h, it's quite funny name because ch9 is chapter nine from the USB standard. And this either defines the structure, the enumeration, all the important value is used from the standard. We have to provide some hooks for device management, some other bunch of hooks for endpoint management. And of course, we use some of the API, USB core, gadget core function in order to register the driver, signaling some events, and so on. So let's have a look at the gadget operation. You can have the pull-up hooks, UDC start and stop hooks. UDC start and stop is called by the core when we're going to use the UDC driver. So during this group, we have to do what is necessary at the hardware level in order to be ready to work. And if possible, we start the VEBAS monitoring, the connector part, UDC stop, the data to stop the UDC. Pull-up, pull-up, the gadget core asks us to connect the data line pull-up. So when it has to activate, we connect the pull-up. But be careful, this connection is detected by the app and then the host. And it's the beginning of the USB activities. With this pull-up connected, we are going to start the bus immigration from the host side. So be ready to handle these activities. On the other side, when it's deactivated, the host will detect this connection of this device. And so it will be no more used from the host point of view. The hooks relate to endpoints. So enable, disable, unlocking a request, free the request, choose and issue a request, and choose a function in order to control the endpoint. Enabling and disabling an endpoint, the endpoint chosen by the court in order to perform the function is chosen among a list of available endpoints provided by the UDC driver. We go that a bit later. So on enable, the endpoint had to be set up based on the endpoint descriptor passes parameters. We have to configure that in order to handle this endpoint. On the disable, the endpoint will no more be used from the core. So the core has to disable. Now that we have to complete all pending requests, giving the request back to the USB gadget core, sorry, after disabling an endpoint, the endpoint is no available anymore. Endpoint zero, the specific in particular one is never enable or disable. It's always enable, sorry, enable, disable. In fact, these hooks will never be called for endpoint zero. Some control operations set out and set wedge. So set out, set or clear the endpoint and feature. An added endpoint will return a star on this transaction. You can do anything more. I am added, so star. The host can use some request to get the status, the endpoint all status, and also to clear the feature if he wants to. Set wedge is pretty the same except that the host cannot clear the endpoint using the hard feature. Only the gadget core using set out zero can't clear the out state. Up. I thought about request. So request from the USB gadget work and simply that exchange using an endpoint. They are changed using a queue parent point. And depending on the direction, for instance in endpoint device from us, again, the host point of view, they contain the data to send. So one request will spawn on one or more data packets depending on the max packet size. And their length packet can be added if needed in order to terminate the data on the USB bus. On the other direction, the data request, we have the data received. So we receive data packets and merge them to the request up to the request size. And if we receive a zero length packet or a short packet, it terminates the request. Data packet receive also cannot be split to have a bunch of the data on one request and the other part on another request. All the data received from one packet has to be in one request. When the request is terminated, we have to give it back to the core. So complete the request. Simply calling the USB gadget, give back request. So the endpoint hooks related to request. We can or we have to allocate a request. So it is a time to set up some more hardware data really close to the request, such as DMA buffer, for instance, an allocated request can be used several times. I mean, it can be queue and complete and re-queue and re-complete, so several times, using the same allocated request. Free request, the counterpart, just free the request. The request will be no longer used. So release the hardware specific resource and remove what it was done during the alloc request. I said that the request can queue parent point for processing. So when the operator wants to process a request, it's just called queue with the request. The requests are automatically removed from the queue at the end of processing by the UDC driver itself. Also, at the end of processing, this end of processing is signaled to the core using the give back request. And don't forget when the request is to queue, start the processing of the queue if it's not already done. On the queueing, the core asks us to dequeue a previous queued request. So we dequeue the request. We have to complete the request using the give back request with the status icon reset. And also don't forget, if we dequeue the first one in the queue, don't forget to start the processing of the next one. We have not never stopped the queue processing, but just dequeuing one in time in the queue. Some of the core API functions we use to the classical one in order to register on unregistered driver, notifying the core about some specific event, the variables event, and USB reset, set the gadget state, give a request back to the core, complete a request. And one other, the setup hook in the USB gadget driver it's quite important because it is the one we're going to call in order to delegate the endpoint zero handling at the core lever, core or ever going upper in the layer at function level. So my simple driver with this data, bunch of data related to endpoint, my private data, my UDC, with the array of endpoint, my bunch of rules or operations related to endpoint and the one related to the gadget. And I need some endpoint information in order to give the list of available endpoints to the core. So this will be used during the probe and we have to be careful the endpoint name it's quite important, you have a specific format it's EPN something with N, the endpoint number that will be used for the transaction on the USB, the endpoint capabilities, saying what kind of features endpoint can support in terms of transfer, control back and so on. And the max packet size limit, so the maximum value possible for the max packet size or the upper limit of the max packet size value. Based on all of this stuff, it's time to probe. So we have to analyze some of the gadget fields, the name, the max speed, the operation and analyze the available endpoints. So we start with a disabled state, it will be enabled later. We initialize the request queue, some endpoint fields, the endpoint capabilities, the max packet size limit, we set the specific endpoint zero, my UDC gadget endpoint zero is my endpoint zero and we build the list of available endpoints for all the other endpoints. Once all is done, just register the UDC driver. I tell you about some specific events on the bus, the verbus, so the power supply and the USB reset. So if a monitor verbus changes, you have to signal the changes to the core and in turn, the core will call the pull up hook in order to activate or deactivate the pull ups. On USB reset, we have to complete all pending request if needed, giving them back to the core. Not so that the speed is negotiated during a new USB reset. So it's really time to get the current speed of the negotiated on the bus. We have to reset the address to the USB default address, signal the reset to the core, and also after the reset, only endpoint zero is available for any transaction, no other endpoint zero. So putting all together, we have the core one to start. So we call the USB start, we're going to monitor the verbus. On the other side, the host, power on the bus, so verbus is on the bus. We detect the verbus and signal that to the core using USB UDC, VBUS and LAR. In turn, the core called pull up. So we prepare for the USB reset and connect the pull up. The pull up are connected, detected by the host, so the host starts its enumeration process, sending the reset. The reset is detected, is on the bus, we detect the reset, we switch to the default address and call the core for signaling this reset. The host continue its enumeration process starting on endpoint zero transaction, some control transfers, they're on the bus, and we handle some of them or delegate them at the core level. So how to handle the endpoint zero control request? Some of them, I fully handle the UDC level. There are the set address, the unique address, the set and get status at device set, get status, so is there no set status, get status at device and endpoint level, set and clear feature at device and endpoint level. All other requests are delegated to the core. In order to do that, we have to call the setup function from the driver's structure, giving the gadget information and control request. The control request is simply the data available in the data packet of the setup stage on the control transaction. So the core performs what you have to perform and queue one request in the endpoint queue for the data stage or the status stage. If the data stage is needed in or out, this request queue is for this stage. If there is no data stage on this setup and status stage, this request is for the status stage. Also, it can return USB gadget delayed status. In this case, it just tells us, hey, I need to queue a status request for the status stage, but I will do that later to the UDC driver while we have to process the queue in or out if it needed and don't forget we have the status stage. So at one moment, we need to send and receive the RedLeverLens packet related to the control status stage. For the other endpoint is pretty, pretty simple because everything is done under a player, the core and the function level. So we just perform the data transfers without having any questions about them. Just process the endpoint queue according to the endpoint direction. A few words now about how to test all of this stuff. So I use TestUSB, it is described by LinuxUSB.org. I also wrote a blog post at Berlin related to these tools. And these tools is quite old tools, but give some results and pretty interesting to use. And it's many executed on the old spot. Old spot, I mean the test tooling, while we are a device, we need the host to communicate with us. So the old spot is just the test tooling in this case. And on the old spot, there is a dedicated kernel driver, USB test and a user space program to ask the driver to perform some tests, just USB. We have to note that the USB test can hang on some failures. And in this case, a reboot is needed. In fact, the first time I wrote a lot of mistakes in my driver, so USB test was really flying somewhere. And so it crashed my system. So be careful, if you use it on your workstation, you may be going to reboot. On the target path, where you have our UDC to test, we have to use, for instance, a gadget zero is pretty sufficient to perform the test. And of course, our UDC, something test. So on the target, just load the gadget and not on the host, load the module, then ask for some tests. We can also perform some tests on the UDC that are supported by the gadget zero. And in this case, we have to load the USB test module using telling it to use the alternate setting. So alternate equal one. I use it so some pre-composed gadget. For instance, I use the master wage gadgets. This one is pretty interesting because it holds some endpoints. So it's really useful to test the odd feature of the endpoint. I also use the ethernet gadget. This one uses transfer size that are really not a multiple of the max packet size. Also, it is very useful to test so the transfer has well spun over multiple packets that the last packet cannot be, is less than the max packet size. All of this kind of stuff related to some kind of size alignments. And I use also the gadget serial because in a pretty basic configuration, each by itself is equaled. So it is really easy to test pretty short packets and easy to isolate some transfers. So a few words from the master wage. If the odd feature is not well implemented, it's really easy to see errors appear. So on the target, create a file back end for the master wage and allow the gadget. On the host, a new USB removable device appeared. So play with this disk for matting the disk, transferring some data to the disk and so on. If there is no error, it's quite good. If there is error, look at the odd feature. The gadget ethernet. What's interesting is you have one UDC USB request per ethernet packet. And also when the UDC completes the request giving it back to the core, the request size is exactly the ethernet packet size. So on the target, load the gadget, maybe a little bit of configuration to set the IP address and then the W get some files from your host and the host run a web server. Look at the ethernet transfer size using first for your shaft instance and be sure there is a matching in the ethernet size and the USB request size when the request is complete. The gadget serial. So each correct type on the host is equaled and when you enter, the world buffer is equaled. I do that on the target, loading the gadget and I did a simple loop back. Cut the device redirected to the device itself. On the host, I open and play with this T2Y using for instance just pick a cotton. So that's the end of my presentation. I hope you enjoy it and it's nighttime for any questions. Thanks. Thanks a lot. There is a limited amount of EPs per device. So, and I know that there is also a limited amount of EPs per controller. Is this only in hardware or is it also related somehow to the driver? In fact, first of all, it is related to the standards. We can have only 16 endpoints for in and other 16 for output and the hardware can handle 16, not more because it is really unusual but also some hardware handle less endpoint than this limits. So depending on the hardware. Yeah, this is per device and now I'm asking about per controller. Say one device implements four in total and other device implements more and the customer can connect many hubs and whatever. Well, where is the limit? The limit is the transaction level you have in order to have the recipient of the transaction, whatever the transaction is, transfer, bulk, isoculus and so on. During the setup packet, you have to set the recipient of the transaction. So you have the device address up to 128. So that's a number of device available on the system bus including apps, apps are devices and you have to set the endpoint number and you can only use the 16. So you have 16 in and 16 out. So you have 16 endpoints per device. Perhaps I can chime in at least on the BeagleBone Black there's indeed a limit on the maximum number of endpoints you can connect to the system. So I run into that limit, so I'm aware of that. Excuse me. Yeah, it depends on what kind of device it is. So a CDC ACM serial requires three endpoints, which is really bad. So for example, I had to use, was using a TIN-C with three virtual serial ports which means it consumes nine endpoints. So I cannot connect more than two of them to one BeagleBone Black. In fact, it depends on each function provided by the device. And even more in a multi-function device, the endpoints are global to the device. So if you have a multi-function device, scanner printer is anything, but scanner printer if you just kind of need two endpoints in the full pool of endpoints device, there is this two endpoints already used by one of the functions. So just a few less than 16 available from the next function. And some function are using quite a lot of endpoints. It's not a question, just to comment just as a data points, STM32 also has limited FIFO space for the endpoints and you need to configure the device tree how you want to do it. So that's also a limitation that you need to take into account. Thank you. So when I tried to play with the gadget, I found a raw gadget interface slash dev slash raw gadget and I found it quite nice to use because all my code could stay in the user space. Is there some disadvantage to doing this? To do what? To just use raw gadget interface from kernel and do the stuff completely in user space? I don't know, but in fact I think from the user space is going to have the visibility and function, function levels on the user space. So I'm not sure they're going to test the really gadget part from the user space except with test USB. And you have a specific module and ask the module for some test. Maybe someone know to comment on that. So raw gadget is meant for debugging purposes only. And if you want your function implemented in user space, you want function FS these days. To the best of my knowledge, for example, one of ADB incarnations was implemented on top of function FS. So there are real life use cases. And every now and then I get asked by people under my blog at Collabra. I blogged about integrating with SystemD. So every now and then people keep asking me about that. So I guess there are use cases, like real life use cases for function FS. Okay, thanks a lot. Thank you.