 Okay, thank you for coming in such a big number, and you're probably all waiting for Christophe to start his talk So I'm just going to let him do that. Okay. Thank you. So, hello. Good morning, everyone My name is Krzysztof Paszak for some non-polyge speakers. I know it may be hard So Christopher Chris is really good enough I work for Samsung R&D Institute Poland and in my daily job I deal with USB support in Tizen operating system. Apart from working for Samsung I'm also doing my PhD degree on Warsaw University of Technology I'm also working with USB related research, but I cannot talk about them now So, now let's go through debugging and our USB devices and drivers First of all, the schedule. In the beginning I'm going to give you some overview of the USB basics For those of you who attended my talk last year, those slides may look quite familiar But for those of you who didn't attend, we have to repeat this Then we will go to plug-and-play philosophy and how it works in Linux And then we will start debugging and getting what we really want, not what kernel offers So we are going to talk about how to force kernel to do what we really want, not what it thinks we want Then we will try to get something more from our kernel We will try to sniff the communication between drivers and between our USB devices Short summary and Q&A session in the end Okay, so We have to verify if you are in the correct room This talk is not going to be about kernel debugging in general I'm not going to show how to use KGDB or stuff like this This is USB related talk, so now it's the time to change the room if you in the wrong one And I'm also not going to give you a cure for all your USB problems Sorry, I'm not a god. I cannot do this I will only show the techniques which can be used to fix some common problems, which you can You know, they're all around here Okay, the USB basics First of all, let's think what USB is about Many people say that the USB is more a network than a bus Because USB is about the same thing as the Internet. It's about providing and using a service Of course only to be different kind of service But we have also cloud printing and cloud storage all that kind we know from the Internet in USB We also have it. The design is only to be different than the Internet. In Internet We have well-known architecture client server. Yep, so we everyone knows it from studies and in USB We have the host and the device The USB host is a machine which is being extended with some additional functionalities Some additional services provided by the device Device is a slave in this design single device May offer multiple functionalities to the host single host May use multiple devices in the same time, but single device may be used only by one host In the same time, of course Okay, some basic entities from the USB world in TCP IP We have the abstraction of ports the destination of the source in USB We have endpoints. It's a little bit equivalent, but there are some difference First of all, there is much less of them only 31 endpoints including end point zero Endpoint zero is a little bit special because as the only one it can transfer the data in both directions ports in TCP IP were bidirectional Endpoints are unidirectional only end point zero may transfer data in both directions If we are talking about the directions USB bus itself is a host centering bus It means that host is a master on the bus. So even the directions Are taken from host perspective? So if we are talking about the in transfer the incoming transfer It means that we are transferring the data from the device to the host and If we are talking about out transfer, we are transferring the data from host to the device Okay, that's the basic stuff. Now. How those endpoints fit together into device Well, maybe endpoint types before that would be better. We have in internet. We have TCP and UDP That's the base protocol. We are using every day In USB, we have four types of transport First of the first of them is control one is the only one which is the bidirectional endpoint So it's reserved for end point zero. It can be it's it's the only one which is mandatory for device to implement So your each device needs to have end point zero and support the control transfer Another one is the interrupt It's used to transfer a small amount of time sensitive data Those kind of transfer reserve some bandwidth on the bus so you can connect only a limited number of them Another one the most popular it's bulk It's used to transfer large amount of data, which is not time sensitive So we have a huge package of data and you say, okay when the bus is free Just send this. I don't care notify me when it will be ready The next one is ISO It can transfer a large amount of time sensitive data, but it does does not guarantee the delivery So we just send the data and you don't know if it has been delivered or not We use this transfer well When it is better to drop a frame Don't hold the whole whole stream and force the retransmission. So video Generally video. It's like UDP. You're streaming the video and you don't care about the frames from from the past Okay, so the USB device live from the logical point of view So device may have multiple and points and points are grouped into interfaces Interface is a group of endpoints which is used to implement some well-defined well sandbox functionality Usually interface has one endpoint like you have in human interface because you are sending only the Information about mouse movements or keyboard keyboard strokes or you have two endpoints and One of them is in in direction and the second one is in out direction So we get the bidirectional communication channel between host and the device Interfaces are grouped into configurations Configuration is a group of interfaces which can be used in the same time It means that all interfaces in active configuration may be used by the host Single device may have multiple configurations But only one of them may be active in the same time apart from all those endpoints which are grouped into interfaces We have the special one the endpoint zero the mandatory one which has to be implemented by each device And it is not grouped into any interface So it looks quite complicated So we have to describe it somehow Because our host when we connect our device needs to learn about the abilities of that particular device So each usb word entity is described by a structure called usb descriptor We have usb device descriptor to describe in our device as a whole configuration descriptor interface descriptor and endpoint descriptor Let's go through the most important fields of those descriptors First of all the id vendor. It's a magic number Which you get from the usb organization if you would like to manufacture some usb devices and of course It's not free you have to pay for it when you buy one id vendor you may use all id products they are used to Differentiate between products which are not Compatible so if you get your mobile phone Probably most of Samsung phones use the same product id Because they offer the same usb functionalities and they are all backward compatible So there we are not changing the id product because your drivers are used to have that id product not other one Then we have information about class The same information about class is repeated in interface descriptor It's not really the same. It's the same kind of field What is class? Well class is Identity of some standard some well-defined communication protocol For example, you have the SSH communicate you have the SSH protocol so a class in terms of usb is a Triple which identifies the SSH protocol So such triple may identify for example mass storage protocol. Here is the list of Available device classes you have the audio class you have communication class human interface and of course vendor specific if you see vendor specific class it means Well, okay, I don't know nothing class information may be used by to choose a driver as the vendor information Then we have some strings the manufacturer the string which represents the manufacturer of course the product the serial and of course number of configuration Each configuration has information about amount of power which is needed From the host to to run this configuration and the number of interfaces Each interface has the class information and the string which can be also used to distinguish between two interfaces Then we got the endpoint descriptor We've got the endpoint address which identifies this particular endpoint like the port number in TCP and Attributes which tell us about The type of this endpoint is it come is it bulk is it interrupt or is it iso? Okay, so how may we learn what we have really connected. I have here Some development board. It's called odroid. It's pretty the same as galaxy s3 But in form of developer board without the screen. I have set up the USB device there using the USB gadget subsystem and now I will connect this to my computer Using the USB, of course, not the Ethernet. I have connected something and now I would like to learn what I have really connected It looks like developer developer board. So I have no idea what I have connected First of all the message. It's always your friend and we should see It's our device. It says that we have connected new high-speed device. So it's usb 2.0 ID vendor ID product The number here we will describe later strings and debugging message about Registered driver Apart from the message we may get some information from LSU SB. So LSU SB And we should see Samsung now minus file means verbals minus the Vendor ID and product ID Okay, and that's the information about our device Those information comes from usb devices from usb descriptors in general if you go through them You will see that it's generally the same information. We have this ID product ID vendor strings and Then configurations we see here first configuration. It contains two interfaces They're both used to implement Ethernet over USB functionality and in second configuration We have some mystery vendor specific interface But I may tell you that it's the generic serial So it's really nothing more than sending a raw data from Serial ports to other serial port over the USB so plug-and-play as You have seen I connected this And it was just working. I Don't need to do anything more So how it happened first of all I connected the device. That's the mandatory point Then my host detected the event of connecting the new device and set some address for it Got some information from the device descriptors Choose configuration because there were two configuration were available and only one of them has been chosen Then it choose drivers for the interfaces and I could start using this device How do we set address? On plug-in each device is default address zero Host decides which address should be assigned and simply Assigns this address to the device usually addresses on the USB bus or simply growing so This formula is the best description of how do we set address for each new device device details, of course all information from the descriptors sent by device and Which configuration? It's not so simple because Do we have enough power? We cannot connect many devices to Single hub, which is not powered Let's say that you would like to connect to USB disks To one USB hub and the USB hub declares that it needs 500 milliampers so the maximum amount and you would like to connect to this hub To USB disks, which also would like to take 500 milliampers So we have one milliampere and you have here on the other side of the hub You have 500 milliampers and how we are going to balance this if hub is not powered So we need to control this in every point of our USB 3 Some broken devices Introduce a configuration without any interface, so we are filtering them usually and If device has only one configuration, we are done Just choose the first one if not We choose the first Configuration which first interface class is different than vendor specific. Why? Where the rationale from the linux kernel says Linux is not the most popular Operating system in the world yet So we are more likely to have a generic Driver for some well-defined well standardized protocol done for some vendor specific protocol Which we don't know so after Choosing the configuration all interfaces From this configuration becomes available So it means that in one configuration you may have multiple functionalities Not only one so in the same time your device may be a pen drive and maybe an internet card But before it becomes a real functionality because before we can get our network interface We have to choose a driver for our interfaces What driver really is well first of all it's a piece of kernel code. That's obvious this code is described using struct USB driver and Usually it's nothing more than a translation layer between USB protocol and Common protocol used by other kernel subsystem If you are implementing the network over the USB You simply need to transfer the generic calls from the network subsystem To protocol on the USB bus nothing more So let's say it's a little bit equivalent of SSH client of our web browser It simply implements a protocol like HTTP or SSH or something. How do we choose our driver? Well in a standard Linux way, it means that kernel has a list of all registered drivers Each driver has an array of acceptable device ID identities So each driver declares I'm compatible with such device choose me for them Kernel goes through that list and if some ID from that array matches it calls driver probe function If driver is if there is no driver You'd have mail you did mail out in the module how you did mail learn which module should be load Each module has a module alias This alias is generated Based on the device ID provided by the driver So each time when we connect a new device We are able to recreate the string and then find the module based on information about the device What kind of information are stored in the device identity? First of all we have the field match flags It informs us which field we would like to match against their suitable defines The vendor ID product ID This is the pair which is used if we are going to write a driver for some vendor specific interface If you would like to implement our driver for some generic protocol, which is standardized We will use the information from the interface or device class and Some helper for implementing vendor specific device. You may choose the index of the interface in the configuration So if your vendor specific device provides more than one interface You may choose to which one you'd like to be in this driver to the first one or to the second one or third one and Some field for the driver the driver information are used usually for something which is called queers It means bypassing some broken devices or some special behavior for this particular kind of device Okay, so how does it look on a big picture? We have our device in this case. It's or droid, but usually it will be a phone or pen driver or something We have our host controller driver for that host controller USB device it represents our USB device as a whole and we have a generic driver for all USB devices and then we will have USB interfaces and Drivers for Interfaces because you as big drivers are probed for USB interfaces not for the whole device Each driver provides usually something to user space. Let's say a network card or input source if it's human interface and simply our user space is using them That's all everything happens automatically, but not always it works perfectly Automation is really good it's really Something would user like if he can connect that his pen drive and everything is working You get a pop-up with pen drive content. It's really great But it's not working in many cases first of all USB is considered now as a security issue Because all devices are using automatically and this is the vulnerability Which is being exploit for example by the bad USB attack You connect the pen drive, but this pen drive has a firmware taken from the keyboard and it starts sending keystrokes to your computer As like you can do from your own keyboard on the other hand You may have combined USB device which introduce itself as a pen drive and as a keyboard in the same time So probably you would like to use only the pen drive functionality, but filter out the keyboard part What if kernel choose run configuration? Which was first configuration which first interface is not vendor specific and What if I have developed my own device and I really want to use the second configuration? Not the first one because second configurations give me more possibilities Then what if we cannot find the driver for this device? You connect your rocket launcher and it's not working because you don't have a device. You don't have a driver and Last but not least What if wrong driver has been bound to this device? multiple driver may Declare compatibility with the same vendor ID and product ID Or we may need some special handling for this particular vendor ID and product ID pair Not only the class information We need some you need some methods of changing those assignments Fortunately kernel provides quite extensible CISFS infrastructure it may be used to get information about connected device it may be used for some device management and drivers information and management and Also provides a device node which can be used to communicate using the USB protocol Usually you don't want to use this directly, but you'd like to use the deep USB to talk with you through your device Okay so if you go to CIS boss USB devices and then execute ls You will see a bunch of numbers and a bunch of magic Directories, so let's try to demystify them first of all all Directories which starts with USB and then has a number represents the host controllers Single computer single machine may have multiple USB host controllers. It means multiple independent USB buses then We have a directory formed like this X is the ID of the host controller which is used for the communication and Then we have physical path to the USB port to which this device is connected For example for this printer We have our host controller Then we have some hub which is connected to the third port Then we have another hub which is connected to the second port and then we have our printer Which is connected to the third port so The name of directory for this printer would be one as a bus ID Then three as a port number in the top hub Then two as another port number and then three as the last one and that's the unique Identity of this device on our machine. There is no other unique Identity string other than this So we may have to totally the same USB devices connected to your computer But this directory will be always unique because you have to connect it to two different USB ports The last type of directories is directory which represents USB interface It starts with USB device identity and then you have configuration number the active one and The interface number from the device that's from the interface descriptor. Okay, so what we can do using our CCFS infrastructure First of all, we may limit the number of allowed devices. So we may say I Don't want any more than one human interface device Just one keyboard nothing more. We may use this also to prevent from some kernel ops if we have a driver which we know is vulnerable and If some device is being connected it may blow up our computer we may simply Say that we are not authorizing this device for usage If device is not authorized, it's left in un-configured state It means that it's in state zero Configuration is has not been chosen. There is no interfaces in CCFS directory. So there is no drivers binding to this device All that kind that stuff can be automated using the USB guard project. It's a security project which Allows you to filter your device your USB devices So you may allow certain groups of devices or anything what you like How it works under the hood? Well, you go to the directory Which represents your USB host controller? You say that okay by default All USB devices are unauthorized. I will do this by myself if I will if I when I check with this device I will decide if I would like to use it or not Then you go to the directory of this particular device and simply authorize it When you authorize your device kernel will choose the configuration Create interfaces and probe drivers for them the next one Here we are operating on our device as a whole, but what if you would like to filter part of USB device functionality We would like to use only one interface out of three or maybe five in order Well, it's pretty the same as the device configurate as the device authorization but now for authorizing the interface we go to the interface directory and After authorizing the interface we have to trigger driver probing for our for our interface why? Well, some functionalities needs more than one interface So before you start probing a driver for that functionality. You need to authorize both of them Usage is pretty the same Choose your USB bus Now you write zero to interface authorized default and You can authorize your individual interfaces and trigger driver probing for that interface What's the difference? Even if Interface authorized default is set to zero kernel will choose configuration for this device All interfaces can be found in ccfs and all interfaces are visible for to the kernel So we authorized some interfaces. We know how to filter them. We know how to filter devices But what if our colon kernel choose wrong configuration? We may have multiple configuration and our colon kernel choose the first one We just go to the device directory Check current configuration and simply write to this attribute to set a new one No rocket science okay so It's really often That some cheap manufacturers Simply set random vendor IDs and product IDs Yeah, you know you buy a pen drive and it says oh I'm manufactured by one two three four five and my product ID is five four five six seven eight Okay, that's great, but you then you buy another one and it say hmm. I'm ABCD and My product number is EF EF FF because yes because I can so some manufacturers are doing this and Unfortunately kernel drivers are not always actual or as actual as we need so they are not probing for that devices It was the case with I don't remember. It was a kind of Lead display on the USB is on some moment They changed the vendor ID and product ID from one nonsense to another nonsense But they did this and you cannot simply fix this on on the USB device because you don't have access to the console so What do you what do you need is to add a new device identity? To driver which is compatible with this device how we can do this. There is a functionality called dynamic IDs You can add any number of additional Device identities to each USB USB driver There are three allowed formats for this first of all is just the vendor ID and product ID Second one is the vendor ID product ID and the interface class and the last one is the vendor ID product ID interface class and Device information, which is a reference as the vendor ID and product ID How it works? You remember on the very bottom You have here the driver information the information about the quirks about the spatial behavior of this driver for this particular device When you are using the last format you can tell your kernel Okay, I would like to add a new entry to the dynamic IDs and Take the driver information from another entry in device ID in that driver Remember that all numbers which you use here are interpreted as a hex in contrast the interface class information in LSU SB is this displayed in digital format in decimal So you have to be careful. Otherwise, you will get the involved value here Okay, how do we add a new device identity to the driver? You simply go to the directory of the driver since boss USB drivers the driver name and Write to the file called new ID you can check the list of dynamic IDs that are only dynamic IDs you cannot check the list of Driver IDs declared by driver in the code So only the dynamic only the one which you write from the ccFS And you can remove previously added device ID Using the remove ID file Okay, so That's what you need to do to add a new device ID and what if you would like to say, okay canal you're smart But I'm smart. I'm smarter than you and I would like to I Would like you to choose this driver not that one So you simply can check which driver is currently bound to this interface You can unbound this driver and simply bind another one Here we should have bind not on it. It's definitely not going to work Okay, so let's do some stuff with it First of all, we have to say which die which directory be belongs to our device We have device our device has address bus free and device nine As USB minus T displays you the physical topology We have the bus free Device nine. So we will have three two directory three dash Devices of course three dash to Okay, that's the directory of my USB device Now I would like my kernel to switch the configuration. So Eco Maybe to see the difference Now I have the USB zero interface When I execute Yep, better now Okay, so as you see now, I don't have the USB zero Network interface so probably I messed up something But not it's the behavior. I expect here because now we are using the second configuration And in second configuration, we have only one interface, which is vendor specific class But I know that this interface provides the generic solio functionality Unfortunately my vendor ID and product ID is not handled by any kernel driver So I need to force some driver to take care of this vendor ID and product ID so I'm here in the driver directory. I can simply eco doesn't matter It's the vendor ID is the product ID and It's the interface class ox ff is the vendor specific class You can see this over here done If I check my de-message now, I Should see that my driver is should bounce to that device But it didn't Okay, so that's the USB debugging You are doing something which should work, but it don't too much terminals too much too much Okay So let's simplify this The previous value of configuration Yep, that's true. Yeah Okay, so now it should work and yep, I have the TTI USB one port So It means that yeah, we managed to get our cell driver working That's huge success. They believe me so What I can do now is for example stop authorizing USB interfaces by default So I mean I can just do eco zero USB free Interface authorized default. Okay. No, it should Hmm, okay now, okay, don't know why Okay, so Now if I switch configuration of my USB device back to the first one when we had the network interface We will simply have nothing because those new interfaces are not authorized to be used by our kernel So let's do this Eco one now USB zero interface and What I can do now is Simply eco one dot zero authorized Dot one authorized and now eco free dash to one dot zero to Of course eco Minus and to do not add the new line character if config No, please Okay, and once again, we have the USB zero network interface Okay, so We managed to put a driver of our choice on the top of some USB device So maybe let's start looking at it. Let's start Sniffing what he is really doing First of all some basics of the USB communication USB is a host control bus Nothing on the bus can happen without host first initiating it It means that your device is not initiating any transfer As long as you will not ask your device for data. It's not going to send you anything Whole USB is a pooled bus It means that your host controller is constantly asking All devices sending or requesting the data USB transfer happen first of all we have a transaction and a transaction is a delivery of single portion of the data of Vue max W max packet size is the value the cloud in endpoint descriptor and Set of transaction From which only the last one is a short transaction or zero length transaction is called a transfer so it's USB transfer and Usually in drivers we operate on a transfer level. We don't care about transaction This is the role of the host controller to split our transfer into transaction. We are operating on the USB transfers USB request block is a base kernel abstraction entity Which is usually one to one matched with transfer. Of course, there are tricks where you can Do that multiple the URB will be put into one transfer But not many driver does it usually. It's a single USB transaction API which is used to send this is asynchronous and most of the drive Most all drivers in kernel are using the USB devices in asynchronous mode and URB is Really a kind of envelope for the USB data. It doesn't care what you are going to send We are simply giving the data and sending it to some particular endpoint So what do we have inside our URB? Of course, there is more fields. I have you know cut this only the most important are here Some handler to put this on the list. This belongs to the owner of this structure USB device because we have to identify where we are going to send this and of course the port number Then we have the status Status is a field which should be checked in a completion routine Completion is a callback, which you will get when this transfer is ready You should check this field and you should check the actual length Because transfer may end up in an error But part of the data may have been already sent So you should check how much data has been already sent and how much data is left What you are going to do with this data? It depends on your protocol which you are implementing Okay, so we say the transfer buffer of course the data which we are going to send and if you are using the Control transfer you have to choose you have to point out the setup package Because control transfer consists of two stage in one We are sending four integers to the device and in second one We are sending a data or we are getting the data from the device So the typical USB driver, how does it look? Well generally from the USB point of view We don't have much to do. We have the probe function Which should check the device and allocate all required resources. We have the disconnect for cleanup We have the completion routine for our urbis, which we are using and The rest of the driver is related to the subsystem which we use to provide some functionality to the user typical box Missing the scripters Many drivers try to get some descriptor from the device and simply assume that device will provide it sorry Your device are not so are not as good as you think so They are going to return errors They are going to return less data than you expect. They are going to do any sick things You should always check if you are submitting a transfer if you try to get some descriptor from the device You should always check if it is valid Then you should check if interface Contains as much endpoints as you expect If you expect two endpoints, you should never try to get endpoint and then don't check the results Because kernel might return now and we have now pointed the reference because I have connected some broken device There is no error handling in complete routine hmm This point is usually a problem if you are running some automated testing laboratory using the USB Well, if you are doing so then well, I Don't know what to say, but you are going to sleep well because USB and running you as some automated system on the top of USB is really asking for problems There are some bugs in completion which are not being checked in many completion routines which are really rare and most of the users don't face them because How long are we using your pendrive five minutes ten minutes half an hour and such automated testing laboratory is running for a month Oh, no, sorry. It's running for a day. Then it's simply sending an error from the USB nothing more in the end Device sometimes gets broken something gets wrong and send some malformed package packages You are communicating with the device using some defined protocol and this protocol may something some sometimes mess up You should always handle the errors which are related to the to your spats and Many drivers simply implements their things which are usually working and Don't care about the error situation about the error handling There are of course some hardware USB sniffers The one from Alice's I have one of those analyzer on my desk. It's really good But it's really pricey It cannot it can Show you and dissect the most of USB protocols most of standard protocols So if you are going to debug some network card some pendrive It can dissect all the classes and that kind of stuff. You can also add your own one Big good can't the same So they are pretty much The same they are going also to show you the physical liar How your transfer is split into transaction how the transaction happens the electrical level of the USB protocol the next one is The open hardware USB Sneaker I Heard about it. It's called open V's SLI. I Heard that people are using it. I'm really really excited to build my own one But I'm not sure if it's really working If there is someone who use this I will be happy to chat and talk about this and maybe order some parts together and Because you know the difference in price I Can buy ten of them for one analyze Yep, hardware sniffers are working definitely. They are really good. They are really good, but they are really pricey So the problem is is the price Okay, so what you can do more you can do some electrical level sniffing you can check the states the J and K states the timing information is really Well try to analyze USB 3.0 using the logical lies Okay, so I don't have such good logic analyzer to analyze even 2.0 So yeah Okay, so that's the hardware part But our Linux kernel has some basic sniffing functionality which can be used to sniff a traffic between Drivers and devices. It's called the USB monitor It's simply nothing more than a logging system for USB request blocks Each time when driver tries to send something it calls submit It means pass this to the host controller and try to send this to the device and we are simply logging it Each time when transfer is finished. We get the completion routine. So we are simply logging it Each time when we try to submit and we end up in an error. We are also logging this event We have two formats available the text one and the binary one It's simply a character device from which you can read the binary data about your USB transfer When you load the USB module You will get one instance of the you of USB mode device for each USB bus Okay, where you should look for the data If you would like to see what your driver is sending you should not look in the completion Why when you open your logs from the USB monitor You will see two kinds of information usually the information about you will be submit and you will be complete If your driver is sending the data in you will be complete you may have nothing In you will be submit you have the whole buffer which is important to you If your driver is waiting for data Then there is no point in checking the buffer on the urb submit Because there is rubbish nothing more than rubbish Okay, so using the row USB mode interface is really not a thing which you want to do So just go ahead and use wire shark. Okay, so let's catch something Okay, so We would like to monitor Okay If you don't want to be polite Okay So sorry, we don't have time I was That's the other bus it's something which you are going to see when you open wire shark all the Deceptors for most common USB protocols like mass storage or something so we will get the higher level of the communication Okay, so That's all from me. Thank you very much for attending