 So, in doing these sort of talks, you never quite know exactly where to start and end. You don't know what your audience is going to be like. So I'm curious, how many out there have actually tried to integrate a wireless driver, a wireless device? And how many of you have had to dig deep into the driver code? Okay, so we have a few. Any of you actually have submitted driver code upstream? Great. Okay. As a note, next time you go fix something in the driver, please submit it upstream. Calla and the rest of us would be thrilled to get it. So and then of course, the other problem is, of course, 50 minutes. So I'm going to have to kind of start with a little, you know, some of the background, some of the stuff, because not everybody here is a Wi-Fi driver expert or Wi-Fi stack, Linux Wi-Fi stack expert, and then get to some of the stuff, more advanced stuff for more advanced people. So, and trying to do all that in 50 minutes, I apologize how much of the stuff I'm going to miss. Okay. But we'll get what we can done. So ideally, this, okay, before I go to that one little piece of housekeeping, a former employer of mine actually gave me a module to give out to somebody in here. So somebody would like one of these. It is called a WB50. Well, this is the radio. This is the dev kit. And basically it is a full Linux computer, a little SOM that uses ASICS-K radio, okay. And so I'm going to be essentially raffling one of those off. So I will start the box around and pass that around and somewhere in the last like minute or two of the talk, I'll go ahead and raffle that off. Okay. So, so we're going to go ahead and do an overview of the basic concepts because you can't really talk about debugging, doing interfacing, that sort of stuff until you've got at least a basic idea what's there. We'll talk about interfacing devices, things like this. And then we'll go through the debugging tools, how to get help, that sort of stuff. Okay. Why am I the person up here speaking to you instead of one of you guys up here? Now, there's probably other people here who has similar experiences. I've been working in Linux Wi-Fi for the last 10 years or so. I've been working in embedded Linux for 17. And I've contributed across a wide, I'm more of a generalist, okay. I've contributed across a wide variety of open source projects. But over the last several years it's been, most of my participation has been focused in Linux Wi-Fi. But at the same time it's been across a lot of the different drivers. Last few years it's been pretty heavily with ASICS KL, before that Libertas, Libertas, ThinFirmware, and I've done a few of the other drivers, too, here and there. And obviously, as I said, I used to work for Laird. The software in this is mostly mine in some form or manner. So, I have unusual special knowledge of their platform specifically. So, anyway, on to the bread and butter here. Okay, let's talk about Wi-Fi chips. Not that you can see it real well, but on this device, this one covered by the can and the sticker, that's the actual Wi-Fi sip on here. Everything else is, you know, processor ram and all that. They come in, obviously, all sorts of different sizes and shapes and capabilities. But fundamentally, they fall into two basic categories, at least as far as the Linux kernel is concerned. You've got your full Mac chips and your soft Mac chips. You might hear soft Mac talk about as half Mac or thin Mac. But fundamentally, your difference is, it's worth upper level that M-L-M-L-M-E layer resides. Does it reside on the chip, either in the hardware itself or in the firm we're running on the hardware, more likely, or is it residing up in Linux's Mac-a-doe-toe layer? And so that's, at least when it comes down to Linux, that is really what it comes down to is in determining whether you've got a full or a soft Mac chip. Even, of course, in soft Mac chips, you've got a fairly wide variety of offloading to the hardware. So there are stuff that sometimes you, you know, some chips do up in the Mac layer, the Mac-a-doe-toe-lover layer. That other chips actually will go ahead and offload onto the chips themselves. But in any case, that's your fundamental difference. That's how I categorize it. Does it use Mac-a-doe-toe-lover? Mac-a-doe-toe-lover, then that's a soft Mac chip, period. That's kind of how I do it. Okay, nearly every one of these chips, I'm sure you've seen these, have firmware blobs. I'm only aware of, I think, two chips where that firmware is open source. Okay, every other one is a closed source blob provided to you, hopefully by the vendor somewhere along the line, with a little luck it is upstreamed into the Linux firmware project and automatically will come with your distribution. If you're unlucky, it's a newer chip and you get that firmware blob when you're integrating directly from Qualcomm or Broadcom or Marvell or so on and so forth, right? And may or may not have other bugs, just like everything else. All of these have their own little errors and quirks and problems. And guess what? You guys know how this works, right? Close source firmware, you just have to live with it if it's got a bug, right? You've got to work around it. You've got to figure out what you've got to do in order to live with it and make up for that problem because you can't fix it. If you're really, really lucky and you work for a company that has enough pull, you might be able to get firmware source out of your vendor. Good luck. There's a few. It happens sometimes. We were able to do that at Laird, but it's very, very rare and takes a lot of negotiating power to do that. Now, something you need to know about firmware. So if you go and you write a driver from scratch, not the ditty of if you're going to do that or not going to want to do that, but let's say you do, you're going to want to load that firmware blob as late as possible. And so you'll see a wide variety of floating strategies throughout the Wi-Fi drivers. Some will load it right on during probe. That is generally not, I won't say it's not allowed, it's generally not recommended. The problem is when you have things that depend on external firmwares that are sitting off in a file system somewhere and you compile that into the kernel, and I'm kernel bring up, you can't get access to that file, you got a little problem there. And so as much as possible, the driver maintainers try to load firmwares as late as possible in the system. There's all sorts of strategies. It's a mess, despite a lot of really great people working on it, working really hard actually. It always seems to trip things up when you're going through this stuff. Believe it or not, there's a whole class of problems caused by the firmware loading times and things like that. So that brings us on to the firmware. Nearly all of the chips as I said, they've got to load that firmware. Even the SoftMac chips in general load firmware. They'll load a thin firmware. Has the lower level capabilities of the chip and then it will pass all of the messages. So all of the Mac messages, the beacons and all that sort of stuff will go on up to Linux to be processed. Your full Mac chip, a lot of things like the beacons and stuff like that that they receive, they process that internally, but your thin Mac chips will pass all that up. As I mentioned, LitFirmware, that's where most of those blocks are residing and LinuxFirmware.get. Interfaces. As far as I'm aware, nearly every conceivable interface that is out there is used somewhere on some Wi-Fi chip or other. I've seen everything from spy to UARTs, believe it or not, but the majority of them, of course, use SDIO, USB or PCIe. Some of the drivers can actually use multiple hardware interfaces. For example, the ASICS-K chip on the device I keep pulling out as an example can do either USB or SDIO. As configured on this particular board, it uses SDIO, they have other people use the chip in a USB mode. And so the ASICS-KL driver, to continue with the example, has both of those interfaces represented in the driver. It just depends which chip you're using. Now, you don't write the whole hardware bus access course because you use the SDIO layer, the MMC layer in the kernel, if your chip is SDIO. You use the USB layer, you use whatever the bus layer, the underlying bus drivers in Linux kernel, those are the ones that you are going to go ahead and use. And so your driver will plug into those, utilize those, and work with those. You don't have to write that from scratch. Now, there is almost always on top of whatever the hardware bus is, there, of course, is always some sort of messaging bus on top of all that, right? The higher level messaging bus. ASICS-KL is called HIF, other ones use other things. But basically, it's how the chip packages things up to send them up to the upper layer where they get unpacked so your driver can do what it needs to. And whether that is command messages or whether that is the actual packets coming back and forth, pretty much every driver has some sort of wrapper format for those messages. Going back and forth, okay? And that is in your driver somewhere, but that will go ahead and use the lower level bus driver. Okay, now, any particular chip or SIP or whatever format it comes in, of course it's going to have other pins, right? As you can see, this is obviously a piece of a schematic. And as you can see, there's other random pins here. Some of them are obvious, you know, like over there on the right-hand side, you've got the SDIO pins. I won't count those as other pins, but if you look on the left side, you'll see a Wi-Fi chip PWDL pin. I'll explain that one later. You can see a wake on wireless pin, which on most processors would get wired into a GPIO somewhere, possibly an interrupt. Down here on the bottom, you can see some Bluetooth pins. Those are utilized for Bluetooth coexistence, for example. So you've got other pins that you've got to worry about, need to connect those up depending on what you're doing, right? Your driver that you have may or may not utilize some of those pins. For example, the Wi-Fi chip PWDL pin over there, that is a power pin, kind of. It's a control pin for internal regulators on the chip. Some drivers will drive that directly. In fact, I tried to upstream a patch that never got accepted that actually handles that pin. Many systems utilize the regulator subsystem, actually, to handle those pins, even though in some cases they aren't really regulators, but they kind of work that way, so it works. And then, as I mentioned, Bluetooth coexistence. This is a pain, primarily because most of the people who work on Bluetooth and most of the people who work on Wi-Fi are not necessarily the same people. And there's Bluetooth chips, and there's Wi-Fi chips, and there's Bluetooth plus Wi-Fi chips. And so this is one of those things that if you want to work right, you're going to have to spend a little bit of time and effort to get working. I've yet to see it work out. It probably works out of the box on some chip, I don't know, but I've never seen it work out of the box properly on any chip I've worked on. In most cases, it's a pain in the butt. So anyway, as I said, integrated, many of the Bluetooth, especially the older ones, will have separate UART pins. So it's essentially like having two chips on one chip, and the Bluetooth are completely separate in many cases. Sometimes you end up with chips that are both Wi-Fi and Bluetooth are going over the SDIO, sometimes they are split off on USB and SDIO. Again, I've seen just about every configuration. Sometimes the coexistence stuff is routed internally, and you never have to worry about or see that. Other times they have separate pins, as you saw earlier. As I said, it always needs setup, always needs setup. So you're just going to have to deal with that. Okay, I imagine most of you have seen a diagram similar to this somewhere on the internet, one time or other. This is my interpretation of the Linux Wi-Fi stack. And get a good look at it, I won't be flipping back and forth, despite as I go through, I'll be describing it. Your hardware is down there at the bottom, user space is up at the top. The kernel is obviously the largest blob in the center there. Most of what we're going to be talking about, of course, will concentrate there in the center. That's what we're really worried about when we're talking about debugging the integration, but that's basically what it looks like. So let's go ahead and start at the low level. We've already talked about this, right? Full on top of Mac chips, we've got a particular bus interface. I'm not going to go any further into that. So your next level up, of course, is your bus driver. I mentioned it uses your standard Linux bus driver. So MMC, USB sub system, PCIe, I recently worked on a PCIe chip. And as I said, I've seen other weird ones. Some of the really, really low speed chips. I've seen spy ones. I've seen one on a UART. In fact, I think actually microchip makes one that has a UART interface. And I believe there's a driver and there's somewhere for that. So anyway. Okay. Now the stuff that we haven't really talked about yet. So now we get a Wi-Fi device driver layout. There is a, doesn't matter whether you've got a full Mac that doesn't use Mac into 11 or a half Mac soft Mac chip that doesn't use Mac into 11. Sorry, said that backwards. Either way, you still have a driver for the device, right? That's at the bare minimum. It will utilize the STO driver. We use the STO, for example, it utilize the STO driver to get stuff physically to and from the chip. Then it will have some sort of abstraction layer in order to unpack the messages that it needs to do. And then it will do whatever it needs to do. In the case of a full Mac chip, pretty much everything needs to be done. It will be done at that layer. And so all the packages will get processed as necessary and then passed up to user space at some point. In a Mac into 11, a soft Mac, it will pretty much immediately send all that stuff up to Mac into 11 where most of the processing will actually be handled. So Mac into 11, if you're ever trying to figure out what kind of driver you are looking at, the easiest way to do is look for IEEE 8211 underscore something. Ops, ALEC hardware and register hardware, those are used right there when the driver sets it up and connects it all together. So those are your initialization functions. That is your key. If you see that in a driver, you're working with a soft Mac chip and that is important later, especially when you're trying to figure out why this does not work when you're having trouble. All right, above Mac 8211 or above your full Mac driver, then you have Config 8211. This is your main configuration API and stands basically between user space and your driver. All of the things that are done go through this API in some form or manner. So things like asking for scans, setting a regulatory domain, setting up connections, things like that, rates, rates are there. Just throwing out a few examples. Those are all right there in Config 8211 and all the drivers, soft or full Mac, will use that. Some of you, I don't know, does anyone here remember wireless extensions? A few of you, okay. So Config 8211 and NLA 8211 replaced wireless extensions basically. There might be a handful of drivers using wireless extensions, some compatibility layer somewhere in there, but pretty much as far as I'm aware, everything has moved over to Config 8211 and NLA 8211. So NLA 8211 is a net leak interface. You no longer use octals. Now you're using the net leak socket interface. And this is how you work with Config 8211 and the rest of the wireless stack. This is how you tell it to do things, okay. This is the management interface. And then that, of course, is not the... Well, we'll get to data here in a minute, but... Okay, so, and then up in user space, so we'll stick on management side for a second. Your basic tools, IW, for example, uses NLA 8211 to talk to Config 8211 to either tell the driver or Mac it or to 11 what to do. That's basically how the flow commands work, and then, of course, the information comes back up over the same sort of thing. For those... Anybody here still using IW Config, please don't do that anymore. Unless you're on an ancient kernel, of course. So, IW Config, IW Priv, actually, I still use it, but I have Config, I'm sure we're all using that, have all been deprecated a long, long time ago. And in some modern systems, a lot of those won't even work anymore. IW is the way to do it nowadays. And if you need to... There's a big warning on IW. I don't know if you guys have seen it. Don't screen scrape this tool. There's a big warning there. I'm grammatically controlling the network stuff to, of course, use native programs and use the netlink layer. But I'm sure all of you, just like I do, still script around IW as input and output. But I will tell you that leads to breakage. Sometimes that what IW prints will change periodically. So, be careful when doing that. I hope we are familiar with WPA supplicant. Very, very large project. Obviously, there's also hostAPD. It goes along with that same project. And so, for those that aren't aware, WPA supplicant handles the encryption stuff. So, it handles the WPA2 and all of the other encryption protocols. That's not handled directly down in the kernel. Though, there are a few chips that do offload some of that into their firmware. And the stack is actually capable of doing that if the drivers and everything are set up correctly to do it. And then, of course, you've got the higher level tools. Network manager, who I love to hate. Conman, there's other tools like that. But, you know, those are there to manage connections regardless what kind of connection it is. Obviously, network manager happily manages Ethernet connections just as well as your wireless connections. So, that's on the management space. Now we have the data side. I'm sure you guys know this, but I'll throw it out there. From the data side, there is no difference. Once you're up in user space, there's no difference between talking to an Ethernet or a wireless device. You're just, you're open in a socket, you're doing standard system calls. And so, all of that goes through the TCP IP stack or the UDP stuff. Whatever it is you're trying to do is exactly the same regardless what interface you're working on. Obviously, the management side is a little bit different, but the data side is the same. Now, that has some interesting, when you go look in the driver, that has the one little interesting little bit of that is obviously you've got two parallel routes for data in the very broad sense going through the driver down the chip. Because you actually have the data packets, you've got a route for that, but you also then have another route for the command, the configuration stuff. So, you do something with IW, it sends something down that link. That gets down to your driver. Again, we'll use sxkl as an example. It figures something out, and then it sends what's called WMI command down to the chip, which at which point kind of the, that's kind of where the data path and the command path kind of merge, because now we're just sending stuff over the bus. And so, you've got two parallel things there. Data path, it's all SKBs, it's all just what you would expect going on up to the network path, okay? So, okay. Interfacing. Obviously, that's a wide variety of different devices. There's some PCI devices there, USB, dongle, all sorts of things. This is just, this is probably about a quarter of the devices that happen to be sitting on my desk when I was preparing this thing. And, yeah, never. Okay. So, we're all here to talk about how to debug interfacing a new device. So, you have a new device. You just bought it, or your customer came to you and says, I want to use this on my weird board, or whatever, or maybe you're a hardware designer you're working with that says, ooh, this chip looks cheap, let's use that. And so, eventually you get your hands on it, you plug it in, and the bus, it will identify itself on the bus, hopefully, if it's that type of bus. And so, it will have a vendor ID and an advice ID that will get shoved out somewhere. Okay. Now, if you're working on a full Ubuntu system, or other similar distribution, insert your distribution of choice, right? And, let's just say it's a USB chip because that's easy to talk about, so you plug it in, it identifies itself, it's something that's already known, and hot plug gets involved, loads the driver, poof, you're working. That's all nice and easy. Okay. Well, what if it doesn't, right? So, what are you going to have to do? First thing you're going to have to do is figure out what that device is. Hopefully, by looking at it, by knowing what you plugged in, you know which bus you're looking at, right? You know if it's a PCI or a USB or an SCI, I would hope. If you don't, I guess start guessing. OddsArt probably was announced somewhere in the message, so we can go look at the kernel log and hopefully something spit out. Or, of course, you can reach directly for LSUSB, LSPCI, or in the case of anybody knows an STIO tool that does the same thing, let me know. But in the case of STIO, I go look in CISFS and I can go find the identifications, okay? So, as I said, the three most common devices you're going to encounter, or the three most buses you're going to encounter are the three, okay? So, you do that and then you've got to go ahead and take a look and try to match that with a driver. So, a good old grep or C-scope or whatever your tool or your choice is, this is a little tiny chunk of code actually out of ASICS-KL. And nearly every Wi-Fi chip driver is going to have a chunk of code somewhere like this that identifies those particular device and vendor IDs, okay? That match it to this. And so, you can go through. Obviously, they're rarely written out in a nice convenient format. So, I'm looking for a 271-301, okay? Well, 271, that's a vendor ID and, but as you can see, if you go down and look at the second chunk there, you're not going to find 301. And so, it might take a little bit of grep-ing and then reading some code. Hopefully, you know who your, the physical vendor is and so that will give you a short cut of where you're trying to go, you can always go grab one from that. Okay. So, you plug it in and let's say it does know what driver is or you already have the driver loaded or whatever. And you'll probably see a trace like this in your DMessage, right? Believe it or not, despite the error message, this is a device that works perfectly, okay? So, this is one of those little things about firmware. The firmware loader system will throw out error messages that aren't really error messages, okay? So, any particular device, and again, using S6KL as the example, may have a whole bunch of different firmwares that can load. And so, in our particular case, this first looks for firmware-5. I won't read the whole path, but you can see it right there. It first tries firmware.5. It can't find that. And the firmware loader will go ahead and spit out an error. And then it will try firmware-4. In this particular case, it found firmware-4 and on it goes. And it tells you, again, in the case of this particular driver, it tells you the capabilities of what the firmware supports, which is kind of nice. Not all drivers do that. The drivers will print it out incorrectly. It all just kind of depends. But, you know, so that's... So, this actually is a good result in this particular case. This actually comes out of this chip. Okay. So, bad result. The first one and the second one is the OGE old, you know, is it plugged in and powered up, right? The first debugging step for everything, right? Now, it's a little bit trickier when you're trying to integrate a piece of hardware you've never seen before to determine necessarily is it powered. If you're really lucky like me, you have schematics and you can find some point on it somewhere that you can probe with your oscilloscope or logic analyzer. You can tell if it's powered. Obviously, you don't need to... Obviously, I'm not reaching for the logic analyzer first, but, you know, go through the basic checks. Okay. In the particular case of the example chip that I keep pointing to, one thing I'm always checking is that chip PWD-L pin is that toggled correctly. Because if that is held low, nothing is in reset, nothing's going to happen, that pin is high, it will live on the bus and talk and communicate, okay? All right. So does it announce on the bus? There's a number of ways to go and look on this. We already looked at one thing, right? So something's partially working if you can do an LSPCI and you see that the device is there, but you can't do other things. Okay, well, at least you know the device, right? The next part is probably, in many cases, where you're stuck. Okay, if you're talking about a dead chip, this is probably your next place. Does it fail to load the firmware? Or more accurately, does it fail to find the firmware? So if you're talking about a new device, you're talking... you probably might not have the firmware. Or you might not have the right spot. Or you might not have it in the right name. The names for the firmwares are generally hard-coded into the device driver. Okay. Common trick, of course, is what I always do. I get a new firmware from the vendor. I name it whatever I want to name it, usually with the version number in the file name so I can recognize it. And then I create a sim link that is the name the driver wants to the actual file that works, it works beautifully and lets you keep track of what version file you have as well as satisfying the driver problem. Okay. Okay, so it didn't... it actually found a firmware, it loaded it. Keep in mind, some of these things require multiple blobs. Blob number one is the actual firmware. Blob number two is often a board file. Okay, they come by... or calibration or... There's number names, depends on your vendor. And this will include the antenna power adjustments and things like that that are needed to keep your device in regulatory compliance. It's part of the equation anyway. And so some of those and we'll get into some of the details later but some systems you got to have that file. Okay. Yes. So, in the case of my example chip the drivers are always... excuse me... let me go back to that one. So you see where it says firmware-5. And then you'll see it actually loaded at the very end of that version number you'll see API four. At least in the case of ASICS KL and actually most Qualcomm chips there is some internal version numbers and there's API levels. Qualcomm has been nice enough to call the file name to include the API level. Odds are, if the API level matches it's something that recognizes, it will work to some degree or other. Unless, of course, you have special stuff. Okay. Example of that is at Laird they do have firmware access. They do put special sauce in the firmware and they do put special stuff in the driver to work with the special stuff that they put in the firmware. Okay, so there's things like that. And then the other, the final thing you can do for compatibility-wise as I said a lot of the firmwares will print out their support. So they'll have a set of flags and the driver will be able to look at those flags and turn on and off different features. Okay, so you might be able to look at the driver and go, okay, it doesn't have the ability to do this feature. The firmware I have says it can do this and now you know you've got a mismatch. Now that just maybe because nobody coded put that feature in the driver yet. Okay. So, for example, or sometimes it will spit out a firmware flag that the driver has no idea what it is and it will just spit out a number and it's like, now you know the firmware supports something the driver doesn't know about yet. So, I hope that answered your question but it's a complicated one. Okay. So we go through next you want to see, does it come up? You know, and here I am doing exactly what I told you not to I'm using knife config, sorry. One of these days I'll learn IP route two commands. Okay. First test to always do and I don't, it boggles me but a lot of people skip this. Don't connect to a WPA2 AP right off the bat. Connect to an open AP. So maybe that means you need to drive over to Best Buyer Fries or somewhere and pick up an AP and configure it as a pure open AP. Call it TestFu whatever you want. Make sure it doesn't connect to anything you care about because everybody in the world could in theory get on it. And then try to connect your device to it. If you can connect to an open AP you're probably doing pretty good at this point. Your device is probably working for the most part and you can worry about the upper layer stuff later. But if you can't even connect to an open AP you know you've got a serious problem. Okay. Alright. And then of course obviously as you're going through you want to check the features you care about. Do you care about Wi-Fi Direct? Do you care about WPA2? Do you care about whatever it is feature does it do 802.11R? Whatever it is that you want to go through and test that. First thing I always do on my testing systems is I uninstall Network Manager. As it tries to manage your device it will screw you up. Okay. Whether it's Network Manager or ConMan or whatever you're using in your device get rid of it first and then figure things out. Because I will guarantee you it will go off and try to connect to something that you don't want to connect to. You'll run a test that is everything actually is fine but because Network Manager tried to do something wonky with it your test fails and it's all over with. Okay. I'm not saying Network Manager is bad. It's not. And I'm not saying it's a problem but it is a problem when you're trying to debug the lower layer stuff. It's something that just gets in your way and gets a little confusing. Okay. And I happen to like it so I stuck it in there. But what it really comes down to is as you're going through debugging as you're figuring stuff out I don't know a single time when I haven't been having a problem that coming up the solution was not pure guesswork at some point. At some point it was I'm working down really really hard and then all of a sudden light bulb goes on and I go oh I should check that. Okay. So when that light bulb goes on and you should check that make sure you go check that please. Okay. That's the first thing you should go look. Alright. There's a ton of debugging tools already available to you. This list is probably I hope you recognize everything on this list if there's one item on this list that is then sitting here for an hour listening to me speak I hope was worth it. So here's a whole bunch of possible options. So kconfig most of the debugging facilities of most of the wireless stuff is off by default. Turn them on. Okay. Asicskl of course has debug. Most of its printk's are turned off unless you turn that on. It also can use the tracing system. Broadcom full mac driver same deal. Turn on kconfig underscore broadcom debug. Okay. So turn those on even if you don't turn those on there'll be a lot of stuff in dmessage. But if you do turn them on there'll be a lot more stuff put on there. That said almost all of them that have this functionality have a debug mask somewhere in there. Okay. Asicskl has a very verbose set of debug messages. If you want to look at all the raw data going to and from the sdao bus you can turn that on. Surprise is turned off by default because you'll get a flood and it's impossible to read through. Okay. But whatever you're doing maybe your notice trickle command is not working. You turn on WMI debug and you can take a look at the processing. The config layer all those usually have individual flags. They're all documented of course in the code in some case in some wikis they're documented. But it's usually a case of giving a particular mask to a either module parameter or in this case of course Asicskl's module parameters can be changed directly so they all have this. Same deal Broadcom FullMac has this. Okay. So module parameters there's a whole bunch of sets so for the Broadcom FullMac driver I gave you a couple a couple things. Debug is what we're just talking about right. It's the exact same thing. You can actually on this trigger driver you can override feature detection. A trigger feature is screwing everything up and causing it to crash. You can disable that when you load the driver. You can even ignore on this trigger driver you can even ignore if the probe fails and just keep going which can be handy. Dangerous but handy. SysFS you guys probably already know this. You can walk the device tree figure out what's where. Sometimes important to do. Debug FS Go look at Debug FS. Enable that. Mount that. Go look because there's probably a whole bunch of super useful stuff there. Okay. I'm not going to just read the slide to you. Some systems will be able to do a core dump on the firmware. I know Asicskl can do that. I know Broadcom FullMac can do that. There's other ones that can do that. You can even in some cases trigger a firmware crash if you know what you're doing and have it do a core dump. Kind of handy. Yes, well okay. Who here knows about Ftrace? Please? Okay. Good. Ftrace is awesome. When you are trying to figure out why something doesn't work instead of littering it with print K's just to go through the driver flow you get the right parameters and you can actually figure out how that driver does its thing. Brilliant, brilliant tool. I love it. This is Kernel Shark. This is just a gooey into Ftrace stuff. Take your pick however you like to do stuff. And then of course, Wire Shark. So everything's working but something's not working, right? So everything appears to work. Ultimately it comes down to does the packet make it to the air? Does it get to the AP? Does the stuff from the AP get back to your device? Fundamentally that's what it comes down to. That's what Wire Shark is for. I'm not going to teach you how to use Wire Shark. I don't have time. But there's lots of videos and documentation on how to use Wire Shark. Okay. Alright. Common problems. And I apologize. I am running out of time so I will blaze through this. We're almost done. Bluetooth coexistence often is a common problem. If you notice wide things are going along just fine and all of a sudden your connection goes kablooey when it comes to latency or throughput and then comes back and then dies again for a little while and comes back. It's not actually dropping or anything it's just you're getting really inconsistent results. I find oftentimes it's actually Bluetooth coexistence. A lot of these devices have shared antennas with the Bluetooth and the Wi-Fi. They use the same frequency domains, they can use the same physical antennas and so some of them to save money will share antennas. And so as you might imagine if you're using, if both devices are trying to use the antenna at the same time you're going to have a problem. Okay, power pins. Sometimes they even have, and I'm not talking about power pins I'm talking about those regulator control switch on switch off pins not just talking about pure power. You got to make sure those things are set up correctly turned on turned off at the appropriate times. Sometimes the Bluetooth and the Wi-Fi have separate power pins and sometimes they interact in odd ways so you have to be careful of that. I've talked a lot about the firmware but you've got to find the right firmware it's got to be named correctly. You've got to have the right board file otherwise you're going to have problems. A common problem you're going to encounter though is multiple drivers. Some of these chips are controlled by multiple drivers. Sometimes those are vendor drivers that conflict with the mainline driver. There's the term open source vendor driver which is an odd one but there's some out there that have a driver out in GitHub has never been mainline but it's still at heart a vendor driver somewhere. Sometimes there's even a driver in the mainline and in staging at the same time. So you actually have both on your system. So you got to watch out for that use the right driver. There's also a good situation sometimes where there's multiple drivers because there might be a thin Mac driver for the same chip. Now that comes down to make sure you have the right firmware for what you're trying to do. And if you're wondering why you would do that it's because some of the full Mac chips might not support AP mode and so some people have created thin Mac firmware in order to use Mac 8 or 11 host APD and all that properly. Sometimes the tools that you're using i.e. iWconfig if you do stuff in iWconfig and iW at the same time don't expect good things to happen. In fact just drop iWconfig please. Okay. Here's a problem that's actually getting some e-mail on the list in the last week or so. We have a couple chip vendors that have ended up using a different device ID on the SDIO bus. This was caused by some SDIO IP going out there and the people implementing the at the end not realizing they need to change those things to theirs. That causes problems by the way because then you have two totally different devices where and multiple drivers that think they both control those devices and mess. Regulatory. This is a problem if any of you have tried to do regulatory stuff particularly here in the United States Europe this is a big pain. There's a system called CRDA which of course is used to work with this. Another problem with this is some devices do have power levels and stuff they might have an EEPROM instead of you loading a board file there's all sorts of ways to trip up on this. If you got something that's kind of working but the power it works if it's this close to the AP but doesn't work when the AP is over there you probably have a problem with your power levels and that's all in the regulatory stuff and then performance says it doesn't work as well as I want or as fast as I want the first thing you need to do is figure out what they're talking about as far as performance goes does it mean it's not connecting at the rates that it's supposed to be connecting to or is the throughput lower than you think it should be there's a zillion different reasons that performance might not actually be a performance problem but you got to figure out what that is obviously my gold standard of figuring out what the actual performance is regardless of anything else is using IPERF figure out what the actual throughput is once you got that at least you've got some basis of having some numbers to talk about what is our end time is it 320 or is it 330 does anybody know so I'm over and I apologize and why does it keep doing this disable that okay so just like two more slides okay getting help so if you email the Linux wireless list and you say it doesn't work we're not going to like that give me details show me what doesn't work enable those debugging features give me the kernel logs with the debugging features turned on do not be surprised if I or another maintainer say here feed this number to the debug parameter and then give us your debug logs we'll probably do that be very specific about the hardware the firmware version what Linux version you're running on if you tell me you're running on Linux version 3.8 I'm probably not going to want to help you very much okay but if you're running something reasonably recent we're certainly happy to help well actually I'm happy to help but if you say it does this and later I go well I can't replicate that and later you tell me you're on 2.6.33 I go oh well yes of course you're having that problem okay so I need to know what Linux version you've got be specific about what works and doesn't work if it does not work when doing this particular encryption protocol I need to know that don't just be you know I go and I connect to an open AP and it works just fine I'm not finding your problem and in certain cases don't be shocked if you're asked to include wireless captures we don't do that too often because not everybody knows how to do that but if you know how to do that and can do that please do it's actually very helpful to us okay getting help whole bunch of links this is mainly so when you download the slides you have the links I'm sure these are all obvious to everybody and then that's it so thank you very much if anybody has questions please feel free to to ask me but I believe we are officially out of time and I think somebody probably has this room after me so yeah