 Anyway, I work for Red Hat in the desktop team working on hardware enablement and I've been spending the last, I don't know, one and a half, two years working on Thunderbolt enablement. Okay, so what is this Thunderbolt anyway? And Intel says it's the USB-C that does it all. And so I think C in USB-C is a very good name because for me it's the first indication that it's a very confusing thing because Intel on the marketing slide put a connector there, right? A lot of people keep getting it wrong and I've been, since I've worked on this project, I've been trying to educate people about the differences between USB-C, Thunderbolt and everything in between because USB-C is the connector type and not the technology. Anyway, so a brief history about Thunderbolt, it was first introduced in 2009, it was back then an Apple thing, it showed up mostly on MacBook Pros and stuff, it could do two times ten gigabits of data transfer, it had DisplayPort 1.1A and the connector it used was a mini DisplayPort. It was not very widely used, I think, then there came version two which used a different controller chip, the Falcon Rich controller, it was also basically mostly shipping on MacBook Pros, so not very widely expected, they opted to just back a bit, mostly the DisplayPort. And then in 2016 I think Intel made a big push to drive this to more general adoption, not only in MacBook Pros but more in the general market. And this is I think to some extent successful, I think in all new modern laptops that you buy, like the Lenovo was from two years ago and all the Dells and whatever they all nowadays ship with a Thunderbolt controller. And the most commonly used nowadays is the Alpine Rich controller introduced in 2016. And then in 2017 Intel made another push to widespread the adoption or make the adoption more widespread that declared it's going to be a royalty free standard, so other manufacturers could also adopt it. And I think there are some AMD motherboards these days where you can actually have a Thunderbolt controller and they updated the specs quite drastically, so they updated it to double the speed to 40 gigabits per second to PC Express 3.0 and they used a new USB-C connector and very recently actually I think a few weeks ago USB-4 was announced and one of the features of it is that it's Thunderbolt 3 compatible. So the new USB-4 which also is going to use the USB-C connector is going to be even more confusing because it can optionally have Thunderbolt 3. This is going to be a lot of fun. So it's the USB-C connector, it has to be that thing, so you can have USB... It features native USB 3.1, so you can have a USB 3.1 connector with the old port. But if you have a USB-C connector it can also only just be USB 3.1, but if it has a little symbol, the Thunderbolt symbol, then it's a Thunderbolt connector. But every Thunderbolt 3 port will have a USB-C connector. And if you're already confused, that's the main point, it's confusing and people get it wrong all the time. But the main feature that it has and that's why it's also quick is it has four PCI Express 3.0 lanes. It can also over the Thunderbolt Fabric, it can channel eight display port lanes, display port 1.2 and the newest chip Titan Ridge 1.4. Of course you cannot tell from the outside that you have a Titan Ridge controller or an Alpine Ridge controller, because it's the same port, it's the same symbol, so you have it or you don't, and you don't know it. With the USB Type-C connector also came USB power delivery, so you can actually charge your laptop via the same port that you connect your peripherals with, which is very cool. You can do up to 100 watts for charging, which is not enough for all laptops. So some of the laptops have a USB Type-C connector and then another little connector that you have to still plug in, because 100 watts is a lot, but not for 15-inch laptops with an external graphic card or discrete graphic card. The newest chip, the Titan Ridge controller, also can act as a USB sync mode, which is only, I mean, it's mostly interesting for docks. So you have a Thunderbolt dock with the new chip and you plug it into your USB Type-C port that is not a Thunderbolt port and it will also work with limited functionality, adding more to the confusion, because now you can like mix and match ports and you get a result, but you don't, you get different results depending on which ports you use. One big feature that they added was security modes, and I'll talk about this in a second, and the main idea is that you drive external storage and graphics cards and docks, mainly docks. The internal architecture of the Thunderbolt chip of all the third-gen chips is basically the same. You connect PCI Express lanes to the Thunderbolt chip, you get the DisplayPort lanes into the Thunderbolt chip. In turn, there's a PCI Express switch, a USB controller, and then the Thunderbolt switch that will wrap all these packages, the USB packages or the PCI Express packages into the Thunderbolt packages and send them on the Thunderbolt wire. You'll notice that there's also arrows not going to the Thunderbolt switch, but they go directly to this Muxer area there, and then to the chip, which is because if you plug in a USB-C device or a DisplayPort device, then they will bypass the Thunderbolt and the port will go into what is called the alternate mode and will directly access the USB or the DisplayPort. It would actually not have anything to do with the Thunderbolt port at this point. Then for external peripherals, if the packages go into the Thunderbolt controller, get wrapped into the Thunderbolt packages, and then at the end, which is basically the same chip that you have in the host, but in an endpoint mode, they will unwrap the packages and then there's going to be another PCI Express switch or a USB hub in there, or the DisplayPort signal, and they will unwrap this and then transport to the devices. And here's a very good example of a UX failure of these USB Type-C ports and Thunderbolt ports. I think this is the T for ATS or something that is currently in production from Lenovo. The very left USB Type-C port, you can charge a laptop, you can do a USB 3.1, but you cannot do Thunderbolt. The Thunderbolt port is the one on the right next to it, the one that does not really look like a USB port because it's together with the proprietary Lenovo network thingy. So I get, I guess, like once a month at least, I get a bug report, the Thunderbolt is broken, and then people realize they plugged the thing in the rock port, and it's very embarrassing for everyone involved. Yep. So the connection mode is another thing that you can do in the bias, you can change how the Thunderbolt actually works, you can put it into USB only mode, and then it will only do USB, you can put it into DisplayPort mode, then it will only do DisplayPort, you can do it in the DisplayPort and USB mode and can do both, and then there's the normal operation mode of Thunderbolt port, which is Thunderbolt 3, where it does actually support all the above and Thunderbolt. And because Thunderbolt is basically PCI Express, and with PCI Express, you can do direct memory access, means you peripherally can basically write and read into the main memory as they please by passing CPU control because that's the whole idea of like very fast I.O., people came up with numerous attacks to the Thunderbolt port, I mean, started out with Firewire from Apple actually, and then with Thunderbolt 1 and up to Thunderbolt 3. There was, I think this year, there was a paper released, and they had a cute new name for it because that's a trend, that you have cute new names for security issues called ThunderClab, and it was actually very good paper, they had an FPGA board simulating an external network card over Thunderbolt and not only did like the very trivial attacks, so reading out in memory via normal DMA, but even for systems which had already I.O. MMU enabled, which I'm going to talk about later, they found attacks and basically bypassed all of the I.O. MMU protection from macOS and Windows. Linux by then didn't have I.O.M. protection, so there was nothing to bypass. We were unsecure anyway. Yeah, so what Intel thought of as a way around is they added security modes to Thunderbolt controller, that's the new thing in Thunderbolt 3, so by default there's no security at all, which is really a bit stupid, but that's the default how most laptops are shipped until Thunderbolt 3, and then in Thunderbolt 3 they defaulted to the user mode, which means that before the Thunderbolt tunnels are actually created, you have to from user space authorize the device, so you plug it in, you will see that there's a device connected to it into the port, but the Thunderbolt connections will not be built, so there's no PCI express packages going over this thing, so there's no DMA, and there's other modes, so you can also set this in the bias, you can set it to display port-only mode, and USB-only mode is basically like the alternate mode we saw before, and there's also secure mode where you can actually challenge the device with a key that you first imprint on the first connect, and then on subsequent connects you can verify that this is the same device that you authorized some time ago, but sadly enough most laptops actually even nowadays ship in user mode, so you get, you authorize the device, but the only way you identify devices via some UUID, and you can obviously fake that UUID if you know it. So how do we do this thing on Linux? The main interface to the kernel is SysFS, so the kernel has device drivers for Thunderbolt, and a lot of stuff is in PCI express, in the PCI express subsystem, and it's all exposed in SysFS, and then there's a new daemon, a new system daemon called BoltD, which talks exclusively to the SysFS interface and exposes the D-Bus API, and then there's a command-line utility called BoltControl or BoltCuttle, and the GNOME shell talks to it, GNOME settings talks to it, the firmware update daemon nowadays talks to it, and the idea was that it's also open for other desktop integrations, I'm not sure if anybody actually has done that. How does the kernel interface look like? Well, it's basically a device tree, so there's SysFS Thunderbolt, and there's a device directory, and in this device directory there's device nodes for all the devices that you have connected. There's two special ones, there's the domain controller exposed, and then there's also another device for the host itself, for the computer itself, and then there's a bunch of files that you can read and write from, read from and write to control how the Thunderbolt devices are behaving, and the first version, the first support for Thunderbolt was in the kernel 4.13, and that was in September 2017, so it's relatively new, actually. The first release was also in December 2017, and then there were a number of features added by Intel and by me to the Vault daemon over the last two years. The most prominent one is probably that we got the pre-boot as access control list for 17 and Vault 06, and then we finally got IOMU in 5.0, but that was like before the ThunderClab paper came out, and it turned out that a bunch of the stuff that was vulnerable and MacOS was also vulnerable on the first implementation that we did in Linux, so we did some more fixes to the IOMU implementation in 5.1, and actually very, very recent, so this landed like two weeks ago or something, what will be in 5.4 is going to be the bounce buffers, and only with the not yet released kernel, we will be completely safe until the next security leak comes up for the attacks that were mentioned in the ThunderClab paper, and you need new hardware for this, so for the IOMU support in Thunderbolt, you actually cannot do firmware upgrade, you actually need a system that is certified for Windows 18.03 or something, so you need a very recent system. And I haven't had my hands on any of those, so we are all still unsafe. So, how does the security levels look like? The current, you set the security level in the bias, that's why you change it, you cannot change it at runtime, you set it once per boot, and that's where it is. It will be communicated to us via the security surface attribute of the domain controller, and then every device has an authorized field and a unique ID, the unique ID we use to identify the device, and the authorized field you can read whether the device is authorized, and if you are a system administrator, you can also echo into it and write to it, and with this leak, you actually also authorize it. Exactly, then there's also, if you have security level 2 enabled, then there's also a key attribute there, and with this key attribute, you can do the device challenge, so the first time you actually authorize the device, you write in a unique key into the field, and then a subsequent connects, you specify the key in there, and then in the device, they will do the matching, and if the key doesn't work, it will refuse to connect. There's a command line tool that you can use to authorize the device, and then in bold, there is a policy file per device, so you can say, authorize it just once, or authorize it, and the next time it's connected, automatically authorize it as soon as you see it, and in GNOME, the shell will do this for you. You don't have to do it manually, but if you run GNOME, then the GNOME shell will automatically authorize devices as soon as you connect it, if you're locked in and you're a system administrator. There's also a GNOME settings panel, which you can use to connect the devices and manage devices. There's also a global flag, which maybe is a good idea if you go to conferences, there's a global switch where you can turn off authorization. So for example, if you go somewhere and you plug in the USB type C cable from an unknown source, then, and you have your laptop unlocked, then we would normally authorize it, and it would grant this device access, so before the conference, you flip the switch, you plug it in, and we don't authorize it, so the display port will always work, because display port will, it's just passed through, but the PCI lanes will connect it, and you will be safe, hopefully. Yeah, there's also two different modes that changed, and we got support for this in 4.18 and 4.19, but I'm just going to skip it, because this is not super interesting. And if you want to use your Thunderbolt devices also during boot or through pre-boot, like to enter your looks password or something, then there's a new feature called pre-boot access controllers, you have to enable it in the BIOS, because it will actually basically disable the secure mode, because there's no key for it, it's just UUID-based, and Bolt will basically do the right thing for you automatically. And last but not least, in 5.0, as I said, we get an OMU support, so if you have the right hardware, and if you have the newest kernel and the newest Bolt, then we basically bypass the security modes. We will always authorize the device, if the kernel communicates to us that the support for it is there, which is communicated via this attribute, IOMU DMA protection. And as I mentioned, there's a number of fix ups that we had to do in the kernel, so if you really want to be safe, you have to have the latest hardware, the latest kernel, 5.4, which is not released yet, yeah, because only there it contains all the fixes that were mentioned in the Thunder Club paper. And if you're more interested in more about this, you can talk to me, or there's a very nice LWN article about the bounce buffers, and the timing attacks, and all the other stuff. Anyway, yeah, USB-Fee is going to be basically Thunderbolt plus USB, and that's it. Hi, so you mentioned that you need to have the latest hardware to take advantage of some of the mitigations. Is that the later controller you mentioned, or is it a firmware thing? It's a firmware thing. Okay, so on Intel hardware, it's using the Intel VTD IOMU thing, and it's a combination of firmware and hardware, and it's so recent that I haven't got I'm begging for like some test hardware, because I released balls with support for this without ever actually tested myself. It's been tested by Dell and Intel that it works, but I actually have never verified that my code actually works. Is there particular hardware you have in mind that someone would be able to furnish that would help you get past that? What do you mean get past that? What is, is there a specific hardware that you need that you might be able to, someone might be able to get your hands on to fix that? Yeah, yeah, well I mean there is hardware in Red Hat, but it was scarce. It's like you have only five machines with the newest chip from Intel that has those things, and like one is in Westford, and one is in Brno, and it's not where I am, and they didn't want to give it away because they also needed for enabling the graphics chips and something. So are the pull requests already in Linux 4.5 master branch now? So they will be in RZ1? Exactly, they are in there. I mean the first, like these bounce buffers were a bit of a, it took a long time to get, I think the first version of this was posted in March, it just took a while until, because there was some overlap with the software based IOMU, and then there was some idea to like unify this, and it didn't actually work out, so it just took a while to get in. And do you have an idea why yesterday there were certain problems with the adapters here? I don't know, I get those questions a lot, but I don't know. Casey? So you mentioned that there are all these great things that the device manufacturers have to do, and as we all know from all the hardware standards they don't. So what sort of things have you seen in the real world that Thunderbolt 3 devices like, you know, just the docs and things like that have done wrong? Oh my god, good question. Well I mean, what I've seen actually is that some of these logos which are required by the specification to identify the ports, right, there's a whole USB specification and the whole Thunderbolt specification which is required to put the logos on there, and some of the logos are actually not correct, so like they're missing, or you know, and then the obvious problem is that there's two different types of cables, Thunderbolt cables that's active ones and passive ones, and you have no way to tell from the outside. So like, I don't know how this is really bad, UX, and I mean, this laptop also, these two ports here are Thunderbolt ports, this one is not, and I mean, sure if you look at the logos you notice, right, but I still think it's very bad, UI. I think on this one you can even just boot from a live USB stick on one side. On this side, yeah, unless you enable the BIOS support for Thunderbolt, then it will authorize the, it's a bit silly. And where can you, do you have this FPGA stuff to test this, or do you know how expensive it is, or where to get this? I know that we internally wanted to get our hands on it, and we couldn't, because QA was eager to play around with it. Since we're asking you all of our Thunderbolt tech support questions, what dock do you recommend? Oh god, can I use off the record? No, so I personally use the Lenovo Thunderbolt dock, but, and I'm okay with this, but I know a lot of people in the office that have had problems with this, and I also like the TB16 dock from Dell. There was some USB issues in older versions of the Lenovo kernel, which supposedly are fixed, but I, okay, they're not. Alright, so yeah, if you just use the dock for like power and display, it's fine if you don't use the internal network card, because that sometimes then craps itself, because it's connected via USB, which I don't, this is, I don't have the Thunderbolt dock from the Lenovo, but I mean there might also be a kernel regression, because this one slide I didn't show, there's different, there's different modes for the Thunderbolt controller, and we only recently, very recently gained support for full runtime management in the Thunderbolt controller, and then new laptops do suspend to idle, and like runtime PN and suspend to idle apparently leads to some of the Thunderbolt controllers on the hardware not to wake up anymore. Thank you, Christian.