 Hello. My name is Matthew Garris. I work as a security developer at Nebula. One of the things we focus on in our product is trying to ensure that every component of the systems that we put together is secure. And that means analyzing various components that for the most part people ignore when it comes to making decisions about their security policies, figuring out things they need to worry about when it comes to updates and the like. IPMI is one of these things. I've been playing with it for the past year and a half or so and some of this follows on from work that was performed by a guy called Dan Farmer. You should certainly, if you find this interesting, read up on his work and I'll mention that again towards the end of the presentation. Now some background. I've spoken at LCA several times in the past. I have perhaps at times has a somewhat unfortunate reputation for the use of colorful metaphors and it has in the past apparently been a explicit part of LCA policy to ensure that I'm reminded that I'm not supposed to swear during my presentations. So I'm going to start with some four lesser words. The first of them is ACPI. Something generally left out of polite conversation. ACPI was actually the topic of the first presentation I gave at LCA back in 2006 in lovely sunny Dunedin. ACPI, the advanced configuration and power management interface, which is a firmware specification that allows the platform vendor to provide additional code, to provide additional data to your operating system so that your operating system is able to better understand the hardware available to it and the capabilities of that hardware and also as a mechanism for allowing arbitrary vendor-provided code to run in the context of your kernel. This is obviously not the only four lesser words I have. We also have UEFI, which I spoke about in lovely sunny Ballarat in 2011 and have mentioned a few times since then. UEFI, the universal extensible firmware interface, which is a specification for providing firmware services that make it more consistent for the booting and management of systems and provides another mechanism for allowing vendor- provided code to run in the context of your kernel. But those are the old and busted things. Obviously today I'm here to tell you about IPMI, which is a gloriously modern specification having initially been released in, I think, possibly just after UEFI had first been thought of. So it's roughly contemporaneous. IPMI is not quite a mechanism for running arbitrary vendor-provided code in your kernel, but there are some subtleties there that we'll get to later. But unlike API and UEFI, IPMI is not generally intended to be used in the context of the running system. IPMI is something few to use to communicate with the running system from somewhere else. What was the reason for doing that? How many of you have been Sysadmins in data centers? Lots of you. Wow, that's amazing. I'm so sorry. Data centers are awful. There's noise. There's lots of noise. There's no way to see it. You're stuck standing in front of a server for an extended period of time waiting for it to get through its firmware initialization. And it's noisy while this is happening. And it's either far too warm or it's absolutely freezing, depending on whether you're standing underneath an air conditioning duct or not. And it's really noisy. Oh, and the phone reception is typically really bad because they're fixed for the Faraday cages. So you don't want to be in a data center as much as possible. Any technology that allows Sysadmins to spend less time in a data center is almost certainly a good technology. Now, that seems like a straightforward assumption. I'm now going to spend the rest of my presentation attempting to disabuse you of this notion. But in addition to data centers being awful places that you would like to try to avoid, it's not merely that the data center is uncomfortable and unpleasant and not a place you want to be. It's also a place you have to walk down to and then you have to walk back up from it or it maybe it's not even in the same building and that's miserable. So as well as avoiding just the data center being awful, if you could avoid ever going to the data center at all, or even better, if you could expose an interface that's allowed you to script many of the things that you would otherwise have to do from the data center, then that seems like an amazing thing. A life in which Sysadmins can spend more time sitting at their desks pretending to do work and still being as effective. It's a glorious future. IPMI, the intelligent platform management interface. Almost everybody who uses IPMI uses it for one thing, which is that you can turn computers off and then you can turn them on again. Since that solves 95% of all IT problems, this is pretty much everything you need. But the specification does have some other useful things. One of them is the ability to monitor sensors and you can call out to systems and you can make sure that they're running within their normal temperature ranges. You can make sure the fans are running. You can make sure that the power supplies haven't failed. And you can do all of this without any operating system involvement because you're actually talking to a small device that's on a completely different network link. So you call out to it and it tells you how the computer's doing. So even if the operating system has fallen over, even if it's sitting as a boot prompt, you can still make sure that the hardware is okay. And there's also this lovely serial overland functionality which allows you to export the serial console over the network link, which means that you can avoid the situation in which you buy a new server and then you find that your expensive serial multiplexer doesn't have any ports left in it. Do it over IPMI instead and it just works except when it doesn't, but never mind. This is firmware, so take everything I'm saying with a large grain of salt. This is roughly what the IPMI specification provides you along with the ability to control what the next boot device is. You can use IPMI to say boost off network, even if the system's otherwise configured to boost off disk, which means that if you want to reinstall the system, you can tell it to boost off the network on the next attempt. Your network server will give us an install image and then it'll still default to booting off disk after that. But that's pretty much as far as the spec gets you. The spec defines these things and it's for a firmware spec. It's very reasonable. It's only about a thousand pages long. A loss of which turns out to be state diagrams. But really. But this is while useful, while this solves many of the problems that sysadmin's face when trying to figure out how they can avoid having to leave their desk. It's not really enough. There are additional things and this is where things start getting interesting because while I said my presentation was mostly about IPMI, I'm not really going to be talking about IPMI that much. I'm primarily going to be talking about implementations of IPMI. In most cases, the vendor has decided to add a significant quantity of additional code, additional features, because that's what vendors do. If you only sell what the specification defines, then there's no reason for someone to choose you over someone who's slightly cheaper. Vendors get really excited by this. The IPMI interface, the IPMI controller on the system is one of the few places where you get to brand stuff as much as you want. No matter what operating system the customer installs, your branding will still be there. There's no risk that someone can forget that they're using an HP when every time they log into the management interface, there's a large HP logo. Vendors have added all kinds of additional functionality and this includes the ability to manage system firmware updates without having to run applications on the system. You can log into the web UI that's presented and you can upload a new firmware image and then it'll reboot the system, apply that, and everything will just magically work. Rather than serial over LAN, which is not really such a great way to manage Windows servers, you can have a virtual keyboard monitor. So you click a button and then a Java applet pops out and connects over something that is often I can't believe it's not VNC. Connects to a service that's running on this IPMI controller that is hooked into the GPU on the device and then scrapes the contents of video memory out of that, displays them to you and then fakes up a USB keyboard and mouse. So you can sit there and through the Java applets you can control your system. So you can manage a pointy clicky installer and system is love that as well. Another fake CD drive. So if you want to install something but you've neglected to actually set up any kind of netboot infrastructure because you're a lazy system. Just quick question. A system is considered a protected class. I'm not violating the code of conduct here, am I? Right. Laziness is a virtue, I'm told. So I'll go with that. So you can upload an ISO to the controller and it will then make it appear as a fake CD drive. There's magic discoverability features. So you can plug in a BMC and then it might actually be running a UPnP server. So your systems running UPnP listeners can then say, oh, this BMC just appeared. Click here to manage your enterprise server. And none of this is in the IP My specification. As almost all the fun stuff is completely unspecified and vendors implement a loss with themselves and so it's all slightly different and gratuitously incompatible and really, really fun. There's a lot of additional complexity here. If you look at the inner workings of one of these devices, you'll typically find that there's a large piece of user space codes that's managing the IP My protocol stack. It's sitting there bound to a network port but it's also listening to a link between the server and the IP My controller or the BMC or Baseboard Management Controller which is this device that's sitting here, providing all the IP My functionality. It's in there listening to that and then you've got some sort of brilliant user space IPC that's sometimes over sockets and sometimes over things that are worse than sockets. And they're talking to each other and then sometimes they've actually moved our pandling out of the kernel and put it in user space as well. I figured this one out after TCP dumping an IBM BMC and wondering why the trailing end of ARP who has packets tended to contain code. Thankfully not kernel code but it was leaking chunks of the IP My management demon over the network every time it sent an ARP request. That's just going to give you a flavor for the rest of this presentation. The magic GPUs are amazing. Most of them look like Metrox G200s except with different PLL setup because every body loves PLLs. These are things where the from the operating system side they look like a PCI device that is a roughly Metrox compatible chip. From the BMC side they're this strange window where you read some registers and it tells you what resolution the operating system has set. You can magically infer, it presumably magically infers that from program timings and stuff and then you just read these values out and then you know well okay I read this block of memory and that's the screen and then I can just pass that off to my Java applets and there's a magic device node for you to do all of that from. It's incredible. I love this stuff. There are web services which are not necessarily limited to just the web UI, although obviously that's the web UI, but there are often management interfaces, scriptable things, devices, endpoints where you can throw XML or JSON or XML in JSON or JSON in XML in JSON. And then things happen and what those things are is sometimes documented and sometimes not. So often a lot of the things that you can do through the pointy clicky interface there's also a scriptable interface over the web management interface and everybody does that differently and there's a lot lot more and I'm not going to go into it because I will run out of time. I will actually start swearing and I do not have anything to drink here. So what's a BMC? A BMC is this as I said the Baseball's management controller. It's sits on, it used to generally be a plug-in card. These days in order to save money it's normally integrated directly onto the server motherboard. It's a complete small embedded computer. I'm going to say small these days they're on the order of you know six to eight hundred megahertz CPU, 256 megabytes of RAM, often a significant quantity of flash because they need to be able to store this ISO image that you just uploaded. So usually at least a DVD's worth of flash so that you can upload the entire windows installed DVD. In almost all cases they're running Linux the CPU varies. I've seen Renesus, SH family, I've seen MIPS, I've seen PowerPC, I've seen ARM. Sometimes vendors will, running the same software snack, go from ARM to a different, will go between different CPU architectures in product cycles. So Dell's iDRAC six is ARM based, iDRAC seven is SuperH based, obviously. Running pretty much the same code, just rebuild. The only exception in the wider world that's not running, so I say almost always running this, I've just realized I didn't actually check whether the Oracle ones are running Linux but it wouldn't surprise me. HP run something called I think Green Hills which is an embedded operating system. Everybody else is running Linux of various vintages, various degrees of quality, various degrees of competence. So these devices are there on your motherboard and I just want to mention the on your motherboard thing a little more forcefully. These devices are built into your system and if something really bad happens you can't remove them and replace them. Your entire motherboard has to be replaced if someone is able to do something particularly unappealing to your BNC. So now going back from this kind of contextual stuff I'm now going to start talking about IPMI as a spec again a little. IPMI has gone through a few spec revisions. 2.0 was released in 2004, 2.0 Erata 1 was released in 2014 which is I think one of the longer periods I've seen between a spec and its first Erata release. But there were previous versions of IPMI and older versions were basically you sent a password over the wire and you logged in. They decided that that wasn't really acceptable because these days we should encrypt everything and everybody knows that cryptography solves all problems. So there are various parts of this that are handled in interesting ways. The spec actually has a bewildering array of different encryption protocols, different integrity protocols, different authentication protocols, but one of the amazing things is that as part of the authentication process the BMC hands you the hash of your password. I'm just going to leave that here because I hate you all. No that's not in the slice of it's true. I'm not doing it because the next slide's got spoilers. Anyway this means that without authenticating as long as you know a valid username for an IPMI device you can connect, pretend to start an authentication session and it will give you a password hash that's not very good that you can then feed in it is at least sorted and then you can feed that into a password cracker and you can wait a while and you can discover that the admin password is hunter-two. It turns out that that doesn't matter because when I mentioned there's this bewildering array of combinations of encryption and authentication and in integrity management one of those combinations which is called Cypher zero and this was explicitly described in the spec and was noticed by Dan Farmer in this article that he wrote on how IPMI was one of the worst things that has happened in human civilization. One of these combinations contains no encryption, no integrity protection and no authentication. As in if you connect you do not have to set up an encrypted session. You do not have to do anything to verify that the session has not been man in the middle and you do not have to provide a password. When this was noticed this is a little bit upsetting especially because a lot of BMCs don't let you change the usernames and the username was then the only secret you had. Most friends have provided updates that fix this. You should really make sure if you manage any servers that your BMCs are running up-to-date software and you should then use IPMI tool to query them to dump the currently enabled Cypher algorithms and make sure that Cypher zero is very very disabled. It's marvelous isn't technology great aren't we as a profession on top of things. So beyond the specification again how do you... What? Oh I'm sorry I've got AMI on slide. That's fine. There are two main vendors of this hardware. These vendors build on top of a variety of SOCs. AMI and Aversons produce them other two main vendors. Aversons are also known for producing KVMs. This is kind of how they got into that industry. AMI are a firmware company. The Aversons ones are used by Dell, IBM and Cisco. AMI tends to be used by pretty much everybody else. Again with the exception of HP who have their own software stack. There's a lot of commonality. If you pull apart the firmware images from say any of the Aversons vendors you'll find that a lot of the code is clearly derived from the same. There's an embedded web server called AppWeb. It has various modules that get loaded into it that manage various endpoints. So if you go to slash data it gets handed off to libdatahandler.so and then there's something in there that parses that request. Vendors can then add additional plugins into that and have their own endpoint management. This is fairly consistent. Bugs that I found in one vendor in this code tend to be present in other vendors as well. But to Aversons credit I've not found anything particularly egregiously bad in any of the common code that they've shipped. Now I'm not saying this is beautiful amazing aspiring code. It's written by a firmware vendor for enterprise customers. But it's fine. It's entirely fit for purpose. Then there's a lot of vendor specific code on top. So on this you tend to get a web UI and people vary in what they use to implement the web UI. Some vendors do everything with CGI callbacks. Some vendors just have thin wrappers that call into the, for instance, Aversons provided functionality that does all the authentication. Some vendors use PHP. Some vendors use an extension of PHP that is incorporated into app web that allows you to not merely embed PHP but to embed C in your PHP. What could possibly go wrong writing stuff that takes untrusted input over the web in C? That sounds like a great plan. That does not sound like a great plan. You can usually SSH in and then you'll get some sort of command line as well as SSH there's normally telnet. These functions you can typically turn them on and off. There'll be non UI web services like the ones I mentioned before. So anyone heard of WSman? I really envy your lives. I wish I hadn't. WSman is a specification that allows you to use web calls in order to get information about system state about so you can, for instance, call out and query whether your BMC has appropriate licensing. You can pull down the system firmware configuration. You can push new firmware configuration into the system. You can upload new license. You can get all kinds of great data out of it. Some vendors don't use this kind of thing and instead extend the IPMI protocol. IPMI has a couple of command bytes which are then for vendor specific commands and then after that you embed a code that represents your company and so there's namespacing of it. IBM, for instance, implement firmware configuration support by implementing something that's quite like a file system over IPMI. You send a open command with a path and then you get back a handle and then you can seek in that and you can read it and you can write with it and you can close it. It's really like a file system. It's pretty awesome. It took me three days staring at TCP dump until I realized that, oh, the stream of characters that I hadn't worked out. Yeah, that's ASCII. It's not enough gin in the world. So quick plug for another project I've worked on. Firmware configuration. There's this URL here. If you go there you can download a Python module that allows you to write Python scripts that allow you to connect to the BMCs on Dell's and Cisco's, pull down the firmware configuration, modify the firmware configuration and then push it back out again. So you can completely configure the firmware on systems remotely and this allows you to do automated deployment of systems rather than having to log into each one individually and change the firmware settings. I'll be adding support for the latest generation HP stuff in the near future. That's not why you're here really, is it? Really you're here because you want to hear me get really, really angry and upset and just leave significant silences. So first thing first. If the code you're writing links against Glypsy, it's software. Okay, that seems pretty obvious, right? You do not get to get out of this by saying, oh, but I write firmware, not software. It links against Glypsy. It's software. Software is miserable. Software is really bad and firmware tends to be worse than software because most people spend less time playing with it. There's much less external QA or bug reporting, anything like that. And honestly, if it works well enough, who cares? It's not like there's any thing security sensitive here, is there? Oh, you end up with situations like this, which is a pseudo code. Please ignore the fact that I'm not bothering to do any memory management here because that would just be an effort. The original code did do memory management as opposed to just scribbling over random pointers. It probably also didn't just call get s, but for the sake of argument, it probably links against read line and stairs, probably a GPL violation anyway. Can anybody see an obvious problem here that isn't a buffer overflow? In fact, missing a semicolon on the last line and as a result, that's the only reason why this code would not compile and run. That is a completely acceptable objection. So you raved your hand. Sorry, system called with random arguments. Apparently I should have a semicolon in the arguments ring. Well, anyway, this is what happens if you log into a Dell iDirect 7 or I think this is an iDirect 7 specific bug. You log in, you switch to this RACADM thing and then whenever you type a command, it handles that command by reexcusing itself with whatever you type attached to the end of the binary name. So if you, for instance, type this, this happens and now you have a root shell on the BMC. This is the BMC that, as I mentioned previously, is embedded into the motherboard and can't be replaced. So to be fair, this does require that someone that have valid credentials for your system in advance. Merely being able to reach over the internet is not sufficient for this and so you can say well they've already got admin credentials for the BMC, they can already do bad things. Being able to get a shell means that they can become persistent. Even if you then notice this has happened and change the credentials, they can log back in and change them back because they've probably back doored stuff. If you attempt to do an install by uploading an ISO to the device, then they could have some codes that modifies the ISO before it's handed off to the operating system. They can block any firmware updates you attempt to apply, so it looks like you updated the system in order to close the back door, but you actually didn't. That's an unfortunate situation, it's kind of bad. You should avoid being in that situation and you should update the firmware on your BMCs. This is fixed, this was reported over 18 months ago, they released a fix over a year ago. How do I know? I reported it. After the fix, you can attempt to SSH in and do this and see whether it gives you a root prompt or not. Yeah, yeah, yeah, okay. It is difficult to prove that your BMC is trustworthy. So I know the TCG specification for measured boot explicitly says that the security of BMCs is outside the scope of this document and they are merely assumed to be secure. Someone decided to do their own XML parsing and this is wonderfully for a login and this means that at this point you haven't actually authenticated. This is your attempting to authenticate and if you say do this, it ends very, very, very badly. Now the good news is that I haven't figured out a way to do anything with this that doesn't result in us attempting to mem copy at least two gigabytes of data and since the system only allocated a 60-byte buffer that is going to seg fault. So it's probably not exploitable. It is sufficient for causing the BMC to fall over for a while because it core dumps and then it thinks oh I should gzip this core dump and store it somewhere and these aren't very fast so when it's gzipping a large core dump it doesn't do much else while that's going on. Other things you'll find are in some cases implementations that are shall we say not written defensively. If you take one vendor's code and do this you can't see any way that this could end badly. Again to actually get at this you need to be authenticated probably. I'm still playing with a couple of bits there. I'm not actually going to drop in the ode here. Sorry I practice responsible disclosure and then there are some things that just make no sense whatsoever. I mean it's a good thing you'd never want to pass more than six arguments to anything. You might be thinking here the obvious thing to do is actually deal with the fact that you're going to have to leave your desk go down to the data center and turn commuters often on yourself because these are not worth the hassle of potentially having to throw away every single server you own because they've all been compromised and are being used to mine whichever altcoins in fashion this week. So fine just don't plug them in that seems like it makes a great deal of sense and then you discover that if you do that they realize oh I'm not plugged in. I'd better bring up IPMI on the main system network port piggyback on another IP address and just steal those packets which means your BMC is no longer on a separate network it's now on your main network and it DHCPed. It's fine because nobody will ever find these things and you think okay fine we block IPMI the protocol at the border so nobody's going to scan and find these IPMI endpoints which yeah okay. They tend to have predictable CNs in their SSL certificates because they all generate new SSL certificates on boot except the ones that all use the same SSL certificate. Every single iDRAC 7 has the same SSL certificate and the private half is in the firmware updates you can download from Dell. So quick quiz here could everybody put their hands up excellent keep them up for a moment if you think it would take more than a week to scan the entire internet and extract every SSL certificate from every system actually a week's ridiculous if you think it would take more than a month to scan every IPv4 address and pull down the SSL cert put your hand down if you think it would take more than a week put your hand down more than two days more than a day more than 12 hours more than six hours more than an hour uh so there's like four uh yeah you two don't count because i told you this already and nor do you uh so i've got maybe five or so people that think it takes less than an hour to scan the entire IPv4 address space for SSL certificates and you know how ridiculous that sounds the answer is 14 minutes and the wonderful thing is not only do they tend to leak the fact that they're a specific BMC for convenience purposes a lot of them embed the server serial number in there as well which means you can scan find all the BMCs that are on the public internet and find the serial numbers for the server they're attached to and you can then go to the manufacturer website plug that into the warranty lockup tool find the date that the server was purchased find the exact model it is from who is you probably know who owns this as well at which point you can call up manufacturer and you've probably got enough information to socially engineer them into telling you the default passwords for that system so HP's randomly generate the passwords sorry HP's default credentials are an eight character alphanumeric string rather than something that's consistent it varies between systems and so that's maybe something you could get an HP tech to tell you but it's almost certainly enough to get HP to ship you some new CPUs or some RAM when you say that it's broken and then you obviously run away with the RAM and CPUs i recommend not doing that thankfully the industry has realized that ipmi is not particularly attractive especially because it's a pain to actually interact with so we now have this thing called the redfish specification which is basically ipmi except it's just a restful http thing and now you can turn your servers off and turn them on again in like five lines of python which is this is one of those cases where i'm going to say it's x but over json is actually better than so just to close a few morals make sure that your bmcs are on a completely separate physical network if at all possible and that that physical network is not connected to the outside world in any way shape or form because someone is at some point going to unplug a cable firewall out all incoming ipmi anyway make sure they're up to date and make sure that you do have cables plugged into them and that it's not sitting there piggybacking off something and it's actually on the public internet and with that we've got just over five minutes for questions could you reliably turn off the piggybacking in the bmcs firmware can you reliably turn off the piggy backing on the host network port uh it's vendor specific how that's implemented uh some vendors provide a way to do this some vendors don't life's awesome next did someone else have a microphone so a lot of these um ipmi bmc chips also accessible via like memory bus from the host what type of extra issues does that cause so one thing that is important is that it's assumed that if someone has administrative access to the server then they also are responsible for control of the bmc they you can communicate already with the bmc over an ipmi interface that's implemented in a keyboard controller style manner so there's an index port and a data port and then you write stuff and then you poll and then nothing happens for a while and then it answers you just after you've given up because firmware but what's important is that since it's assumed that if your root on the system you own the system there's no authentication for that local access you can change the ipmi credentials if you have roots on the system this is something to bear in mind if you're doing bare metal deployments because if you're not giving the customer access to the bmc the customer can just change all the bmc credentials and then you go to reboot the system and also reprovision it and you're slightly surprised to discover that you can't reboot it anymore because the credentials have changed right so you know just typically you can't get to the web UI or any of the extended functionality you've only got the ipmi protocol stuff ibm decided that an excellent thing to do would be to have a usb ethnet link between the server and the bmc so you can from the server gain access to the web UI so you can log it if you get root on the server you can first of all change the credentials on the bmc and then you can probably execute arbitrary codes on the bmc via the web UI and then you can hand the machine back and then the next time someone provisions it you can change what they're running life's awesome uh yes i got a question uh how do you sleep at night and what brand of tonic how do i sleep at night i sleep really well that's good then it's it's wonderful knowing that while this is all terrible it's not my fault right the gym the gym helps another question oh yeah if these bmcs are mainly running linux is there anyone who's looking at like creating a third party firmware that's actually a really interesting question super micro got into some trouble a few years ago because they were distributing binaries for the for the bmcs without any associated source code and one of the outcomes of the settlement of that lawsuit was that for a while they were shipping basically the entire sdk to build replacement firmware for their bmcs now a bunch that was binary only but including the itmi stack but nominally it ought to be possible to take that work and build upon it and produce something that is interesting but the bmcs are very tightly integrated with the hardware that they ship with they have to have a lot of knowledge about gpio gpio setup i2c setup and so on in order to be able to integrate with all the platform management functionality uh you could start a project but the amount of integration you would have to do is probably even more complex than call boot the first 90 percent would be easy the other 900 percent would be really difficult it's uh hp's islo ipmi or it's a different thing hp's does use ipmi it's something that's ipmi but it's not running linux um that but is that the islo is that a separate thing the islo is their ipmi right okay thanks uh the scanning the entire ipv for address range in 40 minutes is that doing something like pulling up a whole heap of aws instances and assigning them separate thing or just look for mass scan and you just want an isp with a large link and who doesn't ask too many so a great way of uh managing this is to find a list of isps who are blacklisted by various people for not answering abuse requests and then find one that is running in a country that has strong credit card protection law and so they won't run away with your money and they're probably a great choice if you want to do this kind of thing pretty good time for one or two more questions right uh you could potentially just gain hire some systems compromise the bmcs and then run mass scan off the bmcs although typically bmcs only have a 100 megabit link so it'll take you a bit longer is this why um ipv6 has been delayed is this why ipv6 has been delayed uh that's actually an interesting kind of conspiracy security people enjoy ipv4 being scannable too much and therefore have restricted the adoption of ipv6 i don't think i'm past the shady conspiracy well i i am just not that one presumably you or someone near you has run that scan just out of curiosity um how many the um did you find percentage-wise bmc that was vulnerable uh okay so i'm not going to stand in front of an audience a live stream and a recorded presentation and incriminate myself but were you to ask me to guess in an informed manner or perhaps to speculate i might say that there's on the order well okay when uh supermicro had when it was revealed that supermicro were running a upnp server that was five years out of date and vulnerable to at least three r3 codes execution attacks people doing some analysis then found at least 35 000 of these on the public internet so taking all the other vendors into account i'd say there's probably at least you're looking at ballpark some around at least a hundred thousand bmcs on the public internet and i think we're out of time there is there one more just okay sure okay how would you compare ipmi to intel's vpro uh how would i compare ipmi to intel's vpro better worse or the same ipmi provides a greater amount of functionality than vpro but vpro does actually have a lot of functional overlap vpro in this case i think we're specifically talking about amt the advanced management technologies that run on the management engine that's incorporated into many uh intel enterprise chipsets this means that rather than having an external bmc there's a small microcontroller actually in your motherboard chipset and it's listening for packets on the network port and if it's enabled it will steal those packets before the operating system sees them it doesn't speak ipmi it only speaks in fact ws man you have this web thing that you can contact and you can make various api calls and you can power stuff on you can power stuff off i'm going to say that based on quality of implementations amt is not the worst thing it's probably about as enthusiastic as you're ever going to hear me be about amt but yeah i'm i'm not aware of specific i'm not aware of significant code issues in the amt stack it seems to basically work it seems to be basically secure ipmi implementations on the other hand seem to be pretty uniformly bad anyway i think that's it so thank you everybody i hope that that was