 My name is Philip Papps, I'm here with the FreeBSD Foundation and I think since there's a lot of confusion about these things I should probably tell you, my slides are a little bit odd, about the three things that are called FreeBSD. Do you want me to be there or here? I can be elsewhere. There we go. I'll stay here. FreeBSD is three things. The FreeBSD by itself is a Unix-like operating system that's been around for quite a while. We are originally derived from patches made by the University of Berkeley in California in the 1970s and these patches have been developed continuously throughout the 70s, the 80s and then in the 90s there was a spirited exchange of views and two different projects were founded. The NetBSD projects on the one hand they were first and then the FreeBSD project started in 1993, so the FreeBSD operating system is an open source operating system with its source code history going way back to the 70s. The FreeBSD project is an open source community that was started in 1993 and we currently have a few hundreds of committers and several thousand contributors from all over the world. We interact on mailing lists and we produce this operating system. We run our own revision control system and you can go and run FreeBSD on anything you like. People run it on servers but there's nothing wrong with running it on your laptop as well if that sort of thing rocks your boat. Finally the FreeBSD foundation, the third of our things called FreeBSD is a nonprofit entity which was registered in the US in 2001 and the FreeBSD foundation is a charity. You give us money and you trust us to spend it well and the FreeBSD foundation supports the FreeBSD project by providing it the legal personality to engage in non disclosure agreements and things like that and we support the FreeBSD project development by owning hardware and resources and things like that. The FreeBSD foundation also pays for my travel to events like this. I'm a director of that organization. So enough about that. Who uses this FreeBSD thing? Well by the end of this talk everyone in this room probably presumably or keep talking to me but there's several large organizations that use FreeBSD and it's probably impossible to use the internet or a corporate network for any length of time without using FreeBSD in some way, shape or form. So there are several companies that use FreeBSD by itself. WhatsApp is a good example of that and Netflix are a good example of that. They take the whole operating system and they run systems administration and they pay for people to run FreeBSD servers, huge numbers of them. But the majority of the companies on this list don't run FreeBSD as a whole. They take FreeBSD and then they fork it into their own product which FreeBSD obviously allows. So Juniper is my usual example of that. Every Juniper router or every Juniper, I think they're switches as well, they run something called Junos which is a control plane for all of their hardware stuff and that's actually just FreeBSD which has been repackaged to run on whatever tiny CPUs are in these boxes. NetApp does something quite similar. NetApp builds gigantic storage appliances with all sorts of exciting file system features which are not available in open source form or in any other product. And they've also taken FreeBSD and they've thrown away most of the operating system and built something around it they call ONTAP. But it's still fundamentally FreeBSD. Apple has done something like that as well. They've taken the FreeBSD network stack, a lot of security features and they've merged it with other secret sauce and they call it MacOS or IOS and it runs on your phone and it runs on your laptop. And so all of these companies run FreeBSD, maybe not necessarily the whole operating system but huge chunks of it. So why do these people run FreeBSD? Well, for one thing, we are a large friendly and professional community. Myself included, he says modestly. We have many active, very active contributors. Many of our contributors have been contributing for FreeBSD for a very long time. So we have FreeBSD developers who've been working on FreeBSD for more than 20 years. I've been working on FreeBSD for 15 years now. I got my FreeBSD commitment in 2003. So that's actually 16 years I'm older than I think. We have people who've been working on FreeBSD since before it was FreeBSD so it's very easy for companies of that size to go and talk to our community and work with us. Other things people like about FreeBSD is that we have mentoring built into our projects, culture and our processes. So when you show up on our mailing lists with patches, we try to rope you in and we try to mentor you into our processes. We try very hard for people to become FreeBSD developers. We like to make this easy. People also like the fact that FreeBSD has documentation. I like to say that you can go and live in a cave or on a mountain or on a small island in the Pacific Ocean and you can get work done on FreeBSD without having an internet connection. If you want to use Linux to use an arbitrary example, you cannot use Linux without an internet connection, access to Google and Stack Exchange and other repositories of terrible advice. FreeBSD has documentation and we install the documentation with the operating system so you can be productive on a FreeBSD system without having an internet connection, without having to engage with all sorts of websites to get the data you need. Finally, FreeBSD is very friendly to business endeavors. We have a friendly to clause BSD license. We don't try to control what you do with your own source codes. We write the operating system. We're done with it now. If you like it, you can use it. You can use it in any way you like. If it blows up in your face, that's cool. You get to keep the pieces. We like people to submit patches and we're friendly to patches, but if people don't want to send patches, that's cool, too. We place no restrictions on what you do with your own codes. If you want to use FreeBSD and derive something from it and change it completely, you can do this entirely or look at it. That's cool. That's where we are good at. We're very good at producing an operating system. We're very good at growing our community and we're very good at documenting what we do and making it useful to people. Unfortunately, historically, we have not been very good at handling security advisories. In a timely way. We do fix. We've always fixed our security bugs. We've always owned up to our security processes. We've always practiced responsible disclosure, but historically it's sometimes taken a very long time for FreeBSD to release patches and to communicate them to the community. This is a particular problem given the fact that most FreeBSD users are unaware that they are using FreeBSD. You have a Junos router. You don't know that you're running FreeBSD. You don't care. Juniper obviously cares. Or you're running IOS on your phone. You don't know that you're running huge chunks of FreeBSD and you shouldn't have to care. Apple obviously cares. But historically, we've not been very good at disseminating security information in a timely manner. There's a couple of reasons for this. The main reason for this is that FreeBSD is a large and diverse code base. There's many open source projects focused on one tiny component like a kernel or a security stack like SSL or some Rachele like Bash. The FreeBSD project has all of those things. So the FreeBSD kernel by itself is already a fairly gigantic beast. The FreeBSD kernel is a network stack, a storage stack, huge load of device drivers and a virtual memory system. Someone who works on the network stack is not necessarily familiar with the storage stack. So if there's a vulnerability in the storage subsystem, a network stack developer is not going to be the person who wants to work on fixing this or will not necessarily be aware of this. The virtual memory subsystems, there's a handful of people or several handfuls of people who understand how this stuff works. And they're very keen on fixing problems in the VM system, but they don't care. Well, they care, but they're not familiar with device drivers or storage to go and fix security problems there. Also, you know, when a security vulnerability turns up, it's usually fairly sudden and then, you know, you need to get the developers, we're an open source community, we're made of volunteers. You need to get the volunteers to focus on the problem. And that's just a kernel. So in user lands, the FreeBSD operating system has a whole lot of libraries and applications in the Linux world or in other open source operating systems. All of these things are maintained by separate groups of people. In the FreeBSD world, these things are maintained by us. And again, the same problem is true. Someone, a kernel developer, me, for instance, I'm a kernel developer. I mostly do device drivers. If you tell me that we have a vulnerability in our shell, I will say, well, that's nice. Someone should go and fix it. That someone should not be me. And that sort of problem appears all over the stack. We have a limited number of people. We have a limited number of developers. And our developers, while they're by and large, they're generalists, they're not really keen on solving security vulnerabilities in parts of the stack they're not familiar with. So the left side of the screen, that's code we produce. In addition to the code we produce, we also suck in code from all sorts of other projects. Open SSL, of course, is the elephant in the room or the big green gorilla oozing fuss all over the ground. We distribute open SSL just like everyone else distributes open SSL. When open SSL springs a leak every other week, we have to fix it in FreeBSD just as every Linux distributor, just as Apple, well, Apple not anymore because they've got core crypto. But just as everyone else who uses open SSL has to patch open SSL, we have to patch it too. And we need to tell the world that we've patched it, preferably not before everyone else has patched it or knows that it's even vulnerable. The same thing applies to open SSH. When open SSH breaks, which is comparatively rarely, we need to coordinate with other open source projects who also distribute open SSH. We need to responsibly disclose the vulnerability. And there's a whole bunch of other components in FreeBSD like that. The five I've listed are just the ones that are most likely to affect people. You'll see unbound on that list. We used to have binds. We no longer have binds. But bind was the classic example of FreeBSD always being late with patches. Binds broke roughly every week in the 90s and early 2000s and FreeBSD was always the last operating system to release a security advisory because the security team does not have bind developers and we're not terribly keen on fixing bind vulnerabilities. And that's just the operating system component. So we've got a kernel, we've got user-landing, we've got third-party bits which we distribute. And then finally, FreeBSD's operating system also has a ports and package system, which at last count has 35,000 third-party applications, each of which will be vulnerable at some point. That's inevitable. Every line of code you write has at least one bug in it. So how do we deal with these vulnerabilities? How do we deal with disclosing vulnerability information and sending patches? Well, again, it depends. The easiest kind of problems to fix are the ones where only the FreeBSD project is affected. So that's the left side of my previous slide. These kinds of vulnerabilities are easy. There's no nondisclosure agreement. There's no explicit embargo. FreeBSD is the only operating system broke. It's our bug. We are responsible. We need to fix it. And we can fix it on, basically, we can fix it on our own time, except, you know, if there's a zero-day sort of thing, then we don't have any time. But we're in charge of our own time management here. Maybe we might need to coordinate with NetBSD or OpenBSD, but their friends and family, the BSD projects, are very good friends. And when something affects FreeBSD, OpenBSD, and NetBSD, it's really easy to coordinate. Another kind of vulnerability that's nice is when there's no major risk of exposure, right? So if it's a fairly subtle sort of vulnerability, we can usually take a little bit more time about it. Examples of that were to one advisory last year, in April last year, we broke VT, which is our system console. I can't remember what the bug was, but it's a FreeBSD only component. No one else uses it. We didn't even have to coordinate with NetBSD or OpenBSD. We broke it. We fixed it. We released a security advisory, and we were done. KLD starts, I think, might have also affected NetBSD. I don't remember. That was 2017. Same thing. We just fixed the bug. We submitted. We wrote a security advisory, mailed it out. Everything good. No problem. Much more complicated than FreeBSD only vulnerabilities are vulnerabilities that affect multiple operating systems or multiple vendors. In that case, there's usually some kind of nondisclosure agreement, and there's usually some kind of explicit embargo where every vendor or every distributor of this piece of code needs to coordinate and release their patches at the same time so that everyone can update and everyone can be safe, and there's no windows where the vulnerability is known, and then it takes several weeks before FreeBSD is patched or some other thing is patched. The response is usually coordinated either by some kind of asserts coordination center or by some private party who discovered the vulnerability, your security lab or an individual developer, and this kind of security response requires a limited disclosure. So we need to, we can't just talk about these vulnerabilities on our public mailing lists. Anyone can go and read them, and we need to contain the risk of exposure because not only will our users be affected by disclosing this vulnerability, users of other operating systems will also be affected, and good examples of this are terrible, terrible examples of this are last year's fun and game with speculative execution, which broke every computer on earth, and then the debug registers, which is a slightly less terrible bug that came up later in the year. So what's the FreeBSD security officer been, you know, how does the FreeBSD security officer work? Well, the FreeBSD security officer is a team of individuals, and they have a huge number of tasks, right? They need to resolve all disputes involving security, that by itself is a lot of work. They need to fix bugs in FreeBSD or patch bugs. The FreeBSD security team needs to write the security advisories, I was on the security team for many years, and writing security advisories is not a fun task, and we need to respond to email pretty much in real time. So if someone writes a bug report saying something has sprung a security leak, the security officer needs to reply to this fairly quickly. Meanwhile, while there's nothing on fire, as much source code as possible still needs to be audited, and we should monitor all of the appropriate channels for any vulnerabilities that are being disclosed. And finally, the security officer is also the only person with access to the PGP key, so if you can't just distribute this PGP key to everyone, the number of people who have this key is limited, so it's a small group of people that needs to do a lot of things, so that's an extremely broad mandate, and there's a lot of hurry up and wait, so you sit there, everything is on fire, and then you fix the bug, and then you just do nothing for a while, and then everything is quiet, and you just carry on with life, and that's not very conducive to a friendly working environment. Also, there's a very high level of technical knowledge required to all sorts of things, so the vulnerabilities are in the kernel, in libraries, applications, third-party things, it's all sorts of things, so as a result of this, security officers don't last for very long, they are security officers, they're very active and very good, but they burn out inevitably, they lose interest, and they go away. Alternatively, you have very enthusiastic volunteers who unfortunately do not have the level of experience or knowledge to be our security officer, so this has been a problem in the previous project for, well, ever since our inception, so how are we fixing it? We're fixing it with new blood, we've looked at our security team, and we've decided to get rid of it and replace it with a new one, and we've split the technical resources from the administrative and vulnerability response requirements, so the people who reply to the email are not necessarily the people who fix the bugs, so we have a security officer secretary now, whose job it is to basically reply to email that arrives at the security officer address, so at least if you have submitted a vulnerability report, it is at least responded to in a timely manner, and then the security officer goes and coordinates the technical resources to actually fix the bug, so instead of having a security team, which is a mailing list with a lot of hurry up and wait, we have a very small security officer team, which essentially replies to the emails and then recruits technical resources as required. The FreeBSD Foundation is also involved in a security team now. The FreeBSD Foundation can sign non-disclosure agreements, we are a legal entity, so the FreeBSD Foundation can sign NDAs, and this allows us to survive security officer churn, so if our current security officer goes away and we get a new one, at least our non-disclosure agreements with large vendors stay in place so we can continue to work with these people. The FreeBSD Foundation also has money, we can always use more money if you have too much money, or if your organization has too much money, the FreeBSD Foundation is a charity and we'll be more than happy to relieve you of this problem, but we use some of this money to pay for the deputy security officer's time, so our CTO or our technical manager of the FreeBSD Foundation has some of his time earmarked for security work. We also pay for the security officer's travel, we feel that he should be attending more conferences at certain events and such, and we pay for one full time employee to basically be there when things catch fire, so we have one full-time developer who is a very strong generalist and works on all parts of the operating system and we just pay for him, and when something inevitably catches fire anywhere in the stack he just homes off, fixes it, and then it helps the security team write the advisory and gets the response out in due course, and this has worked very well for us, this new model, we got it just in time, and I got this wonderful case study, the debug registers were, you know, debug registers were not quite as secure as one would hope them to be, it was a multi-vendor sort of response, we were pulled in very early in the game by Microsoft, they sent an email to the security officer alias, it was responded to quickly, all of the non-disclosure agreements and embargoes were negotiated and put in place, and very quickly we were talking with other vendors who were also affected, which was every vendor of every operating system which runs on any CPU made by Intel ever in the history of time, and there was lots of good collaboration between the technical people fixing this vulnerability, and then finally, once the certs gave their approval, they allowed us to give our patches to one of our downstream consumers, PFSense is basically a re-rolled version of FreeBSD that runs on a firewall, the certs said, yes, that's good, you can send them their patches, which we did, and one hour of the embargo being dropped, FreeBSD's patches were committed and the security advisory was released, and that was several hours before Red Hat, which made us very happy. So this is working out for us quite well, and I originally wrote this talk for a cert event, where I encouraged the certs in the room to come and talk to me about, you know, coordinating vulnerabilities and just becoming part of the club. So if you are a security researcher or are otherwise involved in security in another operating system, FreeBSD would like to play nice with you, we would like to coordinate our vulnerability response better, if, you know, contacts need to be made or if you are aware of vulnerabilities or if you have issues stalled with us, please talk to me, and I will see about moving this forward, because the FreeBSD foundation is obviously trying to continue to improve our security response. And with that, I have one minute and 55 seconds left for questions. This is actually a 40-minute talk, which I have compressed into 25. No questions? That's good. If there are no questions, then I think it's time for the next speaker. Okay, so a big hand, please. Very good. Actually, FreeBSD is quite fundamental to many organizations, really in their storage systems. So I think it's kind of hard to realize how much data or how much infrastructure runs on that. So it's certainly quite an important project. Actually, I was wanting to ask just one quick question there, sorry. How much movement has there been away from FreeBSD say more to Linux with the developer community? It's unknown and unknowable. One of the features of FreeBSD is that we have no idea who runs FreeBSD, and we don't have to know. Well, intellectually, we want to know. We would like to know who runs FreeBSD, but we don't have to know. You don't have to tell us. You use our software, and you're good. We do notice that ever since Linux has stopped being a Unix-like operating system, ever since Linux has this thing called SystemD, we do see quite lots more developers showing up on FreeBSD because it turns out that people like the Unix way of doing things. People like having multiple processes and multiple threads. People like having log files. People like being able to read their log files. We find that there's more people coming to FreeBSD from Linux operating systems, which is good. There's probably also churn going the other way as well. FreeBSD is great on servers and on things that just need to run quietly. FreeBSD is not terribly good at laptops, so people who run FreeBSD on their laptop, they buy a new laptop. Their graphics don't work. They go and run Linux for a while, and then once we have a graphics driver, they come back. So I think there's churn in both directions. Yeah. Also, obviously, they are different markets as well. Linux perhaps is more, less kind of necessarily the back end, whereas obviously, FreeBSD is more like an embedded sort of storage systems. And core infrastructure. Carl, yes. Okay, thank you, Philip. Thank you. Take this thing off.