 Let's get started. So let me give a bit of background of why I put this talk together. I've come in some space, lots of space here. So I wanted to sort of look at the question, how do we deal with security vulnerabilities and open source projects, and what's the best practice, is the consistency and so on and so forth. It is sort of light that software stacks are becoming more complex and like in your traditional stack you have lots of different layers and so that's the idea that if a particular project, a piece of your puzzle and doesn't do security very well, then the whole stack is going to be vulnerable. And basically we sort of took question up there and then we have to wait until the end to get the answer. A little bit about me, I've been contributing, my name is Lars Kurt, I've been contributing to a number of different projects, GCC, Eclipse, Xen, a bit of Linux as well. I've worked in a lot of different technologies and today I do the community stuff for the Xen project. I happen to work for Citrix, I'm a member of the group which developed Xen server, but I'm also the chairman of the Xen project advisory board and the Xen project as a Linux foundation collaborative project. So anybody disagrees with the fact that software bugs happen and we can try to minimize that, but at the end of the day there will be some vulnerabilities. And the picture analogy I see there is I'm kind of using, it's going to use this a little bit through the talk, it's this whole idea of vulnerabilities being like zombies you know which can do a lot of bad things and then you know the way, but you know, we'll get back to that at times to make it a little bit more fun. So let's start with, and by the way, just interrupt me with questions any time, we have an hour so we'll be, I can take one or two questions during the talk, that shouldn't be a big deal. So I just wanted to sort of start with a little bit of terminology. So you know at some point when you write code, you know, bugs get introduced, you know the way how I look at bugs of vulnerabilities in this sense is you know like it's like a broken window in your castle where somebody can get in and you know coming back to this whole zombie analogy. At some point in time, somebody actually discovers you know the bug. If it's undiscovered, or if the vulnerability is undiscovered, well nobody cares, right? Nobody's going to do anything with it. Now this could be you know a bad guy who discovers it and does something you know with it, you know like sell it, exploit it, steal your data, game over. Not very much we can do at that point. But what we're going to look at is really that scenario where actually the discoverer you know isn't as bad, you know they go and report the issue to your project. For example you know via security at your project dot whatever you know email address you know where you report vulnerabilities. And you know that gets us to the point of actually how do you manage vulnerabilities? How do you do that well? How do different projects do that? What are the patterns you know out there today? You know what which are in common use and as some of us those have different tradeoffs. So the way how I look at you know at vulnerability management processes in open source project and it's really a little bit of a team effort and so I wanted to draw back to that you know zombie analogy. I don't know who's seen the living dead or read you know the comic, okay. Right and then you know like there's the scene where they eventually you know they found a prison you know they managed to keep this quite secure. They deployed a lot of techniques to make sure that the zombies couldn't get in you know checkpointing you know make sure that make sure that everything is fine. And in many ways you know what vulnerability management processes do really they're like a team effort you know to make sure that you know you find all the open doors and weaknesses. If you have them you close them as quickly as possible. You make sure that all your doors are locked you know that your windows are boarded up and so on and so forth. You know that your fences have no weaknesses and you know that's kind of that's kind of how I look at you know vulnerability process and then you know it depends on how do different projects you know live up to that. But before I go into this let's just look at some of the common patterns which are around today in a very basic way. So the first pattern is full disclosure. So what does that mean? So you know we we have our we have our buck being reported. You know probably in this case you know to a guy or a small team or security team they look at it and they put a description together and then they announce it you know possibly on a mailing list and work on a fix. Eventually you know the fix is available and then probably make a little bit of a larger amount of announcement and get everyone you know everyone else you know to deploy that that fix. Now the color scheme there is a little bit is important because red means you know a large number of people know you know about that issue and they could potentially exploit you know a vulnerability or bug and during that you know during that red time period and there is a traditionally has been a little bit of a debate on responsible and full disclosure but I'm not kind of going to that you know just Google for it. There's been lots of controversy around it about what's better and what isn't. I kind of wanted to move on from that and push this a little bit further. So I was talking about responsible disclosure already so that's the second pattern which is commonly used and what does that really mean and how does it compare to full disclosure. Well you know basically our buck gets discovered you know somebody looks at it this is all happening in a small very small team in private you know behind that security at list. You know the people who are on a security team will basically probably work together with the reporter and fix the issue and then they will you know create a patch at the end of the day. That patch goes then to something which is typically called a free disclosure list. So that's a list you know of people who have certain privileges you know like it could be it could be a number of key users or you know it's not the entire world it's a sort of select smaller group of people who get to know about the vulnerability and can do something with it and while going to that the kind of different models there in in in some more detail a little bit later because what's really interesting is that there's large differences in how different projects deal with this. And then you know of course you know after the pre-disclosure period you announce the list publicly and then everybody else you know deals with it and updates the systems. So that kind of gives us a little bit of the basic terminology and you know just to remind you this whole process is really about keeping you know keeping your users safe and to some degree also minimizing risk you know in a bigger you know in a bigger picture and you know I mean coming back to that zombie analogy right lots of zombies there on a right and you know are you a little user there on a on a right some is on a left you know trying to fight them off. But I wanted to explain explore first really what does safety really mean you know what do we mean by safety what are the key elements you know of it. Well the first thing really to realize is for this whole process to work actually you need to encourage you know the people who find vulnerabilities in your software to actually report them to your project and and it's really really interesting because actually they're totally in control you know as a as an open source project you know you can't force somebody to report the buck to you you can't force them to not you know do something bad with it. And I really you know of a bust process encourages people and companies you know to report things to you. And we'll get to some of this a little bit later when I will talk about some of the you know lessons I learned this in you know the project I'm responsible for how we dealt with various crisis you know which then led to changes to how we manage vulnerabilities. Then you know the second key part is really that once something is discovered the issue has to be you know fixed as quickly as possible you know you just want to minimize you know the time from when something is known to it actually being fixed. And you know really you don't want to have any unfixed vulnerabilities lying around. That can be quite challenging operationally also depending on the size of the projects how many people you have on your security team and so on and so forth. But we're not going to address that in that much detail. And then you know once once you know once a bug is fixed and you communicate it to the outside world really you want to make sure that you know the exposure time is minimized and you know that your user supply patches quickly. Now there's actually not that much you can do as a project there you know the only thing you know you can do is transparent about you know the issues you have you know make them public and you know maybe in some cases you know PR helps to get users to update their systems but you know ultimately you don't really have any control you know over you know over that side of the cycle. But we will actually cover and talk a little bit later about things like PR and how the media sometimes colors vulnerabilities today and what this may mean for your project and what that you know also may mean for users. So I just kind of wanted to visit bit of terminology we just introduced. I wanted to start looking at some of the of how different projects which are important in cloud computing handle these kind of things just to sort of set the scene a little bit. So actually there's few examples of projects who do full disclosure. So for example and then there's a color scheme in that sort of yellow is there's a problem there white is sort of usually okay and then red that's somewhere where there's a problem. So if you look at the first two lines it's actually really interesting. So in Linux if you discover an issue in the Linux kernel it very much depends on how you report the issue depending on how you report it that it gets handled differently and so there's different groups who you know who handle security vulnerabilities and I have different different processes. So the example you know if you report it via OSS security which is a public list you know it just gets published it's full disclosure. If you report it via a security kernel org they'll look at it for five days they will try and do a fixed period of time and do a bit of triaging but then it'll go out for full disclosure. Other projects you know like OpenStack, Humi and a lot of them they use full disclosure for low impact issues but not necessarily you know for important issues. Then you know we have a whole list of projects which use responsible disclosure you know again you know Linux kernel if you report via OSS security distros that's another you know public mailing list it goes through the full responsible disclosure approach. Most Linux distros, QMU, OpenStack, OPNFV, Open Daylight you know same project and so they all use that kind of pattern today. Then there's I added Docker into that category as well so Docker is kind of interesting because they you know they say they're useful disclosure and I actually know that they do but you go to the website and then you try to find a policy document and you can't find anything. You kind of you have to ask and if you're you know if if you're an important Docker user then you might get access you know to their policy document. And then we come to the area where we actually have problems right. So there's a lot of newer projects or also other important projects where it's not clear at all you know what they do. You know Cloud Foundry for example there's no clearly stated way of how to deal with you know securities, there's no published CVEs, Chorus, same thing, Kubernetes, same thing and it's that that's sort of true for a lot of the newer projects which have you know which have started more recently. I started giving this talk for the first time in in August last year and actually what's happened is you know that the people of those communities you know it was actually even worse then. At that time it wasn't even a way to report the security issue to Kubernetes and there was a Kubernetes guy in it in the audience and that just added that you know afterwards. But you know coming back to the original question that kind of already tells us that maybe there is something you know wrong. There's a lot of inconsistency and different approaches going on today. And part of my talk is also about you know well if you work on these projects you know go back to your communities and get them to think about that and how to address this because ultimately in the long run you know that's going to be bad for you and for their users for your users. I just wanted to sort of look at you know like pick out you know an example looking at why you know this orange this orange bit at the bottom with some of the new projects happens. And there was an example. By the end of 2014 there was a quite severe networking vulnerability in Open Daylight and you know the Open Daylight team it took them nearly six months you know to figure out how to fix this. They didn't have a process. They didn't think about all this you know upfront and then they actually realized that this was an issue for them and once they fixed it you know they basically made a bit of PR about it. So if you Google for you know Open Daylight network and PC World you'll find the whole the whole set of articles around it. But you know really what they then decided was to just look at how OpenStack does things and they just copied their process at the end of the day. And I think we're seeing we're seeing more you know more examples you know of that now. There are a few good off-the-shelf processes out there like the one which OpenStack uses and send users and you can just you know you can just copy them and customize them for your for your needs. So let's just look at this whole responsible disclosure thing in a little bit more detail. And because that's the predominant model today right. And so in a paper you saw before you know most of the projects today seem to fall into that into that bucket. So one thing I didn't really cover you know we had our three phases you know basically first phase was you know fixing the bug, second phase you know pre-disclosure, third you know it's your publication. Now actually what you really want to end up with is some sort of agreed fixed period between you know about being raised reported to you and you publicly announcing this. If you don't fix this it might take months and a lot of proprietary vendors you know they often sit on vulnerabilities you know for months. Something gets discovered and it might get fixed six to nine months later or even years. But we're talking only about open source projects here right. Typically this should be weeks not months right. Then also during this yellow phase when you're actually fixing things and there's a lot of other stuff which is going on as well. You know you have to look at exactly you have to try to figure exactly the conditions under which the vulnerability gets triggered. No other any workarounds, no other any config options. You have to get a CVE number allocated and so on and so forth. There's quite a bit of stuff which happens during this process. Then during this whole pre-disclosure period that's where most of the projects really differ you know quite widely. Fundamentally what you can and can't do with the information which is shared with you. That has a big impact on the users of your projects at different levels and we look at that in a lot more detail because that's really interesting. The key point is you know small differences on how you handle aspects of this process can have very large consequences. So there's your little butterfly effect here. Let's just look at you know an example the disclosure time for example. So you know if you have a long disclosure time you know for example a few months and you sit on that and it comes out you know afterwards there's probably going to be a media story out there which says well you had a severe issue it took you four months to fix it you know you left people vulnerable during this time. So that discredits you as a project and you know actually quite recently there was an article in the register about Oracle where you know where some of these kind of accusations you know were made where they were sitting a long time on some of these vulnerabilities. There are other recent you know examples as well where it took Apple more than six months you know to fix a cross-app resource vulnerability and a bias issue which fundamentally left all their users you know at risk for a very long time. And besides discrediting you know the individual vendors or projects behind it it also discredits the disclosure process you know itself. So a lot of people don't like responsible disclosure because it's often associated with long times between a bug be identified and it being fixed. And in fact you know open source projects seem to be very good in this you know in this area. Now we tend to do this in weeks rather than months. The other interesting thing about this is also long disclosure times also create a disincentive for reporters to report the bug to your project. You know like let's look at security research firms for example you know they make the money from you know from finding bugs reporting them and then getting some exposure in PR after they you know report it. And they don't want to have to wait for months until you know or they will publish a study or paper and they want to be you know they want to be able to do this according to their own schedule and not have to wait you know for you. They might do this once but the next time you know they might just publish the paper without you know disclosing the issue to you. Then another really interesting thing is that if there's a project you if you say it's gonna we're gonna publish this thing within three weeks you know whatever you're setting expectations and that helps you management you know vendors in your ecosystem who might be using you and who might be putting pressure on you. And I have some really interesting stories about this later so I'm not gonna drive on about it. And you know I already covered that most projects typically handle this whole process of it in two or three weeks. So from the time you know somebody reports an issue to you publishing the issue to help you know if you can do this in two or three weeks. So another little element of how projects kind of differ is do you get a CVE number assigned or not you know to an issue you find. And that's really best practice particularly in established projects today who you know who are active in the Linux and Cloud ecosystem. However this kind of can have some unintended consequences and not everybody assigns CVE numbers you know in the same in the same way. Let's just look at this like I'll just so actually the CVE databases you know CVE numbers are just a unique number to look at a vulnerability and then you can find out information about it and there's some really cool websites such as CVE details to get some statistics about it and ratings and so on. And what happens in practice is that decision makers you know like people in big companies who might look at your technology you might you know they can often use that kind of information to evaluate whether your project is secure or not right. Now I'm going to make the case that actually you shouldn't do that because it's sort of bollocks but you know let me get to that a little bit later. So here you see some of the stats you know around vulnerabilities which have CVE numbers which show the history of you know XEN and what you see here is that in the first few years there were very few vulnerabilities right. Now you could suddenly you could think that XEN was a lot more secure around that time but actually if you look at this is we didn't have all this really meant was that during this time we didn't have a process which required creating CVE numbers right. But those kind of assumptions which go into those stats they're not tracked anywhere right. So if you didn't start comparing you know projects based on those kind of information you might get it totally wrong. Now also you know if you for example look at some of the media coverage XEN got all of last year there were so many media stories about one vulnerability after the other but actually in reality probably you know in absolute numbers last year was quite a good year compared to the three years beforehand. So a lot of this kind of stuff has to be taken with a pinch of salt and you have to know you have to have a lot of background you know information to make really inform informed decisions if you if you actually want to you know use CVE number information to make decisions about you have to find out how secure project it really is how many you know how to deal with things and so on so there's other factors as well. You can't really use CVE databases to compare technologies fairly for other reasons as well right. So first of all you know you don't know when those numbers are assigned so we covered this already and you know some projects hardly ever do this you know there's only do this for the most secure ones for the most severe ones. You know some projects don't assign CVE numbers at all but then if you don't find any and then you consume all that project has no numbers so they have no bugs so it must be more secure well you might be totally wrong right. Some technologies can't easily be mapped you know like for example KVM on containers they're not you know they're sort of well known and branded technologies but if we look at it at a component perspective you know what is really KVM and what is LXC right most of the vulnerabilities which affect Linux and you know are classified as Linux vulnerabilities don't show up in KVM or LXC stats and you know sometimes CVE numbers kind of you know affect several products so this all has to be taken with a you know with a pinch of salt. So now I wanted to look at you know where are the key differences and I'll start telling a few interesting stories so we covered you know a little bit about the overall lengths of the process you know we covered a little bit around CVE numbers but what I wanted to focus now is you know is what happens during that previous closure period and what can happen and where are the key differences there and what's key here is that actually what you can and can do if you're you know a privileged user who gets information to you know to security related information before anybody else really depends on what the goals of the process are. So I wanted to look at some of the common goals which are around there so you know the first goal and that's a very sensible one is that you have a fix available before it becomes publicly you know before the information is out there. Another common goal is that downstream projects you know projects are products you know which take your open source product and repackage it in some way or shape form you know if you look at the Linux kernel then all your distros kind of do that such that they can test you know take that patch a package it up through the Patrick packaging manager manager or distribution system and test it in the environment right but more recently we've also seen a lot of other goals and that's where the whole cloud angle you know comes in. So you know one goal for example is you know let's let's say you know let's let's say we're talking about you know a service provider who operates at really big scale you know such as Amazon or Rackspace or you know some of the others. Well actually a lot of them it you know they need to make sure that you know when they upgrade the entire fleet or something that they have staff in place and they have to put plans in place then they have to do some you know certification and testing and that can take a week or two right. So actually for them you know so if you then follow you know a process by which you treat them like any other user then they might be vulnerable or their users might be vulnerable for a week or two until they kind of plant the whole thing you know roll it out and you know deployed it that can be that can be a very big you know issue for their users and for their businesses. And you know the extreme really is that you don't just allow those vendors to to start planning an upgrade you're actually allowed them to deploy an upgrade during that embargo period and really what that means is you know like you know they get the information about the fix they start planning they deploy it and by the time you know to fix the information about the issue it becomes public their cloud or their services upgraded and all their users are safe right. So these are some of those goals and then you know those goals really determine what can be done with the information which is shared with you and you know who can be on that list who can who's privileged and can be on that pre-disclosure list. I wanted to classify this a little bit you know into two buckets now so we had those four goals. The first two I call it at the distro model you know that's that whole model was targeted towards Linux distros who take you know open source code and I package it up and I test it right. It's typically a very small list of vendors you know they get this stuff and then they treat all of their users the same and you know as soon as you start doing something special for service providers you know I call it at the cloud model. Obviously at distro model I was pre-cloud computing and you know as I mentioned earlier it does leave service providers vulnerable. So for example you know if I looked at a KVM based cloud today you know as a KVM based service provider I don't get notified of a security vulnerability before anyone else right. So you know when a vulnerability is published then I have to upgrade my system and you know at some point I'm done and during a time period my users might be vulnerable to that you know to that to that specific security issue. The same might be true you know the same is not true for other you know for other technologies. Then that cloud model that actually really only emerged recently in the last two years and that is something which recognizes the needs of service providers but it's creating some specific interesting challenges. So looking again you know at this model mapping them to technologies right. So the distro model you know Linux kernel follows it if it's reported by a distro list or you know Linux distros do that and most you know most other open-source projects which are relevant in the cloud and follow this model today. Where you know the sole purpose of the whole process is to fix the bug, package it up and test it in you know in the environment of those projects and their vendors who use it. The cloud model that's kind of actually only used by very few projects today. So OpenStack does that for intermediate to high impact issues. OPNV and OpenDaylight do the same. Those projects have a close relationship to OpenStack. They're basically just copied what OpenStack did. Same project does the same and actually Docker also does that too. So I mentioned some trade-offs between those you know between those two models. Well the first one is actually and I covered this sort of not so directly beforehand. It's risk for users right. So why do you want to treat you know cloud service providers differently to you know somebody who just uses your software you know in a smaller environment. Well the main reason is that they have a lot of you know users themselves. A lot more people are basically being put at risk if that vulnerability you know comes out. And that sort of or gets exploited. And that kind of leads almost right into the next point right. So it's also risk to your project's reputation. Now I've been working with the Accent Security team for a long time and actually we got a lot of flak but even so you know when there were security issues over the last year and a half and sort of a lot of very negative press stories. But actually what was really interesting about all these stories right is that at no point you know any users were really put at risk. Any real cloud users were really put at risk right. So you know the stories I don't know where you remember you know like at NF 2014 Amazon had to reboot a portion of their cloud and then the same thing happened again at the beginning of last year. And it was this huge news story. But actually it was a story about Amazon's users and you know other service providers users being inconvenienced rather than being put at risk. Now just imagine what this what the end you know that damage our reputation at the end of the day. Just imagine what the damage would have been if you know if users would have been really put at risk. That would have been a whole magnitude order of magnitude bigger right. Another trade off is about risk of information leakage right. So we were talking about this process earlier you know where a number of privileged companies organizations open source projects get access to information about vulnerabilities before everyone else. Now that's called you know well and a privileged list of people we call that pre-disclosure membership. And the more people you have on that list or organizations on that list the bigger there is a risk of somebody you know leaking this information and it can't going out right. And the traditional you know if you look at some of those numbers you know this door model typically a number of organizations on that list about less than 10 people or less than 10 organizations. If you have to include you know service providers it's a whole order of magnitude bigger right. So the whole you know potentially you know the likelihood of somebody leaking some information and something bad happening you know goes up. And the main argument today against the cloud model is is around you know this whole risk of information leaking. Now interested enough we've been following for nearly two and a half years this model was in the same project and we have about 75 people organizations on a pre-disclosure list nobody has ever leaked anything. Well I mean there have been a few hiccups where you know where coordination issues and somebody might have disclosed some information two or three hours before the agreed time. Because you know they got their time zone conversion wrong but that you know that's kind of that's kind of manageable. So in my view that's not actually really an issue. And then of course if somebody leaks something well then you just fall back and follow you know follow the full disclosure model and you just publish it straight away right. And then you have sanctions as a project as well so if somebody leaks some information you strip them you take them off the list they will never get that information again. It will be damaging you know like let's just say you know a big vendor such as you know okay let's just say Amazon right. If they leaked information which I don't think they will ever do we will take them off the list. And that would be very bad for them in the long run. Another interesting aspect is around fairness and let me just you know like so and that's kind of challenging right. So who do you allow to be on a pre-disclosure list. When we did this originally for the exam project we kind of you know we kind of started out with what everybody else did you know distro model and then we had Amazon you know on it as well. But that created all these questions well actually if Amazon is on it why should another service providers be on it right. Why should one vendor be treated differently to the others. So you know eventually you know this created some challenges within the community and we had a very big you know discussion about this and I'll cover that whole element in the in the full story section which is coming now. Now the fairness problem is actually a really hard one you know to solve because at the end of the day what we have to do now is actually if you're vendor on open source project who's on a pre-disclosure list you have an advantage to a book standard you know individual users right user right. But at some point you have to make you have to draw a line and the only way how you then really can reconcile that is have a have an informed discussion within your community and come up you know with something which works for you. So those were stories. So this section is really about how did Xen come to the project to the process we have now and it kind of gives you a few insights of how the process also works. So we didn't have any process at all really until you know 2011 and basically we just took what Debbie had and replaced Debbie with Xen pretty much. There were a few you know we were having a goal of allowing us to fix issues in private and then allow distros you know to package it up. We did at the time allow service providers to prepare an upgrade so we would notify them that an issue was coming they could test it in the environment but they weren't allowed to deploy it and during you know during an embargo period. And basically you know people who could be on that pre-disclosure list there were Linux distros, open source or commercial and a couple of very large service providers. One thing which was very interesting is we had no we had no fixed disclosure time. We didn't say you know we're going to fix an issue within three weeks or something. You know this was undefined. Now that caused us some really interesting problems. So this whole section about wall store is what you know like the way how this is going to work is we have a you know we have a starting point and we have a crisis and how do we respond to it. So the first crisis was in July 2012 and it was a sinkhole to intose the sweat vulnerability and it affected you know Xen, Microsoft, NetBSD, FreeBSD and you know quite a few other projects. Now what happened at the time is you know that bug was reported to us we prepared a fix you know we wanted to publish it. Two days before we were going to publish the information you know about the vulnerability. A very large predicts you know a large pre-disclosure list member. I can't tell you any names otherwise I might get fired. Their CEO fundamentally rang every other CEO in the community and put and tried to put pressure on them to stop that information from being released. And because you know so many different organizations got involved in this that caused a lot of friction and resentment you know within the community. We in the end at the end of the day stood our ground you know we said we're going to release this on this day but what then happened is fundamentally you know the CEO of a very large organization managed to talk to the person who found the bug and they then told us oh you know I think they may have paid them some money or something anyway the date was being pushed back because the discoverer of the issue suddenly said oh no you know we now want to have this published a month later such that everybody else has more time you know to fix the issue. So that caused a lot of problems you know in the community and the key issue we had is we didn't have that sort of fixed period. So what we did is we started a consultation about how we deal with security in the community. This process was very painful and it took it took nearly a year and it centered on a number of different topics you know like one was the easy one you know timing. So you know we basically now have a fixed schedule where we try to fix an issue within a week and we have two weeks free disclosure period and if we don't if we don't manage to deal with the issue it gets published anyway. And in fact we only had one instance where we didn't manage to keep to the timetable and I'll get into that example a little bit later. It also focused on a lot of these other topics like who should be allowed on the previous closure list you know if we allow XYZ on that list how would that impact other people. So there's a lot of discussion around fairness and that was kind of really interesting because a lot of very small service providers you know they felt that a security issue could put them out of business and they felt that somebody very big you know like Max both. So actually what we did then is you know we looked at this whole thing again we clarified the process to make it a lot more waterproof but then we also said well actually why do we have this restriction why do we the why do we not also allow service providers to upgrade during the embargo period. And the only reason why you wouldn't want to do that is if somebody could reverse engineer you know from a deployed system what the actual vulnerability is. But in the majority of cases if you run a service you're not actually you can't actually poke around in the actual code. You can't see the diffs between binaries so you usually can't you know reverse engineer what the issue is. So why why do we have this restriction in place in the first place. So what we did so we carried that and you know made relaxed that whole process. And we also you know found we made other things you know we made the application process clear we make the whole application process happens in public so there's a public list you apply for membership other companies that people might comment on it and everything happens in public. Also pair vulnerability the state can you deploy you know can you deploy during an embargo or can't you you know what exactly are you allowed to do in this situation. And another really important thing is we put a mechanism in place you know that was a private mailing list for members of that pre-disclosure list to collaborate. And that has actually helped us quite a lot because that means you know like sometimes you know somebody takes a patch and then tests it's in a very old version and then shares it with everybody else so that means less work you know and for the for the security team. So anyway that's been you know that's pretty much where we are now. Then Venom came along and Venom was the first time well it was the first time we were affected by a branded bug but also it was the first time that we couldn't keep two hours three weeks right that wasn't you know we basically didn't have a week to fix the issue and two weeks for Venom to prepare for it. And the reason was that this was a Q-new bug right so what happened is you know the discoverer of the bug you know they raised the Q-new bug through us that didn't want to have to deal with the Q-new security team at the time and we didn't have to wait for the Q-new security team to fix the issue and so we ran out of time. And what that then led to is that you know Venom only had like three days you know to prepare and deploy the patch. And anyway to cut the long story short you know we have a retrospective process and you know what we're now doing is if we don't you know if there's a chance you know if we don't manage to provide a fix in a week we warn everybody anyway there's an issue but that we don't have a fix and please help us right so at least they can prepare and you know what's coming. So you know what lessons you know have we learned from that. Well actually that larger pre-disclosure list hasn't really caused an issue for the project there have been no leaks. We haven't had a single well we haven't had a single zero-day vulnerability you know I mean obviously there might be some which we don't know about but that applies to everyone. The other really interesting thing is that a well-run process builds trust. So you know we've learned so much about the Venom's in the ecosystem by working through them and so we have now this capability to you know to collaboratively improve things as we go forward and fairness you know is a difficult issue and then there's always some practical issues which you can never you know get around such as people will differ you know interpret policies in different ways and you don't always get it right and I'll just skip this quickly. Now one really interesting thing is security in media hype because if you do things in a very transparent way and what will happen you know you're giving the media a chance to talk about this and it doesn't or that isn't always very nice. So and that is something you know really if you if you are running a security vulnerable process for an open source project you have to be very aware of this. So every single time now there's a security vulnerability related to XEN that's probably gonna be a new story about it. You know these are just some examples you know every single time we have a vulnerability there's two or three articles about it and it doesn't matter how how you know whether it's critical or not critical you know it's always gonna be a story. So why is this the case? Well first of all security stories are hot you know they're click baits people just click on them and because XEN is widely used and there's this indirect link you know to Amazon and other big users it makes it even more interesting and you know what I suspect for example you know like Docker you know I mentioned earlier that they don't publish the security policy I have a you know they're also hot project I have a feeling that you know they're worried about that kind of you know that some of the information there might be exploited. It's too easy for reporters to write a new story so you know like whenever we work on a security vulnerability there's a website which tells you that something is coming so that means and it tells you when we're publishing it so that means that reporters they go to this website and then they know exactly you know when they need to look again to write story. Now what's interesting is now we are in the community we have these values around transparency and everything has to be transparent so we can't get away from this. Other projects like OpenStack are slightly less transparent you know about this right? They largely follow the same process but they don't advertise that you know on a certain date you know security issues coming so they get picked on less better press. So coming back to the original you know question you know other security practices robust enough for cloud computing well I think there's a very wide range of approaches and really you know software stacks are getting more and more complex there's lots of layers and there is the weakest link you know in your software stack from a process perspective. There are some good best practices around you know like what we and OpenStack do and more organizations more open source organizations are copying that model so I would encourage you you know to also do that going forward. What's slightly worrying is that new projects don't think about security management straight from the beginning you should really think about this very early from very early on and there's the whole thing about post-snowden you know media pressure and that will never go away you know we were hoping last year that you know the register and other press outlets would get broad-risk and related security stories but that's just not going to happen they will just keep on writing about it and what this is doing is it's actually forcing us as a project to address some issues like you know an example is you know like cloud reboots were very big stories so what we're now doing is as a community we're implementing functionality which avoids cloud reboots you know such that you can just take a security fix and deploy it on a running system and then that whole segment of issues goes away you know we have a similar thing around we had a lot of issues which were caused by QMU so we're sandboxing QMU you know was in the platform so you know what this whole media scrutiny is forcing us with is to tackle some of those common words and where we get media coverage and you know just remove those those angles such that the reputation of the project doesn't get damaged a long run and that's really it if you have questions just go ahead yeah so yeah so I think this comes down to how many you know it comes down to how many users are impacted at the end of the day now you know like traditionally as an open source project you often think about your direct you know your direct users right and you might have a relationship to them but what service providers not you know like you have one user you know maybe one corporate user who runs a very you know a very large you know service and they then also have users and that number of users might actually you know outnumber the number of the direct users you have right so at the end of the day you don't have to make the decision what what creates the most damage to the most people you know for you as a project and that's kind of you know like you have to look at the same way that AWS Alibaba tens you know all those big cloud providers if you take all their users together that's a whole magnitude order of magnitude bigger than you know individual users which and which are around you have more exactly you have more impact yeah so this you know so by by one of those vendors not up that not being updating me being able to update the system in a timely manner you might suddenly impact you know millions of users and magnify you know the problem a lot you know a lot more well some of them do that some of them don't right so and of course you know even if you have something customized you you know you still use probably a common common code base right well but they wouldn't necessarily tell us that right so and why would they write so it's conceivable but you know I don't know whether you know I mean there was actually a series of articles from Amazon that what they're doing is they intentionally do have they run very different configurations to do risk the installation sometimes right so what they I don't know what that means in practice you know that might have slightly 10 slightly different versions of 10 running the same thing such to do risk that you know that's what it sounds like but I don't work for them I don't know any more questions okay thank you very much I hope you enjoyed the talk and thanks for sticking around for so long