 So thank you for coming to this talk. My name is Hosefa. I, for people who don't know me, I am a security engineer with Red Hat. I have been a Fedora contributor for around the last seven or eight years. Started working with something called Fedora Weekly News. And then I did some infrastructure work and I maintain a lot of security packages in Fedora. This talk is on state of Fedora security. Basically, I came to flock last year and since I have been closely working with real security, I have been doing a little bit of Fedora security work as well. I figured out that the security team which Fedora had was not very functional at that point. There were several attempts by several people to try to ensure that, you know, we at least have some structure around Fedora security, but nothing much happened. So I came to flock last year and I had a couple of objectives and my aim was to try to see if those objectives could be implemented. So last year I had these objectives. The first one was security scanning of packages which enter Fedora. We do a very good work of, you know, trying to do a package review of new packages which enter Fedora. There is a package review tool. They are a very good package review guidelines which are probably much better than other distributions out there. But one important thing which is missing from the whole process is we don't actually review the new packages from a security point of view. So technically if there's a package which has a backdoor or, you know, if there's a package which has not been very well written, if it has a lot of security flaws, then the current review process and the current review tool does not do a very good job of trying to figure out if the package is secure at all. I also wanted to have a package exit policy. We have a package entry policy, but we don't have a package exit policy. The thing which came to my mind is there are a lot of packages which don't have a good history of fixing security flaws. There needs to be at least some policy around that to ensure that, you know, if there's a vulnerable package and the maintainer does not care about the security flaws in the package, then we have a way of, you know, trying to get that package off the distribution. I also wanted to work on Fedora Security Dashboard. This is basically a web interface which shows how many packages, how many vulnerable packages we have in Fedora, what are the different security levels which are open against different packages. So, you know, one website where you can go and you can figure out, you know, which are the best packages to use or which are the worst packages to use. So a single dashboard, a single website where, you know, you can see all of this information. And I also wanted to have some way of scanning commits. So this is what I often tell people that Fedora is hard on the outside but soft and mushy on the inside. What it basically means is that if you need to get a new package in Fedora, there's a review process which you need to go through and stuff like that. But once your package is inside Fedora, once you are a package maintainer, then what I could probably do is I can introduce a backdoor into the package and probably, you know, nobody will even know or, you know, nobody will be able to figure out. So it's very difficult. Once you get a package inside the distribution, then, you know, once you have a maintainer or once you are a proven packageer, then you can technically get into any one of the packages. You can introduce a backdoor and stuff like that. A couple of years back, a guy mailed me saying me that, you know, because I'm a proven package, he wanted me to introduce a backdoor into RPM and he offered some money as well. So it's not impossible. It can be done. So we need to have a way to figure out that, you know, any new commits which are done to existing packages are scanned as well. So in this talk, I'm going to talk a little bit about the progress which has been done on all of these objectives. I have two demos to show. There's a security scanning tool which we are working on and which we will make public soon. I have that demo and I have my colleague who worked on the Fedora security dashboard project. So he has a small demo as well. I have a couple of objectives which I want to implement for the next year. There is one on which is called a container held catalog, which is a very interesting concept currently used by Z hat. And I want to work on that. And you know, I want to make sure that we have some similar concept in Fedora as well. So security scanning of packages. We took the Fedora review tool, which is used to review new packages. We patch that and we add a couple of security scanning stuff as well. And in some sometime I'll show you the code and I have a small demo to show as well. What this basically does is when you scan new packages, apart from the normal things which need to be there in compliance with the packaging standard, there are a couple of plugins which does security scanning. Some of the things which it does is number one, it looks at the source code. If the source code is via GitHub, it will look at all the commits. It will try to find suspicious keywords in those commits. So, you know, if there is a keyword which says fix buffer overflow or fix integer overflow, or if there's a cv associated with the commit, it is going to flag the commit saying that this commit needs to be there in the package because it may be security related. It looks at a couple of bugzillas. It looks at the Z hat bugzilla to see if any cv is open. It looks at the bugzilla because the bugzilla does a very good work with security. So, it looks at the bugzilla as well to see if there are any cv is open. It looks at how temporary files are used. So, there are a couple of things which it does and it will report all of these things back to you saying that, you know, I have found five commits which probably are security and you need to ensure that those commits are in the package which you want to introduce in Fedora. So, it does a very good work of that. I currently use the new tool internally and I plan to send all the patches to the upstream as well. The only problem which we have right now is a lot of these plugins are dependent on internal libraries which we use. The libraries basically talk with bugzilla and stuff like that, but at least three or four of these plugins are completely independent. So, I at least can send those upstream. The upstream guys can have a look at it. Like I said, we are planning to decouple those new plugins from the internal libraries so that upstream can use it as well. So, I hope the demo works. So, what I have basically done is I have run the tool before the conference because we have some networking issues on site as well. I have done that also. I will try to run the demo now also to see if it works. I have intentionally chosen OpenJPEG 2. So, OpenJPEG 2 has got a very bad history of not fixing security flaws. So, I have intentionally chosen this package so that we can get some good data out of it and we can see how the tool basically works. I want to show you a little bit of things which we have changed. So, we have added a couple of plugins over here. There is one plugin called PST upstream. So, PST basically is product security. So, there is a plugin called PST upstream. What it does is it looks at upstream source code and it tries to pass the source code and stuff like that. There is a plugin called PST temporary directory. It looks at all the instances of how temporary files and temporary directories are being used. If any of them use predictable file names, predictable file names in temporary files and directories are bad. It tries to flag those. There is a plugin called PST Debian Bugzilla which looks at open security flaws in Debian Bugzilla. A couple of internal things which we use. There is a plugin called PST GPG keys to see if any GPG keys were by mistake packaged with the package. There is one called PST yumrepo. Some of them are internally relevant. Some of them are more relevant to upstream use cases. Can you increase the content? Yeah. So, these are the plugins which we are using. So, to run this tool, I am going to run it with a no build option because we don't want to waste a lot of time trying to build a tool. Let me go out of this first. I think it's called Trivia. I am just going to run this tool with a... I am not sure if we have enough time to let it run, but I wanted to show you a few important things. The message which you see is it's trying to... It tried to figure out the upstream URL from the spec file and it figured out that it was GitHub. So, it's trying to look at all the commits. One very important thing it also does is it looks... If you are using the latest version of the package. So, we have had cases in which the latest version was probably 1.5, but the packageer wants to package 1.4, which is a problem. So, it looks at all the releases which have been done by GitHub and it compares the release with the version which you are planning to package. And this is going to take some time because it needs to download the entire Git tree and then it tries to pass the Git tree and stuff like that. So, what I am going to do is I am going to stop this at this point. And as I mentioned, I zanned this tool before the conference. So, I just wanted to show you how the review file looks like. And this is the standard review file which we have, once the tool is done. What I wanted to show you was the security part. So, these are the various plugins which are done. If you scroll down, then this is the Fedora bits at the end. So, these are the Fedora bits to make sure that the package is according to the packaging standard. So, I wanted to show you the security bits. And the first thing it found out was that upstream code has been examined for obvious keywords and this basically failed. So, it found a lot of upstream commits, which were probably security. One thing which you need to remember is it does not have the ability to check if it is indeed security. It is looking at key keywords. Like I mentioned, buffer overflow, integer overflow, security, CVE. And it's trying to look for keywords like this. And if you go to the end, I am just going to quickly go to the end where it shows you the list of all of these commits. So, if you see this one, it shows you a list of all the commits which it thought was security. If you see the first one, this is probably very obvious from this. It may not be security. The second one may be security because it says detect invalid file dimensions earlier. So, probably it's trying to parse a file which is maliciously designed and it crashes at that. So, this is probably a security commit. So, if you see the second thing which it does is, let me go to the top again, there are no open security issues in upstream bug trackers. So, it goes to the upstream GitHub bug project and it looks at all the issues which are currently open and it tries to figure out if any of these issues may be security. So, I am just going to quickly show what it found. This is the end and then upstream. So, it found a lot of security issues in upstream bugzilla. There's one which says decompress seg fault which definitely looks like security. Then there are a couple of null pointer dereferences. There are a couple of buffer overflows which is definitely a security. A lot of buffer overflows. There's an integer overflow which is security. There's a str copy which is security, integer overflow, malloc overflow, and the list goes on and on and on and on. So, technically if you were to package this for Fedora and I don't think the package maintainer cares enough to go to upstream bugzilla to try to figure out if there are any security issues. Even if these security issues are not being addressed, at least we are now aware that this is probably a package which we may not want in Fedora at all because I am not sure if we really want to give a vulnerable package to our customers. So, if you see the list, it actually goes on. There's a pretty long list. So, after doing that, what it also does is it looks at Z-hat bugzilla to see if there are any open security issues. It looks at, if you scroll down, it looks at debian bugzilla as well. So, I am just going to show you that as well. So, if you go over here, then, yeah, this is debian bugzilla first. There are a lot of CVE issues. So, these are definite security issues, right? So, there are a lot of CVE issues which are open. I can see 1, 2, 3, 4, 5, 6, 7, 8, 9, probably 8 or 9 security issues which are open against open JPEG. There is a Z-hat bugzilla as well, which I wanted to show. I am not sure if it found anything. Probably did not find anything in Z-hat bugzilla. There was. Okay. Yeah. This one. So, a lot of issues open in Z-hat bugzilla as well. I can count 1, 2, 3, 4, 5, 6 or something like that. So, a lot of issues it was able to find. What it also found was it found, so this is upstream release which I was talking about. It basically says that the latest upstream release is 2.3.1 and the one which we are trying to package right now is 2.3.1. So, probably this is the latest one which we are using. It will also list all the different upstream releases which were made, the dates on which the release were basically made. One more thing which I wanted to show you is locations which refer to temporary file parts, right? So, if a temporary file was used or if a temporary directory was used, it will show you the different locations where those temporary code is used. A lot of them may be false positives. So, it is up to the package maintainer or it is up to the security engineer to try to figure it out. But at least we have a list of locations which you can probably kind of go through and with a little bit of automation, I think we can remove a lot of false positives from this as well. So, long list. A lot of these are tests which means that they can probably be ignored or some of this is documentation as well. So, you know, we really don't want, we really don't care about documentation at this point and it's a long, long, long, long list. And yeah. It does a couple of other checks as well. So, I'm just going to the top as well. It tries to figure out if there are any yum depository files which were by mistake packaged. May be something which is not relevant to Fedora. One very important thing, it tries to figure out if the upstream is abandoned. We have a lot of cases in which they are students or they are people who just need to do some college project and they try to package anything which is out there. So, I have had instances in which people package anything which is easy to package so that they can get a package maintainer status because they need to do a college project and stuff like that. So, one very important thing it does is it tries to figure out if the upstream project is abandoned or not. So, how it basically does that is it looks at the last commit which was made and if the last commit is probably one year back or two years back then that probably means that the project is not in a very good state. And yeah. So, this is basically what it does. And there are a couple of more things which we are trying to integrate. One very important thing is we want to query the NVD CVE database. So, that is a database of all the CCVs which are out there. So, what we also plan to introduce is take the package name and do a query in the NVD database to see if there are any hits on any of the CCVs out there. So, this is something which we really want to do. One thing which we do internally is we have a co-vidity instance. So, a co-vidity is basically a static analyzer. And what we do is we send a package to co-vidity and co-vidity does a lot of static analysis. This is something which can be done in Fedora as well. We don't need to use co-vidity. But what co-vidity does is co-vidity uses a couple of freely available static analyzers. Like, you know, if you have heard of S-Patch or if you have heard of GCC also generates a lot of warnings when the build process happens. Right? So, all of them are basically used by co-vidity. So, these are basically open source tools. So, you know, this can be done in Fedora as well. So, yeah, going back to my presentation, the second thing which I tried to work last year is package exit policy. It was approved by Fesco. Some small changes were made instead of four weeks. I think we decided to give six, six weeks. And what the policy basically says is that, you know, if you have a package in Fedora which has a moderate flaw or I think it has an important flaw, then depending upon the time in which you did not add less the flaw, we may remove the package from Fedora when the next branching happens. So, I think for moderate, it is for six weeks. If you have a package in Fedora and there is a moderate security flaw which is open against that package and that flaw is open for six weeks, then the next time the branching happens, we will try to remove that package. Right? So, we have a very good package exit policy right now. So, getting the policy approved by Fesco was not difficult, but getting it done by Zillis Engineering proved to be a very difficult job. And it took a couple of months for me to try to figure out whom to contact and, you know, how to get the work done. So, it seems like Zillis Engineering is very busy. And you need to ping them a couple of times. If you see the ticket, I ping them at least three or four times. There was no response at all. Then somebody told me that, you know, I should probably go back to Fesco and tell Fesco that you guys approved this policy. Can you help me with trying to engage Zill Engineering to get it done? So, I had to open a Fesco ticket and Fesco had to go back to Zill Engineering and tell them that, you know, please, can you guys please get this done? And we kind of got delayed by one Zillis cycle, but I'm quite positive. I have not spoken with Mohan yet, but I'm quite positive that from Fedora 31 onwards we will have this policy implemented. So, let's hope the fingers crossed. Fedora Security Dashboard, my initial plan was to get either outreach or GSOC students to, you know, help me with this, but it seems like, you know, I did not properly know how outreach works. It seems like you need to have a couple of simple tasks which the students need to do, and then, you know, you need to make sure that they are able to do it and stuff like that, which did not happen. Probably, you know, I did not understand how these projects work. Luckily, my colleague Bolo, if I'm pronouncing your name correctly, he stepped in and he did a lot of good, good work. After I finish, he's going to give a five-minute small demo of how the Fedora Dashboard works. My plan is to get a running version and host it on some Fedora site, and we will see how that goes. Yeah, so I have some new plans for next year. So, how many people in the room know what Z Hat container health catalog is? One or two. What it basically is that when we have a container which is available for download on the Internet or from registry.zhat.com or, you know, some Z Hat container website, when the container was last spun, there were a lot of security flaws which were found in those components which were part of the container which was spun, right? So, if the container was spun last week, say, probably seven days back, after that, seven moderate flaws were found in components which were part of the container. So, Z Hat customers wanted to know how safe those containers were. Or, you know, how many flaws those containers really had. So, what Z Hat did was they introduced something called container health index. What it basically does is depending upon the number of flaws which the container is vulnerable to, they are going to call it green, then yellow, then orange, then red. So that when our customers want to download the container, they know the state of the container. I have a web page which is open. I want to show this web page to you. So, this basically talks about... This is a web page which is public. It talks about, you know, the magic formula which we use. So, if the container is grade A, then it means that, you know, there are no unapplied critical or important fixes. Then they have various grades and they have various colors as well, right? So, this is something which they implement. And if you see the container catalog, you know, if you download this particular container, then this is a grade A container from a security point of view, right? So, they have some magic formula which they use, which takes into account the number of days, the number of flaws, the impact of those security flaws, and, you know, they calculate the health index. So, I tried to talk with Z-hat legal guys because, you know, this is probably some of the internal things which they are doing. So, I'm still talking with them. We are basically from a Fedora point of view, interested in the formula which they use. We can probably come up with a very simple formula of our own. But the whole point over here is the various containers which are available for download. The people who download these containers, customers, users, or whoever, they basically know that, you know, what is how secure or how unsecured the container is, right? So, this is basically the purpose of doing that. The second thing is, yeah, so this is basically what the container catalog should be doing. I also want to try to make sure that security updates reach our customers faster, right? Sometime back, I tried to open a ticket against Bode guys to, you know, because normally updates take a little bit of time. Then, you know, it needs to be queued. It needs to get the right karma. After karma is given, it is pushed to the updates, where it needs to stay for some amount of time. The problem with this is that if there is a critical or important security bug, then that's going to delay the amount of time which our customers, or the amount of time which our users get in order to download that. So, I still need to think a little bit about that. Probably one method of doing it is to have an updates, testing, security, depository, where, you know, those updates are still in testing, but because they are security, probably, you know, this is, so I'm not really sure about that, but, you know, this is probably one thing which came to my mind. Yes. We have done this a lot of times. So, if I remember, heart bleed, shell shock, and a couple of other security issues, we had to tell Zillis Engineering that, you know, we have something very important coming up, and, you know, it will be public at so and so time, and, you know, it will take us one hour for the package to be built. The package is pushed into stable as soon as you can, right? But then this is only when the issue is critical, or, you know, when it is important or something like that, but the ratio of critical and important issues are much lesser as compared to moderate issues. So, you know, if there's some method of trying to... I'm not saying bypass QE at all, but I'm just basically trying to say that. Excellent. So, last one which I have not really figured out yet, but I believe Google is doing this and we need to figure out if they are using any open-source tools to do it. They scan all the commits, so they are doing it with Google Chromium, in which, you know, whenever somebody makes a commit, they have an engine behind that which scans the commit, some kind of heuristics to figure out if, you know, if there's something wrong from a security point of view, but then they have a huge infrastructure, so I am not surprised that they would be able to do it, but if there's any open-source tool, I am currently talking with a couple of Google engineers to try to figure out if they can open-source the tool which they have and, you know, if we can... I don't know if I can use it as well, but it is a very useful thing to do if you are able to figure out if any new commits or any new tar balls uploaded by package maintainers, if they have any security issues with them. So, yeah, this is all which I had and if you have any questions for me, I can try to answer or I can get below to come and talk a little bit, yes? Do you really want me to answer the question? It was around 50k USD. That's not much, yeah. But it was five years back, so... and I'm not sure if... I'm not sure if he really wanted to pay or it was a joke or something like that, but after I spoke with my manager, he asked me not to go back and ask him if it is real or it's a joke. So, yeah, basically my point of view is that, you know, once you have access, then, you know, the human is the weakest link, right? So, any more questions? I would like to not just have people's computers get rate-limited because of that. Move into the door, see if I can get permission from GitHub to have higher API access. I mean, we already do so many other weird checks at that level. I know, but that's done at every package build. Which you probably would want to do for those. I think that's too heavy for a package build. Think about how many builds we do. I think that's too heavy for a process for a package build, but it is for a view. It still makes sense. The question is just where do you run? The bug tracker checking thing I'm not that worried about, the one I'm actually worried about would be pulling all the good stuff and checking the releases, because I actually attempted to implement a check like that before for checking, hey, is this the latest release? I got GitHub emailing me after like a week of doing that because apparently I was getting so many API calls and so many random refills I thought I was going to take down the front. So, that's not smart. It's not something I'm relish having lots of people do. That's something we want to have a check for. The question is how many new packages do we have every day or every week and how many people are likely to run the packages review tool? So, that's probably not a good use case. But if you have 10 new packages in a week, 20 new packages in a week, then probably running it 20 times or 50 times should be fine, I guess, or check this. So, the reason why we have this internally in our tool because for any new package which is going to be a part of new release, we need to try to look at the package and the security engineer was, like I said, was the weakest link to the whole check because after reviewing a couple of packages, he got bored. So, and he used to make a lot of mistakes, which is fine because it's a human process. Human processes are prone to mistakes. So, we wanted to ensure that we had at least, and the engineer could spend his time doing other important things like probably looking at the code to see if there are any, if it's an SUID, he would binary trying to figure out if all correct programming practices have been followed and all that.