 Hello everybody. My name is Akshay and I'm here to talk about monitoring and managing security vulnerabilities in open source software. The topics I'm going to be covering is what exactly is a CVE. How do you go about monitoring and fixing CVEs? And just the quality of the CVE data or covering statistics about the quality of CVE data and the tools and its shortcomings. And lastly, I'm going to cover security best practices to keep your product secure. And if not anything else, there are a lot of memes so you can have a laugh at those. So what is a CVE? CVE stands for common vulnerabilities and exposure. And whenever a security researcher or a developer finds a software bug which has a security impact, what they do is they notify an organization like MITRE and that organization keeps track of this in a public record or a database where they enter the vulnerability information in that database. And it gets tracked uniquely. However, if a developer treats that bug as a functional issue and does not evaluate the security impact, in that case it would get silently fixed as a regular ratio and it might not make it to the CVE list or CVE database. So your product might still be vulnerable from a security standpoint of you, but it's not tracked as vulnerability from a CVE dictionary standpoint of you. And then there is a whole bunch of undiscovered vulnerabilities which get discovered over time, might be zero-day exploits or might just be caught through fuzzing tools or static analysis tools. So now that you know the different types of vulnerabilities, how do you go about monitoring them? So if you just take CVE data, then there are open source tools which can be used to monitor the CVEs in a product. However, there are a lot of missed CVEs because of various issues with the quality of the CVE data which we're going to discuss in the upcoming slides. And the effort involved in monitoring using open source tools is quite minimal, but at the same time because of the missed CVEs, the product might not be secure. Then you can combine those tools along with some manual analysis of CVEs and in doing so, you can improve the security of your product, but the effort required is much more. So this covers CVEs. If you want to track all the silent bug fixes which have security impact as well, in that case, you might want to look at the security bulletins, issue trackers, change logs of new releases of software and see what got fixed in the latest version of software and then evaluate if it might have potential security impact yourself. And if you want to go into the undiscovered vulnerabilities, then you can run static analysis tools or fuzzers yourself and then discover them. And there are commercial tools available to help reduce the effort, but depending on the tool, it may or may not help secure your product. So you'll have to evaluate the tool on a case-by-case basis before starting to use them. And why exactly is monitoring CVEs a challenge? So if you take a look at the year-over-year data of the number of CVEs being reported, so it has been constantly growing. And in 2018, there were more than 16,000 vulnerabilities which were reported. So if you have to manually analyze each of these CVEs, then it's a big challenge. So how do we monitor CVEs? So before going into the details of monitoring CVEs, I want to just briefly talk about what are the contents of a CVE. The first one is like a CVE ID which is a unique ID given to the vulnerability. Then there's the description of the vulnerability. And NIST adds some additional metrics such as how severe the vulnerability is. And that depends on is it easily exploitable and what is the impact if somebody does exploit it? So it's a score between like 0 to 10. You have low, medium, high, and critical severity. There's also an attack vector field which means that do you need to be physically present at the device or do you need to be like a local user on the device to exploit the vulnerability or can you do it remotely through the network? It also has information whether like user interaction is required or not. And the last key piece of information which is also encoded is something called as CPE which stands for Common Platform Enumeration. And what this is is like it has information as to what the product is. For example like OpenSSL and who the vendor is and which versions of software is affected. So it could be a range, it could be a single version. And this is a key piece because if you're using tools to automate monitoring CVEs then this is the information the tools would look at to figure out if your product is or if the version of software you're using is vulnerable or not. So having talked about the contents of a CVE, so how do we go about monitoring it? So since like Ubuntu and Debian distros have been successful in monitoring security vulnerabilities and fixing them in a timely fashion, so if you consider their model what they kind of do is they manually review every CVE in the CVE feed and then see whether it's applicable to their product. If it is applicable to their product they go ahead and then like patch it, issue security advisories. And not only do they do that, they also monitor release notes whenever a new version of software is released they subscribe to various security mailing lists and then appropriately in a timely fashion fix vulnerabilities. And if you as an embedded developer are trying to mimic that model it's not feasible unless you have a dedicated cybersecurity team. And the reason being there are more than 16,000 known vulnerabilities which are reported on a yearly basis which translates to more than 300 CVEs weekly and you're also responsible for delivering features in a timely fashion. So it's not a practical model. So what are some of the other options? If you were to do it yourself the first step is to gather a list of software running on the device along with the associated version with it. Once you have that list what you can do is look at the NVD feed which is public or go to the NVD website and then like enter that information there and that's going to give you a list of CVEs and for that particular version of software. And once you have that list of CVEs then you can see whether they're really applicable to your product or not. For example you might not be using a certain kernel feature in which case if a CVE is against like the floppy driver and you're not using like floppy in your product then you can safely ignore that. So you'd analyze each of those CVEs, triage them and make a shorter list as to which ones need to be fixed or some of them might not be important based on like how your product is configured. So that constitutes the monitoring portion of CVEs. This is the website which I was talking about. So you can go in there, enter the vendor and product information. It also has an option for entering the version information. You do that, hit search and it's going to give you a list of CVEs for that particular product and version. And if you're coming from a build system such as Yachto, Yachto has this capability called CVE check class. All you have to do is inherit this class in your local.conf and run your regular bitbake command to build. And at the end of the build it gives you a report which contains a summary of all the CVEs in your build. The only downside is the way it stands right now. Both host and host in the sense like native packages and target packages CVEs are all munched together and it's a giant list. So you'll have to shift through it which is cumbersome. So now you have different ways in which you get a list of CVEs. What do you do with that list? The first thing is you need to prioritize which CVEs need to be fixed. Ideally if you upgrade to the latest and greatest version of all the versions of software you have, you should get most of the CVE fixes automatically. But it might not be practical in that case. So what you do is apply various levels of filtering. So like how I mentioned that if you're not using certain kernel features, the CVEs might not be applicable to you. So you can go through the list. So for example, here's an example CVE list based on IMX Rocko release which is roughly two years old I think at this point. It had by default 658 CVEs out of which 339 were kernel CVEs. So after applying the kernel config filter, it dropped on to like 432. Now if you want to concentrate just on the high and critical CVEs core based CVEs, then it further drops down to 239 CVEs. And most embedded devices you don't have like local users logging into the device and interacting with the devices. So you could probably safely ignore those and look at the network attack vectors, which ones can be remotely exploited. So once applying that filter, you further drop down to like 158 CVEs. At this point, I would recommend going and fixing all of those. But if you're in a time crunch, you can also look at which ones have public exploits. So typically references for public exploits are also available in the NVD feed, or you can look at like a website called exploit DB. So which CVEs have exploits and prioritize those because those are the ones which most likely hackers would first target. So now that you have like a prioritized list of which CVEs you plan on fixing, how do you go about patching them? One option is to see if the latest version of software already fixes it in case it does upgrade to the latest version. Otherwise, look at the reference links in NVD to see what is the commit ID or where does the patch reside for that CVE and then back port it yourself. You can also look at commit logs from upstream packages. Typically people do tag CVE information in those as well. And if you do back port them, then you might have to maintain patches because there might be the number of CVEs over time keep growing and if you continue to back port other CVE fixes, there might be conflicts as well, which might not cleanly apply. So you might be on the hook for maintaining them as well if you do not upgrade the software. Talking more about like upgrading the software versus back porting, first question you get asked is why don't you just upgrade to the latest software? Most time like there are API changes, especially in user space components and that means you might have to change your application and if you're using like third party applications, you might be stuck with older versions because the third party application might be slow to make those API changes which your user application depends on. There might be license changes as well. So you might end up back porting which is complex but if you do do the back porting then if there is a public exploit available just run that, make sure the back port is cleanly done such that the exploit is not able to... such that the vulnerability is addressed with the back port. You can also run package tests in Yocto to make sure there are no regression related issues and make sure the back port is good. The last aspect is the practicality of addressing CVEs in the Linux kernel. If you consider the Linux kernel, there is one release every five days or so for the LTS branches which is good, which means it's getting all the bug fixes and the CVE fixes into the LTS branches. On an average, there's like one to two CVEs fixed but at the same time it means that you need to continuously upgrade and run tests and many times product test cycles are greater than five days. So how do you keep up? So this is where like having like a monthly release cadence or when sufficient number of like high or critical CVEs get accumulated then immediately jumping to the latest minor LTS release might make sense. And the kernel config filtering will definitely help because if the CVEs don't affect you then you might not need to upgrade. The other major thing which determines how many CVEs are there in your product is the number of packages in your product and how old your release is. So if you take again Yachto as an example, if you're running like core image minimal even a two and a half year old Morty release of Yachto just has 33 CVEs and the master branch has like just four CVEs. Whereas if you consider core image seto which includes a lot of graphic libraries and various other packages, it has 336 CVEs just from a user space standpoint of you and this does not include any kernel CVEs whereas the master branch contains 14. So periodically updating to the latest version of Yachto makes sense especially if you have a large number of packages. And it's also crucial that if you're not using certain packages get rid of it reduce your attack surface from a security standpoint of you. So now that you know how to monitor and patch CVEs the next step is the real issue with CVEs is the quality of data in the NVD database. So there are various issues. The first one being like inconsistent naming. So if you consider a package like ARM Trusted Firmware there are CVEs listed in NVD database against ARM-Trusted Firmware ARM-Trusted Firmware Trusted-Firmware-A. So if you did not know to look for all those three products you'd miss CVEs against that particular package. There are other issues where there are typos for example 2.23 might be listed as 2.2.3 in the CPE information in which case your automated tool would never cache that and report that CVE leading to a missed CVE. There might be typos and names as well. And sometimes there's incomplete analysis. So for example they may say all versions up to the latest version of software is affected because that's the easiest thing to do instead of figuring out which commit actually introduced the CVE and in doing so it will result in false positives. So if you're in the older version of software which is not affected you might still be, your tool might still be flagging that package as having a CVE. On the flip side sometimes there are entries saying only the latest version of software is affected whereas the prior versions of software also are affected. And lastly there is the issue of no CPE information or no version information leading to missed CVEs. So how do you address some of these issues? The doctor has solutions for some of these. So for example you can specify a CVE product and what that means is you have your Yachto recipe name and that needs to map to the NVD name for a particular CVE. So for curl, CVE might be listed against curl or lip curl. So by specifying the CVE product saying like curl, space, lip curl is going to search for CVEs against both of those in the NVD database. So this will reduce the number of missed CVEs. If the Yachto recipe versioning scheme is different from what's in NVD you can also have a CVE underscore version which will help you do the mapping between Yachto to NVD. Lastly the Yachto community itself is pretty good at backporting CVE fixes and when they do that they tag that information in the patch headers or in the file name itself. So the CVE check class looks for the patch headers and or the CVE ID in the file name to see whether the CVE is fixed. This way you don't have to look at CVEs which are already fixed which helps you save time. All these solutions are good but it totally depends on which version of Yachto you are on. So taking CVE product as an example, if you consider the Morty release which was like two years back against Watts Warrior there's 22 entries for CVE product which is missing. So nobody backported those CVE products to Morty. So 22 might look like a small number but it means that 22 of the packages are not being tracked for CVEs and that translates to 151 CVEs which are missed out of which 96 of them were high or critical. So you need to consider that when you use the CVE check tool in Yachto and this is just including Pokey. Not even talking about the hundreds of other meta layers out there I haven't analyzed those, just analyzed Pokey and that's the data from that. The other aspect of it is the tools themselves constantly improve and recently Yachto moved from CVE check tool to something called CVE update database and that's only in the master branch and what that does is it moves from like a XML feed which used to do like a string comparison to using the JSON feed and doing logical comparison like greater than or equal to or less than equal to for version numbers and just taking three recipes as an example for the same version of the package with the old scheme and new scheme there's like 10 new CVEs which are being reported and that's only in master branch. So unless these patches have been backported to previous versions you might be missing a lot of CVEs because of that and as the tools improve you get a lot less missed CVEs but at the same time it exposes some of the other issues with the quality of data in NVD leading to more false positives so that's at least been my experience using that. So how do we improve the NVD feed data quality? So first thing is if you see something which is inconsistent or inaccurate just send an email to nvdatmiss.co My personal experience is within an hour they update it and their website is updated, the feeds are updated and it fixes the issue for everybody so everybody doesn't have to keep looking at the same CVE saying like oh this is bad data and continuing to ignore them so that's the least you can do. If you see any discrepancies in the CVE summary or references you can submit that information to MITRE and if you see the CVE product which is being missed for a particular recipe submit a Yocto patch or if you want to improve the tools yourself you can do that as well. So there's also another issue with the CVE data quality that's mainly for the Linux kernel. So whenever a CVE is being reported against the Linux kernel what NVD does is like they analyze that and then report it against all versions up to like the latest version let's say 5.2 for example. Then the kernel maintainers propose a fix, mainline it and then the community or the stable kernel maintainers go ahead and then backport those fixes to all the LTS branches. However, NIST does not go ahead and then like reanalyze these CVEs to say okay this fix was backported and they don't update the NVD database for it. This results in a lot of false positives. Just taking 4.4 as an example, if you query the 4.4184 kernel there are more than 400 CVEs being reported out of which 415 of them are false positives. So just waste all your time looking at these saying okay these are not applicable to me because I'm on like the latest LTS release of 4.4 kernel. So those are all problems with the quality of data. Then there is another aspect which is the delay in reporting the CVE data itself. So it can vastly vary. So taking two of the CVEs as an example, the processes the security reporter goes back and forth with the maintainer in private. After the fix has been developed, then what happens is the maintainer either discloses it on the public list like OSS security. If you're not subscribed to it, I would recommend subscribing to it. That's where most of the CVEs do get reported or else it might get reported on a private list like distros where it goes through a seven day embargo period where Linux distributions can patch it so that they don't have to scramble on the day when the CVE is made public. And then once the CVE does go public, the CVE starts showing up on NVD. And after a few days, the CVE is analyzed and the NVD entry is updated to add the CVE information. And if your automated tools rely on the CVE information, it's not going to report the CVE until this initial analysis is done. So it could be up to like, for example, in this case, 68 days before your tool starts reporting that CVE against your version of software. Here's some fun stats. So from when NVD publishes the data to when the initial analysis is done, which is when the CVE entry is added, it could be in 2017 the average delay was 11 days. 2018 it went up to 34 and then 2019 it went back to 10. So it varies quite a bit there. And there's also another interesting thing where sometimes I talked about these silent bug fixes where a bug gets fixed and years later somebody looks at that and says, hey, this had a security impact and then goes ahead and then requests MITRE to issue a CVE against that. So there's an example for a CVE from 2019, which was actually fixed in the kernel in 2016. So three years later somebody requested for a CVE for it and it got assigned. So Red Hat, for example, tracks such information where vulnerability was actually made public. So if you consider that with a grain of salt for a limited number of set of packages, the average delays were roughly 101 in 2007 and it's down to 25 in 2019. So next time when you see a CVE in the news and you're wondering why your tool is not reporting that CVE, that's probably because the delays in the CVE information being added in the NVD database. So how do you mitigate some of these? So most of these CVEs are already being analyzed by Ubuntu, Debian, Red Hat, and most of them have public trackers available. So if you can write scripts which can aggregate that information from Ubuntu and Debian, then you could leverage or reuse some of that information to report the CVEs in a more early fashion for yourself. And Ubuntu also has commit IDs as to which commit fixes a particular CVE and which one introduced it. So the Civil Infrastructure Platform uses that to reduce the number of false positives in the CIP kernel. So another aspect which I wanted to talk about was Secure Boot. So many of our customers implement Secure Boot, Chain of Trust, and then just forget about it thinking their product is secure. But if you go back and look at CVEs affecting, let's say the processor itself, for example, the IMX processor had few CVEs related to Secure Boot in which it would allow unauthorized code to run because of a buffer overflow issue in the checking of the signature. Arm trusted firmware, Opti, which are Secure World operating system code, also have multiple CVEs reported against them. So it's not enough to just initially design Secure Boot. As part of development, you also need to continuously monitor for CVEs against those. And it's not just software which are assigned CVEs, it's also like processors and processor firmware. So if you look at the Snapdragon 410 processor and look at how many CVEs are there, it's like 246 CVEs reported against the processor and the firmware, binary blobs as well. So you need to look at those and manage those because those are typically not included in your build system like Yachter. So given all the challenges with monitoring CVEs, what is a good strategy? My recommendation is a two-fold approach. One is during the design of the product, bake-in security. And what that means is whether you're locking down hardware by disabling the serial console, disabling JTAG, implementing Secure Boot, Chain of Trust, implementing access control like SE Linux or App Armor, and then the other key aspect is having a secure firmware update mechanism such that when you do discover CVEs, you can go ahead and then securely deploy new versions of software, reduce the attack surface by reducing the number of unused components. And the next aspect is staying secure, which is periodically updating your software versions, managing vulnerabilities, monitoring them and then patching vulnerabilities, also having some sort of audit log and monitoring that as well. So from a tool standpoint of you, if you're evaluating any commercial tools or if you're trying to improve the existing open-source tools themselves, here are some of the wish list such that the effort involved in monitoring CVEs can be reduced. One thing I talked about was kernel-config-based filtering. So you can go ahead and then have some kind of CVE to kernel-config mapping so that you don't have to look at those CVEs if they're not applicable, having ability to add notes for a particular CVE, team collaboration, sharing. Another thing is right now with the Octo, whenever you run a re-scan of the CVE, you don't get a comparison report. So you have to manually diff the results and figure out which ones are new which ones are already addressed, so on. And the other aspect is, are there multiple sources other than NVD being used to mitigate the delays in NVD updating the database? So if you want to see some of these features in action, stop by the times booth or check out this particular link, we have a tool which can help with some of these things. So to wrap most of these things up, the key takeaway is there is no magic bulletin managing and monitoring CVEs here. So you need to design security in from the get-go, allow for firmware upgrades, continuously monitor mailing lists, triage them, triage the vulnerabilities, patch and update to the latest version of software. Try and automate it wherever it's possible, but just be aware of the more you automate because of the quality of the NVD data being bad, you might result with missed CVs and false positives as well. And whenever you do see such quality issues, go ahead and then improve the database by sending an email to NIST or submitting Yachto patches for the tools. And if your product does not have security, please don't connect it on the Internet. That's it. Any questions? Fido. That's like more than four years old at this point, I think. Okay. So I don't think I can answer that off the top of my head, but I think it should be fairly okay because the tool itself is like natively compiled and doesn't have too many dependencies other than like Python and URL lib3, I think. So it should be possible to backport, but Fido is not actively maintained, so nobody's going to do it, so you'd have to probably do it yourself, and if you do, you can contribute back if they accept those patches. Yeah. Any other questions? No? Okay. Thanks, everyone.