 Good afternoon and welcome to this talk about tracking vulnerabilities with built route in Yachto I'll first say a bit about myself. My name is Arnaud. I'm an embedded software architect, but that's just a title So I work in embedded software as a consultant Focusing on Linux operating system integration So the the lower levels of the operating system which includes some kernel work, but mostly integration work, packaging, things like that And as a consultant I've worked for many different customers in all kinds of domains multimedia security, home automation, satellite communications, satellite GPS telecommunications Bluetooth I'm probably forgetting some domains as well Also, I'm a maintainer of the built route project which explains why I talk about built route in this talk But for my customers, I also work with Yachto quite often so I can talk about both here So what am I going to talk about? First as an introduction, why do we track vulnerabilities to begin with? I Hope I don't have to spend too much time on that And then about the infrastructure that exists in the world for tracking vulnerabilities Basically CVEs, then how it's done in built routes, how it's done in Yachto I'm starting with built route because that's what I know best So the Yachto part is going to be a little less in depth and then finally an alternative which is tracking vulnerabilities with a software bill of materials And then finally some evaluation. Well, actually the evaluation is During all the rest as well So why track vulnerabilities? Is there anybody who feels that I should even discuss this? I think I put some reasons in there But I think for most people it's obvious that we need to be aware at least of which Vulnerability exists in the products that we make Although what I see my customers is that it's not done. That's conscious conscientiously I expect that this is going to become more and more important. So on the one hand we have Because devices are getting more connected We will see more attacks and they will eventually hurt sales and then then suddenly it's important, of course And on the other hand, we also have regulations coming up. So in EU is discussing regulations that among other things also require products to be to some extent secure Now what is Important is that we want tracking. We want to track vulnerabilities after the product has been sold already To be able to fix things after the fact So it's not just at the time that we build the product. It's also afterwards that they need to be tracked It's also important to realize and this is a slide I stole from a presentation this morning That's So second is one piece of the Security jigsaw, but also tracking vulnerabilities is one piece. It's this piece here So it's a very small piece of the whole but a necessary piece So how can we track these vulnerabilities? The first thing we need is a way to be aware of vulnerabilities One way to do that would be to do constant audits of all the open source enclosed source software that you integrate in your product But that is not feasible And because everybody in the world is this problem a Kind of solution has been created in early this millennium Which is called the common vulnerabilities and exposures So CVE is a is basically a system for identifying vulnerabilities. It's basically an It's not much more than than an ID than a number So that's the main Purpose of the CVE and then there's a database of vulnerabilities. In fact, there are several databases of Cve's the most Central one, let's say is the NVD the national vulnerability database Which is maintained by US governments Institute called NIST so they provide a server which Provides access to this database They also you can send a mail to them to say that there is something wrong in their database and it needs to be updated But it's not the only database. So there are other databases of Of all the vulnerabilities, but there are also many more specialized databases For instance, all the big distros have their own databases of Cve's which are Typically copied from from the NVD database, but only the ones that apply to them. So which are So any anything we know is only is not going to appear in that database Now with the Cve, you have an identifier of a vulnerability And then there is some free text description of what this vulnerability is but not really much more than that It doesn't allow you to identify what is the thing that is vulnerable and so It started with a bit more formal description of how The contents of the Cve should be structured But in I think something like 15 years ago a more formal Definition has been made and that is the Cve the common platform enumeration Which They have different terminology, but basically which allows you to identify a package And not just a package also a version of that package So the way it works is that Cve is a number with a more structured number With a bit of header They're already at version 2.3 because it's already from quite some time ago. And then you have a vendor and products in a version and then there's some more fields which are basically never used So there's also a database of all these products and all these versions that exist or for which Cve's exist and This database is maintained by NIST as well So also to that database you have web access and Command line access And so these two are linked with each other so every Cve has a list of Cve's to which it applies the original idea was that Every time a vendor releases software They also create a Cpe entry and so that entry is added in the database and then when a Cve is Created They enumerate all the the versions to which it applies Turned out that that didn't work very well So eventually they come up came up with also a way to specify a range of Of versions basically larger than and smaller than And then you can have several Cpe entries to which the Cve applies. I have examples a bit later Unfortunately the this Cpe thing does not work that's great Most of the reasons that it doesn't work is actually because it's from a time before open source software was was really popular And so it's not really Applies well to the open source software of today So one issue is that software packages sometimes don't even have releases So you just have to leave it had and and take some random In git commits and there is simply no way to track that in in the Cpe database The the version comparison simply doesn't work It also doesn't take into account patch versions so in theory if you have the Like open as a cell the upstream version And then the one patched by Debian and the one patched by redhead and so on and so on There should be separate Cpe entries for all of those but nobody does that so you basically have one Cpe entry And you don't know if if a specific package that you have if that Cpe entry applies to it or not It's also a bit cumbersome to create them so in theory you have to every release creates a new Cpe entry and The worst thing is that there is no link to the actual software. So it's just an identifier again which for for The close source World was logical because you you didn't have a git repository to point to But for us we want to actually link it to Release star ball or a git repository or something physical, you know, not physical, but something we can touch And the Cpe doesn't provide that And In addition the information in the Cpe database is often incorrect Fortunately, it's fairly easy to correct it So we can't make a pull request But at least you have a contact form where you can fill in the reasons why the Cpe information is not correct And usually within a day or two they will fix it then so here's an example of what the Cpe looks like in for a real Cpe and why it's not correct so the Here is a list of the Cpe IDs to which it applies. So we have several Configurations and then in configuration two we have two versions to which it applies And all of them are wrong actually So first we see here that's in a description. It says before to 228 zero and before 3.0 but then in Here it says from to 220 up to 310 so the before 228 zero is completely dropped from this range. So if you have a 228 zero It will in fact match that that Cpe entry as you think that your Your software is vulnerable while actually it is not The reason that this happens is because at a time that the Cpe entry was added to 228 zero didn't exist yet and Nobody thought of updating it afterwards And then the Fedora thing is also wrong because in fact This this issue was fixed both in Fedora 36 and 37 so the 36 and 37 Doesn't say whether this is Fedora 36 at a time of first release or with all security patches applied So yeah, it's it's kind of useless information as well Still is the best that we have So we will we will use it in In built route any octal so let's take a look at how it actually works in built routes the command to track vulnerabilities is very intuitively called make package stats The reason is called that is because it's a bit historical There was a package that's comments already and then somebody sits. Yeah, I also want to do CVE tracking and Yeah, we can easily add it there. And so package that's got extended to it with CVE tracking What does this command do? Well, it doesn't actually do it in make it calls a Python helper to do the actual have a lifting It will download the CV and CPE database as JSON files and just save those JSON files as disk and those are cached for 24 hours Because the downloading all of that takes a couple of minutes. So it's useful to catch it Unfortunately, there's no way in the In the NVD database to ask for everything that changed since yesterday so They're the only way to update the cache is to download everything again And and unfortunately also things can change even in very old CVE's because maybe the CPE entry has been updated So, yeah, you can't just say I only download the ones from 2023. No, you also have to download 2009 again second the Selected packages. So the packages that are selected in the configuration file of built routes Are cross-referenced with the CPE database? to to see which versions match and then these are Sorry, these are referenced in the CV CVE database to see which CVE's apply to them And then finally some exclusions are applied to it I'll come back to that why that is and the result is written to JSON into HTML The HTML I can make a demo There is my demo so there's the HTML outputs. It's not important that it's not readable I'll just say you have the package name What you need then is a version number and then here is all the CVE information Here is the CPE identifier, which is Determined by built routes if it's green it means it's a valid identifier It exists in the in the CPE database if it's red there is something wrong with it So in this case, there is no corresponding identifier in the top case the Linux Linux kernel identifier does exist, but the version number doesn't exist In in this nice interface with JavaScript We can actually sort all CVEs to find all the packages which have some CVE applied to them Otherwise this HTML is not actually very useful So it's good for showing in a presentation but and also for manually looking at it But if you want to formally do it, you you will need to use the JSON files and I'll come back to that and so Went over this then the exclusions I didn't mention exclusions yet. So one column is also the exclusions which have been applied so the The box in orange is the list of CVEs Which apply to that version in which have not been excluded the exclusions They are added for various reasons the most important one is because build route carries a patch that fixes that CVE But it can also be to due to other things like the issue doesn't exist in build routes because for instance like the Open SSL random number generator issue that was in Debian That's an open SSL issue, but it only applies to Debian. So then we make an exclusion for that one or the vulnerable code is never even built in build routes for instance in Some packages I can't think of anyone immediately, but have several packages which are in a single Which are in under a single product So in the in the CVE database, but build route only uses a part of it And so then only that's well anything which which applies to the rest is not included And now it's nice to have this list of CVEs, but it has a bunch of limitations To when you want to use it in practice and all these limitations actually Come from the fact that this was created more for the maintainers to see okay We have to update build routes in and apply Patches to fix the CVEs not so much for vulnerability tracking by a user So what are the limitations? One limitation is that you have to explicitly do make package stats, but okay, that's not much of a problem So it's not always there. You have to explicitly ask for it We can't really do it always because downloading the database takes quite some time. So Also the information reported does not contain the CVEs analysis the severity analysis now My opinion that analysis is usually geared more towards Like cloud applications and for your typical embedded application or when you use build routes It's not so relevant. You have to anyway reevaluate for yourself what what the severity is More importantly, you need the full build route source to generate the vulnerability list. So you can't once you've made a build The the build artifacts does not contain the information that you need to regenerate the CV list And you will need to regenerate the CV list after the product has been released. So that means if you afterwards want to To to check are there any new CVEs that have have been created which apply to my products Then you will need the full build route source again You don't need to do a full build you just need all the sources That includes the Package definition custom package definitions But actually only if you want to check CVs for door for those if you don't want that then you can just leave those out But well, it's not not a big deal to include them as well There's no separation of built-only packages Which means if you use for instance open as a cell during the build to sign the kernel then This is also the vulnerabilities to open as a cell are also included Even if you don't have open as a cell in your target device So it's looks at first sight. Yeah, but then maybe we should leave out open as a cell on the other hand It's very all possible that the vulnerability still applies to the signing of the kernel. So it's Maybe not so bad. That's that's even those builds only packages are included But then if you want to maintain that CVE list later You will do an analysis of each CVE and in many cases you will determine, okay, this CVE actually does not apply to my products for instance There is a vulnerability in some encryption algorithm that is not used in my products or there's a vulnerability that Only applies in a certain configuration and in my configuration is configured differently. So it doesn't apply So it's not possible to To add these exclusions later without modifying the source But for a release you don't want to modify your tagged release after the fact. So that's a bit tricky case also, there's no way to Or it's difficult to record configuration specific exclusions But so it's still possible to use it in practice and how do you do that? you Situps your CI to not only do builds but also generates the vulnerability info and then you just ignore that vulnerability info At the time that you're preparing a release Then you're going to evaluate the vulnerabilities and if necessary patch them but For many cases you will just documents. Okay, this has been evaluated as not applicable or it has been available evaluated as low severity So only if there is really something that needs to be patched you do the patching and then after the release you regularly regenerate the vulnerability info on the On the release source code. So you tag your release and then you make sure that there's a weekly for instance CI job that runs on that release stack to generate a vulnerability info and then you sometimes have to check Okay, what's what are the difference between the last time I checked are there any new CV is? And then update this manual documents manually update these documents to see what is Well to evaluate the new issues Now in Yachto in Yachto, it's actually quite similar So the technical aspects differ a little but it uses the same approach of querying the MVD database so You do the CV check by just adding inherit CV check to To your configuration file and then you typically also do include extra CV extra inclusions extra exclusions That's for this the CV exclusions that bilter it also has has in Yachto there most of them are put in a separate file And so what does you have to do then it's also done as a CV e database, but not a CPE database and then it's goes to it looks at the CV product defined in a recipe and Looks looks up everything in a CV database that matches it So it doesn't look at version at all. It only looks at the product and Then after it says collected all of this then it goes to each CV and checks the version With with greater than smaller than matching and if the version doesn't match then It says okay. It's already patched or if there is a patch filing Yachto itself, then it says it's already patched And you can also specify explicit exclusions like in that extra exclusions database and Then it's marked as ignored and so you have a the result is a per package system file with All the CV is that apply to the package at in any version and then for each CPE whether it's patched or ignored or vulnerable So you always need need post-processing to to use this there's also a text file, but it's useful because the chasing is easier to post-process So here is an example what it looks like don't worry too much about the details It's basically contains all the information that I mentioned. It also includes the Cv s CV vs forgot the abbreviation the The vulnerability score In some other metadata Yeah, that's about it. And so as I said the whole list so for Linux we have a thousand CVs I mentioned all of this The limitations are mostly the same as built roots, but just And for the same reason so it was created mostly for maintainer and List so for for a user that wants to track after release What is a bit worse is that vulnerabilities are generated as part of the build So you really have to do a full build to get the vulnerability list In you need all the layers again The exclusions are in the source code again The advanced I mean compared to built roots you do have the vulnerability score so that's an advantage of the Octo approach and yeah, also because the the version matching in Yocto is not done with the CPE database. It's actually a little simpler You need additional tooling, but you also In reality also need the additional tooling in built roots So I gave a lot of disadvantages So I wanted to see if there's something better in we have this software build materials So I checked if that was usable to track vulnerabilities it makes a lot more sense because You can generate the S-bomb at build time and then after the release you can just use that S-bomb to check if there are vulnerabilities And there's also a Google security blog that explains why this is a good idea And it refers to two tools that can be used to do this to from an SPDX S-bomb get a list of vulnerabilities Fortunately, yeah, maybe I should first mention what OSV is I'm running out of time. So this will be very quick It's basically an alternative to the CVE database that tries to solve the issues that the CVE and CPE databases have So it will accurately track the upstreams with actual links to the source or links to the release star ball It also has Version matching that works so you can you can use git commits to do version matching and it will look in the git history to see which whether a specific Git hash is vulnerable And then to make it easier to add vulnerabilities Bitless administration they define ecosystems and each ecosystem manages their own Vulnerability, so you have ecosystems like pi pi npm, etc You also have distro ecosystems alpine almanium exdebian and we have OSS first so completely different ecosystem that takes packages from everywhere Unfortunately each of these ecosystems have their own way of identifying packages. They're compatible with each other, but Yeah, not necessarily And then they have a lot of tooling to work with this Unfortunately the existing tooling doesn't work Why doesn't it work? Well for built-root built-root currently doesn't generate an spdx sbomb And then the one that the octode generates is not compatible with what the the two tools exist basically It's very simple you octode generate something with a cpe But the osv tools don't use the cpe identifier So that part is useless and then you only have the the name inversion info and the two existing tools don't look at those fields They look at different fields It's fixable it just doesn't work out of the box, so I ended up not using that so I can't say much more about this There are a number of other theoretical problems with osv So I mentioned already you have these different ecosystems Which are responsible for for Adding the vulnerabilities and the result is that you don't have all vulnerabilities in osv So in I did some numbers in Cve you have 31 1000 Cve's and in the debion ecosystem for instance you have only 9,000 OS fees of course Cve's are for everything including Sorry, you have 31,000 Cve's which are tracked on debion org Yeah, so these are things which apply to debion, but still only 9,000 of them are in the osv database I'm most likely that's because they're they don't put Cve's which don't apply to debion because it's a different version They will put them in security tracker Don't debion org but not in osv The second problem is with these ecosystems that the same problem is going to be registered in different ecosystems so so we have the The materialist Cve that I showed earlier has three different numbers in three different ecosystems on on osv And then each ecosystem has their own package identification scheme If you filter on package name it more or less matches But sometimes the package name can just be different like live curl in one ecosystem in curling another So it's not ideal It may be in practice is not so bad, but I have not really evaluated it so you need to do Cve tracking and Built roots and yokto have the tooling for that It's not perfect, but it works You do need to add some process on top of that to actually use it and This could become better if we could use osv But yeah, the tooling is not there yet, and it would require some effort to get that tooling to work And I have time for questions What's the state was kernel vulnerabilities? As far as I know the Linux kernel considers every bucket potential security issue But then Cve's are only filed by downstream Projects as far as I know so correct Anybody can Well, not anybody but a large number of people are able to File Cve's for the kernel and so you do have a lot of Cve's for the kernel But it doesn't mean that's something which is not mentioned in Cve Is not a vulnerability so yes, you have this thing that's maybe the Cve tracking of the kernel is not that useful Thank you for the talk and have you any Experiences or advices how often? The devices should be updated in the field specially in embedded fields. They are far outside in the Will dress used and then it's often hard to update in the field It's something that you need to evaluate. So if you If you look at the Cve's and you start evaluating them then actually In practice, it's rarely really applicable. I mean very very often what you have is that and Vulnerability could be used by an attacker but not on its own you also always need to stack several vulnerabilities to reach anywhere and so In in the sense that it could be used somehow it needs to be fixed eventually, but there's no It's not that urgent And also it depends on how Exposed your product is like an home gateway router is much more exposed than than a smart lamp Because the smart lamp is typically after behind nuts and stuff anyway So you really have to evaluate On the case-by-case basis Okay, thanks. Hi. Um, is there a way to match the osv database with the Cpe database or the Cve database so that you can kind of Cross-reference them and get to get to commit anyway for a seat for a Cve Sometimes so in the In this case This one yeah, you see already. It's the Cve number. So that's an easy match this one There is an aliases field in the database and it mentions the Cve number in the alias field So that's also easy. This one just mentions the Cve. Well, actually it's a it's an bunch of Cve's which are in a single security updates. And so The Debian only puts the security updates in there so there can be multiple vulnerabilities fixed in a single security updates and So that is not mentioned in aliases, but it's just mentioned in a text fix the Cve blah and blah and blah so that one is a bit more difficult to find but OSv has a really good search function. So if you search for that string, it will actually find it in that case Possibly there were others as well with the same Cve and I didn't find them because I just used that search function, but I'm not sure if how how useful it would be to do that because okay, you could cross-reference each Cve to the OSv database, but then if it's not in there You still don't know much. I mean could be something that is not applicable at all. So Maybe use a microphone After microphone here. I've also question. Okay in a talk earlier this week I've heard that the NIST server will be down by September. Is there a valuable solution for Yachto then? I Wasn't aware that the NIST server was going down Apparently they are changing the format. Right. So we are well aware and we already fixed that by defaulting to the new format So it will work as long as you have this Supported Yachto branch. It's new information for me. So I will use it in built routes. Okay If there aren't any other questions, thank you for the talk