 All right, so I guess we can get started. Welcome to the talk. It's going to be split in two parts. At first, we're going to talk about some general security metadata formats. And in the second part, Stanna will show some more practical uses of this particular data. So let's jump right in. When we talk about security scanning, we're ultimately trying to answer questions like, is my product vulnerable to this security issue? Is there a workaround or a mitigation for that security issue? Where can I find the fix? And if I applied, how do I know that it actually worked on the system that I'm patching? And what is the risk to me if I don't patch my system? So the building blocks of all these questions is security metadata. And when we talk about security metadata, we have to start with the CVID, which is at the center of most security metadata formats. A CVID is a globally unique identifier of a security vulnerability. Stands for common vulnerabilities and exposures. The project is operated by the MITRE Corporation, which is funded by the US government. It now consists of 153 organizations from 25 different countries who are participating as CV numbering authorities. Red Hat is actually one of those CV numbering authorities, which gives it the power to assign CVIDs to security vulnerabilities. And when we talk about CVIDs, that's just the actual ID part. But the CVEs themselves, of course, also represent the other security metadata around the vulnerability. Here we can see a short graph of how many CVIDs have been allocated since the beginning of the CVU project in 1999. And that number is growing nicely and it's only gonna go up, which doesn't necessarily mean that our software is getting more insecure. It's just that more efforts are being put into security research, security reviews and audits, which are obviously discovering vulnerabilities. So let's look at some examples of what CVEs look like at different vendors. So here we can see a CV record for the Palo Alto Networks, from the Palo Alto Networks vendor for their operating systems management interface. It shows some common metadata that is usually associated with CVEs, like the severity, the CVS score and how a particular vulnerability affects a vendor's products. Next, we have a CV record from Ubuntu. Again, it looks fairly similar. It has some textual metadata. It again shows the severity of the issue itself and how that vulnerability affects all of Ubuntu's Linux systems. And finally, we have a CV page that Red Hat publishes, which at the top of the page, we publish some textual metadata again that allows users to come up to speed on what their vulnerability is about. And then lower at the page, we have a table of how that vulnerability affects all of the products that Red Hat ships. So next up, we have the concept of security advisories, which is just a collection of software updates to security metadata and any relevant documentation. So we're going from a singular vulnerability identified by CVID to a collection of updates that actually resolve those CVEs. What you'd expect from a security advisory is information on which products and components are being updated, what CVEs are being addressed, how are those CVEs fixed, whether you need to take any manual steps or you just run VM update like you do on REL or any RPM-based distra. And what is the overall security impact of the update itself? Producing and consuming security advisories is fairly vendor-specific, so it helps when you can represent them in some machine-readable format because if you are consuming security advisories from multiple vendors, it helps that they all use a common language. And for this, we have a standard called CVRF, which stands for Common Vulnerability Reporting Framework. In its current version 1.2, it uses an XML schema. There is a new version coming out this year, 2.0. They use a JSON schema and they also rename the standard to make it clear that it applies to representing security advisory content in machine-readable format. The common use cases for this is specifically if you are a vendor who is interested in this metadata, it's a lot easier when you can consume it in a standard format across all of your technologies. So let's look at what this looks like in practice. Here we have an example of two security advisories. On the left, you can see Google publishes security advisories for Chrome as a blockposts that contain all the interesting security metadata that you might be interested in, like the fixed version, the number of security fixes, their impact, and their CV IDs, but it's a blockpost. So if you're interested in processing this information, you're left with scraping HTML and trying to parse it out. On the right, we have a hypothetical security advisory in the CSAP format, which uses the JSON schema for Django, which is obviously a lot nicer to process and read. So let's look at a real example of how Red Hat publishes this information. Here we see a security advisory for the pseudo package. This is the web view, which you can find on the customer portal. It lists all the necessary textual metadata. And so let's look at what this looks like in the CVRF format, which we'll break down by a section by section. The first part is some document metadata. This is data about the CVRF document itself. Here the most important bit is probably the ID and the revision history. The document type can also vary depending on what kind of document, what kind of security content this is. Some vendors publish informational security bulletins. Some vendors may publish notices of upcoming security advisories that may note mitigations only instead of noting the fixes. Yeah. The next section is document notes, which is used mainly for documentation. It's pretty much up to the vendor what kind of information they provide in this section. Another important bit here is the aggregate severity at the bottom, which lists the most impactful impact, the most critical impact of vulnerability noted in this advisory. So if there's 10 vulnerabilities fixed in this advisory and nine of them are rated as low impact and one of them is critical, then the overall aggregate severity of that advisory is still critical. Next, we have document references, which is fairly self-explanatory. Again, this is up to the vendor to decide what kind of references they want to provide in here. And the next piece is probably the most important one which is defining the products that this advisory actually fixes. This may look a little bit confusing, but not really, basically the standard defines products as a combination of branches, which are linked together with relationships. So here we have two branch definitions, the first one being the product family, which can be nested through different types and establishes the product as well eight. And then we have a branch for the pseudo package itself and then a relationship that links them together with a textual representation as the pseudo package is a component of well eight. This also gets assigned a unique ID, that's a combination of the two branches before, but the ID itself is actually arbitrary, it could literally be a random string. It's only used later in the next section when you link the ID to actual vulnerabilities, which we can see here. This is one vulnerability, of course, an advisory can have multiple of them. And the important bit here is the association of that product ID to the status of the vulnerability itself. So here we can see that the CV ID that identifies this particular vulnerability has been fixed in the product ID that we established earlier in the product definitions. The product statuses themselves can also note last known affected versions, versions that are not affected and a lot of different permutations of how a vulnerability may affect a particular system. Each vulnerability can also then specify a lot of supporting metadata that you can see lower in the screenshot. All right, so now that we have all this information around affected products, fixed components, list of fixed vulnerabilities and their impact, you could potentially use it to scan your system against this information. However, this format, the CVR format only gets you the information, but doesn't necessarily tell you how to apply it to actual scanning of your system. If you were to create some sort of a scanner based on CVR, if you'd run into potential problems in figuring out how to apply that data to, for example, the kernel package where a security advisory may note a fixed version of the kernel, but you would have to remember to check that that kernel is not only installed on your system, but is actually running to be able to tell whether your system is vulnerable. And that brings us to the second part of this talk where we're going to look at a purpose-built language that solves some of these aforementioned problems. So starting to take it away. So I hope everybody can hear me. I never muted my microphone. So what is Oval and why does it exist? As Martin noted, CVR FC service may be not the perfect solution if you wanna scan or verify what our fixes have been applied to your systems. However, Oval is a language that is designed specifically to tell you whether your fixes, advisors that you have applied or perhaps vulnerabilities that haven't been fixed in your products are or are not affecting your installations, your environments. Oval also covers a lot additional features that I'm not gonna cover, but the base specification, 40 Oval language is about 140 pages, so it's not exactly light. Plus vendors, extensions for various vendors, Linux, Unix, like OSes, Windows, even iOS and Android, which allow vendors to add custom checks that are relevant for their platforms or their products. We'll see why that's important a little bit later. So the old definitions then once you have them produced, once you have those documents can be read by various tools. We'll see in a little bit openscap, which is available freely in Ralph, Dora, et cetera. There is, for example, Clare, which is part of QAIO, which is a scanner behind a container health index that also uses Oval. And there's many other scanning vendors that actually can use Oval as well, whether container or hot. And why this is important is that vendors themselves produce these Oval documents. And then customers who might have mixed environments with multiple platforms, so they might have a mixture of Linux and Windows and maybe Android devices that they manage, they can then use Oval definitions from all of these vendors with a single scanning vendor, they can then scan their whole environment and maybe give them a better compliance overview across mixed, sorry, mixed environments. Now, one important thing to note here is the data itself for how to test whether a product is or isn't vulnerable, whether specific vulnerability applies, is up to the vendors themselves. So Red Hat ideally produces Oval data for itself or for its own product. Debian will produce its own Oval metadata. And this poses a question or a concern that is quite often repeated, which is that how do I know that my vendor is not faking the Oval data to make himself look better? And it's a good question and customers should hold their vendors accountable for providing correct and complete Oval metadata, but who better to know how to verify their products than people who actually developed them and shipped them? Excuse me. So to give you a better idea of how things look, I'm gonna do a very quick demo in a REL7 container that I've prepared in advance just to make this faster. So I have a very minimal REL7 container where I installed openscap and I've downloaded some Oval definitions for REL7, including unpatched CDs, which is gonna be important in a little bit once they show the report. So I'm gonna run this openscap report using these definitions that I pre-downloaded and I'm gonna generate an HTML report, which I'm gonna show in a little bit. I'm gonna run this. This is gonna take about a minute and a half. While we're waiting for this, I'm gonna show you a little bit where the Oval definitions, for example, can be downloaded, how they look. So Red Hat publishes our own Oval metadata in WW Red Hat security. I apologize, my voice is giving out data Oval v2. And this structure then starts with top level main product lines and then many, many additional files, one per product version, usually that provides information or Oval definitions for a given product. This is Debian's Oval repository definition, which again has one file per major Debian version. And this is SUSE's Oval repository. As you can see, the enterprise vendor's repositories are a little bit more complex. And the reason generally is that they have multiple support streams or supported versions that might be even combined or layered on top of each other. And so it becomes a little bit more complicated to provide accurate Oval data. Now, I'm not sure if the, so the report hasn't finished yet. It takes about a minute and a half, but I'm not gonna wait. I'm just gonna show you what I've already generated before with the same command. This is OpenSCAP specific report. Different tools would produce different reports, obviously, or different view of these results. Different tools, for example, satellite or other scanners that specialize in this might provide better overview over like large infrastructures, maybe generic reports across. But for what we wanna show, this should be enough. So I'm gonna jump just to the results. I'm gonna make this a little bit bigger. And so for each result, each line is basically a test that was run against the scanned environment. In these cases, we see that we have an ID which will need to be unique, which we'll see in a little bit. It's a unique ID within the Oval definition file. It's again, can be basically anything, but it needs to be unique within the file. We see the result of the check, whether it's true or false. True usually means that something's wrong, something failed. We have a class of the test that was run, at least for Red Hat metadata, that means we either have class patch or class vulnerability. Class patch is for checks, which are referencing advisories that have been shipped. So as Martin was showing earlier, this references the security advisories and CD metadata, the links that are shown here, which I'm not gonna show right now, go to vendor-specific pages, perhaps in our case. In this case, we know that the container images that I was showing and scanning contains two missing security advisories. So we could have applied them, but we don't have them applied right now. So we should probably run YAM update within the container or rebuild the container with the latest security fixes. But in addition, the definitions inside, I'm using the RAL7, including unpatched Red Hat definitions. They also contain vulnerability class definitions. These are definitions that Red Hat hasn't fixed yet for one reason or another. Usually it's either the vulnerability is very new and so there hasn't been time to really suffix, or vulnerability is low, medium, and the impact has been decided is not sufficient to actually backport a fix. But Red Hat still provides this information for customers if they wanna reevaluate, perhaps maybe they need to bring it up with support and say, okay, this CD is still present. It really is important for us. We really want you to fix it. So this is the report. Now, I'm gonna go and show a little bit what the document itself looks like, the oval definition, the oval building blocks. So it starts with, it's an XML file. So it starts with an XML definition, which has a class ID and a version. Class, I've already explained patch versus vulnerability, but there is multiple additional classes that I'm not gonna talk about. You can look them up if you'd like. There's the ID, again, the very unique identifier of the test itself within the oval document. And then the version, which is not as important, but it's versioning scheme for vendors to say if something changed in the document itself. The second building block is the oval metadata. This is relatively free form and it's not specified too much in the oval language, perhaps with the exception of the affected node and references. As I mentioned earlier, these references that you can see here, they're highlighted in red, that's what gets shown in the openscap report and that might be shown in various other integrations when you have oval scanners. So this is part of this pack. However, everything else that you see here is not part of this pack and it's in this case redhead extension, shall we say. And I'll highlight one particular part which is the affected CPU list. This is gonna be a little bit important later when we're trying to decide which oval files or which overall definitions our scanners should use. Okay, so the next and the most important part of the oval document or oval definition is the criteria. Criteria is a block of rules that can be logically combined and define an actual check, whether a scanning scan environment has a missing patch or is otherwise vulnerable. I will note here that though the comments here are human readable and nice, they are just comments and the oval scanners at least should not use them. Instead, they will follow these test ref IDs which again are unique within the document but otherwise have no specific meaning and are up to vendors to define. So let's follow one of these. For example, the cloud in it is earlier than and the test ref is this string here. So if you follow it in the oval document, we'll find that there is an RPM info test with the same ID as was referenced with the test ref. And within it, if we follow the breadcrumbs, there's two nodes, XML nodes, referencing an object and a state. And again, we have two references to some further objects within the oval document. So following the breadcrumbs, again, we find these two definitions, the RPM info object with the matching ID, which defines that the name of the RPM info object should be cloud in it. And this RPM info state, which defines that the EDR string, meaning the epoch version and release should be less than 0, 18, 5. And if this criterion is true, that means that the test or check has failed. So we can see here, this is how we can combine with the logical combinations. There's not just less than, but there's higher than. So you could have even, for example, version spans to check. Now, there are many more oval tests or oval types of checks in Linux especially. So there's the RPM info, which I've already mentioned. There's the RPM verified file, which can check state of the files on the file system versus database, the RPM database. It can check status of SELinux Boolean variables of selling context on specific paths. It can check system D unit properties, whether they're enable, disable, vote variables, what state they're in. There's uname, there's text file content, which can do regular expression matching in plaintext files. And many more checks. There are checks that are missing. And while I'm doing the next demo, I would like you to think what you see that you would say is missing. And if you can then maybe put it in a chat and we'll see. So this was the previous test that was done. I'm gonna run the second one. So the previous test I ran within the container itself that already had the OpenSCAP installed, which could also be a virtual machine or somewhere else. However, obviously that requires that in all of your environments, you have OpenSCAP scanner installed, which is perhaps not ideal. However, OpenSCAP provides a way simple wrapping script that can do offline scanning of container images, virtual machine images, and I believe some other content types. So in this case, I'm gonna do podman on share to allow podman to mount the volume that I'm gonna use. And I'm gonna run the OpenSCAP-podman command, which is included with the OpenSCAP package. And I'm gonna run it on the image that I prepared without the OpenSCAP installed in it. Otherwise it's a very similar image. And I'm gonna use different definitions. I'm gonna use row seven extras, including unpatched extras are additional plugins for row seven. And I'm gonna run it against this image. In this case, the check is very quick because the amount of definitions in extras is much lower. So let's look at the results. Let's see how they look. This is the extras report. And we see here that there is an additional vulnerability that didn't show up previously in the original report. And that's the Ellip SSH vulnerability that is not fixed. Now the question is, why didn't it show up before? And the reason for this is because we use the row seven definition but not the row seven, including extras. And this shows one of the problems with Oval and that is picking the correct Oval files, which right now, at least for Red Hat can be a little complicated. So let's see what you could do to pick the correct Oval file. If we're talking about container images, Red Hat built container images that are presented in the container catalog have within them a set of files within the root build info content manifest directory, which contain content sets. Content sets are basically a repository name, which can then be looked at a complimentary metadata called repository to CPE. If you look it up, you'll be able to find the matching CPE for a given repo. And if you remember, I was showing affected CPE list metadata piece within the Oval, those can be then matched to pick the right Oval definition to use or Oval definitions. Outside of containers, it's a little bit different because we don't have or we don't have an obvious place to put the content manifest, but you can use contents of the Red Hat repo or you can basically run which repos are enabled using YAM. And the repo names again are the same as the repository to CPE. There are some exceptions with regards to EUS, a long-term support stream, which gets complicated. But even with this, even assuming everything works, we can see that this is not really user-friendly, is it? There are other limitations and problems after I'm done and I'll look at the chat. I wonder if anyone figured it out before. All of the tests that I was showing or the test types kind of rely on fixed paths. So the types of tests are defined within the language or one of the extensions, but that means that custom installation paths are not easily supported. So if you have a zip or some other non-standard non-RPM way to install your product, writing an oval test for it might not be easy because you don't know where the customer will unpack the zip and you don't have a way to scan the scanned environment fully to like find out where it is. In a similar way, there is no support for language-specific package managers. So you cannot ask, okay, give me all the Python paths or is there any library with this name installed in the Python environment with a given version? Which is obviously a little unfortunate. And an additional problem obviously, as we've kind of seen in the previous slide, the complexity of Oval Files or their amount and the need to use perhaps multiple Oval Files to scan a single system makes it really tricky to consume and generate these Oval Files. This is especially true for vendors which have multiple products that reuse and remix each other and perhaps have same package names but different versions where they may be backboard different fixes or the same fix is gonna end up in different versions. And so what could ideally be a one Oval File for all of the Red Hat products? It cannot really be done that way because A, the Oval File would be humongous and writing those tests would be very error prone. It would be very easy to trigger the wrong check on the wrong product. So what's next? What can we do to improve the situation? I strongly believe that avoiding standards is not a solution. Maybe Oval is not gonna be the one and only and maybe we'll need to rethink. But for now, I do believe that maybe we can just extend the Oval Spec to provide better tests or better flexibility that scanners can then use to scan the environment. One of the missing tests that I kind of alluded to is that we can scan container content as I was showing but we cannot scan the container itself. So we cannot ask, let's say you're scanning a virtual machine, there is no Oval test that can say, is there a container image with this digest running? Or even better test would be, is there a container image from this repo and this version running or any previous version or any image that is built on one of those previous versions? So these tests we cannot currently write because the Oval language doesn't support it. And obviously we would need to do a similar version comparison as we do for RPM versions, something like it to be able to compare. We should also have a way to simplify finding correct Oval files and especially outside of containers and maybe find a way to support custom package managers, custom installation paths. That's it. We have some references in the back and I'm very much interested if there were questions or other of things. I'm gonna un-chair my screen. So we have a question in the chat. So I'm gonna read it out loud because the chat is recorded. So Stanislav, this is a really good feedback for the OpenSCAP team who are involved in the Oval standard. Have you had a chance to talk to them about these limitations? Not yet. We, I started working on Oval half a year ago, maybe actually less than half a year ago after one of our team members left and I picked up some things. And one of the things that has been on the back burner for us is providing better tests for containers. And right now there really isn't an obvious way how to solve this. There is a class of containers called container first if you're aware, which is basically containers that contain non-RPMs, right? They contain maybe Golang static binaries that even if we know they're vulnerable and we know that the container image was released in an RHSA in a security advisory, we have no real way to write that Oval test that will scan your environments and see if you have perhaps that image somewhere. However, for container images, there are specialized tools that do this scanning. So maybe the answer really is that we don't bother an Oval and maybe supply something specific for containers, some metadata that can be used for this. It is, for example, it is possible that the CVRF metadata for scanning containers like container first images might actually be enough. Like that information that we already produced might be used and maybe we don't try to extend the Oval language itself.