 Out of all things in the world, it is now my pleasure to introduce you from the MITRE Corporation Dan Adonofi, Dan Adonofi and Anthony Singleton to talk about CBEs, so Anthony, Dan take it away. Thank you very much. Hi folks. Welcome to DEFCON. Y'all having fun? Yeah. All right, now I'm gonna lay some facts on you here. Okay, this is the talk, the whole point of this talk is to give you the audience an understanding of A, what CVIDs are. Okay, so it's, there's a lot of misconceptions about what CVIDs are, how they're used, how they're assigned, why they're not assigned, things like that. So we're gonna talk a little bit about that. We're gonna talk about how you get these things. It's amazingly simple, but it's always, there's always a little catch, right? So we're gonna talk a little bit about what those catches are. Getting a CVID assigned is different from getting it published, so we're gonna talk a little bit about that, especially addressing why you might see a reference to a CVID in the public, but you're not seeing it in the CVID list and why that is. We're also gonna just mention how you can update and work with CBE to improve the content that we have and prove the data that we have so that the whole community benefits. And then hopefully we'll have time at the end for some questions. Alright, so first off, anyone in here who doesn't know what a CV, has never heard of, CBE? Okay, good. Common vulnerabilities and exposures. Okay, so CBE, it is a dictionary of publicly known cybersecurity vulnerabilities. The whole point of CVID is to uniquely identify vulnerabilities so that you know when you're talking to a scanning vendor, some other security entity who's doing a vulnerability analysis on you, your own internal teams, when you talk about a vulnerability, you're all talking about the same thing. It's an apples to apples conversation. And you can use the CVID to pivot around all the different tools and conversations that might be out there. If you look at the antivirus world, antivirus software vendors name things each differently. You know, if you look at what an antivirus is, they have like 10, 20 names depending on whose name them and when they name them and who they think did it and all that. It's very confusing. So CBE was created to address that kind of problem. And so, again, when you're talking about a vulnerability, you're using a name that everyone understands from tool to tool and from conversation to conversation. Yeah, no. I'll talk a little bit more about what data we do have and what we can do in a bit. So, again, there's a lot of misconceptions about CV, a lot of misconceptions about what people expect of us. We are, first of all, a de facto standard. We're not a standard. ITF never said let there be CBE. It was something that MITRE and the US government put together to initially help MITRE, actually. It was an internal project that the government said, hey, this is a great idea. We should do something with it and it grew from there. It is, again, a dictionary and it acts as that pivot point. Now, what it is not, CBE is not a vulnerability mitigation itself. We sometimes get email from people saying, how can I use this CBE to fix my computer? That's not quite the right question, right? It's the question should be, okay, I've identified a CBE. Where do I get more information about it? We're not a vulnerability database. To us, a vulnerability database is something that takes a CV ID, a uniquely named object, and then enriches the information that they have about it. So, for example, the National Vulnerability Database, NVD, they are a consumer of our information. They take that information in and then they add to it. They add CVSS scores. They add other references. They add things like risk assessment or attack vectors, stuff that we don't necessarily publish in ours. And that's them purposely. We're not out there to compete with vulnerability vendors. We're just there to name the vulnerabilities. So, again, we don't include risk assessment. We don't talk about impact. We don't talk about the fixes. We don't care if there's a fix or not. We don't really dive too far into the technical information. That's not what our service is about. Now, that information is useful when determining whether or not something is a vulnerability. We'll talk about that in a minute. But it is not required for a CV ID to be assigned and published. We also, by the way, we're not there to announce vulnerabilities. CV IDs are only published in our list when they have become public already. There has to be a public reference to the vulnerability somewhere out there on the Internet that we can point to. Because we don't want to be the first point of publishing for anything. We don't want to be there announcing a zero day. That's not what we are there for. There's a few reasons for that. Mainly, we just don't want to become... There's a lot of legal stuff. Let's just put it that way. Our lawyers looked at it. Our management looked at it. And they just decided, that's not what we want to do. But this drives a lot of why you might have problems seeing CV IDs published when they're assigned. Again, we'll talk about that in a minute. Who can assign a CV ID? The primary assigner of CV IDs is still the MITRE Corporation. You understand the MITRE Corporation is a not-for-profit organization. We run seven federally funded research and development centers for the U.S. government. CV work is under one of those. MITRE does a whole bunch of things. CV being one of them. We operate all the guts behind the program. We administer both the technology, the operational side, and the administrative side. We work with our sponsors in the Department of Homeland Security and we work with stakeholders in the community to provide this service. In addition to MITRE, we have what we call the CNAs, the CVE numbering authorities. These are organizations that we've delegated authority to to assign CV IDs based on a common set of rules that we call the CNA rules. We're very good at naming things. So these CNA rules define things like who is responsible for what, who should be assigning what, what information they need to share, what information they need to make public. Any CNA must have a disclosure policy for example. CNAs can be someone like Microsoft, Apple, Cisco. It can also be research organizations like Cisco Talos or Rapid Seven. Rapid Seven is actually a hybrid because they have their products and they do their research. It can also be individual researchers, people who have come to us and are working with us a little more closely and are producing enough CVEs that it makes sense for them to get a stack of CVEs at the beginning of each year and then just assign those as they need them. That's the real, the main benefit of being a CNA is that you get a block of CV IDs ahead of time and you have more control when you're a product vendor, for example, over the disclosure process. Right now, if a company isn't a CNA and one of their products has a vulnerability in it, it's discovered, the researcher comes to MITRE and says I found a vulnerability in product X. If they're not a CNA, we'll say yep, looks like a vulnerability, we'll go through the assignment process. If they are a CNA, if the company X for product X is a CNA, we would pass it to them and they would have the first crack at understanding the vulnerability, setting the timeline, as long as they're following their published disclosure process, they have that control. So if you work at a company that has a P cert function, we'd love to talk to you. If you're a researcher who's doing a lot of vulnerability research, we'd like to talk to you. If you're a nation state who has a cert, we'd like to talk to you. We work with JP cert CC, Korea cert CC, we're talking to Europe, we're trying to get these out there to different entities that are involved in vulnerability disclosure to help with that whole process. That's a whole other talk. But I just wanted to get that plug in there. One of my hats is the CNA coordinator, so I have to do that. The CNA ultimately would decide for their own product. Oh, sure. Yes, the question was if we're referring to a CNA, who decides if it's a vulnerability or not? The CNA or MITRE? So it depends. If we hand it to the CNA and they look at it and they say and they agree with the researcher, yes, this is a vulnerability, then it follows the process through. They look at it and say, no, this isn't a vulnerability. We have a process. We have an escalation process where the researcher could come to MITRE and say that CNA over there said that what I found wasn't a vulnerability. Help me. And what MITRE will do with that case is pull the CNA and say what's your side of the story? Understand the researcher side of the story and then make a determination based on the information that is available from the researcher and from the vendor. So it's a we have the final say as the maintainer of the program. But we want everyone in the room working to understand if something should be assigned or not when there is a dispute. Any other questions before I move on? So there's a value to having a CV ID. That value is you've named something. And once you've named it, you can coordinate with other people, talk with other people about it and be clear about what you're talking about. But when do you, when is the right time to actually go get your CV ID? So first thing, you have to identify a new or previously unassigned vulnerability in any product. Now, we have traditionally only been assigning in a very small scope up until the last year. And some of you might have hit that wall where you're like, I found this cool vulnerability in a skateboard. And we said, well, we're not covering that. I'm not going to go into why that was. We've changed. Anything can get a CV ID now. If you're in the IOT room and you find something in there, you can probably get a CV ID for it. If you're in the automotive hacking area, you can probably get a CV ID for it. We're working with the medical community. We're working with, again, our international partners. We don't care. Why do we think that CNA should rule the world? No, no, of course not. But what we do think is that we have this resource and that it's silly that it shouldn't be utilized by anyone else who needs to use it. CV is a lot like an ISPN number in a book. Each book has a unique ISPN number. It's how you identify the books. And they're used around the world. People could come up with different ways of naming things in different countries, but why? If this system works or it's a system that exists that we can improve on, you don't need to recreate everything. So it doesn't matter the product. Talk to us. There are some exceptions. We don't cover software as a service as of yet. So for example, if it's a bug in AWS, it may not get a vulnerability, it may not get a CV assigned to it. But if it's a software that is controlled, that is something that the end user can control, that the consumer can control, or hardware that the consumer can control, it's probably qualifies to get a CV ID. So let's say you've discovered something. The first step in a responsible or coordinated or whatever term you want to use disclosure process is reaching out to the maintainer of that software. So if it's a vendor, it would be a vendor. If it's a project, an open source project, it's reaching out to the people who run the project and get them in the loop. They should be aware of it. If they are a CNA, they have a process to get the CV ID assigned for it. And they will have their own policies about what information they need from you, what information they're going to give back to you, all that. And that should all be documented. And if you run into problems with that again, we can help, we can intercede with that. If they're not a CNA though, you have to make sure that it hasn't already been assigned. So do a little research, do a little checking on the website, see if there's anyone else who's found this vulnerability and has assigned a CV ID to it. If not, then you're good. Now some other cases could come up as well. If you're working with a coordination center, if it's a big deal, big product, and you're working with cert cc, JP cert cc, ICS cert, you want to make sure you're coordinating with them. It really doesn't matter who assigns it per se, but you want to make sure that people aren't assigning two CV IDs to the same thing, right? The whole point is that each CV ID is unique. So if you're working with a coordination center, coordinate with them. It helps make sure that there's less confusion. Note, and as I said before, the vulnerability does not have to be public before you request it. It won't be published in our list until it's public, but if you found a vulnerability and you're going through an embargo process, having that CV ID early means that everyone who's involved in the embargo are going to use that when it goes public and then can use it when it's being dealt with under the embargo. So you can ask for it as soon as you realize it'll need one, but then it won't get published publicly until the information's been publicly given out as part of the disclosure process. So you don't have to wait, but until it's public, we don't put it in our list. All right. So before I move on, any questions about any of that? We have about 65 right now. They range again from vendors. We have a few bug bounty programs. We have some national certs, and we're always looking to get more. Ideally, every vendor is a CNA, every P cert, every vendor that has a mature vulnerability management process can be a CNA, and they should be for their own product. We have one CNA that maybe issues a CV ID a year, and then we have Adobe. And with Flash going away, I think we'll go out of business. It's horrible. So I love Adobe, please stand. So it doesn't matter the size, doesn't matter the vertical, it can be anybody. Right now at 65. In the last year, we went from 25 to 65, so we're really trying to ramp it up. And it is a community so that if you're working in a P cert, say, you get to talk to other P certs, and you get to sort of get around the politics that might be there. It's like, why are you talking to our competitor? You get that sometimes. We want to create a trust community within the vulnerability management crowd so that y'all can talk to each other, and coordinate, and learn from each other. Because I think last year at BlackHat, right, security, we don't compete on security, right? We believe in that. So, yes. Right, so how do you weed out generic, how do you weed out duplicates when what you find maybe generically described already in another CVID, or that might be something completely different. The more information the better that you can provide us, the more information you provide us the better. Okay, because we do have, you know, we can look at what is public, we can do some of that analysis. We hope that you will do that analysis as well. This is a scaling problem. If you all in the room came to me today with one of these, like, I'd have to get back to you, right? But if you do some of the due diligence yourself and come to us, we'll be able to work much more quickly. This comes up with things like fuzzing. We can sometimes get requests for dozens of CVIDs from somebody who's doing fuzzing. It's unclear based on the information that we've been given whether or not it's actually just one vulnerability that's been kicked off a bunch of times or if it's actually a dozen different vulnerabilities. Because the information, they haven't actually provided the information to make that determination or they haven't done that due diligence. We have a set of rules that we follow and sometimes we have to make judgment calls, but that's built into our process. We try to do it consistently. But the more information that you have and you can give to us, and you can say, I don't know if this is a duplicate of this other one. It looks kind of similar. That is information that if it's us, we can look at and analyze if it's a CNA. They'll know certainly because hopefully they know their product, hopefully. So yeah, it's about, it's really that due diligence is important. We don't have access to magic source code that you all don't. And even open source software, I mean there's a lot of code out there and we're a very small team. So the more you can give to us that you've sort of generated through, the faster and easier it is for us to work through it and to get back to you. So I mentioned these rules. We call this counting. Okay. Counting asks two questions. Is it a vulnerability per CVE? Is it a vulnerability? And if it is a vulnerability, how many vulnerabilities are there? This is exactly what we were talking about here. And I'm going to pass it over to Anthony to talk a little bit more about counting. Thank you, Dan. Thank you, Dan. So the first, the run off that, so we have this process in place, this method in place because, can you hear me now? All right, cool. So we have this process in place because in the industry, there's so many different definitions for vulnerabilities and again, even more definitions for bugs. So instead of trying to come up with our own definition of the two, we set a method called counting in place to kind of help you figure out, as Dan talked about seeing if the bug is a vulnerability and then seeing how many there actually are. And so those are the two phases that is actually broken into. So the first phase is determining the number of vulnerabilities. So we do this is going to be how we do it at internal CVE. We start by breaking the report into the individual bugs. And then from there, we determine if those bugs themselves are vulnerabilities. For those that do, we then go to the next step. But for the ones that don't, we then put them together and see that if there's a combination of bugs that will give you a vulnerability. And then we move on. Once we have that listed or figured out, you then determine whether the vulnerability occurs due to shared code, protocol or standard. And if it does, we then merge those vulnerabilities into one because the issue is now in that either the shared library protocol or standard. After you do that, now you know how many IDs, how many vulnerabilities there are and which then leads you into your second phase of figuring out does these vulnerabilities warrant a CVE ID. Well, you determine that because you have to understand that there's a certain value as we talked about earlier in the slide deck, what the purpose and value is of CVE. So for that, we have to start off by saying or thinking or determining whether the information for that vulnerability will be released. So it will be public information for people to have. So it doesn't make sense to have an ID if you're going to restrict access to the information. So if no one can know about the vulnerability, why have an ID? So that's the first thing you have to know is that we're going to release that information publicly. We give an ID to it so now everyone that also talks about that knows that we're actually talking about this one CVE ID. The next thing for that is you have to understand the community must be able to do something to mitigate the vulnerability. So a lot of this comes back to being something that's considered like customer controlled, meaning that, you know, you release a report or an advisory on this vulnerability and now the community can say, well, okay, I know if I apply this patch, I can then remove this vulnerability. So that's another benefit of having it. And then the last one is kind of subjective, but the community has to be concerned about the security of the product. Again, it's subjective. So when you make the request, you would tell us what the impact of that vulnerability is, and most likely we'll agree with you based on your research. And then again, like we talked about a little bit earlier, you want to make sure we're not duplicating a quest. So if you check and make sure there's no, if it hasn't been assigned, we'll sign it. We'll also work with you if there is a duplicate. So the community must be concerned about the security of the product. So yes, it's objective, but it can also be very easy. For example, this bug allows you to turn the color of the page to blue. Okay. Is that a security vulnerability? It depends. If you're working in a power plant and you need to know that if something turns red, it's bad. And if it's blue, it's good. Well, if you can arbitrarily change the color, that's a security problem, right? So there's like this policy that you have to take into consideration. Whereas if it's like a web browser, it turns it blue. That's really more of a bug than a vulnerability. The community must be able to do something to mitigate the vulnerability. That doesn't mean that there is a bug, sorry, that doesn't mean that there is a patch already. It means that it has to be something under their control. So again, back to the AWS example. If there's a bug in AWS's stack that is above user land, above where a user can actually affect something in it, you know, that's not something that we'll probably assign a CVID for because Amazon certainly needs to handle that. They need to market it. They need to spin it. Whatever they need to do, they need to fix it, right? But they're the only ones who can do anything about it. Their users might need to be aware of it. But giving it a CVID, that doesn't have value. Giving it some other way of labeling it that may have value. This is a discussion that we're having with the community in CVE. And if you have thoughts on that, we'd love to hear them. And again, the vulnerability must be public. One example where this is not the case is we don't want CVs to be used for internal bug tracking, where, you know, there's like a nightly code commit. And so you find a bug. You don't want to issue a CVID if like you're going to fix it that night and it'll be gone. And it was never seen by the general public. Again, there's no value there. It's only valuable if it's out in the world and people need to talk about it and know about it. Questions if it's a, let's say it's an old proprietary bit of code that the community can't affect the code, right? They can't patch it. The company's out of business whatever. They can mitigate the risk though. Okay. Either turn it off or put a firewall in front of it or whatever they have to do to mitigate that risk, right? So in that case, yes, we would have probably assigned to it even though, even if the company doesn't exist anymore, there is value in knowing that vulnerability exists because the consumer can do something about it. Not necessarily as much as they'd like, but there's something that can be done. Okay. So all of that discussion boils down to this one bullet. How do you get a CVID? You go to that website. Thank you. Good night. It is just that easy. Well, kind of. All right. So it's mostly that easy. There's a form there and an injection strip. No, no. There's a form there and the form asks for specific bits of information. There is a minimum set of information we need to start the process. We need to have a general idea of what the vulnerability type is. Now, there isn't a list of like canonical vulnerability types. Like the definition for vulnerability, there's tons of different ways to cut vulnerability types. We give some suggestions, but you just need to describe it. Okay. Buffer overflow, cross-site scripting, things like that. Right. Common terms. That's good enough. If you want to get into more depth, you can. We need to know what the product and vendor is or the project that's affected. You need to tell us what you found the bug in. You know. And also the version information. And it's, you need to be clear on when you talk about version, what you're talking about. You can't just say all versions. Because what if a new version came out tomorrow? Would that still be true? So you want to be very explicit if it's all versions prior to 1.1.1 or only version 2.5. If you've only seen the vulnerability in one particular version and you don't know about the rest, just tell us about that one particular version. You could always update it later. Okay. So that's what we require to start the process. You have other information, like we were talking earlier. Please give it to us. We'll take it. It, really anything that helps describe the vulnerability. We don't care as far as whether it's a vulnerability or not about things like impact or the attack vectors. NVD does another vulnerability databases do because again, that's enriched information about the vulnerabilities. But that helps us understand that this is a unique vulnerability because we can compare it with other things that might be out there. And also it helps with the counting question. And public references are always great. Again, in order for this to be public on our website, you have to have public references. Now, if you don't have them yet, that's okay. You can still submit. Again, the minimum is up here. But if you do already have something public or you've found somebody's research and hey, that should get a CV ID. That's a public reference that works too. Okay. Now, any questions about that? Yeah. Okay. So like I said before, we traffic only in public information when possible. So we do have an embargo policy on our website. The embargo policy basically says if you give us information, we're not going to share it until the information is made public. So we actually do sometimes receive information that isn't in the references. And we don't publish that information. But if someone gives us something and says that it's public or will be public, then we'll publish it along with everything else. We try to stay out of the embargo world though. If you're giving us something that's under embargo that's not public yet, we'll take very good care of it. I swear. We'll treat it like our own baby. But again, we don't want to be like having tons and tons of zero days just sitting there waiting for people to get to them. We encourage them to get public as fast as possible. But we don't, we have, there's a lot of sausage making in the background. The stuff that comes in, it gets stored off the internet. Quotes. The stuff that's on our public website, there's nothing there that isn't public. Does that make sense? And also by the way, you can always PGP encrypt it. If you don't trust how we store it, that helps a little bit, not a lot, but you can do that as an extra bonus if you want to. Yeah. They suck. Right. So again, we have an escalation process for that. So you go to the vendor, CNA or not, and you say, hey, I found this bug. Right. Well, if it's, the escalation process only matters if they're a CNA. If they, if you go to the vendor and the vendor says that ain't nothing, you come to us and we give you a CV ID if it follows the counting rules and works. Right. It's a CNA. There's a disagreement. If they sit on it for too long, if they break their embargo policy, right. Because some of the vendors say it may take up to nine months before you hear anything from us because we have to analyze this. You got to respect that. Like our release, we have to respect that. We can't stop you. But we will listen to what our vendors say, but the minute they break their promise, you come to us and we say, okay, what's the deal? And we will push it to some kind of resolution. We're not, we're going to act as a third party. Right. We don't, we want the information to be out there if it's supposed to be out there. Right. So we'll, we'll do what we can to understand the situation and satisfy hopefully the best, what's best for everybody. Yeah. Don't let your vulnerability request fall into a whole or your vulnerability disclosure has fallen to a whole. We, we're not a coordination center. We have friends who are, if it's a really big deal, we can go talk to cert cc and say, Hey guys, are you aware of this thing? But we can also pressure our cna's because we do have expectations, community expectations. And if they don't follow those community expectations, we'll kick them out. So no matter the size, we've had some very hard discussions with some of the bigger vendors about this. So. Okay. So what happens after that sausage? Okay. There's a lot of sausage may being made in the background. But what you see is that you'll get when you email, if you, if you use the web form, you'll get an email response with a ticket number. This should seem like nothing unusual if you've ever used any kind of help desk system. Right. We take that information, we review it. If there's enough information what you've given us to assign it, the content team will say, here's the CV ID, you're going to use. If it's a CNA covered product, it goes to the CNA or we, we help you redirect them. We don't actually take, so back to what you were saying, we don't take that information and forward it. Okay. We'll get back to you and say, you need to talk to the CNA. So that way weren't you, if you disclose something that's super sensitive to us, right, but you are reluctant to talk to the vendor, we'll encourage you to reach out to the vendor and coordinate with them. We're not going to pass your, your, the information that you've passed us to them. Okay. So we want to keep that. If there's a concern about sensitivity, we, we try to keep that control in your hands. Okay. But if you give us enough information that not only is it a valid CV ID request or a valid vulnerability and we have all the information we need, we'll publish it. We'll just send it up in the database until it is published in our database. It is marked as reserved. You've seen this, I'm sure. You've looked at a CV ID in the CV list and it just says reserved. Why? It's reserved because we don't have enough information to publish it in our database. We either don't have public references or the information that we've been given by whoever requested it was not enough to say, yes, this is a unique vulnerability. That's why you see so many of these. A lot of times people come to us and they say I need a vulnerability for X, Y and Z and we get those first three things, right? We get these first three things but that's not enough. You need the references to publish. So they get this and they go off and then they post it on their blog somewhere but they never got back to us with more information. They never clarified and never gave us a reference. We can't publish until we have that. So if you see something and you know there's public information about it, you can tell us and we will immediately get it published if that's what we needed and we'll talk about that in a second. So at this point again, it's reserved unless it has enough information, it is going to stay reserved until it has been populated. So if you're disclosing something, you need to notify us when it's public. We won't know. We have some web scrapers out there. We're trying to find this information proactively. It's really hard. Stuff gets published all over the place and we don't know to necessarily look for it. We're trying to improve that but we need to rely on the community to tell us when the work they've done is ready for publication, ready to be published. Again, if you have an embargo, that's fine. When the embargo ends though, you gotta let us know. We have no idea whether it's under embargo because we're not part of the embargo. So you gotta tell us. Yeah, so again as I said once we have that public reference and we have the enough information to know it's a unique ID, it gets published. And congratulations, you can put it on your resume. Pretty straightforward. It actually isn't. There's a lot that goes on in the background, a lot of coordination, the disclosure processes will vary. If you are in this business, if you're doing this regularly, there are ways to shortcut this where you can send us information more effectively, more efficiently. We can talk about that if you're interested in that. But if you're just looking for one-offs here and there, you just have to remember that we need to know what you know. We need to know when you're publishing information. We're running a little low on time. Yeah, we're almost out. So for, we're running a little low on time. So for, when you come back to us to let us know to publish what you have, we kind of encourage you to supply us with like a description of your research because you guys know the source code and you know the research better than what we do. And so we kind of have some things that are necessary to be in it. So if you go to our public list and you'll see that in the description, it has to be a product. The version problem type is described in the description. And then we also have wanted to note that we only publish information that is already public in that public reference that you give us. And so any information, any additional information that you may have given us in the request phase will not be published when you request your publish. Request is to publish the ID. We only publish the information that is reflected in the reference. So that's another thing to take note of. And then we only want relevant information on it. So we don't want any profanity, no marketing in the descriptions. Just things that are relevant to the vulnerability is what we want to have published in the public list. No profanity in the description. And then lastly, it has to be in English. When you send it to the primary CNA, there are other organizations such as JP cert that are looking to be able to translate for you guys from whatever language they have to English when they send it to us. If you don't have a kind of style, this is what MITRE uses. These are two templates. We actually have a public presentation you guys can actually look at, which will give you more information on how we do things like how do we describe versioning, product types, vulnerability types. And so that's publicly available with examples if you want to look into providing this kind of style when you submit for a publication request. Just really quick. As I mentioned, you know, we need help with the updating of this information. If you see something that needs updating, even if it's not your vulnerability, not something that you've found, but you have information, you'd like to see it in there, you know, certainly, for example, the U.S. government, they look at CVE as a source of what their risk is. And so if they look at our list assessment, based on our information, but if they see that there is a public reference, they can let us know, we can publish it, and then not only do they benefit because it works into their own system, the whole community benefits. And it just helps the program be a little more robust. So you see something, you can update it. If you see something and it looks wrong, you can dispute it. We have a dispute process. CNA or not, you can dispute anything. It just shows up typically as a dispute in the list, but it helps enrich that information. And before we finish up, so we don't care about attribution. Like I just said, you can send us information whether or not you discovered a vulnerability. We don't include credit or attribution as part of a CVE entry. We're not in that business. You can send us a reference. And that reference can be a blog post that says I found this vulnerability. Great. We'll add it to the thing, but we're not going to put it in the entry. We're more interested in these vulnerabilities getting disclosed and the information getting out there than we are about padding people's resumes. That's not our focus. It's a nice side benefit, but it's not our focus. So you typically aren't going to see attribution in a CVE entry and that's why. For CNAs, we will be attributing which CNA might have submitted a CVE entry just so you know that this Apache vulnerability description was written by Apache as opposed to written by somebody else. But that's as far as we will go for attribution. We don't care. Sorry. That said, we understand that's a use case. We understand what the community gets out of CVE. If you value CVE, if you think it is useful, if you think it helps make your life as a vulnerability researcher, as a defender, better, we need your help. We're trying the best we can. Throughput is a big problem. We're doing much better. We're actually on track to have twice as many CVEs this year as we did last year. That's probably around four... What was it last year? I think almost 20,000. We're on track four. That's a lot. Still, not all the vulnerabilities that are out there and not all the vulnerabilities that are out there across all the different verticals. If you're really into automotive vulnerabilities, just send us what you got. We can start building this into the vulnerability management disclosure processes that are out there. Hopefully, this little part will help jumpstart the culture that needs to be out there of disclosing responsibly or coordinately or however you want to disclose so that everybody is in a better shape afterwards. We're going to take a picture, take a picture here. It's all the links that we've talked about. The CNA rules include the counting rules if you want to see what the gory details of counting look like. We're going to slide to grab. We're out of time. I'm going to Anthony and I will be out there as soon as we clean up our laptops and if you have any questions, please come out and talk to us. Otherwise, thank you very much for your attention. Enjoy your time at DefCon. We have time for questions. Feel free to come up to the mic and ask away. Come on. Use the mic. Is the DWF, are they okay? So quick history on the DWF. The DWF is the distributed weakness filing. It is also one letter off from CVE. That's all them. I had nothing to do with that. So anyway, what is the DWF? The DWF is a CNA that covers all open source software that is not already covered by somebody else. CNAs all have scope. They all have the thing that they cover. So the DWF is rolled and they are in the process of spinning up their operations. It takes a lot to run this thing even when you're relying on the open source community to help you. So when you have an open source vulnerability, you can go to the DWF. You can do a pull request. It is based out of GitHub. You can do a pull request. You can add your information in there. I think they're working on a lot of processes that I know some people have trouble getting CVIDs from them. You can contact us and we'll make sure you get a CVID. Again, all of open source is covered. The DWF hopefully once they're up to speed operationally, they'll be covering all of that and we can send people that way and they can get it through there. Until then and even after then, you could always come to MITRE one way to federate this out, to scale it out is to just get another entity that can focus on it and use a lot of different tools that we may not have access to. They're working on it. They're trying. Open source. Yeah, go ahead. Okay, so you can download the entire corpus. It's on our website. You can always do that. Josh Bressers, who works at Elastic, he recently joined playing with all sorts of data. He took the full data set and he's doing some really cool stuff with visualization, looking at statistics and things like that. If you look for him on Twitter, Josh Bressers, I can talk to me afterwards, I can hook you up. He actually just posted a few things about it recently and he'll be posting to his blog about it. He also runs the open source security podcast and I've heard that. But that data set is available. Speaking of that, it's really helpful to learn more about sort of what we're seeing as information is coming in. Again, nothing private, but just information, meta information based on the type of vulnerabilities that are coming in, how they're coming in, things like that. We're going to post that to the website and have that available in a more format, in a way that's useful, hopefully. Keep note, counting vulnerabilities is a black art. The number of CVEs that a product has says nothing about the security of that product. The number of CVEs that you have squashed says nothing about how good you are being a security person. So be very careful when you're counting vulnerabilities. We count one way, a different vulnerability database counts different. So you need to remember that when you're looking at numbers and statistics. You can only really compare us to ourselves. Can't compare us to other people. If you've ever heard any talks from Josh Corman with I am the Calvary, he has a really good visualization of this and he's talked a lot about it. He's been very critical of CVE in a good way. We like Josh because he's saying, look, this vulnerability management is a big deal. And CVE is trying real hard and they're not getting it and the rest of the community, they're not getting it. Hey Congress, let's do some better stuff, especially about medical device security. But there's a lot of good information there from the I am the Calvary folk. And again, Josh Brush is another good source. And again, we'll give you some more information too in the future. Any other questions? Dan and Anthony everyone.