 Welcome everybody to the conformance overview. My name is Chris Edwards. I'm 1 of the 2 leads of the conformance verification matrix subcommittee Robert Daniels is my co lead. He's the 1 running the slides. You can say hello, Robert make sure your comms working. Yeah, no, I was trying to find that so, um, let's go to the next slide. This meeting is, uh, is to give everybody an overview of the face conformance program and the related processes, basically, for people who want to use it. We've targeted this toward basically the software suppliers who want to know more about conformance and the people who want to be verification authorities. The meeting started out as a, so you want to be a meeting. And we had a lot of people coming in that were soft for suppliers, uh, asking questions, um, and we found that more often when we were in that meeting, we were talking to suppliers and having more of a formal informal discussion about conformance than we were actually addressing me how to be a VA. So that's the evolution of the meeting where we're at today. So, uh, we're basically going to do this as a lead discussion. We do have a slide deck that's got a lot of slides in it. We're going to go through a bunch of those. But we're encouraging participation and we might jump down to some slides, which we don't necessarily intend to cover unless people ask those questions. There are more slides in this deck than we could cover in, uh, in the period that we have, particularly because mostly what we want to do is answer the questions for the people who are on the call next slide. So, the presenters in this meeting, um, I'm going to be the one doing probably half talking Robert Daniels, uh, is, uh, as I said, my co chair, um, usually we have members of the operations, um, I see a few of them on the line. Um, there's, uh, members of the VA COP, uh, Stephen Price, who you heard earlier is, uh, probably going to be the most vocal of that group. We may have people from getting started and face outreach speak up on some topics. And then, of course, uh, we may have some suppliers who've completed conformance that may have things to say. So, if you're any one of those people and you're in this meeting, please, you know, if you have something to say, uh, raise your hand. Um, when we get to the, uh, the next slide, we've got, uh, go ahead, Robert, next slide. Sorry, um, so if you have a question, um, there might be times when I will ask, you know, and I'll get to this in just the next slide, you know, show of hands. So, if you know how to use the show of hand feature and WebEx, uh, kind of get used to it, um, we may be, um, asking a couple of questions just to get a kind of a poll. If we were doing it in front of a live audience, this would be a whole lot easier. And that's the way we prefer to do this. Um, but, uh, we're adjusting to our WebEx, virtual world. Um, if you have a question to ask, uh, use the chat feature in WebEx, um, somebody, probably Robert will read that question and, uh, anybody who's on the call who's wants to say something about it could raise their hand. And then we could go into that conversation. Um, there's going to be times when I will delay a conversation because we have slides that are going to come up about that. Uh, and then there's going to be other times when we'll just jump down to a slide deck area that will address that question. So, let's go to the next slide and we'll go through the quick, you know, how many people on this call are new to the consortium and we'll practice raising our hands. Is there nobody? We probably can't see them cause you're not the host. I can't see them. Oh, that's right. Yeah. Our, our, our, our of the results it's about 4 or 5 that are new to the tourism. Okay, and do we have anybody on the call that wants to be a VA? We have 1 that, 1 or 2 that may want to become a VA is not or they're still, hand is still raised from the last question. Okay. And if there's anything in particular you want out of this meeting, just post it in the chat, we don't necessarily need to go through it right now. But if you get something, some question popping up, or if you have some idea right now, go ahead and type that in. The information that we're gathering from the people who are attending the meeting, that's going to guide what parts of the topic we cover. We've got about the first 20 slides that we're going to go through anyway. That can be about a half hour to 45 minutes worth of stuff, depending on how many other conversations we get into. Frequently, we get a lot of conversations that will extend that to about an hour and a half when we start getting into some of the details on things. So, and then there's also the, do you want to be a VA piece, which since we have people who may be interested in that, we'll cover that at the end. Okay, so with that, so the face conformance program, the first piece conformance is to UOC is not systems. One of the goals of the face technical standard is for both portable reusable software components. So conformance to face is that making those portable reusable software components. We don't have conformance to a system that uses those components, just a system that is made of conformant components. Conformance is to the face technical standard. There have been several additions released so far. There's another slide we needed to be updated. 3.1 is the most recent standard that's out there. The conformance program allows for conformance to any valid published addition of the standard. So, yes, right now it's possible that somebody could come in and say, I want to prove I'm conformant to face technical standard addition 1 0. Oh, I think I missed one of my bullets in my earlier slides. This meeting is going to be about the state of things today. So, while there may be movement to try to. What's the word I'm looking for to deprecate the face 1 0 standard. We haven't done that yet. So I'm going to talk about what the current state is. We may get into some other side conversations where we want to talk about things that are going on currently. But we're going to focus what the current state of conformance is when we talk about things in this meeting. So. As I was saying conformance is to any any valid published addition and there is an approved correction process that provides a path to conformance for additions with known problems. So if we look at tech standard 1 0. Everybody who's been at the consortium for a while actually knows that you do conformance to 1 1 for 1 0 actually don't know that we have an approved correction out there for that because it's so old. But we do have approved corrections out for addition 2.1 0.1. If there's problems with you meeting that there's a long list of those. And there's also some approved corrections for 3 0. I don't think we have any for 3 1 yet because it's so new. But then I might be out of date on that. Let's go to the next slide. So you may have seen this slide in the business overview, but basically we look at conformance as being 4 steps. There's a preparation step that the supplier does basically to get ready for conformance. There's a verification step which the face verification authority proves or verifies that the software unit of conformance is meeting the technical standard. Then there is a conformance step or certification step. Sorry. And then the registration step generally speaking. The certification step is very basic. It's it's it doesn't require a lot of more information than you go to the certification authority. You get your certificate and the registration step is also a very simple go to the registry. And report your certificates there and update your information about the product that you've now listed in the registry. This meeting is generally going to be focusing on the preparation and verification steps. Let's go through a couple more slides and then we'll pause for questions. Conformance definitions a unit of conformance is basically the thing you're going to push through the conformance process. Generally that's a piece of software that fits within a segment, but it can also be a domain specific data model. So it doesn't actually have to be executable software face conformance is is meeting 100% of their requirements for a UOC. So a UOC cannot be conformant unless it meets all of the requirements placed upon. A unit of conformance that meets its boundaries according to the face technical standard. We talked about the face technical standard it's it's the document that has the technical requirements. The face registry is a public listing of face conformant products it is not a place to get face conformance products it's a place to find conformant products. The software verification package is the the stuff that the supplier will provide the verification authority to prove conformance and that verification package has limitations which we're going to discuss in a little bit. And then claiming face conformance. One of the primary notes we have here is that you cannot publicly say your face conformant unless you have meat unless you have actually completed conformance and have your product listed in the registry. That's one of the key points of the conformance program. And it's part of the the brand of the face consortium. So face conformance requires being listed in the registry or to tell people that you are face conformant in publicly. I think that's a good spot to stop and ask if there's any questions so far. Chris we have one question about CTS but I think we'll address that later. Okay, so let's go to the next slide is on the participants of the conformance process. And this is kind of terms the software suppliers the term we use for the person who's providing the UOC, even if it's a data model. The face verification authority is one of many entities that have been approved to verify things for conformance. The face verification authority performs a technical evaluation of the product. The face certification authority is the one and only entity approved by the consortium to provide certificates to conformant products. The face regulatory administrator maintains the face registry and the workflow tools, the PRCR process and various other things that you use to interface with the face ecosystem. The face consortium and its role in conformance is that it creates and updates the technical standard and the rules for face conformance. It also executes the PRCR process. The face steering committee approves VA's appoints the CA and LA. It'll handle audits if needed and the appeals when requested. So those are the roles and how they fit within the conformance process. So let's go into a little bit more detail on the face verification authority. The verification is handled through the VA. The VA is a technical expert on the technical standard and the verification process. They are approved by the consortium steering committee. They serve as the technical evaluator of conformance to the technical standard. The VA evaluates those software products through a test execution and then they do an inspection of artifacts for the requirements that are beyond what the CTS can test. And those VA personnel have to be experts on all additions of the technical standard in order to become approved. These verification authority, they're basically the last line of technical evaluation for conformant products. So they really represent that end of the face consortium. So there are multiple VA's approved by the steering committee right now. VA organizations do not have to be members of the face consortium at this time. They can be government. They've got both an army and a Navy VA right now currently approved. They can be internal organizations within a larger company, like a big company that has its own SQE department. You could get your SQE department approved as a VA. We don't have anybody who's done that yet. And then we have independent companies right now. We've got two sign embedded systems and science as independent VA's that suppliers can go through to get their software verified. The list, the current VA organizations is available on the face landing page. So I'm going to ask Robert, Steve, if I've missed anything so far. I don't think so. I don't think so either. Okay. Yeah, the only thing I, and I may have missed this is you were talking about when you can say your face conformance. And it's true what you said. That's when you can say it publicly, but you are allowed to privately say some things. And I don't know if we covered that. No, we didn't. And usually we get a question on that. So I know that if we were in a meeting room and nobody knew we were being recorded that we'd get a lot more feedback and people would raise their hands and say, you know, what's this mean? And that whole public announcement of conformance. That is something we usually get a question on. So, yeah, Steve, thanks for bringing it up. You can go through the conformance program, have your conformance certificate and not publicly listed. And if you're telling somebody in particular about your product and not posting it where everybody can see it, you can say you've got a conformance certificate. Okay, so on this slide. So when you're a supplier and you're trying to go through prepare your product for conformance. You want to establish your relationship with your verification authority fairly early. This is the same as a lot of other processes where you if you have somebody who's evaluating your product for quality, evaluating your product for air breathiness. The sooner you talk to somebody about what they're expecting from you and what your hiccups might be and trying to go through conformance. The sooner you talk to your VA, the more likely you're going to be successful and meet your schedule. You do want to prepare a preliminary software supplier statement of conformance. Go through that document, look at what's in it, understand what's there. And it'll help you guide where you're supposed to be in the technical standard and help you figure out exactly which requirements you're supposed to be meeting. Figure out which technical standard you're using. Find a shared data model you're going to be using for those data modeling parts, which you are most likely going to need to do. Filter down the CVM for that standard and understand which requirements you need to do and how you're going to approach proving that you met them. Then you're going to want to develop your data model and regularly run the conformance test suite while your software is being developed. Running the conformance test suite will help you find all of the little gotchas with compiling to the gold standard library and getting it to link correctly. Yes, this is wrong. Could I add a little there? Yes, you can. You mentioned the VA relationship and establishing that early. And that is probably the most critical thing that you can do to successfully get through conformance. And I wanted to just talk a little bit about that relationship. The VA is not like an IRS audit. We're there to help you achieve your goal of gaining certification. So we will work with you to help you do that. We'll provide mentoring. We'll do a lot of things that could help you along the way. We're not there just to check off check marks at the end and give you a grade. It is an actual cooperative process where we help you achieve your goals. Thanks, Ron. All right, let's go into the next slide. Real quick, we've got a couple questions. I think we get to address right now. So we have a question about the software supplier statement of conformance. And how is that made in conjunction with the VA and supplier? Okay, so the software supplier statement of conformance is a document. It's really a form that's published by the face consortium. That outlines what aspects of the technical standard apply to your software. Generally speaking, the software supplier should make an attempt at filling it out. And Robert, we could probably just drop down to those slides real quick and just come back to slide 13 later. So the software supplier will fill this out and then show it to the VA and the VA can help them through it. The first section is basically, you know, who are you as a supplier? What kind of a product are you putting out? But then it gets into... So basically section one is just who are you? You go to section two, which is on the next slide. You know, what's your unit of conformance? Do you have variants associated with it? Section three is a little bit more detail on how the technical standard applies to you. Which segment are you in? That kind of stuff. Section four, this gets into how you're building the conformance test environment. Which test suite version are you using? Which shared data model version? A couple of questions there. Section five is really a listing of all of the optional conditional requirements within the technical standard and how they apply to your unit of conformance. So to fill this out, the supplier should take their crack at it and then show it to the VA. And the VA should go and help them make sure that that's correct. That's something to do very early. Does that answer your question? Yes, thank you. Hi, I think it does. So while I go back to the previous slide, we have another question I think would be good to answer. And that is you'd mentioned a VA within an organization. And then so the question is, how is that VA accepted or approved? How could that happen? You know, what are the rules? And I think we could probably talk more generally about that now. And move specifically later for the VA's. Yeah, so in general, VA's are approved by organization, not individuals. So in order for a larger organization to have a sub organization within to be approved as a VA, you have to show independence in that sub organization. And then anybody who reports within that organization as a VA could function in that role. And yes, there's a lot more detail on that. Do you want to be a VA section, which will be addressing at the end of the meeting? Okay. One last question, since we're kind of on the VA and PREP, can we talk about how the VA's interact with the CTS or the expectations of the CTS between the supplier and the VA? Okay, so as we said, the CTS, even on this slide 12 here, regularly run the CTS, the supplier should download the CTS. The CTS has the gold standard libraries in it, which you'll have to compile your code against. So really, the supplier has to have the CTS and run the CTS. When the supplier hands over their verification package to the VA, part of that verification package is going to be the test suite configuration potentially, and any notes they've made about configuring the CTS, but the VA themselves is going to download the CTS. They're going to take the approved version of the CTS, they're going to configure it themselves, and then they're going to execute likely the same situation that the supplier did, just to provide an independent review and see what the results are. And of course, the VA is going to be more of an expert on the CTS, and we'll be able to look at those results in more of an expert view. Ideally, the test suite just says pass, and there's no problem. But there are a number of CRs that are against the CTS. So always go back and check the PRCR tool for approved corrections regarding to whichever version you're going against. Make sure you're using the latest version of the CTS. And of course, the VA is also going to know a lot about those PRs. Thanks, Chris. Now we can go. Okay. Yeah, for the process of being a VA, I just saw that come up. We're definitely addressing that later. Okay, so this is the OV-5B for the verification process. It's really simple. So the supplier downloads off of the repository, the face reference repository, the standard, all the tools, everything they need. They do that preparation when they're ready. They send the verification package over to the VA. The VA will perform an evaluation. They're going to check to see if everything's conformant. If it's not, they're going to pass it back to the supplier by noting deviations, telling the supplier what the issues are. The supplier will then update everything and resubmit. Now, generally speaking, we expect the supplier, as a VA organization, that the supplier is going to give preliminary versions that aren't going to pass the first couple of times, just so we can make sure the documents are in the right format and that sort of stuff. So we expect to go through this loop at least once prior to getting a pass. When you get a pass, then there will be a verification results package that's generated that includes a verification statement. So when that verification statement is passed to software supplier, the VA also stores everything in their VA retention repository. Now, one of the other key things on this diagram is that the supplier does not pass the VA the version of the shared data model they're using. It does not pass the, sorry, the supplier doesn't pass to the VA, the CTS. The VA always goes to the reference repository to download that stuff themselves. So let's talk about the verification of the package. In that verification package, the software verification package, that's going to include your statement of conformance, which we just talked about a little while ago. That's going to include your conformance verification matrix, which is of course the listing of all the requirements of the technical standard and how you, the supplier, are answering each of the requirements that are mentioned in that CVM based upon a filtering of what's in your conformance statement. The verification evidence is going to be a collection of requirements, designs, test results, anything that's needed for you to prove the inspection elements of the requirements that are fitting your UOC based upon what's in your statement of conformance. And then you're going to have UOC binaries for testing. And now these are compiled against the gold standard library, not executable binaries reflecting your source code. We'll talk about that in a little bit. And then your data model is going to be all of the communication that you pass across the transport interface has to be data modeled. So for P-TRIPLE-S, PCS, and transport UOCs, most transport UOCs, you'll have to provide a data model of the information that you're actually exchanging. And I'll go ahead and add that the case conformance policy also lists the verification agreement, which is an agreement between the software supplier and the VA. And this agreement is determined by the VA organization that's not standardized. So Chris, I don't know if there's anything else you want to say about that. No, I think that's good enough. Chris, add something here. Go ahead, Ron. As Chris mentioned earlier, if you're going to iterate through this and make multiple submissions and you're engaging early and often, the first iteration or the first software verification package that you may give to the VA doesn't necessarily have all of these elements. That first package may only have the statement of conformance and the conformance verification matrix, for example, because you're not far enough along in your development that you've generated all of the evidence yet. But we can still start working with you for those two things as you're continuing to go further and further into the maturity of the development of your product. So that's why it's important to go early and often and not wait until you get the entire package, but to submit smaller portions of it as we go along. And each time the package gets a little bigger and a little bigger and a little more complete until finally you're ready at the end. Yeah, thanks, Ron. Okay, so our next section is going to be talking a lot more about what is a UOC and what is it we're doing to test a UOC. If we go to the next slide. For the most part, face is interested in the face technical standard face conformance is interested in interfaces not performance. So conformance is a lightweight process to verify that you meet the technical standard requirements. We don't go and verify that you meet your requirements that you have for whatever function your UOC is supposed to be performing. It's not that when it comes to the face technical standard, we verify those requirements. Conformance is primarily involved with those API's which are between segments. So the transport API, the operating system API, the iOS API. Those are the things that we're interested in testing. And generally we do that with a link test. So we're going to take code that was compiled that doesn't have the functions that are the other side of the API. And the conformance test suite is going to test them by generating the other side and seeing if the link works. Now, when there's functional requirements in the technical standard, the software supplier is actually going to test those. And then you'll report in your conformance verification metrics. So basically where the documentation is on how you tested it and what those test results are. They may just, you may just answer some of those requirements with design documents depending on what the conformance verification matrix is calling for for each of those requirements. So the primary thing out of this is the conformance is not an indication of functional software. Conformance means that you've met the technical standard. So I liken it to kind of like a UL sticker saying, you know, I'm not going to guarantee that the lamp lights your room. Just that it's not likely to burn your house down. Additionally, the the conformance process doesn't impose a format or a process for how your software should be verified. So the conformance also doesn't impose that that whatever your organization's internal workings are for documentation can be submitted as evidence for some of the stuff. I don't know, Chris, if you want to expand on that any. Okay, go see. I'm just going to say, we're not, we are not expecting you to write special documentation for this process. What we really are looking for as VA is to have you point into the documentation, you're already going to be writing for these things and say, here's the evidence that I met that requirement. The same thing is true if you have to actually test something, you'll point to your test procedure and say, this is the test that we use the test list requirement. And here are the test results that say that we passed that test. You can have to do that anyway. So in the CVM, you are just referencing and say, hey, go look in this document of ours in this section. And you'll find your evidence. Yeah, and that all goes back to that first bullet there that it's meant to be a lightweight process. We're not trying to add burden unless it's it's practical for meeting the goals of face. So we don't need you to go and generate new documents that state the same thing you need to do to meet an airworthiness goal. All right, more on that in a couple of slides. Let's actually go to the next slide. So one of the other things we need to think about is that because our goal is portable reusable software conformance to face should mean that I don't have to reconform because I ported it. So we take our source code and we build it to a target and we build it to the initial target. We get all that working. We do our conformance and then somebody wants to reuse our software somewhere and we basically prove that face conformance works. You don't have to go and redo your conformance because you ported that software somewhere else. Now, there's a couple caveats on that. Your source code can't have changed. And that actually feeds into the next slide, which is talking about variants. So if I have my source code and in my source code, I've got the necessary compiler switches to compile to power PC or to Intel based architectures. If I've got in my source code, the ability to turn on and off a bunch of optional features that I want to sell as a product line. I can take that product line source code and I can say I've got target a, which is on a power PC and as my features a and e in it. I have target B, which is on an Intel processor and has features B and C. I could basically go through conformance and I could say, here's all of these variations I've got. Do my conformance program upon that initial block and then be conformant to all three of those targets that are on this slide. Now to do that, I need to properly set up my conformance statement saying here are the variants that I'm pushing through. And I need to come down to a subset of those variants that is reflective of everything that my my software could do. And if you look at the possibility of say, I've got 10 options. And I could turn each one of those options on and off as a Boolean that can be 100 different variants. But if I look at that and I say for all of those, if I turn all of them off, I'm executing the negative case. And if I turn all of them on, I'm executing the positive case. And maybe if I just run two variants through conformance, I can cover all 100. Now it's on the software supplier to come up with what that subset is and prove that that they are executing all of the source code or that they are linking all of the source code. With the binaries that they supply. But this allows us to think about units of conformance as these portable reusable components that we want to see in the face registry. Do we have any questions on that? See now I know everybody's just being quiet because they don't want to speak up in front of people. Let's go to the next slide. So this slide. Answers a lot of questions people have about why the conformance policy is the way it is. If we look at the at the various entities involved. We've got the supplier. We've got the verification authority. We've got the certification authority. And then we have the registry, which is really available to everybody. Right. Anybody can go and look at what's in the registry. The green lines on here represent things that the software supplier would normally produce in order to product to create a product. And the first one is basically the description of the product. So what is my sales information? What is it that I'm going to tell people my product is that information actually flows through all four of these columns. When it comes to my executable product. The thing that actually is this product the valuable part. The supplier keeps that the verification authority never actually gets the executable version of the product. And this is to protect that supplier IP. Right. There is no entity within the face consortium or the fake face ecosystem that gets access to all of the executables for every conformant product. And we can also see the source code once again that stays with the supplier. It does not have to be shared with anybody else. Now we have multiple verification authorities and there may be a situation where you want to hire a verification authority to do more than just verification. And you might give them the source code, but there is no requirement within the conformance program for that source code to go anywhere outside of the supplier's own servers. When it comes to your requirements, your designs, your test reports. When they're needed to prove conformance is verification evidence, those go to the verification authority and no further. Now there's some supplier IP in there face consortium allows you to pick which of the verification authorities you're going to use. So once again, there is not one entity that has all of the requirements designs, all of the IP associated with face conformance. And this is one of the reasons why a larger company may go and invest in getting their SQE department certified or approved as a face VA. Now we've got a list of things which you need to do that face is asking you to do, like we asked you to do a data model. That data model that you produce, it's the supplier produces it, they pass it to the verification authority, it goes no further. That statement of conformance that you write, that goes to the VA and then it also goes to the certification authority. So at this point, the certification authority is going to see your product description and how your product meets the face technical standard. The verification matrix that you fill out with all of your pointers into your verification evidence, that only goes to the VA. Same with the test suite configuration information. And when you build your binaries for testing, they are not executable, but you compile them against gold standard library, those only go to the VA and no further. Now, when you pass verification, the verification authority is going to create a statement of verification. That statement of verification will go back to the supplier and it will also be provided to the CA. So when the CA does their evaluation of your product, they're going to be looking at really those three things. What's your product? How does it meet face and has it passed the verification step? Now with this structure, like I said, the face conformance process is doing a lot to protect supplier IP. And it's why we're basically doing verification against source code, but we don't let anybody who's who's verifying it actually see your source code. We do a compilation to a gold standard library in order to judge conformance. I do I want to add one tiny thing to this. It's important to know that you mentioned, and this is true the statement of verification is given to the certification authority, but it is not done so until the supplier asks to have that happen. The VA does not automatically send that to the certification authority. Yeah, I'm not sure if that was covered in the business overview, but generally the steps through the process are all in control of the supplier. So when verification is passed, the verification authority will hand the verification, the statement of verification back to the supplier. So when the supplier contacts the VA to say, hey, a past verification, the certification authority can come to the VA and say, do you have a verification certificate or sorry, a statement of verification for this product? And the VA will either say, yes, here it is or no. Okay. I'm not seeing any questions. A lot of times we get asked how much is this going to cost and the cost of the verification. That's going to basically verification of the unit of conformance. That's going to depend upon how big the OC is and whatever your supplier to VA relationship is. The VA's are not necessarily controlled by the consortium. I mean, they're not controlled by this consortium. They're only approved by the consortium. We're not going to address cost of verification in this meeting. That's really a contractual obligation between two entities that are not controlled by the consortium. The cost to face certification is fixed by the certification authority. We should just put the number in here. We keep bringing it up every time. It's always the same. It's agnostic of the scope of the OC. $995. Ron, I think you cut off on the beginning of that. Oh, $995. It came out as $5. I'll take two. Okay. So for conformance, there's a bunch of documents you can go and look at beginning started guiding guide. Sorry, beginning started guide. You can find information for that on the face landing page. There's the face conformance policy. There's a conformance guide associated with the policy. There's a face conformance authorities plan that goes into what the requirements are for the VA, the CA and the LA. The technical standard, all additions, whichever addition you're going for. There's a rig which comes out for each major addition. There is, of course, the CVM. And for the CVM, there's a matrix users guide. Generally, most of your questions could be answered with those documents. Or you can come to this meeting and we can answer any additional questions you have. Of course, each VA, if you contact a VA, they'll be able to answer further questions as well. So the software supplier statement is a conformance. We talked about that one briefly. The conformance verification matrix. We've got slides on that for more detail if people want to see that. The statement of verification is the one that the VA will fill out when something passes verification. All three of those documents are produced by the consortium. The conformance test suite. The conformance test suite has a user's manual. Those are approved by the consortium. The FACE library tools includes a conformance workflow tool. That's at facesoftware. Somebody got the link for that? I'll find it and let you know. Yeah, I'm trying to remember if it's facesoftware.org. I think it is. And I know it's in one of these slides later on. So, and then the FACE PRCR process is at ticketing.facesoftware.org. And that's where you can find how you can enter new PRs and, of course, find the approved correction list. I would recommend people to check the approved correction list before you file new CRs, of course. And then let's see. The next slide here is on getting into the verification authority more. So do we have any questions overall on FACE conformance before we start getting into details on the verification authority? This would probably also be a good opportunity if anybody wants any greater detail on anything discussed so that we can proceed to those slides. So there's a question on FACE data models or FACE data models required for conformance. Yeah. So that once again, if you are developing something that uses the transport layer to communicate, which means something in the portable component segment or something in the platform specific segment, you will have to have data models for any communication that goes across the transport layer. That is a hard requirement of FACE. If you're producing a transport service. Okay, that's where I was going. Is the transport service? Yes. Yeah, if you're providing a transport service and you're providing the type specific interface, you will have to have a data model for the things that are on the type specific interface. There's a recent CR, a recent approved correction out there that allows you to provide a way of generating new types for your transport. But even then, you need to provide a model for the types that you proved your transport with. There are elements or sub-UOCs you could build within the transport service that don't actually provide the type specific interface and those you may not have to provide a model for. Okay, so let's talk about the FACE verification authority a little bit more. So the verification authority provides an evaluation of conformance to the standard as we talked about earlier. They can also provide guidance to the software supplier on conformance related topics. So the VA can help prepare the software verification package. It's not required that they help prepare it. And when it is prepared, it is the property of the software supplier. But the VA can help do that. The VA can also help complete any of the forms. They can help you fill out the segment of conformance. They can help you fill out the workflow tool. They can trace the requirements to the verification evidence. Now, I know that if you come to some of the VA's, they're going to charge a lot more if you ask them to do these services. But there's nothing that stops the VA from offering those services. It's not required that they do it, but they can do it as additional value. We can or the VA can also help you compile to the test suite. But they must be independent from writing your requirements and designing or developing your software. So this is a standard independence practice. As we said, the current contact information for finding a VA is on the landing page under the conformance topic. So if you want to look for a VA, that's where you go. So in order to promote a common approach to verification, all of the verification authorities that are approved are automatically members of the base verification authority community practice, the VA COP. Organizations wanting to be a VA and ready to form the task can apply. There's no limit on the number of VA's that can be approved. The RFP is at this link and we have a how to become a VA set of slides, which we'll be going through in a little bit. So this is going to start answering some of the VA questions from earlier. And yes, the slides will be posted later and Actually, I'm pretty sure we post these slides, right, Robert? I know they're on the CVF page on Play-Doh. And we are recording this meeting. Yeah, if you give me a second, I will leave this view up and go find the link to the slides and post it in the chat. Chris, we have a couple questions, or maybe it's just one question, one of those slides. So this one says, when the technical standards change, how will upgrades to the conformance, or how does it affect conformance? And will the VA perform evaluation against only the gaps? I guess this may be trying to hit on recertification. Right, so the primary thing here is that verification is to one addition of the standard. So if you were conformant to addition 2.0 of the technical standard and addition 2.1 came out, you don't have to do anything. You're conformance 2.0. If you want to be conformance 2.1, you have to go through and run conformance again for the new addition of the standard. If there are a number of approved corrections that went through the PRCR process, so if you were working on Phase 2.1 and there are PRs that came out that led to Phase 2.1.1, you're conformant to technical standard 2.1. The CRs, the approved corrections that came out against 2.1, they don't have to apply to your software for you to be conformant. So there's no redo. There's no, if the technical standard changes, do you automatically progress forward? Anytime that you want to say I'm conformant to a different addition of the technical standard, it's a new run through the process. Now we can go back to that cost question. How much does this cost? Your new certificate from the conformance authority is going to be that $9.95. Every time you get a new certificate, it's the same price. The cost for your VA to go through and do whatever changes you made to go to that new addition really just depends upon your contractual relationship with that VA. Theoretically, it's going to be a whole lot cheaper. If all you do is change a couple of things, you've already done your verification to a previous addition. So does that answer your question? So I would like to add also, as an example, we have 2.1.1 that's out and 3.1 and 3.0 that's out. If you've written a UOC that you can configure to run with any of those, then you could do that as a variant and say we support these 3 different versions with the same piece of software. So that's another possibility. But as far as something new coming out after you've written your software and gone through conformance, I'd have to agree with Chris. You would have to go through the process again unless, per chance, you didn't have to change any source code. Now you'd still have to go through the process again. Yeah, your conformance statement says what addition it's for. Right. You can't change that after you did your conformance. You cannot change that. But you may not have to make any modifications either. So it all depends on, you know, what your UOC is designed for. Okay. Does anybody have any other questions before we get into the, how do you be a VA? All right. Before I get too far into this, I do want to do one quick. I was going to say that there is one, one other thing that we could do before we, you know, going to be a VA and that is, you know, we have a whole bunch of sites. We could just show them the list of topics and see if it spurs any questions. Well, I, it just may be something someone hasn't thought of yet. Just an idea. Well, let's just, okay, we've got a question. How many VA's are there? They're currently four. Before I get into this next set of slides, I see Brennan's on. Hey, Brendan, I know that there's a new RFP coming out, which is going to affect these slides. But my idea is to how far we are away from that happening. Hey, Chris, can you hear me? Yeah. It is out for BWG Ruby. So I would say definitely by, I can shoot for the December face to face, but by the end of the year for sure. If not before that. Okay. And it's not changing this a whole lot, right? We're just making some fine adjustments. But the reason I'm bringing that up is that if you're applying to be a VA, what we're going to go through is the process today. And by the end of the year, there may be updates that are going to change what goes into these slides. Yeah, I would agree with that. And I think the changes were to address some fairly specific issues from before. Right. And Steve, to your point, they're, and really when we post these slides and we make these slides available everybody. One thing you have to be aware of is that these slides are really kind of everything after the first 20, they're kind of draft. They're really meant to spur conversation. So there's a lot of things in here that may not be fully fleshed out when you're going through and reviewing them. I know that the, let's see. I can just go through the list of sections that that we haven't covered. We have a section on the conformance workflow tool on when you go to the website to work through conformance. We have a section on the face registry and what that looks like. We did the statement of conformance slides. There's slides on the conformance verification matrix. There's a slide on what verification evidence is. A little bit about the binaries. When you compile against the cold standard library. There's some slides upon using the conformance test suite. There's some slides upon the data model. There's slides about the approved correction workflow. The PRCR process. The verification statement. And I think we'll actually back up to slide 72 and do the VACOP real quick. So if any of those topics I went through, if anybody's got interested in seeing the detailed slides and those topics just posted in the chat. And we're going to start getting into a bit more about the VA. Okay, so the verification authority community of practice is led by the conformance verification matrix subcommittee. That's the team that Robert and I lead. We can provide the charter for anybody who wants to see it. The primary goal here is for documenting known issue or common issues in a knowledge base. So the VA's get together to discuss things. They create a knowledge base. Yeah, this is another set of slides that we do need to update because the approved guidance is basically fed back to the CVM. And the, the VACOP doesn't actually publish anything to the public. There is a knowledge center that's restricted only to VA's and the guided discussions are moderated by the CVM team and they're only open to VA's. The VACOP liability to the consortium is that we will provide the attendance records. Come 4586 knowledgeable and helps somebody is not on you overwatched a 4586. Thanks. So, basically, those discussions are private to the VA's that allows the VA's to feel comfortable talking about issues. And if the VA's are talking about something, it, it may not be really. I'm trying to find the right words here. It's not necessarily for public consumption. As they try to figure out exactly what the right thing to do is they'll feed it through the CVM team. The CVM team will reach out to the technical working group. And we'll figure out what the right answer is without telling anybody what the interim solutions are. So those, those VACOP meetings are, are meant to be private and just for the VA's next slide. Well, I'll, I'll let you finish and then if it's not covered, I'll say something. Can add something there. I think it's important to note that even when one VA is talking to another VA, we still have the responsibility of protecting the IP of our supplier. So, we're even if we're discussing, how do we verify this or that we would not use as an example. Something from one of our suppliers that had not been sanitized in such a way that it wouldn't provide information to someone outside of our VA to give them information about a supplier. So even within that kind of setting, the intellectual property is protected. Thanks, Ron. So the VA COP, they do discussions through WebEx and through a web discussion board. Resolved discussions are added to the knowledge base. The VA COP has the contact information for the various members of the COP available. During face to face meetings, when we have face to face meetings, the VA generally meet on the first day. Which is likely a Tuesday, we might be for 2 hours. VAs are not required to attend all of the meetings. They're not required to attend all of the face to face meetings. We don't want to raise the cost of verification by requiring that sort of stuff. And those meetings are once again for VA COP members only. VAs who are not members of face consortium will only be invited to those meetings and not other consortium events. At this point, we don't actually have any VA's that are not members of consortium. The occasional WebEx on topics, we're actually meeting regularly on WebEx at this point. And then there can also be private help desk discussions with the VA to CVM team when there's things that may not be fitting for even the general VA COP conversations. Next slide. So generally speaking, the VA COP, we're going to talk about immediately talking about any approved corrections that come out. Any clarifications on what's in the verification matrix, test suite usage. When normative references exist within technical standard and there are a lot of normative references in the technical standard. What concerns there are associated with those? The general business practices and concerns before the record test concerns. If you have to witness a test, if you are conducting a test, reproducing the test and potential PRs to conformance consortium documents test suites. If a supplier has an issue that doesn't work, the VA may bring that to the VA COP to discuss whether or not that needs to be a CR. Those are the kinds of things that the VA COP discusses. All right. So let's get into the application process. Okay. Before you do, I want to add one last thing. So one of the other purposes of the VA COP is to help maintain consistency amongst the VA's. So if there's ever any question about, well, you know, if I go to this VA, are they going to evaluate something the same as that VA? The answer is yes. And the COP is there to help make sure that that happens. This is how we coordinate. Yeah, that's actually the primary purpose for the COP is to make it so that there is not a huge discrepancy in the way that VA's are evaluating product. And to ensure that everybody who's approved to be a VA that we can communicate changes and other things in a way that we are all doing things in a similar manner. Now, that's not saying that there aren't advantages in individual VA's processes, but the evaluations themselves should be consistent. Okay, so you want to be a VA. So we're going to go through what the VA's responsibilities are, who can apply, how do you apply, what the approval process is, what's in that RFP that you have to respond to, how that response is being evaluated, and what exactly are the 14 items that you need to cover in your response. So as we talked about earlier, the VA serves as the technical evaluation of conformance to the face technical standard. So you're going to evaluate the submitted products through test suite execution and inspection of artifacts. You have to be an expert because you are the final verification of the technical aspects of these VOCs. This is all outlined in the face conformance policy and the face conformance authorities plan. So in those same two documents, to prevent checking your work, you have to be organizationally independent from the development and integration of face software or face conformance software. VA's do not have to be consortium members, but they do need to be technically informed on the technical standards. So if you're not a member of the consortium, you're going to have to find a way to prove how you know the technical standard well enough to be a VA. There is an RFP that's currently out there at opengroup.org slash face, slash face VA. That's got the RFP, which we're going to go through here in a bit. And you basically write a response and you send that to ogfaceadmin at opengroup.org. Let's go to the next slide. So according to conformance authorities plan, any responses to the RFP are going to be reviewed by the VA approval committee committee. The VA approval committee is likely to respond with questions or request for an improved response. Once they're satisfied that the response has met the qualifications, they will make a recommendation to the steering committee. And the steering committee will vote to approve or deny the application. So this basically means that the VA approval committee is not going to tell you no, they're not going to fail you. They're just going to ask you to improve your response. And when you get a response that is sufficient, then you will be voted on by the steering committee. So we're going to go through the RFP and a little bit more. Is there any questions so far? Okay. So the RFP is currently structured in four sections. I believe the new one is going to have the same four sections. There's an introduction. There's a description of what the verification authority is. Section three is basically instructions. And section four is the description on how you're going to be evaluated. The approval committee, when they get the response, is going to evaluate the response using only the response against the criteria in the RFP. So basically they're going to look at section four of the RFP and your response. And that's what they're going to judge you on. The recommendation is that you read the criteria carefully when you prepare your response. The approval committee can use the entire response as an evaluation of your overall capability to meet the duties of the VA. So if you write something in the introduction, the VA can use part of that as their evaluation, as well as when you answer each of the 14 elements. So respondents should prepare a professional, complete, consistent, and accurate response. The VA approval committee cannot judge a candidate based upon reputation or knowledge outside of that response. You don't rely upon name recognition, your company, your individuals. Any of that kind of stuff has to be in the response to be used. Relevant work accomplished should be reflected in the response. Thank you for this quick question. What's an RFP? A request for proposal. All right. So in section three, there are 14 items. In section four, there are the evaluation criteria for those 14 items. So in each of these next slides, we're going to list what the item is that's in section three and what the evaluation criteria is in section four. So the first one, VA's got to be able to access the face reference repository. So briefly describe how your organization accesses the reference repository. And that's basically your ability to access bookstore at opengroup.com more or less. Number two, describe how your VA will ensure independence from software development, software integration activities of your overall organization. So either your enterprise doesn't perform software development or integration, or your personnel report to a management structure that does not direct the development of software products or integration of those products. Number three is describe how your personnel within your organization have demonstrable knowledge of the technical standard and its normative references. Note, the face technical standard has a large number of normative references listed in section two. So either your team members have been involved in the development and or review of the technical standard, or they've documented completion of organic third party training curriculum. Or you've got experience with face technical standards, normative references and its ecosystems. So you have to provide one of those three things. Number four is describe how you plan to maintain that knowledge. Now, a plan can either be an outline of an intended plan to a fully released process. The response just describes it doesn't or have to describe it just can't just name it. And in this case, you both need a training plan for new employees and a plan for learning new standards as they're published. So there are verification techniques that are listed in the CVM. Describe how your personnel have demand demonstrable knowledge of the verification techniques. This can be through documented past experience showing developmental or technical review of software and test procedures. Number six, describe how your personnel within your VA organization have knowledge of the face conformance test suite. And once again, the same kind of, you know, you've been involved in the development or review of the test suite or you've completed third party training curriculum. Number seven is how do you maintain knowledge of that test suite. So once again, a training plan for new employees and a plan for learning new additions as new additions are released. Number eight is describe your organization's process credentials for or methods for developing and adhering to well-defined processes. So do you have AS 9100? Do you have ISO 9000, the CMMI? If you don't have those, do you have an organizational policy and historical evidence of adherence to well-defined processes? How can you prove to the face consortium that you can follow a process? Number nine is describe your processes for configuration control. So your method must cover, you know, recording equipment, serial numbers, software versions, all of that kind of stuff. So number 10 is describe your plan to conduct or witness for the record testing. Your plan must include configuration of the test environment, methodologies for capturing and storing results, recording the test personnel involved, all of that. Number 11, your skills and abilities to objectively assess submitted verification evidence to detect and communicate deficiencies or omissions. So here we're looking for past history of reviewing design requirements, test procedures, test reports, methods for detecting a complete or inaccurate traces and a plan for communicating deficiencies. So number 12 gets to that supplier IP. So how does your, what's your plan to enforce policies and procedures that ensure protection of supplier IP? Now, existence of policies and procedures to generate and honor nondisclosure agreements, hold substantial materials and confidence, that kind of information. Number 13 is your plan to protect confidentiality of all information on applicants and applications. This is different than IP because this is getting to the fact that just the idea that a supplier is coming to a VA could be considered confidential and not for public knowledge. So your plan must treat all communications with suppliers as sensitive as required by suppliers, including the names of the projects and project schedules. And lastly, number 14, describe your policies and procedures within your organization for establishing and maintaining a verification retention repository. It coordinates the face documents and requirements. So this plan must include the storage of the verification statements, conformance statement, digital signatures and sufficient reports and digital signatures. By the way, it's one of the things that's being removed in the new version because that was removed from the conformance policy several years ago. So are you seeing the question? Yeah, so a VA can be a company or an organization within a company or an organization within a branch of the military. So we currently have the Army verification authority is being handled by Army Futures Command under S3I CCDC Emmerdek. The Navy is under, I forget which PMA that is. And then we have Tucson embedded systems savvy, which is related to Tucson embedded systems. And then we have SCIENT as the current organizations. But yes, it's possible for an SQE organization from one of the major suppliers to also be a verification authority. Okay, so the slide that we've got up here, this is a list of all the documents that you may want to go through if you're planning on applying to be a VA. The COP best practices is not published and won't be. So that's one of the things we need to remove from the slide. So the last slide in our deck is the more information slide. These are contacts that you can reach out to if you have more questions. And of course, we're going to say if you've got more and you show up for the next face to face, you can always join this meeting again and ask questions. Usually we get a lot of them. And I know this meeting goes a lot better when we're sitting in a room and able to discuss things more freely. And with that, I'm happy to end us at an hour and a half if there are no further questions. All right, everybody, thanks for joining.