 Hello and welcome to the aerospace village talks. My name is Matthew Gaffney, also known as Gaffers. And today I'm going to talk to you a bit about my experiences with vulnerability disclosure in aviation. And then give you some of my lessons learned should you find yourself going down the same path in the future. So I need to be absolutely clear what I'm going to cover in this talk because I'm not going to cover. So I'm going to strip out any detail of the vulnerability itself the manufacturer or even the technical platform that it's on. This is purely talking about the process that I went through in terms of trying to disclose the vulnerability, what worked and what didn't talk about the timelines and the steps I did take. As well as the frustrations that I've experienced along the way. I'm sure that the, the manufacturer has also had frustrations but they haven't communicated those directly to me. So I can't share those. And then also I call it the differences opinion that we had any kind of third party assistance and the current situation in the future steps because believe it or not this is still ongoing. It isn't over. And I'm going to keep going until I have absolutely exhausted every single avenue, although as you'll see at the end, I am literally at the last stage. So in the background, I was, I'm a contractor and I was brought into to to work in a team that was working on the entry to service at the operator level for any way craft and it was really, it was a role I couldn't, you know, I couldn't pass on. I had already worked with this company before and really liked them and to a go back to them work again and be working on cyber security and aircraft was just, it was just a dream come true. So, you know, I jumped at the chance and started working with them and pretty much threw myself into the role and the role covered. And actually, you know, it's a security wrapper around the whole, the whole program really. So as they're designing policies and processes and procedures, you know, for different areas that required it overseeing the security design of solutions and systems. And those were both the interest and the third party designs as well. So there was a lot of information assurance around what was being delivered. And then, of course, you know, following that the good principle of trust but verify, you know, we got third party and to do some pen testing. And obviously my role was to design that pen test and get it signed off by the boss. And so I went through, you know, I was basically giving, you know, guidance and assistance to to the entire team, you know, when when they requested it. And sometimes even when they didn't like I could see there was an issue or see there was a concern, I would step in and give some assistance. So discovery. So, I think it's important to know I did not actually discover the technical vulnerability. It was the third party contest company that did. I thought it was important to, you know, check my work, because I said trust with verify and and and then also see if any of the issues that a I had missed or they were given scope in in other areas I hadn't looked at because I needed to be looked at by a pen test company anyway. And, you know, we set them on their way. You know, and it wasn't just once we're using a couple of elements in the program. And then on date to this particular one. It was, we need to have a call because we found an issue and when you've engaged the pen test and get a call like that on day two of five day pen test. You know that it's not good. So they shared the details of what they don't be found with us. It didn't sound good. However, the thing to bear in mind is this was a contesting company they, they do have a technical opinion. And in aviation aerospace, that's not the same as a safety impact. And, you know, there's been lots of discussions about safety versus security and what it means, but when you're dealing with disclosing vulnerabilities in aviation, especially you have to demonstrate how it would have a safety impact. So, but I'll come on to that a bit later on. And the pen test company, they're really good actually that they're very nice as well. They provided the full details that, you know, that the PCAP files and including they put a, you know, a proof of concept script as well that they had developed to to perform the attacks. And they find it to me so I could then, you know, verify and maybe invest a bit further. So, the initial disclosure took place in April 2019. And this was, you know, with the manufacturer that the operators have a portal, which is used to for all, you know, transfer information and data going back and forwards. This is in the days before any of the manufacturers, the aircraft manufacturers had a vulnerability disclosure program policy. And basically what we did is, I rather naively assumed that if I uploaded the technical details of the vulnerability, they would look at it where from an aviation perspective and go, oh yeah, that's not right. And maybe there's something we can we can tweak to improve that. I was naive. So the initial response I got was that actually what had been found was an intended functionality and not a security issue. And just to, you know, when I advise to organizations out there, never, ever, ever say that to a security guy, working in as a pentester as an information assurance, you will just get the hackles up. And, you know, like I said, in this scenario, that's what happened, you know, I was like, okay, fine, you don't think there's a safety issue. My gut feeling says there is, but I didn't know enough at the time to really be able to communicate that. So, I started an investigation, but from a perspective of safety. So, in early, you know, June 2019, I started an investigation with multiple stakeholders across the business. And, you know, so I dragged in engineering pilots and operations. Worthiness. Everyone I could think of that could give me their point of view on what this meant. And I basically, and together we formed a number of scenarios and assessed what the potential safety impact could be. And we found that there were multiple scenarios, which, which absolutely would have, if they were successful in, in, in, in the endeavors, would have probably or could have led to a safety impact. There's no guarantee to say that it would. There are other things in place, and I can't talk too much in detail about it because that will kind of give it away. But there are other controls in place, but should any of those fail, then there's a possibility of this, this having a potential safety impact. So, we, we put together the investigation, put together the safety case, and we actually supported it with the manufacturer's own operational guidance and procedures that stated that this, if this happens by accident, it could have a safety issue. So, so we had even our own words really that, you know, if this, if this was an accident, and something about an error, then it could have a safety impact. And here, I was showing a vulnerability that could show that actually somebody couldn't attack it and have the same impact or same effect. So, we closed back, you know, back to the manufacturer with with with demonstrations and the proof of concept I mean videos of doing it and provided everything to. And for two months, they started they investigated themselves, or at least they said they were enjoying that time, we were looking at this from from our perspective is what it meant for us so we were starting to do our own mitigations as well. We were looking at, you know, technical and procedural mitigations that we could put in place that would not nullify, but they would certainly reduce the likelihood and they would, you know, potentially also reduce the impact as well. In July, we received a response that they would indeed, they agree with there was a there was a small issue, and they agreed that they were going to make a small change in the configuration. And this was not ideal, because the vulnerability was still present. The vulnerability had not been fixed. The configuration changed. All I did was reduce the tax that the attack surface down. It was still possible to, in fact, he still is today possible to leverage this vulnerability, if you know what you're doing. And despite not being happy with that we, we reluctantly accepted that we weren't going to get any further than this. And the with these this change in our own mitigations as well that we were happy with the risk at that point. However, I wasn't quite happy. So, manufacturer developed a new version and they released it in early 2020. And at this point, I was, I was not full time I was, I was just working a couple of days a week for the, for the operator. You know, we, we had, you know, things that changed a little bit and, and she's working two days a week. We had less time to work on, on various things and actually focus on, on, on the meat and potatoes of what I was doing. So in early 2020, a new version was released to operators and in the release notes, there was a very small notes, just about the technical change that had been made. Nothing else. There was no information concerning the vulnerability. There was no information as to why the change was made, just that there was a change. In addition to that, there was no security notice that have been issued. So that meant that manufacturers was even obliged to apply this upgrade. They could, you know, still be compliant by running previous version. And so they were not aware of the vulnerability, not aware of the potential impact or the risk, and then, and then to make matters worse, they, they, you know, completely oblivious to that and unable to control the mitigations in place. And I found that very, I found that very disingenuous. I found, I was very disappointed with that because it goes against everything that I come to learn about in a nation in terms of how you deal with issues, you know, you can't know all the problems, you can't know all the weaknesses, but if you do find something that's wrong, you know, you have an obligation to speak up, speak out and inform people and make changes when necessary. And I didn't feel that was happening in this scenario. And also this manufacturer gives operators a number of security recommendations. And there were no changes to that either, even though there was an opportunity. There was no changes to these recommendations to include other mitigations that could come into place. There was ample opportunity for the manufacturer to inform operators of the issues and they chose not to take it. So, April to October, a bit of a funny time for me because obviously with the impact of COVID, the operator I was working for got rid of all contractors. Like all airlines they were in a dire situation in terms of cash flow and absolutely understand that that decision and absolutely hold, you know, no grudge against them for that. What it did mean is that obviously I don't lose access to the manufacturer. What I do have though is a number of contacts in different operators around the world. So, I've got this spare time on my hands because, you know, I'm not working anymore. So I decided to start tracking compliance of the new version, seeing if operators were indeed taking it up. So I started contacting those that I could. I'm literally not very many, but the ones that I could and got back to me said they're not aware of any variability, let alone this one. They're not aware of any ability in the software. And they was actually still using a previous version. And that disappointed me a lot because the emphasis wasn't put on the need to upgrade the operators are still using a vulnerable version, even though there was one out there which reduced the attack service and likelihood down to a level at which the manufacturer was at least happy with. So I then attempted to do some outreach. And this is this I found very frustrating, because, you know, I'm not someone with thousands of followers on Twitter. You know, I don't follow that. I don't. That's not my goal on Twitter. You know, although it might have helped me in this case, but hey, it is what it is. So I tried something on Twitter that didn't get very far and then I tried direct approaches to other operators. I didn't already have a relationship with. And that was difficult because, you know, I'm very rarely if I can't remember seeing a single vulnerability disclosure policy for any of the operators that I could find, or if they were there I couldn't find. So I tried just doing direct approaches through LinkedIn, through email. And as you can imagine, I had very, very limited success, despite my best intentions and best effort. So it gets to October. I've just started a new role, new contract, not an aviation sadly, but it was a green field. You know, site one man security team for, you know, a medium sized company, quite a challenge and obviously I was quite busy, but I decided to still couldn't let this go. You know, I'm quite stubborn as multiple people told me through my life. But I like to think of it as when I get a bit to my teeth, I don't let go. And I thought this was important to have to keep going. So I reestablished contact with the, with the manufacturer. And at this time, at this point they had a vulnerability disclosure policy and program so I thought, excellent, I shall use that. And I waited, and I waited, and I waited, and then after about two weeks, I got a generic reply, which was not really a response. But like I said, I was really busy in my new role, and it kind of went on to the backburners a little bit. And it wasn't really until March this year that I had a bit of spare time, a bit of, you know, you know, a bit of spare bandwidth really sorry to use this, but it was probably, you know, not a few people, but you know, that's me. So I decided to follow up with another email to the same, you know, vulnerability disclosure email. And that time I had no response whatsoever, not even an acknowledgement. So after waiting about four weeks, I decided that I was, I was going to take this to the next level. So I went in and approached an entity at the regulator. So the top top level of the regulator, what I did was this is a branch of the regulator that deals specifically with disclosures of vulnerabilities. So I thought, perfect, these are the guys that can help me get this through. They're going to understand where I'm coming from, they're going to stand by the manufacturers coming from, and we can find a way forward to make operators aware of the problem. So, you know, they asked me to provide full details. So I did, I wrote, you know, you know, I think it was nearly 16 pages of reports. I gave them all the videos and, and all of the results. And they then did their own investigations. They contacted the manufacturer and immediately, literally within a day of them contacting the manufacturer, they responded to the regulator and shut the conversation down. I was very surprised at how abrupt that was. I was also very surprised that it appeared that this entity and regulator didn't really seem to have any kind of say in making sure that, well, if you shut the conversation down and still go direct with them because that's what they asked, they asked that I didn't go through this entity, they asked that I go directly back to them, despite the fact they have been ignoring me. This entity regulator was unable to help any further than that. You know, they, you know, they seem to have the right intentions, but just not the right sway, I guess, is the proper way to put it. So I complied with the request and I sent a new email to the manufacturer and it took them four weeks to reply. And it was basically, you know, whether the operators have been informed of the vulnerability, whether there be a security notice, whether, you know, operators have been encouraged or forced to upgrade to the latest version. I knew they hadn't been because my friends or the operators have told me that, but I wanted to hear it from them too. And they also then put in a couple of things that which was quite interesting. So I did ask them if they were looking because I know they're looking to replace the software which the vulnerability was founded. I know that there is a complete rewrite from scratch going on, and I asked them what the timeline for that was. But they also included a little note there that only airworthiness authorities are empowered to mandate design changes. And I found that an interesting statement. So what they're saying is that they can't make them mandate, mandatory. But the authorities are, it just seems a little bit weird to me that they seem to hold power in this, this, this relationship, but they, you know, they won't move unless the, you know, the authorities are, but they're not informed. Neither of the operators. It just seems a bit weird to me. There's something missing in this whole relationship. And then we get to the final correspondence between them and I. So I asked them, you know, nicely to, to make the update or consider making the update mandatory, given, you know, given the potential impact and the safety implications of it. And that they could also maybe amend their security recommendations which actually find extremely good to include the mitigations that I've shared with them previously. And I thought that they allow me to inform operators myself, you know, either through direct communication or a public disclosure. I didn't really want to go down the reach of public disclosure. I would rather, you know, be kept, you know, more closed loop to what because the validity hasn't been fixed. So it would be irresponsible really to disclose that whilst it hasn't been fixed. And they, you know, at the same time, I hadn't also told them I had already tried direct communications, but I was hoping that maybe if they were okay with that they would give me a list, you know, a good list of people to contact on their behalf. And then I got the response as you can, as you can see nearly, nearly two weeks later, and it's the response actually upset me quite a bit. And by saying that as I had not gone through the proper vulnerability disclosure process with them. There was no mutual agreement understanding reach between the two parties. Therefore they don't support a non coordinated disclosure. That's rather disingenuous because they know for a fact that when I first approached them, albeit at the operator level that they didn't have a vulnerability disclosure program. This has been going on so long that it predates their program. I haven't shared any content which wrong because I give them full details, proof code, everything. So, I found that really, really disingenuous and very disheartening to read. You know, I'm reading the two lines it's almost like a, you know, basically a very curt response to say get out of our way is what I'm reading really. It's actually cut and paste, you know, so I actually lifted this from their email. I'm embellished at all. I've just obviously omitted the words of the manufacturer or the names of the manufacturer. And of course they don't expect their parties to contact its customers on the manufacturer's behalf. So that's them saying don't, you know, don't get in touch with them we don't, we don't support that. So, you know, you might think that's the end of the road. But it isn't. So, it's still ongoing. So for advice with, with some other researchers and, and people who are I, you know, I respect a lot in this business. They have had really good results going direct to regulators. And not these entities that are kind of built for this, but actually to regulate themselves the people deal with the, the airworthiness and the safety aspects directly. So that is going to be my next step. I'm going to follow what they did. I'm going to start with one, but I'm going to do it to multiple and I'm going to make a full case and I'm going to show them everything I've got. And, you know, unless these regulators request it, I will no longer initiate contact with with the manufacturer question. I think there are enough chances now, enough information from me. You know, free dinners, so to speak. So, I think, I think it's, you know, unless I'm a master or forced to I'm not going to, to initiate any communication if they want to get to me that'd be great. But, you know, bear in mind that if they do unless it's about, you know, some proper meaningful steps forward in this issue then, you know, I'm going to be, I'm going to be very receptive. So this, this software is actually being replaced. And when it is, and when it's safe and nobody's using the, you know, the current version, I will think about going public with the whole thing, because not many safety impacts at that point. But, you know, this could be this could take several years. So I have no idea where the software is in this cycle because they won't let me know. I'm not going to lie on my friends in the operators to give me a little note when it happens. And, like I said, I'll give the whole thing it's not a thing is it's not a, you know, uber technical vulnerability. It is something quite basic very simple it has to be. I'm not, I'm not a pentester I'm not a hacker. So, I like playing, I do a fish assurance like learning new things. You know, I'm not a, you know, I'm not an uber hacker like some of these other guys this is not a complicated hack, but there is a safety impact in my opinion. So, that is why I've got a bit of a tease on this one and I'm going to keep doing it. Remember, my wife works in the industry. So I have, you know, skin the game on this. I'm not going to let go. I mean, for you guys, you know, well, you know, I'm sure that there are other researchers out there who are going through some frustrations. Or you're having better results. If you are, please get in touch, share your stories. I would love to hear them. But, you know, I've got some recommendations based on what I did. You know, although, you know, the VDPs are aviation, aviation businesses are pretty new. I would always always always initiate through VDP. They have, they have published on their websites, and they, you know, hold them to it, you know, hold them to their own policies hold them to their own procedures. And also, when you do disclose, remember, you may not have all the details, you know, if you, if you've, you know, managed to get a, you know, a silo of information on one part of the aircraft or the system you're dealing with. There may be interactions with the things that you're not quite aware of, you know, these things are immensely complex. I was lucky I was working at the operator level so I had access to, you know, not just specific software but also configuration details and recommendations for everything around it. So I couldn't make that full determination. But there are scenarios where you may not have all the details and I'm working on something else right now and I'm in that, I'm in that situation so it could be a bit of a two way street between you and and manufacturer. Also, before you disclose research, any safety impacts that they could have. And include these with your disclosure. And there's two ways you can do this, you know, you can look through regulations and, you know, they are quite long. They are, you know, they, you know, they're good for putting into sleep at night. Apologies to the guys who write them and no, I know what policy writing is like, but they are very, very dense they are very, very information heavy. But in there, once you get into it and you start understanding it, you'll find the rules and the recommendations, which will help you make your safety case, or you could do. Also use aerospace SMEs, you know, use pilots, use your engineers, you should EOP engineers, you should have worthiness guys. If you don't have access to them, then reach out reach out to people see if they have friends and contacts who could maybe anonymously because they're going to have to maintain that professional relationships but you know they could provide that information, or their opinion to you on certain things or you can just stay stay in the right direction. And because of those, because of that interaction and that that research that you do you can then start recommending mitigations that you've developed with these SMEs include those in your, in your disclosure. And what I would say is, as you know, for researchers, and absolutely do not do not disclose any vulnerability publicly before the safety impacts are sufficiently mitigated by the manufacturer. I'm in a situation whereby the manufacturer isn't taking their responsibility properly, in my opinion anyway. That is holding you back. That is holding you back because if they decided to fix this and they could, without, you know, replacing the software completely, they could fix this or put some things in place that would make it much more difficult to, you know, to leverage their vulnerability. But they decided not to. So, you know, that's that's on there, you know, but it also means that I can't then disclose publicly because the safety impacts are still there. It's a bit shitty. That's the way it just just the way it goes. What I'd also say as a researcher is don't take no for an answer. Be stubborn. Ask questions. You know, you might annoy them. I probably have annoyed this manufacturer beyond what they they're probably used to in terms of scrutiny from from somebody who's not in their worthiness. But don't take no for an answer. Don't sit down. Don't get fucked off. What I would also say is, don't be as nice as I have. I've given them lots of chances to do the right thing. And, and they haven't. And, you know, it's taken me this long to basically say, right, that's enough. I'm going to go, I'm going to go to the regulator and I'm going to see what they say. And maybe there'll be an update to this at some point, but maybe they won't be. But we'll see. We'll see what the regulator says when they get back to me. There's recommendations for you aerospace organizations out there. So to quote the great Jeff Troy, a vulnerability disclosure program is not a page on the website. You, you know, you really need to, you know, ideally, you know, have these processes built in place and tested in your company. You know, that's that's the minimum you need to do. Ideally, what you need to have is, is that that program is your ethos, you know, that is, that is a, you know, a feedback loop into your own safety program. You know, so absolutely, absolutely take vulnerability disclosure from researchers seriously, you know, do not follow them off. Now, if they turn around and come to you with something that is not an issue, help them understand why. I'm just saying it's not an issue because trust me, someone like me will just get between the teeth and they'll go at it like a, you know, like a dog in a boat. Follow the spirit of EP, even if you don't closely buy buy, you know, and respond in a timely manner. We'll see some of the delays I had in response there with this manufacturer. That's not really acceptable. When you consider that in, in, in businesses, your normal timeline for disclosure is 60 days and some of the responses I had were about as long as that. You know, you have to bear in mind that the research might turn around and say, well, actually, I have done what I think is ethically correct. I've waited 60 days. You haven't responded. And I'm going to disclose and then we're all in a pickle because that information is out there before you, before you had a chance to properly respond and work with a researcher. So respond in a timely manner. And like I said before, if there are gaps in the research, work with the researcher to understand it better because it's going to help you in the long run. I'd also say, do not, do not hide by regulations, even if they return the way that allow you to do so because it's a smart, great behavior. And it's, it's really, it's quite disingenuous to me, to me, in fact. So if you are, if you are relying on security assumptions in your own risk assessments that you put forward for airworthiness, and then don't inform operators of the risk. And sorry, if that assumption is that you assume that the operators doing certain things, and you don't inform them fully of the risk, then how can you really value that assumption? How can you really have confidence in that assumption as being an effective control in your own risk assessment? So, you know, really, really, you know, consider that when, when you're making these assumptions and how you deal with risk in the operator level. So that's it for me. I'll be doing some Q&A in the Discord channel. What I will say is do not ask any details of vulnerability, do not ask me, you know, who the manufacturer is, who the operator is, not going to talk about that kind of stuff. So we've got to talk about the process and advice about how to deal with manufacturers, operators, organizations, regulators. I'll be in there. And if I don't know the answer to the question, I know a lot of people who do. So I look forward to seeing you in the channel.