 All right. Hello, guys. Welcome, everyone. Thank you for coming to my talk, to my TED talk. Sorry. It was one of those. So today we're going to talk a little bit about security. It's not going to be a deeply technical talk. It's more like to discuss what it means to apply these new concepts of security we're seeing all over the show this couple of days to the embedded world. What does it mean to migrate to these new ways of thinking about software development and software management? Just a quick introduction over at Pantacore. We basically do container technology for embedded systems. We try to push, we have all of our customers or people who were trying to imagine their next generation architectures of embedded systems, trying to figure out how to go from legacy monolithic stacks to things that are more modular, faster, more agile, easier to manage, easier to evolve over time. There's a lot of ideas out there on what are the best practices to achieve that. But the context of this presentation in particular is not what we do over at Pantacore in particular with containers, but rather how does this modern way of managing embedded systems have some kind of affectation to the reality of security that is becoming more and more important every day. So the title of this presentation is Uncovering Software Providence in Embedded Systems, which reads real nice, I think, sounds kind of poetic. What does that mean? So uncovering software provenance in embedded systems. So provenance. What is provenance? The idea here is that at the end of the presentation, it's not going to be a super long presentation, but we'll have time at the end to have, I think, a good conversation about what maybe you guys see either at companies you work with or if you have customers and you do consultancy work for embedded systems and these kind of things, like what your customers are starting to talk about. I'll talk from the point of view of the people we work with. So provenance is interesting. It's the place of origin of something. Somebody put it very nicely, the chronology of ownership of something. What is the place of origin? Where did it come from? What am I actually looking at? This is very relevant in the world of software, particularly in the world of open source software. Why? Because everything is just a million things stacked on top of each other. A lot of the times we don't even know what it is. Most systems are running, you could say. And that has become more and more of a trend over the last few years as containers have become ubiquitous in the cloud, for example. Then we have gone more towards this black box model where it's not clear exactly. It's not that it's not clear, but there are holes, let's say. It lives gaps in our ability to truly have fully deterministic systems, knowledge of what's running on those systems. For embedded in particular, this is interesting. We think about what has happened at the cloud. Containers have been great. We've sacrificed, however, that sort of security at the point of development or point of origin for super fast DevOps. That works for the cloud. The cloud is much more easy to also manage if there are issues, if there are problems, it's much more ephemeral. It's just fast. You can deal with these issues at a much higher velocity. If you're thinking about the embedded world, however, then that's a complete different problem segment. The embedded world is not like the cloud. If a device fails, if a washing machine dies, that's problematic. It costs $100 to send a technician out. You cannot just turn it off and on like you could do with a cloud instance. Back in the day, embedded used to be much simpler. It was about more fixed function devices, single purpose. Do one thing, do it right. A power control system or maybe you're all thermostat just did one thing, did it right. It wasn't connected. Those things were not like all over the cloud and accessible from everywhere. None of that. Before it was a simpler world. That has slowly changed. Embedded has not moved at the same speed that, let's say, higher levels of the stack like cloud development have moved. But now we're starting to enter that world where containers are making a push, making a play in the embedded world. Devices are no longer fixed function only. Devices are not completely ergapped anymore. Things are connected. The software stacks are much larger. We're talking about potentially multiple different functions, features inside of a unique device. Everything is sort of like split around in microservices. It starts to be a little bit more like that. You think, let's say, of a washing machine to go back to the same example. Then it's no longer just a program. Maybe it's a thing that sends a lot of telemetry up to a cloud somewhere so the manufacturer can know where and where on the drum, for example, so he can learn about the ways that I wash my clothes. How often do I do it? Do I use liquid detergent? Do I use powder soap? All of these things. There's a lot more going on in our devices than there was going on 10 years ago or 15 years ago. That's great. But it brings a problem. We open the door to a lot of unknown code, you could say, that it's going to be running on devices that it becomes a very complex problem. The bigger the software stack, the harder it is to keep complete tabs on what's running there, really uncovering that software provenance, the name of the talk. What happens? How does this affect, you know, how does this play into the security of the embedded world? Security for embedded. A lot of people will disagree. Someone will agree. It has always been relevant, but not massively important. It has been relevant, but, you know, it's a dime a dozen, all kinds of security issues. Everybody always has in their mind projects that have failed, things that have been taken over. For example, we're going to talk later about the last year, the colonial pipeline, the ransomware attack that probably screwed a bunch of you guys here in the US. But, you know, before legacy, like I was saying, legacy embedded systems were monolithic mashups of a ton of software. And it didn't matter that much what you would do there, because maybe these things were not connected and so on. And the smaller it was, the easier it was to really understand what was running. But now everything is connected. The systems are massive. The surface of attack has multiplied. Simple as that. Surface of attack has multiplied. And when we talk about IoT devices or smart home or whatever you want to call it, and I'm talking more towards the user facing part of the problem, not necessarily, you know, an assembly line factor robot, but like more user facing things. As things became massively connected, that attack surface massively increased. So up to what point is it okay to oversee that security problem at the user level? It becomes more and more important to actually make, let's say, provide that confidence to the end user. Devices are safe. Simple as that. Everybody is always worried, you know, everybody having an Alexa in your house is like always listening to you. But that's a decision you made. You wanted to listen to you. It's fine. What about the other devices you didn't explicitly say, right? Yes, please listen to what I'm saying, right? So, you know, and these are tons of them out there, you know, like there's been a million cases of children's toys with cameras and so on that get compromised, right? For example, right? So a little pet over pops up over there, right? And then, you know, content finds its way to the dark web and you don't want to be that content producer. I mean, that's not the type of influencer you want to be, right? So should we be worried about the safety of our data, private information? Of course, right? We're all worried about that, right? We're just not too aware, right? Users are becoming more and more aware now a little bit more, but more education needs to take place, right? Plenty of big, big hacks come to come to mind, right? Let's see. But the most important thing, let's say, most relevant thing that has happened over the past couple of years was the colonial pipeline ransomware attack last year, May, I think, something like that. I imagine some of you might be familiar with that. If you try to load gas during those days, maybe the, I don't remember if it was the southern states or northern, but it was like 12, 15 states or something like that that were heavily impacted in the gas flow, right? To the east. Okay. And then, you know, mass panic ensued, everybody was loading their stuff, so that created shortages and then price skyrocketed in those states, and it was like, it was down for four or five days, right? Then they ended up paying 75 Bitcoin, it was like 4 million at the time of writing. At the time it was actually paid, Bitcoin had crashed, that's another issue, but not the topic of the stock, right? But the point is, you know, there was nothing to, right? A critical infrastructure piece of the country failed because of a security issue, right? How do you catch those things before they happen? Okay, after they happen, you're like, okay, well, they took it over, okay, fine, let's do a post mortem, let's find out what was the vulnerability, let's patch these things, let's figure it out. But how do we mitigate ahead of the event, right? The potential vulnerabilities that might be on a trusted pipeline of software, let's say, right? Or a known pipeline of software, so that maybe we prevent things like this, right? We see also all the time, you know how bottlenecks spread all over routers? These are a dime a dozen, they happen every year, you know, like, and sometimes they take down infrastructure. A few years ago, there was, not a few years ago, last year there was something with Deutsche Telecom, a bunch of routers went down, right? Okay, that was a bottleneck that went a little bit haywire, just expanded it all over the place, and it just killed the devices. It probably wasn't meant to kill the devices even, just to go in them and actually, well, continue spreading that bottleneck. The, let's say, the trojan wasn't great, it killed the routers, right? In itself, of course, but it left a lot of people without internet. Deutsche Telecom was sort of dead for a few days, right? And these things are happening more and more and more, right? So how do we mitigate, at the point of the vulnerability, become disclosed, or becoming available, or at the point of a security team, just finding out that maybe there's a surface of attack over there, and how do we reduce that time gap of action, right? To really prevent the issue from escalating to something like this, right? That's the part that is hard, right? So the colonial pipeline incident resulted in the federal government, right? Having a very quick response, they had to say something to figure out, you know, like, they had to say, okay, I'm here, I'm going to fix it, don't worry guys, right? So Biden came out and signed the very famous, in the show, the software bill of materials executive order, right? This was in May of last year, about five days after it was, after the ransom was paid, yeah, May 2021, five days after the ransomware was paid, an executive order was issued, right? That amongst a bunch of things, right? It basically creates a bit of a framework for which device-centric companies or, you know, technology companies have to adhere to if their hardware is in critical parts of the federal government, right? So it doesn't apply to everyone, doesn't apply to consumer hardware, technically, right? But it's the first step, you know, towards a, and some of you might have even more detail or might have read the whole thing top to bottom many times, maybe drafted it, who knows, right? But you know, one of the key points is, in particular, this one, right? The providence of open source software used within any portion of a product, right? That's usually the hardest part, because if you have a product that's a hundred percent proprietary software, you know, built for purpose, then you're kind of in control, right? You're kind of in control. You might be screwing up and also creating a bunch of vulnerabilities, but you're in control of that software stack. You know what's coming from where, right? We're talking about open source software, which is now pervasive to every part of everything in our lives, right? We're all using open source software, we don't know it, you know that, right? It starts to become important to understand what issues can it bring, right? But the real issues are the ones that you're not even aware that might exist. You might be making, let's say, the colonial pipeline, right? So some control systems that regulate opening and closing, I'm just making this up, right? But, you know, that might have some old version of some library, let's say, open as a cell, right? And maybe that helped really, you never thought about that, right? And that thing resulted in some kind of compromise path that actually allowed those guys to take over the entire system, right? Or to go back to the to the scenario of your home, right? Maybe you don't care too much about a washing machine being compromised, right? But at the end of the day, the washing machine is in your home network, right? So maybe that's an attack entry to the rest of your home network. You might have other equipment in your home network, like closer good cameras or something that you're like, no, this is air gap. My video is not going out to the internet. But maybe somebody enters your home network through another piece of equipment, like a washing machine, that might have a little library that was compromised, right? How do you discover these things? Well, it's not enough only to have a software bill of materials. Software bill of materials is great because it lists the components. It gives you at least a starting point to try and address the problem, right? But it's not the only... It's not the complete solution, right? It's just the starting point. So what does S-bombs have to do with embedded IoT? How can we apply some of those concepts? What can we take from them, right? So there's a harmful... It's a handful, not harmful. There's a handful of vendors out there that have been creating for the cloud side, you know, code inside, dependency graphs and dependency management, you know, dependency discovery tools, basically, to address what it means to create your software bill of materials, right? And they've been connecting that to very interesting systems that allow you to have automated insights, alarms-based, and vulnerabilities being discovered, right? All these kind of things, how the things work together, right? And really go down to the tiny little details of where might a vulnerability surface to the higher level of your stack, right? The cloud has... It's certainly... The cloud is always a little bit, let's say, more advanced in the adoption of this sort of like forward-looking technologies, right? That doesn't mean that they cannot apply to Embedded 2, however, right? Now, the thing, remember what I said at the beginning, historically, Embedded has always been about monolithic systems, right? Systems that are a mashup of features, right, that is not very modular, not very agile, not very easy for you to understand the blocks that make it up, right? As we go into a more, let's say, advanced future for Embedded as well, right? Then Embedded starts to adopt some of this container technology, right? So we can leverage some of what has been done in the cloud to have the way to get ahead of some of these issues, vulnerability is popping up, right? Not after the pipeline has been taken over, but before, right? So at the end of the day, we can consume some of these technologies and if we mix this up with interesting, let's say, well, software management techniques, distributed architecture for modular software delivery, these kind of things, then we can start connecting the dots and being a little bit more secure ahead of the potential event, right? For example, there's this guy's, one of the logos I showed, Rivenera, right? I've been looking at these guys for some of the projects we're doing with some customers and this is interesting, right? Basically, there's an inventory of the software that makes up each one of the software components, right? They don't deal directly with containers, but if you have a component list, for example, your main data software build materials, if you're, let's say, contracting with the federal government, right? Then it's easy to block that into this thing, right? And give you alerts, give you vulnerabilities as they come, as they surface through any of the security channels, any new CVE that touches this, this, or that, okay, well, there's a system, all of that is indexed, all of that goes into a very beautiful, nice database, API driven, you can connect it to any system, right? So as you start having this kind of tools, right? Then it becomes much easier to connect it to downstream software management pipelines, right? The same way that you would do on the cloud, right? If you had this connected to, you know, to, let's say cloud infrastructure, then you could very easily flag containers that actually have those part of the components. That's what Sneak and these other guys do for actual Docker containers and the like, right? You connect those two things, you can very quickly say, you know, application, A, B, C, whatever, this service deployment or deployment level are vulnerable for this, this, and that, right? Those alarms could be raised, but we could go even one step further, right? On the embedded world is really, as I remember what I said before, if a cloud server gets compromised, at best you're gonna have data, you might end up with data loss, but you know, if you're doing your things right, maybe it's not data loss, maybe it's just, you know, downtime for the service, let's call it, right? But the colonial pipeline, that one, there's no way you can just fix it, right? If it's gone, it's gone, right? So we can connect these things with the software like cycle management tools, with the software updates, software delivery mechanisms we have, right? And maybe the instantaneous mitigation of some kind, right? These are all ideas, of course, of what can happen in the future, like we're not doing any of this, well I mean, we are with some customers exploring this because they're beginning to ask about it, right? These are customers that before maybe didn't care too much about it, they just thought it's okay to just have a security team and they'll sort of keep an eye out on things and then we'll see what happens, right? But it's becoming more complex, right? So they're starting to think, okay, what can we do to think a security as part of the architecture early on, right? Not as an afterthought, right? Security after the fact is always wrong, right? So this is what I was just saying, right? So what if we start to automate some of those systems, right? If we start to take the smart insights from code, from dependency management and so on, maybe we can start to the determination of the origin versus mitigation during runtime, right? We might be able to determine quickly when something comes and try to mitigate it on the runtime with automated tools, right? You might be able to bring down parts of the service yourself, right? You might be able to bring down the, let's say again, colonial pipeline, bring down the delivery of gas yourself before it became vulnerability that somebody else could do for the complete ransomware attack. So you might kill it so that you can fix it, for example, but that might happen automatically, right? It might be something that just air gaps the thing and says, no, it's off the internet, it's off whatever, right? We'll go and fix it now, right? That prevents, of course, somebody else taking over the system, right? So that's automatic mitigation or risk management, you could say, right? Like we said before, as opposed to cloud, real hardware cannot fail, right? So this is not a, there is no room for error with this, right? You have to be better every time, no other answer, right? Also, we have to deal with this kind of thing, right? User-level tampering of advice is always a possibility, right? We cannot get too much ahead of that, but we can identify it, right? If something happened and software changed on the device, we could probably identify it, right? We can identify, there's all kinds of technologies for this nowadays, right? Security verification, secure boot, root of trust, you know, signature, whatever you want to call it, there's all kinds of tools for that, right? So we can connect all those things to have a really automated way of managing this problem, right? So signing software, of course, is nothing new, right? The most important part, right, is maintaining trust across the entire software, right? Remember what we said before, we have all the systems that can give us alarms when new vulnerabilities come to the table, right? But we also have to be able to have a very strict, let's say, enforceable model for our device software to always be considered correct, to be trusted, right? So embedded is starting to leverage a lot of those tools. Our move to modular, to containerization, there are several companies that are doing containers for embedded now, are leveraging these kinds of tools. Everything is signed, everything is tested, everything is a root of trust, a very nice chain of verification, right? This is nothing new, but it's actually, it wasn't there before too much, right? At the best, you had some signatures of complete software stacks and things like that, but it was a little, let's say, laser, not everyone did it, right? Simple as that, not everyone did it, right? Not everybody had the skills, the budget, whatever it was, right? To really go that deep into the trust and to maintain in the trust of the entire vertical staff of embedded devices, right? So as we move into containers, well, an immutable software payload is much easy to become deterministic from the point of view of trust. Like it's much easier to validate, to verify if that immutable container or whatever is what you expect it to be, right? But also to identify if that container has any of those vulnerabilities we mentioned before, you're connecting to any of those pipelines, revenue or sneak, any of those guys, then you can identify, right? And you can identify which devices of your device fleet might be susceptible, might be at risk, right? What might be at risk? So we do automatic mitigation, right? Let's see here. So in our company, Pantacor, what we do, I mean, we take a view of defining the entire state with a very deterministic document that just outlines everything that is on the system, right? With that, we can cross-reference with what the sources of those things can be from a container point of view, or kernel trees where it was built, all these kind of things, right? We call this state revisions. It is our answer not to the problem, but it's just how we lay out systems in our world and how we introduce a degree of security and trust to those customers that work with us, right? It is by no means, you know, a one panacea that will fix anything, no, not at all. All I'm saying is that, you know, for the people we work with, well, we try to create the security trust pipelines based on these type of solutions, right? So as we said before, you know, verification of software is nothing new. We take a view that in our world, embedded systems should all be very modular, like software should always be delivered as immutable payloads. Everything can be very measured, very connected one to the other in a pipeline of trust, right? Our technology to achieve this, of course, is called Pantowiser. That's beyond the point. We basically deliver containers and have everything signed from a very nice root of trust. This is just your normal secure boot type of implementation. But because of this, we have a view of what every device in the field is running at each container level. So it can be connected to the systems where the provenance of that software might raise the right flags to help mitigate those problems before they happen, right? So I don't think there's a complete solution to this problem yet. Like the whole embedded market hasn't really landed into we're going to do things this way or that way, but I think, and you can probably see that in the show floor upstairs, there's so much investment in dependency graphing and code insights, right? There's like 15 startups over there that have a lot of money in the pockets, right? And they're trying to solve that problem for the cloud. So let's use that. Let's use that for embedded, right? Let's use the same technology to deploy software and let's apply the same concepts to keep that software secure, to keep those devices secure, right? This is exactly what we were saying just now, right? So vulnerability alarms coming up from your building materials checks, right? The orchestrator identified vulnerable devices, maybe mitigate by turning some parts of the software off, right? Or taking the entire device off the network for a while, right? These are things that we're doing a subset of that. This is more like what is the idea towards the future, right? Right now, the most we do at the moment is flag some of these potentially vulnerable containers and then you pass that on to the security team of the customer, right? It's available for them to do whatever they want, but in my mind, I think, well, we could connect that to much more automated systems, right? We work a lot, for example, with router, with CPs, HomeGateways and basically Telco, right? And like how much money, the amount of money you can save at Telco if you can prevent that from spreading before it happens, right? That's ridiculous, right? I mean, it's really important stuff, right? So we should be much smarter. We should be much better at this. Think about this problem. Have more conversations like this, right? And just, well, find how to start mitigating those problems, right? So embedded can and must address this, like it says over there at the end, right? It is not just the job of the cloud to keep data secure, no, it is the job of embedded to keep those devices that we put so much trust in secure, right? I don't want my house to be compromised. I don't want my gas to go away, right? And that's it, right? So that's actually my final slide. The whole point of this is to trigger a bit of a conversation or thought process in your own heads later, not necessarily now, but just think about it, right? Because it's important, right? There's a lot of work being done that is just, there's a lot of embedded equipment that is still built in the way that it was done 15, 20 years ago, right? It's just kind of throw away or just mash up of things and now those things are getting connected. It's a huge risk, right? We've been seeing that huge risk for the past few years. I don't think we've seen the worst of the backfiring that that creates, right? We've seen some of these as scenarios, but I think we're going to see a lot more, right? Because everybody talks about IoT. Everybody talks about everything getting connected, your house getting super smart, but the truth is that most of you don't have their house full of smart stuff, right? But, you know, the next washing machine you buy two, three years from now will be connected, right? Everything is going to be connected from that point of view. Not necessarily, oh, it's super smart. It has big screens or anything. No, but it's connected. It's sending some telemetry. It's taking updates, right? It may be not connected to the user itself. You don't see it as a super smart device, but it's certainly connected. Your network can be compromised. Everything can be compromised from any one of those entry points, right? So it's a big problem and we have to start thinking about it. So thank you for that. Let's discuss if you guys have any questions or just want to bring ideas to the table, right? Happy to chat. Yep. Well, I mean, so that's the thing. There are two concepts that depend on who you ask. They will give you a different answer. What is IoT or what is embedded, right? You know, the reality embedded is everything that is not general purpose. Let's call it that way, right? And the same applies to IoT. Nobody really knows what IoT is. IoT just means something that is connected. It's my definition of it. If something is connected, you could consider it to be IoT. It might be a little microcontroller-based sensor or it might be a huge server somewhere. I'll call it IoT. It's connected to a network. It's available. It's reachable, right? So to your question there, what I'm talking about here applies more to higher-level stacks, right? You could say the embedded that is bigger, the embedded that is not microcontroller-based, right? So the embedded that is actually embedded Linux-based, right? Things that are a little bit more complex, multi-feature devices, right? Things that are not only fixed function. I was talking about the washing machine, right? Why was I talking about the washing machine? It's a good example. Washing machines 10 years ago or five years ago, three years ago, you know, 99% of them were just very simplistic microcontroller-based systems. Turn this on, turn that off. Some power control, that's it. Next generations are moving a lot of those functions to higher compute capacity devices, things that are actually running full-fledged operating systems, a little embedded Linux board, right? So these problems start to become apparent over there because now the features are sort of being split, right, in ways that it becomes harder and harder to maintain that entire confidence in the system, right? So no, it's not about microcontrollers, although that's important, too, right? But it's a, let's say, it's less complex or less big, right, problem to address if we're talking about a fixed-feature microcontroller-based device, right? Does that answer your question a little bit? Or at least that's my way of thinking. Maybe I'm wrong. Yes, yes, higher-level stuff. I mean, we're talking about higher-level stuff, so we're talking about Linux stuff, right? Embedded Linux stuff, something that can run Linux, but we're not, I'm not talking, or at least what we do at Pentacore, we don't work with higher-level embedded devices, like we're not really into the Raspberry Pi Plus parts of the world. We're rather towards a slightly lower parts of the world. We're talking about devices that are 1632 megabytes of Nor Flash, 64 megabytes of Nan Flash. I mean, the core of what we do is containerization for those low-spec devices, right? But that's the problem segment we have observed. I'm sure there's bigger things out there and maybe other ways to address those problems, but at least in our problem segment, well, that's what it looks like. Any other insights? Or, yeah, that's a fantastic question that maybe somebody else in the audience can answer. I'm not sure. I'm not sure. I mean, if you have a complete compromise at the root of trust of the device, be it a two-device or something, a TPM, whatever, like whatever that root of trust is, well, I don't know if there's an easy way to mitigate that. Maybe somebody else knows. Exactly. And that brings in the place of mitigation ahead of the problem, right? You might flag something that's weird there, cut it off, right, type of mitigation strategies, right? Yeah. Yeah, I mean, so there's a million ways that the problem about security is that it's not a complete problem, right? I mean, it goes in all possible directions. And there's all kinds of technologies out there that help you mitigate parts of this problem. Like Ricardo was saying, I mean, you can do attestation if the problem is a compromise root of trust. You can certainly do that. You can have trust on mediated access to particular memory regions or subsystems or buses inside the system, whatever it might be. That nothing is a panacea, however. I mean, the most important thing is try to identify those vulnerabilities before they're exploited from a central point of view, right? It's not try to identify what has been compromised, what might be vulnerable, right? And then enact some policies to address that. That's also not going to fix everything, right? But it's at the very least important for us as the architects of that sort of next generation, right? To think about that while we architect those next generation devices, right? It's important, right? No, they are in use. I mean, again, it once again depends on the industry. Some industries just don't have the budget for that. As simple as that, low cost stuff. But if you go to higher cost stuff, which is still deeply embedded, then you will start finding roots of trust. I mean, a root of trust is not necessarily a TPM. A TPM is actually a huge component that provides a lot of features, right? But you might have a little TUF type of thing that's just a very steady root of trust, right? I mean, there's all levels of things you can do, right? We are seeing it more pervasive. I mean, it becomes more available, let's say. And just like your colleague next to you was talking about, ARM trust zone is available in most SOCs, right? Yeah, I think I've heard a lot of it. Exactly. It's a solution for trust. Right. But not everybody leverages. We work with a lot of customers that could leverage trust zone, for example, but they don't care, right? Simple as that, right? It falls sometimes to us to try and evangelize a little bit the importance of it, right? But in their mind, it's like, it's okay, even though there's no extra cost. Well, I mean, there's no extra bump cost. There might be an extra software cost, right? For the development or implementation of that. But the embedded world also suffers from the fact that it's very fast-paced. Everybody, everyone always needs to get a new version of their hardware out there, right? So, you know, it's a complicated segment, but the tools are available. And again, it's up to us to educate, right? Those product engineering teams what are potential best practices for the future, right? All right, I think that's it. Let's not take more time from the previous speaker. I think we're five from the time. So, good. Thank you for coming, guys. If you want to know a little bit more about what we do, we can go to booth B33 at the sponsorship floor. Thank you.