 Welcome everyone to this panel on Edge Device Security. I'm your host, Frédéric Débien from the Eclipse Foundation. I'm program manager there for IoT and Edge Computing. Our goal today is to have a wide-ranging conversation about both the challenges and the potential solutions to Edge Device Security. With me, I've got four fantastic guests in driving this conversation forward. So let's introduce them. First is Dianne. Hello, hey, my name is Dianne. I work as an engineer for Red Hat. I've been in the IoT space for the last five years or so and been drawn into the Edge Computing in the last couple of years. Excellent. Next is Angelo. Okay, thank you, Frédéric. So Angelo Corsaro, CTO at ADLINK. Within Eclipse, I lead quite a few project in the context of Eclipse Edge Native and Eclipse IoT between those and I've been involved with Edge and for computing since the very beginning. In fact, I was involved in some of the super early project in this context. So it's a pleasure to be with you today. Thank you, Angelo. Nicola. Thank you, Frédéric. I'm Nicola LaGloria from Kinetics. My company is members of the Eclipse Foundation and I'm a member of the Edge Working Group. Our company is actually in the OS operating system, embedded device operating system space and we are also contributors to a couple of projects inside the Eclipse Foundation. Thank you. And last but not least is Ted. Hi, my name is Ted Ross. I'm an engineer out of the Boston area. I'm actually freelancing now, working on some open source technologies around the Scupper project, Apache Cupid Dispatch Rubber project. And my interest is not so much IoT but more cloud networking and Edge Computing and the networking involved in Edge Computing. Thank you very much. All right. So first segment, let's start talking in a general setting about the challenges of Edge security. So, Dion, what are the Edge device security challenges and in what way are they different from those of traditional IT? Well, one of the big differences that we have when we talk about Edge Computing and differences between the cloud computing is that, you know, our hardware is much more geographically dispersed now, right? Being the clusters that have deployed outside of the data centers or just a single node of compute, we have devices that are not physically protected. So that stems up a lot of questions about security, ranging from all the layers of security there. So starting with, can you trust that hardware is proper, right? Trusting the hardware layer, trusting the OS and firmware levels of those devices. And, you know, thinking about if someone, you know, tempered with the software on the hardware. And so, you know, going from the hardware to the firmware and operating systems all the way to the microservices and applications running on that hardware. So you can see on the application level, are we, you know, running the container images that are supposed to be run there? How are we, you know, sharing the secrets to those unprotected locations? And then, you know, do we have a mechanism for those microservices to be restricted in their own, in their own, so to say sandboxes and not interfere with each other? And that makes sense. And at the same time, I mean, okay, there are many, many of the challenges that we just discussed that are specific to the device. But all of those devices ultimately will speak to the cloud at some point, right? Those are not self-contained systems. So what role could we say that the cloud plays in edge device security? So in my point of view, so one thing that I didn't mention at first is that, you know, there are also devices connecting to the edge infrastructure, right? That we are not even not considered part of the edge. And what we need to provide is like a single continuously experienced for all that. And that's where the cloud plays a massive role. So we need to have, if you want to trust devices that are connected, we need to have a proper device management and device provisioning of those devices. If you want to handle properly operating system and firmware, we need to have a proper rollout of the framework to a large number of external nodes. And of course, we need to have a trusted way to distribute our applications, microservices, container images and secrets to those locations. So that's all the job of the cloud to provide that infrastructure to basically manage all that in a scalable way. And obviously at the Eclipse Foundation, we love open source. We work in open source all the time. So are there specific open source projects that address the challenges that you just told us about? Are there things that people can download straight away and start building a secure edge infrastructure with it? Yeah. So there is, but my experience for the last six years doing IoT and Edge, so there's hard to find a single solution to all the problems, right? So you have all these pieces dispersed to different projects trying to solve different things. So for example, Eclipse Holo contains a concept of device registry, right? And managing of credentials and for secure connectivity of devices to the cloud or to the Edge infrastructure. Eclipse Hobbit is another project that deals with software rollouts, updates of firmware and basically any kind of software. But we also have things outside of the Eclipse community as well. For example, the Harbor project is something that tries to provide a way to be able to have like a distributed container registry, which can have a really great place into the Edge computing with Docker-based and container-based technologies. Okay, that makes sense. And obviously our Edge computing projects in the Edge native working group have a deep concern about obviously security. So maybe, Angelo, since you're involved in one of those, what's your perspective on this? Well, so I think one of the aspects that we have to integrate when we talk about security in the Edge application is that in the end, yes, the cloud will have a role. But if you really want to scale, we need to make sure that, in a way, we have a change of paradigm and I think into respect. So if we focus for a moment on data, if you think about how we have secured data so far, usually what you do, you build security around it, right? So you make sure that you can't access data, that you don't have the rights to access. That's easy on a cloud environment, easy between quotes. But if you think about taking that approach and moving it to the Edge, in which all of a sudden you have data that is decentralized, well, it's much easier to tamper with devices. So that's not going to work. So all of a sudden, I think, and at least that's the direction we are taking also in some of our open source project at Eclipse is that we have to change the paradigm and all of a sudden, actually, I agree that devices are out there, people might get all of the device, tamper the device and access the data. So the key point is that with that data, they shouldn't be able to do anything. So we are trying to change completely the paradigm for what concerns security at least at the data level. And making sure that instead of assuming and making it hard for you to access, it's making it hard for you to do anything unless you have the right identity credential and so on and so forth. So that poses quite a few challenges. But again, I mean, up to when there will be interesting challenges, there will be interesting problems to solve, which is good. That poses challenges not just from the security perspective, but also from the encryption, which all of a sudden requires some support from hardware and poses also challenges with respect to secret, being able to maintain secret in a secure manner, which as we know is supported by TPM and processor architecture like ARM and trans zones provide also some additional way to deal with it. Especially if you look in combination with some interesting Unikernels that are also coming out in the open source. Yeah, absolutely. And at the same time, yeah, this operating system aspect is important and Nicola, you mentioned you are working on embedded OSs for edge devices. So taking on from Angelo there, what's your perspective on the edge security challenges? Looking at from an embedded operating system standpoint that is where my focus is, basically we have two really great technologies that kicked in in the past 15, 20 years. One of course is virtualization and another one is related to containers. Both of them from a security standpoint, they provide what today is one of the most important aspects in my opinion in an embedded device like an edge node in the edge architecture that is the isolation, the context of isolation, how we do isolate some different aspects that are running on an edge node. Virtual machines and containers are really different. Virtual machine is an obstruction of a complete computing platform, hardware and software. So IO, processor, memory. A container is purely software and you use the same kernel of the host. Actually, a container is a child process of the demo that launched. So there are completely different technologies but at the same time they guarantee what isolation needs to be on a device level. The interesting thing about isolation is critical processes. So let's say that we have an edge device, an edge node and we have a camera that is attached to the edge node. This is a simple case but again we can see that these kind of use cases in two different ways. For instance, from a container perspective we have of course the core OS and then we have a containerized OS and the containerized OS, let's say, is running a software for grabbing the camera frames and then do something with it and then transfer those information somewhere else. The thing is we need how we do enable the container to access some devices, let's say the device, the camera device and then process and then do the job. Well, we can do that in a pretty fairly secure way without a privileged node, without having any container running in privileged node just because we know exactly which device we need to export from the core OS to the container. At the same time, if this is not really guaranteed let's say that we have other devices that are managed by Udev and so we don't have a predicted way to tell which device is generated by the core OS and needs to be connected to the container. We may have some problem over there and there are several recipes to solve that problem but they may violate the isolation principle and also having privileged containers is not really a good practice. So it's about good practices. Virtual machine on the other side, like an extremely solid way to isolate critical application. Let's say that we have an edge device that has some particular important characteristics. Let's say that part of the system are in real time. Part of the system don't have any time constraints like regular operating system or part of the system needs to be bare metal. How can we solve this problem? Containers are great in edge because they provide a great deployment model if you have microservices and some software that is purely software that needs to be deployed uniquely, atomically inside otherwise without worrying about dependencies. But at the same time, how can we enable the same easy way of managing deployments when we have a lot of hardware that is critical hardware attached to the edge now? So in this case, probably virtualization. When I say virtualization, because even Docker is virtualization, it's called lightweight virtualization. But something that is more appropriate for this use case that is the regular virtualization or asymmetric multiprocessing which guarantees the isolation of critical application and at the same time, they allow the edge node to do and to perform really critical aspects related to critical IEO and other critical devices attached to it. That makes sense. And we will revisit obviously the topic of the devices virtualization container a bit later. And one thing I really understand is important from what you said, Nicola, is isolation is one critical thing to think about isolation of processors, resources, even hardware resources. And at the same time, edge nodes, edge infrastructure are connected devices and networking is especially important in everything that we do at the edge. Whatever we're talking about 5G, dash 7, there are one, et cetera. And so my question to you, Ted, since networking is really your bread and butter, so what role does the network play in edge device security? Yeah, that's actually kind of a funny question, because one of the main roles that the network plays in security is to be the source of the problem, right? It's the thing that makes it... Obviously the network is extremely critical and important for what you want to do, but it's also the channel through which abuse, through which intrusion, through which illegal access can occur. So it's kind of an interesting thing. So the network is a double-edged sword, it's providing a lot of the problems with regard to security. So it's very important that the network be set up in such a way that you are making it as difficult as possible to exploit it in ways that you didn't anticipate. All right, and now, that's certainly a good starting point. At the same time, I was wondering, I'm not a networking expert, but I hear a lot about programmable networks, software-defined networks, and a lot of stuff. Are they really useful at the edge, all of those securities, do they introduce more vulnerabilities? And if they are useful, why? Okay, that's also an interesting question, because there's a lot of talk about software-defined networking, there's actually a lot of open-source activity around SDN. In my opinion, SDN is highly complicated and it addresses issues at a layer of abstraction that may not be as helpful as we want. Typically, SDN is really giving you software APIs that access things like MAC addresses, IP addresses, host addresses, ports, routing rules, et cetera. So what I actually think we need is, and this has been stated before, like a different paradigm, but I think we need better abstractions for networking in the edge, in edge computing and in basically cloud-native computing in general. Okay, and that in one way really not necessarily vindicates, but matches my expectation in the sense that as a non-networking expert, I had a feeling from the outside that SDN and stuff can be useful, but yes, it's something that still needs maturing and still needs to progress, to be useful, and at the same time, I know that a few of our members are leveraging or involved in such projects like OpenNAS and OpenSource Mano and ONAP and that kind of stuff. Now, if we take a step back, obviously, as I mentioned, OpenSource is really important to us. So how can OpenSource help improve edge network security? You've been involved in such projects and especially Scopper is a good example of that. So how can OpenSource really help there? Yeah, let me talk a little bit about that because I mentioned before that we're looking for better abstractions. And let me talk a little bit about the weakness of some of the abstractions that we use now. So anybody who's doing any networking at all really knows that when you're talking about addressing things, you deal with a host address and a port. So the host address can be an IP address, but more commonly it's a name that maps to an IP address, which gets you to the host and the port tells you which process on that host you want to talk to. And of course, in the cloud native world, we're talking about things like serverless or we're talking about edges where we may have many edges or many devices that we want to address as a class. The host really isn't that interesting. In fact, we don't want to be bothered with that notion. The other thing I should mention is that the internet as we know it now grew up with client server computing. So client server architecture is what the internet is designed for. So it's really built around literally hundreds of millions of private networks that then connect up into a single public network. So this vertical orientation of the network is very well proven and it works very well and it's very highly performant. But it doesn't lend itself very well to edge computing where we really might want to have some horizontal connectivity. We might want different edges to be able to easily address each other. And that's not done very well at the IP host level. So there is quite a bit, in fact, I think almost all the really interesting work that's going on in this space is happening in the open source communities. So there's the scupper.io project that I'm working on. It's actually also based on the Apache Cupid dispatch router. So Apache Cupid is a collection of projects, one of which is called Cupid dispatch router, which is a high level networking router that works with the AMQP protocol and a whole different set of addressing. So instead of addressing hosts in a large network, you're addressing processes. And so instead of providing host access to a hacker, for example, you deny that access, but you can provide access to specific processes. And it allows you to do things like multicast load balancing in a wide area. But these are not necessarily security related, but it does have security aspects to it. I'll also mention that the Eclipse Iofog project is also using the same basic technology to good effect as well. And their focus is a little bit more on small devices in the edge and connectivity involving the small devices on the edge. That makes sense. So it's certainly really interesting what you mentioned, Ted. And one question I would have for you is we discussed in our edge native working group community meetings various scenarios for device quarantine in the sense that a compromised device could self-quarantine or maybe the network could quarantine the device. So in your opinion, is self-quarantine more useful than network quarantine or the author or both are useful strategies in your opinion? I'm not sure exactly how to answer that question. I think probably both that you would need if a device knows that it's compromised, it will also take itself out. If you know that the device is compromised, you'd like to take it out. So I think both are important and I think that somebody else might be more of an expert to answer that question than I. That makes sense. With the same question to you, Dehan, since you mentioned several device management technologies that we have at Eclipse and especially that's a concern for Eclipse, I'll know that you are working on. So what's your take on device quarantine there? Well, from the perspective of Eclipse phone the device is outside of the cloud. So we are thinking about just the connectivity from the cloud but if you take a look at globally from the head perspective, quarantine microservices definitely does make sense. So limit them in the amount of CPU they can use or resources they can use. Then on the networking layer, what you mentioned is also making sure that they can't do any arbitrary outbound connections and dial-ins to different places. That's all, I think, very important aspects of running microservices at the edge. Excellent, thank you. And now in the interest of time, going forward with Anuture Topic, and this is something we started already discussing with Nikola embedded devices, virtualization containers, and we already established this, the different security roles that virtual machines and containers can play now. When we think about embedded and mobile devices, Nikola, they are increasingly seen as edge devices and I know that some of our members at some point were playing with the idea, hey, I have someone in a factory with its tablet and maybe we can delegate some edge workloads to the tablet that the employee is having and some of the other stuff. But anyway, if we take a more global perspective about embedded and mobile devices as edge devices, what are the security challenges that you've seen in your work that you think are specific to them? From an engineering perspective, I don't think there is any difference between a mobile device and an embedded device. My phone is an RM64 which shares many of the IEO of my embedded board I'm working on doing an embedded custom operating system. So let's talk about, they almost are the same hardware, but there is something that makes a difference between the two of them, is what you want to attach to them. So in the case of a mobile phone, so let's say that in the future, in industry 5.0, we have workers walking around a factory with their phone with their tablet that is an edge node. In that context, because I don't see those devices to be connected to a special machine or some special hardware, I guess that the container technology is what provides this extremely important way to deploy software in one shot. I remember Linus Dalgott say something really interesting about package managing and he said if all the software was statically compiled, we don't have problems or dependencies anymore and so we don't have to go through a lot of resolution issues when we want to transfer software to different devices. So containers brought that concept on a different context, but still they bring that value on a let's deploy something that is consistent and I can distribute those resources across multiple devices and this is great, but again, in my opinion in a real use case scenario inside a factory we have some of those embedded devices that needs to command instruct other machines in the pipeline and this is where actually technologies may be not anymore suitable for doing the same thing. So from a security perspective again for me is how can I prevent a critical process that's control a special machine that critical process needs to be protected needs to be again isolated from the outside. What if a microservice is doing something bad to the CPU there is a DOS attack so there is like a denial of service there is some sort of overflow how can I actually make the machine that is operating safe in that context and try to isolate and shut down whatever is going wrong in a particular partition of the system and again here embedded system are really complex really complex beast because we have multiple core every core can host an operating system it's called like again electric multiprocessing so if I have different cores that address different OS's I may have like a supervised or unsupervised architecture but at the same time in those two different scenarios I have a complete control of which resources are located from another perspective to the specific task well at that point I can play you with virtual machines and I can deploy containers in virtual machines but at the same time I'm guaranteeing that no processes are interfering with other processes so in the case of a tablet in the case of a phone I guess we can pretty much be okay on a very effective deployment model with containers and so I can really have a backplane and this backplane decides what needs to be distributed on the edges and that's cool but at the same time we need to keep an eye on which resources physically are addressed when something is computing on the machine when something is going on on the edge devices and this is where we can play with both of them we can create hybrid systems we can really do a lot of interesting things really work with fantasy and of course with adherence to the reality not really invent weird stuff but again in that context the open source is providing a lot of inputs again it's more about best practices in my opinion but again we have open source technologies for embedded computers and the arm right now is now really important inside the virtualization context Zen that is probably the most important today virtualization technology on ARM computers sorry ARM processors architectures it's an open source technology and it works really well not everywhere not in every platform but of course it's a good baseline Docker is the containerized technology that is working everywhere in the world and other also technologies I like exploring different technologies about virtualization I like monolithic virtualization this is what I like the most but it's very tightly hardware Xvisor is a beautiful project out there for doing monolithic custom hypervisors on embedded devices so again how we illustrate this is driven by probably good sense and best practices but out there there are so many different things that you can play with to just invent your own architecture that may save tablets and computer and regular like mobile devices and also critical devices where something is really important needs to be preserved and take care of devices and going back a bit on the networking team I intended to ask all of you let's say you have to deploy well an edge a solution that involves edge computing so is it better to run that on the same network as everything else let's say you are in a factory, you have edge nodes you have robots so you connect all of that on the single network that is connected to the IT network of the organization or in a separate one or some hybrid model what would be your topology of choice there to ensure that things are secure well I mean first question is what is your topology and technology of choice to make sure that things work right and if you are in a factory our experience is that you can essentially forget wifi so for anything that moves around or needs connectivity and for which you don't have already a dedicated network the direction we have been taking the past few years is obviously industrial 4G and going forward we just see one technology for all wireless one dominant technology for wireless communication inside the factory which is 5G as you know 5G was designed first and foremost for facilitating certain enterprise use cases and that's where it will allow us to innovate to the most 5G will allow also potentially properly configured and used by the higher level software to do some millisecond real time which is interesting for quite a few industrial applications and will facilitate perhaps the connection between the portion of the if you are on a factory floor the machine that necessarily will have to run either on TSN when they get there or as of today continue to run on field bus or similar technologies and then the world of IT that will remain on more or less Ethernet based solution so as you know the conversions that had been envisioned was really with TSN the complexity that TSN poses is that you have to pass you have to upgrade some of the hardware and you have a wire so what we are seeing actually on the on the market is that in several instances there is a preference toward converging toward 5G and retaining some of the field bus as they are and then integrate the rest out of that so the concept of a sort of gateway that plays a role of a bridge between the whole stuff, the mud bus, the backnet that you've got on the floor versus the rest then what I think is the most promising in terms of security and let's say electromagnetic interference or tolerance to electromagnetic interference is LIFI so LIFI maybe some of you know is never used in Hollywood it has been so far too expensive to be used in industrial but what is interesting is that you have no interference with a robot arm that moves and in terms of security or if I close window nothing gets out of it because it's based on light so we are not there yet but for me that's a super interesting future future direction to follow closely and certainly something certainly something to keep an eye on and continuing with you Angelou so we talked a lot about networking we talked a lot about devices so what I would try to get a bit deeper about with you hardware versus software centralized versus decentralized when we consider edge device security we talked about some hardware aspects some software aspects but is it really more of a hardware or a software issue in your opinion well from what I've said it's actually both right because in fact there is hardware, software and network that's really the three because the hardware has to provide us with some basic capability right in some cases in terms of proper either acceleration or computation capability for doing encryption the ability of storing secrets securely and in such a way that it's tamper proof but then once the hardware provides that and we have a solution for that today the software has to leverage it okay and that's where sometimes we have issued today the communication protocol we use as he was mentioned by that before there are lots of things that are not quite right for the use case we want to address so he mentioned a few interesting problems but the underlying problem as he mentions that we need to move from host centricity to name data and as you know we have also some very naughty project in Eclipse IoT and Edge like Zeno which is a name data networking protocol and if you start reasoning about that networks becomes important because the way in which you designed your protocol makes some attack harder right makes load balancing easier makes security easier and makes the combination of decentralization and security easier so I see really hardware as providing some of the basic prerequisite that we need in order to be able to build security but then we need to use those to properly implement security within the software that runs on it at the various level and in the network and one of the important I would say element that is not, I mean it's a facilitator but which we shouldn't underestimate is also the programming language that we use as you know we've always been kind of orthodox with that but we see for instance the rays of rust a good example of programming language that provides you with a good set of let's say invariance that make it easier to build secure software that was one of the reason why also Mozilla introduced it and I think that we have to in a way in our development process to start being a little bit more cautious of all the decision we take down to the programming language that we use potentially certainly that has a big impact and at the same time one important thread that we've seen in the last few years is the emergence of open source hardware you know as a driver for innovation so at the Eclipse Foundation we work with the open hardware group that essentially takes the open source risk 5 instruction set and makes open source processor designs and they are built in the open by virus partners and all of that so the link and you do hardware so how does open source hardware factor in your vision for edge device security? I think open source hardware has helped a lot in terms of accelerating innovation I mean let me give an example right I have an example of open source hardware here not sure if all of you are familiar with this card right so that's a super cool example of co-design right because they've designed this board in a hackathon it uses ARM processor and it's designed for Zephyr which is one of the us we use so for me open source hardware it's interesting because it allows us to do some level of co-design with software and to get in a way you know platform that all of a sudden are very accessible and allow people to experiment and then we can leverage further experience to eventually produce the hardware that maybe is not open source or not necessary and can address the more extreme use cases where you need you know either extended temperature or some specific certification which are required either for security or for safety on some environment imagine an edge server that control the high speed lines in trains right but I think overall as it was done for software open source hardware as an innovation as a concerns of the fact that more people get a chance to review it, test it it helps with security for sure but I think first and foremost where it has helped is in accelerating innovation and the level at which people can experiment with integrated platforms like for instance what I just showed you I think that an interesting thing that Angelo said is about driving innovation today when we look at cheap manufacturer there is a lot of proprietary things that are going on in there many of the things that may be possible are not possible because the cheap manufacturer is not ready yet for something so when you have to adopt let's say a microprocessor on a product you got to also understand what the manufacturer is is giving you to support that processor so let's say that you for instance you choose like a provider and you have a basic BSP that basic BSP which is a lot of code and a lot of things that the manufacturer developed because he has the IP on the processor that he does manufacture it binds you from the beginning to something that may be or may be okay but maybe not so I think that risk 5 and the open hardware community is just breaking that barrier finally we may have a system that is fully transparent since the inception so from a security perspective let's talk about the ARM trust zone so today trust zone is a really nice way to save secrets inside an embedded system but again the implementation the really raw and low level implementation of the ARM trust zone in a real product when I buy a song module it's driven by the cheap manufacturer if I don't have a reference implementation of trust zone in that particular context I cannot use trust zone so again open hardware as I really enjoy so much following what's going on in that space because finally there is more democracy like in software has been like for 30 years now it's time for that same democracy to be on a hardware level providing like a great transparency and a great innovation and I agree with this is absolutely groundbreaking and we see this open hardware and open source model even in the networking space I think although that's not a space I follow a lot but I know there are open source designs for network switches and specialized open source wedges that you can run on that so the potential for improvement and openness is still increasing as we go forward as an industry unfortunately that's all the time that we have but really it's been I think an enlightening conversation and certainly one that I hope that we will be able to continue in further panels and further discussions over time so Angelo, Dihan, Nicola, Ted thank you so much for being with us and discussing this and you all in the audience if you liked what you heard and like to exchange with our panelists today well please join us in our community meetings for the Edge Native Working Group we are an open community welcoming to everyone really willing to exchange ideas and discuss the state of the industry and drive innovation in open source so please join us and we'll be happy to continue to discuss everything that we discussed today and even more with you so thank you for being with us today, my name is Frédéric Deviens, Program Manager for IoT and Edge Computing at the Eclipse Foundation and this was our panel on Edge Device Security so everyone thank you again and I hope to talk to you soon