 So in summary, from the network, we have some requirements to remote access in IoT. You can only do outbound connections. It needs to be secure and turned. So you need to authenticate both ends of it, both the device and the laptop or server encrypted for confidentiality, because there might be some secrets going through there. And the last one there is not strictly required, I would say, but it's highly desirable that you don't have any open ports of the device. So this is not necessarily a vulnerability, but it's to have more defense in depth that you try not to expose ports directly on the device and load network overhead. Okay, so now maybe we have solved it all, right? But if we go back to... So we have the use cases, transferring file terminal, and we know the network is going to be difficult. But there is one more thing that we need to think about, at least if you're doing it in production and you have customers using it, or you're using it for customers, because what could happen? If we start off like we did, probably it doesn't work, the UI hangs, and now the vendor has remote access. Okay, great, this is solved. But what could he do with it? So if you give somebody full access to all devices, bad things could happen. So you shouldn't necessarily blindly trust the person in the other end, at least if you're the customer. And at least in Europe, there is regulation around this, especially if there's identifying information, personal information on the devices. You can't just let anyone get access to that. So what can we do about this? Well, we need audit logs as well. This will tell you who did what to the device, when, and which devices. Specifically for the remote terminal, it's good to have a session log as well, because you can see all the details, what happened at the device there. The second thing is, if you remember the scenario, the customer will reach out because there's something wrong. So it's not necessary that the vendor has ability to, let's say, log in to the device all the time. It just needs to do it when the customer makes a request. So you can build a system where the customer gives permission to log in to the device, for example. So this remote access capability is not always on, but it's on demand. And then role-based access control. How many of you are familiar with it already? Or, yeah, somewhat, two, three people? Okay. So what this means is that you give different permissions to different roles. Mostly on the server's side is where this is applied. So, for example, in Mender's case, Mender can do, let's say, both over-the-air updates and remote access. But you don't need one person who doesn't necessarily need to have the permission to do both of those things. So you can have a support team that can do only remote access to some devices. And then you can have a production operation team that only can do over-the-air updates, but not remote access. And the last thing is maybe a bit obvious, but you can restrict the permissions also on the device side. So it's not like a route permission, but it could be limited to the application runtime environment, for example. Okay. So if we summarize it, the vendor wants to do terminal, file transfer, port forward. The network limits us to do only outbound connections. We need to have end-to-end security and use very little bandwidth ideally. And from a security point of view, we need to audit logs, access approval, R-back, and device side restrictions. So as usual, once you start looking a little bit deeper into a problem like remote access, it snowballs a little bit. Okay. The last part is how can we actually address this? And it's quite a lot, so we can start with the two most difficult things. So if you want a terminal and you can only do outbound connections, how is that possible? Because once you have solved that, then you can look at the other requirements and you can build on top of, yeah, the initial solution. Okay. So we interviewed some users how it's being done in the field today, getting a terminal access to a device, IoT device. And as you can see, VPN and SSH is a very common way to do it. And there are a few other alternatives to like raw socket, web socket, reverse SSH. This is a product. I don't have any relation to it. It was just part of the survey response, remote it. So I'll go through all of these different alternatives and we'll take a bit, look at what the trade-offs are here. Okay. So start with VPN, which is apparently the most popular way to do it. So this will, as you probably are familiar with, allow for inbound connections over an overlay network, the VPN network. So you get rid of this requirement saying you can only do outbound connections because now you're on, you can do inbound connections over VPN. It's very widely used, at least in cloud and desktop environments. And there are several implementations of it. This one is quite new wire guard. I don't know if you heard about it, but it's a bit simpler implementation in open source. It's also has some kernel module, I think. But you also have the good old open VPN, which is a bit more tricky to configure. I've fought it a few times. Then reverse SSH. The way this works is illustrated here. So first you will, on the device, you will create a connection. Yeah. So the red, first of all, the red line here means that you cannot go to the device, but you can go the other way around. So the device can connect to a public server over SSH, so it sets up a persistent tunnel. And then the second step is that on your laptop or where you want to access the terminal on the device, you create SSH connection to the server, which is tunneled through to the device. So it's actually a, yeah. So you get a port on the server that represents one specific device. And it's kind of like a tunnel in a tunnel approach. If you look at VPN, you compare it to it, it's quite easy to set up. You just need to run some SSH commands, so that's not very difficult. But it's harder to maintain. Because now you have all these ports on this public server, you need to keep track of which port is for which device. There's credentials. You have on the devices and the server and the laptop that you need to manage. And you need to make sure that this tunnel is always open as well. So there's quite a few things that can and will go wrong here if you try to use it at a bit larger scale. So I would say only use it for test environments or pilot environments. The third option is a raw socket. So this is quite simple conceptually. So basically what you would do is develop an application on the device that connects to a server somewhere. And then the server will keep this connection alive and it's persistent. And then when somebody wants to use a terminal, it would hand off the terminal to this already open connection to the right device, basically. Of course, TCP is not secured natively, let's say. So you would use something like TLS to handle security requirements. And this is a custom completely homegrown implementation. And there is actually a better alternative, so I would not use this either. And we'll get to that now, which is WebSocket. I want to survey again. How many have heard about WebSockets? Two, three. Okay, so about 15%, maybe. Okay, so this is a specification from 2011, so it's getting a 10-year anniversary. It has, I'll get into what it does, but it has very wide client and server support already, so your cell phone, web browsers, application environments, Goal and Python, all support WebSocket. What it does is that it starts with the HTTP connection, so regular HTTP, I'm sure you're all familiar with that. But then the client will request a connection upgrade, as it's called in the HTTP, yeah, as part of the header, I believe, in the HTTP connection. And the server would open, yeah, upgrade it and create a bidirectional and full-duplex channel. And one benefit with this is that, since you're basing it on HTTP, it will go quite easily through firewalls, but you also have other benefits of using a standard, which is that you can, it supports HTTP proxy and reverse proxies as well, so this is more standardized than the raw socket approach. And in addition to this, it will take care of connection management for you, so mainly this is health check and keep alive to make sure that the connection is always open and living basically. The last option, I think, is off-the-shelf solutions for remote access, and these have good use case coverage, so they will generally cover most of what's needed here, at least the features the user wants to do, maybe not all the security features, but the functional ones are covered, and typically more, it usually does more than this as well. There are a few examples. The plan is not to go through and compare off-the-shelf solutions, but this one, as mentioned already, it's a hosted or SaaS solution, yeah, and they're also had a troubleshoot add-on package, and there are lots others, if you just Google for remote access IoT, I'm sure you'll find quite a bit. MQTT, so you cannot talk about IoT without talking about MQTT, so this is a protocol all the big cloud vendors support. What it is, it's a messaging protocol that uses publish, subscribe, so could we use this? So the key thing here is the publish, subscribe aspect, so if you compare that to WebSocket, WebSocket is just a transport mechanism, so you can actually build MQTT on top of WebSocket, too, just to make it fun. And probably you do not need publish, subscribe for terminal, unless you want to do some kind of parallel terminal or something like that, but that's probably more than you need. So now we covered terminal outbound connections, and these other use cases, you can generally build on top of the alternatives that we discussed already, and I'll go through how they are covered by the different options as well. And where we are right now, I would discard reverse SSH and the raw socket, and so we're left with VPN and WebSocket off the shelf, but I'm not going to discuss off the shelf solutions. So if we look at VPN and SSH, how far can you go? And you can go quite far, actually. So you obviously get terminal with SSH, file transfer, SAP, port forwarding SSH. It deals with outbound connections because you have a VPN. It will work. And turn security, you would think it is secure, and it is if it's used correctly. What you do get is an open port on the device because SSH needs to expose a port, which might not be a problem if you're making sure that it's bound to the right interface and you don't make any configuration mistakes, but you also have more credentials to manage because now you have VPN credentials and SSH credentials as well because it's kind of this tunnel in a tunnel approach. The bandwidth is also a little bit higher because of this. You have two encrypted tunnels, basically, to leverage the terminal. You do not get audit logs, access approval, RBAC out of the box. You can build it. Some of it, like RBAC, is probably quite difficult. You need some kind of server infrastructure around it to manage this. And of course, you can do device side restrictions because in SSH, you can do this, allow users stanza to only be able to log into one specific restricted user account. I wanted to take a look at the web socket implementation, too, and this is what we use in Mender. So at least you can see this is possible. We're very happy with it, by the way. So we can do terminal access over the web socket. It's not a problem. File transfer port forward as well. And output connection, not a problem because we're initiating HTTP from the device itself. And there is no open ports. And for Mender, it's reusing the OTA credentials. In any case, you wouldn't have two pairs of credentials here. You need a TLS certificate, but you don't need both TLS and SSH. The bandwidth is very low with the terminal because you're just sending a character and receiving it on the other end. And then the rest of the story is similar to VPN that if you want audit logs, access approval, RBAC, this is something you need to build on top of it from the server side. Device side restrictions you can do. And just how would it look like if you were to implement it using web socket? This is how we have done it. I tried to generalize the names a little bit, but this is how you can implement web socket terminal. So it starts with the device there. So there's some service that always needs to run on the device. I called it connect. It's called Mender Connected Mender, but its purpose is just to keep that one connection alive. And then, yeah, it sends the HTTP and the connection upgrade and it connects to a service on the server. I call it device connect. So this is the counterpart to the connect service, so they just make sure the connection is alive, which is not very difficult because you're using web socket and it has this built in. On the other hand, you have a laptop and even a web user interface, so a web terminal that wants to connect. One thing I wanted to point out is that since you're using web socket here, a web UI is also very easy to implement because, as I mentioned, web socket is a standard protocol and all the browsers understand it. So you can access it from the browser fairly easily. But in either case, you will send characters. It will connect to this device connect again. And there is one last service here used that it's a message queue that basically circuits those two, so the device and the user using the web UI or terminal. It basically just wires them together. That's the point of this message queue. We use something called NATS, which is a very simple message queue, but you could probably use many others. And then the last thing you can think about, so if you need it and you have some device out there already or some infrastructure, if you have a VPN connection, obviously it's quite straightforward to add SSH. But keep in mind the security nuances here, whereas do you open a port and which credentials are used for SSH? If you have MQTT already, which is a very popular protocol in IoT, you can build terminal on top for sure. I don't know about port forwarding. I haven't tried it and we haven't experimented with that. But for sure you can build terminal. If you have TLS keys, you can implement your own web socket or you can use some off-the-shelf solution. If you have nothing, I would definitely use off-the-shelf solution. And I think I would recommend that you reuse an existing solution. And the reason I'm saying that is because we built a remote access solution and it takes a while, even with a team of developers, getting all this right is going to take a lot of time and also maintaining it as maintaining software is more effort than building the first version. And think through what you have already. Maybe it's easy to add it to what you have. But consider an off-the-shelf solution. Make sure it's built for IoT. Again, if you remember all the requirements around the network and how this can be different from a laptop, desktop environment and so on, make sure it's built for IoT. And rather do what you think is fun to make a product, right? With that, I hope you learned something, found it a little bit interesting, and thank you for your time. And do you have any questions? Feel free to ask them now. Yeah, so why is port forwarding a requirement? So this is a use case. Typically by the end user that you want to be able to connect to a service that runs on the device. So for example, there could be like a web portal. So you have an IoT device, and it has a web portal where you can configure it. But you cannot access. This is only accessible on the local network. But as a technical support staff, you want to see what that device portal looks like, how it's configured. So that's why you would use port forwarding. So you can connect remotely to that admin portal, for example. So that's one use case for port forwarding. If that answered your question, or kind of. Yeah, so if you're using SSH, you get port forwarding for free. So that's included in SSH. But if you use something like WebSocket, it's not, then you need to implement it. Any other questions, or yes? Yes, so the question is, if there are any other questions that you would like to ask, there is a need to manage a fleet of devices or many devices at the same time. I'm repeating the question because we have a live audience as well. But yes, so definitely that's a need. But not for remote access, I think. But for what I would call overyear updates or software updates, which is what Mander does. We do support grouping of devices and deploying to fleets and automatic rollouts for deploying new software. But that is a little bit different use case because in that case, you want to just get the software to all the devices at the same time. So that's overyear updates. But for remote access, you have a problem with a specific device typically when you're doing remote access. That's when you want to log into it or the customer has some problems. So that's why you usually, in a remote access context, you usually just want to do one device. We did ask some questions around there as well when we did the user interviews, but having a terminal open to many devices at the same time, it wasn't very interesting. It's technically quite interesting, but yeah, from a user need point of view, it wasn't that important. Any other questions? Yes, I can repeat it. So it's mainly around maintenance, sorry, the question is what are the downside of reverse SSH and the main, they revolve around maintaining the connection and maintaining the keys and the credentials. Because you will have different ports open on a public server that will correspond to one device. So you can keep track of that mapping, but then you would have keys on the device and on that public server. And you also need keys for the users that set up that reverse SSH. So there's quite a lot to manage, and also making sure all the SSH connections are open as well. So it works, but if you're going to scale it up, it will be more effort and more cost probably than, for example, VPN. And you had another question? Yes. Got it. The question is we built on top of WebSocket and what are the security considerations we had to make. For us, it was quite easy because we had Mender to do over-the-air software updates. We already had keys on both ends, and we already used TLS to set up connections between the client and the server. So for us, the infrastructure was in place, but it's very important that you have unique keys on each device and that you only do HTTPS connections. Because if you do plain HTTP, as you know, people can listen, and if you do this with a terminal, it is even worse because then you could have somebody. If I have a device here and I do plain HTTP, you could sit there and you can see what I'm doing, but you can also change, you can run commands on the device as well. So make sure it's authenticated encrypted, use TLS, I don't see any reason to use anything else for security, and do connections, don't have any open ports on the device. So the question is, is there a way to manage the unique keys on the devices? Because this can get quite complicated, and you're definitely right, this is complicated. There are many approaches to it, but I think it's another talk, maybe. There are some, you can get hardware support for it, so NXP and many other board vendors, Toradex too, I'm sure, have TPMs that you can pre-program with keys, so you don't have to worry about them in your software. You just use what's in the hardware, and then you get the public components you can get from the board manufacturer as well to load it into your server, so that way all the devices will have unique keys in hardware and you know that they're authentic because you got it from the vendor, the hardware vendor on the server side. It's going to cost a little bit more because you need more hardware support, but that I think I hope is the future of this key management because it's a very complicated problem for sure, so that's why most people, they start out with just having one shared key across all the devices until something goes wrong, somebody steals that one key and now they have access to all the devices, so yeah, I'm hoping hardware support will solve this problem, but until then we have to live with it, unfortunately. Any other question? Yeah. So the question is if we profile WebSockets versus the other alternatives, and the answer is I am not sure, but it is very light that I know. It doesn't, I think we're talking about bytes per ten minutes or something like that just to keep the connection alive. If you're not sending anything, it's a very low overhead and definitely lower than SSH, for example. Sure. Any other questions? Okay. Thank you, everyone. Good to see you in person and to all of you viewing as well. Enjoy the rest of the show.