 Thanks, everyone, for joining today. My name is Pedro Oliveira. I'm a solutions architect at SpectroCloud, and I'm joined here by my good friend Ithik from Tevel. And we'll be talking to you guys about an awesome, if you ask me, use case about Kubernetes at the edge for smart agriculture. So Ithik, take it away. Hello, everybody. You hear me? My name is Ithik, the same person, Pedro. I'm from Tevel, Israel. He started up. And what we are doing in Tevel, we pick fruits with flying autonomous robots. Have a look. This is our flying autonomous robots, connected via power supply and networking cable to essential units that process in the sensors and with sophisticated AI, searching, navigating, detecting the fruits. And in the end, as you see, very gently picking the fruits, all autonomously, all done by Kubernetes. Cool, right? But seriously, why to use robot? First of all, to make very long story short, there are no pickers. There are no enough human pickers in the world. And it's seasonally work. You have something like three or four months at the most in a year to pick your fruits. And it's difficult to hire these pickers, so 20% of the fruits are stained to rot. And where in Tevel believes that less people should feed more people. But on top of the crops, the robots are acting as a data agent for color grading, weight, if the fruit is ready to pick or not. And even it's detecting the zest. Before I show you why we choose to use Kubernetes, I want to show you the technologies issues or challenges that we face with. First of all, our robot can do much more things while it's not picking. As you remember, you have something like three or four months in a year to pick. In the rest of the time, it can do pruning and spray and much more other things. As far as it, we want that our robots will be autonomous. We want that our application will be autonomous. So it means that when it has failure and it needs to be recovery, you need something like Kubernetes to do this job. Because, come on, guys, fruits are not waiting for software update. So it has to be fast. I want to show you where we are walking, here our deployment, here where we put our Kubernetes clusters far away from the buildings, far away from your antennas that use 5G. If I will have here one or two megabytes of downloads, I will be the happiest man in the world. It's very difficult to deploy. And right now, in these days, while I'm talking, we are picking in Chile. So imagine that our headquarters sitting in Tel Aviv need to support these guys at the field and risk from theft, a lot of challenges when you walk in a rural network. In a rural environment. Imagine that. Not like AWS or GCP, right? No, you are not managing the cluster at the cloud on the field. No network. So anyway, I want to show you how we started. As a startup, you need to run and run fast. So each developer at the beginning had his own robot. And he needed to SSH to his robot manually script to remote the robot. Different setup. The engineer host and the deployment host. Work is in my machine syndrome started. You know how it is. The manual installation was painful. High onboarding when you want to operate and when you want a new development was impossible to operate multiple drones. It was too many screens. What we did, we used containers. Look at that. We used Docker to easy setup and easy replication. We used Docker to freeze our setup. And we built a small web UI to control everything. But we still had some problems. When you're handling containers manually, it's difficult. Our images was huge. I'm talking about four years ago. The images were huge, heavy, and the build was difficult. Deployment was separately between the operation station and the robot. Multiple robots were still hard to operate. Hey, guys. What about Trigobonetes? We divided our application to microservices. The images was slim and thin. You can move apart very easy. We combined the deployment to one deployment for robot and operation system, operation station. We added service to monitor and control via web UI. Multiple robots now is a breeze. Problem is solved. Kubernetes help us a lot. But when it's come to scale, it's difficult. It still has challenges when you try to scale edge. You need to gather all the statuses by your hand. You need to make sure that your cluster up and running all the time. And you remember far away from the antennas. And it's difficult. You want to care about security. And you don't want to do that manually of a lot of edges device. And we are in Tevel wanted to focus on application. We want to focus on picking fruits, not on operation system, upgrades it. And video, imagine that. Because more apples, more scale. Indeed. So yeah, more apples, more scale. And as we were working with Tevel, we basically identified four needs that we had to solve in order to actually run hundreds of these multipurpose robots dispersed geographically. As you heard, we have robots in Chile right now, also in Italy, and in the US. So the four pillars that we identified are essentially easy provisioning of operations, functionality when it's disconnected, full stock security, very important, and zero risk upgrades as well. And those are the ones that we're going to be talking to you guys about and take you through that journey. But before we do that, let's take a step back. Because you might be wondering, what does the Tevel architecture look like? Or what are they deploying down into the field? And although it does look overly simplified, it's very complex. It's like you'll probably say that very well. But essentially, there's a ground control station that is a Kubernetes cluster, which is operated by an operator. And that's wirelessly connected to these bins. And these bins can manage for up to eight multipurpose robots for the fruit picking or pruning, whatever else Tevel wants that to do. So one ground control station can manage n number of bins. So that's basically the architecture. But now let's look at managing Kubernetes clusters at scale at the edge. So for that, we have spectral cloud palette, which is a Kubernetes orchestration tool designed for managing Kubernetes at scale, which allows you to do that on any data center, public clouds, and obviously at the edge. And what palette does is it has, essentially, for edge, very important, a very distributed architecture in the sense that every cluster runs its own palette agent as well. So we basically separate the management plane from the control plane, which it will make a lot of sense when you have disconnected use cases and when there's no bandwidth. So with palettes, you basically specify the desired state of your cluster, which we call a cluster profile. And that desired state goes from the OS. Like it was saying, they don't want to be playing around with OS dependencies having to install NVIDIA dependencies and kernel dependencies and all of that all the way to the application. And that includes also all the integrations in between. And with that, we call a cluster profile. So with Palette, you basically specify a cluster profile. You then push that cluster profile to a cluster, and the cluster then manages itself. So it reconciles and is able to manage itself with all the configuration and policies as well. So I wanted to just give you guys an understanding of that before we move on. So easy provisioning, but what does that really mean when we're talking about edge? So specifically for Tevils use case, and most probably like 90% of the edge use cases out there, connectivity is always going to be an issue. Security is always going to be an issue. So how do you provision edge devices? And Isaac told me a while ago, he was like, I want to be able to go to Best Buy, buy a box, boot it up, and get it to work. Because there will be instances where there will be the pickers and the bins will go out, and they will have to quickly recover them. And maybe they don't have it staged there, and they have to go and quickly do that. So provisioning, which could be in the field or at the distribution center, is very important. So we made that very easy. So we use Kairos, which is an open source project that basically it's a Linux method distribution that gives you an immutable OS. And with that, we then preload content. With Tevils, in this case, we're using Ubuntu and K3S. And once you boot up and you install your device, you all you gotta do is you ship it out and then the operator, all he has to do is power it on. Power it on, it connects, and then from then on, he can register the device with pallet, and you'll see an example in a second. And then once that device is registered, you can build a cluster as well. And as you can see with the cluster profile that you just saw earlier, and then the updates and day two operations are all done through pallet. So when an operator gets, for example, the QR code, so when they bootstrap and they get a QR code to scan, they will see something like this. This is a real example that we have with Tevils right now. And they will be able to add information to the cluster or to the device even. And you can see here's things like device name, farm ID, and also the region where it gets deployed. For example, in pallet in this case, we are actually using that to select where that device goes into a project, right? And that's how we do some multi-tendency in this case. But what happens when an operator is not able to actually scan a QR code or they don't have a screen or, God forbid, the hardware fails and you have to quickly swap it out. You don't wanna have to go through all of that together, right? So essentially, we also allow zero touch provisioning without the registration. And that is done by essentially, when you bootstrap the device, you can add some cloud init file with what we call a user data. And with that user data, you can specify some tags like you saw before and also an edge token which then authenticates against pallet and registers the device automatically. So now you don't even have to scan a QR code. All you gotta do is your device failed or you're rolling out a new device. You have to power it on and then that device is gonna be automatically registered with pallet. Then the second one, which is disconnected environment and maintaining and making sure that the functionality is still there. So as I said before, pallet has this distributed architecture. So as you can see, you might have n edge nodes, but the control plane and the management plane are basically separated. So what happens is once the cluster is running and everything is going, then the cluster is managing itself. So even if there's no connectivity back to the management platform, the node and the device and the cluster will still maintain its status, its applications, its configurations and all of its policies as well. And this with Table happens a lot. Like it said, one or two megs, sometimes no megs to update anything. So zero risk remote upgrades. So when you push an upgrade to an edge device which is in Chile right now, if that fails, you probably don't wanna ship out one of your engineers from Israel to get there because that's probably gonna take 24 hours. And 24 hours means that they're not picking more waste, more loss as well. So with that, we're using, so as part of Kairos, the open source project that I told you guys about earlier, Kairos basically it's a container based image, right? And what it uses as well, it actually splits your disk into two partitions. So you have an active and a passive partition. And when you push the upgrade, if your upgrade fails, it actually reverts back to the previous configuration. So the upgrade won't break the workflow. This is essentially very quickly. I don't wanna spend too much time, but very quickly this is what our cluster profile looks. This is the live one that we have running right now. And you can see that from the OS all the way to the top, TAVO is using flux to install the applications and maintain all of that. And as you can see, on the OS side, what Palet is doing very importantly is installing all of those NVIDIA dependencies, all those kernel dependencies, and also setting up all the networking for the robots with the bins as well. We're using Netpun for that. And we have, like I said, when it comes to security, some would say probably one of the most important ones, right, it's sick, especially for Tevelle when you have all these devices out there. So because we're using Kairos again, the OS itself is immutable, so your root file system is read-only. You can't install new packages or anything like that. All of that is built within the container image. You can bring it on OS because it's a Linux meta distribution, so basically you can bring any Linux-based OS and it's tamper-proof because of the immutability in itself. And then we have to look at TPM offline encryption. And you guys probably were at the awesome keynote with Saad and Arun earlier, and we announced our collaboration with Intel with Senna Secure Edge Native Architecture, and this is part of that. So this is one of the components. So how do you do offline encryption? So Tevelle, before, they were using UB keys, but if you have 50, 60 bins right out in the field, then you need 50, 60 UB keys plus each operator will have to have his own keys, so it's impossible to manage that scale. So it's an awesome product, but just not for the right use case. So we decided to go ahead with encryption with TPM, so Trusted Platform Module, which is based in hardware. So the TPM will create a hash key, that hash key is then saved into a KCrip Challenger, which actually lives in the GCS, if you guys remember the GCS, because the Intel TPM is in the bins, the KCrip Challenger is in the GCS. So effectively what happens is every time the device boots, it will challenge the TPM and it will only boot if that challenge is successful. Now if someone comes and steals the device, the device won't boot and the disk is fully encrypted as well. And with that, it's like, what have we learned in these past few months now? Yeah. At the end, Tevelle wants to focus on software. We want to pick Apple right, we want to pick them good without bruising them. So we need to be focused on software and not on operation system. So this is why it's so important for us hardware. It's hard, you know what it is. So this is why we choose Spectra. It's the right technical, they are awesome people. It's reduced the times that we spent of building and managing clusters by ourselves and handling operation was, it's give us, it's freeze our engineers time to be focused on innovation and more fruits we're picking apples, nectarines, plums, peaches, a lot of fruits that we want to be focused on. So what next? With the extra time, Tevelle engineer focusing to replace the GCS. We don't, we want to get rid of the GCS. It's the ground control station. And we want to reduce our infrastructure cost. Yeah, and like I said, security again, ever evolving and we started with some of the concept around the center framework that you guys heard before from Saad and Arun. And so we are already doing part of that with persistent data encryption with TPM. But now it really is about for edge, it really is about securing the device from the silicon all the way to the top, right? But how do we apply that? So Tevelle is already using x86 machines. So all of the computational power is coming from the Intel x86 machines. So for us, it's really all about now starting to implement in the next few weeks or so, we're looking at how we're gonna start playing with Intel SGX. So think about confidential computing. So all of the, all of the different workloads, all of the different applications are enclave into memory and also Intel TXT and how your application will only run if the device is actually trusted, right? So if the device gets compromised, you wanna make sure, because it's all out in the field, right? So someone could, if they wanted, potentially if it's unsupervised, go and hard connect to it and try to do something bad to it. So how do you do that? Making sure that it's all safe. And the last one as well, very important around encrypted images, right? So if you think about Tevelle and the fact that if you have devices out in the field and you have to update your registry, how do you do it? If you have a one meg link and you have 10, 20 gigs of images that you have to update, how do you do that? So in Tevelle's case, what they're doing is they have operators that go out and then they manually update the registry. But what's stopping that manual operator from actually corrupting or stealing those images which have their IP, right? So you kinda have to encrypt it not only at rest but also in transit. So looking at end to end image encryption with TPM which already exists as well on the machines. And with that, we'll show you guys a blooper. I don't know if... So this is when we were really making sure that everything was working and we're all very excited. So this is in Tevelle's lab. You can see that we were all very excited the first time that we got it working. And you can see that's Tevelle's UI and also pallet on the left hand side. So we just wanted to make it a little bit more fun. I think we have some time for questions. If there's any questions, thank you guys. Okay, so... If the questions raise your hand, I'll bring you the mic. This is the drone. And by the way, we're gonna be at... From tomorrow on, we're gonna be at S22. So if you guys wanna ask us questions, check out the architecture. We can go... It wasn't too deep into it. So if you guys wanna get deeper, please come by. If you wanna see the drones, this is one of the mock-ups. You can play with it as well. Yeah. And we do have an announcement for someone very lucky. So we were running in our booth a competition. So if you guys have scanned the QR code, then you might be in it to win it. So I need to remind myself who won it. Janki's looking at me like, I've told you before. We're giving a robot the free. No, no, don't. Maybe the next one. And the next one will give away a robot. Okay. So just remind myself of the name of that person. Adding some... You guys write a lot. In order to group chat. And here I was worried about the presentation and I think this is the... Oh, okay. So the lucky person that won this is Nicholas Ferrero. Or Ferrero. Oh, you are here. So you got a mind storm. We gotta keep it in the robot land. You guys have any questions? We got one. Thank you. Always. Thank you guys. Very interesting presentation. Can the robot do the dishes? No, that's not my question. Sorry. My question is how easy or how difficult was it to plug the TPM to the whole Kubernetes stack? Did you have a problem with the libraries that you needed to use to talk to the TPM, the PIKI CS 11, et cetera? Yeah, very good question. So we are... So because we're using Kairos, like I said, open source project. So Kairos has this workflow built already. So what you have to do, so we reused the GCS and we use KCrypt, right? Which is another open source project. So we didn't actually have... It was actually pretty easy with Kairos to do TPM encryption. Once we figured out the architecture and what we needed to do and where basically the KMS, the key management system, was gonna live, once we figured that out, then with Kairos was pretty easy because what happens is during boot or during installation, as we call it initially, Kairos will do all of that work for you in the background and it will automatically encrypt the partition that you tell it to encrypt. So you specify I want to encrypt whatever partition, right? In our case, it's called the persistent partition. And then Kairos is doing all the work in the backend as well. So from our side, it was pretty easy to implement there. So you should definitely check it out if that's something they wanna look at. Thank you, great presentation. We've got one last talk. We had two people on the program committee for this event, Oleg and Tina, who couldn't make it because they had visa issues with travel. But Tina is, has recorded a five minute wrap up for the conference. So if you can start playing that now, we're ready to go. Thank you guys. Thank you guys.