 All right. Hello, KubeCon. Welcome to my session, Kubernetes Running Manufacturing Systems, a perfect match. About me, I'm Moritz Kruger. I'm a scientist for the Chair for Laser Technology in Aachen, Germany. I'm focusing in my research work, especially on microstructuring and metal-suitable printing. And my personal research topic are normally data infrastructure for manufacturing, cloud-based manufacturing systems, and integration of data science into laser manufacturing. So especially regarding the IT infrastructure behind current manufacturing systems and beyond. And normally, if I'm not doing some cloud-based laser manufacturing stuff, I'm going tightsurfing somewhere in the Netherlands or as you can see here on Sicily. But if you know any other good spots in Europe, please let me know. Today I brought the following agenda. First of all, I'm going to talk a little bit about the Internet of Production, the project I've been working for for the last two years. Also, why do we need Kubernetes for laser manufacturing systems? Also, I'm going to talk a little bit about our current lab setup. And last but not least, I'm going to talk about the experience we made with this lab setup. So it's a good, the bad end of the algorithm. So the Internet of Production, what is it? The Internet of Production is one of Germany's largest research projects in the area of manufacturing and digitalization. We have a funding of around about 50 million and 200 involved scientists. And therefore are one of the largest projects in Germany in the area of manufacturing and maybe even in the world. The idea of the Internet of Production is the idea of the creation of digital shadows. So the main vision here is that you have the virtual world and the physical world down here. So you have, for example, the development phase of a product where you draw something or where you plan how to produce something or you plan, for example, a factory and normally use tools for that and these tools produce data. And then the product you want to produce goes into production. And of course, you have machines that also produce data. And later, these products are getting used and the usage also produces data. And the idea of the Internet of Production is that we bundle all of this data together into digital shadows so that we can later use them. That, for example, a drawing engineer in his development phase has access to some kind of algorithm that analyzes the usage data of previous products, for example. Or that's an operator operating a laser machine has access to algorithms that's analyzing the state of its current laser machine. The Internet of Production focuses on the following infrastructure map. It looks a little bit scary, but let me explain it really quick. We have our data infrastructure down here. So basically all of our product data, CAD data, simulation data, machine data, customer feedback data. And this gets aggregated by some kind of middleware plus system. That still is the work in progress, of course. And that binds or combines all of these different data streams into smart data. So here we can run different algorithms and different analytic models to actually make sense of what we are gathering at the raw data level, basically. And all of that, these algorithms can then be fed into the smart expert systems which can help different people in different design phases or in different phases of a product lifecycle. For example, during the development cycle, the production cycle or the user cycle. Of course, we're not doing that alone. We typically have partners in these projects. And this is a short overview of our partners. Not only consisting of, I would say, typically manufacturers like BMW or Airbus, but also digitalizers like Siemens and Bosch and also AWS, for example, as a cloud provider. Well, I'm not here to talk about the internet production. I'm here to talk about why do we need Kubernetes for laser manufacturing systems or manufacturing systems in general. So first of all, what is our laser manufacturing system? I'm typically working with ultra-short pulse manufacturing systems. These look typically in an abstracted way a little bit like that. We have some kind of laser source that shoots from the side in a scanner system. And this scanner systems basically consists of two little mirrors that can be moved for example, an FPGA system to then locate or to place the laser light exactly on some kind of substrate or metal plate. And then where the laser hits, the metal gets vaporized. Normally, you do this with really, really short pulses. So you just vaporize a little bit each time. And the idea here is to vaporize it so fast that the whole plate doesn't get hot. And with the mirrors, you can basically move the laser beam really fast in a predicted pass. And by that, you can create really fancy structures. As you can see here on the right side, for example, you can see, for example, filtration holes that you can shoot into metal surface or also a laser surface that got shot on a metal surface. And as you can imagine, because the beam is moving so fast, a lot of stuff can go wrong. So we typically include sensor systems into that process. For example, video observation systems to observe the process status. One quick example would be the filtration system here on the right. You want to make sure that every hole you shoot is actually a hole and not maybe still closed, for example. And you can pretty easily write an algorithm which analyzes video data, which actually detects if the hole is not through. And by that, of course, you can gain access or you can gain more knowledge about the process itself and how well it was done and so on. And you can make quality predictions during the process. The problem we ran into when you want to basically connect so many sensors onto systems like that is that we have way too much data. The laser beam of the scanner system typically moves around about three to four meters per second. So you need really high frequent data acquisition systems to actually keep track with that. A typical camera system is around about 1,000 pictures per second, up to 15,000 pictures per second. And that's just way, way, way too much for... It's barely enough that one system can handle that. So what you end up with is the following setup is this is from our lab that you have multiple laptops, multiple machines connected to one laser system here. So you can see here one of these laser systems and the laser beam getting guided in here through this fiber. Then you have a camera system here, you have another camera system up here, you have a microphone here, and then you have all these different computers connected to these different sensors and to the scanner system and to the laser, which are responsible for aggregating the data and controlling the system. And normally the problem we run into, of course, is how do you handle that kind of workflow? What we started with was a workflow where we ran around in the lab and were shouting at each other, something like, okay, I just turned on the camera and then the next one said, oh yeah, I turned on the microphone. And then we were like, okay, now I can actually shoot on my substrate with the laser beam. That's basically the scientific work we did, which of course was pretty error prone because a lot of stuff can go wrong. Somebody can accidentally not start the camera system or accidentally name it differently and so on. So it was really hard to handle that data and to set up a reliable experiment with that. So the idea was now, okay, we need some kind of system which bundles all of that together and had some ideas and then we thought, okay, let's try microservices because first of all, they are distributed anyways and second of all, we can go completely on a network system, we can rely on a network and make it really, really plug and play basically. So the most critical problems we wanted to focus on were in the first hand, how can we actually integrate industrially communication protocols like Modbus or VCUA into a microservice-based architecture? Also, how can I actually store all my sensor data automatically because that was one of our biggest pain points? It was just eating so much time that we had to handle the data manually and last but not least, we wanted to make the whole thing as extensible. So the idea here is, how can I add more actuators or sensors later on without influencing the system too much? Yeah, so that's pretty much the motivation of our system. That's where we come from. Now, a little bit about the architecture and the current lab setup, we have at the moment. We started with a really, really general architecture. So we said, okay, cool, we have now all these different services. We want these different services. We want one service that constructs a scanner. We want one that does the sensor and we want, for example, also something that analyzes our sensor data. As I said, for example, the example was shooting a hole in a metal plate. We want to make sure that this hole is actually shot through. So we need some kind of analytic service which takes a picture, takes a look at the picture and then tells us, okay, where are the holes in the picture and do we maybe run or need to rerun the process or the laser beam on that specific point? So the idea here was, okay, we kind of like put the analytic service in its own bubble in its own container and therefore we can basically move it out of the system and then we have some kind of job controller in the middle right here which connects all of these different parts and has the complete execution logic bundled inside of it. And of course, when you need some kind of job controller, you need a job spawner and some kind of user interface to actually handle all of that. If you now map that to our laser system again to answer the question of how do we integrate different protocols is we came up with our three first container-based images with a movement service, a camera, service and a scanner service and all these different services can talk to different, can talk different APIs. So the movement service, for example, can talk Modbus, but doesn't really matter anymore because we are using GRPC to control the movement. So we have a standardized way of talking to hardware which then gets translated into another protocol. We also have, for example, the camera service which talks some kind of USB, API, shenanigans and then we have a scanner service which we can also control via GRPC which connects to our FPGA system and then tells basically the mirrors and the laser beam what to do. And this works really good because we have now our scanner services. We designed them stateless. So when they come up, we know that they are already configured, they are already honed, they are already basically ready to go and the user of that system doesn't need to worry about it anymore. You just sense a signal like move from point A to point B to the movement system and the movement system does that. Same for the laser system. You just send the laser system basically a drawing, for example, to shoot a square and you just give it basically the vector list of that square and the laser service or the scanner service will make sure that this square will just be shot onto the surface. And the job controller is then just basically shifting around all of that different information between all of these different services. So that was basically part one. How can we manage different hardware protocols? Well, basically we introduced a new layer of abstraction by using, for example, GFPC. Of course, REST could work as well, but we wanted a protocol which is binary because of the high throughput we had. So we decided on GFPC for that. The next question we had is how can we actually store our data automatically? For that we need to take a look a little bit deeper into the services that we created. So we're going to focus, for example, on one of these hardware controllers now. And we came up with the following structure. So the idea was, okay, we have our hardware adapter, basically, which controls our hardware. So we have some kind of scanner system from one vendor, and we make a GFPC interface right in front of that so that we can control this system directly via a network hole. And we put the whole thing in a Docker container. And we then introduced the so-called service controller, which basically proxies the incoming client call to the hardware adapter. So what happens is a user connects to our scanner service and says, hey, scanner, please shoot one point. And the service controller takes this request, sends it down to the hardware controller, the hardware controller shoots one point, and then goes to the service controller and says, hey, okay, I just did that. And the service controller will then save the call and the answer of the hardware into a database and make it basically lock itself. So it's basically a really simple logging mechanism. Of course, this is pretty boring for actuators, but for sensors, it gets very, very interesting because you can say something like, hey, camera, please take a picture. Then the picture gets taken and then you save it into your database and you just hand over the URL back to the client and say, hey, I just took that picture for you. If you want to take a look on it, just get it whenever you want, and whenever you want to get it. So the nice part about that is that everything you do is getting locked and saved. And since we put everything in containers, so we have one container for our service DB, we have one container for our service controller, we have one container for our hardware adapter, we can put or use config maps to configure that and we have some kind of automatic configuration management, which can, of course, which Kubernetes luckily already provides. So we don't need to worry about how to handle configuration in that system anymore. So well, what we ended up with in the end was basically a Kubernetes cluster which spends not only from into a data center on the shop floor, but also spends onto the edge directly next to the machine. So what we can see here is that our edge is basically the computer or the server sitting or directly controlling the laser beam or directly controlling, for example, the camera system. Of course, since we are in a manufacturing environment, we can go pretty, we can buy pretty large systems here. So these are normally not, I don't know, Jetsons or Raspberry Pis, but we buy real server-grade stuff because we have enough internet connection and we can also provide enough energy because our data system already needs energy so we can just leverage that. But we can also use or connect it directly via Kubernetes with our data center where we have multiple servers running. And the nice part now is that the picture I just showed you, because of Kubernetes, we can shift all of these containers around. So we can say, okay, cool, I have a camera system which produces 15,000 pictures per second. I probably need the database pretty close to the manufacturing system in order to even be able to save that into a database. While with other services, you can decide, okay, maybe it's not that important. Maybe I don't want the hassle of actually having that kind of database right next to that. I can put it into the cloud, for example. And that is really nice about Kubernetes that it gives us the flexibility of shifting all of that around and see what actually is happening and design your system that you can very flexible change depending on the data stream you're using. So you can plug in a new camera system which produces a lot more data. And then you say, ah, cool, I just need a database right next to it. Damn, it's there. So that's really, really nice. And later when your process, for example, is not running, you can then decide, okay, maybe I migrate now my database from the edge nodes, for example, into the data center. But then it's not time critical anymore. You can take longer. Okay, coming to the good, the bad and the ugly. Well, the good is we have a CI form manufacturing system that's really, really nice because we went full container on that so we can now just build, deploy, and push all of our stuff directly onto the manufacturing system. We have automatic sensor data acquisition with high throughput, which is also really, really nice to have because before that we had to handle that by hand basically. Because of Kubernetes, we have a single point of truth for configuration because we use config maps for everything. And that is really nice, especially from a scientific point of view because we know what's going on. We can roll back. If somebody changes some kind of, I don't know, frame rate or the, I don't know, acceleration of the movement service, we see that. And we can use Git to versionize it. By the use of GRPC, we actually prevented vendor login. So the system I just showed you works on, I think right now, three or four different movement manufacturers and also on two different scanner laser manufacturers. And it's also scalable to probably a few thousand of nodes so you could run your complete shop for all of that. And you also get remote control, which wasn't a thing till now at our institute, which brings me directly to the back. Well, you get remote control. Of course, since we are now completely network based, we open up for hackers and so on. But yeah, Kubernetes is, I think, a very nice system to handle that. It gives you a lot more overview of what is actually going on. But still, we are running a laser system in the network. So yeah, we have a little bit of risk here. And this is especially a problem when you take in consideration user management and safety stuff. If anybody knows of some kind of way of authorizing, of some kind of way of proximity based authorization, please give me a call. I'm really, really interested in that. And of course, one of the larger problems we have is that we have now a distributed state management, which makes stuff a lot more complicated. Normally, we had one machine connected to multiple things. That is way, way, way easier to handle than multiple machines in the network, which is running unreliable. Okay, the ugly. Well, you run into a lot of concurrency problems with micro-services that you don't have to handle before when you're not using micro-services. Also, industrial protocols are pain in the butt. It's just crazy how complicated industrial protocols is. Also, most of our hardware is still windows-based. We are currently picking a lot of Linux stuff, but a lot of that is still windows-based. I see the whole industry is a little bit shifting, but it's still a lot of windows, which isn't that good for humanities. And we have finally automated testing so we can test our machines automatically pretty easily. Summary. Yeah, micro-services for manufacturing are really, really cool. It makes a lot of fun programming your manufacturing machines with that kind of architecture. It's very, very scalable, and the management of your manufacturing machine parts become quite easily. And it also gives you the possibilities that developers can hook into that system. Yeah, that would be all from my side. Thanks for listening. If you have more questions, please send me an email or ask them directly in the questions and answers directly after that. Also, I would really, really like the DFG for sponsoring this scientific research via the Internet of Production. Without that, I wouldn't probably be here today, so thank you very much for that. And see you in the comments.