 Hi, thanks for the introduction. I'm quite delighted to talk here today about Yeah, in those building industry 4.0 applications and rapid prototyping with micro python So last year I saw a nice talk on micro python and after that I was kind of curious when the next project would come along where I could Yeah, convince my boss that we need to invest in that and get started with it so we had a company that That makes injection molded parts contact us and basically They said hey, we need we need your help with the interlogistics. So They have these machines that produce plastic parts for the auto industry using injection molding and Then at every machine there's a worker that basically takes the parts and stacks them into these boxes here and and Then the boxes get picked up by a forklift and then stored in the storage facility and there they They work around the clock so in three shifts and they have 40 machines in there in their production facility and it's all in one big building and Right now they just split the production facility to five zones And then they have five forklift drivers in every shift and they drive around until they find a full box and they take it to the storage facility so there's like no communication or organization other than Dedicating one driver to a certain zone, right? And that's very inefficient obviously And they wanted us to help with that because they have trouble finding new forklift drivers and So the goal was that instead of yeah Dedicating in a static way the drivers to the zones There should be some dynamic allocation and it should Also the drivers the number of empty runs should be reduced, right? Because these forklifts are quite expensive. So if they drive around a lot like with no load and They waste time, right? It is should change and they should this is be like a uber for forklift, right? It should it should be on-demand only And so yeah, we first talked with them I convinced my boss that this is going to be our first microbiome project and Then we said, okay, what what is the logistic process, right? So how does it work? So basically the forklift driver is free then he should The machine operator has a full box. He should push a button then basically the forklift should appear at the machine Pick up the full box bring that into the storage facility And then pick up an empty box and return that to the machine because of course They need a constant inflow of new empty boxes as well and Yeah, we We then looked at what what hardware should we use right because a micro python? we need some hardware for that of course and At in the beginning we thought all this will be like an android project right where we build like some kind of smartphone app and then And then we can do that with an app But the thing is that the machine operators they work lofts So they cannot use capacitive touch screens So you might know that from the winter depending on where you live that when you work lofts You cannot read your mail, right outside and So we looked around for micro python hardware and we found the m5 stack platform It's somewhere from Asia and they open source the hardware So we thought oh, this is good because if we start building that and later It turns out that we need like a thousand of those devices Then we can just take this open source plan adapted to our needs and build our own hardware We could also use the facility Wi-Fi network so that was quite helpful because Otherwise we would have to install Wi-Fi network there, but the m5 stack has Wi-Fi is a STM32 So it's like one of the really Yeah default microcontrollers for micro python stuff It has yeah, it's powered through USB. In fact, I have one here and It also has a small battery for When the power goes out it will last maybe five or ten minutes depending on how much your code uses the screen and how energy efficient you implement it It has Bluetooth. So We thought that might be interesting because The customers warned us that they didn't want two drivers in the same zone at the same time because they have narrow hallways and They are worried that the number of accidents will go through the roof if this Yeah forklift drivers start meeting in the in the narrow parts of the Of the place so we thought okay, maybe that's cool with Bluetooth later on We could build some kind of collision avoidance system or use these Bluetooth Smart tags to to figure out if if there is a driver in a certain section of the course and The best thing is that yeah, they are only like 50 euros They also run by default they run like Arduino See this Arduino see flavor, but you can flash micro python on them and they they provide you with a version of micro python, but In that version you have to yeah, I write your code online in their online IDE and obviously we didn't want to do that Right because we thought we had maybe some commercial opportunity there later on so we were not really keen on developing this and some like web GUI Where the servers are hosted in China writer? so In the end We went with this there's this guy on github. I don't know. Maybe it's also here called Lobores, and he has a really nice Micro python firmware for this STM 32 baseboards, and we use that and that worked out of the box So now that we had the hardware Right. We also needed some kind of message protocol And we decided to go with MQTT. So who here has used MQTT before? Okay, it's pretty good So it's a very famous. They are published as prior architecture based Yeah, message protocol. It's very lightweight and it's quite common in IOT applications so Here's an example with a with a temperature sensor where basically The the sensor publishes this and then there's a server called a broker that will hold on to the message and Until someone subscribes to that topic. So the channels are called topics in MQTT language and The messages have this quality of service flag that basically says Deliver the message at most ones at least ones or exactly once That's quite good because in our case we wanted to make sure that only one forklift shows up when there's a box to be picked up, right? and the other thing is that there's MQTT libraries for kind of every Language, yeah, so if we wanted to later on build a web app We could do that because we could just send the messages that is needed to trigger like the forklift pickup Through a HTTP MQTT client So the system looks as follows So basically we have these machines right that produce parts and they all get one M5 stacks mounted to them and Then on the other hand we have the forklift drivers that also they all get the M5 stack If you are thinking about how did they power this in the forklift so the forklift like cars have the cigarette lighter Plugs so it's really easy to to power stuff there And then we need something in the middle right and we decided to go with a raspberry pi So that runs the MQTT broker that basically is the message exchange place And then on top of that broker we built a couple of services. So we built something called the dispatcher Which basically takes the the messages from both sides and then Creates jobs and then assigns a forklift to a job and it keeps track of the jobs that are currently running and so on and it does the scheduling so We implemented basically first in first out, but you could later on implement more fancy scheduling algorithms right? Then we had a locker component because we wanted to know what was going on so that was a good hint in the previous talk actually and We built an admin panel Basically for the the manager of the plant should be able to see What is going on in this plant right? So there you can see like which machines are producing which how many open Jobs there are at the moment and so on And he can assign the forklifts to machines there. I will show that later in a demo and all these Orange boxes we we put into microservices, of course So then we went off to to start a development and we were thinking about okay, how should we Have a go at this because Yeah, I guess if you do some embedded development You know that if you have to upload a firmware every time for testing It's quite a tedious process and in the beginning maybe you make bigger steps But then in the end you make a lot of small steps and then if you have to test after every step It becomes quite time-consuming and annoying. So we thought the following Let's implement a prototype in Python like regular Python test that and Then basically later we just copy it over to the hardware because it's micro Python right should be sort of the same thing and We're done and today. I will tell you about the the learnings that we had from that So first of all it started that we needed some kind of GUI library But so the the micro Python firmware. There's basically functions where you can like activate single pixels You can write text and you can draw boxes and so on but there's nothing that's really like Organized GUI library So we ported something that we found from one of the Arduino examples And we also wrote a simulator for for TK so that we could test on our laptop It looks like this. So basically we have a top bar for like showing the date or the version You have the center widget where there's the information for the user and on the bottom We have this like yeah, basically the button labels because here you can see the buttons are on the bottom right and Yeah, might be nice if the user knows what what he's doing when he pushes a button So we put the git version on top. So this is something that I could really recommend everyone to do because sometimes Yeah, things go wrong and you have the wrong version on your device and you're wondering why the change that you made is not showing So this is really a good hint and We put something called a Wi-Fi bar So, you know that from your phone just to show whether we are connected to Wi-Fi or not Because we figured in the end when we drive to the factory to deploy the thing and there's Wi-Fi problems Then yeah, we want to know and not like yeah search for the problem in the in the dark. Yeah and Then yeah, just fun labels Basically, so what happens now when the machine operator pushes the button? So I have the code examples here Just more for the sake of completeness and I will upload the slides to the git lab repo That's in the in the description of the talk so you can find them there in the next couple of days so Basically at first we we just said the text for the bottom bars And then when the operator pushes the button the following thing happens So there's a function called announce packet and that basically sends MQTT message to the broker saying like hey I'm machine number one and I also have like an integer ID it's pretty one and There's a new job and then The the dispatcher Oh, sorry, and then basically there's also this packet showing up there That's quite important because the people that operate the machines If if they push the button and yeah It takes like a little bit of time to draw the box But if you don't show them any feedback, right? They might get impatient and Then you have like a hundred packets that need to be picked up in your system because they'd like keep pushing the button So we did some visual representation of how many packets are waiting So then the dispatcher receives that request and the first thing it does is it just says, okay You need to put this into the into the queue and then in the next step basically it looks at It it sorts it so in this the queue job method basically it sorts the The job into the right queue because we said that there is this zone system, right? So right now they have five zones, but maybe later we can reduce that gradually So there's one one task queue for every zone So it sorts that in there and then in the next step it looks at the available forklifts in that zone and it Assigns a forklift right so it schedules the job basically and then it lets it lets the Machine side now and the forklift side now by by publishing messages, right? And then the following thing happens so on the machine it will say the I'm sorry This is in German all the text because German company so but it basically this means that the forklift is coming soon so and then the machine operator also knows that okay like There was the packet rating but now a forklift has has been assigned and then On the forklift side. Yeah, I'm sorry for the bad contrast, but we did the two different background colors It still says like oh, there's there's no No job so the message has arrived, but the GUI has not been updated, right? So the forklift basically we did a model view controller there and It it first it there's a job object that has that describes the state So it could be like pickup requested job is scheduled job is picked up and so on and so first it Updates the state of the job And then it it beeps so that the forklift driver gets a audio Feedback and then in the next step it updates the GUI And so first it will update the text saying like okay You have to pick up a package at machine one and then it also it has to it has to also update in the next step The bottom the button text right because then it should not say pause, but it should say like okay I loaded the job And then a forklift driver can confirm that he loaded a job, right? So he pushes the button and then basically yeah, we again update the GUI and we let The dispatcher know and I think the rest is quite clear right so this goes on then for some more iterations like But I will show this now in a demo anyway, so We built a simulator which is quite nice because that allows me to let me see so I can do this And then I can now put the Sorry But also I have this so so first maybe There's the admin panel here right so the left side you can ignore this is basically just a representation of of How the place looks like and that there's the four machines on the top and the red dots basically They indicate whether that machine is producing or not, but in the simulator that doesn't work But here on that side you can see that there is There's two machines now no three machines and one forklift so the cool thing is that With the simulator so with the simulator we can just start multiple instances and But also because yeah, well I brought my own Wi-Fi access point I could connect to one of the actual devices So you can see that it allows you to do this hardware in a loop testing, right? So in the beginning you simulate everything and then step by step you go you go more towards running things on actual hardware and now Basically you can make like new zones here, so I made this zone to and you could put the machine in here Right by dragging and dropping and you can just say okay. I just want machine number five to receive the So to communicate with forklift one You can also rename the devices if that's interesting to you and and That way basically they the company that we did this for they had the opportunity that way to to still control being in control of Who drives where and so in the beginning they could start with their five zone system and then slowly as they begin to Develop trust towards the system. They could relax their their zoning Yeah, and Then so, okay, I put them back now just to and so basically there you can also see that disadvantage of the Advantage of using MQGT because so also this web GUI it uses JavaScript and it communicates with MQGT with the dispatcher right and So now maybe you want to see how it actually works so So basically the buttons are emulated by key presses So you have to trust me that so I I now I'm on this machine there on the top so machine number one and then push the Q button and then Yeah, it says the forklift is coming because at that moment. There was no other forklift. It goes quite fast and then Basically here you have this like red notification dots that count how many packets are waiting at that machine. So The manager has at one glance. He knows what's going on in his factory, right? And on the bottom you see a bit more text information. It basically says, okay, there's a new job It's it has been assigned to this forklift and so on So now the forklift driver drives there and he pushes the middle button and then basically it says, okay He loaded it it immediately updates and then He does all that and then he brings the replacement and so on and then he's done and the job disappears And so now I can actually on that hardware push the button That takes a bit longer, but it also works. I can push the button again Now I have two jobs and then maybe I also Make some jobs here. So there's some stuff going on. So now this poor forklift driver, right? He can I mean, you can come up after the talk and try it out yourself. Maybe it's more fun But basically the nice thing here is that the simulator works together quite well with the real product, right? So they share most of the code except for the GUI library So basically on the one hand you you import TK and in the other case you import this Micro Python GUI library that we wrote Okay, so So did our workflow work as intended? Yeah, not completely. Yeah, so we made some really Yeah bad mistakes in the beginning and I want to share with you the lessons that we learned so the first thing is that Micro Python has no type hinting support. So in the beginning when we started that project. We wanted to be really good Yeah developers and we we wrote all the code with type hinting and Then after this long period of developing when we first like copied it to the hardware, right? So like Yeah, we figured out. Okay. Crap. There's no type hinting and then we spent like one day Removing our mess The other thing is that You can upload code in multiple ways. So One way is is to To use the serial interface there's like a file system emulator But that's quite slow. So that's fine when you want to replace like one file But when you want to upload the entire code Completely and we always did that because we wanted to make sure that we don't mess up there and make some mistakes We found out that it so for our code it took I think like three minutes for one board So that's quite a long time if you want to test out a small in a small change, right? So that's what really paid off to to test on the laptop But then later we found out that you can actually like include your code in the micro pattern image And then you can just flash the entire flash storage in one go and that takes like three seconds No matter how or four seconds no matter how big your code is. So That's something I would really recommend you also Especially it helps you in that phase where you need to roll out. So once we went to that factory and we had to like Roll the system out there. So we tested it there and it actually ran like for four weeks without a single support call So that that was quite nice We needed like a lot of units to To update a lot of units at once right and then this flashing really paid off and the other Yeah, hint is before you begin your development Make sure that the libraries you plan on using are Available in micro Python because that's also something that we didn't consider at all. We said, oh, yeah, you know And the laptop we have PIP and PIP has sort of everything and but then in the end we had to either like write a functionality or self or like find a different library so there is something called micro PIP and And there you can see whether the packages are available The other thing is that you can run the micro Python interpreter also on your on your local machine So if you have something where there is no graphic user interface That might be quite smart for local testing But I think for us it didn't work because of the of the GUI Because then there was no way to simulate the display or at least no easy way And so I would say the workflow should look like this First check the dependencies for your projects or like these libraries that they are available build a simulator implement it And then Simulate as much as you can test as much as you can on the laptop because there is just really fast and then later if there's Problems that you tested for on your laptop that show up later. You know that in the end is a hardware issue Maybe and then profit And so in deployment it yeah, it looked like this So there's quite a rough environment But these things are super cool because we just used the TESA power strips and we glued them to the machines in the end we made a Android app for the forklift drivers because they didn't like the small display. They wanted a bigger display and Now you might ask yourself that but you said you're a data scientist, right? So what does all this stuff have to do with data science? So one I'm Embedded systems guy by training and then in the PhD. I started data science, but Yeah, secondly if you want to do data science, but you have no data, right? You need to like develop your own means to to record data first and I'm running out of time So I will skip to the interesting bit So in the end because we we always Notified a dispatcher when the job state changes we could like make this Histogram for example is a cumulative histogram of the waiting time so the time between the machine operator pushing the button and The forklift showing up and then you can derive that Yeah for 80% of the of the packages basically takes less than two and a half minutes for some guy to show up and and take the package Then you can do the same thing for the turnaround time and you can see that the turn around so the complete cycle from picking it up to delivering the empty case Yeah, it's also below five minutes for 80% of the packages So that's where now the interesting stuff for me as a data scientist starts but in the beginning there was no data to analyze, right? So Yeah, that's that's why we why we invested the time in building this and also it's a lot of fun So I hope I could maybe motivate you to start your own Your own micro patent project with this and I'm happy to look forward looking forward to your questions Now so thanks Thank you very much Brian. So we have time for a couple of questions I can see that already people perfect. Please go ahead. Thank you Florian for the presentation. It was really interesting I have actually two questions. So at the beginning you mentioned that You are not going to use phones because of gloves, but at the end you used phones. So that's Yeah, so the rationale behind it. The second question is How does the dispatcher server? Assigns a machine to a full clip So how does it work? Is it like you put like already a machine and the full clip In one zone and then it's automatic or or is there is like a logic in the server? Okay, so so the first thing with the smartphones They told us that all machine operators wear gloves But that's due to that some of the parts that they make they have glossy surfaces So you don't want like fingerprints and grease them But the forklift drivers they don't wear gloves and also it turns out that many of the of the machine operators Actually didn't wear gloves. So that's something that the executives think think happens, but then like So in the end we also built a web version for calling the forklift you can see it here Because they had this touch screen this place, right? So they could choose between and like using the hardware and the touch screen For this the rationale was just the bigger screen size and also they wanted I Have the high slides here. So they wanted something where they could Make the system memorize where the where the box from that machine was stored in what storage place And that was much easier to implement in a web app than in micro I mean just the amount of time to to invest So we did it in a Two days or something. Yeah is that's where the MQT stuff really paid off because you can just switch Yeah, it doesn't it doesn't matter and then the dispatcher it works with You have the zones and basically the manager decides what forklift drives for which machines and Then the dispatcher looks for the next three forklift and just assigns that but you could do that You could do better than that for example, you could when you know that forklift a deliver something to machine one And you have two jobs waiting like from one from machine two and one from machine 25, right? Then maybe you wanted to work on the job for machine two because it's closer. So you could do that Thank you. Sorry. That was long answer Thank you very much. I'm sorry. We have questions, but unfortunately we are running out of time But the good news is we have a coffee break. So there's time for more. Yeah, so you can come here for me It's fine. I can stay here also. So let's give a last one of applause for Floyd on this