 Hallo iedereen, bedankt voor het joinen. Dit is over, zoals ik het project Coldplay calle, of hoe je CNCF-projecten voor enige muziek in het aandacht van mijn huis, Elevator. Ik ben Erwin de Kijzer. Ik ben een DevOps-engineer bij Day. Ik ben een Elevator-engineer bij Night. En dit is mijn Elevator. Dit is door in de livingroom. Het spedt op heel significant. Ik denk dat het twee keer zo snel gaat als het eigenlijk gaat. We hebben onze huis gebouwd een paar jaar geleden. En het belongt op een oud vrouw die kunnen niet meer werken. En ze hadden een elevator installeerd in het huis. En we hebben het huis gebouwd. En eigenlijk om het elevator te brengen is het echt expansief. Dus we hebben een elevator in het huis. Dit is in onze livingroom. En het gaat van de grondloor naar de tweede. Als je 0-1-2 kan komen. Ik weet niet hoe het in Englisch, Dutch, Amerikaan is. Ze proberen het anders. Dus twee vloers van reis. Het is echt gebruik. Mijn moeilijke stof. Zoals een washingmachine. Voor de mannen die vorige presentaties hebben gezien. Het is een washingmachine. Ik heb over het Grafana gereden vijf jaar geleden. En dit was toen het gebroken was. En we hadden het van de huis gehouden. Dus een beetje een background naar het project Goldplay. Ik denk dat ik vijf maanden geleden heb gehaald. Ik heb lunch met de collega Rue aan de Bruin. En we hebben over fun projecten over het huis gehouden. Dus ik maak de washingmachine. Ik heb mijn garagedoor geautomateerd. Ik maak de powerusage van mijn huis gehouden. En we hebben een brainstorming over wat kunnen we doen naast. Wat kan ik doen naast. En hij wist over de elevator. En de elevators waar we werken. Ze hebben alle aandachtingen als je aan de vloer rijdt. Een heel makkelijke aandachtingen. En ik dacht, misschien kunnen we muziek op de elevator. En na de visite in Cubecon in Detroit vorig jaar... ...waar we voor de plane en de airport in Detroit wachten... ...halen mensen over. Oh, let's meenemen voor Cubecon in Amsterdam. Like my home country. En er was een lot of talk about... ...how to sell Kubernetes, why you should use Kubernetes. Kubernetes is awesome, I thought. Let's do something different. So I thought, I have an elevator in my house. I'm gonna submit it. And also gave me a very good reason to actually execute the project. So this is some pictures of the elevator. This is the door in the living room. You can even see our couch here. This is when you look up in the elevator. So you see the full shaft to the actual roof of the house. This is the door to the first floor... ...and the door to the second floor is not visible here. And this is the electronics that are visible... ...when you remove a panel in the elevator. And I know enough about electronics. I saw enough YouTube videos that I don't want to mess with this. But it also brings us to the goals and rules of this project. I want to know where the elevator is... ...so I can have proper announcements. I want music when the elevator is moving. So announcements when the elevator stops at the floor. I don't want to break the elevator... ...and I don't want to mess with internal electronics. Because I'm not an electronics expert. This is actually very interesting if you look at it. Here you see lights that correspond to the doors. This panel knows everything. It knows that this door is open and the other doors are closed. But I suck at electronics. I had multiple issues with soldering just two wires together. So I'm not going to touch this. So it's basically a black box approach of monitoring the elevator. And this is the first attempt. This is an ultrasonic sensor. I hooked it up to Raspberry Pi. I quickly got some Python code working to produce measurements. The distance seemed quite stable if I put the sensor on a flat surface. There's not a lot of jitter. If I move it around, it measured all the distances in the home office quite well. But this also brings me to my first failure. These horizontal support beams in the elevator shaft prove a problem for the ultrasonic sensor. It's basically the sound waves travelling in a certain direction. It is a cone going up. And these horizontal support beams seem to produce echoes. Not very reliably, but they do it sometimes. It's already too much. The other thing is, the top of the elevator platform that I want to measure is quite a small target for the ultrasonic sensor. It needs a bigger target. And the elevator shaft is too high. If you try to measure more than four meters with the ultrasonic sensor, it becomes unreliable. So it's a literal vertical scaling issue. So I come up with something else. This is an infrared sensor, the TF Luna. It has a range as reported by the spec sheet of up to eight meters. Accuracy of two percent of the measured distance. So at eight meters that would be about 16 centimeters that it could deviate. Frame rate of up to 250 Hertz. And there's multiple ways to connect to it. You can have a serial connection where measurements periodically come in. Or you do like a trigger based. You ask for a measurement, you get a response back. En it can even have interrupt based measurements where it's constantly measuring. And if it has a changing measurement, it will let basically one of the pins on the output will turn high. So you know that something changed. In the end I chose to use the trigger based mechanism. This is the setup. So here this is the elevator shaft. This is the measured target. The top of the elevator platform. This is everything that moves. Here's the TF Luna's mounted at the ceiling of the elevator shaft. Like the full distance is about seven and a half meters. But the top of this platform is five and a half meters. And in software there's a measure component. I written it in rust. It runs on the Raspberry Pi. That takes the actual measurements. Then it sends them through nuts. And then they are picked up by PromViter. That I wrote in Go. And it does remote rides into Prometheus. This is the message that goes over it. And the reason that I chose remote rides with Prometheus is because the elevator doesn't get a lot of usage. It's sometimes not moving for a few days on end. And I don't want, I don't need a very high resolution of metrics when it's not moving. Like once a minute is more than enough. But once it starts moving I want to have a very high resolution. So when it's moving I think it rides about once a second into Prometheus. So you can very clearly see how it's moving. And when it's not moving you get a flat line. So this is the first result. It actually seems to be working. This is a full ride from the ground floor to the second floor and back. And it takes about 55 seconds to travel this distance. The speed seems pretty constant. And also it's actually measuring the opposite of this. Because the sensor is mounted in the ceiling and it's measuring the distance to the platform. So when the platform is at the top it's a very low distance. But I correct that in rust already. And also this part is when it's very close to the sensor. I think it's about 2 cm distance that it's measuring. There's also not a lot of noise. So the minimum reported distance of 20 cm seems like a safety measure. But 2 cm is also going fine. So I actually had reached the first goal already. I know where the elevator is. Some more charts. This is the absolute speed. So I do a promquel query of the elevator height. And you see that it's moving about 10 cm a second. Which also correlates pretty well with that it takes 55 seconds to go 550 cm up. And going up and going down they're both almost exactly the same speed. It's a tiny bit faster going down. But not enough that you can actually feel it. It also doesn't matter if there's a lot of load on the elevator. If it's empty, if it's full it's always the speed. And now when I have the measurements. I can also do alerts. So I set up some alerts in Grafana. That I know when something is wrong with the elevator. If it's stuck between floors. The way the elevator works is that you have to hold a button in the elevator to move to the floor. So if you release it you can get stuck between floors. So if it's stuck between floors something is wrong. I don't think it will happen anytime soon. But with two small kids. I want to know. If the elevator moves too fast I also want to know. It's probably not going to happen. Because there's I think like four redundant safety systems in the elevator. But just to be sure. Also if it moves too slow. I don't see it happening anytime soon. But it's probably a degradation in performance of the elevator itself if it happens. And that's just fun to know. And if the elevator is out of bounds. So if it suddenly goes into the basement that we don't have or too high. I really want to know. It's quite unlikely that this will happen. If it happens it's probably an issue with either a sensor or my codes. Which are both far more likely. But I did run into an issue. So after I had this setup for a few weeks. I noticed that there was quite a lot of jitter. So you see the line is going up and down. It's about 5 millimeters to 10 millimeters in a full day. So it's not a lot. But it's enough that I started looking into it. I also got myself nerdsniped here. I started to think about the elevator. Is on top of the earth, the earth is rotating. Maybe it has to do with the direction that the earth is traveling in. And the earth if it's aligned with it. Maybe that gives different results. Space time is shrinking. I don't know. I tried to calculate it. I couldn't work it out. So if there's any astrophysicists here please. Please come talk to me. But also I realized it's a cyclical nature. It has to do with temperature. Maybe it's infrared temperature. Maybe it has an influence. And I went over the spec sheet of the TF Luna again. And you can see here. It reports the temperature of the chip itself. In the amazing resolution of 100th of a degree. And so let's add the temperature to the stuff we track. And after a few days of running. This chart came out. The red line is the temperature. Blue line is the height. And this is a slightly cleaned up version. I take the average over 15 minutes. So the jitter per second is a bit less. And you can see that there's actually pretty often. It correlates with like a drop in temperature, drop in distance. It's still like the actual changes are very small. You go from 2.7 to 2.5 centimeters here. So 2 millimeters. But it does happen when the temperature drop. But it also seems to happen a bit too much when the temperature drops. Like also another first limit. The temperature seems to drop in full degree increments. So here it's 20, it's 30 degrees, 29, 30. It seems to, it reports it in 100th of a degree. But it's very clearly just measuring it in full degrees. And the slide drops probably come from me taking the average of 40 measurements before posting it. I do that within a second, but that's why you get values in between the temperatures. And every time the temperature drops or rises, the distance also changes. So it looks like the TF Luna is already doing some kind of error correction based on the temperature. Because every time it goes from 30 degrees to 29 degrees and back, the distance changes 2 millimeters between 30 and 31 at 3 millimeters, between 28 and 29 is 4 millimeters. So it's a bit too much of a coincidence for this to happen. In the end, temperature is not really useful to make the measurements more reliable, but I did find it very interesting. Also to detect if the elevator is moving. It's also not that relevant because these are changes in the order of 1 millimeter a minute. En once the elevator starts moving, it's 10 centimeters a second. So it's like 3 or 4 orders of magnitude difference. So let's get back to the software architecture. In the end, this is what I came up with. Basically a microservices approach. On the Raspberry Pi, we have two components, measure and speaker, that do the actual measurements and speak the announcements. Then we have the scientist, which is of course a gold play song dat does the actual thinking of the whole project. So it takes all the measurements and decides should we play music, should we update the user interface, should we send it to the Prometheus. And these two have to live on the Raspberry Pi. The others can live wherever because they only have to be on the same subnet en the subnet in this case that I use is related to tail scale. I'm not sure if anyone knows about tail scale, but it's an awesome product that basically allows you to build an overlay mesh network between all your own devices. So no matter where a device lives, if it's on the tailnet, it can connect to other devices on the tailnet. So my laptop is in the same tailnet as those devices. I even have had something running on fly.io. They can also live in the tailnet so it makes all the connections a lot easier. Also has ACLs and it uses a wire guard for underlying safe connections. This also brings me immediately to the second failure. This is too much. I'm an elevator engineer by night. I had a J job. Maintaining five components at a time just became too complicated. I had issues with restarting services. What should I name services? The scientist was pretty easy. How do I name the other services? So I thought to myself, there's no problem in distributed computing that a bigger monolith cannot solve. In the Kaiser right now. So this is the new architecture I came up with. It's called Paradise. This is a monolith in Go. Also a Coldplay song. It does everything. It writes to Prometheus and Grafana is of course used to read the results. So on the evening I started rewriting everything. I quickly got a nice monolith together. But there was something curious with that. So the Go version it also uses I2C to connect to the TF Luna just like the RIS version. All the commands were basically the same. It's like hiding and reading memory blocks. But oddly enough after running for a few minutes or a few hours the results got still or nonsensical. So in this case this measurement is exactly the same for four minutes and there was always a lot of noise. So that was quite strange. I restarted it. It seemed fine for ten seconds. Then it returned minus 15 cm. So it's below the ground at that point. Then it jumped up and it comes tail again. But when this happens I could kill the Go monolith and rearen the REST program and it got those same still measurements. Even using the command line tooling to directly read the results those same still measurements. And I couldn't get it to work. I started looking into the sensor. There was a reset command basically triggers a reboot of the full sensor. That worked for a few seconds but then it got still again and then I thought maybe the sensor is overheeding. I started adapting the time between measurements based on the temperature of the sensor so when it got too hot back off longer. In the end I couldn't get it to work en I was talking to my colleague Ruan again throughout the project we kept in touch and he was like you can't give it a taste of crap and then just expect it to not hate you for taking it away. For those unaware, Ferris that's the unofficial crap mascot of Rust and I think he was right. So I adapted the infrastructure again and this is the software architecture 3.0 We now have Paradise that is written in Rust that only takes measurements and writes them to nuts and yellow is the one part of the dualith that takes all the measurements and articulates the speaker updates the user interface and sends measurements to Prometheus and this turned out to be very stable this has been running for a few months now in our house we had never had issues with the sensor anymore so now came the hardware this is the top of the elevator shaft at this point the elevator is I think 20 cm from the actual floor so this is between floors this triggered an alert I'm standing on a step here to take pictures here you can see I think it's a backup electric motor to take the elevator down if the main motor breaks down something like a fuse box this is the TF Luna here you see wires running to a hole that was already in the ceiling so I think the people that designed the elevator thought about this use case and made sure there were holes for this and this is the speaker that I connected it is a Bluetooth speaker but I connected it with an audio cable Bluetooth adds quite a lot of latency and I wanted the time between starting to move and the music starting to be as short as possible and everything is connected with magnets on the one hand because I fucking love magnets on the other hand because I don't want to break the elevator and so I attached all this and I wanted to try out if the speaker was working so I SSHed into the Raspberry Pi and I started taking commands like speaker test some stuff with al some mixer and I couldn't get it to work I got strange like sacfold errors device not found, no sound card present and I couldn't figure out what was going on so I started googling and I noticed I have to update some kernel flags and I restarted I also had to recompile some things in the end I think I spent at least 4 hours but I couldn't get it to work and then a few days later I SSHed into the Pi to reset a paradise but the system D unit was not found what the fuck is going on then I realized that I have two Raspberry Pi's in my house so I was connecting to the wrong one there's one in the garage that triggers the garage door and one on top of the elevator and when I SSHed into the correct Raspberry Pi it worked immediately I will now give you a short overview of the elevator so this is the control panel iPad is not included just to show the user interface and if you look up the actual hardware there this is the corkscrew going up that is used by the backup motor to take the elevator down you see the TF Luna you also see two small lights because the camera sensor of my phone does pick up infrared but in reality you see nothing here because it's proper infrared and this is the speaker and the cables I will also show you the user interface it's a bit bare bones but it does update in real time and this is the full html slash javascript code for the user interface and it uses server sent events to update the user interface it's basically like a web socket connection but it only goes to the client from server to client these are go templates this is the index html page and you see that it includes the template height which is this template so when you request the index html you get the full page and then it sets up a service and events connection to the event stream and when a message comes in on that event stream this div here will be swapped with the incoming message and the service and event stream is basically this html snippet so it's the same template that gets included here it just gets updated with the new values this happens with code channels whenever the measurements change so it's quite real time and also if you take a look here it's very close to a rounded value again then the music patriek really wanted me to elaborate on muzak when I mentioned this project when we were having lunch about half a year ago he's like oh you have to talk about muzak he's like what is muzak basically muzak is to background music what google is for internet search so it's music with all the exciting stuff taken out of it so it's very pleasant to have it in the background so there's usually no one actually singing like the high notes and one of the first things I did for this project was actually starting to look in some free to use muzak but in the end you should also look it up cause it's like a lot of elevated music out there and it's like the really cheesy one it's like you should look it up but in the end I decided to use a different kind of music and the first one to guess where the music is from wins stroopwafels in the Netherlands and if you take this to the full stack stand you also get some nice things so please pay attention and just shout it out if you're recognizing also when the elevator starts it's quite a loud sound so be prepared for that does anyone recognize it it's too bad stroopwafels for me now Maxpain Maxpain it's the elevator sound from Maxpain I understand if you don't really recognize it cause I think the game came out over 20 years ago now but so actually I now have 2 out of 4 well maybe it's 3 out of 4 I still didn't break the elevator and there was only one more thing to add was the announcements and it actually turned out not to be that complicated cause I had all the logic in place I know when the elevator is moving I know when the elevator stops I have measurements of where it is so I know what floor it is first let's go over the challenges of this project so I think the biggest one it's the major one for me is using Go with I2C and the TF Luna I still don't know what's going wrong there maybe I'm triggering some kind of race conditions but I couldn't get it to work also SSHing into the wrong host when testing audio is a very big issue I also thought that I broke the Raspberry Pi several times I even bought a new SD card and I was about to plug it in when I got to the top of the elevator shop and I noticed that the network switch was not just disconnected so I plugged it in and everything started working immediately again I think two copies of the measuring software are running at the same time also produces some unexpected results this is cause I use a trigger based mechanism and a few times I didn't shut down a system D unit when testing and then there were two programs triggering the measurement and reading the result in tandem and that gives very weird results as well I suck at soldering so basically these three are all related to the real world I really like software cause it's much more predictable than having to do stuff with the real world another thing that was complicated once I started talking to embedded devices audio device is that cross compiling becomes complicated usually Go is very easy to cross compile once you have embedded stuff it just doesn't work so in the end I decided to use the VS Code remote development environment so I actually develop on the Raspberry Pi itself and I compile everything there it's a bit slower but it does work so it's a very easy trade off to make and another big challenge is now it's sniping myself I I'm still convinced that the rotation of the earth has something to do with the measurements it's also my next project coming up I want to measure it over a longer period of time to see if that also changes with the seasons and good morning and I had to tell myself also multiple times that I don't need a completely automated pipeline to transform pictures of the graphs that I take that I draw on a notebook into nice inverted graphics for in the presentation I did make it though it took me another day but I did get it to work so this is actually the end result and I will show you how it all came together so this smile this is truly the smile of someone that spent way too much time on doing this all the code is open source at ongithub feel free to leave feedback about the session with the QR code thank you all so much for joining