 Εντάξει, ευχαριστούμε πολύ και εγώ, και αυτό είναι γιατί είμαστε φίλοι για αίσθησης. Και καλύτερα, είμαστε στην πρόσφυγη. Βέβαια, όχι. Και η δημιουσία είναι ότι καλύτερα είμαστε στην πρόσφυγη. Και έγινε στο ίδιο στιγμή στη Βατουρίλιο. Όταν δημιουσήσαμε αυτή η πρόσφυγη, κάποιες χρόνια, Άρξιο asked us to create a fully operational IoT core for a field management purposes. You know, cars sending signals and the user is able to control his fleet. For those who don't know IoT means Internet of Things, device sends some data packets to server and the server is able to provide top indications. At first we thought that this was a small step for us and a great step for IoT. But this was only the first impression. Back in reality when we tried to design and implement our core we came across several requirements as we expected. Of course, in our way our integrity was difficult. About these requirements now. The first one was that we had to use a plugable device in order to get all the signals from the car and send it to a server. The second one of course was to found an IoT core, a server basically which can partial this data and store the base. We conducted a fair amount of research and we found cheap and reliable solutions for devices, OPD devices for us with enough documentation to install a module inside the server. And of course we also found cheap and reliable solutions for IoT cores with live servers. This wasn't so hard. We purchased the device, implemented module according to the given documentation but we also came data packets to our server. Our main requirement now was to create a module that some services basically which will be able to save all this data to our database. Of course, as Pythonists, we use Python for this purpose but then we weren't as experienced as we are now and we made a very simple Flask application which can be easily found on the internet according to main tutorials so we will not share this code. Though we are afraid that we might fail we tried and built this application. We named this module ingest because it receives data and delivers it to the database, it feeds the database basically but if something went wrong back there we may lose our data. This was the main problem now. In general, as you can see the interesting part for an IoT core is the same, no matter what language, no matter what technique you like. A device sends a data packet to a device gateway then the device gateway parses as data it sends it to an ingest module, to every ingest module and then the data is saved to the database. So, sorry, it looks good in an ideal world without error cases or bugs. Unfortunately, in the real-life world where we live programming projects are characterized from imperfections and flows. For example, what we can do when a data packet fails to be saved in a database. Here comes the midi cloud. This question is answered by single-sendage with many perceptions ensuring data integrity. Of course. We think that it's important to create your own scope before your development. For us, data integrity comes with two basic principles. First of all, and most important, correct and not unintended storage. In other words, no corrupted data, no duplicate data and everything saved in database. Second one, ensuring data quality. Look deep to your use case and check your database periodically for quality errors. With quality errors, we mean the correct value which doesn't make sense. For example, speed comes with negative value. This, in the long run, will prevent you from bad user experience. We'd like here to note another two important principles. This may be outside of the data and data definition, but from our point of view are equally important. First one, services integrity or working services. Take this one seriously. Just think. If a data packet is corrupted, you will lose only one data packet. If a service in a chain of services and modules is down, no data will be saved in your database. And second, devices integrity or working devices. You have to look for some important characteristics in order to choose your device, which will be integrated with your server. Please be careful with this one. Here we think that is important to tell you that for us, data integrity has nothing to do with data security on this level. We may disagree with us. Data security is equally important for us too, but it will take double and more time to include it in this presentation. It is also a step forward for the junior developers who want to be involved with IoT. So, here and back again, as Bilbo Paddings used to say. Actually, this talk has so ugly title that it could be combining Python with blockbusters without any bits. It could be also IoT tips and tricks. Before this small scope explanation, we are ready to tell you about our first implementation for data integrity inside the IoT core. Of course, our first thoughts on how to deal with service downtime or corrupted data, temporarily saved data to our file system, basically our failed events. This was the main code of our very, very first ingest module. You can see a while loop running, parsing some files which have inside failed signals from the car. We thought back then, yes, we've done it, that's solution, but no, not at all. It wasn't a solution for us. We got some metrics and we had about 95 or 97% of CPU usage. Quite a number. We even tried to put a sleep command here, but it didn't work. But we learned a lot back there. And I think as master code, I used to say the greatest teacher failure is. So, after some research, we ended up combining new technologies with a Python program. First of all, we refactored Python ingest with multi-threnting. So now every time our signal run on its own thread and we have concurrence to many new words. Let's make a play. First of all, asynchronous. What is asynchronous? With asynchronous programming we respond to requests outside of the main program execution. On a high level, this means that when a request comes to an instance or a module, it will be pushed in a message broker or something like that and will respond later. For us, it was very important to understand this term. We reduced our load and we had better performance because we could do heavy tasks later. Next term in this queue of new words is concurrence. You can see the first example here with no concurrency that in order to start the second task, the first task has to be finished. What you can see on the second block. In a system with tasks running, the second task will not wait the first one to be completed. But it will be executed not necessarily at the exact same time. This means that the first task starts in its own thread and then a second thread starts the second task and they nearly enter the same time. But what is a thread? On a high level and from our point of view the unit of process which can run independently. Of course, there are multiple ways to achieve concurrency and as a result you will have better performance of your IoT core. Here we included multithreading and async.io as we already said multithreading requires multiple threads and different tasks not necessarily at the same time. On the slide we try to represent at least we tried I think here. We tried to represent the main difference between multithreading and async.io. As you can see on the first example we have three tasks. They nearly end and start at the same time. On the other hand we have async.io. Async.io is about a single thread running some tasks. It can stop and resume the execution of each task according when it reaches an await command. So in the example here we have task 1 which starts then it's called reaches an await command so it will stop then task 2 will start and task 3 and this will continue until all tasks are finished. Now on the other hand for our interesting part we wanted good performance that basically our car signals comes and has to be saved as soon as possible and here we used multithreading for the first time but could we do this with multiple processes and not multiple threads? Of course, after a specific point this would be necessary for your system. You see you may reserve more resources than even if you use multiprocessing for the first time. Our advice here could be this if you know what a single thread can do then you will understand easily what multiple threads can do and if you know what multiple threads can do you will understand but a single process can do. Then it's easy to understand what multi-processing is. Back on our code as I said we used then back then multithreading we chose a thread pull executor in order to limit our open threads that's exactly why we are adding max workers 3 because we wanted to have a limit on open threads after this we had a simple socket in logic we planned to report while listening for incoming data if this data counts we will have a dangerous executor we submit this data to the executor and if it fights an empty thread it will save the database so here maybe you could do a small recap just but first I would like to tell you that threading without limit could be a little bit dangerous inside the system but I agree Sure Till here we discussed about the interesting part in this part a data bucket comes from device to device gateway then device gateway creates a socket connection and sends data to python ingest and then python ingest try to save it in database and of course our solution for a concurrent and fast interesting part was multithreading with a thread pull executor we also discussed about but that's something that we will expand later also we talked about the device integration part there are multiple solutions with higher or low cost according to your needs we would like here to say that we have already prepared to get her repositories with all the code we are showing in this slides the first one is about the python ingest module and the second is about IoT data integrity module we have made it with a simple make file where you can install three containers and then you can have a small IoT logic running inside your local PC now after this important works back on our main thoughts as we said back then when we were junior developers we were confused on how we can guarantee that all the signals no matter what the services condition is they are stored in database for this we connect both device gateway and python ingest with a message project our message project is rapid mq as you can see but what is rapid mq on a high level and rapid news stands as our middleman which temporarily keeps our signals after a failed save attempt and of course it has two main actions publish and consume with rapid mq or the message project you can build various solutions but in our use case it was this specific one now about the publishers so if a message has to be published to rapid mq we have our publisher responsible for this task as you can see on the slide all these four steps connection, channel, declaration and publish are required for this process back on the code we used pika to connect with rapid mq pika is pretty reliable and has good documentation of course you can choose other libraries to connect with rapid mq here back on the code we get a channel of the created connection and we try to declare a queue not that the queue maybe already exists we would like this queue to be durable in order to remain there if rapid mq is started also we would like the message to be persistent for the same reason that's why we added delivery mode too finally we send the message of our singlin to the new by using basic publish on the other hand we have the consumer part this consumer part is part of a module that we will show to you later in general consumers are responsible to fetch all published messages from rapid mq again we have to create a connection to get the channel of this connection and of course we are waiting to get all the iconic messages from rapid mq because all the signals are coming as messages in this queue and this consumer so far we explained separate modules each of them has its own importance but they are not combining a simple very simple iot core we wanted here a module that will or a service which will orchestrate organize all these different services inside an iot core and will encapsulate also some tasks to ensure the previous principles we used here a syncio for this purpose you see our interesting part has not so many failures and we nearly had the thread waiting all the time for the incoming messages so we wanted to assign some more tasks here if you search around the internet for a syncio you will find a term cooperative multitasking that's exactly what I am trying to explain to you we have an event loop running as you can see in this slide it gathers some tasks and presumes the execution of its task so a syncio is about an event loop running some tasks this can be done with multiple tasks concurrently through the use of a weight command here in our code we create an event loop called event loop no imagination I call here we set it up as our current loop and we gather all tasks with the syncio gather command and of course we run our loop with the run for ever command of course there are multiple ways and various solutions here to start a loop but you will practice and you will learn how to start your own loop now the total flow of our very beginning IOT code will be this but believe me this flow works we have the organized module here it awaits to consume some incoming messages from Rabbit and QQ then it pushes back the Python interest on its free time it applies all checks with periodic tasks to the database it could do a lot of things I think here it could be monitoring the server many other things but we would like to show you some of our ideas before those ideas fans I think that this flow looks better than ours our flow is not so beautiful why? I think George here the arrows are more green yes this here are a little bit darker would you talk to our team for color accuracy so about our ideas the first idea was to create periodic tasks think about a while loop running the task and stopping it for a specific time out as you can see on the top of our code an infinite loop is running upon every iteration it stops for a specific time out but how we can benefit from periodic tasks so far we only show the first and most important principle of our scope correct and not intended storage on every failure offer the interest part a message broker take the signal now with this example here we check our data basically our cut signals for quality issues not that this data is already stored in database this will ensure the second principle of our scope data quality imagine for example a broken device sending values for a longitude or latitude which are unexceptible it will be a serious user experience problem as you can note on the code again their own coordinates method checks for unexceptible values of coordinates we may disagree here someone will say that we have to check for quality before save we suggest not just think that checking for quality before save will reduce our interest in patch performance so we implemented smart and periodic checks after signal save on organized module you have the time to do this checks but you have to be careful you should avoid checking the same data again and again for quality issues another catch with quality checks is one of our additional principles device integrity if a device sends you your own coordinates all the time it has to be chained so here you may want the mechanism to inform your user or your customer support department about this problem so now till here 3 or 4 principles are sold correct storage with message protein quality with quality checks and also devices integrity with the same or even different checks of course George and as you can guess the last principle of our PPSCO service stability or working services i think here this could be done with various programming with multiple languages or anything you like but here now for example we wanted to show you all functionalities gathered inside the simple module also we have a list of services we bind report if this port is already used and services up i think it's not completed but you will catch the other meaning of this method again we have a periodic usage here with a while loop and async IO slip command with a given time out but here George i have a question first do we need so many modules inside our flow of course another step forward here maybe a little bit outside of the junior maybe a little bit outside of our junior days you will see this flow actually now our rabid MQ is our middleman it has 3 queues of course our main queue which passes all income data to the python ingest we have a dead letter queue where the unacceptable messages are saved and will not push back to the ingest and at least we have the failures queue where failed messages will push again to the ingest of course here we have no organized module it's been merged with the python ingest which can do both things since the events will be saved to the database and of course apply some periodic checks but here George what do we need to code everything maybe you could use some ready solution some tools that will help us sure like cellar here you could actually use plenty of solution in order to save the time and resources cellar could do this work for you and we actually use it but let me tell you this it was very important for us to understand what is actually happening inside our IoT code don't be afraid to break it just practice first you have to understand we would like to wrap up this presentation and say that we did it to show that IoT is possible then if you don't have so much experience don't be afraid to be involved in such project the only thing we know there is no perfect solution so we would appreciate important feedback from more experienced developers on this subject nevertheless we think it's very important for everyone to always look for better practice and solution so this is it Fanes no more bad movie inspired humor George at least we tried but some friends from Hollywood will have serious problem now and of course thank George Lucas for his movies before we say goodbye here we would like to thank some special people for us of course our team of Beturilo developers for their unlimited support and for their important ideas those are colleagues from other companies Bill our system administrator and of course our rest of starter community for their daily support of course here we have Harris all the content is in the scripts before it is his as we already said we prepared to give her prepositories with a very simple IoT logic I think it's important to expand it to clone it to refactor it and to know what could be done inside an IoT core I think George when we first used Severely we didn't even know what this tool can do after some fair amount of debating then we realized what we could do with this task queue so I think this is it if there is any questions we are ready to answer them ok thank you thank you guys so yeah we have time for questions so if anyone wants to ask talk to Bihar only the first presentation for us yeah I hope you will enjoy it so I have one question from Javier Martinez he is asking who do you who do you do when you have to pass messages between coroutines we did this with consumer pass messages with IOM QP it could be found on the second of the first Github repository we have all the code gathered there ok cool any other questions I can also open the microphone if someone wants to ask let me tell that we try to use multiple solutions here because we want to show you all possible ways to achieve concurrency the idea is to to help you think how you should think about the solution ok one more question why rabbit and not Kafka of course you can use Redis, you can use Raptem too you can use Kafka we don't have real time data so we can use Raptem I think if we had live writing on our fleet management this could be done with KF2. ok the last one, so we have time if anyone wants to ask one more question ok i think that was all, congratulations thank you very much thank you for your questions thank you