 Welcome all to how to make Cervesto with Python here. I have just quick while Give him some claps. Well, thank you for being here. Welcome all My name is just quick wall. I am a back-end developer at elements interactive. It's a company from the Netherlands I'm going to talk about how to Make a how to brew beer with Python more or less So there are some things that I would like you to to live here to leave this room with some ideas and Some of them are I I would like to know you to know how to build an IOT back-end Which are the technologies the protocols and the tools that are out there? And I will do some comparative so you get to know what's out there and also which is the one that may fit you best if You have to do a project like this Some back-end considerations that people will not tell you about But you as a back-end developer you need to make sure that they are taking into account You will meet a full running architecture for an IOT back-end And of course you will learn how to brew beer Actually, you're not sorry if you want to leave the room you can skip 30 minutes So first I want to talk to you about mini-brew which is the project for which we developed this platform Mini-brew is an amazing machine You can actually brew beer with it and you it will guide you since getting to know new recipes from a mobile app And then you can choose the recipe you want to brew It will be sent to your home And then you will you will just pour the ingredients into it and and the machine will take Care to do everything from the first until the last part of the process And it will also teach you how to how to brew beer even if you have no idea at all So at the end you will start being having no knowledge about it and after several brews You will be a total expert this machine comes with a mobile app The mobile app is going to show real-time data Of what the machine is doing at this moment. It's going to show What is the current temperature of the ingredients that are inside the machine? What is the current step? What is the target temperature in case you're in a very cold place a very warm place? And it takes longer to get to that temperature. Etc. Etc. It will also help you To see if there is any error in your machine. For instance, it loses connectivity anything like this It will also tell you and And It will also Allow you to start and stop brewing sessions. So you will actually only interact with the machine through the mobile app So this is not everything that the project has of course because otherwise I wouldn't be here giving a talk Actually, there is Python in the middle. Yes So well not only Python also some others Some I will focus the presentation on actually explaining to you what do you have to put in the middle of those two things? For it to work seamlessly So let's get technical First I want to share the project requirements that probably when you get a customer or you want to build something like like this You will you will have You will need to we will need to deliver real-time data. We want to have from the mobile phone We want to know exactly what the machine is doing in real-time on or real-time. That's fine as well But quite close We will need to have security in the communications We're all actually sending sensitive information because the recipe sometimes would be copyrighted and we cannot we cannot Let anyone see it We will need a frustration Because in case somehow the the channel got compromised nobody would be able to see what actually is going through the through the channel Authentication if we got somebody knocking at the at our API or our our platform saying it's a device Is it actually a device and if one device gets compromised can we disable it? To we communications of course because we not only get real-time data from the mini bruise But we will also send actions to the mini bruise. So we need both both ways communication Resiliency the system has to be resistant and if in case it falls it needs to go back fast because people expect responsiveness in there in their devices and also it has to be lightweight in the IOT usually we got and This is the case as well. We got really constrained environments And this makes us have to mind every kilobyte that we use in our in our project, especially in the in the device also, we have to take into account that the Packages that we send information that we send the bigger it is the slower it will be So that's also two considerations on lightweight that we have to take into account And some other project requirements that we may get us the last known status What happens if a machine suddenly goes offline? We need to know what was it what it was doing right before that will have to ask the buggy the bagging of the machine of course for the production a Production personnel staff they need to be able to take a mini brood that is not in correct state and be able to Debug it send different channels of communication only for debugging purposes We will need an admin site also for Being able to set up the users the mini brews their keys everything I'm a while at API. I don't need to explain that if there is an app Rainbows etc by this I mean that this is a project that is going on is growing So we have to take into consideration that we may get new requirements that we didn't get at the moment that we were Delivered the first round of requirements. Actually, this is usually in every project, right? But in this case as being a startup. It's more likely that this will happen Then the other thing that I mentioned before as a back-end developer You need to take into consideration some things that not always will come as requirements For instance, you need to take into account scalability, which is a word that is overused It's in every job offering. It's in every every company does a scalability every project is scalable But actually in IOT on this project. We have to take into account when building the architecture Proven technologies by this I mean that we cannot use a technology that is kicking right now And in one or two years is going to be obsolete. We cannot afford that Small tech stack. This is important for several reasons. It's going to be Easier to maintain for the people that goes after us. It's going to have less errors Connection tests are going to be easier. We're going to have a much easier life and Error tracking we've there is something going wrong. We want to be able to debug it This is the usual that we do in Python, but before entering a project. You have to take it into account Reduce the data transfer as I mentioned before reducing data transfer will do two things one make faster messages and the other will make Cheaper bills in in our clouds a provider And documentation the most important one because we all love documentation and we all love to write documentation So this is something that we have to take into account actually nobody writes documentation So this is the phase that you may you may do when you see this list of requirements and you never tackle the project like this So let's go. Let's do something and let's go step by step. Let's go step by step and let's start With the communications protocol This is this is a requirement that this is a decision that depending on how we make it now It will affect the possibilities that we have later to for technologies or software So we're going to analyze very fast the the protocols that are out there There are some of them that are already out of scope like we are not even considering them because of throughput But I will mention them that because you could you could probably build IOT like this like HTTP PPP DDS a MQP MQP is good for servers, but but not so much for IOT Then we got MQTT MQTT is one of the big players in the in the protocols for community for communicating with IOT It was developed by IBM and then given to the Eclipse Foundation for as open as open source MQTT is especially tailored for constraint for constrained environments And has some good things from a MQP for instance has quality of service where you can decide if you want to to deliver What the message one and I don't care what happens or you want to deliver message one and only one So you have a range of qualities between that. That's a very nice feature. It also allows routing for Based on topic on topics which allows for spreading messages through several receivers, but this is this is good There's another one. There's COAP COAP was developed by the constraint resources group at IETF and it's also one of the big players in in IOT protocol for yeah for IOT obviously so COAP is slightly different it uses more client server configuration and Uses HTTP burbs even though it's not HTTP. So it doesn't have its disadvantages Both are targeted for very constrained environments, but Even though they are very similar in that they are very different in some other things So for instance, they are very different in the way that MQTT uses a broker in the middle And everyone connects to it as a client. So then you can do books up and all of these kind of stuff and COAP uses a client server both ways. So then that's a big difference already Another thing is that MQTT uses long live TCP connections While COAP uses EDP. So this is already another big difference For these reasons actually probably you could do with both but for these reasons we we chose MQTT So now that we have our protocol, it's time to to look for a solution So out there you will find a lot of Already like companies that they claim that they have a backend ready for IOT So you don't have to program anything and that's kind of true more or less So we're going to analyze one because there's not time for for analyzing more I want to analyze one of them one of these comprehensive backend solutions and see if it fits our Our our requirements. So I'm going to to analyze IWS IOT from Amazon and see if it's actually a good Good choice for our for our case I'm going to go very very fast through the through the architecture because Amazon declined to sponsor my talk. So I will go very very fast so There's some things that we like from from Amazon for instance authentication authorization registry This one of the requirements that we had we needed to authenticate the the device the machines one by one Individually in case one was compromised Amazon provides that MQTT of course Here's the broker Device shadows. This was also a requirement for us. We needed to know the last known status Here it is provided out of the box And an API also we needed an API for our apps. Here it is out of the box as well The small disadvantage the small disadvantage being picky is that you're going to you're going to have to use a very generic API and maybe it wouldn't be as Reduced as if you would do the API yourself for only your purposes But anyway, it's documented because you will always wondering that probably and and it's out of the box ready for us some things we don't like The SDK well, this is a really small thing, but the SDK used a couple hundred Kilobytes and that was actually too much for our constraint device So we couldn't really use the SDK action. Yeah So in probably in another project, this wouldn't be a problem, but in our case it was Then there is the rules engine Amazon. This is not a problem in itself. This is good But it's good if you're using other Amazon products Amazon makes it really really easy to connect things between Amazon You can send information said rules notifications everything queues But only if you're inside Amazon, then if you're out you need to do quite more work to connect it to your external part of the platform Another thing that we didn't like too much is that the applications have to connect Using an API and an API would doesn't sound really too real-time, right? Having to do an HTTPS call every time that we need to to get the latest tick of information doesn't doesn't sound too good as well So let's analyze the project requirements as we had them Let's analyze what Amazon is is is achieving already out of the box. So we got security. It's a secure channel We got authentication. We got two-way communication. We got good resiliency Amazon claims. They are very good at that And I think they are we got the last known status. We got an API We got a lot of scalability as much as your wallet can afford We have proven technologies. Well Proven technologies Amazon has not been out there for long enough to be called the proven technology But I mean IOT but There's a full team working on it So we can be quite sure that it's not the support is not going to be dropped in one year or two years So we that's why we take the proven technology because if they have bugs they'll fix them At least we expect and documentation most important. So some things that Amazon doesn't provide us It did it. We're not happy with the way that they they were giving us the option of real-time data We don't have a fixation because the data that we're sending is Jason. That's the format that is used Lightweight for the small reason that I commented before that we couldn't put the SDK into our device But actually because we're using Jason. So we're actually sending more data than we should. I'll go to that later Rainbows, etc. I didn't market but actually a market read because it will be hard for us to implement new Requirements is that they do not fit nice into Amazon because then we'll have to do them ourselves Outside small tech stack. There are some things that we will not be able to do in that stack like admin side debugging All of that so what happened that we at the end We will have to have our own admin side our own stack of technologies for having that running And then we will end up having two big pieces of software the one that is related to Amazon and the other one So that this is why it cannot be considered like a really small And then the reduced data transfer if everything has to go through the server one and the other we're using Jason for this it's not going to be considered like a reduced data transfer so The conclusion for this is that you can use it But only if it really matches what you want to do. Otherwise, let's check how to set up our own Solution so first thing is to get a broker So we need to get a broker that supports MQTT and we have like a lot of options a lot of them There is every week or two weeks probably there pops a new company that has an MQTT server So I'm just going to show a few ones active MQ most keto and EM QTT burn MQ I've MQ cloud MQ rabbit MQ. So There's not a lot of creativity in the names for these kind of companies But I mean you could choose any that you wanted some are Pay you pay them and they are already Deployed the others you deploy them yourself But we chose rabbit MQ for going very fast for some reasons Well, it's been a top player for many years. It has scalability proven both vertical of course and an horizontal It can convert from a QTT to other protocols, which is a goodie. It's not something that we require But MQP is something that that servers like There is no payment for use and we were familiar with it So these also counted so that's what we chose this but probably if you chose any of the others it would be good as well Now we got an extra bonus for doing the the broker ourselves And is that now we can do pups up and the minimum we can send the information to rabbit MQ And the devices all of them that they want to listen to the actual brewing session They can be listening to to to rabbit So what the information for real-time doesn't need to pass through the server anymore? So what is what is Python doing while the devices get real-time data chillaxing? There's no need to do anything Now what? Let's talk to that broker. We got a broker right, so we'll use Python now finally There is a library from Eclipse from and I promise myself I wouldn't show any code because this talk is beginner and also it's it's very short in time But now that I already showed it. Let me go very fast with it so It's so easy as you only need to import the MQTT module and then you just set a Few callbacks on connection subscribe to a topic on this connection on message do something with that message You connect to the broker and voila. You get it But it's so easy to get an MQTT server in Python So we got one thing fixed Time to look at the API In Python, of course So options here I could go through all of the options and actually there would be a lot of fights if we started that discussion Only discussing this slide could take like a full talk and only discussing each one of them would take a full talk as well It's one of them. So I'm going to just go very fast and see which one which shows okay, so bottle tasty Django tasty pie flask Falcon Django rush framework And there are even more Actually, we chose Django rush framework because we of course we were familiar with it We already had a part of the API with that And and well, it's scalable. There is loads of documentations of external plugins and apps And there's everything you may need you can cache everything So it's a really good and now with the latest version that I just got to know a few days ago There's really good There are some goodies like automatic documentation and stuff that you should really check out So that's it. No, sorry okay, so How are we doing right now after the choices We got real time data now. We got security because through broker you can also connect to SSL To we communication thanks to MQDD and we reduce the data transfer by skipping the going through the API Some things we also got is the last known status because we because we program it Now it's our API We got the bagging we can use the bagging and we can use the admin site to help the Staff debug the machines. We got the admin site with Django, of course, and we got the the API we mentioned We also got error tracking because we are familiar with tracking errors with Python So that can be considered that it will be easier for us to the book Python than to the book any other stuff so some of the things we already got by using those those two is Resiliency and scalability, but this depends of course on how you do your DevOps how you're going to deploy You're going to the plane clusters and etc. Etc. But it's full of documentation of how to make the system scalable because they are very famous technologies We got rainbows, etc. Sorry. You got rainbows, etc. because Now it's Python and we can do everything we want with it. That's we all know that right and Proven technologies, of course, otherwise, we wouldn't be in this conference small tech stack Now we got only these two pieces of software and that we really understand And some things that we still don't have We don't have a fiscation the messages are still being sent to Jason We don't have authentication the everyone can connect to the broker and just start messing missing up there And we cannot say it's lightweight on the device because we just sending messages to MQTT. That's super light We don't need any library for that and and it's also But it's not lightweight that we're not sending the small amount of information. We're actually sending more information that needed and Documentation, it's fine guys. It's full of documentation So let's start solving the remaining parts authentication in Python as well This there is a really nice plugin for rabbit MQ. It may sound a little bit tricky. Let me explain it It's a rabbit MQ authentication back in via HTTP Okay, what does this mean? Now when the device When the machine wants to connect to the broker instead of the broker just saying yeah, you can connect or you can Connect it will pass that information to Python and Python will check the database and save That the that machine can actually connect or not. So then we don't need to have all that in the broker It can be in our normal database nice And response and remember it's a long-lived TCP connection. Just once that's really cool This works for devices as well. So we checked another one. There's authentication now So let's go for the last one a few skated and lightweight messages So, how are we going to do this with Python? No with Google. So If you don't know protocol buffers yet, you should if you use any kind of API It's a protocol from Google that actually Well at this could take also another talk just explain But I will try to to just go very fast and and explain it but as you see in this example Jason that the device could send to to the server or or To the machine could send to the server or the device you see there in each one of these messages There's going to be a lot of repeated information You're going to send the ID every time the ID string You're going to send the timestamp by string You're going to send the data action sensor 1 sensor 2 disease Information that is going to be repeated over and over and over and that's going to consume data And that is also going to make it slower. So we don't want that. That's also is really easy to read How are we going to solve this with protocol buffers protocol buffers actually what what what it does is that This specification of how the message is built is shared between the parties that are communicating, but it's not transmitted through the wire So actually you got the specification on both ends and you only send the data that is different every time If anybody from Google was here probably they would shoot me because it's not really like this But this is an example. So so you see actually what is being transferred. Yeah This Google also didn't want to sponsor my talk. I don't know why but so I'm not going to show you how it works but this is Really easy, you know, just build which fields your message will have if they are required optional So then you can change the requirements They are always backwards compatible and then you will not have problem if you're upgrading the message that you sent and somebody's using an old spec So how are we doing now? Of course, both things checked We are happy with the solution now This is a very very small final architecture very very simple, but You can see that we got the web servers We got the the three blocks that we talked about some endpoints for rabbit MQ and MQTT listener and responder and then we got the Django API For the for the devices and we also got the rabbit broker for for devices and machines for the web servers for to communicate Of course database gets cash and and everything that is usually just in a in a web in a web platform So the everything is scalable because everything is in the cloud. So and everything all of the toast technologies are easy to scale So this is it This already it's I was Probably my boss is going to watch me on YouTube So I I want to tell you that I will work at that really cool company called elements interactive We work at Barcelona in Spain and Almeria in the Netherlands. So if you want to do cool projects like this Be sure to enter and or just contact me after the talk and I'll be happy to to explain what we do and how we do it So, thank you very much for your attention. Got any questions. What about unit testing? I hope you're right. Sorry. What about unit testing? Yeah, I hope you'll write them, right? Yeah We got them. We got them Actually documentation the question the question is which Test runner are you using? We are only using the default one. I mean, okay Is it a pie test or no test or right now? We were using the test cases, but We were thinking of trying by test, but we still didn't do it So right now we got a very very good coverage, but on test cases How does the beer taste like? Hi, I have one question Do you have a hardware embedded security because I wonder what if someone hugs the API and make like an increase the Temperature impression makes like a remote bomb essentially Or set the mini-brew on fire, right? Actually the mini-brew is also intelligent and and it does a validation of all the recipes and so there is no There is no way that anything that could break the mini-brew is accepted Actually, if you send a recipe to the mini-brew and it finds it it's not good It will just answer that it could not be started with some information, of course But but yeah, there's no no risk for bombs if you want to buy it It's safe anymore Have you thought about using thrift instead of Google protocol buffers if yes, then What's better about protocol buffers? I have I don't know protocol buffers better than Sorry thrift Apache thrift. I Haven't I don't know about it. Okay. We were pretty happy with protocol buffers. We because they had everything we needed the The benchmarks were very good. So actually we just found it. This is really cool. Stop here So actually we didn't try that. Okay, but I'll write it down I was just wondering what kind of data do you send from the mini-brew? It's it's actually I'm I'm I don't really know exactly because that's more of a hardware thing But there is like a zillion sensors around all the mini-brew pressure If there is water flowing around which is the current status if there is action spending from the user What is the next thing that it's going to do? I mean, there's a lot of information Usually that the Jason I just showed it was like a ridiculous example because it actually may much bigger So so yeah, that's Have you tested what the scalability limit of your stack is have you tried to topple the thing? No, not yet. Not the top. We're actually working with mini-brew sending Like if I don't remember wrong. It's sending. It's mini-brew that we're testing there quite some of them quite some PCBs testing at the center and Each one is sending like 60 times more information that it will send in in the real world because we're really really logging everything into a very Detailed so right now we can handle already with a really small Cloud architecture and I mean really small We can handle like hundreds of mini-brews already. So I when we go to when we just make a normal a Production deployment it's going to be able to handle Thousands and of course we will we're going to implement the scalability clustering and everything Thank you all for coming. Thanks, Jessica