 I'm going to be talking about basically IoT applications today. I put the most click-baity title I could find, so that my talk gets selected. A couple of years ago, a year ago, I worked on a project for a large enterprise client, where we built an application for home security automation on Azure. So this talk is just a distillation of our learnings from that experience. Before I get started, how many of you have worked on an application related to IoT, or have tried out things in that area? Okay. Quite a few people. So if you're new to IoT or you're not like sure if this talk will be relevant to you, I split the talk into two parts. The first part is an introduction of what a basic IoT application looks like, and the second part we would get into some details about how a solution like this can be implemented, right? So before I get started, what's an IoT application? Things talking on internet. Yeah. Kind of. Different things, devices which are communicating to each other. Why are the internet? What's that? Aircraft. Aircraft? Aircraft is a bunch of IoT. Yes. There are applications in aircraft which use IoT. Even your car has probably lots of sensors which are continuously sending information to the cloud, which either helps the user who's a driver in the car or it could have other applications like car manufacturers are constantly collecting data from your car which helps them build their next car model, right? So there's a lot of examples of IoT applications. The example that I've taken is, let's say you go and check into a really fancy hotel and because it's a fancy hotel, you get access to an application on your phone, which they allow you to download, and that application gives you access to some controls in the room. So you can use the app to maybe turn on or off lights in your room, or you could have more complex applications like you can check the temperature in your room, maybe you can remotely control the AC, you can turn it off if you forgot to turn it off. If there's a locker in your room, maybe you can control it remotely through your app and see if it's still locked or not. So you could have any number of use cases, right? So if it's a hotel, then there could be many users like this who get access to the app who are controlling things in their respective rooms. They could be administrator in the hotel, who is looking at all this data that's being collected, which helps him understand how the energy consumption is in the hotel, and what needs to be done or if someone's left their lights on or things like that, right? So this is an example application. So we're going to dive deep and see what are the different parts of this application. So there's actually six levels in the application. The first part is called sensors and actuators. So these are actually edge devices which are there in your room or in the physical space. So if it's a car, it could be sensors in the car. If it's a room, it could be a contact sensor, or it could be a light sensor or temperature sensor. So there's two classes really. So sensors are devices that are measuring some physical quantity converting it into electrical signals and either storing it or sending it somewhere else. And actuators are devices which actually receive input and convert it into physical signals. So an example would be, I think I have an example. An example is, for example, you could set up something in your house which says that whenever the temperature goes about 25 degree Celsius, automatically turn on the air conditioner. And if the temperature falls below 20 degree Celsius, turn it off. So the device that's measuring the sensor, that thermometer is a sensor, and the actuator is actually something that's controlling the air con, turning it on or off. So these two devices are actually good enough for basic applications. Like if you have a reverse parking sensor in your car and it's wired to the camera in your car, it's good enough if your sensor is directly connected to the actuator because that's the range of functions that it's expected to do. But if you want to do slightly complex functions, then you introduce the next component which is called as a device or a control unit. So some common things about sensors are they're usually really small devices, low power, low capacity, they can transmit data only to a limited range or bandwidth. The devices or the control units are slightly more powerful devices and depending on how much money you're willing to invest, this could be like as powerful as a really small computer. And these devices can have business logic in them. They can do slightly more complex operations. They can execute rules, right? So the control unit can do some sort of localized decision making and data manipulation. It can connect to multiple sensors. So if you have, if you want something to be a function of multiple sensors, the control unit would be the device that decides what needs to happen, right? So we'll take a slightly more complex use case. So you are in this hotel and each room has a bunch of sensors and a control unit in each room which controls what happens in the room. So it's continuously getting data from the sensors and maybe doing something with the actuators. Now as a building administrator, you might want functions which are at a higher level, not just localized to your room, but maybe at every floor. So for example, if there is a fire in one room, you might want everybody to be alerted immediately and maybe the sprinkler to turn on immediately on the floor. So you could also have higher level control units which connect to multiple lower level control units. These guys also double up as adapters and gateways. So what these do is your control units can talk using different protocols or different contracts. So these devices will have the ability to be the adapters which translate this information into one type of information. They're also the devices that connect to the cloud as you can see there. And so if you're building like a platform on the cloud, these are important because your cloud application can be agnostic of the kind of data that's being sent in because the control, the edge gateway here takes care of the translation becomes the adapter, right? All right. The next level is the cloud services. So the cloud services are really important because the more mature your application gets, pushing all the business logic to the cloud lets you, gives you a lot of control actually in what you're doing with the data you're collecting. So the use cases that you can use the IoT devices for almost grows exponentially when you push all the logic to the cloud, right? So for example, our application was to, so we built a product which you could buy off the shelf. So you go to Walmart and you buy this product. It comes with four sensors. You can go home and set up the sensors. So these are contact sensors. So you set them up on your windows and doors. So if someone breaks into your house when you're not there, this sensor would automatically trip and it would send a signal to a control unit. So the control unit was more powerful. It would receive data from the sensors. It also had a camera. So it would take photos of the house at that point and it would transmit the data from the sensor and from the camera to the cloud. So the cloud can do a bunch of things. So the cloud basically stores the data from the camera separately. It creates an audit log of the event that happened that the sensor got tripped at this time and alarm was raised at this time. And based on how it is configured, the cloud can also do, so it notifies the end user through an app on the phone or something but it can also like call security services. Like it can notify the police or the building watchman or whoever. So the application, the complexity of the application you can build can actually grow exponentially if you push all of this to the cloud. The last part is the mobile and web applications. So basically the cloud service provides a platform on which you can plug in multiple channels. So it need not even just be mobile web applications. You can have voice activated devices. So you can have, you can say things like, I don't know, Alexa guard my home and maybe it'll enable things. And when you are home and you don't want the alarm to go off, you can say Alexa, I'm home. And maybe you get a notification on your app where you enter a passcode and the house is, the security system knows that it does not have to guard your house anymore, right? So the applications can be any number and because they integrate with the cloud via an API, you can just add on channels to the level six, right? So putting all of this in perspective, this is what we call an IoT stack. So there's six levels, starting with your sensors, activators, devices, going up to the adapters on-prem gateways, the cloud platform services and the mobile applications. Okay, so apart from this stack, so for the purpose of this talk, we are going to specifically focus on scalable software platforms and that means that we will focus on level four and five, like what goes on in the cloud and how do we architect the solution, right? So the reason we call out the cloud application so specifically is because I think we're all used to building web services or web applications at scale, but when you're building an IoT application, it's slightly different. It's not another crud application. It has more complexity built into it because of the IoT ecosystem. So there's four things in my view that distinguishes this from other applications. So this is the scale of the application that you're dealing with and what is the representation of the edge device in your cloud? What's that going to look like? And there's like different ways to model this and based on your use case, you could build your application to be effective. Security is obviously a big concern. So how do we protect our application? Especially when the devices are no longer in your control. It's not a server that you can SSH into or it's not, you can't get logs on Kibana or whatever, right? So we have to figure out how do we guard this application? And the last one is more of an open question. So CI CD is continuous integration, continuous delivery. So when you ship the device to end users houses or when you ship it into a car, how do you continuously upgrade the device and how do you make sure that all the connected systems are working together when you roll out an upgrade, right? Okay, so let's jump into scale. So the application that I just spoke about, the home security automation application, we essentially shipped it to many parts in North America, which means that the end users bought the device. So they had lots of contact sensors, but they also had the capability to add more sensors to the solution, right? So we were getting a large volume of data and so I'll give you an example of the data. The devices would actually send pings every five seconds to say that I'm alive, I'm alive. This is my state, I'm alive. Especially because it was a security domain, we needed to process these pings because in case you don't get a ping, that is also a breach of security, right? So we were getting constant data every five seconds. And the application was only going to get sold more and more, which meant that the number of devices that were connecting to our platform was going to increase more and more. So we did four things to deal with scale, four specific things. So this might not apply to all use cases. This is what worked for us. So the first one is the gateway there. So our application was actually deployed on Azure Cloud. So we actually leveraged a lot of managed services that are available in Azure. But there's actually equivalence to all of these on AWS and GCP. So you can choose to go with whatever solution makes sense to you. So the gateway is actually literally the first point of contact into your cloud application. So we use Azure's IoT Hub, which is like a queue, but it comes with a lot of advantages like it comes with 99.9% availability. It also comes with an API that we could use to connect to our devices. So basically it can take care of device registration. So when a new device gets added, the gateway actually knew how to identify the device, register it into the ecosystem and give it a unique ID. So if I wanted to say, turn off the AC in ThoughtWorks, I can just give a device ID and the gateway would translate it into the AC in the kitchen on the 11th floor of this building, whatever. So it abstracted that out to us. It also helped us with bi-directional communication. So in an IoT ecosystem, there's actually two types of operations you can do. The one's called a command and other is a query. So a command is when you are trying to change the state of the device. So a command is turn off the light or turn on the light. And the query is when you're trying to see what the status of the device is, right? So is it on or off? That's a query. So we would send a command to the device and the device would respond back with a state change. So the gateway handled by directional communication for us and it also provided some sort of abstraction over different types of protocols. So that's the first thing we did. The second thing was we had a service to process telemetry. So what is telemetry? Does anybody know? Any guesses? Something I'll log. Log, it can be used for logging. It's kind of like data, metadata that the device is sending you, right? So it's just essentially saying, I'm alive. This is my state. So depending on your application, you could send any data that you want as telemetry, but you would usually try to keep it minimal because you just need to know that your device is out there functioning, right? So the telemetry service is like the first point of actual contact in our application. It was a service we built. It was a microservice. And it processed the continuous stream of input we were getting. So this ping was sent every five seconds and the telemetry service would receive it, see if there is an anomaly. If nothing, it'll just store it and move on. So this helped us because if you wanted to scale this at all, this was the first point that we would scale, right? Instead of scaling all the other services, this is the first point we'd scale because the amount of data that the telemetry service actually pushes to the rest of the services would be less. We could also put any sort of complex rules we wanted in there. And it took care of updating the latest state of the device. The other thing we did was we had a time series database which would capture the change in state of the senses over a period of time. So one of the other use cases we had was to show this in a dashboard where you can see like what your consumption of energy was like over a period of time. Or if you want to reduce how much energy you're consuming, you can compare to what you had last month. So this time series database, it uses time as a first class citizen. So any data that we wanted for this dashboard was pushed into the time series database. We also did caching, which was... So actually there's different techniques to do caching. So we'll talk about this a little bit in detail in the next slide. But essentially this is the same rule of thumb as you would use in any software application. So any pre-processing we could do, we did it and pushed it into the cache. And some use cases when the rest of our system went down, the cache was actually up and alive. So the end users were not affected. So these were four things we did to deal with the scale of the application. Specifically in terms of caching and telemetry, we used certain models to represent our device on the cloud. And this actually helped us in our application. So I'm just calling out three specific things that we tried. So the first one is called as the actor model. Has anybody heard of the actor model? Yes. So this is loosely related to the actor model in Akka, where each device is represented as an actor. So an actor is essentially like a long-running process. Each process has the ability to store its current state. And if there is no processing happening, it will go to sleep. And when there is an incoming request, the process will wake up. It will fetch the last known status from the database, and it will be able to process things. So we actually tried four versions of the platform. So in one version, we try to represent all the devices as actors. And each actor has a queue into which you drop a message to communicate with the actor. So the advantage is if we have multiple users, when we have multiple users, each user's device would be a separate actor. So there's no context that is shared across actors. So your data is within the bounded context of the actor. And you can also easily model a hierarchy into an actor. So if you see the control unit can connect to two types of actors, a thermostat actor and an AC actor. It can also connect to a fire alarm actor, a sprinkler actor. So the control unit can actually send messages to the actors to perform actions. And because we did this on the cloud, we were able to persist the state of the actor and retrieve it easily. So this is another view of the hierarchy of actors in the application. So if you are trying to represent a hotel, the hotel actor would be at the top. Each floor would be an actor. Within each floor, each room would be an actor. Each control unit would be an actor and so on. And this is an example JSON representation. So if you see each actor will have certain generic fields properties. It will also have some specific properties. So the floors are children of the hotel actor. And they are specific properties. But you also have ID, name, document type, which is common for all actors. And your document type would tell what type of actor it is. So this we actually use this in conjunction with the Cosmos DB, which help store the document, retrieve it very fast, and process the data. So the next thing that we tried was to use something called a device twin. Has anybody heard of a device twin? Has anyone heard of shadow devices on AWS? Yes. So it's kind of similar to that. So this is an Azure concept. So it's actually very straightforward. So it's a physical. So you have a physical device. This is the virtual representation of that physical device. So you have properties. So each device will have properties. And a property has two components, a desired and reported properties. And it's got tags. So what would happen is, let's say you have a light bulb and you want to turn it on or off. Right now it's off. So the reported property is that the light bulb is off. And if the user wants to turn on the light bulb, they'll use the app and they'll say turn on the light bulb. This command would come to the cloud, which will change the desired property to on. The reported is off. So this would be then sent to the device. The device understands that it has to change from reported it would change the state. And then it would communicate back. Then reported would be off or on, I don't know, depending on what the user wanted to do. So when the user wants to query the data, the data would be returned from here. So I have asked the light bulb to be off. And then I just go doing my business, having pizza or whatever. And then later on, if I want to check what the state of the application of the light bulb is, I just look at my app. And the app is actually fetching the data from the device in the cloud. It's not going to the actual physical device to get its latest state. So light bulb might be a very simple representation of this. But when we are dealing with complex ecosystems, complex control units, which connect to multiple senses, this is kind of like a cache, which is retrieving the read data from the cloud itself instead of going to the physical device and fetching it. The other aspect is your device state can actually change very frequently. And it may not be controlled by the user's inputs alone. It could be environmental. It could be other factors. So every time your device state is changing, it's actually stored on the cloud, but the end user can fetch it and read it when they need to. Right? So the design state is different to the report state. If you... Yes. ...the state is syncing. Sorry, could you repeat that? So I say turn light off. But that's my design state. Yeah. Now, why ask the cloud? Is the light off at that point? Do I have to go to the device to check? No. So we assume light off even though we're not sure what light off? Yeah. Or before the light's actually off, before we get acknowledgement from the device that the state has actually changed, it will actually say that the light's still on. So because that's the last known state of the device and the device has to confirm that the state has changed, then it would get updated. So if you check after that, it would change. Yes. Was there any issue with the actor model that prompted you to try out the device to be... So actually we went a little bit too far with the actor model, where we tried to represent even the users as actors. It was all right, but we used it in conjunction with... I forgot the name. It's called... I forgot. It's this algorithm that's used by Google Docs, for example, right? When you change the document, the document gets... It's called differential sync. So the document update is almost real-time. So we used something called differential sync. So we used the actor model in conjunction with differential sync, which meant that we took one contract, one representation of the device, and we used it across the app, cloud, and device. So there was only one way to represent the device, and all three layers understood it. So whenever there was a state change, we would create a JSON patch out of the state change, and only the patch would be communicated to the cloud. The cloud does some basic validation to see if the user has access, et cetera, and then pushes it to the device. It was very interesting, but it became very unmanageable. And also the contract was bound across three layers. So we had to push an update to all three layers when we wanted to change, which is why we switched to another model, the device model. So this is the representation of a device twin. It's just a JSON representation. So there's a couple of things which are interesting. So one is tags. So tags is some information that the device actually doesn't care about so much, but it helps you identify which physical device you're talking to. There's also the properties that we spoke about, desired and reported, and some we're changing some frequency, I think, here, or some other data. So this is a sample representation of a device twin. The last thing, this we actually didn't try. We did a lot of research on it. It was very nascent. It was still early stages when we tried it. So this is Open Connectivity Foundation. This is actually a consortium of different companies. It's not one org. It's a group consisting of many orgs, lots of sponsors. They are trying to come up with a specification, like an open specification, that will help represent devices. So, for example, if you have a Google Nest home control system, and you want to connect it to like an Amazon speaker or a Philips light bulb or whatever, it's actually a little tricky, because traditionally these companies use proprietary protocols, proprietary contracts, et cetera. They're trying to make it open and define the specification so that the devices are interoperable. So end users can actually interoperate the devices. So according to the OCS specification, this is how you would represent the device. There's two parts to it. There's resources, which are actually properties of the device, and there is functions which let you control the device. So, for example, for this device, there is a state which is a switch, whether it's on or off. What's the dim setting maybe, and what's the brightness? And then there are functions which let you control this. So what they have described in their specification is that any capability that the device has can actually be split into two parts. There's core capability, and there is like domain-specific capability. So the core capability is anything you would do, you would need on any IoT thing. For example, how the device, how the device gets discovered, what sort of protocol it uses for messaging, or what is the messaging structure it uses, any sort of crowd notification operations on the device, addressing, how do you uniquely identify the device, things like that. So the thing at the bottom is common for any type of IoT device in respect of which industry you are in. And then you put vertical profiles on top, which are specific to the domain you're trying to operate. So home security could be something specific, or you could have something specific for aircraft, so cars, whatever. So they allow the specification to be extended by adding a component for extensibility, but the core specification remains the same. So the device, when a new kind of device is ported onto the cloud, you have a standard specification to use to add the device to the set of devices you already connect to. So you can check out their website. They have an editor and they have some documents and tools, frameworks that guide you around this. So the next thing we're talking about is security. Security of the application itself. So I added this slide because the last time I spoke about it, there were a lot of questions about security. And to be honest, we didn't actually deal with all those things. We chose security concerns that were very unique or specific to our application. So I can just give you examples of what we did to deal with certain scenarios. So the first thing was like I mentioned, it was a home security application. So we had to make sure. So one of the use cases was as an end user, I can give control of the devices in my house to another person in my house. Like I can administer and add another user to get control to the devices in my house. But it had to be very careful about who you're adding and how you're managing the devices and what sort of roles and access that user gets. So for example, in our application, only the homeowner could add other users. Even though I could be the homeowner, even though my husband owns the house with me, I'll be the only person who's allowed to add users because we had to strictly control who has access to the data. Because this application actually had a camera which you would keep in your house. So you had to be really careful about who's getting access to the data. So like any application roles and access controls are really important. Another use case we had, which was very significant for our application was we kind of used to factor out for device authorization. So like I said, we had a camera. And the first thing you would set up in your device is the device actually connects to the phone via Bluetooth. And then as a user, you can use the application on the phone to connect your device to the internet, to the Wi-Fi. But before we do that, we have to actually authenticate that the user who's trying to set it up is actually the user who bought the device. It's not the neighbor trying to set it up to connect to their Wi-Fi and control the device. So what we did was we generated a QR code and we sent it to the app. The app screen would show the QR code. Then you had to scan the QR code using the camera. And the camera would send whatever it read to the cloud. And the cloud would validate that they're both the same. And only if that's the same, the device is allowed to pair and then connect to the internet. So other thing that we did was we had to set up sensors on the windows and doors. And the sensors had like, they could emit light or sound. So they would emit like 3B, so 3 light signals. And you have to like enter the number of times the signal went on and we would validate it. So these might be rudimentary validations, but we had to make sure that the user who's setting up actually has access to the physical devices. The other thing is what I mentioned briefly earlier. So in a home security application especially, absence of telemetry was very important. So one of the things we had to deal with was you have set up the application in your house and it's locked and you are at work. And if someone actually breaks into your house and physically disarms the device and breaks it, what are you going to know? Because it's not sending signals anymore, right? So in order to deal with that, what we did was if there was no telemetry for 3 consecutive pings or how much ever we could configure it, then that would actually be an alert and the user would be notified. But the user had the option to check on their phone and see if there's a camera feed and they could decide if it was an actual threat or not. And then we also had rules which would escalate this. If the user is in a meeting and they're not looking at their phone, they actually had to escalate it as an alert to the police. So we had rules around that which would sound the alarm. The next thing is firmware vulnerability. So when we built this application, we were working with the three business units. So someone was manufacturing the hardware in Taiwan or Japan, I think Taiwan. And the firmware was being written in Bangalore and the cloud application was written in like Czech Republic and Bangalore, many places. So actually when we put everything together, we had to do some work to make sure that the integration was seamless. But then we also had to think about you scenarios where how do you update the firmware when there is an upgrade or when there's a known vulnerability and you want to push this to all devices, how do you do it? So we'll talk about this more in detail. But this point is here just so that we start thinking about it really early on when we start designing the application. It cannot be an afterthought. The next two things are things we would do normally. So the kind of data that we're collecting and where we are storing it, who gets access to the data and how do we treat sensitive, confidential information is really important. So when we were building this application, we actually had automated tests which would run, which would take like a screenshot from the camera and make sure that the camera is working, all the sensors are working. But we discovered that the camera was actually in our work area which was taking pictures of people who were on the team, which was actually a violation of our privacy right. So we have to be very careful about where we store the data that we collect and what sort of laws and legal compliance we have to be compliant to. Last one is protocol security. So when you choose the end device and the gateway, there's actually a lot of choice of protocols you can use like AMQP, AMQTT, Wi-Fi, Bluetooth, whatever it is. So the protocol that you choose will need to be considered based on a lot of things. What's the range to which your device can emit? What's the size of data you're trying to emit? What's the bandwidth? And what's the, does the device have, can it connect to a power source? Does it work on battery power? So there's a lot of things you need to keep in mind, but security is the protocol, secure enough is also something we need to keep in mind. Okay. Last part is about CI CD. So CI CD, like I said, is continuous integration, continuous delivery. How many of you do CI CD on your project? Right? So you'd use like Jenkins or, I don't know, some tool like that to do CI CD, right? We actually had difficulties because we were not sure how to push updates to our firmware device because it's not as flexible as a server and we had to, so when we roll out an update, we have to roll out the update to multiple devices simultaneously, so thousands of devices at least, and we also have to put in scenarios to handle rollbacks in case the update fails because these devices are not physically accessible to us. So what we did was something very simple because our device was actually really powerful and it could connect to a power source. So what we did was it's almost like a web server was running on the device, so we would push an update to the device and it would try to install the update, run a series of automated tests locally. So we had to run like four or five automated tests locally, and if the test pass, then it gracefully assumes that that's the latest version of the server and removes the last known deployment. If the test fail, then we would get notified, but it would go back to the last known version. There's definitely more sophisticated ways of doing this, but this is probably an area that's not been explored too much. So some things that helped us was our devices had the ability to receive updates on the air from day one, so we could test it out and roll out patches. How do we monitor the deployment? We actually had logs, the logs would get pushed to the server, but we couldn't do monitoring or logs to the scale that we would do in a software application, so I still don't know the answer to this. These are some questions to consider. So we sent a package to the device on the air and the device had scripts that would kick in. So when the device would periodically check if there is an update, so there's like a predetermined location on the server, on the device, so it was powerful enough, it was like a web server, so it could do a lot of things. So the script would kick in at some frequency, I think it was once a day, it would kick in to see if there is a new deployment applicable and if there was, it would try to install it and run these tests. Another approach that we discussed was to check a version API to compare that the version it has and the latest version that's available if there's a newer version available, it would do a pull. So who does the API update the control? Because that affects all versions depending on how many million device you're talking about millions of device across GPS and their time zone differences and this thing can just fail somewhere. Exactly. So who controls this thing? There's a lot of vulnerability in IoT. So even the firmware, so depending on the manufacturer and it depending on the batch, so it's not just the manufacturer because each batch, the firmware is rolled out at different times, so it depends on when the update is in issue. So all these things if it's not considered and the worst part is the IoT, you can just kill the whole stream. So does it quantization and the control? I'd like to point out, you know, because in the OECM there's nothing stated about all this interesting thing. So I'd like to hear your opinion. Thank you. So I only know theoretically how we achieved it. I wasn't actually involved in rolling out the upgrade because the upgrades to roll out an upgrade was it could be because of a vulnerability or a patch or it could be a business decision to roll out the upgrade. So I think we had some control over the geography of the devices we were rolling out to. Like we could plan and roll out the updates to specific sets of devices. So actually one part of the application that's not represented here, actually the device management service was like a registry of all devices that were connected to the cloud. And along with the device data it would have lots of information about the version of the firmware, the version of the hardware, the geographical region where it was deployed, things like that. So we could actually control it by I think the last I know the updates were actually triggered manually because someone had to take a decision and sign off and say roll out the updates and they would roll it out in phases to different regions. But the actual mechanics of it, I'm not too sure. Theoretically this is how we achieved it. Yes. Did you have to manage more than one generation of hardware? Version one and version two? No, actually not. So the engagement that we had with the client ended just as soon as we went on. So I'm not sure if, I'm sure we would have reached a situation where we had to manage multiple generations of devices. But when we rolled it out it was to one set, one generation of new devices. But we actually, there was another team which was supporting thermostats which were in different generations and they used to have a lot of issues. So they used to do a lot of testing beforehand before rolling out updates. So we actually had a board of thermostat devices of different generations and different versions. And we used to run automated tests every evening which would actually test basic sanity functions on all thermostats. And that was like a major check box you had to meet before a release or a roll off. So this is a sample architecture. This is one of the, oh sorry. What is your definition of simple versus basic? Is it the same or not the same? Most likely the same, sorry. Basic is different, simple is different. Is this in context of something I just said? No, because simple it means that we know it's subjective. Because basic, it means it covers the total spectrum. That is basic. Because if you meet the basic then you save. But when you simplify it it depends on your own use. Oh you mean the tests? Yeah, definition of simple. Your definition of simple. In what way is simple? So the test would not try to, we did not try the test to cover all behavior exhaustively. We identified critical parts that needed to be covered. So what is different with basic? I didn't intend it to have any difference. So this is the sample architecture. Oh sorry. Yes. Oh yes, sure, sure. So this was, so the company, the enterprise was a dotnet shop. So we used most of dotnet services. So the news, we had some legacy code which was on dotnet framework. We also wrote some new services which were built on dotnet core. We used a lot of azure managed services. So we had azure IoT hub. We used azure event hub, Cosmos DB. And yeah, that's pretty much it. Azure API gateway. Yeah. So actually the documentation on Azure for the IoT gateway, the event hub and other things are actually pretty good which is one of the reasons we decided to continue with it and leverage what's already there. So this is the sample architecture. So this has been abstracted out but some important things to call out like our application had sensors which would connect to the control unit of the device. Some of them used proprietary protocols. Some of them used industry standard protocols. The device would connect to the app, the initial stage for setup using Bluetooth. But the device also used AMQP to connect to IoT hub through which it would send the data. We had a bunch of microservices. There's only four or five represented here but we had a lot more services. Each service had specific responsibility. So there was one service which took care of device management, device registration, the telemetry service and a bunch of other services for user management, command query, et cetera. We had Azure API gateway as the gateway because we had to connect to external services and end user applications and we had dashboards and app, a bunch of apps. Yes. Why did you choose AMQP over MQTT? So that decision was already made when we started on the project. We wanted to use a secure protocol, maybe not AMQP. But at that time Azure IoT Hub had the support for HTTP, AMQP and MQTT. So we were debating with the idea of using MQTT but this was a decision that was taken by people with the higher pay grade. So that's essentially all I had. Thank you. I'll take any more questions now.