 So hello everyone, thank you for attending our call our our talk My name is Stefan Bunciak I'm a manager of quality engineering and in the middle-ware department along with me. There's a miracle Senior quality engineer also in the middle-ware department and we'll talk a little bit about testing IOT applications We'll walk through some of the challenges we faced in our research project and I'll do some of the more boring stuff. Let's say more generals general concepts and stuff and then we will talk about our Our proud our open source project But yeah, that will come later on so Let's get started When we talk about Internet of Things we generally think of something like this a lot of Sensors bunch of actuators Connected via gateway Sending sending data via that gateway to some data centers for processing and analytics decision-making and stuff like that, but During our let's say research and and our work We found out that the things are a little bit more complicated than this and we sort of Discovered what more of the IOT let's say architectures and find great topologies of IOT applications And stuff like that. So, yeah, you will see that in the in the following slides But even with this Let's say a little bit simplified Simplified topology of IOT application or ROT system you can you can already see that There can be a lot of a lot of challenges so when you want to comprehensively test such such distributed system, right? First of all, there's the environment itself, right? There's a there's a lot of Network topology and a lot of network protocols in play. There's a You need to take into account that the network is sometimes Limited in its bandwidth and stuff like that. It's not like perfect conditions in your in your data center There is a lot of different technology stacks different devices and stuff like that Then there is also the speed of performance, right? when you have Let's say hundreds or thousands of sensors generating data It's a huge load for the for the IOT system to actually Process and and work with it. So that's one important aspect or one important challenge And then you need to take into account Then there's the thing of the integration itself, right? You have a big distributed system Sometimes very very resort demanding to deploy or to actually Sometimes put together, right? You have the you have the sensors you have the actuators of various types You have the gateways you have the Business logic in there you have the analytics parts and you need to basically spin it up Everything at once in order to comprehensively test it. So it's not just Testing the small part, but you need to take into account the whole picture so that the IOT system works as design as you want and One of the last things that we observed is that there's a huge let's say Almost let's say unlimited even and it's kind of the of the data and the data inputs that the System needs to process and and work with right so Coming from the coming from the sensors. So Ideally you would somehow create only a only a subset of these data to Put into the to put into the IOT application for testing so that it travels all the all the important aspects of the application but not not like test every single possible input because that would of course create a huge amount of test runs and Texas test executions required for comprehensive testing so With these challenges in mind We teamed up with our university partners from Czech University We put some thought into it and put together a project proposal titled quality assurance system for Internet of Things and submitted to Czech Agency of Technology for for support our project got accepted and the work basically kicked off in 2017 And The let's say it consists of two two important aspects of it Methodological part and and the technical part as you as you can imagine the technical part is handled mostly by the redhead engineers and the methodology part is Taking care of by the by the university people so Our journey began at the at the very At the very beginning of the project We basically had to have to define what the what the IOT system or IOT application Basically is how does it look like what are the important topologies we need to have in mind what are the protocols what are the Deployments use what are the technology stacks that that are being used in the in the industry How does it all work together? You know how to even How to even test such such a distributed and heterogeneous environments? So there were there were a lot of a lot of questions in the in the beginning Let's say Yeah, one of them being also how to how to effectively effectively automate such testing, right? at the end if you are developing a distributed out of the system you want it to be automated and and Run all the tests and ideally in a some sort of continuous integration Environment, right? You don't want to run the tests in a manual fashion. So there was a lot of a lot of questions We had we had to answer before some some real work could even begin So we basically started with the with the question of how the typical IOT system or IOT application look like What are the topologies? And we basically came up came up with five of those very different ones And Mirac will walk you walk you through all of them in a little bit more detail. So So as the fan said it started As a question, what is the IOT application? So we went to the papers, right? we start digging digging and after hundreds of hours and searching from Reading hundreds of papers. We found out that we can categorize the Last majority of the architectures into five different categories. So first one and that's the most common one as you know it Is the gateway centric? Application or deployment that's the deployment. You probably knew no from your house In a way that you have some gateway deployed and then all the Is it better now can you hear me still all right So this is the application you probably know the best because that's the kind of application you can have in your house You do have some single Gateway where all the amenities all the things that you have in your house are connected to which collects the data and Then you somehow interact with the gateway in a way like okay When I want to open my window, I will just click in my application at home and do Window will open something like that. So pretty Simple application pretty easy to test even but then there are different like the distribute in mesh That's a there's a kind of application that is really similar to the microservice architecture as you know it now so a lot of different things that are sending data all over the place and Communicating among themselves like one actuator knows that it has to listen to different sensor and when Message emerges it will react somehow Next slide please Then there is a connected internet of things. This is something More related to the industry way how the internet things Works there like if you are in some I believe the correct Abbreviation is IIo T industrial IOT Is handled so you have different Parts of your I don't know factory has a different separated Networks that are Mostly single purpose and then they communicate via some gateway together and they interact Together and Then there is the last part and that's a collaborative IOT application. This is the architecture. This is the architecture that is used mostly in the Biggest larger scale deployments like Like smart cities etc. So you do have a separate part separate gateways That are somehow interconnected Let's say one gateway is collecting data from I don't know the traffic control Stuff another gateway is getting from I don't know the I see part like from the From the police etc. And then they communicate somehow on a larger level like between the gateways And there's some user who tries to collect all the data together and combine them Then next slide please then there is a last architecture and that will be the wireless sensor network, which is somehow Out of the scope but also in the scope because this is more related to the technology than to the architecture itself because you can't see the wireless sensor as some part of just one network and Or maybe as even as a single device If you would if you would look at it It's a single device that provides you data from the whole network, but it has to be in mind as well for the property sign So yeah, that is all okay, so With these let's say first questions answered like what would the IOT application or IOT architectures look like We wanted to address the challenges outlined in the previous slides So the first the first and most important is the heterogeneous environment, so we we knew that we would need something that would Simulate the networks simulate the various kinds of networks to various topologies that we would also need something that would emulate the devices the actuators and and simulators and And at the same time it would support The emulated devices to be configured in a way that we we wanted to behave like like so we can actually Tell them hey, you're going to be a temperature sensor and you will behave like like no more than normal day in in a Boston let's say but At some point we need to be able to tell him like hey I don't know an earthquake arises So it needs to start behaving a bit differently for a short period of time so With with these features in mind We need it we need it we need to start implementing Features like this then there's the matter of performance again, the ideal thing would be to Take an existing tool right and and integrate with it So we do not reinvent the wheel so We picked up the perfect tool Because it's developed developed by our colleagues at the Red Hat in Brno. So there was our obvious choice Then there is the matter of integration and into a scenario. Let's say Again, we wanted to take a de facto standard and in this case. It was the journey to framework So again, we do not even reinvent the wheel only build build upon that's already in the world out there and last challenge we outlined was the Huge amount of data variants and configurations of the system That's something that our university partners has experienced with so we basically built upon their knowledge upon their experience and and expertise and Our colleagues came up with a tool named oxygen And Let's say the research research guys the university put a lot of a lot of thoughts and a lot of Expertise in implementing mathematical knowledge and stuff like that Theory of testing and stuff like that into implementation of this tool It's a it's a model-based tool for Generating test cases in an automated fashion where you basically where you basically Put into the tool the entities the relations between entities and the operations model it via you and my activity diagram and will automatically generate the test cases that would Test the system comprehensively, but in a most efficient way that you don't need to don't need to Work with all the possible inputs, but only with a subset of it In case you are more interested in it Feel free to grab the bits and and contact the contact the university people because it's mostly being worked on on it by the by the University University researchers and professors Not really our our domain. What is our domain though is the automated testing framework? Via this one We want to address all the remaining challenges in in I've been testing First of all We had to put in place some of the some of let's say Hard-stop requirements first of all, of course they had to be open source, right? The second one is that we wanted to build as I mentioned on a on a de facto standard technologies that in our case was the unit 5 Then the most important features, let's say which were the Simulators and generators network simulations and and data generators so that we can automate Network deployment automate the data generation to properly test the IOT application And the last one it was that it had to be see a compliant so that we can look it up to a See I to like Jenkins or or travels Travis. So yeah with these additional requirements in mind We went on and wanted to design the system As you can see we also put a lot of thought into it So it ended up like like this one Not something you would call a simple diagram, but as you will see later on Fortunately, it's turned out to be a bit more simple But yeah, this was the first first design we came up with and and Deliver to the check agency of technology. It's part of our final report in the first year So yeah, I just might add if you go back one slide that The colors are there for reason because for every single component you can see there There was a different diagram that The defined interactions more comprehensively and so This was basically a high-level day So, yeah, nevertheless Even with this system design in place We were very brave and went to the implementation phase And basically set up open source project call Patriot and Mirac will tell you Walk you through some of the some of the details of our proud open source project Okay, so about the Patriot So as we said, we started with the architecture. It's a market lecture and we have the design You saw on the slide before So the goal of the Patriot is to give you some tool some framework that will allow you to build up a Testing solution that will enable you to Simulate somehow the network or the network topology that is going to be used by your application emulate the behavior of the Different kinds of data providers like are the sensors or to emulate the behavior of actuators that you are going to have Deployed in the real application. The reason why we are simulating this is a simple to have this Kind of testbed ready That's really expensive and the tests are hardly Repeatable, etc. Because basically you will end up with some Technological some physical Demonstration where you will have to have Human who will look over it write down the results of the test cases and then you can measure them So that's why we and we started with the simulations, etc Of course, it has some disinvented disadvantages like you can't test the itself But you can test its integrations if you have its mathematical properties Yeah, so we ended up with five corner stones the component for network simulation component for And the data emulation or data generation test runner for the integration testing which will allow you to Do the integration testing itself in a most simple way Some extensions for reporting and metrics because in this scenarios The basic ones like only plain X units, etc. Are not Sufficient for you because you are dealing with distributed environment and components for integration with some performance tool Because we need to test that as well. So as you can see we pretty narrow it down In a way how to design it so we ended up with three basic components, of course That's a work in progress, right? So in in some next release it will grow but right now we ended up only with three components and we are happy enough But all the interactions that were defined are there somehow hidden, right so We do have the free basic stuff Of course, there will be the performance in next release that will be somehow connected to it, etc and also the simulator part will probably Divide also to more providers than there are now Next slide please So the framework base framework base. So we started with JUnit 5. Why JUnit 5? Well, because for the testing You don't want to use some low-level language in general because if you if you would start with something like C and We would try to build a A comprehensive system for testing and you would end up with Framework that will test your application, but after you have written something like thousands of lines of code To write single testing methods, right? So and why why why Java and JUnit 5? Because JUnit 5 is pretty new technology which is highly customizable and And extensible and we know it the best, right? So that was the second reason So The base defines the APIs for interactions. So it defines the API for the network simulation In case you would need to extend it for your specific Model or something like that it defa it extends the basic properties of JUnit because JUnit itself is still unit testing framework So there are stuff like random random orders, etc. But even the JUnit Developers are extending it so you can do some stuff with it and we are doing some stuff with it as well so we extended the properties to add some things like skipping the test execution When another test failed for instance You don't want to test your function For the division if it failed on a single scenario, right? But you have to think of it in a more complex way in a bigger system So that's why we added this or break points Okay, I started this test, but if this test failed, we don't need to continue So that kind of thing then there is a single management place called Patriot Hub Which contains all the things created by the framework to allow you to manage it. So let's say you want to In between the test you want to set up some environment change like okay right now Part or segment of network will fail It will be disconnected. You can do it while it Connect to it set you are disconnected and it will happen and then you can resume your testing and test it What is going to happen in your environment? And it is the start point of the framework to operate so Other parts of the framework will be basically I'm more probably useless to you without this Next slide please. Okay, so the network simulation. I would say this is the biggest part of the Patriot framework itself So it provides you to link to emulate and where to are and deploy your application to define the topology of your network application like Spawning up several networks and several network segments and then deploying specific applications Or a specific part of application into it It does have the full control over the network, so at any point of the test run or In a before phase after phase anywhere you can say something like okay right now You are changing your IP address or you are going to behave Maliciously somehow like okay right now. We are going to send only 50% of packets So like that you can this is the thing that will allow you Right now it's completely It's using only docker. Why docker because we need we needed to have some easy to use environment for proof of concept and The place how to even define the API so what do we really need to have in common to Get to work in state But currently we are starting we are working on another I will get to it later But we are working on other runtime as well Okay, next slide please so this is This is somehow Capture of how the network simulation operates so the test runner itself or test runner all the automation around this runner can Create a simulated environment in a way that it will create several networks In order to it's pretty simple. That's another reason why we use that and then it can deploy the applications in this case look at this Square or rectangle. I didn't put some labor to it as a single container How we operate is to every single container that is running in the simulated environment We add a simple thin layer of abstraction to it That's our part. That's our API which allows us to connect to it and somehow modify its behavior We can do it either Why setting up the network interface the virtual network interface itself or why are the IP tables which are there present as well And then there is a second part, which is router the router itself you can imagine it as a normal L3 router so it's a it's a Container that is connected to both networks and has the routing table that allows to forward the packets from one to another via this via this Control interfaces you can then set Things like okay. I am completely disconnecting my application from the internet. What is going to happen? I'm going to Disconnecting the single application or you can tell to your router. Okay, you are not rooting this application anymore and in case you would test something like I Don't know Of the information via different route. So that's it how the network simulation operates And then there is the data simulator. So for the data simulator our initial thought was to create some Simple tool that would behave like a Lego if you know it I'm not sure if I can say Lego if it's a train mark, but Please don't tell on me So we thought Or our thought was to create something that would Easily enable you to build your solution as you want the base of the Network simulation is some mathematical property of your device for instance when you would be simulating a thermometer There are mathematical functions that defines how the temperature behaves during the day our The our data simulator then incorporates mathematical library that will allow you to in combination with Some random data generator Combine it with your mathematical property and for instance by the distribution function define what data will be sent out So this is the first place how to get there How to get the data that or Seat for the data that will modify the behavior of the generator Then there is a second part Which which describes where you can describe how the Data which are typically some number some double or something like that are transformed into some meaningful Number like for instance if you would go with a Thermometer your mathematical function will only define the The increments or decrements of the value itself But it will not Set the zero So then you have one single transformation In the in the input is some double. I want to transform it into the real The real temperature so then I have to apply some modification to it and then there is last part how to how to Transform these data into something that is transferable over the internet. So Wrapping up home solution into Let's say MQTT message and send it out or into a rest call or into a co-op call something like that Okay, so those were the three Constants of the application What's next for the 2.0 release? The Patriot So as I said this What we have done right now is basically more of a POC than a production ready solution We are aiming that our 2.0 by the end of this month and we should Provide or we will find some better In the parts of framework for integration testing what we have right now in mind is More support for all the execution In a way that you could describe The dependencies between the test classes and test cases as a tree for instance you for instance one of the things we are going to implement is Label or annotation after test after not after test after That will allow you to say to a test class that it should be or it has to be executed after a different test class is done And things like that Second next thing provide better integration of the framework parts framework parts right now It works together, but we don't have any support for simple usage So yeah, you can use the network simulator and the j-unit extensions and the data generator But right now it doesn't basically know about They basically doesn't know about themselves We want to add some layer of abstraction over it in order to make it work more together Next thing since the reporting and monitoring is really Highlighted feature that is expected from us by other By potential end users, we are going to add some preconfigured Kibana That will be connected to the elastic search for the collected metrics so you can see the metrics of the Execution itself and the last thing first cooperation between Between the rest of framework and perfect is expected for the truth. Oh Okay And more plans like comprehensive demo which we do not have so you won't see it in action unfortunately We also want to extend the network simulation itself first of all to spend over more docker hosts right now We are able to operate only over one docker house. We will want to make it more like Cluster for testing also we want to add some things like cooperation with the hypervisors for instance KVM in order to start and to manage virtual machines either as a single segment of a network or only as a one application and We want to introduce open open tracing into the framework in order to provide the data from the run itself. So Whenever for instance application itself is over open tracing or system on this is over a full-time tracing We could add our data to it and then the debugging would be even more simple and the last thing Is the description of a city in demo file. So away how we could describe what is going to be deployed and how The demo because it's cool, right? So yeah, we do have the definition of data generators already We are right now working on the definition of the simulated environment but we want to wait with the definition of the simulated environment to the Other parts of the of the framework like the connection to different providers, etc And that's probably all from my side. Yeah Yeah, that was the plans of Patriot project itself, but It plays a role in a let's say a bigger project that started in July 2017 and should finish in By the end of the 2020 We are currently in the in the phase when we have a POC ready as Mirick mentioned and still still in the in the process of completing the required features, let's say We should be completed by the end of this year And we are already in discussions with potential users in in Czech Republic both from the methodology perspective and Patriot framework itself, so that's something that weighs us next year this pilot testing or pilot pilot usage In in a customer and real environments And of course bug fixing and stuff like that and we will evaluate the project itself and in the in the 2020 hopefully it will be a successful one. So that's that's from the project timeline In case you have any questions feel free to feel free to ask Or you can reach out to us later Either via email or visit our web page or fork our github repo and yeah pull requests are very welcome In case you have Interested in testing out the application. So yeah, thank you very much for your attention And if there are any questions You can ask them now System under test