 En we creëren heet om je huis te lopen. Dus een beetje over de producten, we testen met een robot framework. We ontwikkelen een nieuw smartroomcontroller die veel radiointerfaces, wifi, 868Mhz en zelfs een ZigBee interface. Het is een bedrijf met een Android en iOS app. We supporten veel verschillende protocolen om hier van een apparaat te bevinden. We kunnen ook met apparaatjes van andere vandaars gebruiken. De meeste feature is zone control. We kunnen de temperatuur in de ruimtes controleren, individueel en niet als een hele systeem voor je huis. We maken ook de apparaatjes controleren. Het is de onderdeel van de apparaatjes die je ziet. Dat zijn de elektronici die de HMI geven. Controle de burner, pumps, fans, valves. De apparaatjes is met een businterface om de thermostat te connecten. Het heeft ook een veilig kritische deel, omdat we meestal gasapplies zijn in de sterkere regulaties. Dan een beetje over testautomatie, want als ik een businterface ben, als tester begon ik in een project van een prachtige product. We hadden een app en we waren aan het uitleggen. De manier waarom we dat doen, met manier testen, in ieder geval over en over, hadden we denk dat we iets meer nodig hebben voor dat. Dat is waarom we begonnen kijken in testautomatie. De eerste is waarom we want testautomatie. We willen feedback. Als je iets vervolgd hebt, wil je snel feedback over wat je vervolgd hebt, of dat werkt. De tweede is, we willen automatisch een repetitive regressie test. Er is niets minder behoorlijk dan over en over dezelfde job. We hebben ook tijd om de release te creëren, want nu is het met app-developement, maar ook voor ontdekend producten. We hebben de omgeving om te updaten, zodat we heel makkelijk kunnen creëren fixes of expanden functie. Dat leeft ons in de onderdeel, dat we willen ingewikkeld hebben in producten-developement, en dat we flexibel zijn in creëren producten-configuraties en de levering van frequent updates. Hoe hebben we dat gedaan? Het was een lange manier voor ons. De dingen die zijn op de gedeelte, zijn wel welk knowne voor je. Maar als je testautomatie begint, het klinkt veel makkelijker dan het eigenlijk is. Wat moet worden gedaan als je met testautomatie begint, moet je eerst besluiten wat je wilt testen op welke niveau. Je hebt natuurlijk unit-testing gedaan door de developer zelf. Integratie-testing, het testen van partijen van je producten, en in de eind, de acceptantie testen, voordat je de producten releaset. Voor elke niveau moet je een testarchitectuur creëren. En wat je hier ziet, de testpyramide, als we op de testautomatie begonnen hebben, was het meer of meer of minder upside-down voor ons. Dus als je unit-testing hebt, dat zijn de meest makkelijkste testen om te creëren en te implementeren. Als je in je testpyramide hoort, naar de end-to-end testen, die testen, want je werkt met een zwart bocht, is het heel moeilijk om te implementeren. Je moet de systeem van de buitenkant veranderen. Je hebt niet veel kennis over de binnenkant, dus die testen zijn heel moeilijk om te implementeren. Daarom moet de meestal van de testen op unit-testing zijn, en alleen de end-to-end testen moeten als mogelijk minder zijn. Als we naar de robot-framing kijken, als we begonnen hebben, hadden we de testpyramide upside-down. En dit is eigenlijk de keuze waarin de robot-framing kan helpen. Een robot-framing behoort zich over... je kan veel interfaces overleven, maar het bedoelt de reale product van de functie. Als we naar de robot-framing kijken, een kleine historie, de robot-framing was eerst ontwikkeld in Nokia. Het is nu door de robot-framing, en natuurlijk kan je het gebruiken voor veel verschillende manieren. Maar het bevindt het als een test-automation-framing, dus het maakt je de toolen om proper testen te doen. Het meeste is dat het heel extendbaar is met libraries, en het bevindt je ook met goede functie-reporting. Bezins van de robot-framing is het besteld als een Python-modul. Dus natuurlijk moet je Python, en je kunt het easily extenden met andere toolen, ook Python-modulen. Bijvoorbeeld gebruiken we Modplotlip, PySerial, en je kan veel van andere deel nemen. We gebruiken Aconda als een package-management systeem als een setup of testenvironment, PyCharm om een robot-test en Python-librariëtes te rijden, die worden gebruikt door de robot-framing, en natuurlijk Jenkins als een build-automation-tool, waarin we de testen van bevinden. Dan een beetje over hoe we eigenlijk de robot-framing gebruiken tijdens de testen. Eerst zal ik een beetje over de smartroom-termostat vertellen, en dan zal mijn collega Windel over en je meer over hoe we testen of de controle uniten. Wat is de room-termostat? Het is goed om een beetje van hoe het systeem functions, dus we hebben de termostat in de midden. Het kan de verschillende ruimte controleren met elektro-electric radiativels. Het heeft een buskonexie naar de applies. Natuurlijk hebben we de app en een backend om bevinden, en alle soorten parten. Dus als we het testen willen, dan zie je al wat interfaces naar de room-termostat. We moeten wel denken over een systeem hoe we het testen kunnen. Dus hier in deze foto hebben we de room-termostat, de verschillende modules die erin zijn, de main-control uniten, de wifi module, de HX8 module, we hebben nu de ZigBee. We hebben natuurlijk de connectie naar de applies, we hebben een backend connectie en we hebben een app. Eerst hebben we besloten, ja, toen we besloten over de testen, we hebben gezegd, oké, als we integratie testen, we willen alleen de room-termostat testen. En we willen de app separate testen. We hebben ook een testset te testen samen, maar dat leeft ons met de volgende setup. We hebben de app gestopt, we hebben de HX8 module gestopt, want dat is externelly gevolgd. We hebben gestopt, of applies, en we hebben alles geconnected tot een testexecutor. Als we de robot framework op de top van dat hebben we een testarchitectuur, die we hier kunnen zien. Op de top hebben we of testcases, die we proberen per functie te geven. Aangezien dat, zijn er resources, robot framework supply's met een manier om generische keywords te defineren. Bijvoorbeeld, we hebben een resource om de room temperatuur op 20 graden te zetten. En we kunnen dat in alle verschillende testen gebruiken. Met de resources zijn we heel handig, dus je kunt het uitleggen, of je hebt een heel mentaal testcase, want als je iets veranderd in je logiek, kan je het in de resources veranderen en moet niet veel testcases veranderen. Als we terug naar de onderdeel gaan, we hebben natuurlijk de robot framework zelf. En onder de robot framework hebben we de verschillende libraries. Deze zijn de libraries die we implementeren, maar de robot framework zelf komt ook met veel verschillende libraries die worden gegeven door veel mensen die robot framework gebruiken. Wat we hebben gedaan, is dat we een library hebben gecreëerd voor onze HX8-communicatie naar de radiële files. Dus we kunnen actief controleren hoe de user een radiële file gebruikt. We hebben een simulator die zegt oké, ik simulate, ik presse een button op de file en dan... Our library provides that to the terminal status, so we could give a reaction and robot framework captures the reaction and checks if that is the desired function we want. By implementing this, in this way we can set up a dynamic environment, we can... Sorry? I have a question. Before going to the implementation, can you give us a small example of what we intend to do in the previous video? Because right now you are describing what you intend to do, how you intend to implement in that figure. What we integrate here is the behavior between the whole firmware of the main control unit. Sorry, you wanted to know what we are integrating in this figure. What we are integrating here is our firmware of the main control unit and the firmware of the Wi-Fi module. These are both supplied by ourselves and we want to be sure that they work fully together. Is that... And to end testing is... And to end testing is that we also include the app, so we change something on the app and we want to know what happens. For instance in the room, the user changes something with his or hers app. Okay, then back to the implementation, so we built a library for every interface we have. So here we can create a setup, for instance our house has four rooms with four thermostatic radiator valves or with only one thermostatic radiator valve in a room, multiple radiator valves and so on. So we can differentiate in how the setup is the user uses his or her thermostat. Also we have, on the other hand, have simulated or appliance connection. So we can simulate different appliances users have an open-term appliance for a competitor on-off appliance or also different appliances made by ourselves. Then in the middle we have the API, the app users. And we also have a debug connection by which we can control the device itself for instance to let it reboot to certain modes and so on. On the first slide about how we, how you should test and the bottom, it said create a testable product while for instance this debug interface is very handy for us to automate or test and it's really necessary to do good testing with our product. Then at last from my part a little example about how we handle the different interfaces when you send something out with a robot framework you expect an answer back the easy way to do it is send out a comment wait 5 seconds and read back if there is a result but your system behavior changes over time as you expand your product and make a more complex setup so in that sense testing should be a little dynamic in the beginning we counted a lot of failing test because of small changes small timing changes and so on when we looked better into or a test we found out robot framework has a very nice keyword what is used over here and wait until keyword so please you ask robot framework please repeat this keyword, this part so read the line from in this case the serial interface check if it's successful and then continue if it fails it checks again in 1 second so you can say in this part we have we tried 10 times with intervals of 1 second we check if the profile has been set in this way because we are also we have a little bit external dependencies for instance what is used here the setup of a wifi connection you cannot exactly predict the behavior you are encountering so we used this keyword a lot in our interfaces and our testing became much more stable than it was before this was it for my part my colleague Wendel will continue on the appliance control unit alright well thanks everyone I see a lot of people so that's good news so you care about testing and I think that is good because software always contains bugs and that's not so nice so I will talk a little bit about the appliance itself you can see it here so this is a schematic overview of a home we have a bath tub and some radiations for the heating house and in the bottom you see it's a combination boiler so it provides you with warm water and that is what I will zoom into so if you look at this boiler this is a more or less schematic overview of the inside of a boiler so it's not it doesn't have a lot of IO it has a fan, a pump, a valve you see here and then it will connect to the radiator of to the warm water supply also we are very fond of acronyms so over there you see the dhw for example that means domestic hot water and so we have really a lot of acronyms but I will try not to use them and to control this appliance we are developing a so called control unit it's a piece of electronics with software and it controls the appliance and what I will talk about now is how we tested the software of this control unit so what is the software architecture inside this control unit we have made use of a hardware abstraction layer which means that we can run on two different platforms on top of this hardware abstraction layer we have the behavior of this control unit and we have some application behavior saying when a certain request for heat comes in we have to do this, move this pump to this valve that kind of things and we have two other main parts which is the bus communication it's a sort of serial communication and a burner control which controls the flame really the burning of the thing and for the microcontroller on the right that is what we will put on the control box itself so if you go in the right flow the end result will be a hex file which we can flash on the microcontroller and that we use CMake and IAR for IAR is a compiler proprietary and on the left we use GCC and the result of that is that we have a DLL and actually both we use for testing so the right we flash in the device and we test it on the hardware itself and the left we also can test of course on the left you have to deal with we have some IO and some interfacing so you have to step that out so why did we do this why would you create a PC build well to my opinion it has a lot of benefits to do this because when you create your code such that it works on two platforms your code is more cleaner so it has less pollution of header files for certain targets you can test actually really a lot of your functions without that the hardware is already available you can test for example the communication protocol and also you can accelerate time so if you flash it on the real board you really have to wait for example we have functions that for example every day we should run a pump so we put it in the lab we wait one day no it didn't work and if you do that on the PC we will know faster if that function is correct so at the moment we can accelerate about 26 to 50 times so it is pretty good so we can simulate a day in half an hour and that saves us time of course there are also disadvantages the obvious ones you have to do this work so I would say do this next to that we simulated the control unit software itself on the PC platform we also created some dynamics in simulink so it's a crude approximation of what a system will do in reality we've done that with simulink and then also this dynamics are then there is code generated from this dynamics and we also put it in a DLL so we now have two DLLs so how did we test this two DLLs with robot framework so we have written our test in robot framework I will show that in a minute and then we have written a custom robot framework library which connects the two DLLs that I just described and one cool thing you can see here is that we can run this test.robot we run it through robot framework and we can configure if it runs on the simulated environment so to say or on the real thing so what we can do is run the test first on the simulated environment which is of course close to reality but not a real thing and then when we are satisfied we can do one last test on the actual hardware and validate if the product behaves as we want so a little bit about the simulator set of how this works what we have done is that we we run the physics of the physics DLL and we copy all the values that are measured we put it to the control unit simulator we let the control unit do its thing and then we copy the outputs of the control unit again to the physics and this goes on every 10 milliseconds it is repeated we implemented our libraries in python by the way so now for the live demo effect yes so the backup slides are there but I don't know if they are needed so what I want to show is do we need to duck no, it will be okay so the function that I want to show you what we are going to test is what we called the chimney mode it means chimney means the exhaust on the roof of a home and what happens is we install a appliance in a home and you test this by hitting a button and it goes full power and then it shuts down and you know okay everything is connected correctly so it is a sort of a test mode so we are going to test a test mode so as Alex already said we used a pie charm to edit our tests so I will first show the test I am going to run so here you see the robot framework syntax if you know robot framework this will be familiar to you if you don't know it the intention of robot framework that these tests should read quite naturally so how it works is that you have some keywords which indicate some steps so what we did we grouped our test suite into logical functions so we have about 13 function groups en this is group 8 and then we have a test 1 of this function and this test the chimney mode and what it does is you can read what the steps are to do this test so we set it to 100% then we verify that the function is activated then we remove this request we check that it is gone and also we check that it is going back to stand by mode so actually a pretty simple test what you can do with python for example that is really handy you can control click and go to a keyword and then this is a keyword which sort of implements this step and in this step you see okay if you want to wait for stand by you have to do that by looking at a certain variable and it has to be a 203 for stand by function we have this status code so we'll wait for the 203 for say 110 seconds it's quite weird unit but we use that so now I will run this test for that we use con emu to run our test because to run a robot framework it is handy to run it from the command line en as you see 3 tests are executed from this chimney test en if everything is well I hope I don't know how to do that so now we come also to a one particular nice feature of robot framework which is the report you can see the report here it is a log of the test we just run of course I don't have any hardware here so this is the configuration for the simulated version of the previous setup you see that you have all kind of statistics how many tests pass, fill how much time it took so if I zoom in on the chimney test you will see here in the report zoom in a little in this report you can see exactly the same steps what you have as described in the test green here means ok red means not ok in this case it's ok if you look into you can also expand this keyword and then you see that for example this keyword we used wait for emsverbal which wait for the status for some time you can look really into it and also what you can do is in there you can see what messages were passed between the test and the control unit so that is really normally when it's green you don't look at it when it's red you want to know hey what happened and then this comes into play and also here you can toggle between certain levels of in the report and one thing we did was let's see so since robot framework is written with python we also used the mudplotlib library to include some figures of the temperature say the temperature is over time so since we started here a test mode which goes full power you see that the temperatures went up quite a bit from 20 to 40 degrees of course this is on a simulated environment so there is no way of telling if it is precisely as it will happen in real time but it gives you some a good feeling of what goes on and in the end you see that the temperature stays the same because we stopped the appliance so it will be hot for some time you can see here more at the top this is the burner power in percentage so how much power did we apply to this system so first it's a little bit that's actually a little bit weird but we also have the test mode itself so I don't know what that first thing is but you can see here what the system did over time so that brings us to the summary so what did we learn when doing this for the last I think year robot framework is a good tool to test embedded systems you can have keywords you have a lot of libraries you have python, ecosystem all these things you have the python, IDE and the test itself can be really readable and also what we did was we run the test on the host system which accelerated our testing times what we also learnt was that if you have a lot of tests you also have to maintain these tests so that's also what Alex said in the beginning with this pyramid really use a lot of unit tests and don't use robot framework too much I would say because then you have to maintain all these tests so if you have say 300 tests I think in our case we have maybe 200 different test cases that's a lot of test cases and if something goes wrong you have to really search where it goes wrong so maybe you don't want to have too much of them and also if you have we also developed our own libraries those also contain bugs sometimes so yeah that is we want to keep that as simple as possible so that was maybe what we learnt what we want to improve where we want to go we want to be able to test the room thermostat for example with multiple different appliance types so that we can check if the software works against the variant of other hardware and in there we want to better organize the tests and for the control unit part what I really would like to do is speed up the simulations even more to get the test time down and also make this test more robust to variants of products since we have a lot of different say we have a lot of control boxes which are all small variations and what I would like to do is we can have one or maybe a couple of test tweets but that they run and that they can handle this alright 10 minutes left that's good and we have time for some questions so thank you almost now last thing you were saying about speeding up tests we've got very similar stuff going on here these are our host development things but you can sometimes explicitly build a set of tests where you take time out entirely so there is no virtual time and then you can run them as fast as you like it's simply how many logical steps rather than speeding up simulated time write a test where time is not one of the variables that can make a huge difference yeah that's an option but maybe that's more a unit test what you would then write sorry so the question was to speed up time you can also yeah the question was you can speed up time but you can also eliminate time right so currently the system that we created is really focused on time so that would be hard but I would do that what you say in a unit test I mean just we've got an example here where we're moving a strangely similar piece of hardware to you where we're simulating moving a valve and there are a whole bunch of tests we can do which merely make sure it does the logical correct steps timing is not relevant to those you have a sequence of steps yeah alright yeah so you've covered the flat of functional tests what about non-functional ones like do you have these ones is it somewhat useful for banks so the question was do you have these are functional tests functional tests and you mean non-functional tests can you give an example of a non-functional test no we don't have we don't have the non-functional test we don't have the performance of the boiler we still do that in a lab environment so that we have the real thing we test that so then we really know what the performance is so it is mainly focused on the functional things yeah do you evaluate the correctness or the fullness of the test I didn't see any mentions to code code no oh yeah so the question was how do you evaluate the completeness of the test suite yes so we don't we don't do any mentions I don't know if so we don't use code coverage yet but I think that is a very good idea to do that so then we can see how much we cover we do the code coverage we do that when we run a unit test we do the code coverage this test we don't have the coverage enabled we should more or less have requirements coverage with this and that's very hard to track at least in the way we were yeah obviously people don't change their central heating systems very often so how long do you expect to support these products internet things while I buy today do you expect to support these products so the question was how long do we support these devices how long do we support it 10 years and appliance is longer there are also regulations for that how do you evaluate the simulated environment is similar to the real devices so the question was how do you the simulated environment is equal to the real devices so how do you do that we did some measurements in the lab and what we did was the simulated environment we applied the same inputs and compared the outputs to what we have measured in the lab so that's basically how we did that anymore oh yeah in the back so the question was did you find in reality some extra things that you didn't find in the simulated environment that's a good question so in our simulated environment our temperatures are really smooth we don't have any noise so in our reality there's always noise so so this is something that we found in our labs for example we had some noise on our sensor that we didn't spot in the simulated environment because everything was maybe too ideal I need to fix it exactly there so we can shake hands we thought about that so the question was have you tried to simulate the noise in the test environment I'm a little bit opinionated about that because I don't like randomness in the test so if you would do that then you would add some random variations to the test and then you have the situation in the test once it fails, you run it again it works I don't like that but you could do that so I think we should do something like that so you can get round I agree what you say but it's definitely going to work if it gets through the 10,000 test and then you can go with the randomness so I think what we want to do is what we also want some random inputs to check if that will disturb it so that's more question for Alex the question was what is the role of the backend and do we test that as well well that's a little bit an organizational organizational answer we have there because what I showed is the part of the product we create the backend is maintained in a different team we use for different products so they also have their own test but what we do continuously in our integration test is we use the backend in the test so when things go wrong it also could be in the backend so that's it's not ideal in the repeating the test cases and seeing what goes wrong but just to be sure we don't miss anything we use it actively in our testing yes we also test your question was if we also test the api and yes we do because that's a way a robot framework interfaces with or thermostat also by using the api and setting comments like the aptos yes your question was if I understand correctly how do we manage the different platforms we run the test on the simulated platform on target testing those are the finds which we can give to a robot framework on startup so we have a configuration file which configures the environment you want to test but the actual switch between the platforms is in our python library so a robot frame we did not implement any switch actively in robot framework but we implemented it in the python code ok