 Hello, everyone. Thank you for attending this presentation. Today, I will be talking about lessons learned in creating a mechanical ventilator with open hardware and open source software. My name is Luis Herrera. I am senior consultant developer at ThoughtWorks. I have been working in the technology industry for more than seven years, in which I have been doing system migrations to the Cloud, implementing continuous integrations and continuous delivery process, implementing scalable architectures and building project. So specifically at the beginning of the pandemic situation, together with a group of expert engineers in the area of automotive and electronics, I was working on the building and extension of an emergency mechanical ventilator. Across this process that ended with a medical approval, we faced and solved multiple challenges that I'm going to share with you on this presentation. So I'm going to cover these four topics. So first, I'm going to talk about the project that we developed and the outcomes that we accomplished. Also, then on the topic number two, I'm going to talk about evolutionary software architecture, but related to microcontrollers. So I'm going to share it with you, how we got medical requirements, and how we translated it to architecture characteristics. On the topic number three, I'm going to talk about this strategy oriented to combined hardware and software. So here we found some challenges about testing, and I'm going to share it with you, these challenges because it's something common on the IoT project. On topic number four, I'm going to talk about continuous integration and deployment for microcontrollers. So we know that this is a common practice for software development, but when we add hardware, then we need to find new mechanics to do it in an scalable way. So let's start the project development. On the month of May and April, together with a group of expert engineers and with Oxygen Project Community Support, we made an emergency mechanical ventilator that today is published on the Oxygen Community page. So we did this in order to solve this problem. So if a person present is serious symptoms caused by the virus, either lead, they will have access to an intensive care unit bed, which have mechanical ventilators and other emergency mechanics. The problem in our country, Ecuador specifically, is that at that time in May, they were only 256 intensive care unit beds. We took into account that in several cities, there was not even one intensive care unit bed. So if someone presented a respiratory problem, there will not have an intensive care unit bed, and therefore a mechanical ventilator at their disposal. Faisal with this situation, we were encouraged to build a mechanical ventilator from a base of a device tested and validated in Spain called Oxygen Project IP, which is open hardware. So on the process of feeding the device, we use it next stack. As I mentioned, we build it in basket on the Oxygen Project and since that we started with the first versions, we build it the wooden version and after some tests, we move it to the IP version, which is designed to be built in factories. Okay, so on the hardware, we also use Arduino to provide configurability and another feature to the device and this was the value that we added. We extended it and made it configurable. On the other hand, we also use a Raspberry Pi in order to provide a graphical user interface as it values related to the volume, pressure and airflow in real time and also to send data to the cloud in order to provide stretchability. On the software side, we use a platform which provides multiple benefits like a library for testing an agent to connect with remote device. Also, we use C++ in order to develop new libraries and to extend what we had found that was open source. Also, we use a Docker to containerize application developed on Node.js and get actions to implement continuous integrations and continuous deployment violence. Okay, so let's start now with evolutionary architecture and before that, let's talk a little bit about what means architecture. The industry had made a great effort to precisely define software architecture. Some architects refers to software architecture as the blueprint for the system, while others define it as a roadmap for the system development. So based on the definition made by Mark, Richard and Needfor in their book, Fundamentals of Software Architecture, we see that in this definition, software architecture consists of the structure of the system combined by the architecture characteristics, the system must support architecture decisions and finally design principles. So following these four principles, Misionet, ideally we can define a good software architecture but as we know over the time, the business requirements change and so the technologies and even good practice to develop software can become bad practice. So then once I have built an architecture, how I can prevent it from gradually degrading over time. So here is when the need of find a mechanic to prevent important characteristics of architecture is born, the term of evolutionary architecture by Needfor, Rebecca Parsons and Patrequat that says, an evolutionary architecture support guided incremental change across multiple dimensions. So in your project, we were deciding on each step that architecture characteristics will be all guided of what we need to implement and what pattern we could use to do it. And of course, to do it incrementally, and of course to do it incrementally, signs that we will be voting because at the moment, at that moment, there was not a medical guideline to assess emergency mechanical ventilator. In this building process, our clients were doctors and paramedics of the intensive care unit who in the initial stage told us that the device should have a mechanic to at least measure pressure and frequency with which the first four characteristics of the device appeared. And that were configurability, extensibility and accuracy. And configurability we have already covered signs that we were using Arduino microcontroller. So we could parameterize the device functionality with that. So let's start with the first one. Sorry, with the second one, this is extensibility. This was one of the features that also later allowed us to iterate it quickly because of the easy of adding controls and functionality at the software level. And we cover it by creating libraries for the different sensors and controls that we had in the device. Then how we cover accuracy. To meet this characteristic, we added to the device a led display of 16 columns at two rows, which allow it to allow it to the user in this case, the doctor who was managing the device to know that he was configuring in real time. In this case, time by using the potentiometer. In this case, yeah, using the potentiometer. On this part in the future, we got a good learning. One was because on this screen, we were printing a technical message on the road one and we were showing time two seconds, as you can see on the slide. And that message never was interpreted by the doctor and paramedics as frequency. Since that, they'd read that message in a different way. So another learning was at the building of process. As you can see the image, the device is composed by multiples spread and both. And each one have a functionality. If you put one of another size, the device after a couple of hours of active work, it is start to failing. And that was a good lesson for us because we were using another size at the beginning and you can imagine the results. So we started this journey on March 24th. And as you can see on the image, we built the wooden version that was one of the first version released by the Oxygen Project. And we worked remotely. So we found some ways to collaborate in a distributed way since that at that moment we were living in different ways, but in different cities, but we were building an embedded system. So we combine in this way software and hardware. So on April 5th, we delivered our first version of the device and we got the doctor's approval. But this was a city approval. And in order to get the national approval, we should go to the capital city and run tests on the sophisticated laboratory. So on this process, Kia Motors decided to join into the project and sponsor us in order to collaborate with improve our changes and then build it on the Kia Motor Factory to donate to the hospitals. On April 11th, we ran the first test on the capital city, but at that moment signs that the Ecuadorian legislation, there was not a document or law that could tell us what are the guidelines that we should follow. We got not the approval. So our device was not approved at that moment. And it was evaluated based on the doctors and paramedic experiences. They decided that in order to provide their approval, the device should have another functionalities, which they found are essential for doctors and paramedics like C in a real time on a color screen pressure, the volume, the frequency, the A floor and visual or audible alarms. And as you can see here, the requested frequency, that was something that we already have done, but signs that we were using a tech language, they're not accepted that because they thought at the moment that it could generate confusion with the end user. So we translated the client business language to the technical one, and we found these three characteristics which are traceability, file transparency, and simplicity. So how we accomplished the first one? So this characteristic, which was also essential to know what was happening since the first time that the machine is turning on, and also for that to grab on the screen, we could accomplish it by transmitting or sending data by the serial port from the microcontroller to the computer. In this case, the computer was the Raspberry Pi. So on the microcontroller, we define it as a structure to transmit the data and we use it across all the system. So this was the way how we could accomplish this characteristic. Then file transparency to meet this characteristic at the software level and combine it with the hardware. We use it as some global variables that were accumulated and owns. They passed the limit values that we set at the start of the device, it emitted the alarm signal to turn on the red light and the booster. The booster is a kind of device that emits a zone. So we call accomplish it this visual and audio alarm requirements in this way. Then we have simplicity. So to accomplish this characteristic, we added the buttons to the device, which will allow to the user to set the control mode either by volume or frequency. So this way we have two auctions to run the device. And also another thing that we added or another auction that we added to the device was to select the installer card come. So basically we added that functionality because the device is composed by a ambut bag and by a can, this can do a pressure to the bag and with that this emit the volume of the airflow and the airflow. Another functionality that we added also with these three buttons is a possibility to reboot or to reset the device. Then another functionality that we also added was or another functionality that is of course related with architecture was the testability. And now we are going to talk about it more indeed because this is a really big topic since that we combine this software and hardware. So testing in microcontrollers is a testing process to check the functionality, the functional and not functional attributes of both. Software and hardware in that if we combine these, this becomes embedded system in order to ensure that the final product is free of deficit. The main purpose of embedded testing is to verify and validate whether or not the final hardware and software product meets the customer's requirement. So software testing in embedded system is an excellent approach to ensure safety and critical applications, such as medical equipment, aviation, automotive industry and so on. And maybe this image is familiar for you because we commonly see it in the software testing but we can use the same reference for system that involves hardware. Although the cost is higher for science that is higher for this kind of project that involves hardware, science that usually not all the developers can have the device for testing purpose. All right, so how are test unit testing on embedded system? For embedded system on this test type, developers test at a model level by writing substitute objects that we know as mock and to replace, in order to replace the rest of the system hardware and software. So in this case, the developer mocks both. And at this point in the development cycle the test focus on the logical performance of the code and are usually executed quickly. Here we can see an example of how we can run test for embedded system. So then let's talk about integration testing for embedded system. And this have a unique feature which is focused on the actual device on which the software is running. And this makes a little more complex to write test because it is really complex to simulate conditions compared to the unit test. This takes a little longer to be executed. And you need, and another thing is that you need to have access to the device to run it. So if you have a big thing, definitely you need to find a way to have access or to provide access to the device and in order to run integration test. So here we have another example of how we run test in our project. This is specifically integration test. So then we have in this process when we were doing this different type of test. We didn't talk more deeply about end-to-end test because on that process, that kind of test was manually and that was done by two doctors, a paramedic that was, and that were the stakeholders that compromised it with the project. So they were doing that end-to-end test and as I mentioned before, they were doing manually. So basically that kind of test usually, usually that was longer test. So that take almost one or two days to accomplish that end-to-end test. But on the unit and integration test, we found some challenges that I'm going to talk a little bit. So the first one was hardware dependencies. It's one of the main difficulties that we face during integration testing, signs that in some case, there is limited access to the hardware. In some case, emulators and simulators can be used, but they might not accurately represent the behavior of the actual device could give a running idea of the system performance and the usability of the application. So then we have open source software library and most model of the IoT area have open source library. And regularly this comes without automated testing. So there is a great range of test combinations and resulting scenarios that you can do when you implement it in a new hardware that you are building. And that was something that happened to us. So something that we were using some open source library after we implemented it, we went through how we could test this because it was a different combination that we did at that moment. And the scenarios that we wanted to cover was a little different of how the library was built at the, or was, yeah, was built at the beginning. Another aspect that this is related with the FETs that is software versus hardware. Another aspect is that when you are developing software for newly created hardware, there are possibilities that you could face with hardware failures at the beginning phase. So the FETs that you could phone are not limited to the software, since that it can also be related to the hardware. We faced that every time that we changed it in our project, every time that we changed the version of our device, that was two times that we did this change. And simple things like install a boot zone, could face in this part. So, yeah, and sometimes you could think that it's software, but the truth is that it's something related with the hardware. Then we have reproducible the FETs. The FETs are more difficult to reproduce or recreate on embedded system. This makes the build process procedure appreciate each the FETs sustainably, more than the standard case. Apart from gating as much, that has a sensibility needed to alter the system to find the basis of the FET. So this happening also to us when we change it to another version, as I mentioned also in the previous challenges that we resolved it. And this was more specifically when we installed a sensor that was working with inverted logic. So when this was detecting something, it was emitting zero and when not, it was emitting one. So we thought that we had at the moment a software defect, but since that we have some tests, some automated tests, we could see it in a fast way that the problem wasn't the sensor. All right, let's go back to the journey that related with the test of the device. So after the doctor's recommendation, we started working on the improvements together with Kia Motors, with the support of Kia Motors. And when we run the next test on April 27th, yeah, we run this, but at this moment the device was not approved. At this moment, they recommended to us to do another change. But finally, on May 12th, on Ecuador was published for the first time a normative or a law that have the guidelines on which any initiative related with mechanical ventilator could be assessed. So it was really helpful for us because for the first time, we have something to follow and to build our device. And this was also helpful for another initiative because we, at that moment, on the country, aware of multiple initiatives related with mechanical ventilators. So based on the new legislation or law, we did some improve in our device and we started to have progress on the approbation process. We did our final test on another laboratory and it was successful. So then we were ready to deliver our emergency mechanical ventilator. And that moves us to the next part that is, as you can see here, these amazing people did possible what we accomplished. And also, as you can see, all these iterations that we have in this short time, in this short period of time. But in this image also, you can see that we had, or that we did three different versions starting from the wooden version and then to the most sophisticated version. So we delivered our device on the Esmeralda City where this project started in an official ceremony where was present local authorities. And this is how looks the device that works with the unbooked bag and to account. So let's play the video. So then let's move to the next part that is continuous integration and continuous deployment. Within that, what made possible, iterated in a fast way, definitely was the implementation of continuous iterations and continuous delivery process from the beginning. So after the development that we made, a new create, I mean, create a new feature or extended it as what's really easy. So in this part, we created a kind of workflow. As I mentioned at the beginning, we use it, it had actions in order to integrate and deliver the software to the hardware. So how looks this process? So basically, let's say that a developer complete a feature or a standard one, this developer do a push to the repository. So then we have on this part, we have the key cut actions that is listening for any change on the main branch. So basically when a new change is done on this branch, this file is triggered and this file and start getting all the code that have the main branch. Then this proceeds to install dependencies. And as the first step, this start to run unit test and for that unit test, basically what we need at the moment is the core C++ language and some libraries. So this is how we can, how we accomplish it through on this unit test. Then after this pass is done in a good way, I mean all the test pass in this step, then as necessary, the violent front integration test. And for that, at the beginning, we thought that we could use Arduino and some libraries, but through was at the moment that as I mentioned, also something that of the challenges that I mentioned before, in order to run integration test, we need to device or we need to communication with the device. So in order to accomplish this part, we use it platformer platforming agents. So basically we run platforming agent in Raspberry Pi and this was connected to the internet, but also this Raspberry Pi have connected an Arduino by USB. So every time that a new developer was doing a change, the pylon was using an agent and this agent was talking to the Raspberry Pi and saying Raspberry Pi have access to the real device, all this integration test were working in a good way. So that was how we accomplish it, run or have a communication with the real device. And then also this step worked for the deploying firmware, saying that we have connected to the real device and the unit test, the integration test was working in a good way. Then the last step that we had is to deploy the firmware. So how looks this pylon? So the good part of Github action is that you can define it with code. So we define it by using this pylon as code and we added another part, I mean, we added the part to communicate with the device as you can see on the line 29, the way that we can communicate with the agent is by using a token that you generate the first time that you're, I mean, after you do that login you generate that token and then this is the way that how the pylon can talk to the device. The most important here is that you can get it by using secret. So there is not always, you know, to dispose this information. All right, so this is an example of how the pylon runs. So this is the first step where it's building and running tests. The first one is unit test. And this is, well, this is fast, but then we have iteration test where the pylon communicate with the agent and this takes a little more time. I mean, if we translate this more seconds, basically. And then, as last step, we have the field work deployment. It is communicating with Raspberry and then it's uploading the new field work version. This step, yeah, this takes a little bit time. It's done. So this is an example of how we were integrating code and doing deployment of the field, the field work to the hardware. So if we analyze all these or what we accomplished on this project, we have these outcomes. So we did basically five showcase in three months. We push it, you know, to have one load for validation use of emergency mechanical ventilators in Ecuador. Also, we did basically three versions of mechanical ventilator. And finally, we delivered it. We had the validation of the correct operation of this device in Esmeraldas. Also, the key factors that we found on this project was the commanding of a stakeholder for the evolution of the product. And these stakeholders was where the doctors, technical staff and sponsors. Also, another key factor was the design of an evolutionary software architecture. So this allowed us to iterate in a fast way and build an extended device and features with confidence. Another key factor was implementation of this strategy. So this was a decision that we take at the beginning and was something helpful for us across the project. And then finally, the implementation of the strategy for framework and continuous integration and deployment was also another key factor in this project. So finally to recap, we have three points in this project that we want to share with you. And first one is it is very important to define the language of the business at the beginning of the project in order to show the value that you added on the each stage. So this is related with the first phase, I mean, with the first test that we did in your project. So at that moment, we were using technical language to show this frequency into the device. But at the end, our clients that were the doctors and paramedics, they never understood what we were trying to say at that moment. So they say, you don't have frequency because you are not using the correct language. I mean, the business language. Then another point, the second point here is for the evolution of the IoT product, the community of stakeholders is very important. And yeah, that was for us very important because also the doctors as I show it to you on the test part was they were the persons that were running the end-to-end test. So they were validating the current functionality, the current behavior of the device. And also they were at the moment basing on their experience, validating that since that this was a new device, validating that this could be something helpful for any doctor on the contrary. Then we have the point number three, and this is the implementation of automated testing, integration, and continuous deployment of the rapid delivery of the valued and give the technical team the confidence to add a new functionality or send the product. So I would like to share it with you. This is an example IoT project set up basically on this repository. You can find the set up or an example of set up where you will have features like unit test, integration test, and pilot for any IoT project that you would like to build in the future. And here we put how we also, how we beat the set up in your project. And then we have Q&A. So feel free to ask anything related with the project. I will be asking questions also on the social networks. That's it. Thank you.