 Hello everyone, this short video is about the data's changes to the DHIS-2 to RapidPro integration middleware. This middleware implementation has three key goals and deliverables. The first goal is to have a routine synchronization of RapidPro contacts with DHIS-2 users. Here we have synchronizing DHIS-2 users having valid phone numbers to RapidPro as RapidPro contacts. The second goal is to have a webhook consumer to receive aggregate reports from RapidPro and transfer them to DHIS-2 as data value sets. The last goal is to have automated reminders to RapidPro contacts for overdue aggregated reports. We can identify three key components in the high level architecture of this implementation. So obviously we need a RapidPro instance and a DHIS-2 instance and then we have the custom made camel and spring based middleware. So when we are synchronizing contacts, the middleware reads users having valid phone numbers from DHIS-2 and then it creates or updates RapidPro users and when receiving aggregated reports from RapidPro users via SMS or any other communication mechanism available for RapidPro. The RapidPro flow calls the webhook implemented on the middleware and then middleware translates that payload into a DHIS-2 data value set payload and updates the DHIS-2 instance with the corresponding data values. So let's first look at how we can start the middleware. I already have the pre-built DHIS-2 RapidPro JAR file under the target folder and then this is how I can start the JAR as a service. So let's first look at how we can start the middleware. I already have the JAR file pre-built under the target folder and this is how you can start the JAR application. So I can specify DHIS-2 URL and the credentials for DHIS-2 instance. I'm using the publicly hosted Play DHIS-2 instance and these are the credentials for that instance and then we have RapidPro hosted under one of our domains and then I have to specify the API token for the RapidPro instance. And you can obtain the API token by navigating to your profile. Under here you have the API token, you just have to copy and paste it and then I have specified the sync schedule expression which is for contact synchronization. This specifies how often we need to do the contact synchronization. So in this case I have scheduled the contact synchronization to happen at every 30 seconds just for the demo purpose. And I have created a SH file by specifying all these parameters just for clarity. I can start the program by running this SH file. Then it's going to start a Spring Boot application and Apache Tomcat instance and expose the webhook and the rest of the services via a port that we specify under the configuration file. So here is the configuration file, I'm not going to go through all the configurations here because we have them on our documentation. The only thing to note here is the server port. So if you want to change the port, you can change the port here and rebuild the file and then it will run on the desired port. Okay, now let's look at how the contact synchronization works. So in order for contact synchronization to work, the middleware picks only DHS2 users have invalid phone numbers and they will be periodically synchronized to RapidPro as RapidPro contacts. And the existing contacts which matches with the DHS2 users will be updated if there are any updates to the DHS2 user. And apart from updating the contact number, in order for the rest of the flows to work, we need the DHS2 organization unit and the DHS2 user ID. So the middleware will synchronize those details as additional fields into the RapidPro contacts. So let's go to the contacts page of the RapidPro instance and we currently don't have any contacts and our middleware is currently running. So let's log in to the DHS2 instance. So currently the DHS2 instance does not have any users having valid phone numbers. So let's go ahead and update some of the users to have valid phone numbers. So I'm going to select this user and add a valid phone number here. So as I mentioned, the contact synchronization happens at every 30 seconds. So let's wait for it to complete one round of synchronization. And please disregard this error, it's actually a warning where it says some of the contacts in the DHS2 does not have valid phone numbers. In order for a phone number to be valid, it should have the country code. So in the play instance, there are some contacts which does not have country codes. Okay, now the synchronization happened once. So let's refresh and see whether it has synchronized the contact that we just updated. So it has synchronized the contact and if I go into the contact, I can see that apart from updating the phone number, it has additionally added DHS2 organization unit ID and also DHS2 user ID. So these IDs will be used in the aggregation flow to determine the organization unit of the events that are submitted by this particular user. So let's add or edit one more contacts. I just have to add a valid phone number. Let's wait for the middleware to complete another round of contact synchronization. It's updated to and it's going to have the same custom fields updated. So that's about contact synchronization and now let's see how we can achieve the second goal of the middleware, which is getting aggregated reports from RapidPro and updating them as data value sets in DHS2. So the main job of the middleware in aggregated values synchronization is reading the aggregated values from RapidPro, which comes in this format, which is a very minimal format which can be accommodated within an SMS and then translate them to a format which is compatible with DHS2. And as I mentioned previously, the second task of the middleware is to set the correct organization unit into the data values by reading them from the corresponding user. So in order to demo this integration, we have created a dummy RapidPro flow, which is about collecting information about OPD attendance. So this is the expected message format for this flow and we have the prefix APT. This is used by RapidPro internally to identify which flow should be triggered when a message is received. So the first thing we are doing in this flow is removing that APD command. So we are removing the APD command here and then sending the rest of the message through the flow. And then we process the OPD new attendees, which is the first number in my message. And then the second number of the message is OPD total attendance. And then we do a validation here because the total attendance should be greater than OPD new attendees because the total attendance include new attendance and the old attendance. And if the validation passes, it will extract the second number into the OPD total attendance and save that into this result name. And then it will extract the OPD expected mothers and then we have OPD missed appointments. So likewise, it will extract all four numbers into different variables. And again, we have a validation. And then here comes the most important part of the integration. So at the end, so now we have extracted all the numbers and assigned them to flow specific variables. And then we call the webhook. So the media that we have started has been mapped to a domain name to make things easier when integrating with RapidPro. And then we can call this webhook by specifying dataset ID of DHIS2 in the query parameters. And in addition to the dataset ID, we can specify report period offset, which is optional and defaults to minus one. The purpose of report period offset is, as you know in DHIS2, when we are reporting aggregated events, we are reporting, I mean, it allows only to report the final period or the period that just ended. So that's why we have provided the capability to provide the report period offset and default that to minus one. And then for testing purposes, optionally, you can specify or override the organization unit ID. If you don't specify the organization unit ID in the query parameter, the middleware should read, will read that from the user and plug it in. But due to the limitations of the RapidPro simulator, we actually can't specify the additional fields for the dummy user that is being used by the simulator. So we don't have the capability to specify the organization unit ID or the DHIS2 specific custom parameters that we have added for the synchronized contacts. So that's why we have to specify an organization unit here just for the demo purpose. So now let's see how we can create or configure the DHIS2 to accept these aggregated values. So I'm going to demo only the OPD new attendees case here. So we have to go to the maintenance app. And then we have to create a new data element to capture the OPD new attendees. So I will name that SOPD new attendees and as the short name I will add something. And then the most important thing is we have to match the code of the data element with the result name that we specified in the RapidPro flow. So I'm going to copy it and then add that as the code. And the domain is going to be aggregate. Yeah, those are the only configurations that we need to do at the moment. And then since we are using the data value sets API, we need to create a new data set. And since I am demoing only the OPD new attendees, this data set will only have one data element, which is the one that we just created. So the data set name can be something like this OPD attendance. And we will be, I mean, I will set the period type to weekly, can be monthly as well, anything. And then I will add my data element into the newly created data set. And I will assign the data set to an organization unit and save and I might have to update some sharing settings as well. Now let's see whether we can access this new data set from the built-in applications of DHS2. So I assigned my data set to this organization unit and if I scroll down, I can see that it shows the OPD attendance data set and I set the period type to weekly. So it is showing me a bunch of periods, weekly periods, and if I select that, it provides me the capability to add values for the OPD new attendees data element. So now since I just created the data set, I might have to copy the ID of the data set and update that in the webhook. So this is going to be the ID of the data set. And I believe the organization unit has the correct ID, it has the correct ID already. So now we can use the simulator to send a message and then it's going to go through all those validations and it sends us a summary of what we have just entered. So we have sent 10 new OPD attendees, 15 total, 5 EMTCT, and 3 expected mothers and at the end, if the webhook sends us a 204 status code, it's going to send us a message back saying the result has been sent to the webhook. And if the webhook does not send a 200 error code, it's going to send us a message saying that the webhook call failed. Now if we go back to the data entry application and if I select the OPD attendance and select the minus one period, I can see that it has received the value that we just sent from the rapid proof. So if I need to update that value, I can do that just by sending a new message. So let's make it 12, so it says results sent to the webhook. Now if I select that period, I can see that it has been updated with the new value. And also if I change the offset to minus two, I can allow the users to enter values for one period before the last one. So let me send the same message. Now if I select the week 31 that has been populated with the value that we just sent and the rest of the periods are empty. So that's about the co-functionalities that we provide by the DHIs2 to rapid-pro integration middleware. And in addition to the co-functionalities, we provide some tools to monitor and troubleshoot the errors that could raise from the middleware. So we basically provide two tools for this purpose. The first one is HavTO which is a JMX backed tool where you can use to read JMX stats reported by the middleware in a graphical user interface. And the second one is a built-in H2 database where we send all the errors or the dead letters. So in order to access the HavTO interface, you have to go to the management path of your domain and the default username and password are DHIs2 rapid-pro and the password is same. And here you can see that as we are using camel to do this integration, HavTO lists down all the camel-specific routes and it also shows how many times each route has been executed and whether there were any failures and things like that. So for instance, the DHIs2 route has been executed three times and there are no any failures. And this is the route that handles aggregated values. So if I send one more message through this route and if I refresh my HavTO interface, it's going to show as four messages because it counts the last message that I just sent. And similarly, you can use this interface to explore other monitoring tools that are provided by HavTO and one more important thing is you can explore the error law. So since I mentioned that we are continuously printing some error logs here regarding the invalid phone numbers that are available in the DHIs2 instance, we can see that error message continuously getting updated in real time on the UI of HavTO. And the next tool that we provide is the deadletter channel which is provided through the H2 database here. If there are any message failures, those will be sent directly to the deadletter channel table. So right, at the moment, we don't have any failures. So if there are any failures, this is where you can look into those. And if you want to manually replay them, that's up to you. So that concludes the demo on DHIs2 to RapidPro integration. And in this short video, we have discussed about the architecture of the implementation and the co-functionalities as well as the monitoring tools that we provide through the integration middleware. So thank you very much for watching.