 Alright, so today I'm going to talk a little bit about capacity building along with my colleague Pamela. So he's going to present a bit on Sri Lanka, and I will talk about some some principles overall. So, I'm just going to refer back to our tracker house that we kind of initially started some of our discussions with to see some of the court areas where capacity building is involved now, you know you could make the argument that many of these other areas might also be impacted and we're just kind of going to talk about some of the key pillars and how we might make some considerations around these, you know, when you're making a training plan. So as I mentioned, I'm, you know, I will be presenting along with Pamela so I will present kind of an overview just of some of the kind of common adult learning principles that I think are relevant in this context. So in particular, I'm going to try and relate them to various tracker implementations various tracker concepts, then we'll get Pamela to present on some use cases from Sri Lanka in terms of the training he's performed. And then we'll just wrap up the session with some information on the DHS to Academy program in particular, some of the modifications we're making to the tracker configuration curriculums that are available. All right, so before we get started and jump into the kind of core of the presentation, I have a number of questions that I'll be asking you as we go through this presentation. So, if you can go to the mentor which you should all be familiar with at this point, and use this code in order to answer this question, I just want you to list, each of you will be asked to list three qualities of your ideal trainer. You know, what do you think your ideal trainer should have in terms of, you know, their various qualities when you're when you're kind of working with them in the group, just bring it up here and you'll see the code here as well. Okay, so you can log in on your mobile or use the use the web whatever your preference is. And we'll just respond to that so let me see some of these responses come in. So as this is coming in, you can can already start to see some some similarities between the responses knowledgeable is a particular one that you'll see repeated patients as well. I like to understand participants. I think that's important. I want to touch upon a couple of these as they're coming in. So, so there are a couple key things that I think are important when we're when we're doing training that, you know, sometimes we do kind of tend to forget a little bit. I'm Kim mentioned understanding participants and I think that that is kind of a kind of a critical one. Being empathetic to their needs. I think many of us forget as we come more and more fluent with various tools and concepts. What it's like when you're first starting out to learn something and being empathetic to those types of needs can be a bit challenging. You know, the farther and farther we get down the track in terms of our expertise regarding a particular subject if you are particularly knowledgeable that emerging as a key concept you know the harder it is to kind of make things kind of a simpler in a way that people will understand we wanted to have a bit of a bias in terms of how we present things in terms of a kind of a more advanced way that might be difficult or challenging for those who are just getting started. I think another you know in terms of being knowledgeable I think that's also, you know, a very good one, you know you want the person to be, you know, quite knowledgeable in the subject. I think it is important to also understand when you're doing training you might have members of your team, however, that may not be as knowledgeable in the subject area but do have a lot to contribute in terms of formulating training concepts and helping you kind of conduct the training as well so there are some relatable skills because someone being knowledgeable about a particular subject and someone being able to kind of adequately then share the knowledge of that subject with others. These can often be, you know, difficult different skills that we kind of see as we come in, you know, regarding various concepts but but I think that these are overall great and, you know, preparedness is another one I'll talk about that as we go through the session good communication. You know these are all quite relevant in terms of, you know what we would think so I think we have already some agreement here in general with the type of trainer that we're kind of considering, you know when we're working with capacity building. So those results will continue to come in and we'll make a recording and I'll share those with everybody at the end, okay. All right, so I kind of wanted just to touch on this a little bit when we're working with capacity building. Actually there's there's two separate areas that we are that are actually part of this framework, and we often focus on kind of the left side of this diagram first right even I kind of like misled you in the way to kind of say well, what is your ideal trainer. When we're dealing with capacity building there's both learning solutions and these are, you know, various training solutions that we might discuss. And then we also have the secondary part and this is often ignored. This is when we're when we're looking at performance solutions and this is really looking at, you know how we evaluate those who we work with hiring practices so if you have a budget that you're putting together, and you're able to make considerations around, you know various partnerships that you're perhaps, technical assistance that you're bringing in, or even your own internal staff, in terms of looking at their own roles and reviewing that and ensuring that you're getting the right person to fit that particular role that you that you are kind of you know for seeing would have some impact for the longer term in the future. Now for the sake of time and kind of to just to ease this session a little bit, we're going to focus on learning solutions, but I have posted. Or I will upload to the Google Drive, some more information on performance solutions because I think it is something that is ignored a little bit in terms of, you know, when we when we think of capacity building kind of automatically jump to training in a lot of cases right. And that's even you know the way that things are posed. You might get a request for example that you know I need training on subject a right, and is that really the solution that's going to lead to the best outcome right and we often don't think about this in such a way that, you know we go through the process and kind of try to understand better, what will lead to a successful outcome, when you're doing this type of kind of knowledge building, or whatever it is that you're trying to achieve right. So I'll make sure to post some some more resources on this. So you can get a better overview of what this represents, and the different types of performance solutions performance assessments that can be performed in relation to capacity building. All right, so I just wanted to introduce that as a concept, but for the sake of time, we will focus mostly on learning solutions but just to make you aware that there is a second part of this that you know should be considered as well. All right, so we're going to go through a scenario, as we kind of define this little bit, and basically our scenario is looking at a tracker program right so in our scenario we're implementing a tracker program for coven 19 vaccination. An initial training was performed where a training of trainers approach was used. This was cascaded down to your district and facility staff right so you performed training at the national level for the teams. National team went around for or did this online perhaps. However, it was necessary and tried to train the district and facility staff. Now you're now noticing that several districts are struggling to report their data. So, so what would you do, you know, we're going to talk about some approaches that you could potentially take what when you're kind of going through this. All right. So I wanted to introduce a bit of a model that we use in learning practice, and this is a very common model that's been around for some time. There are some issues with the model but it's a good kind of framework for us to understand the needs of our learners and kind of, you know, deal with how we're going to kind of respond to the various requests that we receive. The model is called the adding model is just the acronym of what we see here standing for analysis design development implementation and evaluation. So, if we can, we can run through these steps and I'm going to explain each of these steps but it doesn't need to be a large process in the sense that you know if you're kind of well steeped in the needs of your own implementation. And a lot of this will come out naturally, without you having to go through, you know, a very detailed process of course I will discuss on some of the detailed process if it is a very new task that you're considering. But you know it doesn't have to be a long drawn out process in order to go through this right. So we're going to touch on each of these topics and kind of relate them to is in particular, you know how we would kind of work with something in our in our scenario that we presented earlier. All right, so the first part of this is looking at the analysis, and what we're trying to do is assess learners needs. So we have if we're if we're doing some type of training, or maybe even if we're doing some type of performance analysis where we're trying to make some job modifications etc. Okay, what we have is our desired results right there's there's somewhere where we want to end up. And we have our current results so this is where the group is now as you're assessing them. And then we have to kind of assess well what do we need to do to take people from where they are now to where they want to be. Right. And when we're looking at this we need to be a little bit realistic, right with our expectations. So in some cases, we have to be a bit careful, you know, regarding the pace of what it is that we're trying to achieve. If we do it too fast, we might lose people and then the process to achieve these results, it'll break down a little bit right. But of course if we can kind of separate this out in ways that make a little bit of sense. Then we can hopefully get get to our target area. So there are many different types of needs analysis you can perform to determine core problematic areas. So I have actually posted this resource on the Google Drive folder, just so you can get a better understanding of what different types of needs analysis there are. I'm not going to go through all of them at this point in time. But you can have a look at this resource just so you understand this a bit more. But once again, this does not need to be a drawn out process. Right. The core of this is to understand the needs of your learners and you might understand this already right. In some cases you might want to try and you know if there is a particular bias leading you in some direction, then you might want to just have a quick discussion with some, some of the individuals involved in your training to understand that a bit better. But you know if you already have a general overview of some of the gaps that are present in your system because you're working in that system all the time already. Then it's not something that you need to get into a lot of depth regarding you know all these various concepts right. But the whole idea is just to identify the gaps that are present with the team that you're working with. Right. I've also posted. So for this Academy, a couple years ago when we were coming up with it, we did perform a bit of a needs analysis on this I posted that as well. So you can see what a detailed needs analysis would look like. So that will be available in the Google Drive if you want to have a look at that okay. So the next part of our model was looking at design so this is the design of many different aspects of your course alright. So, going back to our example as part of our reporting, we want to teach people how to use event reports, right. And we want to teach people in particular how to create line list that include data from more than one program stage. Alright, so I'd like you to go back to the mentor meter. And if we can answer this next question, right. So if you were to create a learning objective for, you know, in terms of teaching people how to use event reports to create line lists that include data from more than one program stage. What would your learning objective be for in such a scenario. Alright, so just give everyone a little bit of time to formulate their response. And then we'll discuss that a little bit more. Okay, so we see some other responses coming in. And it can actually be a bit difficult to develop a learning objective right so we have some that are coming in that are you know a bit more specific, and some that are you know, a bit, a bit vague, you know. So, so when we're developing learning objectives actually specificity becomes quite important in terms of developing these all of the other responses come in, as we continue this discussion. When we're looking at learning objectives, at least internal learning objectives so this doesn't necessarily need to be shared with your larger audience. But we can actually break down learning objectives into three predominant areas. Right. This is called, this is a very useful model in terms of developing specificity for the learning objective that we're working in with. And this actually becomes kind of a critical part in terms of defining, you know how how we're training and how we're measuring success. Right, so that is the big thing. If we have kind of vague learning objectives like, Oh, by the end of the session you will learn how to create the vent reports. What does that actually entail right. What do you actually want them to do by the end of the session or want them to be able to achieve within that kind of, you know session of working with event reports right. So the three conditions are the condition so what will they be provided with. Okay, or in what scenario are they going to be asked to perform this with, and you try and reflect a real world scenario as possible so if they're not going to be given any aid they're going to be asked to do this from memory, then that needs to be kind of a focus in your session right. If you're if you're going to provide them with some type of manual or job aid or some other standard operating procedure. You know, then that's the kind of condition of them performing this right they will have something to refer to. Right. The second part of this is the performance. What will they actually be expected to do. Right so in this case I put the learner will construct an event report line list of enrollment type. And the third part is the criteria, and this is often the part that is gets a little bit muddled when we try to define our learning objectives right. You have to be clear in terms of what defines the success of that individual when they're you know when they're trying to meet the goals of this session. Right, so we have to be a bit specific here. So we can measure or observe their progress against what it is that we're trying to show. So in this case I put to the extent that the event report includes parameters for the organization unit period and data from at least two program stages. And this might seem a bit involved in terms of, you know, when we're when we're thinking about what we're doing, but it gives us kind of very clear criteria on in which we're going to measure their success against right. We don't necessarily have to perform a test or anything. I mean order to to kind of check how they're doing, you can just, you know, observe them and kind of see how far they've gotten. And if they're able to do what what we've kind of outlined, then you know we can mark this as a success. So this is much more for own internal use. So we can, you know, understand exactly where we want our learners to be. So this is a useful model to use before we actually go to develop our course material, and before we actually go to deliver our for our own internal purposes in particular like you might not want to present the learning objectives this way to the learner right this is maybe a bit too much detail. But for our own internal purposes to really understand what it is that we're trying to do. You know it is quite helpful to as a frame of reference, you know for the subsequent portions of our material design and our kind of learning scenarios. When we create these learning objectives, there's actually many different models we use in order to define them correctly. And there's this pyramid that I'm presenting to you that is a common model we use in order to kind of match what we're trying to achieve with ways that we can properly measure and kind of define what it is that we want the learner to do. So at the top of this pyramid we have the ability to create new things without basically much guidance, create construct design assemble these are the types of words we use to kind of describe this. In a particular way, at the kind of bottom we have just remembering right so if we just want to show somebody, and we want to be able someone to be able to recall that from a, you know be able to recall that concept, you know that that's kind of the kind of a military level, but it's something that we build upon as we go through this. So this model, you know if I go back to my learning objective, you know I highlighted that word construct, right because the idea is by the end of this we want them to be able to create some of these on their own right. We don't want to provide a bunch of a hand holding basically at the end of the session. And with that in mind, you might have to go through some of these other principles, and before you get there right. So if we understand, you know what event reports are they're going to have to remember all the different steps associated with, you know, selecting the data or unit the period. You know, moving between the event and enrollment type line list. So there's a bunch of different things that are associated with actually creating that report on their own right. And you know we might have to work sequentially through some of this in order to reach that final kind of output or outcome that we're trying to achieve. So we sometimes use this in order to help us define our learning objectives and frame them in such a way that would be useful in the context of what it is that we're trying to do. So when we're looking at the design phase as well, we're also trying to assess the delivery method. Right. So how are we going to actually deliver this course. And because that can affect you know when you go to actually create your material that might affect then how we're actually going to actually, you know, word or material how we're going to create our material. What that might look like in a little bit more detail. So on one side we have kind of in person training job aids manuals things like that. And then on the other side we have online solutions right and, you know, due to the nature of the current situation. So this is the direction that a lot of people have gone in of course there's a there's a number of different ways though that we can deliver online training that can be useful right so right now we're having we're having sessions right so we're here together as a group. And we're all kind of gathered that way but you know there can also be other methods that can be used depending on what's useful in this scenario that you're looking at right. We can also have you know just basic online media content where you know you kind of assemble all your content together and share it with people. We can also call online blended sessions, which are a mix of self paste sessions right so you, you give them some material to look at, and then you meet together, you know, maybe one once every so often. And you go through the material you discussed right so you're not spending that time together to look at conceptual material, you might go through some practical considerations and discuss what they've reviewed and see if there are kind of any kind of prevailing issues from the material that they reviewed. You can also do the same thing with this kind of online or in person blended approach so what I mean there is you would have a mix of online material, it could be self paced or it could be and then you would come together as a group as well in person. And they're you know they're just some things as you've probably noticed by now. During this whole course, it's sometimes difficult to convey certain concepts, if through an online format and in that case you might, you might have some in person sessions as well. Right. So there are multiple ways to to kind of deliver this online content is kind of up to you to assess what you think is most appropriate. This is like a huge refresher training for a number of health staff, for example, if you have a program on data entry, right, then some of this might be helpful if it's kind of available as a refresher training, you know, on an ongoing basis, versus if you're trying to introduce something new, like a COVID-19 vaccination tracking system, you know, then you might have to come up with something new for that particular program, and you might want to be a bit more hands on with the delivery of that because this will be a new concept both for yourself as well as anyone being introduced to that concept right. So just kind of depends on the scale and scope of what you're trying to achieve and you know how useful it is for that repetition as well. All right, so for for online tools I just kind of put a reference for you for some of the available online tools that are available to build certain types of course material. I'm more familiar with, you know, we're using zoom and slack in this in this course and then you know you're probably very familiar with some of these tools are also Microsoft teams which many of you might be familiar with as well. In addition to this we have what we call learning management systems. We didn't use one for this course because we weren't really doing too much individual assessment, but if you are doing individual assessment it can be quite useful, especially if you're delivering the course online. And you want a method to be able to track your students progress and share the various materials with them. These types of things can be useful. I tried to highlight at least one Moodle the first one it's an open source system so you know there's no payment associated with it. So it's quite useful in that sense, but there's some other ones that are also quite useful and quite nice for you to compare. There's also a number of e-learning or authoring tools that we can use to develop our material. I've just put some references there for you so you can you know you can have a look at those on when you get a chance but there's a number of tools basically we can use to to build our course if we choose to you know doesn't mean we have to go through this route. They're just there is kind of supplemental pieces of information if you are interested. Okay, so. Next question. Without referring to your notes, can you name all the 17 concepts associated with the tracker house. Yes or no. Okay, just curious to see how our, our retention has been and be honest it's not a, it's an anonymous response. Can you name them all sure G. No, I can't. That's intense. We're we're seeing you know people are being honest with this and that was my expectation about this. Oh, we have one person. That's great. Okay, so we can see that the majority of us, myself included, you know, are unable to do this right. So, when we're developing material, there is a concept about our memory retention, right. The longer time goes on. The less we remember things right. So imagine if I were to ask you this question in a week. I want to remember some of those concepts now. But if you don't refer to your material. And in a week I asked you the same question. You probably remember even less of that model. Right. So, when we're when we're kind of building course material when we're developing, you know capacity building tools. It's important to provide people with enough context enough information that they can find what they need and refer to it later on. Right. So the whole idea is of course, you know, in a real life context, you don't need to remember all those aspects of that of that house. We have it as reference in multiple presentations in our documentation. You know, so if you want to kind of gather and gain an understanding of that, then you're able to. But imagine if we just presented that house on a slide somewhere, and we never shared it anywhere else. Right. It'd be very difficult for people to then go back and refer to it or really remember all the different kind of points that are associated with it. I mean, we should think about things in the same way. We have to realize that especially if we're not using a concept immediately, that the likelihood of someone remembering this is going to be quite low, right. You know, especially if you're dealing with a complex process, right, you know, you're showing someone how to configure a bunch of components of a tracker program, or you're showing someone how to create a bunch of advanced reports, or, you know, you're showing someone how to how to do some type of scheduling or some other process within your data entry, whatever it is that you think might be a bit complex or new for somebody, whether or not they're going to use it immediately, or you know they're going to kind of use it, especially if they're going to use it, you know, maybe a week or two or more from the point of when you're showing them this concept. Right. Then we have to kind of consider, you know, what the memory retention is of that audience and how we can kind of mitigate this a little bit as we continue going forward, right. So, when we're looking to develop materials, there's kind of a simple timeline for training materials that that we need to consider, right. So we have the before portion, and this can be minimal right you might want to provide a bit of material beforehand, but not you know it kind of depends on the training if it's completely new concept, you know, I would say we can be careful here in terms of what we provide. Okay, then then during, and this is where the largest focuses right, create a number of presentations exercises things of that nature for that during the training. But then what happens after the train right and that is a component that we often do not spend enough time on right where they're not given enough material to review after the training is over. You know in this case we have you know the full implementation manual both for Android and for tracker. So we there's a lot of reference for you to look at after training is over on top of all the presentations and everything that have been included. But the whole idea is if that stuff isn't there, you know, then it's something we really have to consider, you know, to make sure that we're able to meet the needs of our learners adequately. And as I mentioned, distinct processes or tasks, tasks that have a lot of steps associated with them. So if we're showing someone how to maintain a tracker program, for example, and it's the first time they're being exposed to such a scenario. So if you just do a demonstration of some kind, and don't have any supplemental information for them after the fact, you know that they might struggle a bit to kind of remember, you know all the different things that's all the different things that they've been shown in relation to kind of doing that in their day to day job. And of course if they're not going to be doing that for a little bit after then the likelihood of them remembering, you know what you presented during those demonstrations is even less. Right. So if possible, we try to take these kind of complex processes and break them down to those their core steps. And we can use those core steps to serve as reminders for that process that you're showing. Right. This is this is a bit more challenging than you think. But but the idea is, you know, the problem is, you can make materials too long, and people won't refer to them. It's too simple, and then it's not enough information for them to really retain or remember the concept that you're showing. So you have to find a kind of medium that kind of prompts people, prompts people's memory in particular to remember what it is that they've been shown with enough information that, you know, they can do the task that you show. But you know, not too much detail that it's just going to become, you know, a huge document that no one will refer to you because it's so dense. You know, people just don't want to read through it every time they're performing a particular task. So secondly, if you're doing a training of trainers right so that was our scenario at the beginning right we did a cast dating training of trainers, but now we're having trouble with people entering data in from the district right. What are you providing to them when you're conducting the training right. I hope you're seeing that things tie together a bit in terms of understanding the needs of our learners right we sometimes make assumptions about what it is that people need to understand, and then we don't provide people with the correct information going forward. So do they need training on the subject. For example if you're introducing that new tracker program. Are these people who have worked with tracker for some time and can enter tracker data, but haven't performed a lot of training in the past. Do they kind of need that full overview of how to enter that data how to work with the information within that program, and then also a discussion on, you know how to actually deliver that material. I think one thing that is often, you know, unfortunately, forgotten as part of this process is understanding of the program itself, right. So this is, you know, if you're dealing with in a health scenario that the kind of context the public health context of that program, and the different parameters that are associated with the work full of that program now that has been discussed in quite a bit of detail over the course of this Academy, but the idea is to make sure we marry these two concepts. So if you're doing a to T, you want to make sure you provide them with enough materials that if you're going to cascade this down that they're given a package of sorts that allows them to conduct that training successfully. So some examples of additional guidance, they can be in the form of standard operating procedures on various manuals and job aids, step by step guides video demonstrations, for example. But when we're making these types of things, we want to make sure that we go over them as well so if I'm introducing a standard operating procedure for example on conducting a training. And you know, as part of your training you actually want to discuss the various steps that are associated with that, go through some mock-ups and errors with them in terms of, you know, what that would entail when they go and actually conduct the training right. We find this is often a bit rushed. And then you know, unfortunately, later on down the track, you kind of see some problems emerge as they go to conduct training later on. Okay. We're still in this developing stage and you know, it can be a longer stage that the more prepared we are for this training, the better thing, the better your outcome will be, right. So, a well received model used in DHISU trainings I think is especially for trying to teach something practical, right. This is something we follow in a lot of our, a lot of our practical academies and you know just a lot of our country trainings as well. I think there's still sound. There's also a little sound. Oh, hello. Yes, you're back now. Thank you. Could you please repeat the last 15 seconds or so. Thank you. Yeah, sure, sure. Sorry about that. All right, so these first two steps, explanation and demonstration, they might be tied together a little bit. So if we go back to our specific example on showing people how to use event reports. And we might want to pull up a couple example event reports that have information from two stages. We might also want to show the program that we're working with, you know, hopefully before then you would have explained the program. And you know all the individual program stages, for example, that make up that program. So people understand the context of where you're pulling that data from, right. So if I'm building an event report from two stages, people should understand the various data elements, etc. And then those two stages so they can pull the right data, right. So we're trying to build on previous concepts it might also help you with your own kind of sequential understanding of how you present things right. Then we have some practice sessions and depending on the complexity of the concept. You know this is something we struggle with I think a fair amount. The concept that I think has been coming up a lot lately is this program rules. We see this on the community of practice when we're doing trainings itself. You know it's something we've continually struggled with in terms of how we get people to practice this correctly, and how we kind of deal with the transfer knowledge. There's always changing rules and the context of those rules are different right. So, so this is a scenario where we don't have a generalizable framework necessarily, because everything is specific to a local context right. So there are areas I think that we struggle with, even as a, even as the DHS2 team, right, and that we're trying to improve upon. But in any case as a general model, you know we kind of perceive this as important. Right, then we have the evaluation phase, and I put test in brackets, but as I mentioned before, we don't necessarily have to test everybody right. And we have to be careful, you know, a lot of people like this approach of testing. But when you're introducing a new concept to somebody and you know particularly someone who's not that familiar with what it is that you're discussing, you know testing someone that can be a little bit much in some cases right you don't want to, you know, create some type of barrier to learning the topic, or you don't want to make it overwhelming for somebody as well. There's a lot of observational testing right you can have people present things back, you can just observe the steps they're performing to make sure that they're able to carry out the same models that you're doing. You could have ungraded guided exercises that you know allow them to follow the same steps that you're showing for in order for them to achieve the final output. So the evaluation doesn't have to be a graded evaluation in all cases, or like a formal test of some kind. You just need to be able to observe that they're able to achieve the final learning outcome. Right so I've just linked a couple examples of materials, you know that can be used for reference. So we have the implementation manuals on the documentation site for both tracker and Android available. And also some generic modifiable end user material for both end user and trainer, trainer's guides for COVID-19 surveillance vaccination and adverse events following immunization. And also some specific videos that we made for in this case for Vanuatu for their rollout. So just so you can get in a sense of what's what's provided at various levels. So we have kind of global scales regional scales and then kind of national scales as well. All right, and you can have to look at those whenever you get a moment if you think they're useful. All right, so we also have a framework for when we are developing our assessments, right, and if you remember these words here on the left side, they match that kind of pyramid that I showed you at the beginning, right. So depending on the complexity of the of it of what it is that we're trying to convey, and we have to select an appropriate method to evaluate them. Right. In the context of using DHS to some of these, you know, are not always relevant so for trying to create something right and like an essay or oral test to be replaced with, for example in oral demo, you know, where we get someone to show us exactly what it is that, you know, we've asked them to do to make sure that their understanding of this concept is clear, right. So that could be the replacement of the oral test, right, or an essay test could be just actually following along and doing some type of exercise. But we have to make sure you know in some cases like if I were showing someone to create an event report, and then I had a, I only had a multiple choice kind of quiz, right at the end of the day for them to perform I didn't have any practical exercises or anything like that for them to do, you know, that's not appropriate necessarily to guide or gauge their understanding. Right, we have to make sure that they're able to actually achieve what it is that we're trying to do in a practical way. Right, so we just want to make sure we're matching whatever it is that we're trying to show people with appropriate assessment methods, when we're kind of going down further down the track to making, making sure people can perform the concepts that we're trying to teach. Okay, so this framework is more just to kind of reinforce that concept, then to be some kind of prescriptive scenario, right. All right, so there's there's different levels of learning assessments that we can kind of create as we're going through and performing our training program. So we have in some cases the diagnostic or pre assessments. This can also be useful for our needs analysis really quickly just to understand what the strengths weaknesses and knowledge skills are of our learners, prior to actually going through the topic right. It also helps to develop the framework for kind of measuring success right can measure before and after and kind of get a better idea of where they are. So this is a very, very different type of assessment called formative assessment. This is used during the course of the learning activity. So this is just kind of chunking out your various topics and assessing them on one particular component. It's part of that analysis training for tracker, you might go over through creating various tables, you might also create some, some inputs using pivot table for example using program indicators right, you might want to assess each of these components individually. Then we have the kind of final assessment and we call this a summative assessment. Basically this is performed at the end of the activity right so this might be an exam of some kind might be a group exercise, you know in our case we're having you guys present back the project planning template as an example. It's not great and necessarily but we hope it will provide you with some critical understanding of how to use that template going forward for your own implementation. So it's always good to have something that kind of sums up the sum of their knowledge basically at the end. And once again it doesn't have to be an exam or test or anything like that if we think about kind of traditional schooling. But it can be just something that we can observe to ensure that people are able to grasp the concepts that were that we're showing them. There are some practicalities on the DHS to side right when we're developing our material and training. So, if we're doing some type of practical DHS to training, you will need at least one corresponding DHS to environment where you can conduct the training, right. The environment should reflect the real real workflow of the program that you're introducing so introducing a new tracker program, right, it should be the same right should be identical. So in terms of environment, you know, learners should be able to make mistakes without causing any problems to your production systems and data. Right. So, in a previous session, I believe Marcus talked about various environments that you might have for long term maintenance like a development environment, for example, right. You should also consider having some type of place for training, right. This doesn't have to be a dedicated instance for training necessarily, because it can be temporary or permanent depending on the frequency of your trading. So you can always take a backup of that development environment. And if it's clean enough perform the training there. Okay, as well and then just restore the development backup after the training is complete. But you just, I just want to kind of emphasize not to do training in your production instance, especially if you're dealing with some type of configuration topic. You know that can lead to a lot of problems I think down the line. Right. And in scenarios where you don't have data, for example, like you're rolling out a new program. And you don't have data and you want to perform some, some training on analysis, you know, then you might also need to make some extra considerations there. You know, do you populate the system with some type of data that you can actually use by just entering some cases or something like that, you know, and if that's possible and that will kind of be more effective to your learning. So of course. Okay, so next question. I regularly test all of my training material before I present it during the training. Oh, sorry, there's an option three, but really should be just yesterday. Did you test this before you presented it. I did. Yes. Except for the option three, apparently I forgot that option three great. Okay, so that's good. It's most of you are responding that you do you do perform some testing of your training material before it's actually presented that's that's great to see. So, I just created a quick checklist summarizing some of the concepts we've discussed. So when you're actually going to conduct the training. Now I know in reality doesn't work like this always, but the idea is to be be as prepared as possible. Right. So summarizing that we've discussed, we have a clear understanding of the needs of our defined learning group. We have all the fine learning objectives. We have decided what it is that we're going to discuss. We have decided how we will deliver these topics right it's going to be online in person etc. We've developed all the required materials and assessments for our learners to go through. We have prepared or populated any of the required supplemental tools. This might be your DHS to instance. You've prepared all your zoom meetings you've, if you're using a learning management system maybe you've uploaded all your content needed to support the delivery of that training material. And we have reviewed and tested the components for each of our topics. You know at this point in time you should be kind of well suited to deliver the training. Some of this happens in parallel right I know in reality it's not always the case where everything is prepared and ready for you to deliver during the training right. So you might be preparing things kind of in parallel to conducting the training, especially if it's the first run of that training right it often happens where you know just things happen. Other things happen you're so busy. And then you're kind of forced to develop things in parallel. You can still follow kind of similar steps, despite this and think about it, you know, on a session by session basis if that's the case right. So if you don't have much time, right and the practical considerations of this or implications of this are obviously that you can't go through all of these things, you know, at once. I mean you would have at least, you would have at least had some idea, you know what the topic is right, and you know, hopefully by then as well, what some of the material that might help you along with that. So if you are able to, you know, even if you're working late into the night before the training which, you know, we've all, I think, done at some point in time, right. You can still run through these concepts really quickly in your mind, just to ensure that you're able to kind of meet the criteria you need to make that session successful. Okay. So it doesn't have to be a long drawn out process if you don't have the time. Of course if you have a bit of time it's better to be able to go through through these things and assess them adequately, but you know if you're not able to do so. And you kind of are really crunched for time. Just keep these things in the back of your mind, because you know I knew I know that does happen and we're all kind of guilty of this occurring in the real world right. All right, so the last step is to kind of evaluate right so we talked about assessing. And this is different from evaluating right so when we're assessing we're asking the learner to do something and observe them and this is part of our evaluation, but there are other steps to evaluating right and I've identified kind of four levels of evaluation right. So the, the, the first level at the bottom here in red color is our reaction. Right, and we've been asking you for this the whole training right we ask you for feedback every day on all the sessions we want to know to what degree you reacted favorably to the material we're doing, you know, was it good was it bad, or their improvements we can make, or the things that you appreciated more than others, you know and this helps us to identify, you know, both within the training itself and for subsequent training know what we could improve upon. And what the learners, you know, kind of grasp upon if there are aspects that they're kind of missing out on during the training. That's also something that needs to be addressed. Right, the second level is this learning level so to what degree did we kind of acquire the knowledge that we were trying to share with you right. So we've created one quiz, we're creating another one for you to share with tomorrow. And we're also going to have a look at your project playing templates as well. Right. So this is all part of this, this learning kind of framework that we're looking at. The last two are harder to assess, and are often ignored a little bit right. So this kind of is after the training to what degree are participants able to apply what they learned when they're back on their job. And the level four is the results to what degree do targeted outcomes occur right so when I mean targeted outcomes, we're looking at kind of overall organizational outcomes so if you're implementing a tracker program for example, you know there are a number of outcomes associated with that obviously you want the data to be a good quality you want to be able to make make sure you have all your cases entered. You want to make sure that data is reviewed and use that dashboards are created. Right, this is looking more at organizational change and there's often. It's often difficult to make a, you know, a causal relationship between training and kind of this overall results phase, but we'll just talk about this really quickly right. So the first and second level I think they're well understood. You know, you gather some feedback on the training you also assess your learners during the training. So here's an example of the third level and how we can kind of generate information on the fourth level right. So as part of that house so I don't remember every part but the very top was data use okay as part of that house right so as part of data use, you know data quality is always linked to that right. So here's an example of modified this slightly right for for a behavioral kind of assessment on on data quality, right, and there's a number of different items here that are available and this will be kind of assessing the behavior you've done the training. So as an example if we went through our training and one component of that was data quality. You know you might want to see how their behavior changes when they go back to the regular job right and if it doesn't change then what was the cause of that right. So this is just a quick checklist and this is meant for you know someone else doesn't have to be a supervisor necessarily could be a colleague or peer right. So this will be beneficial if you have a peer assessing your, you know assessing what you're doing, and it's not meant to be kind of positive or negative, you know, saying oh they're not doing this and therefore they should be punished. No, it's really meant to try and understand if the if the training was successful right. Are they doing some of these things now on a routine basis, or you know how they just forgot and everything that was conducted during the course of the training right. And then to look at kind of this more higher level organizational change actually a lot of what Anna and Brian mentioned in their sessions, regarding evaluation is more pertinent. Right, you could try and summarize results from these type of behavioral evaluations to determine you know, at a larger scale was a training successful, but but actually you know you probably be referring to more of your evaluation frameworks to understand this in a bit more detail. So you'd have a bit more evidence to kind of substantiate the link between the training and the kind of final outcome. All right, so I just wanted to present some of these concepts. You know I went through that the entire model at the beginning here. Right. And so you could get an overview of what some of this represents right. And you know there's various ways we can apply this to training. And of course, we can entertain some questions at the end of the session. But I'd like to hand it over to pan mode. So he can present a bit on what he's been doing in Sri Lanka. Right. Thank you very much read it. Let me share my screen. So, in the previous session we learned about what are the different concepts of capacity building and what are the best practices that you can apply in in a country setting. So let us see how we have in Sri Lanka we have tried to apply different concepts and best practices of capacity building with the given with the provided limited number of limited amount of resources that are available in this lower and middle income country. So, for this presentation I'll be mainly sticking into two tracker programs one is the nutrition use case which I presented to you last week. And the next one is the COVID-19 tracker implementation which also I discussed in a previous day. I will not go into too much information about the background of these two programs but I will only talk about the capacity building aspect. So, few things that we have to keep in mind. Now, usually when we talk about capacity building the first thing that strikes us is like, it's about training. But in a larger context, especially in a country context it's not only, I mean it has a lot, a lot other things to do beyond the scope of training, right. It's a rather long term process. So, in this long in this broader topic of capacity building. We may have short term activities like maybe conducting training programs or providing material and resources, but then in addition there's a long term part of it as well. When we talk about capacity building, you always have to think about covering all the levels, right. It's not only about building capacity at national, it has to incorporate national district and sub district level, as well as the service delivery level. And also capacity has a lot to do with governance as well. So, what I'm trying to highlight is, in a country context, there has to be some governance mechanisms right so if you have some governing stakeholders like the Ministry of Ministry of Health, or maybe the central government. You will have to merge with the initiatives that they are having and also there has to be a proper advocacy that happens that takes place with the governing entities for us to build a capacity. So, this is exactly what we tried to do in Sri Lankan context. So, the major factor that I would like to emphasize in Sri Lankan capacity building is this high level health informatics capacity building initiative which which started about a decade ago. So, what we have seen in this country context is that mostly we can be very ambitious in implementing their various informatics programs. But the main issue that happens is the government stakeholders and people who are placed at district and sub district level who will be ultimately implementing the systems. They need a lot of capacity. I mean, you should in country context, the people who are right now holding these positions are mostly from administrative background or purely medical or health background. They will not have the sufficient knowledge and exposure on health informatics. So, in this country, they took this as a major major initiative, I mean to build capacity in health informatics so what they did first, they recruited some medical doctors, I mean these medical doctors were from kind of public health aspect. So, what they did first they had this medical doctors trained on health informatics by providing them with a master's program, which ran for like two years. And finally, after completing this master's programs, they were absorbed back into the Ministry of Health and they were placed at national and district level to implement information systems and also to build capacity at all level. So for example, the national level health effects program graduates, they'll be working mostly on the policy and the governing aspects of information systems whereas like people who will be mostly at district and sub districts level. They'll be focusing on very specific information systems and building capacity, I mean capacity as in at end user as well as ICT staff and other supporting staff capacity at all this level. This has been some, I mean this has been almost now continuing for like 10 years. So with this the country was able to build capacity at all levels so that in a short span of time, if the decision to implement information system is taken, they have the relevant capacity up to the community level so that they can quickly expand the system and roll out in a really fast manner. And then the capacity when you're talking about the capacity I mentioned like there are multiple levels. So one thing in Sri Lanka is that they have the local his group, so basically his group is well connected to the global his network. So here we are talking about some resources who have years of experience at local and international level. And also they are very well connected with the DHS to community as well as University of Austin. Right, so they even get regularly exposure to various training programs and capacity building activities that is conducted by University of Austin. So, these set of experts are the other first group, the country tries to contact in case if they're trying to start a new project, possibly on DHS to or even a general health information system, maybe on a different platform, because they have a significant amount of knowledge and expertise on implemented systems. They are the first kind of people that the Minister of Health tries to reach to. And in addition, we can also talk about program level or as national HMIS level core team. So we need this. Now why I mentioned it's not the national team because like for example in Sri Lanka we have multiple vertical programs like malaria program TV program who try to kind of implement their system in a vertical fashion so they have the national team and they have the district and the service delivery team with their own staff who try to implement this system. So for this of course when you're talking about this core team. It's a kind of a small team, like you may have 10 no less number of people in this team, but interestingly they have to be from multiple competencies right so we are not only talking about people from ICT background but you need to have some domain expertise from health, as well as healthy informatics. So this mix of people will actually function as the as the core trainers of this system, and also they will be involved in conducting training of trainers programs as well as material development. And this group are the ones who will be directly connected with the district team who actually take care of all the training programs and capacity building activities at the field or community level. So that is why this team has to has to have kind of multiple expertise together for them to have a successful implementation. And of course the end user level that's the next level on our window came out Sri Lankan context what we have seen is like, we really need to have a better engagement with the end users because the thing is like, you may have this field health workers for example, on the public health advice will be collecting data for multiple programs like it may be the nutrition program the maternal program orders even 19 control program. So, what we have seen is when you try to kind of build our competencies in them, like for example, if you try to build that build ICT competence so that they can use a mobile device, and maybe they know how to join a zoom call right. So this will really be helpful for not only for that safe for example nutrition program not only for the nutrition program but even if they want to introduce another program, this capacity is going to help. So we are in. So because of this we are talking about cross sector capacity building. I mean it's not a very hard thing to do, but it's just that sometimes we don't realize that how important it is to build this core competencies on ICT. So the very, very, very specific competencies we build on how to use the information system. And also the other thing like when we talk about the end user level, you, you talk about kind of a high turnover. So because of that we may need to conduct more refresher training program. More frequently compared to other levels that is national and district level. Right, so before the pandemic. This is how we try to conduct training programs, especially on tracker tracker related programs. So now, our general understanding is conducting training programs on tracker is much difficult than training people on how to use aggregate. So because of that, especially at the lowest level we tried to have multiple training programs with smaller or manageable size of a group of participants. We totally depends on how many facilitators or trainers are available. So what you're seeing here is one training program we have we were conducting before the pandemic for the nutrition program. So the one full day training program that we planned contained some knowledge building sessions like these are the theory or like I mean some presentations but much of it was for hands on training and also having some exercises and to let them use the tool and to kind of get I mean from them we will be trying to get the issues that they might they might encounter and we tried to kind of have a better discussion amongst the participants so that we can get a bit I mean all in all it becomes a really fruitful training program. So this to do that we had to have a manageable size of participants and that was the main focus before the pandemic. But then again, things change. The pandemic came in like it was it was from early 2020. But we were kind of ready with the level of capacity that we were building at all the way up to the field level because even before the pandemic, we had this online technologies like zoom which was available to the sub district level. So during the pandemic, just like what we are doing here is mostly shifted to the online training program. But then again, one major issue which we had in getting field health workers is that like, I mean what's the amount of knowledge and competence that is available for them to use these tools. So more often, at least this I mean at the level of the field level, we had to get down this field health workers at the sub district level to conduct this training programs. I mean, not always but in case if you detect that in a particular village or like some area that we don't have enough competencies or they don't have the necessary infrastructure to join these meetings online. But then, but then again like this has been one method that we have been trying so hard during the last 18 months and we have had kind of like moderate amount of success in conducting this online training programs. And of course, about user guides and user material. Now the thing is, like, we can't just ask the end users to refer to the DHS to main training and user guide right it's too comprehensive. So what we try to do in almost all of our tracker programs is to have this kind of single page user document, which is very colorful and concise, but the major difference between this one and one we use for aggregate programs is that for multiple programs because multiple steps are involved we may have to have a couple of documents not just one single end user guide but we may need to have multiple end user guys, but this they can have next to their desk can use as a data entry companion. And of course we are also using some training videos. Now what you're seeing here is the COVID-19 immunization tracker training video that we have designed and uploaded it into the YouTube. So what we ask is in case if someone can't join the online training program that we conduct, we just ask them to review this and also in case I mean at any time when they are entering data if they encounter any issues at that point of time also they can review this video, which is available online. Right. We also have a learning management system, which is available for a couple of health programs, especially the nutrition and the mch program. We have this system called ebridge, which is managed by the family health bureau. So, this is a, this is an example of a learning management system that should I just mentioned in his presentation so we are using the free and open source model platform. And we have designed this learning management system which has like not many at the moment we have around three programs. So here the health staff can enroll to the already there with the training programs which are already there, and they can follow this structured training program. Now this program only difference with the previous slide which I showed the video is that here we have a structured training program where we start from basics, and we get them do some hands on exercises some activities. At the end we have an exam so that we can give assist their competency and what a certificate. So this is this is again something quite new for all of us and we are really excited to see how our participants and the health staff is going to respond to this new learning management system. The next important thing is about user support and engagement. So, here of course like at like as any any training program any any tracker program, you need to have multiple levels of support available. So here we are, we have devised a couple of methods first thing is the user guides, like which I showed before, and that day I mentioned to you about this peer support so what we do is after training program. If, if by any chance any participants encounter as an issue. We encourage them to directly. I mean, not to contact the national level support staff directly but to contact one of the peers like I mean maybe a fellow community health worker and try to see whether they can solve that problem at that level. But they can put to the Viber group so this instant messaging group that we have. And in that case, these Viber groups are usually managed at the district level so that they can post in their local language, and even discuss the issues that are quite related to the region. And in case if this fails we also have other solutions like this remote desktop and the mobile solutions which are available in both the nutrition and the COVID-19 surveillance program. And also the next thing is the monthly conference. So here of course it's more about data use. So, I mean like data use is one major component of capacity building, because if the end user or the participants just don't understand the value of using this data or engaging with this information system, the system is not going to sustain. So here we tried with this particular exercise that we have, we call it a monthly conference. So what we try to do is, in this nutrition program, they are the end users use a mobile device to collect data. And in the end, a monthly conference, where they were like say a group of 20 community health workers. They kind of join this program which happens every month at a public health doctors level. So they are, they talk about various topics. So what we have tried to do in the last one to two years is that we have a agenda item where they discuss about the issues encountered in using this nutrition program as well as like what insights. They can get out of the data that is collected from their area. So they are asked to make a PowerPoint presentation. Just one person will present per month. And also, the public health doctor will go through the data that has been collected. And there's a discussion where he tries to promote the use of data that has been collected at field level. And again, kind of a larger level discussion that we have is called reviews. So this review programs, which has been even even present like last, I mean, I know this new method of review review programs we started two years back, but this is a concept that has been there for decades. What usually happens is the national level staff, like there's a group of around five people, they go to each district, at least once a year and review the district level data and have a discussion with the district level stakeholders. So these review programs is again another concept to promote data use. So here we talk about different data quality issues, practical issues, and also how to I mean like we share the knowledge that we have gathered from other districts with the with the So here, now with the COVID-19 pandemic, this physical traveling has been really restrained so we don't have a team from national level we don't go to each of the districts physically, but nowadays we are conducting it online. So what happens is we select a particular district on a given day, and we ask them to log in and get connected with Zoom platform. We have all the indicators that we are reviewing, which is available to national level as well as the district level in DHS to dashboards. So, so the people who come to the session they know like what are the indicators that and what are the items that we are going to discuss and review the program. And with this of course we have been able to ensure the transparency and previously these meetings have been like have even led to some conflicts, saying like the data that is interpreted by the national level is not the same data that the field level has connected and things like that but with these online platforms and especially with the DHS when the dashboards, this review exercise has been really easy to contact. So, these are few insights that I can share from the experience that we have around the capacity building data use and review in Sri Lanka. Thank you.