 In this section, we're going to talk a little bit about the people. What are the core roles and skills that are actually involved? What kind of people do you need for different hosting configurations? So what kind of work would they do? One of the things that we like to emphasize is that you're not just looking typically at a system administrator, you need to think in terms of a system administrator team. The DHS2, because it's Linux-based and open source system, this role involves knowledge that is typically different to what you often find in commercial enterprise software administration, which very often is windows-based. You need system administrator skills regardless of how the DHS2 system is deployed, unless you're using that last option of software as a service. It should be done as a team, not only one person, to ensure stability and continuity of system support and to reduce the risk or system capture. The reason why we emphasize this is very often we find that there's one person on the team who's the system administrator. If your system administrator becomes sick, if he gets run over by a bus, or if he simply decides that he's no longer happy with the salary that he's getting, he's in a very powerful position to influence the continuity of the system. So most good, stable implementations would have at least two system administrators who can stand in for one another. Often you might have one experienced person, and then you'd always try to have at least an understudy alongside who's building up the requisite skills. Frequently because these skills are not necessarily going to be available in-house, some of the skill is outsourced to local, country-level, or regional HIST groups. This allows for some economy of scale, as one HIST group can support multiple countries, and HIST groups can also, besides providing technical assistance, provide some sort of mentorship so that they are building capacity within ministry to eventually take over. System administrator skills are required for self-managed DHS2 system, regardless of whether you have it in the basement or whether you have it in the cloud. If you're self-hosting the system in the sense of you've bought your own service and you've set it up in your own server room or data center, then there are additional skills required other than simply system administration. Typically you're going to require a network engineer who's able to manage the network infrastructure, things like the domain name, service, the internet connection, firewalls, the like. You will need an infrastructure engineer who understands working with server hardware, data center infrastructure, air conditioning, fire controls, and the rest of it. What we've presented on this slide is kind of an example of, this is actually taken from a system administrator job description. What a system administrator is typically going to do is fault finding analysis, analysis logging information for reporting and performance, proactively monitoring the system performance, planning capacity for different patterns of load. He's going to be able to create and modify scripts or applications to be able to automate certain tasks. He's going to provide input on ways to improve the stability, security, efficiency, and scalability of the environment and would be able to conduct training and contribute to training materials related to server deployment and management. It's a typical skill set you're looking for here. Somebody who's passionate and interested in this kind of stuff, someone with fairly advanced scripting skills, including bash scripting, Python scripting, a reasonable track record, three plus years. We've sat here as a Linux system administrator, bachelor degree in computer science is desirable, perhaps with sufficient experience. It's not prerequisite. You want really in-depth knowledge of most of our systems run on Ubuntu server currently 18.0.0.4, mostly 20.0.4. There are systems running on additional, on different flavors of Linux, but for the most part, we're dealing with Ubuntu servers. It's important to have good knowledge on containerization within Linux. Using either LXC also Docker is becoming increasingly important. Good, strong track record and experience with the PostgreSQL database, focused on taking backups, restoring, monitoring, etc. Good hands-on experience with Java, Java virtual machine applications. Knowledge of provisioning tools like Ansible, considered advantage, knowledge of configuration and administrating of DHS to also be considered an advantage. It's possible to take a system administrator that's never seen DHS to and to train them how to work with DHS to easier than taking somebody who's familiar with DHS to and training them how to be a system administrator. Strong problem-solving communication skills. A lot of system administration is really about problem-solving, and so you really are looking for somebody with that curious, inquiring mind. Fluency in English is often an advantage. We know in some environments that's not always going to be easy to get, but in terms of interaction with a broader community, it is certainly an advantage. Looking at this, you can see that this is a fairly highly skilled kind of person that you're looking for. It's important if you're putting out job descriptions to understand the skills, and it's an area where we can often assist. If we'll reach out to the University of Oslo, we have assisted, for example, in assessing potential candidates looking through CVs and occasionally even taking part in interview panels and the like. So, yeah, the skills require system administration. As we've seen above, require really years of experience and work to develop. We do run a number of one-week DHS to Academy courses on system administration. Typically, we do around two a year, but a one-week Academy is really not sufficient to develop the significant skills to manage a DHS to implementation. So these courses are really meant to provide the specialized knowledge about DHS to system admin, to people who already possess some fundamental skills and knowledge in system administration in general. The reality, of course, is that often people with those required skill sets are not readily available and might be hard to recruit. In those sorts of cases, you really do need to get technical assistance from somewhere. One of the models that we've talked about is taking advantage of TA that may be on offer from regional or national HIST groups. And a recommended approach would be to enter into some sort of long-term mentoring plan where you have identified people who have some potential in this area and they might attend academies and they would work together with your technical assistance resource to gradually build up skill. What we see is where you have good candidates in fit this sort of role. Realistically, it might take about a year of mentorship before someone would be ready to fly solo, if you like. In this section, we'll talk about after looking at all the considerations above what is said about making a plan. Things that are often overlooked when making a plan. People often focus on the server. We get asked often, what size of server do I need? How much RAM? How many CPUs? How much disk? Unfortunately, it's sometimes very hard to say, first of all, what size server you need. And also often it's not exactly the right question. Because the server itself is only one part of the bigger picture. I mean, there are other aspects like where is the server going to be? And what are you doing about backups? And who do you have that's going to manage the server? These are questions that are often overlooked. So I'm going to just run through a few of them here. It's possible to roughly estimate what the server needs in terms of its physical resources based on similar projects that have been done in the past. But we need to plan to review these typically within the first one to three months to assess the actual resource needs. Because every application, every implementation is different. There will be different types of load. And the important thing to do is to learn and understand the load you're getting and how well your performance is doing. It's worthwhile to reach out to Oslo because we have some experience looking at a lot of different implementations and say, well, this is what we're trying to do. And typically what we would do is say, well, that's similar to what these guys have done or it's similar to this is what they are using. So what you're going to need is probably something similar to that. In general, because we know that whatever first guess you make on how much resource you're going to need is only a first guess, the needs will only make themselves really clear over a couple of months. It's helpful to over budget at the start so that you can adjust if need be. This sort of oscillations of adding more resources, taking resources away will taper off or become less over time as the system stabilizes. Certainly by the end of the first year, it should be clear what the requirements are. Though it's worth bearing in mind that this can also change over time and typically would change over time as additional programs are added into the system or, for example, individual data collections added to an aggregated system, or there are shock events that can happen like, for example, the mass immunization campaigns that we saw around COVID. If you're going to have to implement something suddenly around a large program like that, you have to bear in mind that the resources you might have already dedicated would not be sufficient or might not be sufficient. So unfortunately, some flexibility is required in general budget over rather than try to make a tight fit. We've seen that there are different modes of hosting DHS too. Depending on how you go about which mode that you go into, there's going to be different budget considerations to take into account. Very generally, we can say that if you're using infrastructure as a service, such as using Amazon Web Services or LinNode or what have you, commercial hosting providers, it's usually going to be the cheapest option, though there are skillset considerations, as we've seen above. If you're doing DHS to software as a service, there's a limited number of companies that offer that, that specialize in offering you DHS to pre-configured. Usually costs a little bit more to do that, but it's certainly the easiest way of getting started because you're paying for a whole lot of infrastructure management at the back end to be done by somebody else. Cloud hosting in national data centers, sometimes it's a legal requirement to do this. From a budgetary perspective, it tends to be more expensive simply because these companies or government departments are not offering service to tens of thousands of clients, so it tends to be a little bit more expensive. The quality can be a bit variable depending on the maturity of the data center compared to commercial providers. Physical hosting, which is something that is often brought up as the beginning of a DHS2 planning project was to buy servers and build data centers and what have you. It tends to be quite an expensive thing to do if you've got to build a proper data center conforming to minimum data center architectural standards. It does require quite a lot of specialist skills and specialist architectural considerations. Typically you're not going to build a data center for less than a million dollars, probably more in many cases. Sometimes it's investment that can be viewed as worthwhile, particularly where the data center in question is not just for DHS2 but can be used with different health programs, different softwares and different government agencies may not simply be for ministry of health. Because that's such an expensive proposition, what's often done is people will try to make use of existing data center or an existing server room, converted storage room within the building or what have you. In general, most examples of what we've seen of installations like that have been inadequate in terms of environment controls, the cooling systems and power supply reliability systems, water leakage, road and control and the like. It's a fairly complicated operation to try to retrofit buildings or rooms like that and we wouldn't normally advise implementations to go that route. What we have on this slide is exactly a real example of a country who were transitioning, in fact, from one cloud provider to another. They had problems with their existing cloud provider, which actually was becoming quite expensive, but also the service wasn't very good. So they moved to a new provider. This one I think I can tell you, it's actually Linnote. It's quite a large installation. It's a big country with they've been running DHS2 for the last 10 or 12 years now. There are probably seven or eight different DHS2 instances. There's a staging environment for those. It's a training environment. This is an example, I guess, of the virtual machines that were purchased to meet those requirements and the monthly costing of each. As you can see, it's a number of large 64 gig servers, some smaller ones. The billing is monthly, so the figures there, you're seeing a US dollar monthly fee. Importantly, the backup plan was considered from the outset here. The most host providers will give you the option of snapshots of the virtual machines and so they decided to pay for that. In addition, they've got longer term S3 object storage, which turns out to be quite cheap. This is for handing all of them backups and they would be paying about $200 a month for that. So that gives them a monthly infrastructure fee of just under $3,000. They plan that on the basis that that would probably be consistent over the next two years, growth following two years, it may require that some of those servers would get a bit bigger. But it gives you an example of what a monthly cost would be for a fairly large DHS2 implementation. Similar examples we've seen of people putting similar infrastructure together using a national data center have come up a good bit more expensive than that, perhaps $15,000 in the region of that. But those kind of arrangements will vary a lot from one country to another. With commercial providers, you can get quotes which would be much more expensive than this. You can also get quotes which are much cheaper. Generally speaking, you'd want to be sure that you're dealing with a provider that's got a good reputation and consistent performance. Some of the things that are overlooked when one of people are making a budget is that it's not just one server running the DHS2, there's backups that need to be done. The backups need to be stored somewhere and often pushed off-site archives somewhere else. This requires additional services which need to be taken into account in the budget. Systems for testing and staging and training are really important for a living implementation. Again, it's not just having a production server. If you're going to deploy a new version of DHS2, for example, it's really important that you test that first. So you need to have another server where you can test that on. You're going to need to often have training sessions where you're going to get users needing to access the system. You do not want to be using your production server for this testing, staging and training. You need to have some spare capacity to be able to spin up resources for that. You need to budget for a fairly major operating system upgrade, database version, things like that. That might typically be every two to three years. Countries will typically run each DHS2 system, an HMIS system or an HIV tracker or a TB tracker, COVID-19 surveillance system or COVID-19 vaccination system. They would typically be running on separate servers. Unusual to run everything together on one large server. It's much easier to manage if they're broken down, but each of them need to be provisioned and need to be managed and budgeted for taking into account the ballistic larger picture. It's worth bearing in mind, particularly regarding test systems and staging systems and training systems, that they don't need to be on all the time. So you don't need to budget to have them running all the time. They may only be used for perhaps one month in the year. And so you've got some flexibility there. If you've got your own physical hardware, then you need to have excess hardware in order to be able to spin these things up on demand. Couple of closing thoughts. Remember that decisions that you make on hosting, whether you're in the cloud or whether you're in a national data center or whatever it might be, are not necessarily final. Often there are migration plans. Country may begin with one hosting option with a longer term plan to transition to another. For example, we see cases of countries that have begun by a hosting locally and are planning to transition that into the cloud. And very often also we see the other way around. It's very quick and easy and relatively cheap to start up on the cloud. And often it makes the most sense to do. But you might still have a longer term plan to bring that back on shore into a national or a ministry data center. So yeah, it's not a permanent decision. It's quite possible to have a hosting plan with a migration possibility. Many countries have a longer term ambition of building national data centers and infrastructure. Often because it's not just around DHS, it's about developing IT infrastructure capacity in general in the country. And so it's natural that there is this ambition in most places to build the skills required and the infrastructure. And as I say, it's quite possible to have this ambition in the longer term while still avoiding risky hosting decisions in the short term. So as a very common model is to start off with cloud hosting and bring it on shore as the prerequisites get put into place. In the previous slides we've looked at different ways in which people can set up their DHS either using the cloud or locally or whatever it might be. An important point to make is that, and this is something that we've found in many cases, even though you might have a longer term aspiration for hosting your DHS to a national infrastructure. Because that's a complex undertaking it can take some time and investment to do. It's not a reason not to necessarily delay your DHS to implementation until you have everything ready. Often countries have found using the cloud as a very quick way of getting started and over a period of time it's possible to transition. And we've seen people go the other way as well where people have started off hosting in nationally and over time moved their infrastructure over to the cloud. Moving from the commercial cloud into a national infrastructure often requires there's a different skill set that's required. There are a whole lot of behind the scenes things around managing infrastructure which were taken care of before by your commercial provider which you need to make sure you have in your national environment before safely moving. The other thing to bear in mind DHS-2 systems as we have discussed a bit earlier usually it's much more complicated than just a single DHS-2 instance. Often there are multiple DHS-2 instances and often there are different security considerations around each. For example a very sensitive tracker server you might have stronger security considerations around then perhaps your aggregate system. And it's quite feasible to have some servers in one type of environment other servers in another type of environment. This process of moving systems from one place to another and we saw in the in the costing example of our country that moved from one cloud provided to another. It does involve some quite a bit of technical work related to configuration of the new environment, migration of the data and very careful planning around managing the downtime. When ideally you want to have a transition which is seamless from the user's perspective minimizing downtime and unless you make a really good plan you can have a situation where you may be faced with a week or two's downtime which often is not acceptable. If you plan it properly the whole process may be may take you between one to three weeks but it can be done in such a way that you have perhaps one or two hours downtime while the final database is moved from one environment to another. So moving moving data between systems it's an operation that needs to be planned carefully and budgeted for the technical work involved. One of the things system owners can be legitimately concerned about when they host their system on the cloud or with some commercial cloud provider is what happens if for some reason they get cut off from their system. This can happen through not paying the bill or some other way in which the relationship breaks down. So generally a good strategy for dealing with that kind of risk is to make sure that you've got access to the database outside of the cloud provider. This can be done by regularly bringing a copy of the database back to a server in country. If you want to do that and it's generally it's quite a good thing to do you need to bear in mind that there you do need to provision some kind of server to receive the data and this has to be an automated processor something that's on all the time and there are costs to bear in mind in terms of the data transfer transferring DHS to databases can be quite large. An alternative strategy to that again not keeping all your eggs in one basket is to use a different cloud provider. So even though you have your primary account with a cloud provider where you keep your backups and the like you can also arrange to keep copies of those backups somewhere else sometimes that can work out cheaper than bringing the backups on short. Impact on project budget this is really important for planners hosting a DHS2 system can involve adding budget lines which often we find are missing in a typical project plan. Cloud hosting including as we mentioned a sufficient margin to adjust the hosting plan based on the actual server needs after the project is deployed. It could include purchase of physical servers which importantly is not just a matter of buying servers it's also budgeting for the surrounding infrastructure and installation and very importantly budgeting for human resources. We know that in all of these models except for software as a service you need to have a system administration team which means typically at least two people with system administrator skills often these folks are not going to be minimum wage employees they can command fairly significant salaries. Some costs would be recurring which need to be planned for as well. Most obviously in a cloud environment is your cloud hosting bill and we've seen many implementations get themselves into trouble because the bill has become due and they've not been in a position to pay so really important that that's planned for. Physical servers you need to bear in mind that once you've bought your physical equipment it's not going to last forever typically things like hard drives you're going to look at replacing approximately every three years or so technology keeps improving they become faster they become more reliable particularly if you have your your servers you know not very well environmentally controlled situation whether whether the the heating might be a problem or the power supply may be going on and off you can expect the lifespan may become considerably less. As mentioned above the other important recurring cost bear in mind are the salaries and training and onboarding of new staff and the like. In summary DHS2 hosting considerations involve both the technology and the people who manage that technology. DHS2 needs to be hosted somewhere software software needs hardware to run there are pros and cons of of physical versus cloud hosting and they need to be reviewed critically in light of the specifics of each implementation what the context isn't the country what the plans are longer term most of the hosting challenges in fact turn out to be people challenges you need to budget for the necessary roles and ongoing training capacity building for these type of roles are not short term one week training courses they take quite a lot of time and mentorship you need to include flexibility in your plan for adjusting the provisioning up or down sometimes based on the actual needs and remember that your hosting plan can and more than likely will change over time.