 Okay, so a little bit about these server setup tools. Some of you may have seen this slide before. It's a little bit of background that DHS2, the web-based version, running over the web, I think we're first implemented in Kerala in India around about 2007. It started to become quite widespread, running it over the internet from about 2010. Kenya, Ghana and Rwanda were very early examples of running DHS2 on the web. One of the things we discovered quite early on that many and probably most people who had the awesome responsibility for setting up and running the service were quite inexperienced at doing that. And so the challenge was to try to take the best practice that we had from our documentation and to incorporate that into something that was reasonably easy to deploy and to learn. And yeah, try to make it as easy as possible to do the right thing and as hard as possible to do the wrong thing. That's the reason why we set the SSL up as a kind of default. I created a set of scripts, which I can't remember when, it was around 2014. It was a long time ago. For just setting up basic DHS2 environment on a Linux server, we did a couple of early server academies based around those tools. I say still in why, I don't think it's any longer still in widespread use. I think there's still a few people still using the legacy tools, but hopefully not so many. One of the things we found as things have grown and the systems have become more complicated and particularly more and more people using Tracker, where there's more security concerns and also more performance concerns. It was quite hard to manage the way that it was being done at that time. So we're set about developing a kind of new generation of those tools. And a lot of people on this group have played a big part in some of those. Call it DHS2's NG, if you want to have a better name, should come up with a better name, I suppose. The fundamental difference, I guess, with the old tools is that we don't run everything on the same server in the way we used to. The old tools, we had a Linux server and you ran your reverse proxy and your Postgres database and your Tomcat instances and everything was running together in the same memory space in the same CPU slice. We needed to have a way of breaking those things apart a bit in order to be able to scale better and in order to be able to secure the system better. What we started doing a couple of years back, I think there were one that was probably the first example, predates these tools, really, is we put all those pieces into separate containers, right? So the reverse proxy should run in its own container, the Tomcats should run in their own container, and the Postgres should run in its own container. The container technology that we're using, something called LXD, I know most people who are familiar with containers probably come across Docker. Essentially, all of these things all use the same basic Linux kernel feature, which allows the partitioning of processes that they can run in some kind of isolated environment from one another. The Linux kernel feature people are interested in, something called C groups. So what we're looking for the next generation tools is a much stronger emphasis on security. That was largely because there were more and more people using Tracker, but also we had a couple of incidents of hacking because of poorly configured servers. So one of the advantages of putting each of these little components in containers is that we can also make them very simple and try to define them very tightly. So one of the things we've done is gone through CIS security benchmarks for different components and tried to go through the checklist to make sure that they are as close as possible to reasonably compliant with security best practices. The target, if you like, what is aimed at is installing a virtual environment within a single hosting machine. That's the simplest use case, and I'm going to talk a little bit about variations. But in most cases, that is what most people need. The heavy virtual server, they want to create a stable, secure DHS installation on it, running on a cloud VPS. Sometimes we make some variations to that architecture, and I've talked a little bit about that in a subsequent slide, as it is also frequently installed on physical infrastructure. It's quite easy to set it up as hopefully you'll see later this morning. And I've made quite a few fixes over the weekend to make it even easier than it was. But having an easy setup is not a substitute for having good experience and training. In fact, in some ways it's a bit scary. If it's very easy for someone to, with no experience at all, to take a blank server and install DHS2 on it, then it means that you're going to have a lot of inexperienced servers running. Server administrators running very critical systems. So yeah, please bear that in mind. The fact that it's easy to set up doesn't mean that that's a substitute for gaining experience and skills. It's probably not, it's not suitable for all environments. It's got a particular target in mind. I know like BAO systems, for example, Hisp, South Africa, people running hundreds of DHS2 instances. If you're running a big operation like that, then first of all, you don't need my tools to do it. You should have sufficient skills that you'll develop something much better than what I've done anyway. But for a large majority of use cases where you've got a relatively small number of instances, we found that it works reasonably well. Okay, so kind of hardware variations that we've seen. Commonly, they generally fall into one of the following three, I think, commercial cloud infrastructure, right, running on a lin node is very common, running on AWS, running on Contabo. The other thing that we commonly see is where it's running on virtual machines on some kind of national infrastructure, right? Often, government may have a national data center, Ministry of Health may have a national data center. And the DHS2 servers are installed in there. Almost always on VMware, it turns out. It's not my favorite environment. There's almost always performance problems we find. That's not something that's particular about VMware. It's usually the way that it's being set up. Sometimes, DHS2 is directly installed on physical hardware, right? There's quite a few places I can think of currently where physical hosts have been acquired and DHS2 is installed on those. So there's quite a lot of variety out there. I don't really put these in any particular order of preference. There's quite a lot of advantages and disadvantages to each of the above, right? And what you end up doing, it largely depends on what's convenient, what kind of skill set you have, and what constraints you might have in terms of policy, legal environment, et cetera. So for a lot of places, for example, it might be convenient to run a system on a lin node, but legislation might be that systems are run in country, so you have to look at different options. Important considerations to bear in mind, depending on what you're doing up, is, first of all, it takes a lot more management and technical skills to operate your own data center. So I know a lot of places are keen to operate their own data center. It's not a bad thing to do. I don't discourage it, but you do need to make sure that you've got the management skills and the technical skills to be able to do that. Putting up a server on a lin node is a much, much easier thing to do. And there's all kinds of things around the hardware infrastructure, the environment, the network, things like that that are taking care for you. Be very careful with VMware, over provisioned VMware, should I say. It's quite common that VMware environments tend to get over provisioned. So you get told you've got a server with 16 CPUs, but those are virtual CPUs you might not actually have the resources that you think you have. I have a typo there. The other thing we sometimes see is dodgy licenses, people running national data centers on VMware, which is not fully licensed. And then you come up and come with some limitations and difficulties around that. Probably the most important consideration is, regardless of what your environment is, how you do it, what is your backup plan? Where do you take your backups? Where are they stored off-site? And how do you go about making sure that that happens? Frequently, it's quite common for people are starting up DHS-2 project from the beginning and they go to the budget and say, well, we need a server. Just having a server or the server is not sufficient for running a DHS-2 environment. You also need to have a place where you're where you're putting your backups, which is not on the same server and preferably not even in the same data center. Either way, you need to have a backup plan. Your backup plan is going to be different depending on what kind of environment that you have. Okay, is that a basic architecture for those who don't know of a DHS-2 installation? You've got a couple of required bits. The main bit is your DHS-2 instance itself. This is why it's typically a Tomcat servlet engine, which is running the DHS war file. Depending on your setup, you may have more than one DHS-2 instance. We recommend that besides your production instance, you might have a training instance, you might have a staging instance, etc. You could have more than one DHS-2 instance running. You need to have a database in which to store your stuff. Nowadays, that database is always a PostgreSQL database. It's possible to have more than one database. You can have different instances store their data on different databases. You need to have some kind of reverse proxy. Either Apache 2 or Nginx, even HA proxy I think is another common one. The reverse proxy performs a number of important roles, probably the most important one being SSL termination, but also providing a routing to your different containers. You need to have some kind of monitoring system so that you can keep an eye on the health of your system. There's again a lot of choices for what you can use for monitoring. What I installed by default with little installation scripts that we're going to look at is something called Moonin. It's not the most beautiful monitoring tool in the world, but it gives you most of the basics that you need without much particular additional configuration. The way that we've set it up in the environment describes each of these things run within their own container. The host machine, I think of this big round box here with the host machine. The host machine should be very, very simple. There shouldn't be many services running on this machine at all. Typically, most important thing you want to have running on here is your host space firewall. I configure UFW default simple wrapper over Linux IP tables. Things I like about it, things I like less about it, but it works. And you need to have your LXD hypervisor running. This is the thing which basically allows you to run your different containers. Beyond that, you don't want to have too much running on the host at all. Keep the host as simple as possible because your host has got to be as secure as you can make it. Because if your host gets compromised, then essentially each and any of your containers can be compromised. The advantage of running all of these services in separate containers is we can limit the damage. If any one of your particular containers get hacked, you can take some precautions to make sure that those compromised containers can't affect others. Okay, I've got on this slide, you can run this on Ubuntu 2004 or 1804 host. Currently, my latest set of scripts that are available on GitHub are primarily tested on Ubuntu 2004. So if you're starting out fresh, start out with 2004 LTS. They can be made to work on 1804 as well, but I've not tested recently. And so there may be a few small teething problems. In theory, they should be able to run on Debian as well. But again, I've not had the opportunity to test it on that. So my suggested environment is to use Ubuntu 2004. Either way, when working with Ubuntu, particularly with a production system, you should work with what's called a long-term service edition or LTS edition. They're typically the ones with the 04 at the end of them. So 1804, 2004. The next one will be 2204. They get released every two years. And the advantage of using an LTS edition is that you'll continue to receive updates for, I think, up to six years. Whereas if you go for one of the latest releases in between those LTSs, then you'll find that your system needs to be changed, reinstalled, or upgraded much more frequently. Okay, so that's the basic layout of the various containers that we see. I just realized that while I'm in full screen mode, I can't see the time. We are already running out of time. I'm going to overrun a little bit and take a little bit of the next session. So we'll stop at, let's say, in the next five minutes and pinch a little bit of the next session to finish off. Okay, a couple of variations. This might be the last slide. I'll fit it in five minutes. A couple of variations that we see in the above pattern. Very, very common. People want to use an engine X proxy instead of the patchy two. This is a discussion that's been going on for some time now. I still need to make a strong configuration using engine X. I've done it many times manually, but I just don't have an automated script for it yet. I'll try and do that, in fact, over the next couple of days when I get a chance, because people ask for it a lot. I prefer a patchy two still, but there are some advantages of using engine X as well, and people increasingly are more familiar with it. The other variation you often see is the proxy server already exists somewhere in the environment, and so instead of configuring your proxy there inside your LXD host, often its proxy is somewhere else applied in the environment. The other thing commonly see is database provisioned on a different machine. This is often done for performance reasons, adds a little bit of complexity when you do that, particularly if you're running in a cloud environment, if you want to run the database on a different VM, because all the traffic between your Tomcat and your database is going to run unencrypted through the cloud provider's network, so ideally you need to make SSL connections from the Tomcat to the database. It's not too difficult to do, it just adds an extra little bit of complexity. The other thing that I'd like to see more often, but it's not very common at all, particularly people running tracker databases and particular people running databases which carry quite a lot of PII, as they call it, patient identifiable data, personal identifiable data. Good practice is to not only have encryption in transit, like what you get from using SSH and SSL, but you also want to have encryption at rest. What that means is that when the system is powered down, everything on the disk is encrypted. It's something that we should do more of. It's not very difficult to do, and we might get an opportunity over the coming weeks to run through the process of doing it. The risk is, of course, if people don't know exactly what they're doing and they lose the keys to the encrypted volume, then they can lose access to all their data. So be careful with encryption. The other common thing that we see, it's not so common, but it's getting there, often you have more than one of these environments. Rwanda, I think of in particular, I think Amzai is probably on this call, maybe Andrew. There they started off and they heard an environment pretty much like what I described in the previous slide. This run four, five, six Tomcat containers on it and realized that they were running out of resources. So they made a second one and now they have a third one. The other thing that is a minor variation to what we do today anyway, that's not so minor, a fairly major variation, is that instead of using the default file system that's provided, is you set up an LVM or a ZFS file system, which provides some additional, many additional advantages. In fact, particularly the ZFS, but you need to have a pretty good understanding of how ZFS works. Feel comfortable with your file system before really attempting to install and use it in a production setting. It works very, very well. It's particularly good for running databases with encrypted volumes, as I was pointed above, because the latest version of ZFS includes a native encryption on it, so it's very easy to create, to set up a volume for running ZFS. The other thing that we often see increasingly is that besides that list of containers that I had in my diagram, there are many other types of containers that people want to have or need to have in their environment. I've got a slide here which talks about some of the common ones that we find, but I think because of all of our orientation stuff, we kind of run over our session. So Martin, I don't know, can we pause here? Recording us on it again. Okay, so we mentioned earlier that the installation scripts by default only creates the minimum number of containers that you really need to be able to run DHS-2. That's the reverse proxy, the database, simple monitor, and your DHS-2 application instances. But there are quite a number of additional types of containers that you might want to create, and these are typical ones that I found myself creating over the last couple of years, which are not part of the default installation. Some of them are now becoming so standard, in the sense that we should find a way for people to be able to easily script. Email gateway is very common. People want to be able to send alerts. Somehow email is one of the best ways of alerting. It's quite easy to also link an email gateway to something like Telegram or something like that. SMS gateway, which we set up quite recently in Rwanda, so that all your instances can make use of a single SMS gateway for sending and receiving SMSs. I've got some nice examples of setting up small containers just for running ad hoc pods and ends of services. Something else which is increasingly becoming important is the ability to run Docker containers alongside. One of the best ways of doing that is creating a separate container in your architecture, which is simply for running Dockers. Again, you can have more than one of these. We may get a chance later in the month to talk about some of those other container types. It's very useful to know how to make containers yourself, and then you can put together environments which are more suited to exactly what your requirements are. There in mind, I think as I mentioned before, the fact that you've got a simple setup script doesn't mean that you don't need to understand as much as possible about the way your environment is working, particularly if you want to create new containers. There's a little bit of tour of what's actually on those containers, but I've got to keep this brief. We'll do it in more detail just before lunch anyway. The Postgres container by default now is running Postgres 13. It had been running Postgres 12 up to now, but there's all kinds of big performance gains to be had by running Postgres 13 as a database, so now we've made it to the default. All the configuration for Postgres, you'll find in that file. Usually after installing it, again, we'll see it a little bit later, you want to go in there and tweak a few of those parameters to tune up your Postgres server to suit your environment. Tomcat is running Tomcat 9. It's running under system D. It's quite well secured, I think, and that adds a little bit of complexity to some things, but again, we'll see it a little bit later. The home directory for your DHS-2 home is opt DHS-2. That's the default on DHS-2. It's actually a little documented feature. If you don't specify the DHS-2 home directory, then by default, DHS-2 is going to look inside opt DHS-2 to look for your DHS.conf and things like that. The proxy server, for each of your upstream locations, they're actually created as little snippets inside that folder. If you were using nginx, then we'd create that similarly as etc. nginx upstream. Again, we look at what those snippets look like. There's a ufw file, which is running on each of those containers, basically to restrict access to the minimum for each of them. For example, your Tomcat container, you need to be access it from the proxy, but they don't need to be able to access one another. Each of them need to access the database. Whatever the minimum requirements are, what containers need to interact. We've set firewall rules on each of them, so that is taken care of for you largely. You can set resource limits on your containers. Again, we're going to look at this shortly. One of the advantages of putting them into containers in the first place is we can make sure that we can restrict the amount of RAM each of them can use. We can restrict the amount of CPU that each of them can use and the like. I'll show you some examples of that in a minute. Logging. Tomcat uses the SystemD journal for logging. The best way of looking at those logs, in fact, is using journal control rather than rather than browsing through Catalina.out. Again, I'll get the opportunity to show you how that works a little bit later on. I've configured a little bit of customization on the proxy so that we get some performance logging on it, which doesn't come by default. Monitoring, as I said, we use a system called MoonIn is on there by default. We can change that, and one of the things that we want to do is to also create an automated setup for things like Prometheus and Grafana. MoonIn is not very beautiful, but it is easily configurable out of the box, and it gives you some very useful day-to-day monitoring of things which are important to you. I'll show you some pictures shortly. Increasingly, we've found the need for profiling. If your application is particularly if it's struggling for one reason or another, performance-wise or something else, GlowRoot has been a tool which over the last year we've found incredibly useful to put on to your containers. I'm going to provide some assistance to make that much easier to configure. As I said, this various other monitoring solutions could be used. MoonIn looks like this. You see these graphs look very 1980s, but you can see your basic statistics of what your Tomcat is doing shows you the amount of threads and how busy they are. If those busy threads start to start to reach a peak, then you know that you've got an issue somewhere. MoonIn will also monitor your engine X or your Apache, give you some idea what your daily load is, what your weekly load is, what your monthly load is. These are some very useful graphs around what's happening with your Postgres. This is an example of a SICK system, because you can see all that yellow stuff are connections with just sitting idle in transaction. If you see a system like that, it's an example of where in fact the Tomcat is overloaded for some reason, and it's not able to open a lot of transactions, but it's not actually servicing them. This particular graph from MoonIn is very useful for diagnosing basic database issues that you might be having. I leave that, but now it's a boring slide. Oroot, as I said, is something that increasingly we're finding very, very useful. It's quite easy to install on your server. It gives you a really quite insight into what's happening with particular web transactions, what's happening in your JVM, what sort of slow traces. This is just a screenshot, because it's not very interesting looking at a demo server, because there's no traffic on it. I might request some folk later on whether they'll allow us to have a look at Glowroot on one of the live servers, but I don't want to do that without getting permission first. Where we plan to go from here, and this is, some of these plans have been sitting for a while. One is the installation scripts themselves. Currently, there are rather horrible mess of bash scripts. They pretty much work, but they're becoming harder and harder to maintain. They're a bit ugly. We want to change this to using Ansible. One of the things that you might find is that LXD actually is very Ansible friendly. People are familiar with Ansible. They don't know what I mean. You can make connections to your LXD containers natively from Ansible, so you can define them all using Ansible Playbooks. I'd like to try to change all of my current bash setup scripts to do it like that. I've been told for a long time about getting email to Telegram on alerting. The transition from Apache 2 to Engine X, for me, it's not really a transition. I still like using Apache 2, but I know a lot of people want to use Engine X. Again, that's something I want to get actually working probably before the end of this workshop, so that if people want to use Engine X, they can. We've talked already a bit about additional containers and the documentation. Documentation currently is a little bit sparse. It's an ongoing project to improve that to make it a little bit better. That ends this section, really. I've got a little bit of background reading here, which I advise people to take a look at. The installation tools themselves, which we will look at this morning out there. This is the primary documentation site for LXD, that's for Glowroot. We didn't talk much about ZFS. Advise you to take a look at that. Orc. I don't worry too much about Orc for the moment. Okay, so that's pretty much my very, very brief overview of the architecture from a high-level view. As I said, if you have particular questions or issues you want to raise, please do so in the Slack channel. I want to move over quickly to doing a little bit of a hands-on demo of LXD.