 Good afternoon, everybody. My name is Bob Jelliff. I've got an hour's session here this afternoon to talk a little bit about installing DHS2 on service. So I've got quite a complicated presentation really because I'm going to jump between slides and video clips and typing at the terminal. So hopefully we'll manage OK. A lot to get through. I might have to skip a few bits, but let's see how we go. Right, a little bit of the overview of what we're going to go through this afternoon, a bit of history of the tool set around DHS2 on service. And then a description really of what we're calling the next generation of DHS tools, a bit of the architecture, what the main pieces are. Then I'll go through briefly the setup. I actually recorded the setup and then speeded it up so it's actually less painful to watch than doing it in real time. Then a little tour of the different containers involved, talk a little bit about the logging, the monitoring that's built in and then next steps that we plan to do regarding the tool set. So without further ado, let's, OK, yeah, acknowledgements. I thought I'd better put this up front in case we've missed it at the end. Many people have contributed to the code and ideas which have been incorporated into the tools that I'm going to show you today. I've mentioned a few names there, Stephen Ogavitsa, Stephen Alcaya, Tuzo, Morton, Barnabas, many others. Some of these tools, at least the early version, have been around for a number of years, quite a lot of people using it and all the users generally provide feedback which improved the overall quality for everybody else. A bit of history, now DHS2 as a web-based application, meaning something that's accessed over the open internet, was first implemented. I understand in Caroline, India around about 2007, started to become fairly widespread from 2010. Kenya, Ghana and Rwanda were very early examples of putting DHS2 on the web. And one of the things we very quickly discovered was that many of the people within ministries or within his groups who were responsible for setting up and running the servers were very inexperienced. Often it was their first time learning to do such a thing. And so the challenge was really how to take the best practices that we have written up in the documentation and how to incorporate that into something that was reasonably easy to deploy and probably more importantly easy to learn. With an emphasis on making it easy to do the right thing and making it hard to do the wrong thing. The DHS2 started off the first generation created around about 2014. There was some earlier version I guess but pretty much 2014 as a result of providing assistance to countries who were deploying DHS2. It's essentially a set of scripts, mostly bash scripts at the moment for creating DHS2 instances on Tomcat. It was inspired by, there's a package in Ubuntu that people may be familiar with Tomcat 7 user, Tomcat 8 user, Tomcat 9 user, etc. So it was kind of built on top of that but customised to make a few setup things easier to do. It formed a basis of an early series of server academies starting around about 2016. It's still quite widespread use but it's not really adequate I would say nowadays for managing the security and performance of more complicated systems. Particularly as more and more people now are using Tracker, they've got databases which has got personal identifying information on it. The security of servers have become much more important than arguably five years ago. Also the internet has become let's say a more hostile environment as the years have gone by. So over the past two years we've generated a next generation of these original tool sets with a lot of community supports. And that's what I'm going to describe today. What we call DHS tools, NG, next generation. I know Lars doesn't like that name. We can think of another one we will. But the primary differences with the new generation of DHS tools which is different to previous one. First of all we're using containers. So all of the different components, the proxies, the Tom Cap, the database, things like that, each one in their own separate container. As I've mentioned before we've got a much stronger emphasis on security than before. And I think each of those containers have been defined with reference to security benchmarks, particularly we looked at CIS. We're probably not, definitely we're not 100% compliant with the CIS benchmarks, but we're fairly close. The objective really, the main use of these tools is installing a virtual environment within a single host machine. So it's typically a cloud VPS. They've been tried on Linnodes, AWS. Another tool which I was introduced to by the Hisp Uganda people, SSD node, which is a very low cost VPS. Contabo is another one that's proven quite popular. That's easily and also quite frequently modified to support not just a single host, but often we have an environment where the proxy may be on a completely different machine. In the database, there are often reasons to put it on a separate machine. If you have different disk setups and things like that. So it's a frequently installed on physical infrastructure. There's quite a few of those around the world. It's quite easy to set up, as I hope you'll see in the next couple of minutes. But the fact that it's easy to set up is not really a substitute for strong experience and training. You can't really expect knowing nothing at all about Linux and system administration to start running a complicated DHS to environment, just because you've got a simple setup process. It's also I would say very suitable for probably running less than 20 DHS to instances. If you're a large hosting company like BAO systems or Hisp South Africa or something like that. If you're running hundreds of DHS to instances, you're going to develop your own customized tools for doing that. This is more geared towards mostly it's it's single country deployments. A country may have a half dozen instances. Sometimes their aggregates separate to their tracker and a training instance or a staging instance and that kind of thing. Normally not more than five or six different instances. Okay, so the architecture of the setup. I guess it's aimed at using either Ubuntu 20.04 on 18.04 host. We started with 16 moved to 18. And now I'm largely recommending people use Ubuntu 20.04. Mostly because in fact file system support on Ubuntu 20.04 support something called ZFS. We're not going to have really the opportunity today to talk too much about ZFS. But it's got quite a few cool functionalities relating to encryption and this compression and the like, which are useful in some circumstances. Even if you're on an 18.04 host, all of your actual pieces like your reverse proxy, your DHS2 instances, Postgres database all run inside their own containers. And those containers are based on Ubuntu 20.04. So even with an 18.04 host, you'd be running the same 20.04 containers. The containerization technology for want of a better word to use is something called LXD. It's been around for a long, long time. It's similar in some respects to Docker but also quite different as well with LXD and LXC containers as they call them, the containers themselves are more like fully fledged virtual machines. So each one is running like a complete Ubuntu operating system on them. But they all share the same common kernels so they start and stop very quickly. The packing density is very good in the sense that you can have many of them on the same host, etc. It's a requirement that you're running a host based firewall on the host. The UFW is what's built into Ubuntu so that's what we tend to use. So all connections in and out of the machine are passing through the firewall. All of these things like the Tomcat instances and Postgres instance and monitor are essentially inaccessible from the outside. All the connections we allow through the firewall are an SSH connection to the host machine and an HTTP and HTTPS connection which gets routed straight through to the reverse proxy. Okay, we can come back to that diagram later if people have questions on it. Okay, so I had a little demo planned. The first step is not really particular to these tools or any other tools. It's a small bit of guidance I suppose for people setting up a raw machine. It's about securing your SSH, enabling some simple firewall rules, etc. Then we're talking about installing the tools themselves which involve basically pulling a set of scripts off JITUP and then running an install script. And then we look at creating a DHS2 instance. As I said, I've prerecorded these so I could speed them up. The problem when speeding them up is then I get a voice like a chipmunk on them. So I've killed all the sounds, I'll just narrate over them so they're like silent movies at the moment. So first of all, setting up what you do with a host machine which is kind of newly born. So we call this a birthing process, a term that arose in February during the training of trainers academy. Let me just stop. So this is based on a linoad. So I've set up a simple linoad. In fact, it's going to be used for Oolove later for running his courses with students. So it's completely fresh out of the box at this point. And what I've run through here in about two minutes is just setting up SSH on it. That's moving on it. Okay, so initially there's no user account on it. So I've logged in as root. The first thing I'm going to do is you'll add a new user. In my case, it's Bob Jane. I added my user and then I would make myself a member of the sudo group. Exit, try it out. SSH in as Bob Jane. Check that sudo works. It does exit again. This time now I'll put my SSH key on there. So that next time I log in, I don't have to use a password. Right now we try again and this time I'm straight in without a password. Now, I've got myself a user. I don't need to log in as root again. I can do a few things to the SSH config. I usually just do three things. I changed the port. It's not strictly necessary. Change root log into no and make sure the password authentication is set to no. Restart the SSH service. Now I should see it's listing port 822 rather than port 22. So I need to enter like that using SSH with minus PA 22. The next thing I would do is start updating packages. Because it's a new machine, I'll do a dist upgrade. But that's all my packages updated. And the next thing I will do is set my firewall rule to allow port 822 in and then I'll enable the firewall. At that point, I can relax a little bit, go off and have a cup of tea. I've essentially secured the machine. I haven't installed anything yet, but we've set up the SSH and the firewall so that the machine is basically ready. Okay, so this next little two minute video, I'm going to show you the installation of the tools themselves. Now there's a lot of ways of doing it. And I guess what I'm doing here is the simplest way of doing it's not necessarily the best. I have a small script called LXD setup which you're going to see you're going to run here, which essentially will set up the LXD hypervisor and then install the basic containers on it. Normally if I'm working, particularly if you're working with physical hardware, I wouldn't automate the setting up of the LXD itself. I'd customize that a little bit depending on the disk type. But let's just watch simple setup. Right, first of all, I didn't do it earlier, so I should do it now. Let's set the time zone on it. This machine is going to be used by Ulavin Oslo, so we'll set the time zone to Europe Oslo. Then there I clone the GitHub repository with the tools. And the first thing I do, I'm going to copy the sample config file and enter a few details, just the top three lines is all we need to do. This config file is going to be used to generate what's required. Okay, so this is much speeded up. Usually this process would take about 10 minutes to run. I think in this little video it takes about two minutes. I've speeded it up four or five times. If you can follow what's flying past there, it's actually creating the three containers that were defined in that config file. Just about to make the Postgres container now. Because this is based on Ubuntu 2004, it'll use Postgres 12 by default. We've used Postgres 12 for some months now. It seems to work very well. Once they're all set up, I'll have an opportunity to show you around them a little bit. The last container to set up is the monitoring system. It's a very simple monitor. I know there's more beautiful ones around. Moonin is a very simple monitor, which gives a lot of very useful day-to-day information to system admins out of the box. Okay, so there's my three containers created. The last step I'm doing here, I'm just installing the SSL certificate onto the proxy. I used to have this as automatically part of the setup script, but people made so many mistakes with it that I thought it was better to do it manually. All right, so that's the setup. I'll put video online so people can have a look at it. It's also described in the documentation of the tools themselves. At this point of the setup, we actually have three containers created, a proxy container with its SSL certificate installed. A database, which hasn't been tuned yet, and the Moonin monitor. So how do we create a DHS2 instance on that? Again, a very speeded up video coming up. Creating an instance. Can I pause there? The first thing I did here, inside this setup directory, I should have done this as part of the previous installation step. There's a number of service scripts, scripts which are used to create new Tomcat instances, scripts for looking at the log files, looking at the database, etc. Which need to be installed. So I run this thing, sudo install scripts that actually copies scripts from this scripts directory and puts them into user local bin so that they're available in your path wherever you are. So currently, before we start creating an instance, we can see there are these three containers that we have here. Command, that's about to come up here. I'm creating a new instance. Create instance. And we give it a name. HMIS, for want of a better word. And often it goes about its business. It's going to do three things really. It's going to create a database on the Postgres container. And it's going to set up a Ubuntu container running Tomcat 9. And this process takes about five or six minutes to do, typically. So I've speeded it up here. So this should run through in less than two minutes, I think. Done. Okay, now you can see there's a fourth container, HMIS container. So we need to deploy a war file to it. So I'm going to put the latest 234 war file to it. There's the command DHS to deploy war. The reason, I'll just pause the video quickly. The reason for having a custom command for deploying the war file rather than just downloading it and putting it in the web apps directory. It's something to do with the way the Tomcat container has been set up. For security reasons. We don't allow war files to be automatically deployed. So that war file is actually unzipped into the web apps directory, which is owned by Roots. So there's a couple of steps involved with deploying a war. So it's very easy just to have a little script to do it. Again, to try and help people make it easy to do the right thing and harder to do the wrong thing. So that's what it's doing now. It's deploying the war file. I've got a little command there, DHS to log view, which will look at the log of the HMS container as it starts up. Get a few exceptions in here, which are related to the rolling file appender. I need to trace down why that happens, but in fact, there's no problem with it. So up it comes and we should be able to now go to our course.dhs2.org and go to our HMS and there it is up and running. Secured behind HTTPS and running 234. Yep, there we go. Okay, so that was a very rapid run through the install. Let's have a closer look now on what actually has been installed. One of the things that I like to do after one of the nice things about running working with containers is you can configure various resource limits side. What I typically would do, I've actually already done this. I'm not going to show you again is set a few resource limits on various containers in particular. The Tomcat container called HMIS, we make sure no stage can ever use more than 90% of the CPU. Sometimes DHS2 can go crazy and start burning up all the CPU. If that does happen, we want to make sure that the host doesn't become completely unresponsive. Same thing with the Postgres. If the Postgres gets extremely busy, we still want to be able to administer the machine. So we'd make sure at no stage you're going to use more than 90%. The other thing I often do is I'll set a memory limit on the Postgres container. We've only got a four gigabyte machine here, so there's not much to play with. But we want to make sure that because Postgres, if it sees all of the memory, if it can, it'll try to use all of the memory. And often that makes it difficult starting if your Postgres container is running already. It can be difficult to start your Tomcat containers because Postgres has already consumed all the memory. So we need to kind of calculate how much memory you think more or less the database needs. And it's a good idea to set a limit on that so that it can't exceed that. There's many, many other configuration options and things which can be limited. A little bit of link to some documentation there. Okay, so we have four containers currently. We do a little bit of a tour through them. As I said at the start, there's quite a lot of attention to the security on this. And one of the things that we've tried to do is make sure that each container is only minimally accessible. So for example, the Tomcat container should be accessible from the proxy because the proxy needs to forward request to it. Similarly, the Postgres container should be available to the various Tomcat containers. So those have been set up automatically, but let's go through them one at a time. So we start off, look at the Postgres container. As I've said, it should be based on Postgres 12 by default. The only thing I'm going to show you here in the interest of time is the configuration file because that's typically what people want to go to. So let's, I'm off the video now, I've gone live. There's our containers. I want to go, I can execute any command inside a container like this. So I'm going to go to the Postgres container and just execute command hash. That brings me in as root into the Postgres container. And the most typical thing people would want to do soon after installation would be to set some configuration items. Now, often people do this directly inside the Postgresql.conf file. I prefer to do it here as a separate configuration file for a number of reasons. First of all, it's easy to see the changes that you've made, but also you can keep different copies of configuration files. For example, you might configure it slightly differently if it was being used to restore a gap or something like that. So these are the most common options actually taken from our implementation guide. Maximum number of connections, people often ask about this. Well, in this case, this machine is going to be used as a training machine for students. By default, if we don't change anything else, Tomcat will try to make 80 connections as part of the connection pool, often a couple more for doing things. So 200 is too high a limit, 100 is much more reasonable. Now, there's not much tuning you can do when you only have two gigabytes of RAM. We definitely want to turn this down to maybe 512 kilobytes, about a quarter of what we have. Workmen, that should be fine. We won't have too many connections. Anyway, maintenance workmen is probably a bit high for the amount of memory that we have. I'm going to turn it down a little bit. And the effective cache size of this machine is not very big at all. Most it's going to be about one and a half gigabytes. These settings generally we can keep. I'll put this in by default. It'll log any query on the database that takes longer than five minutes. That's something that you can customize that if you're troubleshooting. Okay, so before we start in the database to get those connections, I should stop the Tomcat and do that now. I can just go and stop the HMIS container. Stop the Tomcat, restart my database container, and its new settings will be in effect. So let's go and have a look inside the Tomcat container. The Tomcat contains probably the most interesting one. We'll really have a lot of time today to go into a lot of detail. Let's just have a look at the main outlines of it. That's not running because I stopped it. Okay, let's start it. Let's say because these things are containers, they actually start very quickly. It's up already. So because we only have one Tomcat per container, we've based this around the Ubuntu system Tomcat installation, which has got a good thing to do because they've done quite a good job. That's most of the security around things like file permissions and the like. It's a bit of a pain to do all of that manually. So this is just the system Tomcat. So most of the configuration that you're going to get is from places like this, such as default Tomcat 9. That's where you set things like your Java apps and the like. Nothing much else set here at all, but that's where you would set it. The other thing, the place where you might look if you're configuring, if your server.xml file, so that's where that would be. There's a few things that have been done in here. You notice, for example, server port equals minus one. Some of you might know what that does. Basically removes the control port, which again, it just improves a little security thing. This is your server.xml. You can, given that this is going to be a small system, with a very tiny amount of RAM being used by students, I would turn down the maximum threads there from 100 to maybe 20 or 30. But I'll leave it for now. This is the famous Relax Query Chars, which some people might have come across. Otherwise, that's pretty much it in there. There's a few things in. This is interesting, I guess. Here, notice that we've got unpack wars equals false, and we've got auto deploy equals false. The reason for that is that if your web application gets hacked, and it is probably the most vulnerable part of the whole system, if your web application does get hacked, what you don't want is for that Tomcat user to be able to modify its own application. So the Tomcat user has been locked down as far as possible. It's able to run the application, but it's not able to do anything else. Okay, what else is interesting in here? The logging, I'm going to talk about logging later, so I won't bother with that. Yes, the DHS2 home directory. DHS2 home directory is in opt DHS2. That's where your dhs.comp file is, which hopefully has been set with the right permissions. Yep, it's actually owned by Root, but readable by the Tomcat user, which is about as good as we can make it. What I've done in there is, again, just giving you a bit of a template. I'll just take a quick look inside the file. A little bit of a, basically all of these options are essentially taken from the reference guide. So they're all in there, commented out. It's a bit of a mission to keep it up to date every time new versions come out. By default, it's just set up like this. It has its database set up connection string set. That's various things. I would, for example, turn this connection. This is an error in the documentation. The maximum size of the connection pool used to default to 40, I'm thinking now if you're forced to 80, I'm going to specifically set it to 40, et cetera. There's various things that one can set on here. The base URL session timeout generally is too long for most purposes, but I'm not going to tune it now. I'm just showing you where the bits are. Okay, so what else was there? I was going to show you the firewall rules on there. The firewall rules should all have been set up. So your Tomcat user, you can see, it can be accessed by the proxy, which is 192.16802. It can also be accessed from the host. That's not strictly necessary up there because sometimes I run commands on the host to get information from the Tomcat manager. This one here is on all the containers. 4949 is open to the monitor. Monitor uses 4949 to get its information. Okay, what else can we say? Where is my slide? The proxy. The proxy is in an interesting state of flux because as people who know me well will know I was very fond of Apache 2 as a proxy server. I'm still very fond of Apache 2, but I'm transitioning to engine X. I've explained that a little bit. I think in my last one or two slides why at the moment the tools are using Apache 2 by default. That'll be changed over the next week or two. The way that the proxy is configured, and this will be similar on engine X, is that we created a folder in here called upstream. These are the various snippets, if you like. So the HMIS snippet is the little location block, as they would call it. This is the bit where we tell the proxy that if somebody goes to HMIS we need to push them through to the Tomcat running on 192.68010. Similarly, the monitor has got its own little snippet so that if you go slash moon in, it'll get you to the monitor. Any other interesting thing about the proxy, which I'm going to talk a little bit later about these logs is I've configured a particular access log which has got two features to it. It's got, first of all, it's tab separated. We'll see in a minute why that's useful. And also we're recording the request processing time on it. But let's go back to the slides. Yeah, logging. We could have a whole presentation on logging. Unfortunately, we don't have time here. The Tomcat, because it's the system Tomcat, it's actually uses the system journal and syslog for all of its logging. And the journal log on Linux system is access using a program called journal CTL. It's actually a very powerful way of looking at your system logs. It's that useful that in fact I made a little wrapper for it. It's called DHS2 log view for looking at the logs. I'll show you that briefly now. It also logs to syslog. One of the advantages of logging to syslog is it's quite easy to configure centralized logging. DHS2 log view. If I go log view, I want to look at my instance. I can say just show me everything that's happened in the last 10 minutes. Because I remember log files can get huge and they'll show me the log file from 10 minutes ago. You can show me everything that's happened since 9 o'clock this morning until 9.05. There's a particular five minutes that I'm interested in. So I can pull that particular five minutes out of the log when I'm going to look at just what happened between 9 and 9.05, which probably was not very much. Yep, not very much happened at all. It's just sitting idle. These are scheduled jobs being kicked off. Using DHS2 log view is a very convenient command for looking at your log in all kinds of different ways. The proxy, I'd mentioned that I'd made some changes to the default access log to make it easier to perform for text processing and querying. I don't know if I'm really going to have time now. We are running a little bit to show you a few examples of why that's useful. I've got a file here of little orc one-liners which I used to process the log file, but let's just take you through them one at a time quickly. I think I can just run it from here. One-liners. Okay, so we have a file called perflog. This one is not from this machine. This one is from a production machine. Unfortunately, you'll probably see it in the log file. It's very big. It's only from today, but it's actually half a gigabyte's worth of log. It's compressed. So we're just going to z-cat it out first to show you what it looks like. That's the log format. As you can see, it's pretty much the same as the standard log format. I've separated everything, and I've also got a record of the request processing times here. So one of the things we can do because it's tab separated, using something like orc, orc is just convenient for doing really simple things. If you're going to get fancy, you can use perl or the scripting language of your choice. I think Ben is the scripting language of your choice, Guy. But I'm going to show you a little bit of orc. We know that field number six is the status field. So if I was only interested looking in my log file at the status field, I could run a simple orc program like that, and it's just going to extract the status field. Not really very useful. What's maybe more useful is to know what the requests were and what their status was. So this is just extracting from the log file now. You see, I'm getting all the requests and their status. There's a lot of requests in this file. It's like 1.6 million. So I'm probably just interested in the interesting ones, the ones which are not giving me 200s or 300s, but the 400s and 500s, I can make a little script like this, which says $5.05 and $6 as before, but this time only when $6 is greater than 304, and that's going to give me all the 404s and 500s and 401s and the like. This is often something useful. If you're looking at your system, try to find out things which are misbehaving. You see something here, which is very weird. You see this, get API script, whatever that is giving me a 500 quite often. That's obviously a problem. Something to be looked at on that machine. This particular machine spends most of its time, in fact, putting API events. So if you wanted to look at just everything related to putting API events, it's a little bit like grepping really. You could use a regular expression to only print the lines, which have matched the regular expression and just print $5 and $6 of it. And there we've got all our event puts, and you can see most of them are ending in 200s, about 302s. Not very interesting information the way it is, but what if we look for putting API events, which match an error code of 420, error code of 429. 429, if you look it up, is the HTTP error code where a server has essentially limited to the client. So they're saying that you've exceeded the rate limit or connection limit for this particular server. We can see there's quite a lot of cases where this server has actually returned a 429. How many cases? Well, we can count them. One way to count them, instead of printing the rows out, we just make a variable, increment it and then right at the end, print them. And we see that we get 28,000, which is quite a lot of them. There are 371,000 requests, which ended in 200. Sometimes it's useful to just know the big picture out of all of the requests on the server, just what the breakdown of status codes are. And again, you can do that with a simple one-liner or script. We just say take that $6 in an associative array, clan on the fly, increment it each time we see that status code. And then at the end, after processing all of the lines, we print out the array. And that'll show us all of the status codes that we got when putting API events. And you can see as before, the 371,000 success is 28,000. That's probably about 80% of all the requests were actually rate limited. And then a few other interesting ones that you're getting there. Because we've got the request processing time in here, we can also query on that. That happens to be field number 10. So I can look at all of the requests which took longer than 60 seconds. And there we can see them. There's actually not so many on here. I can see that getting track entity instances is quite a nasty one taking up to three minutes. Possibly needs an index on there. But in general, for 1.6 million requests, the number taking longer than 60 seconds is actually quite short. Okay. So that was a little diversion, I guess, around why it made sense to make a separate performance. Making a performance access log means that it's very quick and very flexible to be able to make interesting queries into your log file. I plan to put together a couple of orc one-liners to do this kind of thing, because it's actually very, very, it's probably the fastest way of processing a log file. Okay, monitoring. There are many ways to monitor a system. Many monitoring solutions out there. We've been using Moon in really for quite a long time. It's not a very beautiful thing. It looks very clunky really. But the main reason for using it is it's very, very easy to configure it out of the box. And it understands already with plugins, Tomcat, Postgres, Apache, Nginx, et cetera. That's very useful for the day-to-day monitoring by sysadmins to get an idea of the health of the system. For more detailed profiling of the JVM, we've found Glowroot is very easy to install and use. Thanks to Dan Cocos for that one, introducing me to Glowroot. We're actually installing it on most of our servers now. Obviously other monitoring solutions are possible because these things are just generic containers really. If you want to monitor them using Prometheus or Nagios or SolarWinds, you can do it. It depends on whether you have an existing monitoring solution in-house. We will need to set up a Prometheus monitoring container sometime soon. Now that we know we have specific Prometheus monitoring functionality in DHS to itself. I'll just show you a couple of pictures of the kind of graphs that are useful. Using Moon, this is showing you the number of threads in the state. Engine X request, engine X status. Postgres connections. This is an example of a very troubled server. All that yellow stuff are Postgres connections, which are ideal in transaction. That's a sign usually that the DHS2 application is run out of CPU. So it's open transactions, but it's not acting on them fast enough. Okay, I'll skip that. That's a little command that we have as part of the tools to look at your non-idle Postgres connections. As an example of the kind of thing you get with GlowRoot, I'll say GlowRoot is not installed by default, but it's quite easy to do, and I will make some documentation around that. Where are we going from here? Well, one thing I plan to do, we've talked about this for a while, just not have the time. Currently, all the installation scripts are rather hairy, nasty, almost bug-free bash scripts. It's going to be much more robust and much more easier to maintain them over time if you just convert them to using Ansible. Ansible plays very nicely with LXD. There's an Ansible LXD connector. So it's not a very big operation to redefine these machines using Ansible, and that's where we'll see it in the next edition, if you like, of these tools. I've not talked about alerting today. Typically, we use email for alerting, but we've had a lot of problems in countries with setting up their email MTAs, mail transport agents, getting banned for spamming and not having the ability to create the right DNS records, etc. So one of the solutions that we're looking at, I know it works, we've got a proof concept already, is using email for alerting, but then instead of actually sending the email, we route all the email to a telegram box so people can get their alert messages via telegram. We had a complete transition from Apache 2 to Engine X. The reason I said that there was a reason I wanted to do this is because there's two modules on Engine X, Limit Rate and Limit Con, which turn out to be really not just useful but actually essential when dealing with huge installations. We've had recent issues with the state of Uttar Pradesh in India. I don't know if anyone from Hisp India is on. But with 28,000 Android devices, now we know DHS2 can't handle 28,000 parallel connections, but we've no way of limiting it. So using Limit Con or Engine X, we can actually quite finely control the amount of parallel clients that are able to get in and thus not bring down the server. Currently, what you've seen defined are just the bare minimum number of containers. There are many other containers could be put together. Central logging container, I've made one of those before. It's quite easy to do, but I haven't automated it. Listening to the presentation yesterday on ICD-11, I can see some value in creating a little ICD-11 container, etc. Also, we plan to have a virtual academy before the end of the year when we get into a lot more detail. Alright, some additional readings. Those of you who want to look further into things, you really want to read the LXD documentation. We didn't get to talk about ZFS, but if you want to do things like encryption at rest, compressing your file system because your short of disk, etc., ZFS is really the way to go. If you like the idea of processing your log files using an ancient language from 1987, that's the classic book to read. That's it. I think all we can fit in in a short hour. Sorry if it's been very, very rapid, but I wanted to leave a little bit of time at the end to answer some questions. Thank you, Bob. We had a few questions coming through on the community of practice. Maybe if you'd like to address a few of those, that would be great. Okay. Oh, I don't have the link open to the community of practice. The link is in the chat. The link is in the chat? Yes. It's convenient. Let me get to the chat. I can share the screen for you. That's also okay. I lost the chat. I got the participants, but where's the chat? All right. Can you see my screen now? I just found the chat. Okay, some questions in the Q&A section. Okay, so starting with Carlos. Well, I did get a quick run through. Okay, there's about four questions. Let's take them one at a time then. So Carlos has asked, hi, thanks for your efforts and time. We're doing a server installation. We're checking since 232. Why do we need to do a post-cursor post year? Yeah, I don't know exactly what you're asking Carlos. It might be that what you're saying is that the documentation around post-cursor installation is not quite up to date and accurate. It might be the case. We'll go back and check that. But as far as I know, certainly if you use these tools, then the installation is up to date and should run 234. I've got a link on one of my slides there, where you get the updated repositories to install the tools. How do we tweak the Windows server? That, Walid, I'm afraid I can't answer. You're going to have to ask a Windows person. I don't think it's very, very common to be running DHS in Windows environments. I know a few people are, but I wouldn't be the one to ask, but I think that's the best way to get good performance out of it. I think one of the biggest problems on Windows probably is that post-cursor. Post-cursor is not designed for a Windows environment. And yeah, it's been successfully ported, but it's probably not going to get the kind of performance that you get on Linux. Morton, does install scripts copy the files to destination to the JIT repository? No, no, it actually copies the files, Morton. Is it recommended from time to time you JIT pool and rerun? Yes, that's exactly what I do often. Pull the updated repository and then rerun install scripts to get the latest set of scripts on there. Any other questions? Anything in the chat? Bob is an interesting guy, since Ben. That's right, Ben is an interesting guy. I'm just running through the chat here to see if anybody has asked me anything. Bob, there was a few more questions on the community of practice that you didn't see. Oh, okay, sorry. John Lewis, we use your server script on training server. Basically run locally without an internet. Yeah, John, this comes back to my very first point. A long time ago, people were very commonly running DHS2 without HTTPS at all. Often they would just run Tomcat on port 8080 and throw it on the internet. There's still a few installations I see do that today. Part of the reason that people were doing that is they didn't really know how to set up the proxy and they didn't know how to get the HTTPS set up correctly. So what I'd done, I guess, in fact, from the previous version of the tools even, was made sure that it was really very easy to set up HTTPS. But unfortunately, the downside of that is now you've got a bit of work to do and you don't want to run it without HTTPS. I can maybe document how you would best do that. The idea is to make that difficult for you. So I'm glad you're finding it difficult. Mamadou, hello, thanks for the presentation. How are backups work with this architecture? All right, that's a very good question. Backups are not configured automatically. The reason for that is because everybody's backup strategy is going to be slightly different. Basically, there is a command, I think it's called DHS2 backup, which will back up your databases. And what I would typically do is to install a cron job to keep a rolling backup, typically keeping seven days of backup and then six weeks of backup and then the first day of every month backup, etc. But where to put those backups is the thing and how to synchronize them with some off-site storage. That unfortunately is going to vary too much from one installation to the next. That's why it's not currently automated. We will provide a bit of documentation or guidance perhaps and what's a good way of putting together a backup strategy. Stelio, thanks for a great presentation. Can you please share it? Yes, I will do that. Why are we moving to engine X? With great sadness. I love Apache 2, particularly mod rewrite on Apache 2. I don't think there's anything as good as that on anything. But as I was explaining right at the end there of the presentation, there's two things on engine X around the rate limiting and the connection limiting, which is just a bit difficult to do on Apache. And they are so essential when you talk about very big installations that we just have to move. Carlos, all apologies for typo. My question is about database installation steps on the implemented documentation. All right, Carlos. I know the implemented documentation often has fallen into a bit of disrepair from time to time. We go back and check them. You're probably correct that there's some problem with the database installation steps. What I could suggest you do is have a look at the DHS tools installation scripts and you can see the steps that are done in there are basically the steps that you need. All right, any other questions? Scrolling, scrolling, scrolling. Ben is talking about me driving a motor home for Ireland to Norway in the winter. I'm hoping then to get it to Norway before it gets really, really icy. So November, Brexit permitting.