 Thank you. So, I guess we can start now. We have numbers. Yes. So, welcome everyone to this weekly call server implementation meetup. In this meeting we do discuss issues of administration. We started last year, the last month, December, and October, November, December. But then we proceeded to break, two weeks breaks before the year ended for the Christmas holidays. Last time we had this call, Bob was interviewing our security, Michael, and today's call is kind of going to extend on the same. We want to talk about ways that we can improve our server security. At least on a fresh install we make sure that at least a few security, basic security controls appeared too. So, I prepared a few slides that we're going to go through while I'll be explaining those security checks. So, let me share my screen. So, yes. So, when we talk about server security, there are security models that we do follow. And one of the models is a model where we want to make sure that whoever accesses the server is authenticated and they are who they say they are. We also limit what they can access. And by that I mean, even though you really are the person that you're claiming to be and you have access to the server, it's not that you cannot, you can access everything on that server, but only you are allowed to access and when by that we follow the principle of least privilege. By that I mean you, you are given access to components that you really need access to and then you are denied everything else. And there's also an aspect of accounting. That means everything that you do is accounted it's written somewhere and in case of an incident that can be tracked easily. It also can be can be implemented in multi layer. That is, you want to look down your server, not only on one layer, but on multiple layers that is from the host level to the application level. And in our case, our standard deployment of DHS to instance is done on on on LXC containers, and with that it adds an additional layer of security. And even within those LXD containers, we do run a filtering network filtering. It's a firewall, which blocks things that we don't want to access that particular container. And on even application level, we don't just we still do limit what can be accessed. Our deployments mostly is automated. We are using Ansible Scripts. One of them and and and and by scripts, which automates the whole process of deploying the applications and with automated deployment, these security checks security controls are just implemented on flag. You don't have to go back later and and or you don't have to do that manually. If you adopt deploying your app or the application DHS to application using automated automated way then these checks are just fixed on on fly. So we have components that we want to to secure. And first of all, as much as you deploy your application on a given host, there are components that are related to the application. In our case, we have proxy Tomcat server database and and finally backup well backup is not one of the components but it's a security feature that we want to in every installation to implement. However, things that we do have installed during the automated installation proxy Tomcat server and the database. So we want to break down our security discussions into these details. And the first one on our list is that is the is the host OS. So mostly our deployment happens on the on the Ubuntu server and all deployments can be running old LTS releases. But then LTS releases are just supported five years, like for instance Ubuntu 18.04 LTS release, its support is, its support is expiring this year. So that means if you are running your DHS to instance on 18.04, you do have to upgrade to at least the next LTS which is 2004 and it is supported until 2025. So on the host level, we at least have a firewall running and it is uncomplicated firewall just to demonstrate that I can I can look into one of the servers and show you. And one of the servers that I do have an instance running is this one it's running on some cloud environment. So it has a firewall running and I can demonstrate that by showing you have W status. So it has a firewall running. And as you have seen, the status shows that the firewall is active and only one port is allowed or is just open from the from the internet and the port is a 22 that port is specifically for SSH traffic. Anything else is really is really locked. If you try to access any other port from the internet, apart from port 22, then you will not be able to access it access it. Of course, the server is listening to other ports. It's not only SSH listening right now. We can check with SS command. We have other services running even SSH is listening on port 22, but that is not open on the on the firewall. That means even if you try accessing your, your secure shell on port 22, you will not be allowed in because that is locked on the on the firewall level. So, even though SSH is is the only way it's only get away that we can access this album. It needs also to be to be secured. Password authentication, key based authentication, and we do advise that we adopt key based authentication and we disable password authentication. So that is the case in our installation. And you can enforce that by editing it see SSHD configuration file and turning off password authentication and enabling only key based authentication. There are other security practices that you can also adopt like you leveraging on failed to burn to which really checks or monitors your SSH attempt logins and it blocks access. It's going past a certain number of times. And with SSH also, you want to use a different port, not necessarily an added security practice, but it in a way because both on the internet and even the attackers do try to access your server on on the default port, which is both 22. So you want to limit the kind of looks or those access attempts and use a different port so that even though they try beating your server on port 22, it will not be listening on that port and it will be on a different port. You want your server to be patched timely, you could enable automated updates and upgrades. And you, as I mentioned before, you should make sure that your server is at least running supported LTS release. For instance, right now, 16.04 is not on that list. And the last on that list is 18.04 LTS, which is going to its support is going to expire very soon. So on a fresh install of your server, we recommend it to be minimal. However, sometimes you might have an OS that has packages that you don't you don't really use. So we recommend also that you do ensure that you have packages that you use and and those that you don't use to delete them from the system. This is going to help you. At least at least packages that you have. It's just one of the command that you can use to list the packages that you have. And here you could potentially note those packages that you are not actively using, and you can delete them with sudo apt removed command. Next, on our list is the is the Tomcat Tomcat is the is this web application is it's a it's a web app. It's a web server that's used to run Java based applications and our DHS to is is Java based and it's run on top of Tomcat. So on a fresh install of Tomcat, it has extraneous extra files that you don't need like root. You want to delete that on your web apps directory and ensure that you have files that you really, you really require. It also, it also comes with the default install is not appearing to the best configuration configuration permissions. For instance, I'll file. Let me just demonstrate on the login. Let's see. This is, of course, the application is running with Tomcat user. So he wants to make sure that Catalina home or Catalina base is is owned by the admin user. And number two, that is is is also within Tomcat group, it's owned by them. The group is Tomcat, and that's read write permissions are restricted only to root user and Tomcat group and others are not given permissions to read write or execute to that to that directory. Let me find Alexi EXERC, HMIS. Yes, you can just take this. This is one of them. It's Catalina home for this, at least this container. It's owned by Tomcat. The fact that the directory is owned by root user and group is Tomcat. And on top of that, only root user is given access, read write execute access to that directory and group Tomcat is given read execute permissions. So others are not given are given zero permissions on that on that directory. So we also make sure that our Tomcat instance is is is running behind proxy. And we can do use, we do use either Apache tool or IngenX. And we are basing basing our guides on the on the CIS benchmark security checklist, which is a hard one here, which is something like this, that it gives you the checklist which you have to do when you have your Tomcat installed. This is just a small list. It goes to 10. Something. If you can see the next part of this CIS benchmark, it goes up to 10. And these are the checks that you need to follow after you have installed your DHS to instance. And these all checks are automated, of course, with the DHS to installation script, that after the installation is finished, you're going to have your Catalina home adhering to the permissions as stipulated on these CIS checkmarks. So, even, even disabling the shutdown port. Of course, our apps are running on LXC containers. All our applications are running on LXC containers. And on top of that, they are running container based firewall. Let me just show you that. So within the container itself, we just limit access to them to the application and we are limiting access from specific IP address. As you can see here, 17219.2.2 is just an address for the proxy. So we want our Tomcat instance to be accessible only from the proxy. And if the IP address is different from this, then it cannot hit port 8080. So port 4000 is for muning and it's only accessible again from the proxy. And then 4949 is the muning node and it's only accessible from the muning server. So, as much as we have post based firewall, we also filter traffic on the container running that particular instance. Same applies to the Postgres container. We want to limit access to that Postgres container to be only accessible from instances. For instance here, this is the Postgres container that plans database and you can check its firewall status. It's running and it's only allowing access from one. These top two are the instances and those are actually really Tomcat instances. They're allowed to access the database. And partly is them, of course, the port 4949, which is opening its access from the muning server. So, yeah. Do you want us to ask questions or we'll make comments? Yeah, you can. Just on the Tomcat setup. The biggest, one of the biggest problems we see is when people download the binary Tomcat, often they might do this because they want to run a particular version and then they just get the tarji zip file and then they untar it. And that's usually where you find all the permissions and everything is wrong. Generally speaking, the Ubuntu and Debian packaging of Tomcat is actually quite good. So usually if you just deploy using apt-get install Tomcat 9, most of those controls are actually already implemented the CIS controls, which is good. Recently, they started disabling the shutdown port that used to be the case up until version 9. The only thing I think that we do with the tools on Tomcat's container is changing the ownership of the web apps directory. Because that's one thing that is left owned by Tomcat with the apt install. So that's something a lot of these, I think the thing to bear in mind is that a lot of work is done in that Ubuntu package. And if you're not going to use the package, but are going to unpack the Tomcat yourself, then you really have quite a lot to do in terms of verifying all the permissions and stuff on different directories. Exactly. Then that is when you need to at least appear to that checklist and make sure that those controls are met. And we don't implement all of the controls either. And I think this is maybe something Michael is also going to be working on. We're kind of cherry pick, I guess, what we figure are the most important ones. One in particular that I'm thinking of is the, what do they call it, running the security sandbox thing in Java. Yeah. We currently not running, I think it's probably down near the bottom. We don't, we're not using client cert authentication, for example, because we're not, we're not running HTTPS on the Tomcat. And we also, we're not running it under the security manager would be quite nice to run it under the security manager, but there's quite a lot of things that are potentially broken when you do that. Anyway, that didn't mean to interrupt. I was just adding a few thoughts to watch it. That's welcome. On the firewall stuff. And you showed the one of the things that's important to bear in mind because you've got quite strict access to that Tomcat container is if you're going to set up something we typically going to see this if you've got an integration layer or something. If you've got some API based, you know, maybe it's Apache camel base root that's doing integration with your Tomcat from a different IP. You do need to remember that you're going to have to create a separate firewall rule for that is by default, you will be able to access it. Yeah. So, yes, the firewall rules running on the on the on the DHS or other instance container looks like these ones. And these are very strict. And as Bob mentioned, if you, he wants to open access from some different integration content, then you have to add an extra rule there. Okay. So next is the is the is the database post classical database. Yeah, so, as I mentioned, there is firewall of course running within them, the database container. So we also with with the configuration files, edit them with Ansible edit them PG HP configuration and ensure that access is only allowed from a given IP address. In our case, it's, it's instances running DHS to application. So, as much as we are blocking on the on the firewall level, we do really, again, restrict on the Postgres SQL application level. And that matches not only the IP address but which database do you want to access and with which user. So it's restricting not only with the IP but with other details. The URL is edited during that deployment and it's on HTC Postgres. The last line so during the automated deployment. These two lines are added depending on the number of instances that you want to to install. So those two line restricts. The way your database is going to be accessed, it only allows access from, from hard-coded IP addresses. And number two is to which database and with which user. Yeah, so even if you have a user added in your database, and you want to access it from network and its entry is not here, then you will not be able to access with that user. So, yeah. That explains the three components that I have within the database that firewall is enforced and PGA configurations have had coded entries that allows access to different IP address with some given user and and its password, of course, is from the application side. And next is the proxy. We do not access our applications directly. That is Tomcat app directly from the internet. You can go through the proxy, reverse proxy. And in our case we are using Apache to and and and engine X for that matter. And with proxy you could implement other security controls. One is you could limit access to your, your services, say, with geographical location. If you want to say you don't want if you want to block access from certain region, then you can enforce that on the engine next level or even Apache to level proxy levels for that matter. You could also, if there is a known vulnerability, and its patch is under development, then you could also potentially block access to certain endpoints which is vulnerable until its patch is, is, is available. It's on, on those proxies that we implement strict transport layer security, TLS and SSL. And with proxy, you could also potentially monitor the access logs and and and and error logs. So it adds, it gives you an extra layer of security and and it abstracts your, your applications from the external access, you know, because users and clients to interact with the proxy and the proxy do proxy for your request to the backend application. So most of the, our cases we have more than one running instances of DHS to you could have production instance, which is used for production purposes and another test instance. So if it was not for for the proxy, then how else would we proxy somehow route those requests to different applications. So you have one DNS sub domain and he wants to match the request address and proxy for to the two different backend applications. So it's also from the proxy that we do routing to the apps that we have running, and it can be more than one. So, yeah. Next is backups. We want to plan, you know, backing up or planning for backup is also a security that we want to ensure that we plan even before the installation, because without backup and you have an incident and it's a problem so just doing or planning for backup, planning for backup is one thing but then you need to also test your backup and ensure that in case of or other your recovery works because it would not make sense if you have a backup that is not working. So you want to test your backups and ensure that they are working and ensure that you have your backups pushed somewhere else. Well, you could use a server running somewhere offside and you do your local backups and push to that side. And of course you can automate that so that it runs. It's a stipulated time times following a policy that you you do configure and you could also push it to cloud. Well, the cloud providers made the cloud providers to provide S3 bucket storage so you can push to that can procure that service and push your backup to to that endpoint. So backup is an interesting topic and we're going to have a future conversation around backups and even use cases, how do you guys implement backups in your environment. So yes, that is all I have today in this quick presentation. Any questions? I got a quick comment again, if you don't mind. If you go back to your host and the UFW status, the exit from that one, you are using, you are using allow to port 822 slash TCP. We're good but it can be a little bit tighter still you mentioned a bit about fail to ban, you know fail to ban will will protect against the repeated attempts to brute force on the SSH port. But in even with UFW instead of using allow if you use limit. So your firewall rule instead of instead of saying allow does limit instead, then it will limit. I think six attempts on the SSH port within 30 seconds. If you do more than six attempts, then it will temporarily ban you. So it's kind of got a, it's got to behave a little bit similar to. Yeah, yeah, like that. That's actually a little bit better than just doing UFW just doing allow. Yeah, exactly gives you some of the benefits of fail to ban without having to install fail to ban. Small point. Thank you. So yeah, so this this this checks that I've just talked about most of them are automated with them. If you use our tools, which is the HS to unstable tools. So you don't have to to kind of take the checks one by one when you do your installation. However, if you you're planning to do manual setup where you do set the post cares. Proxy and and and even your instance separately, then you need to at least have a checklist which you follow to make sure that your deployment at least meets these basic security. checks. And, and, however, if if you want to just go away with that and and and be able to kind of have a secure system on a fly, then you have to, if you use the tools, then it's going to fix that for you. So any other comments from people I see they still got a few people in there we've been paying attention throughout. Matthew, Mohammed, Moses, Naseef, any thoughts? How does this is anything there which is new which you're not doing in your own installations? Are you using the DHS tools? None from me. Alice? Alice? Are you trying to say something? I can't hear you. No. Tito, have you had any thoughts about? I know you need to chat with Michael probably a little bit about this, but you know the CIS checklist you refer to. There's many, many, many of them. And what we do with DHS too is we kind of pick the core set of ones that applies that we should at some point write down a checklist rather than just have the tools which are implementing all the controls. Well, we should write down the checklist of all of the controls that should be applied to the Tomcat and the proxy and also the database. I'll have a discussion with Michael and I already have a list that I do use when developing the DHS tools, ensuring that I meet most of these security checklists. But I'll have a discussion and come with Michael and come up with a list that at least we will be following at least for the installation. However, this list has a lot of checks which we don't use all of them. So we need to go through this list again and decide there could be some of them that we're not doing them, we should be doing them. Part of it is, so it's a matter of, I did this with the proxy and it took me almost a week. There's like 170 pages of controls on the Apache proxy to actually go through page by page control by control and deciding which one to implement and which one's not. So I've done for them for the Tomcat, but then not not all the controls are implemented. And I'm planning to also check one for Postgres and one for for Indian X proxy. And at least those that that will not break anything are at least configured. Yeah, I think we need to get this onto a spreadsheet somehow, which is a bit awkward because it comes in a PDF, but where for each control, we can say it's implemented or it's not implemented. And if it's not implemented, why it's not implemented. Okay. Because it's not a requirement to implement everything, but we, if you don't implement something and it's got to have a particular reason for it. Where, for example, application specific logging. Now that applies if you've got maybe different war files running on the same Tomcat, right, so you're running a couple of different applications, then each different application would have its own logging. And in this case, because we really only have one DHS to running on one Tomcat, we don't necessarily have to implement that control. So we've got a justification for it. Similarly, with 6.1, we don't have client cert authentication, and we don't have 6.2, because the SSL termination is done by the proxy. So not all the checks here really, as you mentioned, we can implement now because already some. When we don't implement it, we need to have a justification for why not. So that might be useful spreadsheet to make. Otherwise though, thanks Tito. That's all for me. That was useful. So I'm planning to go one by one, and I'll prepare the spreadsheet for all the components that we have. One of them is of course Apache Tomcat, and I'll check one for Postgres and one for the proxy. Then I'll have at least check all those, the security checks that at least are recommended by CIS. Then I'll have a meeting with Michael and discuss extensively, which ones are we going to implement and which one are we are not going to implement. And at least if we are not give reasons. It sounds like a good plan. Moses, you have comments. Oh yeah. No, thank you very much. I think this has been very informative. Okay. So I think for today we can stop there. Unless anybody else has a comment, additions. Well for me, thanks Tito. Thank you everyone for joining. Have a good day. Thank you.