 Okay. So thank you everyone. So this week we are going to have a discussion, a discussion with Rado. Rado is our DHS2 infrastructure DevOps engineer. They have instances of DHS2 that basically are for development and even our infrastructure. So he's part of the team that supports the same infrastructure and they have interesting setup architecture. So today we want to have him take us through what we have and the technologies that they are using in the infrastructure. And Moses, sorry, Rado, welcome. And that is a quick introduction about him. So he'll just guess if he has anything or slides or what, but he's going to give us a preview introduction of what we have and again, you know, also that we can ask questions if we have any. Yeah, thank you. Welcome, Rado. Thanks, Rito. Hey guys, my name is Rado Ivanov. I'm just one of the DevOps engineers working in the DHS2 development team. And today I did prepare a small presentation for you guys. Hopefully we can have good discussions throughout the presentations and after it. And I guess I'll try to share my screen now. Can you guys see my screen? Yeah, I can see it. Okay, so today I'm going to be talking about the DHS2 infrastructure and more specifically our internal development infrastructure that is used by the development team for testing development, demoing new features and so on. As I said, the DHS2 infrastructure is indeed all the servers, all the services that we administer. And I'm going to also talk about some of the challenges that we've had in the past with the infrastructure and we still do to some extent. And I'm also, excuse me, and I'm also going to talk about some of the latest initiatives that we've been embarking on and in particular this is the push to have most of our infrastructure as code. I also talk about the challenges and new things we have with moving the DHS2 in Docker and running in Kubernetes. So what is our DHS2 infrastructure? As you can see, we have a lot of things that we administer. We have test instances, a lot of demo instances, again, a lot of training instances. Only some of these are public facing so you would be familiar only with probably like play.dhs2.org or maybe demos.dhs2.org. But we do have a lot of service that we use internally for the things that I already mentioned like testing, development, demoing and so on. Other notable services that we administer on our own is App Hub or maybe it's also known as apps. This is apps.dhs2.org where we host all of the DHS2 front end applications both custom and developed by us. We also have an internal Jenkins server that we administer and run on our own infrastructure and by our own I actually mean AWS but it's all internally managed. Other things that you might be familiar with is the releases.dhs.org website where you can download any number of more artifacts of the DHS2 application. Databases.dhs.org where we distribute the Sierra Loan testing database that is kind of widely accepted as the DHS2 testing or development database or at least the most used one. And the documentation of course docsdhs2.org. Some other services that we do use but do not administer ourselves is of course Github. We recently moved from our own hosted Gira to Gira on the cloud so this is now in the hosted services section. We also use browser stack or rather we did use browser stack but I listed it here as it's still there in some places. We do use Netlify for testing front end applications. And the DHS2.org website itself, it's not hosted on our own infrastructure, it's hosted by an external company and we pretty much just provide the content there. So some of the challenges that we've had and still have is mainly the amount of servers and DHS2 instances that we have to keep around for people to be able to test, develop and see the new features that we have been working on. We have servers running on AWS on OpenStack and as well as some physical machines that are hosted on some hosting platform but are physical machines and not cloud platforms. And on these hosting and cloud platforms we have a lot of servers. I've only mentioned play and demos here because these are kind of the most widely known public facing ones but we have about 23 or 25 I think servers that we keep around all spread across AWS and OpenStack as well as the physically hosted ones. And on the screenshot on the right you can see this screenshot of play.dhs2.org. You can see that currently we have what is this 4, 7, 5, that's 12 DHS2 instances running on that server. So you can imagine that we do have a lot of DHS2 instances running on the 25 servers that we keep. It's in the hundreds, maybe nearing 200 at this point. So you can imagine that this is quite challenging to maintain, to keep secure and just to keep running in general because yeah, it's a lot of servers. As I said, the main challenge with running so much instances on the number of servers that we have is the maintenance and particularly like operating system updates, package and software updates that have been provisioned on those servers. Plugging updates. Also, it's a big challenge to have user and policy administration on all these servers basically who can SSH where, who can see what and so on. It's also a challenge to keep tabs on these flakes that is being used when it's going down, when it's actually, when it's actually all full and so on. Same goes about the memory. And of course, with the latest DHS2 versions, we've been kind of changing the development workflow. And now we have some new processes and workflows on that. We would like to enter trying to support. Basically, we are trying to shift toward testing of each pull request instead of testing full features or full releases. This basically means that we want to have a new DHS2 instance running of each pull request that you create so that the people that are supposed to test it can focus on just the given changes there. And it's not requiring a long running server that you have to update constantly. Also, we want to have and actually have some very realistic environments. Compared to some of the implementations of DHS2 across the world, mainly for kind of performance benchmarking, we've been helping a lot of implementations with performance and scalability improvements and issues across the years. We do use a lot of large databases that need a lot of resources to run. So this is a challenge on its own. And for the realistic environments, we also use them for demos and functional testing because sometimes issues do arise when you have a very kind of test dedicated environment. That's not really what's out there. So you can have unexpected results when you actually deploy it somewhere that people actually use it. And recently there has been a big push with metadata packages. A lot of new metadata packages are being developed and released. So this is another challenge that we are working on improving and making better. Before jumping into infrastructure as code, I suppose I can ask if somebody has any questions. If you guys want me to kind of go deeper on any of the things that I've went through. And if not, I'll just continue to infrastructure as code. Okay, I'll take this as a no. So infrastructure as code. Why do we use, and what is the reason that we want to have our infrastructure written as code? One of the main reasons is reproducibility. If your infrastructure is written as code, you can easily recreate it. Either the whole thing, let's say a whole cluster, a whole server, a whole service, like app hub, or just parts of it that you want to update and so on, which minimizes the risk factor. And also the fear of updating can potentially breaking something. It also really helps with scalability. You can create any number of copies easily of the infrastructure models that you have created. For example, you can create a production, a staging, a testing environment that is absolutely the same with just maybe different names or slide changes in configuration. It also helps with having your infrastructure be immutable. So we, everywhere where we have infrastructure as code, we do not do kind of in place updates and upgrades. If something has to be updated, it's usually all destroyed and then brought up from scratch again when there's something that we need to change. And as the saying goes, treat your servers like cattle, not pets. Maybe this is a bit offensive to the servers, but they can take it. And of course consistency. It's very easy to make human errors when you're when you have your infrastructure dependent on some manual interaction basically either clicking around on AWS or maybe running some scripts individually. It's easy to to make a mistake and maybe later it's hard to understand what actually went wrong. So infrastructure as code really helps with consistency and having identical environments. Also version control. You can you already I suppose know the benefits of version controlling your code and the same benefits goes for the infrastructure as code concept, you can go back in time and see what was changed who changed it. Maybe if you want to go back and so on. This makes it much easier. And, yeah, infrastructure is called just in general removes the manual labor of creating things on the cloud or physical servers and kind of minimizes or eliminates user errors. Now, now I want to briefly talk a bit about Docker. I've outlined some very basic Docker best practices and I'll talk about them in a second, but I just wanted to say that we've been pushing toward running the HS to internally exclusively in Docker. And I can't say that we are done at this point we still have work to do, but it's a big journey. So it does take a lot of time and running your application Docker, as you may know, has a lot of benefits, but it's also kind of challenging to get it done right. Yes, Gerald. Did you have a question or just testing your mic. Okay, I'll take this as testing your mic. So, some of the best practices that we, and these are very basic best practices of course the list is non exhaustive. So just Google Docker best practices you'll get a lot more information on the topic. I suppose some or most of you guys are already familiar with this but these are kind of the things that are foundational and we are creating based on that. Very important for us is that we don't run our containers or images as route. So this really improves security and so Gerald is saying that he cannot hear anymore. Can somebody just confirm that they can still hear and it's maybe just Gerald that cannot hear. I can hear you. So, yeah, I can hear you. Okay, sorry, Gerald. I suppose you can watch the recording later. So yeah, we do not run this route anymore. We did run our images. Oh, perfect. Nice. We did run our very old images as route before but it's no longer the case. We're kind of proud of that. We do not run actually we do run our application as process ID one, which basically means that this is the only process that is running within the container and this is kind of a must do for for Docker containers. If you want to have multiple services running in your container then it's maybe better to split it in multiple containers. It's a very good idea to have simple Docker files or in our case we actually no longer have a Docker file at all. We use Jeep. I have provided the link here but I'm not going to open it right now. Jeep is a, I guess, a tool developed by Google. It's mostly Java centric but I think it can be used for other languages as well. But it provides a better way to build images without using the Docker daemon at all. It makes an artifact like the war files, the DHS to war files and basically adds it to a base image. Like the Tomcat base image that we use and this simplifies things by a lot and makes the image much more simple, secure and easy to maintain. And it's also a very good idea to use verified base images. In our case we use a Tomcat base image. So we don't have to worry about securing Tomcat. It's already done by the Tomcat developers on their own. So it's a very nice and easy way to do the right thing basically. And with Jeep now, the latest images that we have been building there, as I said, built with Jeep. This really helps and makes the boot time of DHS to faster. Notably before, DHS 2 in Docker was booting rather slowly and by slowly I mean we're talking about minutes. So this now greatly improved and it's kind of half a minute boot time, which is a big improvement. And also this is something that we can be proud of, I guess. And the Docker images now are a lot more secure, as I said. This is again partly due to using Jeep for building them as well as some changes that we made recently with the sole purpose of just improving the security within the Docker images. Before going into the next topic, which will be Kubernetes, I again want to ask if somebody maybe have any questions right now. They want some clarification or anything. Right on maybe a quick question. Yes. Have you considered not bundling the DHS 2 into the image at all and just running the kind of secure verified Tomcat image and providing the exploded war file as a volume? Because it strikes me that, you know, you make very big images and particularly if you're exploding the war file within it. And if you're running, if you're trying to run, for example, a load sharing environment with multiple Tomcats, my inclination would have been to just provide the war file or the exploded web app as a volume. What is the thinking behind going the way you've gone? Well, I suppose that what you're saying here with the exploded war within a volume, this is kind of, it is a way to go about it, but I suppose it's not the most efficient way. Yes, the images are maybe bigger right now than they would be if we would be providing just a Tomcat image and exploding the war separately. But the idea here is that we just get a DHS image. And if you have Docker, it's very easy to start DHS 2. It's basically about simplicity. And, yeah, it depends on the use cases, of course, as you said, on certain environments where you would be running multiple DHS 2, multiple DHS 2 containers. And there might be better ways to go about it. But maybe you were kind of referring to a Linux container environment where you have multiple, separate, virtualized environments within the same server. No, not really, Radha. I'm just thinking of running a simple Docker environment where you don't put the web app into the image. So you don't have a DHS 2 Docker image as such. You just have a Tomcat image. I understand that it's probably simpler the way it is when you just download your Docker image and run it. I think so, Bob. And I think another kind of point is we want to, we want to recommend an image that we've tested. And I think if we, obviously we could then recommend a specific Tomcat image with the DHS 2 sort of link to that. I think, you know, if we compose this together into a DHS 2 image, then exactly what we've tested is exactly what is in production. So I think there's an element there too, that it's simpler to manage that as well. But I think it's interesting, the solution as well. I was just wondering what the rationale was, whether people considered doing it the other way. But no, good. I understand why it is the way it is. I presume you do explode the war file in the Docker image there. So what you download is... The image has an already exploded war file in there. And basically the explosion or the exploding of the war file does not happen in boot time. So this is one of the reasons why the boot time of DHS 2 is now much faster. Before the way we built images, the war was exploded every time you start the image. So this added some overhead when booting. It just made things a bit slower. So now we have a pre-exploded war basically within the image. Yeah, there's also good security reasons for doing that as well. Yeah. Sorry, it didn't mean to interrupt your flow. No, of course. I just felt like I'm going maybe a bit too fast through the slides. Okay. So Kubernetes, why do we even want to go there? Why do we want to use Kubernetes at all? As I said in the previous slides, we do have a lot of servers and a lot of DHS2 instances running. So this is kind of a natural decision to go there because Kubernetes is kind of the go to orchestration platform for containers. And you get a lot of stuff for free once you get going. Of course, there's no free lunch, but maybe it's a cheap lunch. It takes some time to properly create your infrastructure to run Kubernetes and it just takes a bit of time to learn the management things that are behind it. But once you go there, it's pretty much very easy from that point on. And you do get a lot of stuff for free. Notably, you'll get like load balancing, proxying can ingress management for free, which you don't even have to think about as a developer. We as the DevOps engineers are the ones that take care of this and the DHS2 developers can pretty much just deploy their applications, DHS2 applications and not have to worry about any of this. Of course, with ingress management, you also get SSL termination and Yes, and also scalability, both horizontal and vertical scalability is possible, maybe vertical is a bit more limited, but horizontal scalability is what you really is one of the kind of the main selling points of Kubernetes. It's really easy to scale your container applications. It's very easy to handle more traffic and so on. And if you have the proper configuration setup, this pretty much happens automatically. You don't have to do anything. If you have more demand, then you get more applications running, more servers running and so on. So this really helps with the current number of servers that you have. As I said in the beginning, we are somewhere in the hundreds of DHS2 instances and some of these are running all the time, even though they don't have to. So within Kubernetes, this is very basic problem to solve with automatic scalability. You just, as I said, you get as much as you need pretty much. And you also get a lot of packages for pre-made software for free. And like you can very easily install logging packages like the Elk stack, Grafana stack and so on. This also goes for metrics. These two stacks that I mentioned also provide metrics, tracing and so on. Pretty much you can easily install anything else that you might meet within your Kubernetes cluster besides the main application of interest, which in this case is DHS2. And if you do it, it's also easier to have your infrastructures called pretty much everything in Kubernetes is a YAML file. So it's a lot easier to manage and I did skip to the next slide and gave you a sneak peek. But yes, there is a lot of complexity with Kubernetes. But maybe the reputation kind of precedes the reality. And the reality is that with a couple of tutorials and some workshops and just some extra time, it's really easy to get started and get you along with. So it's not as serious, it's not as hard as people say. Yes, it is hard, but it's also hard to run without Kubernetes. The number of servers that we currently have, it's unimaginably hard to manage, let alone by just a few people. And it's kind of really easy to not notice issues in time and so on. So clusters are indeed complicated, but it's a lot more complicated running a big infrastructure as our home without Kubernetes. So as I said, it was kind of a natural decision for us to go there with the whole push to run DHS2 in Docker. And yeah, this is what I have prepared for you today, guys. I suppose now it's a Q&A time. So hopefully this was somewhat useful for you, maybe really useful. If you have any questions, you can either ask now or reach out to any of us later, either in the community of practice or in the development slack. Okay, I'll take you back to the Docker images and stuff. So I would like to know if you just build your own images or you start from a template like Tomcat image and then you build your own custom images and then where are you posting those images? So yeah, we do use a verified base image and we use the Tomcat verified image. So currently I think we promote Tomcat 9.0 as the main stable version, but we do also have 8.9 that we built. Yeah, and as I said with GIP, which is the build tool that we're using, we don't even have to build the image with the Docker daemon. I don't want to go into too much details, but basically how it works, you get the base image, the Tomcat base image and you add an extra layer with the DHS2 war within it and you're good to go. So this is how we build the images and we host our images on Docker Hub. You can find them on, I think the username is DHS and we have both core and core depth repositories, which I suppose the name reveals what they are. So yeah. Thank you. Another one is with Kubernetes. Do you, for instance, have two applications say demo, you have two instances of demo and how is load balancing handle, are they going, are the two applications going to write the same database? No, so we, right now we use basically single instance deployments within Kubernetes, they're not load balanced so every DHS2 instance has its own database. We do have it as a plan in the future, but I mean, it's not really a very common use case where you have load balanced DHS2 instances using the same database. So this is not our main focus right now, but it is something that we would like to explore and develop more in the future. I was going to ask a little bit around that as well. First of all, thanks for the presentation and I know you give a very short notice. Thanks for inviting me. Now, when I think about your primary challenge, and it's really to do with, you know, running hundreds of instances, so it's quite complex operation. But you are less concerned, I guess, about the data. I mean, data is, because they're mainly for demo purposes or for testing purposes and the like. I mean, compared to probably many of the people on this call, they will have less instances typically. I mean, a typical national setup, you might be talking about three to 10 different DHS2 instances. But in most of the production case, I guess it's the database side of it, which is much more critical in lots of ways. Is it on the database side that we tend to find most of the performance issues, but also it's the database which contains the assets which need to be protected and the like. Can you tell us a little bit more about what you're doing on the database side? Have you done any performance scaling work in terms of, you know, database replication and load sharing on the database side? So, to start from the back, database replication and load sharing, this is not something that we have explored, at least to my knowledge. Maybe if you can correct me if I'm wrong. But yes, it's, you're very right that usually implementations are very data heavy and the databases there are very big compared to our internal usage of DHS2 where databases are smaller. We have pretty much the Sierra Leone database, some other specific databases related to HMIS, LMIS and so on. But indeed our internal struggles are so much different to the implementations out there, still with the performance. Sorry, I couldn't hear you very well. Okay, maybe this was just a mic check. So yeah, as I was saying with the performance databases that we are using for testing, we are kind of trying to mimic the real implementations right there, right that are out there. But of course, it's not a one-to-one comparison because even though we do have very large performance databases, some of the implementations out there are much larger and it's actually hard to get realistic data. Given that this health information system, it's also not easy to just ask implementations to provide their database, even if it's anonymized and so on, it's still a challenge. So, I think, I think I can compliment that, I read what you're saying in terms of answering Bob. I think one point is we're still quite early in the transition to Kubernetes, right. So, although we've been doing some of those things kind of manually or in our previous setups, we're trying to get the sort of Kubernetes setup right first before we start expanding on the different sort of configurations that we can support. But I think we will want to support those sort of configurations if not only for testing and exploring those areas and improving and optimizing our own guidelines for these kind of use cases. And indeed, I mean, maybe it wasn't really clear from my presentations, but as Phil said, we are kind of early into going into Kubernetes, even though we have a lot of things available and usable right now, it's nowhere near complete. We still have a lot of things to explore and figure out. And as Phil said, our main first goal was to just serve the internal development needs that we have. And then we can kind of look further both into performance testing, database development and so on. So I don't know if this really answered your question, Bob. Yeah, I know it does. I was interested, I think what I got from what Phil was saying as well. The intention is that the performance testing stuff, some of what I know Gintari set up on sort of separate machines. The hope would be that that will also come under the umbrella eventually as part of the kind of Kubernetes infrastructures code setup. Yeah, indeed. And, you know, I said that most of the national implementations wouldn't be at the scale in terms of number of DHS2 instances, but there are in fact quite a few actors who run a lot. I do not see anyone from Hisp, South Africa on but I know they probably run in the hundreds of instances. I see Dajo was here from Hisp, Western Central Africa, they don't directly run many instances, I don't think, but they certainly assist with quite a number of countries running a number of instances. So, yeah, this question of how to manage many DHS2 instances, it's not irrelevant for country implementations at all. No wonder it has at least 12, maybe more. Yeah, I mean probably Kubernetes is a bit of a stretch for even the kind of the implementations that run a lot of instances, but for internal, for our internal usage, it just made sense. But nonetheless, just running DHS2 in Docker, maybe through Docker Compose because at that point in time Docker Compose is even recommended for production. I think this is a very decent and scalable way to run multiple DHS2 instances in a very controlled manner. What about the security management of Docker images? This has been always my reluctance in a way. I mean in your case, for example, most of those instances are quite ephemeral, they don't last long, they're cattle as you say. For long running instances, you have to have a mechanism to ensure that your Docker image is secure and remains secure in terms of things like operating system updates, Java updates or things like that that might come along. What's your strategy of dealing with Docker security in that sense? So I guess our strategy is just following the best practices, not running the images through, running just one process within the image. And since we started building our images with GIP, I think they're much more secure than they used to be. At this point, as I said, we are just using a verified Tomcat image, which is tested a lot by the Tomcat development team. So I think we can... But if there's a security release on the base image, for example, there's some major flaw in Tomcat and then Tomcat produces a new image tomorrow. Do we have a process around that? We don't really have a process with automated process about that right now, but it's still kind of... We do keep an eye on those base images that we use or if there's some big security issue, we would know about it. And we would basically just build new images based on the new image that Tomcat releases. But some of the challenges, security challenges that you said, like keeping the operating system and so on updated, these are still challenges regardless of whether you run the HOS in Docker or just as a standalone war file. You still have to kind of secure your own infrastructure, the service where you're running them. So this doesn't really change too much of what you have to do with hardening your own infrastructure where you're running the HOS too. But it does provide kind of a layer of security for the HOS to itself. And Gerald had two questions. I'll first go to Gerald. He's asking Kyrie building the HOS to work on Kubernetes. Yeah, we are building the HOS to Docker images to run both in Kubernetes or in Docker compose whatever orchestration tool that you want to use. I think it's pretty much the same. The images are usable anywhere that you can run Docker on. And he's also asking whether during deployment the pods are the pods restricted to use specific IPs because he sees that as a challenge. And no, I don't think we use any static IPs for the pods. They pretty much get a new bot internal private IP and public IP for each new bot that is run. So the IPs are basically recycled. And Chinsang is asking, what do you say is the best way to scale a DHS instance vertically or horizontally? Well, depends on whether we are talking about running DHS to Docker or outside of Docker. If we're talking about DHS running inside Docker vertically, of course, you just have to have a bigger machine. If you, if you're talking about running within Kubernetes, then there's this thing called the vertical pod autoscaler, which can depending on conditions that you care about like traffic and so on. Resource usage, you can increase the size of the notes that run the DHS to pods. So this also happens automatically. Same goes for horizontally, which is even easier. Of course, as I said, in the context of DHS to we do not currently load balance multiple DHS instances to use the same database or horizontal kind of scaling is. It's not something that we are doing right now. Maybe we will start doing it in the future. And if you're talking about scaling outside of Docker, then again, it's, I suppose the best way is just to get a bigger server, which does sound easy in words, but can be challenging in practice, I suppose, especially if you're talking about country implementations. You can have a fixed number of services and so on. Sorry, what if you're running a cluster of servers and say you have two servers right now, and in future you want to add another one to the cluster. How is that going to help in scaling? Is it going to be practical? So right now, we are using automated automatic cluster scaling. So the number of nodes or servers that we have within the cluster depends on the kind of the demand. So if there is no demand at all, and we are just running kind of the core services that keep the cluster going, we usually have one or two servers running at any time. And if there's more demand, if there's more demand, it kind of goes to three, four, five, again, depending on the demand. But this happens automatically and we don't really have to think about it. And this is how to scale. Sorry, how do you monitor the demand patterns? We don't really monitor the demand patterns. We just kind of leave the cluster out to scale to do its own thing. We don't have to worry too much about the patterns because it happens automatically. We are still early in the Kubernetes usage, as I said. So with increased developer, with more people, more developers using the Kubernetes cluster and DHS to in Docker. Internally, we are probably going to start looking into usage patterns and have some way to deal with it. But at this point, it's not something that we have to really worry about. I think BAO have got quite a bit of experience with horizontal scaling. And one of our plans, one of the weeks ahead, we're going to invite someone from BAO to tell us a little bit about how their experience has been. So that will be interesting. Yeah, that would indeed be very interesting. I think BAO have a lot of relevant experience with managing and running DHS to as a service for people. So they do have a lot of knowledge that they run in the community, including the developers. We definitely plan to get them on here in one of the coming weeks. Yeah. Good. Okay, Moses has a question. Where do we see DHS to in the next 10 years? I see it's becoming versatile like self-abating caps like Android and iOS. I don't think I'm the best person to talk about DHS to in the next 10 years. Maybe if you can talk a bit more about the future roadmap. But yeah, indeed, the future is bright for DHS to for sure. Yeah, no, it's a really good question. There are elements where we are trying to make it easier and easier to be able to update DHS to or components of DHS to. I think most of you guys are probably aware of the shift over the last couple of years towards continuous delivery apps as we call it, so that apps can be updated independently. There are some kind of designs over the next years to give more flexibility in the sort of backend architecture to make it even easier to kind of extend things in a similar way and to update things in a similar way in the backend. So I think we're definitely moving in that, you know, that direction of the versatility. But there's a balance, right, in terms of making things remain robust and secure as well. So yeah, I think the other thing that we are likely to see is the breaking down of the application into more of a micro service architecture. DHS do has been this large monolithic application for some time. And I know, for example, lead architects like Austin are actively looking at ways of breaking down the application into more separate services, which of course will add to the complexity, which is why it's really important that we learn from experiences like you guys are going through. Because we'll, we'll find DHS to becoming a much more flexible application but equally a more complex application to manage as my crystal ball gazing for the day. Yeah. So Moses also asking why is development mobile, why is mobile app development concentrated on Android and stopped on iOS. Yeah. Yeah, I think you know the main user base is with Android in terms of mobile. And that's one of the reasons why the primary focus is there. And there have been, you know, some support of sort of progressive web applications which means that things work on mobile devices from the web as well so that I think it's considered that that will provide some of the support for other mobile devices but I think it's still considered that the, you know, the primary use cases is Android that that's the availability of devices in the field. As far as we're aware, in most cases. I don't know whether Bob can comment on that. I think you said it all feel I mean the interesting thing about Apple iPhones and is that senior people in government tend to have. Certainly the case I know in Asia people tend to like place like Vietnam. Everybody above a certain age pay grade will be on on on Apple and everybody below that pay grade will be on Android. And so sometimes the call the call for iOS support from from East Asia is quite strong. But yeah, I think the footprint of Android generally in our world is much, much larger. Yeah, that's currently at least I could change. Yeah, probably not going to change but it's also asking do we sorry Bob. Was this also asking, do we expect DHS to web application becoming full time offline like on capture up, but on the web. I, you know, I think we've been trying to increase the support, the offline support in different apps. So I think that will continue to improve. Yeah, so, you know, it will always require a connection, or eventually, in order to kind of send in the data and get back data but definitely we've been trying to to increase the offline support. Because we know that's a really important factor in a lot of implementations in the field. Yeah, thank you. Thank you too for the questions Moses I think they were very helpful. So it's at the top of the hour. Thank you for the session. Thanks a lot for inviting me guys I was a pleasure talking to you and hopefully the presentations and the Q&A that you just had had some good insight and I was helpful to you. So, yeah, as I said, if you do have any other questions, please reach out to us and we'll try to help. Thanks for what we're hoping to do next week is talk a little bit about about troubleshooting and monitoring with with Glowroot. So if anybody's interested in that currently that's what we're thinking we will have the talk for next week sounds good. Thank you. And at that point. Thanks everyone. Thanks to you. Thanks. Thanks everyone. Thanks everyone. Bye bye.