 Welcome everybody. I see we have 32 participants in so far and two people sitting in the waiting room. Hopefully they'll be let in soon. The session today entitled scalability and performance tuning and we only have an hour so we're going to touch quite briefly on on very few issues I guess in what's really very broad topic. I'm fortunate today to be joined by Gintari and Daniel and Jason and Dan. I'll explain a little bit what they're going to present in the next slide. So I've just got a couple of introductory slides framing a few problems. Gintari is going to show us a little bit about some of the work that's been done back at base at University of Oslo around performance testing. This is work that's been inspired I guess by the huge challenge that was faced with moving into the COVID-19 response particularly vaccination campaigns where numbers got really really big. Lots of interesting developments there in terms of how we have started to be able to test more realistically more repeatably and the like. Daniel who's actually joined recently part-time assisting me with with sort of country service support. Daniel's working with a company called Solid Lines. He's got really great experience working with a very large databases with PSI and like. He's done in the last month or so a bit of a comparison around using Postgres 13 and Postgres 10 some of the results of which are kind of surprising but interesting I think for for people who care about their databases. Jason who I'm going to go back some years now. Jason Phillips from Hisp South Africa. Again long and battered experience working with Hisp South Africa managing quite a number of or quite large DHS2 instances. And then we're going to leave a bit of space I hope at the end towards open discussion. I know there's lots of lots of questions and interest around this topic and we're fortunate to be joined by Dan Kokos as well from BAO who will ask lots of difficult questions too as well. So I don't want to take up too much time. I want to leave time for everybody else but some of the challenges as I see it is I think probably one of the biggest challenges that the DHS2 is deployed in so many different ways right there are people running from in the basement as we say to in the cloud I mean there are some places where where there's a service sitting somewhere in a room in some ministry and the DHS2 is deployed on it and somehow or other they've got it onto the internet. And there are lots of particular challenges around managing a setup like that. Increasingly we we see more and more within countries that there's a government national data center which provides the enterprise service across ministries and DHS2 is deployed somewhere within that. Very common model is using cloud VPS systems. LinNode historically been very strong still very widely used. We've got good connections with LinNode. AWS used by great numbers of people. That should be Contabo not Conalbo. Contabo is a well recent player. It's proved to be very very attractive large because of cost. Very very good value cloud VPS solution. Deadly service has been used in a couple of countries as well. Through to the whole software as a service model which I think there likes a BAO, Blue Square, Hisp, South Africa I guess to a extent of offering software as a service type of offerings. And there's advantages and disadvantages and different kinds of challenges involved in all of that spectrum. The other challenge I guess we find is that there are very very different levels of skill and experience of the the the poor people who end up with the responsibility of having to host their DHS2 systems and often scale them into places where they haven't been before. Transitioning from managing quite small to large aggregate instances which has been the kind of skill set that people have built up over the last 10, 15 years or so to suddenly moving to managing very large tracker implementations. And where we've seen this really kind of bursting the through the ceilings is on the vaccination campaigns. Been looking at Nigeria for example the last few days. I mean then 250 million Nigerians and if you're going to vaccinate a reasonable proportion of that and track them all in tracker we start talking about very very large systems. DHS2 software as those of us who've been in the breach trying to manage it will realise is sometimes a very unruly donkey. It doesn't always do behave quite the way we expect it to do. The rapid increase in scale that we see often exposes what were I guess previously undiscovered performance issues. And so managing a system like HS2 is not just routine monitoring. There's a lot of troubleshooting involved as well. Particularly as we start stretching the boundaries of things. Sample of the kind of issues which is probably fairly random there are many others. Traditionally analytics generation time has been the big performance thing right. People managing big aggregate databases and they start the analytics job at midnight and hope that it'll be finished by five in the morning before the users start getting in. That's still a challenge in a number of places. There are still very large aggregate instances. For people who don't realise it skipping zeros in analytics tables has been a really major breakthrough in improving the time that takes to get through. The problem of zeros in databases I think could be the topic of a whole discussion on its own. We don't have time to get into it now. The other thing that we find and you find this because we're monitoring there's often you'll find that different API calls API calls cause different types of distress and the kind of distress that we saw a lot the last two three years is very very high Tomcat CPU usage associated with with particular API calls and one of the learnings that we found from that is the kind of general rule of thumb if you like when you see really really high CPU usage usually the solution to that isn't to add more CPU. The reason the CPU usage is high is because the system is struggling with memory and we've had a few quite inefficient API calls which use gigabytes of memory and I think we've done quite a lot over the last year in particular to hammer a lot of those on the head and again a lot of credit going to Gintari and her testing environment for identifying and dealing with some of those. The other thing we find expensive back-end database queries usually related to analytics some of these are really quite difficult to deal with and yeah lots of diagnostic skills and and hunches and things required recent problem that's been rearing its head for me at least certainly the last six months to a year has been this issue of database connections being made outside of the data of the connection pool in DHS too we have this this connection pool which is used for probably 90 percent of all the queries but there are a few places in the code where we make connections outside of that pool and we've seen cases where those connections start to get out of control and reach the reach the maximum connection count on the database and bring the system down. The kind of issues that sort of come up on a on a frequent basis I mean they're probably more but these are the ones that were in my head at the time of making this slide. As I said at the start I mean managing DHS too is a mixture really of of sort of routine monitoring and then frequently quite a lot detailed troubleshooting. The different tools people use for the real basic stuff things like moon in and monitor I always install moon in just by default not because it's beautiful it's actually quite ugly but with very little configuration I get most of the the useful information that's required on a routine basis. I think there's a bit of competition at the moment between the Prometheus Grafana combination and the Elk stack combination. I mean those are both two much more beautiful platforms in some ways much more flexible and I know they are both Prometheus and Elk uses in you know our little panel today so we might get a bit of discussion around that. Profiling particularly for troubleshooting is is really important. Your kit is something that's favored by developers really detailed stuff. I found Glowroot thanks to Dan for this Dan introduced me to Glowroot last year and I found it totally invaluable for fixing all kinds of issues over the past year. It's worth pointing out maybe that knowing what to look for is much more important than having pretty tools. I've seen lots of places where they have very beautiful tools but they have no idea what the data means. Besides monitoring, alerting is really critical and to be honest I think it's very often a weak spot in a lot of our implementations. I mean generally it's quite easy to set up things like email alerts but a lot of implementations find it quite tricky because of the DNS setup that's involved with having a reliable email alerting system in today's world. Yeah so we co-present us today as Gintari, Daniel and Jason. I'm going to give them spotter. We didn't discuss in advance the sequence to go in perhaps we'll just go in the sequence that we have here starting with Gintari who's going to tell us a bit about the performance testing that we're doing back in Oslo. Daniel as I say is going to give us just a very brief overview of some of the the analysis he's been doing recently on Postgres and Jason's going to tell us a little bit about what they've been doing down in Cape Town on the monitoring front. What I hope to go to after that is a bit of more open discussion and questions. I've got a couple of discussion questions here which we might get to. The big one for me is how do we balance that a very complex DevOps type environment with a kind of limited human resources. People who are just learning how to administer systems who are maybe taking on these challenges for the first time. Things like Kubernetes, things like load sharing with Tomcat and Postgres. I will say that in almost every case I've seen people starting off trying to build configurations with with load sharing they almost all have failed and performed much worse than if they had done it more simply to start with. There's a lot of understanding that's required to to put more complex configurations into place. A lot of learning we still need to do around that about how to be able to package those things. I mean if we're really going to recommend more and more sophisticated deployments then how can we package them to make it easier for people to use. But there's just some sample questions from me. The discussion may well develop differently. But what I want to do now is to give over to Gintari and let's move forward. Because you guys confirm that you see my presentation. Gintari, we can see your slides. Perfect. I would like to stop sharing my video because I do want to show you guys performance. I know it's just me Gintari. I can't hear you. I can hear you Gintari. Does everyone else hear me? Now I got scared. Okay I assume you guys do. So yes I'm going to stop my video because I'm going to be showing you performance tests in action later. So we have been doing performance tests for around two years now and we are testing the releases. But this year has been special because of COVID pandemic of course. It actually made us to encourage us I would say to do some more extensive performance testing. We wanted to make sure that DHS2 can reliably support mass vaccination campaigns that the packages, metadata packages we deliver support large scale of data and we wanted to be proactive and find issues as soon as possible instead of you guys coming up to coming to us with issues. And yeah we were aiming to give you some recommendations for your implementations but later on that. So steps we took. First we started with database. We did not have any large scale databases that we could use for this initiative. So we had to basically start from scratch. We generated a lot of org units and users around 400,000 org units and users but gradually we actually scaled down on that. We imported four metadata packages. First of all of course it was COVID vaccination package and then we also used COVID case-based surveillance, COVID events and API packages just to be able to you know cover different scenarios. And then we generated data. So in the end we ended up with three million tracked entity instances, more than six million enrollments, eight million events and a lot of data values and tracked entity attribute values. Then we got ourselves a quite powerful server. It had 32 cores and 128 gigabytes of random access memory but we quickly saw that this wasn't showing us issues as fast as we wanted to. So we scaled on on that as well. We got another server or AWS instance that was that had only eight CPUs and 42 gigabytes of RAM. I'm sorry I'm hearing some background noise. Then we came up with the workflows that we thought it would be the most common in the vaccination campaigns. It included all areas of the HS2. So Android, analytics, tracker, aggregate, everything. And we implemented tests for that where it was missing. I'm going to show you the test board we're using. We're using a tool called Locust IO as a test load generation tool. It is an open source tool that is written in Python but we do use Java integration. So right now I'm running the tests and I have very low load just to be able to show you guys how it looks and what kind of information we're getting from it. So here, Martin did you raise your hand? Is it possible to zoom in a little bit on the figures? Thank you. Yes. So the figures, is that enough? Yes, thank you. Great. So this is how the tools dashboard looks. It gives us information about the endpoints we're testing, the response breakdown, which one of them are the slowest, which one of them are the fastest. It also gives us some nice charts that shows how much response times depend on user count and similar. But this information is usually not enough. It is enough when we want to compare the releases in terms of performance. But when we want to find issues, we actually have to monitor the test front better. So Bob suggested us to use this tool called Chloroots. Now I know that it wasn't his recommendation but we're letting the tool and it's very helpful. It allows us to find a lot of issues. I'm not going to talk about the tool itself a lot because I know that we will have a presenter speaking about monitoring. So the tests, I want to quickly go back to the tests and tell you guys that we write tests in the way that they can be runable against any implementation or we try to. So the test generates data and API calls based on the metadata you have in your database. So if you would be interested to collaborate with us on that, you can just drop me an email and we can make it work. We can benchmark your databases and instances. Okay, back to the presentation. So with this framework and performance tests, we were able to make a lot of impressive bug fixes and increased performance by a lot. So what we found was actually what Bob talked about earlier. It was slow database queries. There was high CPU load on the Tomcat and similar. Also we found some our own apps making very excessive calls like skipping paging or requesting all fields when they didn't need them. So this is the chart that you might have seen on Monday. Marcos actually presented that on what's new on the HS2 session and it just shows the performance improvements that we've made on Tracker. As you can see the improvements are very impressive and especially what's related to Android. We were able to speed up sync by a lot. There's also some findings that we have not fixed yet and that was because it needs quite a bigger effort I'd say. So that's related to general Tracker CPU usage and Tracker being very heavy on the server. We want to improve that. And also analytics. We saw that analytics is not very fast when it comes to large volumes of Tracker data. So we're going to be working on that soon. Hope so. Yes, and recommendations. So I already mentioned that monitoring is very important. You should definitely get that. And also one thing that I'd like to recommend is to not be scared of upgrading the solution to use out. You use that support the HS2. So Java, for example, we did find that Java 11 works best with the HS2. So this is our official recommendation. Yes. So that's all I wanted to share with you guys. In case you want to contribute to our performance test, please visit this repository on GitHub. And if you want to collaborate on performance testing your instance, just let me know. That's great, Kintari. I think very informative and interesting. And yeah, I hope people do take you up with your invitation there. That we could help by using our test environment. And country implementation can actually deploy it themselves and test against their own test databases or staging instances. Very good. Thanks very much. Thank you for inviting. Who do we have up next? Daniel. Now Daniel, I know you've got a lot of facts and figures there. So we have to squeeze you into 10 minutes. So good luck with that. I want to go quickly through the slides of the results. Okay, let me share my screen. Yeah. Can you see my screen? No. Let me... Yeah, okay. Okay. Hi, everyone. I'm going to talk about the performance between PostgreSQL 10 and PostgreSQL 13. And for this, we have run several tests for different servers with the DHS 233 6 and the others with 236. And each of the servers with different PostgreSQL version. Okay. Well, the tests were about the database sizing, some of them about the analytic tables update, some load testing about the track identity instances, about the upload and download the TI's, because this caused a high load in the system. And finally, well, some execution plans about the some sample queries and some conclusions about the test. Okay. Well, these are the specifications of the servers. We have used AWS instances with the ACPUs 64 GB of RAM. And all with the same operating system, the web server, Tomcat, and the different PostgreSQL versions. The database we have used was the same for all the servers. It was a production database, lurching off to draw conclusions. We have more than 65,000 data elements, a lot of programs or units, TI's, events, etc. Okay. The first test were about the db sizing. And as we can see, one of the main advantages about using PostgreSQL 13 is the size of the database. It's up to 42 percent smaller with full analytics. And the difference, as we can see, is higher if the years of the analytics are increased. The next test were about the analytic tables update generation. And well, the performance with PostgreSQL 13 is a bit lower, about 10 percent more or less. Well, these are the results of the test, about the results tables update. And about the analytics tables update, the timing is similar between 233 and 236, although it's better with 236. But it's better also with PostgreSQL 10, as I said, about the 10 percent lower. And if we go to the timing, we will see that the main bottleneck about the analytic tables update is when the analytic tables are being populated. As we can see, with this task, when the analytic tables are being populated, for example, for the events, it takes about one hour more. So it's not very good. The next test that we ran were about the load testing, about the upload and download the TI's with a different number of concurrent users to increase the load in the system. And then we'll get the average response times. And as we can see, the performance is much, much better with the latest release of VHS2. And of course, the improvement is about the 20 percent better with PostgreSQL 13. Okay. And more or less the same with the post test, about the target entity instances. It's much better with 236 and PostgreSQL 13. Okay. And finally, we have run some tests about the execution plans of some sample queries. Okay. The performance is better with PostgreSQL 13, but not with the analytic queries, as it was expected. And for the other kind of queries, the performance is better with PostgreSQL 13. And the gain with queries about the target entity instances, so for example, with the relationship item is much, much, much better with 236. And of course, with PostgreSQL 13. And finally, that's a conclusion. We can say that the upgrade to PostgreSQL 13 is recommended. And of course, the latest releases of the HHS2. Because as we can see, the database sizing is up to 42 smaller with full analytics on PostgreSQL 13. And, well, although the analytic tables update takes about the 10 percent more, more or less with PostgreSQL 13, the overall performance improvement outweighs the loss of that efficiency with the analytic generation. The global behavior is better with PostgreSQL 13. And of course, with 236. Thanks, Dan. I mean, there's a really interesting bit of analytical work that you've done there. I'm a little bit relieved because I've been recommending people to go to 13 for some time. I was worried that you were going to come up with the wrong conclusions. I'm kind of surprised that the analytics generation takes longer. But it is what it is. The disk space saving is hugely significant, I think for a lot of people with disks, actually quite expensive part of their plan often. But maybe we'll hear from Jason and maybe from Dan. I don't know where they are currently in their thinking on database versions. But let's move on to Jason's presentation and then hopefully we'll come back to this question. Thank you. Okay. Thanks. Interesting. You should say that, Bob. Funnily enough, Drive Space is one of the issues we don't really worry too much about in South Africa. It's one of the things we have an overabundance of. But I suspect that's specific to our least environment. All right. Good day everyone and welcome. Thanks for attending. And Bob, thanks for inviting me to talk to you today. Also, thanks to Renea Rousseau, an infrastructure colleague who helped put this short presentation together with me. And apologies again on behalf of Pochlaki Maloy, the current infrastructure manager for Hisp, South Africa, who was unable to attend today. That leaves you stuck with me. So let me introduce myself. I'm the ex-infrastructure manager for Hisp, South Africa. And I've been with Hisp since 2013, so nearly a decade now. And today I'm going to talk to you about the importance of performance monitoring in larger scale DHIS deployments. I say larger. We have a reasonably complex environment, I would guess, by comparison to the average NGO or NPO type environment, certainly. But I'm well aware that in a corporate scale, we're definitely small potatoes. But in our peak in 2018, we had around 400 servers. We're now at about 250 through some efficiencies and change in our design. And that's mixed, what we call the Bob model, which is a mix of bare metal and virtualization in our environment. At the time, we were hosting around 220 instances of DHIS2, hosted across 54 physical Intel Xeon E5s. And our total environment had around four terabytes of RAM. And amongst that lot, not only was DHIS2, but other applications as well. But we were hosting about 80 unique DHIS2 databases. And those were from multiple countries. And as I'm sure you surmised, they were mixed. Originally in the early years, certainly back 2013, 2014, 2015, most of our databases were routine or aggregate databases. But over the years, as Bob mentioned, the focus on tracker and event has increased. And so today, in fact, our tracker databases outnumber our aggregate routine systems. So the HISP South Africa infrastructure team has isolated between about two to seven people over the years. But we are supported by a large data science and data management team of over the years, varied between 12 and 25 odd individuals. So I'm sure you've all heard great stories throughout the conference of excellent work and success. Today, I'm going to tell you a slightly different story. It's one of trial, tribulation, tantrums and trepidation. And I share the story with you all in the hopes that you'll learn as we did from our eras and work towards ensuring they don't become mistakes. Any error is a good learning opportunity. And to paraphrase Robert Kiyosaki, the best way to predict the future is to study the past. So let me start by saying it's my experience that there is no normal. As Bob and the other speakers have mentioned today, there is a wildly different performance and behavior animal built into two different versions of DHIS, such as a routine aggregated national instance, and compared that to a, for example, a tracker-based instance that is, you know, receiving data from mobile devices in the field via the API. Likewise, over the years, depending on the version of DHIS that you've used or the specific configuration, the type of database that you use, the type of usage that it will experience, whether the traffic is more aggregated or whether it is highly transactional, you've got factors such as the resource allocation variation between the web server and the database server, which will have a massive difference on not only your system's performance, but the is dictated to some extent by the type of system that you have. And obviously there is a vast difference between an instance serving a population of 5 million versus a population of 55 million and over the years, the hardware has changed significantly as well. If you compare the CPU market of a 2002 era E5 Xeon with a modern day AMD EPYC 7700 series processor, they are just worlds apart. You can condense an entire data center of E5 Xeons into one or two EPYCs, 30 or 40 machines, and get comparable performance, albeit potentially with some troublesome bottlenecks here and there, but purely talking CPU power. And then of course, the other thing to consider is your connectivity. What is in place for your particular project and that will definitely affect your system performance, doesn't matter how fancy your server is or how slow the server is if the people connecting to it are doing so through the equivalent of a straw or using edge or some intermittent or poor connection type. So this absence of there being a normal, in my opinion, means that there is no shortage of ways to correctly monitor and track your system performance. There's definitely no shortage of wrong ways to do it either, but there are lots of ways to do it correct and they won't all be the same. And sometimes it will be inappropriate for certain circumstances, but absolutely appropriate for others. So my story of Woe begins a few years ago and it was a national mobile-based tracker capture program, sometimes five or six years ago. And it had an initial deployment with a couple of hundred devices to be scaled up to a national scenario. Each device would then be in the hands of a mobile healthcare worker capturing hundreds of records, which would then sync up daily. The program's timeline from concept to launch was extremely aggressive. So I meant to interrupt just a brief warning, you've got about two minutes. Sure. It was extremely aggressive, it was to last several months, but kicked off just a couple of weeks after it was conceived. So this rapid deployment meant that we didn't have a baseline, we didn't do enough testing. We couldn't use hibernate caching and database replication because of the time we didn't know how to. And so we were ill-prepared for what happened next. We had no performance monitoring capability at the time. We were using a company called monitor.us to do all our performance monitoring, and they had just a couple of months before decided to switch over to a paid model. And the cost implications of monitoring such a large infrastructure were just prohibitive, so we couldn't do that. The result, because Bob spoke earlier about the behavior of the API and the certain circumstances, back then it was worse, it's a lot better now. And because of the short timelines that the project was operating under, and because we had no way to monitor properly what was happening on our systems, we were caught unawares when we began to experience considerable performance issues, degradation in performance, and then eventually repeated tomcat death. So we had multiple systems load balanced to try and solve the problem through hardware at the issue, which is I think everyone will recall at the beginning of the session, Bob said that is not the answer. He's quite right. And we had to learn that the hardware, because we just did not have sufficient data to identify the problem and respond to it in the most appropriate manner. So we had a lot of people spending a lot of time propping up tomcat instances after they'd fallen over. And the result was a significant cost increase for us in terms of what that project cost us. It didn't need to, but it wound up costing us a lot more than it should. One minute. I won't be much longer. So that's where my message of understanding the problem is key. And some advice what we've learned from that experience. What is your normal? Achieve your baseline. Look at a combination of both long term or medium term and live telemetry. So we use Elk and net data, but there are lots of really good tools out there to do that. Use the pasta plan for the future and the standard systems spend the time required to do the testing in advance. And make sure that a good monitoring system is in place, which will decrease your response and investigation time significantly, which in turn reduces your overhead. Do stress testing, load testing. We're excited to speak more to Atari and experiment with the various configuration possibilities. And the six P's proper planning and preparation prevents poor performance. And we've since had some success in the ICSP program. There are speakers who have given presentation on this far better than I can. Thanks, Jason. I think we're going to have to leave you on a high note here of success. I'm glad we ended with the success slide. I know that the journey has been hard. We've only a few minutes left. I wonder, Dan, could we ask you maybe to respond to what you've seen or even just to add something from your own experience at BAO over the years, which might be useful to others to hear? Yeah, I guess that after seeing that presentation, I know I'm going to get a lot of requests to upgrade to PG-13. I think the problem with that is that sometimes it'll help you, sometimes it won't. The tracker improvements I think will be incredibly helpful. One of the things that we've been working on with monitoring is we're using a tool, of course, it's commercial, but it will work pretty well called Datadog. And what that allows us to do is it's built more for the model like Netflix where you don't have individual instances doing one thing, which is kind of the Glowroot model. What I can do is look at RAM uses across all 200 servers and see what's going on there. And so that's one of the tools that we've been kind of digging into to help find where the bottlenecks are, depending on versions is one of the things that we've kind of run into. Yeah, that's interesting. Datadog I know is very, very nice. I learn a lot from looking at the Datadog documentation pages. I've not had the pleasure of working with the tool. There's quite a bit going on in the chat. I wonder is there anybody want to ask any particular question to anyone who's presented? This is the point where people tend to go quiet. Otherwise, Dan, can I ask you something? Particularly around load balancing. I think you guys have got more experience with doing load balancing than anybody else. I've seen lots of examples, as I said at the start of people attempting to do it following the documentation, but not really understanding what they're doing. And as a consequence, making performance far worse than better. I don't know what your experience has been. I know a little bit. I'm particularly interested around replication of the database, for example, and difficulties with that, with analytics. Any points or advice? Yeah, so with the horizontal scaling, we did a bunch of testing. The first project we did it on was with datum. So we had the resources and a pretty huge database to work with. And what we found is somewhere between once you hit six nodes, the overhead for everything else kind of makes it a moot point. And so what we found is three to six nodes seems to be where you want to do the horizontal scaling. The main bottleneck in that case is the database. And so you always want to have a replica that you can fail over. Database replication, as everyone knows, is incredibly difficult. And so for that, we actually leave it to Amazon and we use their RDS products because they probably have more engineers than we have staff just working on that particular problem. I will say the one thing, and we discussed it with Lars, is as it stands right now, there's not a good way to autoscale up and down because the DHIS2 comp has to know which instance talks to which instance. Is there any other questions about kind of the horizontal scaling? Oh, thanks, Dan. Yeah, I think that the Postgres replication is still really a vexing issue for many of us. And as I say, a lot of the cases where people have tried it, it hasn't worked out too well for them. It's really attractive, this idea of letting RDS take care of it. I'll give an example of, granted, this was quite a long time ago, but I worked at the travel company Orbits, and they had a team of probably 200 DBAs. And at one point, I asked about, do you do replication for databases to kind of take the load off? And he said it's just easier to have a larger database, a larger database server than it is to deal with replication. Space, and I think that's your experience as well, is it? You guys were looking at it, they were tapping with Glocky a number of times around replication. I don't know. Yeah, so we do use it, but I agree with Dan, our experience is probably by and large, and with the exception of very specific circumstances, it's more efficient to just use more resources, throw more resources at the problem rather than using replication. I was going to say the other example use case that we've seen is where you have the production database doing all the data collection, and then you use the replicated database for reporting. But the analytics process is pretty intense. And initially, we found that the replicated database had to actually be larger than the original database just to be able to keep up with all the analytics lines coming in. I think what we're all looking forward to seeing is taking that whole analytics generation task off the domain data collection database completely. And I know that there's some movement in that direction. I know Lars is actively looking at the problem as well. In Terry, in your performance tests, you haven't tested running against load balance Tom Katz or against replicated database at present, have you? No, not yet. But this is a plan for, of course, going forward. And we are definitely looking for any kind of recommendations that we can test. We are planning to test, of course, several setups and just see which one works best. So yeah, later. If I may have a question for Daniel. When you did your PG 13 versus 10 testing, how many years of analytics data were in your data set? We make tests for one year, two years, four years. And I don't know much year for the full analytics. I can't remember that. You're going to make him run a query live now. No, no. It was more, it was more along what I had mentioned in the chat is that we noticed, basically, you need to tie a CPU to the number of years. And we started getting to the point where you know, most systems are for eight CPUs. And so once somebody hit 10 years of data, we started running the bottlenecks and try to figure out what was going on. And we basically found that it was like, as analytics generates its tables, you need basically a CPU because it's each year basically gets a dedicated CPU for while it spins out the analytics. I saw that in the chat. It's actually quite an interesting little rule of thumb. We need many more of these rules of thumb, I think, to try to at least take an initial stab on what kind of resource allocation to throw at what kind of shape of database. I think we've reached the top of the hour. We're probably going to be kicked out soon. So if just in case we are, let me just thank everybody who's presented and thank Dan for making time to be available as well. It's been interesting for me.