 maybe a better title will be this, how to survive to have a right. Today I want to talk about four points. The first one is, well, giving you a little context about the system, about the problem. Next, we want to talk about what we did and what we're trying to do in order to solve the problem that we have. And at the end I want to talk about some conclusions and hope that it can be very useful for you. So, well, I'm from Mexico and maybe not Mexico because of the tacos. So I bring this picture in order to give you some idea about the variety that we have right now at some places and the kind of food that you can find there. Of course, we also have here, this is a place with the name of Sochi Miko and the name of those boats are Chinampas. So if you want, you can go and you can drink and have a party up there. We also have nice places like this one, that name is Oaxaca. There's a lot of culture. Of course, you know, the ceramics, that's one of the most typical places. We also have, this is downtown in Mexico City. We also have put sometimes crazy things like baseball area in the Socalo, that's the name, in downtown. We also, of course, have new buildings and great architecture. I think that for us, the most important thing is that we have, as today, one of the largest groups of elixir in the world. I'm very pleased to be part of that. In fact, in October, we are going to have the second earthland factory light in Mexico City. Of course, for us, that's also really surprising like the last year because we sold out in three weeks. That was crazy and I'm really happy because of the community. We can see that right now here that we are a lot of people here. And this year is going to be in this venue. In this venue, we are going to have the conference. So, of course, everyone is invited to be there in exactly two months. So, okay, let's talk about the context. In Mexico, we have a role about doing everything right now using electronic invoices. Everything started in that year and right now it's mandatory. Two years ago, it was mandatory that any intersection, everything that you do right now in Mexico needs to have an electronic invoice. And that's more or less the amount of people, the total tax payers that we have in Mexico. And the government right now is pushing supporting more documents. For example, bill certificates, they are trying very hard to put everything in an electronic way. So, it's going to be more and more important to have assistance to support that kind of demand. This is a statistic about, for example, the transactions, the number of transactions that we have right now. Well, this one is actually one year ago, at the end of last year, we were there. Yeah, the name of the company, the Versa. Yeah, and at the end of this year, we are going to be there. So, as we have, like, we're going to have those millions of transactions by the unit teacher. The company has that amount of users. And, more or less, we have 41 transactions per second. So, maybe a little, you are going to take it right now. Well, that's not a lot, right? When we know about Iran, we almost hear about that you can manage a lot of transactions. So, 44 transactions is not a big deal, right? We can do maybe with a number of systems. But then, why? Why use a lecture? For us, what's more about availability of the system and more about cost reduction? Yeah, those are the most problematic things that we have in that moment. For example, for our clients, it's a very critical part of the process. Yeah, we need to deliver the invoices online on time. Yeah, the clients can use money, a lot of money, if you don't deliver, for example, the invoices on time. Yeah, also, the best authority that is the SAT has heavy funds, if you don't deliver the documents. Yeah, for the client and, in the case of us, some statistics and reports that we need to give to the authority. Yeah, so the system needs to be available all the time. Yeah, any date, any time. Yeah, and the last one, we were paying a lot of money for infrastructure. Yeah, then the system was a Java system. And we have, well, if you see the bill at the end of the month, you really want to be scared about the amount of money that we need to pay for that, to support the operation. This is the growth that we had in the last years. When we started here, more or less like in the middle of that year, we started working, we were there in the company, we started working in the company. And that was the value of transactions. So, you see, we got a lot of growth just in the last three years. It was really, really crazy. The investing process is really simple. Yeah, you need just to receive the reports from the clients, pay the date, the credentials, payment status for those clients. After that, you need to dilate the documents or the data that they give you. For example, you need to check structure, you need to check some business rules. Of course, there are a lot of them. Then you need to create the invoice. There is a format that you need to follow. And there are very, very specific rules about the characteristics of that document. So, you need to create the document. After that, you do a little signature for that document. And then you need to deliver that document to the authority, to the client, and that's it. So, it looks simple and usually really simple, the flow. But the business rules that you need to check, for example, for the document can be both very, very complicated. So, once upon a time, yeah, we advised, we arrived to the company and we found this one. Yeah. First, we have a system. In the right, they're going to find the data servers that expose some web services, especially using soap. We have a database manual SQL server. And we also have a hardware security module where we store the private keys that the authority gives as an authority for creating the digital signature. So, we need to follow that rule. We cannot store that certificate in a server. It needs to be in a very specific hardware, in this case, this appliance, this hardware security module. And we have two different kinds of clients. One is the batch, clients that needs batch processing and all the ones that need that use online processing. For the batch processing, we also have a program and that one was also written in Java. So, you need that program to the client and the client installed that system in their servers and they can start writing the documents, the data, and then start sending that information to our services. Yeah? That was really a pain to maintain and support because you need to check for, in the case of the clients, you don't have control about infrastructure. So, sometimes you have problems because the Java virtual machine is not the correct version, all the operating system, all the libraries. So, it was really, really, really a pain to maintain that part. This is the configuration for the servers. So, you can have an idea about the servers that we have. We have two physical servers with these characteristics. We have a lot of Java servers with also this configuration. Yeah? And when we arrive, we say, what? Windows Server and Apache. Oh, my God. No, I really wasn't expecting that, no? Because you normally expect, I don't know, Apache running in Linux, on Linux, and if you have Windows, you are running the Microsoft Web Server, but that's a really, really real configuration. This is good. We did some tests in order to check the Java system and his behavior, and we found that. When we start to stress the system, it crashes. We were really, really scared, yeah? Because as you remember in the graph, in that one, we would start to get more and more and more clients, more transactions. And we saw this and said, oh, my God. We had a problem. And yeah, we had a lot of problems, yeah? We saw that the system had a lot of time times because of memory leaks and also because of manual deployments. We have multiple copies of the system, yeah? Literally copies, different servers. And the only reason for that was to manage crashes and isolate those crashes. Of course, those copies were not synchronized, of course. So some of them have different versions of the system running. Some of them with amounts of difference was crazy. And of course, that consumes a lot of resources, because you have a lot of copies of exactly the same system. So of course, we have the normal problems. We have a lot of problems trying to fix in order to fix bugs or adding new features or testing. Yeah, the testing cycle was very, very wrong, because everything was also manual, right? We learned that's also very, very bad, because inside the system, we found that we have a lot of very general rules mixed together with the clients' requirements. So some clients say, OK, I want to have this validation for my embraces. But those specific rules will mix it with the general ones. So if you want to modify and do something about the business rules, you are going to need a lot of luck to avoid some problems deploying into the system. We also have a very, very, very job-like library that we were using for PDF generation. And that's a very old version of the library. And of course, it has, you know, leaves have a lot of problems. And in general, we have updated libraries everywhere. So we were thinking when we saw all of these things, of course, first, yes, say, please, OK. And we feel, of course, I went to my boss and explain and say, how can I tell you that we have a very, very big problem? So the question is, well, what we did? What we did with all of these things? So of course, the first thing was this, yeah? I also told my boss and say, well, of course, we need to write, right? Everything is wrong. We need to start. We start from scratch, yeah? Everything will be OK, yeah? But we know that story, yeah? Right. Why? Yeah, because of this, we have technology, of course. That's, I think, that why we are here. And almost every time, it's nice to try something new, yeah? In this case, a lecture. And all is not new, but we want to do something different. And of course, the first idea is to rewrite everything. But we know that it's going to be hard. It's going to be very difficult to try to replace a legacy system, especially a system that is in production for a very long time. We are talking about more or less pangels working that system. That system, of course, has a lot of patch books that were fixes in the correct moment. It's going to be hard to replace that system and taking into account every edge case, no? And, well, we have developed advice, for example, from Joel on somewhere that they told us about that. You need to be very careful about thinking about rewrite everything from scratch, yeah? And we also know about this, right? We also know that we know about the literal effect, no? So you just dive in the legacy system, and then you start from scratch, and you end with two legacy systems, no? And now you have two problems, no? And you have to maintain those two systems, and it's going to be a real world problem. So the next question is, do we really need to rewrite everything? That question, and we decide to follow this strategy, OK? First, fix as much as possible the old system, yeah? Don't give up. Have some hope about the system, and say that, well, maybe it's not so bad. Maybe I can find a way to fix this thing, OK? But in the meantime, I'm going to start to prepare, yeah? And in this case, start the training with the team, yeah? Start working with the team, doing some work with the airline and the lecture for the team, and try to do that simultaneously. Second, replace the Java client with a good program, yeah? I'm going to talk more about that. And start small projects in the airline and the lecture, yeah? So some programs that they are not critical for the operation, yeah? But can be very useful in order to understand the technology, especially for the team. So we start doing that. Next, start the bit we write if you still need it, yeah? After we try to fix the old system as much as possible, and then if I wait, I find we really need to change everything, well, OK, we are going to be ready, no? Because we did the training, we did the pet projects, or we did some small projects, or so give us more experience about what we can do. And the last one of course, yeah? Get fired, yeah? So it's also a good plan, right? So I say, OK, let's try it. How long can it take? What can be wrong, yeah? You are very optimistic here, no? So it's a line, it's a lecture. Everything will be fine. So we start fixing the Java system, OK? And we made some changes to the system in order to support the new load, yeah? And the team that was responsible for that was did a really, really good job at the moment. We started updating most of the libraries, operating systems, application servers, everything, yeah? We have worked some parts, some small parts that we everywhere and say, no, those parts are really, really bad. We need to try to encapsulate those parts and we'll write everything inside, yeah? And replace those models. We also improved their deployment system. It was really, really crazy how the teams were. They, for example, in the case of the Java system, they, before they were taking the class files and moving those class files backhand and putting those files into production. No job, no, no. If you have a change in the data center, please don't do that. Especially when you are trying to fix a system, was a really nightmare. Was a nightmare of more or less like six months that all the team was really, really suffering because of a lot of dependencies, a lot of things that we didn't know about the system of those dependencies. We especially have a problem with some clients that use a specific version of a protocol. And when we try to move the systems from one data center to another, we start to receive complaints from some clients saying that they can connect, they couldn't connect to the system. And we say, why? Because it's the same. But when we checked the configuration of some protocols in the data center, we found that there was a very different version of that protocol in the new data center. So that was the problem. We need to go back to the other one. No, was a nightmare. So we also removed some of the copies. We were able to remove some of the copies of the system and try to centralize some logic in just one services. And the good part is that we were able to manage the growth from the next two years using the old system. So we were fine. We more or less saw some of the problems that we had. OK, that was about the services. The other really, really pain that we had was the client in Java. So we decided to just go for that part and for those reasons. It's very easy to deploy. You can, in just one binary, we include, for example, SQLite, a web server, everything inside the binary that was more or less like a megabytes of size. So for the client, we really was very easy just to copy the file and put it on the server. And that's it. No external dependencies, no problem with the version of the operating systems. That's nice. And we all have good support for concurrency. It's very, very fast. So that's OK. And well, we don't need high availability. So it's fine. I don't have any problem just going in that part. We started doing the training for the team. We created a two-day course. And we gave that for free for one and a half year. And we have very less than 100 people in the classes. So it was a very nice experience trying to teach Elixir and around to more people. Now we have a seven-day course. That is nice. We created the Mexico City Elixir Group. Yeah, I'm noticed on that time. And also it has been a very, very great experience. We started with small projects. We created Proxies using Dynamo. Just before Dynamo announced that it was deprecated. So we said, we loved you. Thank you. But later we replaced that one with Phoenix. So we said, OK. We're fine. Not problem. And I think that's one of the good things about starting with small projects. Because after Dynamo was deprecated, we can replace that part with Phoenix very, very quickly. So maybe a problem with that. We also have a high-waist secure module emulator that is in GitHub. I can give you the repo, the link of the repo. That is very nice. Because everyone and for Elixir is awesome for implementing binary protocols. So we were able to implement an emulator of the presence of the real hardware. So we can remove that dependency from development and from testing. Because we implemented the complete binary protocol for the clients. So when you, for example, want to start the development for the system, you can click on the complete repository and you can start the system. And that system is going to also run the emulator. And then you can start to send in the messages and do everything like the real one. So that was great. Because now the team can have everything inside the machine. We also do some changes in order to support Postgres and SQL Server. So for development we use Postgres. When we deploy, we use SQL Server. So that was also great. Because all the team has, for example, Max. So it was very hard to create the virtual machine, install SQL Server, and blah, blah, blah. So we just put the adapter for Postgres and we were fine doing everything with Postgres. And then for QA and production we can use SQL Server and everything was okay. The time for the big rewrite. First, we had luck, honestly. Because the authority defined a new version for the invoice standard. That was the best news that I can get in my life. Because awesome. All the clients need to upgrade their systems for the new format. So it's the perfect time for doing this. It's the perfect time. And of course we still have progress. With calls, with slow development cycles. So we start creating a team of five people. Two during the development stuff. One working on the deployment stuff. And two more during the testing design and test automation. In this case, for example, in the case of the tests we have two persons that have a lot of experience doing design for testing. The cases, everything. And then they learn Elixir. So they learn the people that do everything about the automation. So that was really... I think that was good to do that. We wrote a first version in six months with a part-time team in just one Phoenix app. Yeah? So of course that was wrong. So we wrote the system one time to use OTP. So we started to split the system, just in OTP, and started to say, okay, well, yeah, it's not... Again, Phoenix is not your architecture. It's not your... The way that you need to define everything of your system. So we started to put some parts in OTP and just isolate Phoenix in one part. And we needed another change in order to support umbrella projects. Yeah? And isolate more Phoenix. That one was really weird, because that took just one week. Yeah? Because for that time, we had a lot of tests. Yeah? So just when we were able to do all the changes, and everything was working, yeah, as normal. And today we're really happy because of Chris's keynote announcement that he talked about how to support now Phoenix and you had the support for umbrella projects. Well, we did a very similar thing in the system. We are going to see the architecture right now. So this was the previous system, and now we have this. Mm-hmm. Now we have the web services in Elixir. We were using... We are using ports for calling Python and Sanacea library for XML schema validation. We still use the Java part for pending web creation. The reason of that is because we have a lot of templates using that version of the library. And it's really that I need is going to take us a lot of time to relate those templates. So that's why we decided don't do that. We also have the emulator and of course SQL Server and the normal appliance and the client. So normally with the system you're going to find that that part is just the one related to Phoenix. Yeah? Anything else is just OTP and modules, normal modules. Yeah? So we have the request. Phoenix receives that request. Then we go for truly the emulator, the validations, the same basic process, the signature with the signature and then we deliver the documents. And so we have very, very defined models to do all the parts. Those are the servers. Now, with Phoenix. And this is the final timeline. Yeah? It took three years. Remember? This is the, more or less, the time and the main events during the last three years. And right now we are doing the deployment in production. In fact, we're going to support very important clients right now in Mexico, Mexico City. And we are very happy because we were doing some tests and some benchmarks and the results are really, really great. So, the last part. So the new system. So the PDM generation still uses Java. Okay? But we no longer use application servers. Yeah? So... Now it's very easy to deploy it also that part and we invoke that service from Elixir and it is working fine. For the scheme evaluation for XMLs, we use, as I said, a Python library and a LAN port. We're getting better performance and latency. And portless for development and SQL Server in production. For deployment, we are using Ansible and LAN releases. And we have right now a test suite of about almost one thousand tests. Yeah? And it's really great because it takes just five minutes. Five minutes to run and we take a lot of cases creating documents checking the validations checking the digital signatures, errors. Almost everything. So, now we need to do a change. We can just run all the test suites and we can be okay that everything will be fine. We can also develop and test everything without any external dependency. And we also create a single sign-on service also in Elixir. That was another project. And now that single sign-on service is used by other applications internal systems, ones in PHP. Don't laugh, please. And there are other ones in Rails and there are a lot of internal systems that we have also. We get flexibility to open requirements. No memory leaks. We also have a centralized management system. And the last one. Yeah? We just check the amount of code that we have in the old Java system versus the new one. And we found that. Yeah? So, that's nice, no? A lot less of code that we need to be maintained. So, we also are very happy with that. This is an idea showing the Java version. And this is the Elven and Elixir version. Mm-hmm. So, with less resources, especially memory. Yeah? So, it's nice. Remember the block? Okay. Very different one. Yeah? So, we're also fine with this. In the service, we have right now two cores. And of course, Elven is using all of them. And we are using, in fact, less than 1 GB of memory for doing those tests. Final words. It can take a long time to do these kind of things. So, be prepared. Yeah? You will need support from all the team. You need to define that training path. Team really loves Elixir. Yeah? Java developers, Ruby developers, PHP developers. When they start to learn and understand the advantage and the things that LHRCs can give you, they really love Elixir. The patient. Yeah? Sometimes you need to wait for the right time. See? But you need to be ready for that moment. Yeah? And you need to support the LHRC system for a long period of time. Yeah? It is awesome for running protocols. Also, Java need to try everything. Yeah? From the LHRC system. So, be open to reuse legacy codes. Sometimes you can try. Yeah? So, but that's fine. And this one. LHRCs is going to orchestrate other systems. So, for example, in the case of Python, we are using ports. So, when we will start the system in LHRC, the LHRC system will start to create operating system processes, containing the Python program. And then we send messages between the processes from LHRC to Python. And everything is okay with the scheme of validation. So, we will start from one related project to going crazy with different people and finally using umbrella projects. I think that the umbrella projects are awesome. So, don't be afraid of changing and direction and make the change. And, nevertheless, I think that 3 months is okay to have someone to understand and use OTP. That's our experience. Yeah? I think that more or less is the time. So, thank you very much. I hope to see you in Mexico.