 My name is Elad Liv. I'm a platform engineer from Apps Fire, and I've been there for the last two and a half years. I love distributed system and databases, and I've been doing it for a while. I like to call myself an RSS junkie because I always open my morning reading my RSS feed. On today's talk, we are going to see some of the unique challenges that Apps Fire is facing. We are going to see what was the motivation to make this kind of move. What was involved in the migration process itself? What was the architecture that we chosen? And for the end, we are going to see a small retrospective about the two years of using GitLab because actually, this migration happened two years ago. So what is Apps Fire? Apps Fire is a mobile attribution and analytical platform. We basically help app marketers to get better decision on the running campaign using the data that we measure for them. If you have a mobile device, you'll probably have Apps Fire SDK installed by one of our clients. We are currently installed on 90% of the devices in the world, and we are trusted by the worst best companies like Slack, HBO, Alibaba, and Walmart. This graph can represent our incoming traffic growth, but can also represent the amount of engineers that we hire, the amount of microservices that we have, and so more. We are facing more than 1 million incoming HTTP requests a second, which sum up into 90 billion events per day. We have crazy amount of projects and microservices in our system, and we have more than 300 developers in the R&D organization. So what did we have before GitLab? Due to historical reasons, we didn't start with GitLab or GitHub. We actually started with Bitbucket, and we used the hosted solution. Our founders decided to go with Mercurial because it was just much more easier to go faster with it because they thought that the command line that Mercurial is providing is much simpler than the Git one. So why did we decided to move? First, we didn't want to stay in a hosted solution. As our company growth, the demands of our clients are growth as well. We couldn't put ourselves in the risk of opening one of our repositories to the outside world. It happened to us more than once with the minor repositories, but it was just too easy to do it on a hosted solution. So we want a solution that will seed inside our VPC. Latency. From time to time, we faced latency from the Bitbucket service. Some of them did happen because of our network configuration, but it started to cause our builds to fail, and it's something that we cannot live with. Limitation. Bitbucket is limiting you to 1,000 calls per hour, and it was really easy to pass it. So why did we try? We did try to use a self-hosted Bitbucket solution, but it was just a closed source. You couldn't really know what is going on inside of it. If you're facing a bug, you didn't have the ability to know if it's something that happened because of your configuration or it's something that is wrong with the product itself. It just felt like a black box. We considered of using GitHub Enterprise Edition because everybody basically know how to use GitHub, but it was really, really costly, and it gave us a minimum ROI. So why GitLab? We decided to go with GitLab because of few reasons. The first one is the GitLab growth. GitLab has shown great growth and maturity over the years. The factor became the most developed friendly product out there, and got adopted by more and more companies. We really appreciate the transparency approach that GitLab is taking, and everything in GitLab is basically public as default. You can see the issue tracker. You can see the code itself, and many, many more. I highly recommend to read the pager duty onboarding notebook that GitLab has. It's both educating, and you can also learn a lot about the GitLab organization itself. To sum it up, GitLab was the best fit for us. So the migration process. Sorry. OK, the migrations process. During the migration process, we ask ourselves some key questions. First, API support. There's a lot of things that happening during the builds on our services, and we wanted to make sure that the new GitLab API really contains all the endpoints that we need to make those decisions. Architecture. We are a growing startup, and we are in a hyper-growth situation. How we create an architecture that will send up 10x from today. Education. Can you change your developer's main tool? Tooling and integration. Do we have a sufficient tooling to make this move seamlessly? So let's dive in. During the migration, we have few things in mind. We have to save history, commits, and tags, and we need to do it in the most fast and efficient way possible. After a short research, we find a tool which is called FastExport. You can see the URL down there if you want to use it. This tool is basically taking a mercurial repository and convert it into a Git one. We started with a few in-teams repositories, and we saw that it's working well. But now, how do we scale it? How do we move our entire repositories to GitLab? We decided to create a tool, a tool that will help us to do this move by providing a self-service tool to our R&D organization. The tool was a one-liner, so it was just easy to use it. It was idiot-proof, so no one could make a mistake. It kept everyone in sync using a designated Slack channel, as you can see in the picture. And lastly, it was really safe. You cannot override someone else a repository by mistake. The basic flow of the tool was like this. First, we check if the repository already exists in GitLab. Then we notify the right team using a Slack channel. We close the old repository to writes in Bitbucket, and we create the repository in GitLab. We are converting the repository from Mercurial to Git, and then we push it to GitLab. At that end, we log into the channel that I showed before. I want to focus on one thing here, this one. It's really, really important to close the old repository to writes in Bitbucket service, because it happened to us more than once that the developer used this tool to migrate his repository from Bitbucket to GitLab. But other developers didn't know that the repository was moved, so you don't want to get into this split-burn scenario. And I suggest you always close the old repository in Bitbucket, so everyone will push to the right place. During the immigration, we got some additional benefits from it. First, we had the ability to do a small killing up. We found out a lot of dead services and a lot of the repository that no one is using, so we had a chance to download them and just upload it to S3 to store it for the future. And second, everyone basically took a part in the process. We built our developer trust in the platform group. To sum this part up, let's see what helped us. First, that we create a self-serve tool. We couldn't do it alone, and we needed the help of our R&D organization. Second, the transparency. Everyone could saw the process as it goes. We built our developer trust. We tried to make sure that everyone is aware of the benefits of moving to GitLab. And at the end, we set deadlines. I know it's not the nicest thing to do, but we had to put the deadline to the immigration. Education. As I said earlier, a big part of the immigration was education. We tried to make sure that everything is covered and everything is well documented. At AppSwire, we're using Guru to document internal things in the organization. And as you can see, we created several documents. For example, we created a document for issues that could come up from the migration in Jenkins. We created one with basic configuration and so on. Sorry. AppSwire is giving all the developers free access to Plua site. So we found out two good courses and we sent it to our developers in order to strengthen their knowledge. Another resource that I will highly recommend is this one, which can be used as a cheat sheet, pun intended. It's just a website that have a lot of edge cases that could happen using the command line story and how to solve it. Now we are moving to the interesting part. We asked ourselves, how many application servers do we need? Should we use EFS? Should we use NFS? How can we make it highly available? What is the best solution for DR? And lastly, how do we back up the GitLab? This is the architecture that we came up with. As you can see, a developer is basically reaching to a Route 53 address, which has a console service behind of it. This console service is built from two GitLab instances. We use the managed service for Elasticsearch, for the Fuzzysearch. We use RDS as a database and we use our own Redis for caching with the Redis Sentinel. And I want to focus on one thing, which is this one, the EFS. At first, we thought that we can save our souls from using the old Rusty NFS service and just use EFS. We thought that EFS did a long run since AWS launched the service, and we thought that it will be fit for us. Apparently, we was wrong, because EFS is not handling well with tiny files like Git is creating. So at first, it worked really well. But after a major part of our developers moved to GitLab, we saw that things start to break. And as always, it happened on Friday night. So eventually, we did use NFS with NFS replication. And I want to focus on one more cool thing that we did. We have a daily restore, actually, hourly restore to S3 with a replication, but we also have a daily restore to a Google Cloud platform. It both allows us to actually test our backups, but also we have a separate GitLab instance waiting on other Cloud provider, which really help us to be Cloud agnostic. You all know the phrase, if a tree falls in the forest and nowhere is in route to hear it, does it make a sound? So my version of it is, if a backup is taken without anyone who will still test it, does it count as a backup so always test your backups, self-serving tooling. Like I said before, the first tool that we created was the migration tool. But we also created a lot of other tools in order to support this migration. We created a small one-liner that we actually built based on a request from one of our teams. They wanted the ability to connect their repository with Slack. So we created one-liner that will do it really simply. And like I said, we set deadlines for the migration. In order to keep things in track, we created another schedule task that basically took all the projects from Bitbucket and all the projects from GitLab. And check which repository migrated to GitLab. After the process finished and they had the list of the repositories that didn't move yet to GitLab, we posted it on the relevant team channel. As you can see, for example, hey, data team, you didn't move those services. And you need to do this as fast as possible. So basically, we created some GitLab So basically, we created some kind of a shameless list. Another cool thing that we did is to create an in-house GitLab API wrapper. During the migration, we saw that we have a lot of code duplication in our services. In AppSphere, a platform group, we write code in Ruby, in Python, in Go, in Bash, and so on. And we saw that we have a lot of these duplications. We saw that a lot of people just implement their way, for example, to authenticate a vault, to manipulate the GitLab API paging and to filter the payload, and so on. So we decided to create a central base that contains all the related metadata that we need. We decided to create a wrapper, and not those kind of wrapper, this kind of wrapper. We created a central place that will contain all the GitLab metadata. As you can see, all our services and builds are talking to this service. It's updated by using GitLab system hooks, which is a really, really cool component of GitLab. And we use Redis as a cache layer. We also had an internal scheduler that detect data integrity. You can all see the system hooks that GitLab can post. You can find it in documentation. You can do crazy stuff with it. And it's really easy to debug it using the UI itself. It's really amazing. And if you want to read the full technical blog about what did we do and how did we do it, you can visit our blog post. And lastly, we have the ability to see if everything that we did really was good for our growth. Today at AppSpy, we are using GitLab for many, many things. We actually, a lot of teams ditch other products, like Jira, and started to manage their task inside GitLab, which is really amazing. But we did face two bugs that I wanted to show you today. The first one is this bug, which is already actually still open. One of them was because of the restores that we are doing to GCP. We found out that while we are making those restores, GitLab just take the repository directory and move it aside and call it repository.all. Some kind of timestamp. As you can guess, after some certain time, disk space became not available. We solved it by creating a small script that will clean it. But, unfortunately, this issue is still open. Another one is this one, which already fixed by GitLab. We found out that while taking a backup, some of the developers continue to push code to the repository that just been back up. So we face a lot of error, like, five changes. We read it. So GitLab solved it by announcing the GitLab backup strategy. So we just used the strategy copy, and it solved the problem. And that's it. Thank you all, and you can find me here if you have any questions. Thank you. Thank you.