 Hello, everybody, welcome to the functional update for GEO. I'm Stan Hu, no longer the GEO manager, but I'll get to that in a minute. For those of you who don't know, first of all, what GEO is, it is this project for GitLab that essentially does this. To a primary GitLab instance and a secondary GitLab instance, one of them is essentially a mirror of the other. And we can talk about more why if you want to know, but that's really in a nutshell. The team has undergone a few changes in the last few weeks. Rachel Neenabar has joined as the engineering manager for GEO now, so she'll be taking over with a lot of the responsibilities I had with GEO. Really happy to have her board. We've got two people who've moved from GEO to other teams. As some of you know, we staffed up GEO a lot this year to make GEO generally available to our customers, but also to do this Google migration. And now that the Google migration is over, we can refocus some of our team members to do other parts of GitLab. So thanks, Nick and Brett. You guys did a fabulous job, especially with coordinating this migration. Really happy to have you on the team the last few months and looking forward to working with you guys in the future. The team itself is this. Other than the changes I mentioned, it's pretty much the same as a few months ago. Still a full set of people to do a lot of great work. So yeah, let's talk about the Azure Google switchover. And as many of you know, at the summit, it happened about August 11th. And I think it was successful. We can all celebrate. And I was really proud of how the whole team, GEO, especially and the rest of the groups around GitLab, worked together to get this done. So yeah, this was a mission to accomplish. We moved over 200 terabytes of Git repository and Wiki data from Microsoft Azure to Google. And if we did our jobs right, and most people didn't even notice, right? We had a little downtime on the 11th, but as in that came back up and most people just picked up where they left off. And that's, to me, a success. And there's over five million projects on GitLab.com. So it was not a trivial matter. There were a lot of edge cases, weird things that were happening on GitLab.com that we investigated, we fixed, and we shipped it to our customers. So we moved, you know, we had millions of LFS objects, attachments, CI build artifacts, all sorts of little files everywhere. And GEO transferred a lot of them. And eventually we moved them all to object storage to kind of ease that burden, but if we proved that GEO could do that. And as I mentioned, just working on GEO was not just about working on a separate product. We were actually finding a lot of strange things with GitLab itself. For example, you know, error cases, you know, if repositories didn't exist, we'd see a lot of errors when people were kind of cloned them. So we dealt with those and we fixed a lot of issues, everything from database constraints to files not being in the right place. And so I think it was a great way to just kind of dig deep into figuring out whether we had everything in GitLab.com. And when we migrated, we actually double-checked that, you know, files were actually migrated properly. So as I said before, this is a huge team effort. It was a cross-functional group, coordination between the infrastructure team, database team, distribution team. Everybody was involved with this. So it really hats off. I'm really proud of what we did, but also how we also worked together. You know, there's so many different moving parts to this and in the end, you know, it took us a better part of this year, but we did it. So I'm really happy to close this chapter in the geo story and kind of talk about the future of geo. You know, after the migration, we actually discovered a number of things that geo should have done, but did not do. So for example, if you had a project that you created and then you changed your default branch later on in the process, geo didn't actually notice that. And there's actually, this is an issue within Git itself with what you do when you do a poll, you don't actually get the updates to this default branch and we shipped a fix for this, thanks to Douglas and 11.3 and we're talking to the Git maintainers on how to make this sort of part of Git itself because this is something that we solved kind of this three step dance, as I mentioned the issue, that really should just be done by Git. And we've got a lot of great responses within the Git community, but yeah, that makes a lot of sense. And Christian Kudair is looking at that. We found that when people were creating MergerQuest from forks, not always to commits. So normally what should happen is that the source, the target repository that you're trying to merge to gets synced with all the commits in that fork. And we found that that wasn't always happening and Mike shipped a fix in 11.3 to make sure that that would be taken care of for next time. And the last thing that we noticed is that we've got about 20,000 push mirrors on gitlab.com and when we migrated to GCP they actually stopped working. And the reason is because when we create a push mirror we assume that the git remote has been added when we create that push mirror, but that's not actually true in the migration because when we migrated, we didn't actually reflect that, we didn't ensure that that remote actually existed. So again, we shipped a fix that says every time we start this push mirror make sure that remote is there. So nothing here was in super major but the default branch was kind of a surprise. We should have got that earlier. But on that I think everything went pretty smoothly and all these things were rectified pretty quickly after the migration within weeks. So this is a feature, this feature we've been talking about a while is making the GEO more friendly to users because right now if you are working on a GEO primary you have to know that or a GEO secondary you have to know that what's the primary? What's the secondary? So if you're pushing to something you have to know that you're pushing to the right repository. So right now if I'm a user and I decide to push to a GEO secondary you get this error message that you can't push to read on the instance go to the primary. In 11.3 which we're gonna ship on the 22nd we've changed the picture here. Thanks to Ash McKenzie a lot of the hard work gone over the last couple months to make this more transparent. So if you're working on the GEO secondary you decide to push you'll actually proxy to the primary behind the scenes and then to the user it looks like it just worked. As you can see with this error message it looks like it goes to the secondary win in fact it went to the primary and then eventually the secondary will sync and get all the updates. So it's really cool if you wanna see in more detail how this works behind the scenes the issue there has the actual diagram of where all the API endpoints are working in this workflow. So that's again happening in 11.3 we just deployed the version on gitlab.com that would support this. So really excited about this feature it's a big feature. So things that we need help that are stalled with the first priority and I think everyone on the team would echo this point is that we need GEO backup and running on gitlab.com. We saw a lot of issues at scale when we had it running for the last year we found a lot of little problems and also big problems. This drinking our own wine really really helped and can underscore the necessity for this and we've been talking with the infrastructure team about doing this and it's just a matter of scheduling time and getting this done. So again, just wanna reiterate the importance of this because right now we don't have any feedback for 11.3, for example, other problems that we introduced and we can catch them locally but it would really be nice to have it on gitlab.com. A lot of the performance database queries in particular that we saw during our deployment would be helped a lot by upgrading to Postgres 10.0. Postgres 10.0 has this feature called aggregate pushdowns and what that means is if you do a select across two different databases it does some intelligence and actually makes that fast. Right now we have a workaround that basically is this really ugly query that's megabytes long because we don't have the support and so this is sort of again another cross functional task. We need help from the distribution team. We need help from the database team, infrastructure team to get this deployed, figure out the upgrade path. Not a trivial matter because every time you upgrade what they call a minor version of Postgres it's a significant undertaking. What are we going to do in the future? Well, I really see it as kind of breaking down these four different categories, reliability, disaster recovery, performance and UI UX and we're working on a number of these different issues. A lot of things related to just making geo kind of rock solid for the enterprise. So, for example, the reliability is the hash storage or we've been talking about this for a while about having immutable paths in our repositories and we're going through the stage roll out on GitLab.com. We've finished the first stage and then we're going to be continuing the next week to roll it out more and more to different projects. You know, on the DR front, we did a lot of manual things and this whole switch over because of time and because we're trying to understand like what we need to do, what users need to do and so we've got a runbook of what we did and we're looking at that and figuring out what are these things, what are the lessons we can learn from these things that we did manually on the console and how do we make that part of the product? How do we make sure that, you know, users don't have to worry about, hey, do I need to reset my verification status? Can we just do this automatically? As I mentioned, kind of alluded earlier is we have, you know, a lot of performance issues in terms of database queries because, you know, we have got five million projects. We're doing a lot of queries to figure out, hey, do we have all five million projects and that can be really expensive if you're talking about millions and millions of rows and so we're looking at different options there. You know, one of them is this feature in Postgres called logical decoding, had a long discussion with the UNGRESS consultants to figure out like, is this a right solution? Can it work? What are the limitations? So there's a lot of good discussion in that issue about that. Push to primer, I talked about that earlier. There's a lot of improvements that we can make with that feature that we just shipped. For example, a lot of the proxying is done in Unicorn, not in Workhorse and that can tie up a Unicorn worker if you've got a lot of busy pushes. So we're gonna be looking at moving that into Workhorse and other ledger improvements. And overall, there's UI UX improvements that we can always make and just make it easier for the admin to understand what's going on and GEO and so there's an issue there that kind of shows you the screenshots and things that we're thinking about. So that's really GEO. Happy to open the floor to any questions, concerns. Yeah, everybody's excited about push to primary. It's not push to secondary. Yeah, I guess it's technically push to secondary because it doesn't look like an actual push to the secondary. To anticipate needing security app reviews, I actually think we do. I think it'd be a good thing because GEO is sort of this feature that has access to all the repositories and all the Wikis, all the issues. We really should be looking at what are the holes. We've taken a lot of efforts to make sure we don't expose tokens and things like that. We use rotating token that's generated for the session but there may be potentially other things that we need to look at. So yeah, I'd welcome a security review as much as we can there. Do we think we'll make the GEO known and get a lot of public in the sense of recommending? That's a good question. I don't know if we, I think when we talked about in the migration, we didn't wanna do that because of a number of reasons but I think it's definitely a possibility if people wanna clone or poke around or secondary. There may be other reasons why we may not wanna do that just for just reducing the security scope and things like that. But what region are we thinking? I think that's up for debate on that issue. I was just saying, well, we probably want it in the US West just because if the East Coast goes down, maybe one on the West Coast, but there's an argument to me made that, hey, maybe Europe is a better place for that. Well, we do more than one. I'm not sure. I think we're looking at one right now. I think the main reason may be cost here. We've got over 200 terabytes of data and having one replica already is pretty expensive. But other than that, there's no other reason why I can't think of why we can't have multiple replicas. Okay, then other questions? Nope. Yeah. First of all, great work. Like that the fact that the migration went well and there were so many moving parts. It was great validation and really cool to see how ready GNO is for mission critical apps. If we're gonna use it for DR, I'm gonna prevent mass deletions. Obviously, we're also gonna do just plain backups, but are we gonna have something in place? I'm not sure what the easiest way is to do a mass deletion and get a lot, but are there operator errors? We're gonna make sure that don't get replicated or something like that. Yeah, that's a good question. I mean, one thing we could do is have a delayed replica, for example. So it could be lagging an hour or 24 hours or something so that things don't get actually reflected until later. That could be one avenue, but as far as mass deletion, I think we should probably create an issue around discussing if there are things that we can catch. Someone does a drop table, maybe you do something like you avoid that, but that's a database issue. But if you have something where you see a ton of delete events, maybe you should flag it and prevent that. Cool. And we had the problem with basically the cursor moving and missing things. Is that, is the issue geological replication case, issue 7420, where we discussed that? Yeah, we're discussing that in the logical decoding case. We've had a long discussion about, hey, does logical replication or decoding work? And yes, it does, but at the same time, it introduces another thing that we have to configure a more disk space that happens. Because you're basically, you're adding another replication slot. So it's like you're having another streaming replica. So there are some problems with that. You're adding more disk space as the replica goes offline. So we have to make a decision of, is it worthwhile to do that? Or there are, I think, Jose, potentially even volunteered some interesting ideas of, hey, maybe you can timestamp all your, all the events so that you're actually, when you look at, when you're looking through which events you have in process, you actually look at the commit times. And maybe that's a more immediate solution rather than try to go down this path of introducing logical replication or decoding. Yeah, simpler sounds better. But now that GEO is getting better, like we've got a lot of finishing up to do, but are there plans to, what are the plans for the size of the team? Yeah, so as I mentioned, I think the current size, that we already had two people move off the team this month. As far as I know, there are no other changes right now, but I'll leave that to Rachel and Tommy to discuss whether we need other people to be prioritized on other projects. Cool, thanks for the update. All right, thanks everybody for your time. Have a great day.