 Well, everybody can everybody see the screen. I can't really see with great. Thanks This is thank welcome to the engineering update This update will be actually focused very much on geo But I wanted to start with introducing new team members on our engineering team. We've got One platform member Michael front-end what we have new xux lead Tim and When he joined us as part of the core team and on the production team we hired two two more people illa and Jason both in Europe and Asia and We have a new UX lead Sarah Vestalov. So I'm really excited to have you guys on board And now I will launch into a completely different topic because a lot of people have asked me about well, where is geo today and Where is it when is it going to be ready and what does it do? So I'll first start with the pop quiz. What is git lab geo in the first place? So is it a get last master plan for world domination? Is it a feature that improves productivity? Because you can clone from different zones. It does it to provide high availability with a short answer is Really two through five You know, we love world domination, but that's not really the goal right here it's A tool we want to use to be able to migrate from Azure to another cloud if we need to but we also want to have a live backup That we can fail over to in case something happens on our cloud provider so in this slide five shows the current diagram for What geo does today if we have a primary on the left? That might be like your git lab.com instance and then the second area is on the right, which is basically a clone and which we right now we replicate repositories LFS objects and Up user uploaded attachments and we do this most of the the bottom two things LFS attachments. We do it over HTTPS The repository side we just copied things over a standard git clone The one thing I should point out is one thing we added and I know was this geo tracking database So normally we have on the primer We have our normal postgres database on the second area. We also have a postgres database that is a read-only Secondary of that database so every change that happens on that primary gets automatically reflected on that secondary Via standard postgres replication, but we've added a second database that actually tracks on the secondary like what it has and what it doesn't have So This is how we determine on the secondary that we don't have project X X Y or Z So next slide. I'll show you exactly how that works So, you know on the left the secondary has a list of the projects it should have and on the right It has like this project registry that indicates the ones that it actually downloaded And so if it notices that it missed project three and four It'll go off and try to clone that and we put a lot of effort into approving that over 9099192 As I said before, you know, we replicate the the projects and the project wiki repositories LFS objects we copy over HTTPS attachments And things like SSH keys. We also propagate In 99192, we've improved the UX a little bit. We you know, this is the the status screen that you see if you go to a geo admin page It's not exciting But it's better than what we had in 91 and we're constantly trying to prove this But the big step forward here is that we actually could start to monitor the progress of our synchronization I alluded to some of this and I accomplished first from 90992 You know, we added the secondary tracking database before and I know we never uploaded We never tracked which uploads were actually going into the database So if you attach something to your issue that would just get uploaded and you wouldn't we wouldn't actually know about it Except for if you looked in the file system, we happen that you happen to upload this PDF or image Whatever you upload it. So now we actually track that in the database so we can actually figure out what has actually been Changed on the primary I Mention the repository backfill sync. So when you bring up a new secondary It figures out it a doesn't have these projects that goes off and tries to clone them It tries to figure out what has been updated and then goes and tries to do a get clone a get pull We added support for downloading LFS objects attachments We've made it a little bit easier for developers to set up geo and also made a little bit easier for admins to set up It's still a bit of a challenge, but we're again trying to prove that over time As I showed you before in the earlier slide We added a diagnostic screen and API to get the status of the replication from the secondary And we've had customers try to use geo and they run it to number issues and we've been Working with them to iron out some of these bugs and thanks to people in the support team to vet and Gabriel and Douglas for jumping on these customer calls and and working through them So I want to talk a little bit about sort of the The model of replication right now. We you know, there's two ways to look at it We did you have a push or pull model for replication? We actually have both right now We have on the on the push side We we have these system hooks that really web hooks that when an event happens We fire and we go off and do something so for example when you push a commit On your primary we fire system hook and that gets that gets communicated to the secondary Secondary says oh that repository has changed and does a poll, right? So the good the good thing about that is something that happens on the primary gets Immediately reflected on the secondary assuming you have a network connectivity The bad part about that is that you're not guaranteed that these hooks actually go through Nor you guarantee that they arrive in order All right So you consider project deletions and creations that becomes problematic if you have something that's supposed to Delete a path and then create the same path again. You get that out of order things could get messed up. So That's one reason why we're going to consider moving away from it in the poll model we have Asynchronous workers sidekick workers that go and just repeatedly run they they run every few minutes and decide has anything changed And then they go and try to update either the repose or the attachments or whatnot And the good thing is the regular schedule They're not dependent on user activities. So you have an idle instance. It should just pick things up Again, the downside is that you may not get updates fast enough You know, you might hit up you might do a push and you might wait Minutes for it to update and that's not necessarily a good experience for some users. So That's kind of how we have both right now, but we are going to move away from that So as I mentioned earlier, this is this is kind of what happens when you when you push a commit on the primary on the Left it gets handled by the post receivers handler fires off a webhook But again, if that connection between the primary secondary is not there This doesn't work and it will retry over time. But again, that becomes that you get into ordering issues and and I don't see issues and things like that. So What are we doing about that? We're talking about Creating an event log. So, you know what happened on the primary and you can kind of replicate it on the secondary And these are events like you create an SHK SSH key you push and all that and so we have some system and the second We figure is that okay, this stuff happened. I need to go and Resynchronize like this project or I need to download this file and so forth So that's the approach we're gonna be taking now and hopefully get away from this system You know, this again, this is a design that we're pushing on for 93 But I'm hoping for suggestions So just to summarize what we're trying to do next for you. It's reason why it's not quite production ready is that we really want to you have Geo useful for us internally as a way to Move and back up get lab.com. So for us to do that again We got to get away from the web web hooks and and move to this whole event log system One thing that we do on get lab.com that we don't document really well is that SSH keys are actually managed by the database rather than the single file That's authorized keys files that most users are using today And that's a problem because it's we don't have a sense it's a single source of truth for some of our customers because You know, you've got keys in the database and you also got keys in the file That's generated periodically and that that's challenging for Geo because you really need a Single source of truth to make this effective because if you need to update this file constantly, then you have to preserve ordering and all that so or Really gonna I'm Geo trying to focus on documenting support for that and making that easier for customers And there's a bunch of other a task that won't get into them. You can look at the issues and talk and look at them in more detail That's really a summary. Are there any questions? Yeah, Jim you mentioned why? Spokes. Yeah, that's that's a little bit of solving a different problem the distributed get problem That's something that we that is on our radar, but not for really for Geo right now That's replication from like a get low level abstraction So something like giddily once we have that in place will allow us to consider. How do we do replication at that level? What do we think is 939495 that's a good question a lot of this stuff is challenging So we're still trying to figure out the time frame of 93 I think we're gonna push on creating the event log and making it Making Geo a bit more robust for enterprises so that you know if a customer has an instance in China and one of them in New York If they're disconnected for a week, we can pick up what we left off and that's really the goal to make it Solid so I think 93 will make a lot of progress on the event log probably a lot of a lot of things to work out there You know if we get that in 9394, I think we'll feel really good about getting closer to production readiness I'm hoping You know it's hard to a pretty good time frame because a lot of the work Comes up as you do this because it's not an easy problem because basically we're inventing a replication System for git lab an application level application system But you know, I think we'll feel really good about it when we start actually using it for git lab calm And we start finding issues and and working them out and working with our customers that actually have production deploys of Geo Ndr Any other questions do we think anyone will love all the repos from one continent? That's a good question And that's why I have this item of adding support for selectively replicating some groups or projects You know, we've heard from customers that they don't necessarily want to replicate everything So we do need to have an option to say look Only pull these things down and that's an optimization that we can add But yeah, I definitely think that's a use case, but right now we're focused on just making this work Does the dr? We're currently Like I said, it works at a basic level The challenging them about dr is things get out of sync You get run into errors and things like that and you've got a review robust So at a basic level, you know, we have primary secondary. We've been testing we have customers Sort of using it For 9-2 it'll be a much better place Like I said, it will keep repos up to date assuming nothing goes wrong but the challenging part is things do go wrong and practice and geo's gonna have to tolerate these failures and try to recover and and You know, for example, the repo gets broken and we can't download it for, you know, 10 times We should try to just wipe it and try to redownload it again and see if a clean state will bring it back to life So things like that make geo a challenging product Great. If there are any other questions, thanks for joining Yeah, I'm glad this was helpful I know this was this was really focused because a lot of people are probably are not aware of what geo's happening in geo and so instead of giving a broader update I decided to focus on that so perhaps next time I'll do a broader update. So thanks everyone for joining and See you next week. Have a great weekend