 All right, it's a full hour here. So welcome everyone to the Functional Group Update for the distribution team. Today we're going to go through accomplishments, concerns, plans, and then finally, questions. So we're gonna start with the Yamleba's GitLab package. A couple of items we worked on in the past couple of weeks. We have let's encrypt on by default, but when we say on by default, we mean whenever you set HTTPS and you do not provide certificates, we're gonna try and fetch let's encrypt certificates for you. And furthermore, once you actually have those certificates, we are going to periodically check whether they expire or are they expiring and we're gonna try and ought to renew them. We also spent a lot of time in upgrading internal libraries in the package. That's actually expanding every time we add something, we actually need to test things thoroughly. And I think we upgraded something like seven or eight libraries that have impact throughout the package. So that was a pretty big item that we had to go through. And one thing that we've been having, basically since the beginning of the package, we didn't really have a good way of enforcing upgrade parts. So when we would have a breaking change, we recommend in our documentation that you go between major releases, but what we realize more and more in the past couple of months, users just jump between releases, they don't check changelog, they don't check blog posts, they don't check messages, they just go from, for example, eight dot one to 10 dot six and things get broken. The problem is we also don't have a good way of making sure that even when a user jumps like that, we can't easily recover them or other user can't easily recover from a failure. So we worked a bit on enforcing upgrade parts. So now with GitLab 11 that is due next month, we'll abort the package installation as soon as you go from an older version to 11 and you didn't upgrade to then dot eight. Next item we also worked on is the GitLab Provisioner, which is a combination of an Ansible script and Terraform. The reason why we worked on this is we needed a way to quickly verify whether HA is working correctly when we boot things up because of the breakage we had a couple of months ago and this is progressing very well. Few items left there to do, basically we need to connect the Provisioner to our CEI and we'll have automatic provisioning of the full HA environment on demand so we can test things before the release. Couple of concerns, we keep adding components. When I say we, I mean, why community as well. We are adding libraries that are not big per se, but every library we add, we need to go through a certain checklist. So for example, whether a license is acceptable, then whether it can be built on multiple different environments and then finally, every library we add adds a test route that we need to cover. We also, with ever since we introduced Prometheus, we also are adding various exporters and I think recently we added the, I forgot the name, something for alerts. It slipped my mind, I can't remember right now. And that actually requires more configuration. So every time we add configuration, we jump into a problem where user can have any certain number of configuration options configured in a certain way and we can't easily cover all of those cases. Last time I mentioned that we worked on triaging or rather that the whole team is doing a weekly rotation on triaging the issue tracker. And that is uncovering a couple of interesting things. We have quite a number of actionable items to go through. So what counts as an actionable item? If something is set for scheduling, that means that we need to devote some time to it, whether reproducing the problem or looking at the impact of the problem and trying to fix it. So that list is growing and with a number of items we can actually resolve per release. This is one item we are going to have to keep an eye on as it can go out of control easily. However, the issue tracker is under control now, we are not growing like we were growing before. So I think that's a good work done by the team. The triaging also uncover that we have a bunch of HA related issues and that's a worry because with GitLab.com running in the HA configuration and our customers, the biggest customers running HA, we will need to work on that with some more priority. The last one is a bit of a concern for the whole company. I personally feel it's a big deal and that is geo is difficult to set up. This is not strictly a problem for the distribution team but if you go and take a look at what needs to be done to just install geo, I think it should be quite apparent that something is not going right. The geo team is working as best as they can but of course they are responsible for the features as well. So as we get merge requests from the geo team, we review the diff but we don't have the time to look at the whole picture of how this thing is evolving which kind of shows with the documentation we have on setting up geo. So this is something that we will have to worry about very soon as a company as we start selling more geo setups. Moving on to the cloud native home charts, we worked on preparing for beta. Beta is going to be announced very soon. So as preparation for that, we changed the name of the project, we are now just GitLab. So the chart is now in the project named GitLab and the big repository of all the charts we had, we split them up into separate projects. And now when you actually wanna install one of the charts there, you can get also versions charts. So quite a lot of nice work done there. When you install the new chart, you're also going to get GitLab runner configured out of the box. It is going to be configured with some restrictions. So you won't be able to run in a privileged mode because it's deployed on the same cluster where GitLab is deployed, but it's going to be enough for you to quickly spin up GitLab and try either one of our demos or just to test how things work. Similar to GitLab runner, we have monitoring as well, a bunch of runners, well, sorry, not runners, but the exporters was added to the chart as well. So the whole story we have with the omnibus package, there is pretty much covered. And we exposed almost all configuration that you can configure in GitLab YAML and we added the outgoing email support so you can connect to external mail services. One thing that we didn't add yet and we moved it out of the scope is we are not going to be supporting relative URLs, which is a feature of GitLab for now. And we are not focused on incoming email support, which means that we don't have the feature parity to what GitLab.com offers at the moment, but this was a decision in order to make sure we can actually hit our beta dates. Concerns, as we were working on GitLab runner chart, we realized no one is actually maintaining the chart, which is a real shame because that's the only GA chart that we have. We had a number of community members submitting merge requests and no one reviewing them. And when we changed some things to get the support for the new charts, we managed to break a lot of people's workflow, including our own, with setting up out to DevOps, which was not really great. So that's a problem that we need to resolve somehow with other teams. And we also have a bit of an issue with what is our target audience with these charts. We started with GitLab.com being the prime target, but as we started working on this more and as the focus moved more towards GCP migration, we were left with trying to test things to the best of our knowledge. So now we are running into issues that are pretty much similar to what we currently have with the omnibus package. We are targeting single small installations with allowing options to have a big installation like GitLab.com. That's not currently working for us really well with the omnibus package. So I wonder how it's going to go with the charts. But we will see the reason why we are focusing on smaller users at the moment is with our users contributing their time testing this thing. We already uncovered a number of issues that we previously didn't look into just because our workflows are not covering them. We wouldn't be catching them on GitLab.com anyway as well because we are not using the same features there. So this seems to be a right move for now to get to a usable chart. Another thing is our transformation from a monolithic application is challenging. It touches all of the teams in the company in engineering at least. And it seems that it's really difficult to get on the same page on some of the priorities of these issues. The problem with that is we can't really slip with our deadlines at all. We don't have enough time obviously. So sometimes it feels like we have a bit of a head on clash with some teams when we start asking, hey, how do we do this now? We can't move without this problem and that causes issues with planning with other teams. But on good points there, I have to mention that Alessi is doing an amazing job together with Jakob Vosmeier and Nick with direct object storage uploader. That is a really big change inside of the GitLab Workhorse and that is progressing. And we are currently dependent on that feature before we can release the charts in beta. Another thing is backup restore task. We realized that we won't be able to deprecate our GitLab omnibus chart without people backing up their data and exporting it into this new environment. So we try to get some work done there. Ahmad was working on getting Gitaly Endpoints written there but we also realized with the help of Jakob that we need to finish that story completely. It wouldn't be really helpful to just do one thing. So basically with what's left of Gitaly team, Ahmad is working on getting that backup and restore task done. And as we are progressing further, we are discovering more and more of upstream issues. When I say upstream, I mean items that we didn't even know were a problem before because of the shared file system that we had and the fact that every process could reach almost everywhere, we didn't think about it. And now we have problems like we can't really use the GitLab import feature. We can't use project templates and a couple of other items there linked as well in that list. We're over 10 minutes as stated in the handbook and we take questions now. Yes, of course. All right, if there are no questions, I think we can then end this functional group update. Thank you everyone for listening. Did you want to mention maybe a little bit of the help that we provided to the production team Maureen or? Yeah, we can mention that. We worked on getting the GCP migration on the road a bit. We had Ian work on setting up the database and we also had Jason look into some of the issues we've been having with some of the notes in the previous release together with a couple of other production team members. All right, great. Well, thanks everyone and have a great day.