 All right, welcome everyone. I think this is the second edition of the package group office hour And we're excited to do this during the hackathon So I guess without further ado, I mean Tim I think I'll just turn things over to you and then let me feel free to share the screen and we can start the conversation Okay, great. Yeah, so thanks everyone for joining and Today I really wanted to talk about a few things just a quick overview of what we're working on now So I'm going to pull up the and share my screen and review the direction page From there. I wanted I have an open issue for the hackathon actually for potential community contributions that range from off issues to contributing new package managers to smaller updates and And then I was planning on going over some recently released features that we have so it might spark some interest in ways to continue to iterate on the GitLab product and Then Nico was actually going to give a short demo of a new feature that we're working on In 12 that will come out in the upcoming milestone and that will continue to iterate on moving forward as well So without further ado, I'll share my screen Okay, and everyone could see that right Okay, so where where I'm at right now, this is every stage at GitLab has a specific section and stage direction page and Ours is under that because package falls under CI CD ours is under the CI CD section and then the package stage I do think it's important to overall state what our goal is for the package stage and for us Our goal is to build a set of features that within three years Will allow our customers or at least 90% of them to store all of their packages and images on GitLab Removing the need to have any external third-party vendor. We want to be a single application for all of the package and container needs and So we have a few in order to accomplish that goal We need to unify the container and package registry to create a seamless experience for building testing and releasing code But a big part of that is actually driving community Contributions, whether that's from our customers or whether that's from the community We need to make you know We have a really aggressive roadmap and timeline and so it's really important that we work with the community to make sure that our product That it that people can contribute and that they want to contribute and that you know They feel like what they're doing is valuable and Then of course we need to perform improve performance and reliability of builds by reducing reliance on external dependencies So that speaks to an initiative a cat the category for the dependency proxy Which I'll talk a little bit more about when we get to some ideas for contribution Okay, so These are our major categories as part of our stage We have the package registry and this is really the different package manager Formats that we support currently we have support for npm Conan maven and coming soon is new get and thanks to doshan Composer which is a big community contribution that we're that we're close to getting through What we're working on now is a combination of adding New package managers, so we're working on adding like we mentioned new get and php Next on the list for the things that we'd like to add is python So we're actually just starting to do the investigation work on pi pi as the package manager for python Another highly requested feature and it's something that we don't really support yet is any Linux distribution So Debbie and an rpm is a big focus that we want to add Probably in the next Probably in q2 of 2020. We'll start to shift our focus to Debbie and an rpm and then For obvious reasons that get labs since we are Ruby shop It's would be really valuable for us to support Ruby gems for dogfooding purposes, but also ruby's a popular language So we want to support that as well There are in the documentation you could see a list of suggested contributions for other package managers, but these are our top priorities and Some other things that we're working on and is you know one I kind of called the big thing for the package registry is new get and and Python or the pi PI Which is next. Okay. Let me go To the container registry Couple of Highlights for the container registry that we're working on now The biggest thing we're working on is improved is lowering the cost of storage on behalf of our customers first and then on behalf of get labs second So one of the problems with the doctor container registry is how garbage collection works. It's pretty slow The open source doctor project hasn't been updated in quite a bit. So we're actually now at get lab We are making updates to that project And we are first optimizing how garbage collection runs. So this includes how the the algorithm Marks and deletes images and blobs and Then separately from that we're also after that we'll be working on online garbage collection because the current system requires The registry being set to read only or down mode which for get lab calm. That's impossible We can't set the container registry to download or to re only look for an extended amount of time But separately we're also attacking the problem from, you know, how do we help administrators? programmatically manage their container registry and that was It's one way that we're doing this is by enabling docker tag expiration and retention policies and this is something that Nika's gonna give us a quick demo in a few minutes on but I think it's it's worth just calling out the big thing We're working on the container registry right now is is lowering the cost of the storage and giving tools to administrators to programmatically manage storage costs and Then the other category I want to specifically call out is the dependency proxy This is a feature that was actually the MVC was launched a few milestones ago It's not been broadly adopted yet because it's dependent on using the Puma web servers and get lab comm uses unicorn for instance, so I Know the infrastructure team is working on rolling out Puma to production One thing that that we're considering is switching the logic to use Workhorse, which is another type of upload system that we have But there are two really important or a couple of really important features that we would like to add for the dependency proxy So one is adding authentication so we can enable using it with private projects This is a great idea for a community contribution And and the other one is the ability to add to delete items from the dependency proxy or just purge it because you could imagine We don't want to get into the same boat We are with the container registry where we see ballooning storage costs and and not have a good answer for for cleaning those up Okay, so I think I'll pause there on the direction updates is there did anybody have any questions on that or Could I think maybe we could switch to Nico quick for a quick demo and then I could dive into some potential options for community contributions after that Okay Nico I'll stop sharing if you want to do a quick demo. Yeah, I'll screen share clearly the Expiration policy is something that run on a temporary basis. So we won't see it running in action. We can check the UI though First of all, I want to clarify that This piece of UI this feature for now is not enabled for older project. So project created before the expression policies released This is to avoid like accidental deletion of tons of tags from projects project that have a lot of tags and images But let's take a look of the UI There is a new section in setting CI CD And it's this little form that allows you to set up several rules to retain and expire tags The form comes pre-compiled with some sensible default. This is actually a project that I'm working on. So it changes It's a little bit change it The meaning on that you can select is like seven days. This means that Tags older than seven days can can be expired when you should run How many tags should we retain for every image from 1 to 100? And then this is a regular expression that is going to match them the project name There is the ability to save and turn it off if it's not necessary anymore As I said, I can't really show it in Working because at least seven days needs to pass for a tag or actually needs to run Iran's I think every midnight Around midnight So there is not more to show But we do have an extended documentation about this and we actually explain All the steps that the algorithm is going to take I think you can already find this on gitlab.com. Clearly the feature is not available before dot point eight. It's Is out Any questions Is there also a way that you can update this outside of the ui like can you use the gitlab api to do this? Yes, absolutely. Um We have documented Two basic usage here and more documentation can be found year so this falls under the project edit api and There is a new Dictionary to pass a new ash and we are allowed to set the same setting as you can see through the ui What what's next for for this? um I guess the next step is going to be on the project level, but you can confirm that probably On the repository level level And i'm just wondering, you know thinking about um people here that might want to contribute Is there any what was some is there any interesting pieces of this that you could think? Oh, this could be a good avenue for people to contribute to this code or or maybe something you've learned Well, um, there is one um, we made some assumption in the ui specifically That we will go with sensible default and drop down, but these are all things that can actually change and become More rich selector most of those can be combined date range selector Or you can end up to pick up a time or or this kind of improvement and This is all stuff that we can work on from now This is regarding ui. Um, maybe regarding the api in the back and somebody else can chime in or if not Like steve maybe and some ideas Yeah, I think along that same level of just more customization and sort of the options that are available Are sort of some of the like, you know easier ways to approach this as some some small improvements that can be made right away And From there like nico is saying our next step is going to be taking a look at this not from the project level But from the repository level so one um a little bit more granular View and I think there will be a lot of opportunities there to also just Be able to increase like some small Pieces of functionality in some ways that might be really helpful Awesome. Thank you. Thank you steve and nico. That was that was really helpful I will um Share my screen again and I put together this issue as per raise recommendation It's something that the runner team is doing and I was definitely interested in it Um, I created sort of a meta issue that just details when are we having open office hours? And by the way, we should probably create a new one of these for next month because we're trying to do these monthly and Get get in front of the community more often and then I just added a bunch of Comments and different threads for potential contributions So the first one is these are bigger projects. So this is Something similar to you know, here we highlight composer which joshin's been working on and I'd love to hear How that's been going and if you have any if you're willing to share any lessons that you've learned and That would be really helpful But I kind of highlighted the top five that we're looking for for contributions right now um, and then also unfortunately the the person that's working on the The the cargo mr and terraform are I don't think that they're here right now But these are in motion as well um Justin, do you feel like saying anything about the composer contribution you've been working on? Well, uh, currently it looks pretty good. So, um gg did A few tests yesterday, I think It's been looking good so far so The only issue he's having at the moment is like, um Composer creates packages by uh tax So he was trying to like set up a new branch and building a new package from there So and that didn't work for him But basically that worked only for the master branch So we have to look into that but I think so far Composer is doing or looks pretty much done at the moment so That's a big step forward. I think Yeah, so it's in maintain it's in maintain a review and just going through some final testing. It sounds like yeah, and well, there's maybe a thing for The instance entry point at the moment um We're currently or gg decided to basically disable the instance entry point for composer because it would pull in all existing packages So it's kind of like a performance issue so A little bit of investigation yesterday to see How we can get around this so we can enable this feature for gitlab.com But for now he's like He said he will only add a checkbox for enabling the instance entry point for like the enterprise version And not for gitlab.com. So gitlab.com will not have Instance entry point That's the plan for now. So but we're working on that later on once it's it is released. Yeah That makes sense. Do you have any suggestions for those watching at home for getting started on a contribution like this and Any learnings or best practices that you picked up along the way along the way well, I would basically When I started there was maven and I think npm So I should have talked more to like the creators of that feature So and basically just focus on reading the docs Reading it once reading it twice wait settle down for a week and read it again Because I think I made a lot of mistakes on the way just by fiddling around and Not reading the docs properly as I was supposed to So yeah That makes sense. And I think we learned, you know, this being one of our first contributions of of this type of Like a new package manager, right? I think we learned as a team what it means to support those community contributions and you know having ggb available for support and you know being able to Help guide those through has been I think has been really helpful. Would you would you agree? Yeah, yeah, it's really helpful. And I think it was may rather doing like the review every now and then which is always been like Big help everyone was just trying to help as much as he could so But in general, I think I should have been like more robust on or just like talking more so People start to listen more because I tended to be like really quiet when I was programming And try to get the future to a certain stage And then I came up with saying like can someone please review so people started like to look at this and When I just kept on pushing like changes like each and every day So let everyone know early what I'm doing at the moment So that would make to me Would have made things like easier for me. I think so everyone can like just be Nice and tell me what I'm doing already wrong. So I don't finish like the wrong writing of the code So that saves me time and maybe gets the feature Done early and not as late as what happened to composer now So, yeah, um, yeah a joshan Dan craft engineering manager nice to Put her name to your face and thank you so much for all the work you've done and contributed It's it's pretty awesome to see you Giving your own time to sort of contribute here and I know that I know that this mr has been open for a while And you have put a lot of energy into it. It's really really appreciated As a group I certainly want to try to engage and assist the community wherever we possibly can And so I'm always interested in feedback if we could be doing a better job with that We've been iterating internally on how we engage the community to try to make sure that we're providing the support necessary for people without trying to take things over or take it out of someone's hands or Try with you know, we're trying to avoid a situation when someone feels like they're not able to own that change that they've created And they really feel like they're part of the solution that ends up being made for the the sort of particular feature set Um, I'm always keen to hear feedback if we can do it better um And in our case what we've done is we've updated our On our handbook page for the package group it's sort of updated and in an ongoing fashion been updating How we engage the community and that's now inclusive of like anything up to like Actually taking over the mr in cases where the community member feels like they're not able to contribute or they would like that level of assistance But it can be really tricky for people because we're really just trying to help So anytime you have feedback for us, please let us know I'm more than happy to iterate on how we do this and what works for different people in different circumstances I don't know if that issue that tim has shared makes sense the office hours issue But if you want to contribute there, I'll put it there would be awesome We also have an issue describing our experience learning how to implement package managers for ourselves Um, and that sort of started by steve and contributed to by david And other members of the team. I mean this could be really helpful to have your feedback in there as well And just to say again, I'll stop my ranting But like just thank you so much like we recognize that this is like Ashing projects for people and it's super awesome to see that from the community. It's really exciting for for me I don't speak for other people, but it's really really cool. Thank you Yeah, I mean I also want to echo. I mean this mr has been going on for for a while and it's amazing You know and that you you kind of kept kept at it and then I think part of the challenges is I mean I mean daniel and tim you've been here for a while But you guys had a pretty lean team for a long time, right? I mean it's it's great to see other like engineers on the package team Uh, so we have people like myra like sort of helping out. So I'm sure that didn't help But yeah, I mean that's that's sort of the reason why we wanted to do these like office hours, right? So we I mean for for new teams like the package team can can better support community members and And and have community members be part of the conversation. So It's not going what what what people have already already said about, you know, um the work that's already been happening Great. I think thanks for all for that. That that was really helpful There is a a list of suggested Contributions that kind of a more full-length included in the issue So if you're interested in any of these other ones like cram for instance for any of our data scientists out there and Puppet is something that I've been hearing a bit of interest from the community on recently so If you're interested in this this list is available in our documentation. You could just google For packages and suggested contributions Okay, so the next section that I have is all about Permissions and access and deploy tokens This these issues are really interesting if you're interested to see how get lab handles permissions and off I think steve is on the call. He's worked through a lot of these issues since he's been here since since may he's gotten to work through Probably some type of example of all of these issues and you know, we see a lot of customer requests for this Some highlights on here are Having, you know, how did the deploy tokens work? So basically having read-only access to the packages api right now deploy tokens are Limited to reading the the code repository or reading the container registry We don't have it. They don't have access to the package registry And so for a lot of people they want deploy tokens for the packages Adding write registry permission to deploy tokens and this could probably be expanded to say write container registry and write packet registry Having global deploy tokens Issues where private tokens with working with the container registry And then another thing is we know as we add support for each package manager or fast follow to whenever those issues are released or ensuring that the CI job token Is unable to work with that specific feature. So for instance, we're about to launch new get and I think it'll be available on dot-com And then next couple of days The next thing we'll want to do after that is make sure that the job token works with that Um, so yeah, oops. I have two repeats on here, but uh, steve Did you want to say anything about working with the off and tokens about anything that that was particularly interesting or exciting that you learned about? When you worked on those features Yeah, um, I think working in the the off section and permission section of the code is really interesting because um, Like with a lot of these changes oftentimes the changes aren't that big But the code the underlying code is probably some of the more complicated code that I've run into At git lab Just because you know authentication and authorization tends to be a little bit more complex Um, so it is a lot of fun to kind of explore and see how it's uh currently working And sort of it's almost like a little bit of a detective work on finding like where does this change need to happen? And then once it all clicks it's kind of Uh, a little bit easier to make the change and test it out. Um, so it is a very different Um area of code versus, you know creating a new package manager or creating a new feature for container Registry, so if if if that's a little bit more of the style that Some people are interested in where they just like to kind of go exploring and understanding Existing code and not necessarily starting completely from scratch. It's a really fun area to get some of that sort of exposure Cool thing. Thanks, baby. Yeah that I've following along those issues It's been fun to see the different use cases that have popped up and different considerations that come into play when you're When you're editing this code because it touches so many different pieces And along with that you do end up getting to also interact with the security team and a few of the other different areas in uh git lab, which can be Really like informative and and I'm pretty fun Yeah, I agree Okay, so the next one is I've noticed for me is uh Has been a great way to contribute to git lab But not being a developer is being able to try and help improving the documentation Something I consistently hear from users and from our customers. They love git lab Actually, this is a quote from a recent customer meeting. I love git lab I don't love your documentation Uh, so this is an area that we can always improve on and that includes doing things like creating templates Or if you have a use case, that's that um is important for you Or that that's working for you like Add a template forward or add it to the documentation. We see a lot of these contributions come through and they're really helpful Um, I put in some examples here for having some Docker templates Um, one thing that I see a consistent number of issues for is How do we support private registries and what documentation exists for that? That would be a great community contribution, but Yeah, if you're looking if you're a first-time contributor and looking for a way to contribute to git lab I think improving the documentation is a is a great way to start And certainly it's a good way to get your feet wet without having to even change any code And to build a little bit on that, um If you're kind of worried that you know, like maybe you want to Make the documentation better by adding some content But you're not really confident in how to properly word it or organize it with the rest of the documentation Um, I wouldn't worry about that. I would create the merge request And then one of the technical writers at git lab will be like super happy to help Um figure out how to actually bring the right voice to it and get it into our documentation in that way Yeah, that's one of the things One of the things I've seen uh projects do to encourage Um, are to enable people who are coming in from the outside Um of the project is they have they have labels or tags for issues that are that are basically like You know, this is a relatively simple. You don't need a lot of context To to actually be able to do this successfully Um, it's you I hear it's low hanging fruit. Um, it's a common one or something of that nature where it's just like You know You don't have to understand the entire system to make the change effectively So I wonder if if that's something we could explore just to An advertise like hey, this is a good way to contribute. It doesn't it's not you're not going to spend weeks spinning up That's I I think that would be really helpful up for at least for me Ray, do we have any label like that? That yeah, I mean we a good label is um Like a good for first time contributors Um, so that label is certainly applicable here and then the only asset I have is I mean for if you create an issue with that label, uh, just You know, it'll be great if you can sort of be Available as a resource for for you know, helping community members sort of get over the first time because I mean I think the first MR is usually The trickiest one because you're learning the processes and learning the tool So if you can sort of you know, if when you add the label just volunteer and say look if anyone wants to work on it I'll be happy to you know help you and work with you. So Um, I mean, I I think that would be a good label to start with But let me know if that you know if that helps Okay, I like the low hanging fruit label I wonder do any objections to using that because it maybe it's like a little bit more than a first time contribution Which would but it's at the same time. It's not as involved. You're not going to spend, you know An extremely long amount of time on it Yeah, or I mean the other thing I asked people to do is when they search for issues like look at weights I mean not everybody assigns a weight to uh to an issue that that people create But you know, I mean added like a weight of like a one or two That's usually a good sign that you know, it's not going to take an enormous amount of Work to sort of get started That's helpful too. Yeah. Yeah, I think the issue maybe we're doing it just by the weight is I've waited an issue one or two but With the with the context of like Knowing how to test and verify that and knowing the container registry. It's in that context Right, it would be a lot higher for someone who who hasn't gone through and knows how to verify the change is correct Yeah, I mean that's I think in general that's a challenge way It started it's somewhat subjective and then there there isn't I mean you need to factor in the context as well um, but Okay, we could Let's uh, I'll take that as an action item to one at least put first time contribution on items that could be uh easier and then to consider maybe Adding another label that would be something like low-hanging fruit or something like that Well, if you have any example projects where you've seen that, um and can call that out that would be really helpful to share Okay, so the next category is jupyter notebooks So this is uh, if you're not familiar jupyter notebooks are um, they're ipython notebooks where you can run code In cells it's common use cases are for data scientists for doing data exploration and analysis The advantage is that you can have one cell that's written in sequel and then you can switch to python or scala in the next cell It's also i'm hearing a lot of recently of people using them as runbooks So being able to specify specifically like how our release is done and there's a bunch of uh I shouldn't say there's a bunch of issues with how uh items are rendered within the jupyter notebook so you can see here that images are not being displayed or Markdown is not being displayed correctly uh, if you're interested in jupyter notebooks, this is a great way to contribute because uh, it seems like These all work within the jupyter project normally. It's just not working within get lab So these could be cases where there's a fairly straightforward solution And again a great way to contribute if you're a user of uh, python notebooks and and want to contribute to get lab The next thing that I wanted to talk about is just the package registry user interface So, you know, if you want to contribute to that there's one example is being able to add a package to a project from the ui And there's an issue here that kind of go goes over that But also, you know, if you're a user of the feature and you're going and you see anything that can be improved I think, you know Just starting a contribution is really useful And actually a lot of our front end improvements and and we've made a lot of them in the past Seven or eight months since uh, nick and nico joined Um have been them basically saying I would like it to work this way like as an npm or maven user I really want to have this functionality And then just creating the issues and working on it. So it's been really useful Okay, a couple a couple of more. Um, so I mentioned the dependency proxy a little bit earlier This is a pretty aggressive one But if you're in the mood for having a big contribution that That touches workhorse, which is the way that git lab will do file uploads Updating the dependency proxy to use workhorse is a probably a big contribution But something that would definitely be valuable and would help get this feature out sooner And then adding authentication support so we can extend the feature to work with private projects And then like I mentioned adding the ability to delete and purge items from the dependency proxy And then there's just a couple of other general improvements that I've added here Extending extending package JSON links to work with the package registry And allowing publish to group For the maven endpoint And then this you know, this is a similar thing. You could do the same thing for conan or nougat You could basically extend to The pull or publish roles to either the group or instance and we have different use cases that we could follow up on that Okay, I'm you have one more thing to share. Just to talk a little bit about Some things that we've done or that are coming soon or that have been done in the past couple of months that are worth highlighting So in 12.8, we have a bunch of new features That are coming out. Actually I should start with let's see here So I mentioned the nougat repository. This is awesome. This is our most highly requested The most upvoted feature for our stage. It has over 150 upvotes Uh We have improved garbage improved performance for the garbage collection algorithm for s3 and we're seeing orders of Orders of magnitude improvement. We're seeing It was for deleting 15 000 blobs. It was taking two hours and now it's taking like 93 seconds So it's some great improvements that are coming coming soon Nico demonstrated the expiration policies that is hope will Likely if not, hopefully make 12.8. I know this is in maintain a review now We have improved performance of the delete api for the container registry front end So in terms of front end performance, this has been one of the most poorly performing controllers on git lab and it's it's resulted in a lot of support issues And so this will be really valuable for our team to to get these out and We've this is actually available in production now Where you can add package metadata to the ui So if you build an npm or maven or conan package using ci you could you could visualize Or it'll show you which pipeline which branch and which commit was responsible for it in the package details page And then a couple of other things. So we I believe this is Out and available now and calm and will be in 12.8 But we resolved an issue with the garbage collection for digital ocean on s3 This was an issue for instances that were using sef storage We added support for npm tags So you can now use the npm disc tag to add tags to packages and there's that's available now on com I'm really excited about this one this delete corrupted manifest it Of common thing that happens in docker is the manifest will get corrupted and We have seen many many issues opened up where people have tried to run garbage collection And then it fails because of these corrupted manifests. So this improvement will actually Ignore them or it'll actually it'll delete the corrupted manifest as part of the garbage collection. So it will no longer error out the garbage collection when it fails and a couple of other things just maybe worth calling out and We talked about using ci job tokens. So we we've made that work with the conan repository. We released the conan repository And then also we've been improving the user interface by doing things like adding snippets to install the package And then overall improving sort of sorting and filtering for the package registry as well Okay, that feels like a lot of just talking so I'm going to stop there And I should say do we have any Questions or comments You know see a couple of other familiar community members like Mateo's Brian and others if you have any questions Let me feel free to jump in or you can even type him on the chat window if you don't want to verbalize it One one thing I guess I'll just say is you know if you're thinking about contributing and you know I think that's just we are here to support you. You know, that's not going to it's a little intimidating at first to say You know to look at the documentation on how to contribute, but you know, we are here to support so I think that We can get through there Yeah, there was a question from Brian. I guess. Yeah Tim guide Yeah, so yeah, Brian saying what are some good issues to get started with? I It depends What you're interested in if you're looking for small issues to get started with I think some of those off issues and like the token Stuff is good because it gives you a really good sense of how to contribute to get lab. There's A lot of examples you could pull from and it'll definitely get a lot of feedback from from us and from the security team as well But maybe does anyone on on the any of our developers have any ideas for a good first contributions I think that like iterating on the in iterating on the api is also a great way to get started Because it's like a if you're well if you're a front end developer, I guess it's a different How would be a different answer? But if you're a back end and you want to contribute to the api It's a pretty good way to do it to just like add a different endpoint or expand the functionality and like right just linked to the issue that has a That has a bunch of different options for the things you can contribute I'll be sure to check out Maybe the deploy tokens Issue it seems to have a lot of popularity and maybe something that could be a great start. Thanks Thanks, and if there's anything That you're not sure about whether or not it is like a You know Something that'll be easy to get started on or seems a little confusing Feel free to to ping anyone Tim any one of the developers on the package team or just give it a package label and and one of us would be Happy to just sort of you know start a discussion and talk through what might be involved and you know Help out as much or as little as as you'd like really Thanks. All right. I think Ray, what do you think should we should we call it? We're about and yeah, I You have nine minutes left, but if there are no other questions Uh, I mean if for people that are watching the recording, uh, I mean if you have any questions, I mean I would encourage you to just uh add questions or comments directly on the issue That we have for the office hour and then I think the other thing you already mentioned We'll try to schedule this like once a month. Um, so hopefully Uh, you know few few weeks before whenever we have this call In march we can open another issue and then start a conversation there Great. Yeah next month. I think we'll be demoing the Composer uh integration. Hopefully there you go Well, thanks everyone for joining. This was uh, really great and and look forward to keeping the conversation going in the issues and and uh, in merge requests All right. Thanks everybody. Thanks. Cheers