 Recording is on and wish everybody a good morning, good afternoon or good night around the world. This is a special edition for GSOC office hour where we will answer questions, solve doubts about two project ideas for the Jenkins Google Summer of Code. And we have several mentors here on the call. Just a brief presentation for myself. I'm John Mark Mason. I'm located in Brussels, Belgium, Europe, and together with Alissa, who is on the West Coast of the United States, we're both org admins. And we have also Bruno, Bruno Vrachten, who is located northern of France. So these are the organizers and we have mentors around the table. So we walk first. Basil, can you just present yourself? Who you are? Where you located? Sir, I'm Basil, and I'm located in Northern California. Okay. Mark. I'm I'm Mark and I'm located in Colorado in the United States, not far from the Rocky Mountains. Great. So morning for you too. And Loggy, where are you? Where are you located? Yeah, hi, I'm Loggy. I'm located in Sri Lanka. Yeah, I should remember it each time. So and how do you pronounce your first name? Loggy. Loggy, yeah. Okay, need a training for that. So welcome. So without further delay, we're going to discuss the first project idea, which has quite an impressive name, so exponential back off and jitter for agent reconnections. I think this is a subject where Basil has good knowledge about he proposed this project idea. So first of all, could you present the project, what it is, why it's important? And then we'll open up for a question for housekeeping. The presentation of the project should not exceed 10 minutes. Sir, sir. Well, as the as a user of the swarm plugin, there's been multiple pull requests over the years to solve this problem, which is that when we starting a controller, all of the agents reconnect at the same time. And this can be a scalability limitation. If too many agents are connecting to the controller simultaneously, the controller could reject some of these connections. And essentially that means that the restart of the controller would would be only partially successful if age if some of the agents were not able to reconnect. So people have reported this with both the swarm agents as well as the standard remoting agents. So swarm is effectively a wrapper round remoting. And the the problem is basically to this is basically a scalability problem, right? And the idea behind the solution would be to introduce jitter so that these agents aren't all trying to reconnect at the same time. The main challenges for this project are number one, there's two different implementations of this. So the code is not already in a clean state. There's essentially the duplicate logic in the standard remoting framework as well as in swarm. So so that's already going to be a challenge. Then the other challenge with this project is demonstrating that this jitter is actually having the desired effect on a busy controller. So it's one thing to implement an algorithm that's defined on paper. But the other question is, is this actually beneficial to users? And the only way to really determine that is to do some kind of scalability testing and to see, okay, well, how can we reproduce this scenario where the controller is being overwhelmed with agent reconnections? And once we can, we produce that scenario, is this addition of jitter actually making a difference? That is, is it improving the success rate of reconnections? So that's really this project in a nutshell. So we have already just for my general culture. So does the remoting part have already the exponential back off? That means that every time an additional time is added to the retry? Does it already have that? I don't remember. I think there's some, there's definitely some stuff missing from remoting that is present in the swarm version and and vice versa, maybe. In other words, these two implementations are not at feature parity with each other. So, okay, that's part of this project is to get these two different implementations up to feature parity. And I can't, I can't remember what remoting is missing. It's definitely missing something. Mark raised his hand. Basil, could you, could you describe for those who might not be familiar with what the term jitter means in this case, give a little more, a little more depth on it? I had to go do some, some looking to see, okay, what's jitter in this in the context, etc. Yeah, jitter in this context means adding a randomized time delay to spread out, to spread out load. So in the project proposal, there's a link to an AWS block post that has a nice diagram of this, which kind of shows that adding jitter will kind of increase the, what's the best way to describe this? It'll adding jitter will, will ensure that there aren't these big columns of dots in the same time span. And if you, I mean, I just, just go read the AWS block post, but they have a diagram of this where they show how essentially adding jitter distributes the load in a different way, so that, so that the server isn't overwhelmed within any particular time period. Okay, so it's just jitter is a random, random startup delay to the initial connection and that, that by doing randomness when a thousand things connect, we don't get 1000 of them in the same exact second. Right. The idea is spread it around. Okay, thank you. Thanks for the clarification. Sorry, I should have read the paper. No, it's sorry, it's spreading it around within a limited boundary. Okay. With a limited or unlimited limit, a limited boundary. Right, say between so and so many seconds, choose where you're going to start. Yeah, very good. Retrying is a retrying is a deceivingly complex topic. Because if you if you look at there's like a one of my favorite libraries for retrying is tenacity, which is a Python library. And they have a really good documentation. But if you look at the tenacity documentation, you can see that there's all these different ways that you can retry things. And there's not some of them are better in some use cases than others. But it's not obvious to anyone who's who's new to retrying that there's this much complexity behind the topic. But you know, for example, you can, you can sleep fixed amount of time between tries, you can, you can increase the amount of sleeping between retries, which is called exponential backup, which is done by TCP, you can add jitter, you could there's all these things that you can do. And they make sense in some use cases, but not others. So the use case that we're talking about is the thundering herd problem. And so that that is the case that we're discussing with controllers being flooded with agent reconnections. And in that use case, the identified solution of industry wide is to use exponential backup and jitter. So that's that's the medicine for this problem. That's medicine. I think one of the important dimension about that is to demonstrate that it has the desired effect. So the the the testing. So there are the usual testing, which is unit test integration testing. But here we really need to do an end to end test of that feature. Yeah, the testing is probably the hardest part of this actually, because I mean, it's it's relatively straightforward to refactor duplicate code and to add a loop that goes and maybe not trivial, but it's something that's well, conceptually, it's not there's nothing complicated about adding a for loop, you know, to some code. In practice, it's going to be hard because this code is a mess, and it's it's extremely messy code. So and then, like I said, there's two versions that fit as well. So in practice, it's like one of the most difficult for loops that you're going to be adding. But but the testing is really the hard part here because how do you really demonstrate that this is making a difference? You know, the there's it's not easy to set up these kind of scalability environments, even if you have a lot of computers, but especially if you don't have a lot of computers, if you're dealing with a small number of machines, maybe even one machine, you know, it would be be fairly challenging to kind of replicate this scalability limitation where you're seeing agents being dropped. So that that that's going to be a difficult part of this. And I'm not sure. Yeah, so in your previous life, you experienced these herd effects, which were sizes. Yeah, I was running a big Jenkins deployment at a previous employer where I experienced this firsthand. And I've seen other users on people have submitted the pull requests to swarm to add jitter. And and I've even seen social media posts from Jenkins users talking about this problem. So, you know, one idea might be to try to, you know, connect with one of these users and try to get them to deploy, you know, a test version or something like that. But yeah, that that isn't certain that we be able to find someone like that or that they'd be willing to to deploy to their production environment, some of this code. And what was the number of agents we were talking about? Are we talking about 10,000? Well, my case, it was hundreds. I'm not sure what I'm not sure what the Netflix case is. I think it's hundreds to thousands. And there were also some people in the swarm plug-in filing pull requests. And I tried to ask them for more information and they never got any. So. And excuse my curiosity, but and what is the the the effect of the herds? Yeah, the effect is if an agent can't reconnect, then that job is still running on the agent, but the controller has lost communication. So, when the controller is coming back up after the restart, you won't be able to get any status updates until it reconnects with the agent. So, you know, for I think I'm not sure if remoting does retries at all right now, which is I think the scary part. I think remoting by itself might might not be doing retries. I don't I don't really use remoting directly because I like I like swarm personally. But it's not really the standard way of doing it. Swarm definitely does retrying, which is, you know, critical in this kind of scenario. But then again, you know, if you're if all the agents are retrying at the exact same time, then that could create a worse problem, which is that the controller just gets completely hammered with connections at the same instant, you know, every 10 seconds or every 30 seconds, which is why that which is why jitter and backoff are both needed. But yeah, basically the effect is that the that the controller could be made inaccessible if it's receiving so many requests at the same time, or the jobs wouldn't show status updates until that reconnection takes place. Last question before opening the questions. I'm keeping the clock in my eye. This project can be huge. So there's a lot of work possibilities. There are a lot of holes you can start jumping rabbit holes that you can go. Can it be cut in different phases or different and say have ideas for another edition of Google summer code or do you see a strategy like that? Yeah, absolutely. I mean, there's there's no harm at all in just simply consolidating the duplicate code that exists today and not included and not having any regressions, right? That is a change that could easily be accepted without controversy. If the if the existing code is de-duplicated into some sort of common function or common class or whatever, the challenge there is that a command line interface needs to be added to express some of these ideas, right? To to express the the backoff period or the amount of time between tries, the number of tries, etc. So in designing CLIs is always tricky because it's effectively a contract that you can't break once you once people start using these command line options. So a lot of effort needs to be, in other words, there's this is challenge where even if you're only implementing a subset of this you need to design the CLI for the whole thing before you can implement a small part of it because designing a CLI is one of those things where it's effectively an API that people commit to. So you can't you can't just do it incrementally. You've got to design the whole API even if you're only implementing part of that API up front. So I think that's that's the only that's the biggest impediment to breaking this up into incremental tasks is that the CLI really needs to be thought out this one completely before anything is implemented even if even if the implementation is only partial it's easy it's easy to leave out CLI commands or subcommands but it's much harder after you've already implemented a CLI to go back and say oh well you know actually this I need to rename this argument or I need to add some some mandatory flags to this other like you know it's a lot harder to do that. There's also another interesting work package is being the how to test it and there might be some thoughts designing and tooling. Here I will stop this presentation part I would just would like to give the word to to log E. If you have additional things you would like to share or questions or or points don't forget to unmute you. Yeah so I would add upon that some points so this project is very vast so the Remoting module is very integrated and there are a lot of dependencies at the plugin as well so there will be very challenging tasks that when we try to push these code changes to this upstream repository so when it comes to the production level of these implementations then we need a very hard testing for this and students can do their own research because this kind of like we can we can use the exponential for now like that's a very industrial standard for this retry mechanism even in my company XYZ so we are using the exponential retry for even for the microservice intercommunications so exponential good choice but students can do their research own research and they can propose what whatever it works we can implement that yeah but it's a very vast project and and also interesting projects so that's why I joined some inter great so to take the analogy again a big mountain okay who wanted to jump in oh okay so it was not the fact from from my side open up the question to the rest of the audience Nethra Vandit or hello everybody I have some questions related to how the unification of Jenkins remoting and so on plugin is to be implemented like currently remoting waits for 10 seconds after every after every attempt to connect to the controller currently it waits for 10 seconds so and the swamp again has has a linear linear back of linear retry algorithm and exponential back of exponential retry algorithm so do we have and it all and so on plugin already has the CLI function where we can override the default exponential back of algorithm so first do we have to do we have to implement the things that are already in so on plugin into the remoting module and then unify it yeah that's exactly right and and the swarm CLI is missing jitter yeah so altogether the the first step would be to design and I would not consider the swarm CLI to be a work of art to be emulated it's evolved haphazardly and the standard for introducing a CLI change to remoting is a lot higher than the standard for or I'd say that yeah swarm has just evolved in a very messy way over the years but like the main the main challenge is coming up with a good a good CLI and I actually wouldn't be afraid to break swarm consumers because I don't think too many of them are using this or even if they are it could be announced you know ahead of time but the key is to design a CLI that does make sense and is easy to use that includes all the features that swarm has and jitter which swarm does not yet have and to really find a way to express that to the use because I think right now in swarm the exponential backup it's a very strange CLI it's it's not an intuitive way of expressing the arguments and the the the intervals it's basically just there as a historical accident so that's where I would start is really figuring out how do we express these options in an intuitive way and then from there how do we retrofit these existing implementations to align with that intuitive CLI which would only exist in in theory and then in practice we can slowly evolve the existing implementations to fit it's going to be easier for remoting because there isn't any support for this so it's a clean slate but that's both good and bad in the sense that it only gives us one chance to get CLI right and then for swarm there's a little bit of so you know it's swarm if we need to change the CLI to really fit this ideal that we could do something like have a transition period or something where these old options keep working but we deprecate them and try to move to the the options that we think make sense so designing a good CLI designing a an intuitive CLI that can be used by a lot of people without making them think too hard about what options they need I think that's a big part of this thank you Ben so are there other questions or yeah yeah I have some more questions sorry for taking so much time oh go ahead my question is about the stress testing framework I searched the web and I found about Apache Jmeter like we can use that to mark something like that where 10 or 20 agent we can we can mark 10 or 20 agents using that I I got I got to read that from some blogs on the web but it's not the optimal solution but it's it's something that we can use to first test testing on remoting and the CLI part I think because it is a one chance to create the CLI if I if right now I try to use remoting from the CLI it it says you are using it from the console use it interactively so currently it has something like of a CLI implemented but but it does not work it just says use it interactively okay so like does the remoting already have it because the code is really complex just like you said that is something I got from the high level overview okay so Vandit did you have a specific question so I think you you raised a few points about the the testing framework so you explored a few ideas yeah Apache Jmeter and having read your your initial draft so I think there is is an interesting topic to to research although there's choices that will have to be made one of the things that you can start and I'm I'm talking to the audience here is to define so to to know how do we know that we have the problem right so this is the first question and the first research item that can be part of GSOC so you don't need to have it but state the problem there so how can you say there we there we are now we have the problem then the second part is to think on how can I increase the load or create a load where I can demonstrate that a problem is caused by this retry a load and and this would be already I believe looking at at basil already one big step forward in starting to solve the problem and and then the the other steps are so I think seeing from the reaction from basil I think I'm I'm on solid ground there say now I need to to look at a clock yeah the question to be accepted to be accepted ultimately our bar for except well by accepted I don't mean from a GSOC point of view but for for any code change to remoting to be merged our standard would our standard of approval would be for it to be demonstrated to solve some sort of problem and to not cause regressions and the demonstration so the demonstration that it's solving some sort of problem would either come from the results of a stress test framework or from an end user who's experiencing the problem trying out the build and reporting success and that's a lot harder because end users are they're not very good at trying out beta code from us and even if even if they're willing to they are not always they're not always good about reporting back the results or sometimes their results are questionable and you wonder if they really deployed it correctly etc etc so when you're not the one doing the testing yourself it's a lot harder to convince yourself that the change has had the desired effect it's a lot easier if you're also the one doing the testing so that's why I'm really hesitant to rely on other community members to do testing because it's very hit or miss with them sometimes they're very responsive and test your build right away and other times other times it just hits at that end so yeah I mean first for us to first to accept a PR to remoting we'd want to see some evidence that it's having a beneficial effect and a real world use case not just a not just a a mock test so whether that's whether it's a local test or an end user like I said we generally wouldn't accept a pull request unless we can we can tell that this is better than before and it's not causing a regression yeah so I read from that that starting with test and digging into the test framework is a first good step for that project yeah and you know like logie mentioned you know maybe jitter isn't the best solution to the problem you know or maybe there are other solutions that you could experiment with and choose which one has the best effect if you've got a way to increase load like that so I like that here I'm I'm sorry we'll pause this discussion we'll try to keep 10 minutes at the end will not exceed the hour promised because these are interesting questions and interesting discussions but I'll be so we'll keep 10 minutes at the end for questions I'd like to cover the second project so the GitLab plugin modernization I think mark you have things to tell us or Basel who would like to talk about that project I think Basel is more credible I'm happy to talk about it if you'd like but Basel you want to talk about it at a high do you want to introduce it at a high level and then I can talk about some of the challenges with jersey and rest easy oh yes yes yes oh and that's okay Basel has just noted a sort of awkward mistake I made so yes let's highlight that first so at the high level the GitLab plugin is important to Jenkins consumers who are using GitLab as their scm system it allows them to do all sorts of interesting and useful things updates of of merge request comments status changes etc and so the GitLab plugin is quite useful however the GitLab plugin is outdated in many of the things that it does and much of its implementation and there are specific things that need to be improved in that implementation now it's important that we have a strong integration with GitLab just like we have a strong integration with GitHub and BitBucket scm providers matter deeply to Jenkins users so we want the GitLab plugin to be very strong and right now it's it's not had an active maintainer for a while it's not had if Chris Stern just barely started on it again and we're we're deeply grateful for that but there's a lot to be done to modernize this thing I think that's all I wanted to say Basel do you want to take the next step which is more more relevant technical detail yeah the GitLab plugin is communicating with a given GitLab server through its REST API so GitLab has a REST API I think it's I can't remember if it's JSON based I think it's JSON based and essentially anything that's communicating with it goes through that API including the GitLab Jenkins plugin and we've had this fairly constant thorn in our side for the last few years that I've noticed which is the way that the Jenkins GitLab plugin communicates through this REST API it's using a very old library called REST easy and not only that but it's also using a very old version of it yeah um and this this library has been a bit of a challenge to maintain it's as far as I know it's the only thing in Jenkins land in the Jenkins ecosystem that is using this particular REST easy library and this library is not very easy to work with but it's also just very fragile and there's there's something to be gained by being on a more common path with the rest of the universe right we're we're effectively I don't know if we're the only people in the world using REST easy to communicate with GitLab but we very well maybe the only people on the planet doing it this particular way and as a result we're subject to a lot of regressions and problems that simply don't happen to other people because other people are using a more common path that we're that we're not on um in particular the the standard way of communicating with GitLab from Java is to use the um there's a there's an official or a semi-official Java library um I forget what it's called now but um I'll look it up GitLab for J um so that that's a very nice library that um gives you these Java classes and you can invoke methods on them and it will figure out in the back end how to translate that Java method call into a JSON request for you but that that GitLab for J library is that's that's the common path that the rest of the universe is using to communicate with GitLab we we've actually already bundled it as a Jenkins library plugin and it's being used by the GitLab branch source plugin for Jenkins it's being used by the GitLab authentication plugin for Jenkins so this this is kind of the standard code path that we'd like to follow uniformly in all Jenkins plugins including the the GitLab Jenkins plugin so the the idea behind this project would be to migrate callers from rest easy to the GitLab for J library and what that would accomplish would be that we'd be following a more standard deployment paradigm we likely wouldn't run into all of the regressions and bugs that we encounter with rest easy we just encountered a new one this week in fact so that's really the project in a nutshell is bringing us into a more standard deployment pattern by eliminating the use of a very old library in favor of the more commonly used GitLab for J library so so the buzzle buzzle's comment changing from one library to another my brain shows big signs that say there's probably work involved there and it's we don't want to regress so there's testing involved there's familiarization with hey how do you use this thing there's probably interactive exploration needed very likely that additional automated tests will need to be added in doing this because probably discover oh whoops here's a piece of code that called rest easy had no interactive or no automated tests those kinds of things and major service to the Jenkins community doing it yeah I can feel that so interesting work useful work and substantial work yeah as it turns out this week we just we just upgraded Jersey and Jersey broke compatibility with rest easy so I filed there's now an issue filed with Jersey to restore resty but I could tell that the person who was responding to the issue was like oh this is very interesting using Jersey with rest easy and I think they used the phrase very interesting and I was like in the British sense of the word yeah that's one way that's one way of the hell right right interesting in the sense of completely dumbfounded what were you thinking why are you still why are you still doing this exactly exactly yeah I can envision but yeah this this is just it's just a thorn in our side for a long time this it made sense I assume when this was originally developed but it's just no longer a viable library to use good okay are there questions from the audience about yes so I have a question go ahead it can be a silly question but I just wanted to know what are these corner cases which should be handled during this I didn't understand those authenticated proxy servers unauthenticated proxy servers all those corner cases which this project is talking about yeah yeah any kind of any kind of library that does HTTP calls is deceivingly simple because you might think oh well you know I'm just calling a REST API from an HTTP client and that's usually very straightforward but there's all there's always tricky cases that go along with HTTP clients for example proxy servers a lot of companies that use Jenkins require all communication to go through a proxy server and what that means is that the HTTP client library needs to be configured to use the proxy if it's been defined by the user so we have we have a place in the Jenkins UI where you can specify a proxy server and from there that proxy server needs to be plumbed through into in the in each Jenkins plugin to whatever libraries are performing HTTP requests and it's not just it's not just the presence of a proxy server some of these proxy servers also require authentication and some of them you know there's also like it gets even more complicated because there's HTTPS proxies that have TLS certificates that need to be valid so it can become fairly challenging to test some of these cases and you usually don't find out about it unless you break it and then some test some user of Jenkins is like oh well I use Jenkins with a TLS proxy with and TLM authentication and that stopped working when I upgraded we've been we've been there I know the one who invented corporate proxies should burn in hell because it's been a nightmare I got a lot of gray hairs because of that and Bruno lost his yeah so because we we already know proactively that without any explicit testing that would probably break so there's a need to do explicit testing of some of these scenarios and you can find you know Docker images of these proxy servers I've I've tried a few of them none of them are particularly great but I've you know there's ways that you can set up a local proxy and test communication with it and and such but it's it's usually an extra step of testing that would not be intuitive to someone who's just using this using an HTTP client and in the usual way it's always it's always an extra level of thinking and testing to support this that's right did that answer your question yes yeah somewhat I guess I'll understand it better when I start actually understanding the project in people's way you have other questions yeah so it's related to the proposal part so in the proposal part how how do I elaborate these steps involved in this project like so like how do I explain it there so did you already pull the proposal template did you look at it yes there are some some guidance there so in the beginning you expose so you do a summary and then you expose what is a problem that you're going to solve with that project and then you're going to describe the strategy so what you're going you're going to do and so and this you can use whatever technique you want to explain some drawings some steps so there there are various ways and you can try them out and you submit your document and say do people I'm very good guinea pig for that because I don't know the details of the projects if I don't understand it quickly then others will have problems so okay try to explain it to the good way is you have a friend at university or so and you have just 20 minutes to explain to him what you're going to do okay and just use that and there is also a part where you're going to explain how you're going to split your time okay during the summer so I'm going to do first that I'm going to do that and this will allow us to compare with other proposals and say okay this person has a sensible plan has understood the problem okay yeah go start do a first draft the first one is like going into cold water it's it's it's uneasy it's not not fun but start writing something and then you ask people for the opinion does it make sense okay yeah I know it's it's scary but here we're here to help you and and we we want you to understand learn from the process yes yeah got it did you understand the problem or are the other details I understood what what mark and basal explained so I'm even tempted to start looking into it which you shouldn't but it's other things you would like to know yeah I have understood the most important stuffs the only part which is to give me is the testing part like should we like do tests is often to make sure the process is like going well integration test unit test and all those stuff yes so believe an old man like me and so what has not been tested will fail so testing is important testing should be automatic so check everything mark has also opinions on that one so I'll stop so you're in your comment to hey automated testing absolutely and run the run the automated tests run them as a way to get started watch how they behave on your system maybe run them in a debugger and watch how they how they execute certainly look at the integration test that are in that plugin if there are and see how they operate don't forget and this is the crucial one that I tended to forget I had a bias towards test automation and I sometimes failed to do the interactive testing that would expose bugs that were very obvious to me and not very obvious from the user interface but not checked by automation so it's not just automated unit tests and automated integration tests there's also an element where the contributor here must plan to do interactive exploration and testing interactively to be sure that it's well behaved so mark the the plugin has integration tests that use wire mock and wire mock is a library that effectively it it's a box HTTP layer where it expects a certain HTTP request and will provide a canned response in lieu of a real get lab server that would be that would be doing real things in the in the back end so some challenges I could foresee are the input to the to wire mock i.e. the expected requests those might change those might change slightly when you're switching from rest easy to get labbed for J without necessarily being wrong it just might be like putting arguments in a different order or putting like new lines after each JSON field or or something like that you know so the the expected input to the tests might change but not necessarily in a wrong way so the test might need to be updated just to adapt to that new style of input and like I said it's all it's all mock based so it really has no bearing on whether it's going to work in production or not like I said I don't think wire mock is going to test anything to do with proxy servers or anything like that that's all something that can only be tested interactively well and and whoever implemented the initial definition of those mocks probably did it based on the git lab version at that time and git lab has certainly evolved right get lab the plugin has existed for a long time thankfully thankfully git lab has a pretty stable version dpi I'm not sure what version we're using through rest easy but I think it's fairly recent that would be another thing to look into though is whether I actually don't remember whether git lab for j is using the same version of the api as what we're currently using with rest easy I think it's probably the same version but they might be our rest easy logic might be using an older version in which case if that's true then upgrading to get lab for j would not only be using a more modern library but it would also be using a more modern api endpoint if the git lab server itself and that could that could require the tests to be further that would require even more updates to the tests if they're if they're mocking an older version of the api so as for the other projects testing understanding what we're testing are the tests correct is the first important step to get yeah I tried to get I wouldn't even bother trying to get the tests working initially I I just go straight for testing on a real server and then because a lot of the challenges in getting the automated test suite working are going to be merely technical rather than actually going like fixing real bugs like ideally your test suite is going to help you find real bugs but in this case it's going to be mostly a hindrance the test suite is going to be mostly just more code that needs to be adapted to this new library rather than a signal to whether your production code works or not unfortunately so I think that's safe to assume then that it's okay to delete tests if you find that it's not helping you that there isn't any religious or okay some it's somewhat okay I mean but also even even when they're only mocks they are still useful in terms of exercising the java code so ideally we wouldn't delete them unless we have unless it's prohibitively difficult to adapt the mock right okay good next round do you have other questions you seem very interested about that go ahead okay so as discussed before so if there is any version difference between REST API and the GitLab 4j API then it would take more longer than the expected that's what they meant right yeah there would be more more mocks that need to be adapted if if the if the existing mocks are using let's say GitLab API v3 or something and GitLab 4j is using GitLab API v4 then that would require more changes to the mocks I mean that's a very easy question answer we could just go and look at it look at the code and see yeah okay yeah I guess so don't succeed we have eight minutes to go somebody who didn't speak yet is Smith if I pronounce it correctly do you have questions or hi hi everyone no I don't have right now like I I'm interested in exponential back-off and jitter project I started looking but I have not yet researched a lot because I have a job simultaneously so I've got much time so yeah I'll pick up the pace and okay so if you have a question or a doubt in the next four minutes or five minutes you can't shoot otherwise if somebody else has a question I have another question maybe the last one I wanted to confirm the order of things that should be done for the exponential back-off and jitter project first I should should I work for should I make the CLI then stress testing framework and then exponential back-off and jitter in remoting and jitter in swarm plugin should this be the order of things the challenge with that order of things is that without a way to demonstrate the effectiveness of the change it would be unlikely that it would be it would be unlikely that it would be that a pull request to remoting would be merged if it's if that pull if a pull request to remoting is adding a jitter feature that has not been demonstrated to make any difference and it's creating new API surface area then it would be unlikely to to be merged so I mean it would it would be fine to merge a remoting pull request that just that just unifies the CLI with swarm let's say but in order in order for that to be designed the entire CLI needs to be thought through even if only part of it is implemented so yeah order is very tricky with this project I mean ideally the ideally the best way to start would be to find a way to reproduce this scalability problem and then to kind of iterate from there to find solutions to it and then from there even if even if no code change gets accepted into remoting just having a way to reproduce this scalability bottleneck would be a valuable contribution in and of itself I have the same feeling so the stress testing framework should do that so I can demonstrate if the changes I am proposing will help and will help in making remoting better yeah and I mean looking at order of operations from a dependency point of view a major design and testing dependency is having a way to reproduce this problem and to see if various solutions are effective or not and then from an implementation point of view the first task that I the first task from an order of operations point of view is designing a CLI that is intuitive and meets people's expectations I would I would not look at the swarm CLI as a good example that's more a good example of what not to do but even because even if only part of that CLI gets implemented even if even if you only implement retries and you don't do back off you don't do jitter it's critical to design the CLI to it's critical to design the jitter and back off parts of the CLI because those arguments could affect the naming of the arguments for the parts that you are implementing so API design has to be holistic for it for an API to be intuitive so I would really start with I'll probably start with some kind of performance testing or stress testing framework or methodology from a design point of view and then from an implementation point of view I'd start with the CLI design even if even if only part of that design would get implemented yeah I'll write back down those my questions we're three minutes from the end if we have a short question we can go ahead otherwise go ahead Nessa can I just get a small sum up of like from where should I exactly start and like how should I go through the project if a guidance like can be provided so we're talking about the GitLab improvement yes yeah GitLab plugin modernization so if I'm going to venture venture how I would approach it Nethra so the improve a plugin tutorial on jenkins.io guides on some things that you can specifically do to improve that plugin look through that that through a improve a plugin look through that tutorial see which of those could be applied to the GitLab plugin if they've all already been applied great okay go ahead yeah you're talking about the schedule build modernization right which is that the tutorial well so that's what's mentioned in the video but those same techniques can be applied to any plugin in the GitLab plugin I believe may even still have some remaining items in it that you could you could begin modernization using those small and simple steps okay there's no real there's no real algorithm for how to implement a project like this you just got to rip out the old code and put in the new version and it won't even compile right away and you have to get it to compile it's going to be kind of frustrating and then once it compiles it's not going to work you have then you have to get it to work and that's not going to be that's also going to be frustrating and will require you basically debugging some library that you've never basically debugging some code that you didn't write in order to to integrate with it and then once after all of this frustration you think that you might you might be done but then you have to deal with proxy servers and authentication and all of these other insane you run away mainly complicated edge cases so I mean that's that's essentially how the project would go just rip out the old library now won't compile at all then add the new library still not compiling but slowly convert every method call from the old library to the new and eventually after doing that long enough it'll compile but still not work and then from there get it to actually work in production then get the tests to work tests probably need to be adapted the tests are going to be like I said earlier they'll be almost useless because they won't actually help you catch problems in production they're going to be more of a liability and that they need to be updated so that's I mean that's why this project is is something that has languished for so many years or months because it's just not the most it's just not the most fun thing to do but unfortunately software maintenance is full of tasks like this that are critical to the health the long-term health of and projects like Jenkins that have been around for 15 years tend to accumulate a lot of these types of things because the longer that your software is used the more the world around it is changing and your software needs to I mean in the 1980s you would never need to change software because the internet didn't exist so you could just keep it on some old library forever but now now that your software is communicating with other software on the internet it's always got to evolve and adapt and that's more true for 15 year old software like Jenkins than it is for some newer projects so it's part of the blessing and the curse of being an older and popular software project so you end up with a lot of stuff like this that needs to be adapted over time but it can also be a rewarding challenge and I find this kind of work rewarding myself because it's got a high impact on a lot of people and it also teaches you how to be good at reading code which is in many ways harder than writing code and it's a valuable skill to learn I've definitely had a lot of benefit in my career from being able to read other people's code and understand it and that's something you're going to have to do a lot of in this project so it's not all pain and laborious work it's there's also a lot to be learned and a lot of valuable real-world maintenance experience to be gained and one more small doubt this GitLab plugin has like around 57,000 installs so does it anyway affect the project? What do you mean with that? So if you do a mistake that you're going to make a lot of people angry? Yeah Don't pass a regression Now, Nathara, I'm going to tell you don't be afraid of that it's if I take the analogy of climbing a mountain if you're doing your work seriously and you don't stretch too much take unnecessary risks and people are there to help you Okay it will go well so this is not something that should refrain you to tackle it okay it's a steep hill and there are consequences but there are also big rewards of that because you're going to help a lot of people and the coaching in this project will just make sure like Basil stressed and saying you need to demonstrate that there are no so not you but the project needs to demonstrate that there are no regressions that it works as advertised and so these are securities as you build but don't be scared by the number of people Mark, go ahead and you're you have a safety net the lead mentor and the other mentors will do their best to give you feedback that helps you understand if you're making a mistake now I have the fear you have and we should learn to live with that fear because it's a good it's a good and motivating fear we shouldn't let it paralyze us but rather we admit what I'm doing matters and it matters to 50,000 installations and that is a great thing to matter to that many people and you've got the safety net of the mentors there to help you but you're your own best safety net right tests perform interactive tests check that it behaves the way you expect use a debugger to watch it run to be sure you know what's going on all those things are very healthy responses to the fear that you described yeah I mean the first thing I would do is step through in a debugger the whole code path through rest easy like from when it from when it's making its initial high-level call going into rest easy going into get lab it's basically getting familiar with with what it's doing in the old version and then that way when the new version doesn't work you can start stepping through it and you'll have in your mind a pattern of what it should be doing and then you can debug the new version and see where it's going off the rails because even what even after it compiles it's not going to work right away it's going to it's going to need some debugging but yeah unfortunately the automated tests are going to be a more of a liability than a than a help in this project right okay here we're five minutes over budget so I'm sorry to have to interrupt because it's late in the eastern hemisphere thank you for staying that that late thank you thank you very much for the mentors that joined the call and for taking the time and it was a very very interesting presentation for the contributor candidates so start working on a proposal start to put something on paper start interacting you need to start somewhere and so that we can help you to to improve and and get ready to ask questions on on Gitter channel or better on community.jankins.io and try to be specific in your questions the very general questions tend to not catch attention of people so well where should I start I want to do it but what do I need to do oh no make your homework but I think the people here that that were around the table and I have done that so good thank you very much everybody have a nice rest of the day thank you very much for your involvement and don't forget we have weekly office hours where you can join ask your question exchange we'll try to have mentors regularly also on the call I'll be there so at least I can be the the relay for your questions thank you everybody bye bye thanks Jan-Marc thank you bye