 Welcome. This is the Get Cash Maintenance Project. It's May the 27th, 2022. Topics for today. So, Hrushakesh, are there questions that you have that we want to be sure we address? What topics are of concern for you? I'm still concerned about the architecture itself, you know. Like, how are we going to proceed? Okay. And so let's talk some more about that. So I think you had said that you're okay with the idea that cash maintenance happens in the background, right? So it probably runs on a separate thread. Async, yes. Async inside the Jenkins, in the Jenkins controller, running a separate process, get process to do the work. I'm thinking of a queue so that I put the task into the queue and then you use another thread to dequeue everything from that queue to run the maintenance task. And I think that makes sense. And I believe that there is a concept of queuing inside Jenkins that you can use. So use a queue to, and the concept there would be to keep basically a relatively few maintenance tasks running concurrently. So we don't want to overwhelm the thing with too many maintenance tasks. Basically I think a maximum of five maintenance tasks would be put if all of them are set at the same time, you know. Okay. So I have a question here. We want to use a queue. As far as I understand queues are used when you want to decouple two concepts. You don't want to have a tight coupling between two things. So you use a queue so that you push something and then there's a producer who would push something and there's a consumer who would then take it whenever they have the resources at the time. It's in their control. So, since we know that these tasks that we're running are going to be scheduled at some frequency that we've configured ourselves. Is there a way, is there a reason or is there a possibility that our initial let's say the first job or processor hasn't finished and there would be other processes that have started happening. And that is why we need something like a queue to make sure that that doesn't happen and we're managing it. We want a queue for that. I am not clear why we would use a queue here. I thought of a queue because we have discussed in the meet that we want to run the maintenance tasks sequentially. So assume a queue which has two or three maintenance tasks, like a prefetch, a commit graph and a GC for example. Whenever I check whether the queue is empty or not, if the queue isn't empty, I dequeue the first element and then I run all the maintenance tasks. I run that maintenance task on all the git caches and then after that's initial, then I continue for the next maintenance task. But this thing happens only when all the maintenance tasks are scheduled at the same time, like same cron schedule, like every minute or every hour. That time only this case happens. Otherwise, if it's for different cron syntax, that time I think only one maintenance task would be pushed into the queue. Still my question is that, like Mark has written, we will do control resource consumption. So if we were not sure what would be the amount of time a process takes or we're not sure about the incoming processes that are going to come while my existing process is running on the controller, then we would want to use a queue which would make sure that we don't overload the controller. But if we're, for an example, if I know that I have exposed an endpoint that is going to be, we're going to push, let's say, 10,000 messages and that is how you've built your controller to consume 10,000 messages and process them. Let's say I push a million messages within a second. So you need a queue or something like that to not overload your system. But when you know that the processes that are going to run are going to run in a finite time and you yourself are the originator of the next set of tasks that are going to come there. Then why would you need something like that? Well, but you made an assertion there, Rishabh, that I think is not precise, right? In that the maintenance tasks we actually don't know their duration. Get GC of the Linux kernel could be minutes. And in fact, it could be much longer than that if the computer is under memory pressure or relatively lower performance. And whereas fetch of most repositories is seconds or take the other end, fetch of the Linux kernel over a slow link could be an hour or more. It's a two gigabyte repository and so if we were trying to, if we had just the beginnings, the first few commits and we did an incremental fetch and it's now retrieving two gigabytes of data, that is a very unpredictable duration. Yes, that's true. So that means then I am not sure how much time it would take and that is why I don't want to. Yeah, so I want to keep them separate and I have a layer in between like a cube, which would make sure that I consume a task only when the first one or the previous one is complete. At least that was my mental model. Rushakesh, does that fit for you or what were you envisioning? This is what I was thinking, but you know that again I was thinking of an edge case where, you know, assume like I do a GC maintenance task on all the maintenance, you know, get me get caches when I do that. Okay. And assume there's another maintenance task which is scheduled hourly. So, you know, and if the GC task and another maintenance task like a pre-fetch both have been set at the same time, so two maintenance tasks would enter the queue. And assume the garbage collection takes another one hour. Okay. If there are too many repositories on the Jenkins controller, then more tasks would be added to the queue and the previous maintenance tasks also wouldn't have been executed. I think that would be a duplication of maintenance tasks on the queue. So, but the condition you're describing sounds like sort of what I would expect, right? If garbage collection is processor intensive and is going to take 60 minutes, I don't think we want the pre-fetch that was scheduled after it to begin until the garbage collection is done, do we? Yeah, yeah, we wouldn't. But assume the 60 minutes has finished. And you know, again, the Cron syntax executes for, you know, after 60 minutes because we set pre-fetch. So, there would be two pre-fetchers in the queue or, you know, another commit graph in the queue, which would, you know, there would be a duplication of two tasks in the queue is what I was telling you. Ah, I see what you're saying. Okay, right. So, you're concerned. Okay, so how do we, so I guess the question there is do we need to handle duplication, duplicates that arrive in the queue that are in the queue. So should the queue, as now I have to make a match to my Jenkins job experience. So, a Jenkins job, a Jenkins freestyle job, let's be very precise. Jenkins freestyle job with its parameters. Acute Jenkins freestyle job with its parameters will be replaced, i.e. not executed, if a new job. Acute Jenkins build, use the correct word, if a new build is scheduled with the same parameters. So, this is the way that the Jenkins freestyle job handles the duplication problem you were just describing. And so, I think mentally, are you are you suggesting we may want to allow that in the queue, if, if a particular repository has a pre-fetch assigned, and another one is scheduled, we discard the second one because we've already got one scheduled. Yes. So, discard duplicates rather than add, then add them to the queue. And I think that makes sense. Yeah, it wouldn't make sense adding it to the queue, both of them learning at the same like one after the other. Yeah, yeah. Well, for instance, don't schedule a second GC. When a GC is already scheduled. Second, a second GC of a repository when a GC is already scheduled and would we say already scheduled or in progress. So if we're already doing a garbage collection, scheduling another one of the exact same repository seems unlikely to be helpful. Vishab, you look like you're perplexed. Does that make any sense? No, I was, I just, I was trying to understand asynchronous processes in Java. I was just looking at different ways to do it. Yeah, yeah. No, I was just lost a little bit. Okay, great. All right. So I guess that makes sense, you know, this, this kind of duplicate. I was thinking that we would have a way to identify a particular job or a process using a unique identifier and we that would be calculated based on the parameters that we've defined for a particular process. If the identifiers match, then we probably could discard it or whatever the way we could do it. Okay, so let's hang on, let's take that so a maintenance task should be we think should be uniquely identified by the repository it is processing the task it is performing. And then the idea is only one in the, in the queue at a time. Only one you only. Yeah, so the queue contains unique at unique tasks. Okay. Ruchakesh was that helpful to have that discussion or is that just making it more confusing. It's fine, but I have a doubt. Is there any way of getting you know, the, the path of all the cash is on the Jenkins controller? Yes, there is. Yeah. So is there a way so is there a, at least I'm pretty sure there is because to list all the caches on the controller. If, if worse comes to worse what you do is list the directory contents. But I think I believe there is actually a cache, a cache method in, let's see, is it reshub to your members at SCM API. Because, or what maybe yeah it was last year's project wasn't it. That we had an SCM cash on the controller that we were relying on as a way to answer size questions. Oh yes, yes, hang on. I think we can find it just a minute. You've prompted me. So someplace in here, there is a concept of a cache. Get tool chooser isn't this the one that knows how to find the size of a repository. Yes, it will calculate the size of the question then choose it. And the size where is okay size of repo. So here we go. So there's a repository size cash, and that size cash if I remember right, knows how to look inside. Yes, here it is. abstract get SCM source dot get cash entry. There is inside the get plug in this get cash entry method that for a given repository URL, or here get cash durr I think is the one we want right the the directory we want to the folder on the file system that contains the cash. And here it is so abstract get SCM source dot get cash durr will for a given repo now that doesn't give us. The list all of them but I bet inside abstract get SCM source we can find that just a minute. And if not, you could add a method. It certainly knows what the caches are somehow. Okay, so keep one lock per cash directory. Get remote. Get includes. Cache browser. Maybe we look for a file. Cache durr okay so get cash durr get cash durr is here. Here it is iterating over the. Okay, so it says cash durr give me one if it's a direct not a directory created cash entry. Oh okay that's. Cache increase the location. Yeah so well so here let's let's look at it. I need to use emacs instead so we can also use it for operating system functions so here let's look at it on my installation. So this is the file, or this is the directory the folder that that code is looking in, and you can see it here it's got a bunch of caches in it. So if we look at this one, here's a dot get file with a config. This one is Jenkins pipeline utils. And if we look at this one will hope it's not the same thing pipeline utils private. So, so there definitely is an abstract get SCM source. It knows the list of local caches. And now I, I didn't see an iterator that lets us iterate over all the caches. So for that you may have to extend this class to give it away to walk across all of them. No that's not it. Yeah so I think you may have to extend abstract get SCM source so that it, it knows, it gives us a way to ask for the next cash folder or an iterator over the cash folders. So who should cash did that answer that question. Yes. Okay, so abstract get SCM source. All caches find a cache for a repository abstract it SCM source. No obvious method to iterate all caches. You could add one. We would have a cache and people get SCM source. I can see that the gish get cash entry has been populated by the get SCM source remote feed. That is how abstract get SCM source is expecting. So, so you made two statements there one was, you use the word every and I'm not sure that it's every. At least, I think for instance if all I do is run a freestyle job. I won't get any cash for it because there isn't any benefit to that cash. Instead, I'm running a pipeline job. In order to do in order to do the execute the Jenkins file of the pipeline job that pipeline job has to check out a copy of the of the repository on the controller. Now, if it's got access to the GitHub API, it can get the Jenkins file without requiring a full clone of the repository. Likewise with Bitbucket I believe in giddy, but if it's using generic get it doesn't have those API is available so it has to ask for the whole repository. And in that case I would expect it to be cash. So, so did that answer your question, Rishabh or is there something I missed. I just I was just looking at the way they are trying to get the cash entry because that is how we get the cash directly. I was just looking at how that cash entry is being tied to the get SCM source. That's it was just. Okay, all right. Okay, so Russia cash, are you. Are you feeling comfortable with with that idea that you may have to extend abstract get a CM source to give it away to iterate over all the caches. I tried out once. Okay, great. So if on your Jenkins installation, you can look or on a Jenkins installation you can look and see examples of this in the caches directory. Now you may say, Oh, but I'm not sure I've got that a caches directory and if, if you find that you don't have a caches directory. That may mean that you haven't run a pipeline yet, or haven't run a multi branch pipeline. If it would help you, you are welcome to use a toy environment that I maintain as a Docker image. You need a wider sample wider of Jenkins configuration. See this thing and I'll paste it here. There's a repository I keep that has things that help me. And our, I think has it has interesting configurations and many job definitions multiple plugins installed, etc. And if you're, if that helps, you can use it and you start it with Docker underscore run and build it with Docker underscore build it so it requires Docker container but it what it does is gives you a it's a Jenkins controller with with several interesting jobs. Now I've got a much more complicated version of it that's stored with credentials embedded in it, etc. And that one is private I can't make it public because it's got credentials stored in inside. So back to the the cat iterating over the caches on the controller. Rishikesh you're okay if with the task hey look through abstract get a CM source. See if you can identify how you would iterate over the entries there and and decide if work needs needs to be done, etc. I'm fine with that. Okay. I just I thought that it would be beneficial if there was a document which defined the architecture of the SCM. I remember Mark shared that when I was doing my project mission. It was written by Stephen Stephen Connelly, I believe, which outlays the fundamentals of how Jenkins is implemented. So that would be helpful and just trying to find that document. Yeah I wonder if that's. Okay, so let's see for a guy thought that was in the, in the, maybe the writing an SCM plugin. No. Well, here's okay here's oh yes here we go SCM consumer guide and implementation guide. Okay. So, so here's this one. Writing an SCM plugin. And then if we follow that. So, there. It's this one the consumer guide so those who are calling the SCM API are here. And it gives an overview of the different concepts and, and how they are used. And then there is the implementation guide here that we hope the get plugin is a correct implementation of of the SCM API if you find something mistaken there by all means, let us know. So is that what you were alluding to Rashab or did I is is there more than that. Okay. So the question that I have related to this, I think you've answered already or not, is that the way I see it, this work for this handling these processes is that this is above the layer of a particular job or a pipeline. So when I was working in the get plugin. I remember that I always had this concept that I'm going to get a job, which could be a freestyle could be a pipeline and then I have to do some work on it and give my research whatever that is the function that I had to write, and that was the context which I was working with the research works what I understand is we need to go above that layer, we need to go above that because we are actually sitting at the, sitting at the place where we decide when and where the control, the jobs are going to run or whoever is publishing these jobs. So, does that come. So now my question is I'm sorry that it's a long way to question is that is this within the purview of the get plugin. Is this code. Can this code reside within the plugin or does it have to be above that some core infrastructure Jenkins. Good question and since since the get plugin has things that it configures at a global level. I think it's safe for this to be inside the get plugin. But, but for me it's, there are things that the get plugin configures globally, and this will be effectively another significant large and interesting global configuration where. Yes, maintenance tasks should never be more than this many and this many running concurrently. Yes, maintenance tasks should prefer to start Sunday morning or things like that that's what I was thinking anyway who should cash does that. Does that align with what you were thinking were you thinking something different. See, running the task. But, but you know, the, there was some other problems which I was thinking about like, you know, when we are scheduling the task. There are trans syntaxes where you can run maintenance tasks every minute. Okay, like there are syntaxes that I was thinking can we prevent users from running maintenance tasks like that. No, there should be like a base. That's like, you can start running tasks only hourly, and then you know, starts from hourly only you can't run any maintenance tasks below one hour, like you know every 30 minutes or every 15 minutes, or every five minutes. The minimum base, you know, for syntax for one should be starting hourly, you know. So, and I have no objections to that. One of my worries to go along with that is, if I imagine a Jenkins controller like CI Jenkins.io, a large Jenkins controller. CI Jenkins.io has, let's see, probably 2000 multi branch pipelines, each with from five to 50 jobs. So with that number we could have as many as 10,000 jobs on that, that large Jenkins controller. If, if I asked to schedule even hourly 10,000 maintenance tasks may still not complete right, because if they're all the Linux kernel or they're all variants of some large repository. They just be a long time so so isn't this safeguard that you're suggesting should we have that we've got to have that form of safeguards somehow without making it time based but rather making it. Don't overload the Jenkins controller. Is that am I being clear in my phrasing or have I have I misstated something. Here schedule 10,000 maintenance tasks. All of them will be happening sequentially they will that's at least that was my assumption my assumption is because we're queuing. We queue them they will be sequential so it shouldn't overload the controller, but the user may say, Hey, I've got 10,000 maintenance tasks in the queue. And I'm only processing 100 maintenance tasks an hour. I will never empty the queue that that that was the thought I was seeing and so I think we may have to have a way to maybe it's maybe not safeguard them and alert them. Should we, we have. Should we graph the queue length over time. So admins can see if they've got if they're scheduling too many. And I don't know if that makes any sense but we've we've got the concept of these graphs over time that already exist in Jenkins elsewhere. If we look at the system load graph, bring it up. Think I can find it. Let's take a look at this one so no, no, I don't remember where it is. Oh wait a second I know where it is it's under system load statistics here we go. So this, this picture shows shows the, let's look at it. Yeah, here's a good picture. There's executors that are online busy and how long did the run queue is. And it is, it's something like this might help the administrator know if they had scheduled maintenance too frequently. I don't know that we have to do that but it's, it's an option. Ruchakesh you're quiet I'm worried that I'm, I'm blathering I'm saying things that aren't helping. What can we do to help. I was thinking of this, you know, a situation, because as we're running meeting and stars sequentially. Okay, so assume now I'm running the comment graph meeting and stars. All the repositories of the comment, all the repositories on the Jenkins controller, you know, would run the comment graph maintenance. So I don't think it is required to put it in a queue once I finish this then I take out the other maintenance starts from the queue and then again run all of the, you know, maintenance starts on the cache is present on the controller. I don't see you know why we need to add the gift caches in the queue. Okay. All right, good. All right. So what you're saying is no need to add the caches to the queue for. I was like just add the maintenance starts with like information like you know, such as, you know, the prefetch the comment graph the GC, and once you remove the information from the queue, then we can iterate through all the repositories present on the Jenkins controller and run, run all of them. Oh, oh, I see so your. Okay, your concept is is different than the one I was envisioning I was envisioning that that the iteration was by repository, but your concept is, let's, let's admit that the task is garbage collection. And we're going to garbage collect all repositories. And isn't there a danger there that that someone may say I don't want you to garbage collect my Linux kernel repository but once a month. Okay, then. So we need to have some kind of set the future settings and the, you know, maintenance y page. I think it's a good question do we iterate over the over tasks. GC prefetch. What was the graph calculation one. I made it commit graph thank you commit graph dot dot dot. Or do we iterate over repositories iterate over tasks and process all repositories. I mean that has the nice city that has the very nice thing that the user interface is much more straightforward than isn't it by iterating over tasks you say, how often do you want to garbage collect and, and it's one single setting we're not worrying about how many repositories are there they're cached. That's that's a very elegant idea I don't know why I didn't think of that that's very good okay iterate over repositories. And, but the problem with iterating over repositories is that then me needs much more complicated maintenance much more complicated interface user interface. Good, very good okay. So prefer to iterate over tasks to perform all of a single a task on all repositories. Good. Okay, well now now with that small said, does that. I guess we still need a queue don't we even if it's going to be a relatively small queue. Go ahead, Richard. Yes, so my question is, I think that's a, this is a great point but what I want to understand here is it. There's a trade off right if we're not letting the user decide the repository where they want to run this maintenance task. How do, if you're going to. So then the assumption is that our system is not only care about the size of the repository or the parameters of the repository we're going to care about running a maintenance, pushing a maintenance tasks and then executing it over all the repositories. But then I believe we discussed there were issues with that for an example if we're running GC. And if we start GC as the first maintenance tasks and then we're going to run that all over the system, since that being a combination heavy tasks. So do we want to then have the awareness to understand. How do we make sure that whatever tasks we're running because I understood initially that since the user has the ability to decide the task and the resource on which that task is going to run on. The pipeline or the, sorry, the repository, then it would be easier for it. It's it's a work that we don't have to do, and that is to make sure that we're not over utilizing the, the utilization of the controller is not happening. But now since the user does not have an option to decide the repositories, we need to make sure that either we have the intelligence within our system to figure out the kind of tasks we're running and the kind of utilization that they will take once they started to run because we're going to run them across the the whole controller right with all the repositories. I'm not sure I understood that. Could you repeat the last party of your sentence again it was, if that we need to run the, the thing we will need to check is that we are running the maintenance task over all of the repositories, but that that is not overloading the system. Ask your question again we shop sorry for my not being able to keep up. I think I'm thinking my I'm just thinking out loud probably so I'm just thinking if that is something that we need to worry about or not. So my question was that if we're not giving the user an option to decide on upon which repositories they want to run these tasks. Then what we're doing is we're just making sure that okay this is the schedule I'm going to run a GC on all of the repositories. Or I'm going to run commit graph on all of the repositories. Okay, I guess it's it's not a question mark I guess I think this there is no problem doing that. I was even thinking I was thinking of you know, a way where you know, so a user can, you know, enter, you know, a regular expression or you know the minimum size, and by processing each maintenance, each get cash, we can check all these conditions which the user has put and then check if he wants to run that, you know, get a repository, you know, run maintenance tasks on that repository skip it. That was what I was thinking one day like a filter. Okay, yeah that that see for me I think I think you would already have something useful and helpful if we had a single page that said here are the five maintenance tasks. Would you like to them to start, and how frequently would you like them to run that's already something quite useful. And, and that will, I think help performance already of operations then, and if you then later extended it with added the additional filtering if you'd like, or, or do something you know something more sophisticated but the first first round of iterate your garbage collection how often commit graph how often prefetch how often isn't that already something that that would be valuable to users, and, and you could do it independent of of the of all sorts of other things. I agree iterative approach is always best. It's better to, you know, open the first thing. So start start with. With a simple. Yeah, okay so so for sure Kesha are you okay with that idea then I think I think this was your key concept and I had missed it iterating over tasks, I like that a lot. That was what I worked on I even made a class diagram. I can share it in the login channel. The UI also I have made a sample UI it's present in the, in my proposal. Okay, a sample UI. I haven't used the latest UI things and that but that that's something you know, like that's like a start. Well, and I think that's a great. That's already a great thing that you could begin coding now is implement the sample UI be ready to show it and we could have a conversation about hey here's how the UI, because the first experience the user has is with the UI so I like that. Good. Now are you and you're familiar you've seen at least the design library on week. Good okay good. I like that very much. Now I apologize we're almost at the hour and I'm, it's late my night. I'm, I'm a little bit weary. I'm unavailable next week because I'll be out of town Rishabh, you'll provide the zoom link. It'll be a different length than the one that's in our meeting. And the two of you can meet separately now do you have permission to to edit this, this. Oh yes you do good. Okay. You've got a way to edit the notes. Be sure you record the session, and then after I return I'll help so that we can upload it to YouTube so that we've got it recorded publicly. What about. Yes. All right. Next session in a week. Mark will miss the session. Okay, the shop send the URL record the session. Great. Russia cash anything else that we should discuss in the last few minutes before before I go go to sleep. I have nothing else. Okay, now I apologize that the recording from our last session isn't available yet. I will do my best to get it ready within 24 hours. Maybe what I should do is let's end now. I'll go upload those. At least then I can give you a pointer to them uploaded. Even if I haven't finished all the processing of them because then at least you've got access to them. Would that would that work okay for you. Yes. Great. Let's do that then Rishabh anything else from you. No. And Russia cash anything from you. Okay, then I'm going to go ahead and call the color session done for today. Sorry, our session is on Wednesday, not on Friday.