 Welcome. It's the 13th of July, 2022. This is GitCache Maintenance. So Roushakesh, what are the topics that you'd like to be sure we discuss? Today, I wanted us to test the software on your machine because I tried doing it from my end. But it wasn't coming. There was a bit of lag when I was using your machines. So that was one thing. One more thing I wanted to discuss was about the presentation. Like next week, we'll be having the presentation, right? So what do I need to focus on and what all I have to be working on? Good. Those both sound good. And few other doubts I had regarding, you know, like we discussed about pre-fetch. We weren't able to use pre-fetch because we don't have the credentials of the caches. So how do we complement that? Okay. Good. Yeah. And very good. So pre-fetch. Let me take some notes to remind myself of the, so we've got GitCache Maintenance. Here we go. Hello, Rishabh. Okay, so. Hi, Rishabh. Hi, Rishabh. So taking notes today is July. Okay. 13 in your world. Okay. And you're part of the world. Rishabh. All right. So review again for me. The topics were, let's see, presentation for July 21, 2022. The midpoint. Testing. To confirm behavior testing. The plugin implementation. And then there was a third. Oh, pre-fetch related. Oh, right. Using credentials. Right. Operating. Or maintaining caches. That. Have. Private repositories. Right. And there's one more thing. You know, we were adding caches to the hash set. Okay. But we didn't, you know, add logic to delete those caches from that set. So, you know, assume if there are too many caches present on the Jenkins controller, and if I don't restart that Jenkins controller, then, you know, caches would be added to it. And even if user deletes it, it would still be present in that set. So I don't know if that would be a performance issue or, you know, it would consume a lot of, you know, memory. Okay. So there's no process that removes. Cache entries. From the hash set. Okay, good. All right. Any others that come to mind. No, nothing. Okay. All right. So, so shall we testing could probably take the entire time. I assume we want to time box it to not take the full hour on testing. So how much time would you like to spend on the other topics? I want to like a more preference to, you know, running the software on the, you know, on your computer that is one and the presentation. So that that would be like the most important thing session. Okay, so what if we what if we were to allocate half half to each so 30 minutes to each of those, would that be okay. Yeah, that will be great. All right. We show up any topics that that you want to be sure we discuss. Nothing that comes to mind right now. Like I was reviewing Rishi cases PR and I have half with it. So I would say once I agree with it, I would probably have something once I agree fully. Some questions. Great. Okay. Well, and I, I started my reviews. My reviews were focused rather on. Fixing the compilation. So it now compiles again. And resolving test spot bugs warnings. And I was trying to work on test, test failures. So I actually introduced a test failure that a lobby that we should go in and fix to get rid of one of one of the failures I created, but I think it's good. Let's do some interactive testing. I think that's much more valuable than the automation, automation I can do independently looking at, at failures. Those things can be done by us as individuals, whereas the, the experience of testing the, the plug in. In, in a live environment is, is much more interesting, I think. Yeah. Okay. All right. So how about I'll share my screen and let's, let's start through. A series of, of testing tasks or testing ideas. And talk about which ones we want to want to try. Okay, so. Let's see share the screen. Okay, you should see the cash maintenance document. So any, any suggestions, I assume the things, some of the things we need to test is we need to test. The, the, the multi, the operations that are available. And that includes prefetch. And there, as you said, it's public repositories. We know that private repositories can't work. So there's probably no point in, in testing private repositories there. Then we've got commit graph. There was a garbage collection. There was a fourth operation. Help me with that. Lose objects and incremental. Okay. And is that a single operation or two operations? No, there are two different operations. Okay. Good. All right. Incremental repack. All right. Good. Now commit graph though should work on both public repositories and private repositories, right? Yeah. Except prefetch all of them are local operations. Great. All right. So now some of the attributes that I think we want to be watching is I think we want to watch the load on the system. While the operations are being performed. Right. If we were to see that something was very heavily loading on the system that so things to watch system load. How is the, how is the overall system performing. Oh, execution time. That's all right execution time of the, of the maintenance operation. Good. Are there other things should we be worried about delays in starting the maintenance operations? Not really. Okay. Any, any others like that that we need to need to be of, they need to be of concern. I don't know if I think system load covers it, but would there be a relationship in the execution time of the jobs that are running in check-ins and the maintenance tasks that are running in the background. That's a good question. So impact of maintenance. On other jobs. Yeah. So for example, we could start a. Get plug in test job. Start a, whoops, let's switch this. It's determined. Okay, there we go. Let's start to get plug in test job. I get client plug in test job. Start. And then start a maintenance job and watch to see. That there is no. Oh, probably should also scan a multi branch repository. Because that actually uses the caches. But that is multiples. Yeah, that makes sense. I also, as I, as you were talking about caches, I have one question. We can ask that later or after this exercise is done. This is related to locking caches and getting the logs. Oh, well, but that seems like something we should be watching for right is cash logs during maintenance. Right. So we should be able to see that. Okay. So what are some of the other operations attempt a. For instance, attempt a repository scan. During a long garbage collection. Yes. I just wanted to ask what is our, I mean, what do threads do if, if, if a thread who has acquired a log for this cash, if it dies or if there's an issue. While locking it, what would other threads do? Would they wait for it until, and unless they get the thread or is there a specific time for which they wait and they move on? Because I did not see any kind of logic like that. Trying to acquire a log and then and waiting for a certain interval and if it's not there, then moving on. So, but, but I thought that the operations were. Coming from a single thread. So for instance, there won't be a concurrently running commit graph and pre-fetch. Is that correct Russia cash or have I misunderstood? Yeah, there is one single thread which is responsible for maintaining, you know, for running all the maintenance tasks. So now it's so assume there's one thread and it's going to run pre-fetch. So what it's going to do is when it's running pre-fetch, it goes through all the caches present on the Jenkins controller. It locks the first cache. It runs the task, then it unlocks it, then it goes, goes and grabs the next next, you know, cash lock, executes it and then unlocks it. So that sounds like. So my question is that you are acquiring a law that means there is an assumption that some other across the whole system, some other thread could try to look at the cash while you are working on it, right? I'm not just, this question is not just for the scope of the maintenance tasks that are trying to acquire a lock on the cash. The question is, let's say GC acquires a law. It started to run, but for some reason it takes, it's taking a long time, right? And a mighty branch job wants to acquire a lock on that same cash. So what is the behavior? What would be the behavior of that program? Would it wait for that to acquire a lock? And if it would wait, then assume a user is trying to run a maintenance job and it would take him, let's say five minutes to do that. And now since GC is going to run and GC is going to take 10 minutes, his job is going to be delayed 10 minutes. And he would not know that. He would not know why that would be happening. So I did not see any kind of logic. So I saw that there's some logic in the abstracted SCM class where we use a re-entrant lock. We populate the cash, the map where we actually store the locks, right? But when we try to acquire the locks across the whole program, whole kit again, source code, I don't know if that is something that we have kept in mind. Since this code was written early on, right? This was written by Steven. I am assuming it works fine. And there's not an instance where a thread enters, acquires the lock, and then for some reason it cannot free the lock. And the other threads they were just waiting or what are they doing that kind of behavior? It was something that I could not figure out from the code last night. I think it's a good question. It's a very good question, right? So things we should be checking for are our locks released when the maintenance task is complete. Okay. Any other observations, Rishab? This wasn't. Okay. So let's, I think what we're going to do then is grab, what I was going to do is grab the, so this is pull request 1277. So I'm going to go grab the build of this. So pull request 1277. And I believe I had already updated the get client plugin. Let's take this one. Okay. Manage Jenkins, manage plugins. So first things first, let's check to see what version. Oh, okay. This, this one has. As I've been updated with the, the newer version of the client plugin, but haven't restarted yet. So I'm going to paste this URL. This is the get plugin. RC 4832. Okay. Now we're going to restart. This is as usually it's at least a minute. Because there are many jobs on this. So while it's restarting, let's take a look. Are you okay if I spend just a minute while we wait for the restart to fix my mistake in the tests? Sure. Okay. So the mistake I made is evidence here in this test result. We look at the test result. I extended or I did some work on a test for test get version. Get test, get, get version. And the mistake I made was I said that it was not allowed to have a value of zero on the lower range of a version number, either the minor, minor number or the patch number. But that is allowed, right? 3.0.0 will eventually be released as a get version. So zero has to be allowed in the second and third position. We don't want to let it be in the first position because we're not interested in supporting any get version older than 1.0. So I'm going to do some quick fixing of this mistake that I made. And as soon as we see indication that the, whoops. That's the problem. I don't want to annoy me until I get rid of it. There we go. Okay. So yeah. All right. So this has my latest changes in it. And let's see where I made the mistake. So here's in test get version. The mistake I made was that. Okay. It asserts that there must be more than one entry in the version arrays. So that means 2.0. In the cases that I know 2.17, for instance, is an allowed version. No dot, no dot beyond it. So if the major version is, it must be greater than zero and less than 99 was my assumption. The minor version is greater than minus one and less than 99. And likewise the patch version should be greater than minus one and less than 99. So that changes seem reasonable. That's two of you. Yeah. Okay. Let's do a compile maintenance task configuration tasks. Okay. Okay. Now let's go back here and I bet we're ready to out no wrong one. This one is ready. So let's check that we've got the expected version of the plugin. So manage plugins installed plugin. Get the client plugin version is RC 3100. And the get plugin version is RC 4832 RC 4832. So as far as I can tell, we've got the right one. Now let's prove that further by going to. Manage Jenkins and here we see get maintenance. Yes, clearly we have the right thing. Great defaults to hourly. Now we had said that we wanted to test. Is there one of those that's more valuable to you to test first. Rishikesh. I already tried testing, you know, running tasks every minute. Those things are running as expected. We can try it once again. Okay. Yeah. Now, and how were you, how were you able to observe it? Through the system log. Yeah, yeah. Okay, so if we look at the system log here, and you already have a system log defined. Yeah, maintenance. Yeah, perfect. Okay. So when I open this, it should show me some of your, some of your, okay. So loaded get me now. Do we need to adjust the logging level at all? No, no, once it's fine. Okay. So it's logging all records. Oh, okay. Good. So you're being very verbal verbose. Great. All right. So let's pick one then how about. Let's do prefetch. So prefetch if we make it minute by minute. Yeah. And then save it. Okay. Yes, I really meant minute by minute. And I do a save. Yeah. And now it would, it would, it would execute. Okay. All right. Now, all right. That surprises me a little assigned hourly to commit graph. Because I was trying to assign it the empty string. Oh, like, okay, can you go back to the get like to the UI ones? Mm hmm. Like, I mean, the meeting has to ask you why. Oh, sure. You bet. Here we go. Okay, got it. Right. So if I want that blank, I have to actually blank it. Thank you. Okay. Good. And that makes sense. Okay. Now if you look at log records. You'll have to scroll the previous logs. Yeah. Right. Okay. So incremental re repack faults. Assigned incremental. Okay. Loose objects execution faults. Commit graph execution faults. Prefetch execution faults. Assigned every minute to prefetch. Oh, interesting. Now. Wait a sec. I did assign every minute to prefetch. Okay. So it's telling me the truth. Good. All right. And it's running a task prefetch. On this repository. So let's go take a look at it. Shall we? So as a user, the way that I would understand if. What I've actually executed runs or not would be to come to the logs and see the logs are to be planned to actually put something in the UI to be able for the user to understand that. And the second phase of, you know, that you saw a thing that was the, you know, that was what proportion, you know, in the proposal that I would, you know, add a UI to display. How the maintenance tasks were executed. Like what were the, you know, the data related to how, whether it passed successfully or not. So that doesn't have to come back. Okay. I see nothing in pack. I see maintenance.lock. I assume that's a lock that either command line get has applied or we've applied. But first cash, I don't see any pack files. Oh, is this an empty repository or something? It seems to be. Although it looks so get remote minus fee. Oh, okay. Interesting. This is that should not be empty yet fetch. Well, there would be a brief edge, you know, fire here. You say there would be so. So this, you know, this. If you go to the refs, if you go to the refs folder, there should be something related to. Okay, so let's look in dot get in the refs directory. Hmm. Hey, and I don't see anything. I'm going to try to get fetch. Are you okay if I do a fetch? Yeah, yeah. I'm not sure why this is could not read. Okay. So why is that the case? Let me fix that. That won't help us because we need. We need this one is this is a good example of a large repository. So get bare Linux. And there is no way for us to for the maintenance tasks to get the credentials and make connection with the remote deposit. I think that was on discuss. There is one way of, you know, user entering the credentials. Because the abstract get SEM has a, you know, abstract method, which takes in, you know, a string credential ID, but I am not sure how do I implement that with the logic of how we are executing the maintenance tasks right now. Because that is for a single cache and we are running on a group of caches. So we would be needing, you know, a group of credential IDs. So. Okay. Okay, so I'm going to do a. I need a copy local here of, and let's do a minus, minus mirror. I know stable slash Linux. Okay. Get clone why too many arguments. So apparently I don't know syntax just a minute. So let's do a clone mirror. Oh, not in your minus minus reference. Okay. Okay, so that first one that we were attempting is not going to help us because it's empty and it's going to take some work to get it filled. We'll want it, but let's let that run. So back to the log. Right. Let's see you. Refresh the logs. I think you'll get more, you know, more caches. Okay, so let's do it. And we now see. This was the one that I was facing when I was trying to execute the fetch itself. This one. Yeah, yeah, yeah. Okay, so it says prefetch added to queue and it's now saying, hey, it's already in the queue. So it does this tell us that it's actually working. So initialize executing prefetch on this one. Let's go. Okay, I think what I understand what's going on right now. Okay. So basically the maintenance task has been added to the queue like pre such has been added to the queue. And now what what's happening is the main, you know, it's running on that cash. And then we terminated the maintenance tasks and stentaneously, you know, to update the value. And I think the lock hasn't been released. So I think that's the reason why. Okay, well, so let's go look right. If we look here. And that is, oh, that's the same location. All right, so. And where did I find the maintenance lock. In the back. So it's in objects pack. No, it doesn't see doesn't see the objects. Yeah, I should be in the objects. Okay. Yeah. All right, so you're okay if I delete that. Yeah. Okay. And I don't see. I see a fetch running. And an SSH running. Okay, so it's, it's trying to do the fetch there. And I think what you're saying is then should I, should I stop this task here? Well, yeah, no, let it run. That's the task which is, you know, continuously running. Okay. So I'm not understanding it. Should I, should I leave it running or kill it? Oh, you, you can terminate it right now. Okay. All right. I'm not sure I can actually terminate it. Because it may be inside the Docker container. Oh, okay, I got it. And then this is an SSH command. That's that one I can terminate. Okay, bin dot get is a private repository. So somewhere here. And this one I've got a maintenance lock right at the moment. And it is 90 megabytes. I'm 90 megabyte repository. So it's not a small repository. And it's saying do the fetch or it's saying it's somewhere saying do a fetch. Is it still have the lock. It does. If I were to do a fetch here. Will it honor that lock or is that lock only honored by by Jenkins. That is by Jenkins that maintenance logs. Okay. All right. So if I said this, I should see some data traffic. There was a little bit. And now it's got some things that can be garbage collected. Okay. So what we've got running right now, if we go back to this. And look to see, we should see the log file that tells us. Okay, so it's using the official get maintenance command. Executing the prefetch. Should we go there? Yeah. Okay, I don't see a maintenance lock here. Now it's possible that it just completed. Oh, there it is. What am I saying? It's right above where I'm typing. Okay, good. So we should be able to see something in the log that tells us that it's making progress on that. Now I need the directory name again, just this is the directory with that directory name. Here we go. Cash lock has been acquired. Executing the prefetch. And then the next log should be releasing the log, but I don't know why is it not releasing the log here. Well, and let's go look to see if we've got a process running. Yes, it's still running. But would it take this long? Particularly in getting zero CPU seconds. Okay, it started at 34, so it's four minutes old and seems to be getting no progress. So would you mind if we try the same command? Yeah, sure. Just to see if we can understand why it's not. You know, you can run the maintenance task itself, you know, the above command get maintenance run. Oh, okay, so I can just say get maintenance prefetch. Yeah, you know, dash. And then the last thing we were running was what we were running prefetch. Okay, prefetch. All right. Get maintenance. Okay, so I don't have access to a get version. That is the same version as what's inside the Docker container. Oh, can you go back to the logs once again? I can, yes. Here? Oh, here. Okay. If you see if you scroll a bit, you know, and all. Like, can you see that? Yeah, there would be a log which states you see greater than 2.30. Then why, why is this long? So, because my Jenkins instance is running inside a Docker container. Okay, so, so we can do, we could go inside the container like this. Docker exact exact. Let's Docker exact minus minus. What is the, oh, minus I. I minus T. Give this and then bash. Okay, get minus minus version here is 2.30.2 whereas on the outside of the container, it's the older version. All right, so now we want to go to that location. Jenkins underscore home slash. Get dot get. And the place where it's stored is objects. Yes, there it is. So it's still in maintenance. And now if we look at the process listing, it's doing this prefetch thing. And so what you were saying was, Hey, I should be able to do the prefetch thing myself just by saying get maintenance. That's my task. Run minus. Run. Is do I need minus minus? No, no, no. Okay, so just get maintenance run. Refetch. No, no, no, minus minus. Task is like the commanders right there. Got it. Okay, like that. So get run. No quiet. So that we see noise. All right. Maintenance, objects, get objects, maintenance exists, skipping maintenance. Okay, so it thinks it's running. But when I look at the processor time that's been allocated. It's not doing anything with that. So, have we specified a time out here for the maintenance task? No, no, no, there is no time out. So then when you've acquired a log to the cash and now let's say this is running and not exiting any other thread was trying to use that cash would wait for the log site. Yeah. Yeah, there's a deadlock situation or something. Actually, I don't think it's a deadlock. I think it's this is an authenticated repository and it's blocked. Because when I try that get fetch command, the exact get fetch command here. It prompts me for using a password. Oh, it's remote minus V. So. Oh, right, right. Okay. In the, in the user environment, I had access to my SSH private key. In this environment, I do not because I'm inside the Docker container. So this is an authenticated repository. So how do I, is there a way for me to end this running process and cause it to go to the next process or am I. Oh, okay, I assume if I kill it, I assume if I kill it, it will, it will continue with the next with the next, the next repository. Is that a safe assumption? No, I think you're the problem right now is we are not able to, you know, run pre-fetch on authenticated authorized repositories. Right, right. Exactly. Due to which we are waiting continuously, you know, for, you know, authentication is what is my assumption. So I think we can kill it by going to the UI and clicking on terminate. That would kill the thread itself. Oh, okay. But now that would terminate all processing, right? Whereas what I did should have just shown one message. Whoops. Now, why is it restarting? Okay. That I don't understand. This was the case which I faced when I ran only the pre-fetch command because I, when I tried the same thing and even I found the same error. I wasn't able to debug it. Well, and, and when I try to run that pre-fetch command, the command that it's running, I can see why it blocks because it, when I do that. Oh, I'm back in the, I need to be there. The cache is this. Okay. There it is. And now if I say get remote minus V. If I say get fetch. It prompts for a password. And this is where a maintenance task would get stuck if it's trying to. I think so. I think so because what we see here is. Here's the maintenance task. Interesting. No, it's, it's moved on. To Jenkins bugs, which is a public repository and then Jenkins bugs private, which is a private repository. So can we see the logs once like. Something. Okay. So the logs here. Nope. Nope. Where did I put my logs? Well, must have, I must have made a mistake and click the wrong thing and exited from the log page. System log, get maintenance. Chugga chugga chugga. I need a faster computer. I always need a faster computer. Okay. So here it says. It's running doing the maintenance run on this repository. Shall we go look at that repository now or would you like to see something different? Okay. I'm going to do that in a minute. If you, if see, if there is no access to that repository, actually there should be a log displayed like that. Like about there's a log, please make sure you have the correct access rights. And that we definitely did see, please make sure you have the correct access rights. Because it says permission denied fatal could not read from remote repository. So, I think that's what we should be seeing here is inside this repository. It should show that it is stuck. Okay. It's got the maintenance lock. And if I do a get pull. It will. It should show that it is stuck. Right. And I think I think that's what we should be seeing here is inside this repository. It should show that it is stuck. Okay. And if I do a get pull. It will. Okay, let's use. No, get fetch. Now it's prompting me for password. So I can't work on this one either. Are you okay if I kill this one? Can we try changing the maintenance task to something local for now? Sure. Absolutely. So if you do that, we would switch. Right. We go back here. Okay. Okay. Disable prefetch. And add a commit graph instead. Yeah. So let's go to commit graph. And save it. Okay. And that one. Save. Okay. I had let it time out. So I have to have to bring it back. So no prefetch. And we do want commit graph. And we want it how often? Okay. Save. And can, can we clear those previous logs once? Yes, absolutely. Save. And now let's clear this log. Okay. Now log records. I'll just take like one minute. Okay. All right. So now we're ready to save. Do I need to terminate here and then start again? No, it's fine. Okay. All right. Log records. There we go. Okay. Maintenance configuration data successfully stored. And the thing it's performing is. Incremental repackage. No, no commit graph like this. This is all the data which has been stored the logs. Okay. Can we refresh this page? Yes. So just so commit graph added to maintenance queue. So if we refresh, I think we would be seeing. Some logs. Okay. So commit graph added added to maintenance queue. I'm not seeing it there. But if we look inside the inside the machine. We're currently running a prefetch here. As a result of a maintenance run minus minus task equals prefetch. And if we look at. I was going to look to see if we could see the parent process ID. So parent process ID of this one. SSH talking to the remote is. This one is talking and it's prompting for a. A password is prompting to get authentication. So we've once again found a case where I think. We need to kill this process. Oh, Mark, I think I understand what's going on. Like once can you go to the maintenance UI and terminate the. Oh, okay. Because in this case it's holding the lock. Yeah. Because actually the prefetch task was still present in the queue. So we didn't. You know, Okay. So now if you look at the log here. Will we see an entry in the log that tells us that the queue has been cleaned. If you scroll. Okay. So what I see is. The initialize executing prefetch. So it's attempting a prefetch now. Even though I'm not. Executing is that is am I reading the log correctly. There should be a log which says, you know, killed the thread successfully. No, something like that. Long search. Okay. I don't see threat or many did execution, something like that. Okay. Terminated execution of maintenance tasks. Terminated scheduling of get maintenance tasks. Couldn't run prefetch message. No. And can we like now run the, you know, come and get off maintenance. Oh, good. All right. That's a very reasonable request. Okay. I'm going to clear this log. Okay. Show me what's happening. And we have to click on execute and the UI. Right. So I just wanted to be sure I had a clean state. And as far as I can tell, I do have a clean state. So now we're going to attempt to execute commit graph every minute. Okay, commit graph added to maintenance queue. And now it will. I'm going to look at the process. I see a fetch running here. Still as a fat operation. Oh, right. That should, we should be able to kill that, right? Because. The maintenance, the commit graph task. And now the commit graph will iterate over all the repositories. Yeah. Come on. Okay. So if I were to go look at the process. Because on its own, I don't know. Now, why would it kill it on its own? Because I tried because when I clicked on when I click on the terminate button, you know, the thread running that prefetch should have, you know, but this is, this is a separate execution. Right. Yeah. So once again, we've got jobs that are trying to run. And here it is. It seems to still be starting new jobs. Something is starting new job. It is too strong. I don't know what it would mean in this context, but okay, I, it looks to me like I have successfully killed all of those jobs. Now will I see something that tells me I did that terrible thing in the. Oh, wrong, wrong window. Sorry. Let's bring up the correct window. Here we go. We are looking for this one. Okay. Log records. All right. Lock the cash. Version that says it can use it. Executing commit graph. Okay. So we should be able to see the proof of that in that repository. Right. Yeah. Yeah. Okay. So now and. No, that would be a comment graph. Okay. And it would be at the top level or is it in. No, it would be in. Info. So objects info. There it is. Okay. And this is a binary data file. Right. Yeah. Yes. And I think in the logs, it would show the execution time as well. And. Oh, okay. So let's. Let's go look at that. Yeah. Here we go back here. So in the logs. Okay. So if we take. That. We see. Oh yes. Here it is. Execution time. Nine milliseconds. Okay. Good. All right. So. So this definitely does seem to be. Seem to be doing the commit graph. At least the case that we tested. And here we have a. This one. Which has. Run and. Now completed successfully, right? It says. Hey, we're run a batch file. Run a shell script. Lock the cash. It then says, Hey, this is a modern enough Jenkins. Or not modern enough. Get version to do real maintenance. And then it says. Start running the commit. The commit. Graph task on this repository. And there it is running. And it ran for. One less than one or less seconds. Okay. And then it unlocked the cash. Okay. And then it says, Hey, this is a modern enough Jenkins or not. Modern enough get version to do real maintenance. And then it unlocked the cash. So that, that looks promising. Right. That seems like it did what it was. What it was claiming to do. Yeah. Now let's let's look at execution time. Oops. Okay. 12 milliseconds for that commit graph. Seven, eight. Five, six. Five. Seven, six. Okay. And I apologize to the two of you, but I'm running out of energy. I'm, I'm getting weary. Sorry. It's, it's getting late my time and I've been up. Did we accomplish positive things for what you want to push cash and should we schedule a time to do this again? Two days from now, for instance. Okay. I think that would be good if you schedule it two days from now because I'm worried about few things not working. So, and then there is a presentation on next week. So what I'm going to do then is I'm just going to open up my calendar and I'm going to move. This, this meeting. To two days from now. So I'm just going to edit it and shift it so that we will meet in two days from now. Okay. So I'm just going to do a presentation office hours. On what for you will be July the 15th. Because I try, I wanted to do this on my own, but I don't have access to those, you know, that doctor container. So that was, yeah. Exactly. I mean, there are, there are all sorts of subtle things going on. There that it's, it's just better if we do this together. We'll find more things together. I think that's a very good thing that we did exactly what you suggested this. We've got a plan for how we approach it. We've got things we should be watching. So I think. I think it's a good, good thing. Now, one of the things that I haven't. Arrived at yet was how to get. A good test case. For. Things like. Things like garbage collection. So because for instance, here, I just copied the. Stable Linux configuration. And what we'll see is it's all in a pack in pack files. So there's no, no, no loose. There are no loose commits to, to. To be. Tied by garbage collection. And I don't know how to generate a repository that has lots of loose commits other than just creating the commits. And maybe that's what we have to do. There is one way of, you know, removing all objects from the pack file. You know, I tried that once where I don't know all objects from the pack files and put it in, you know, an objects folder. I could run the maintenance task only for that sake. And then everything again goes back into the back first. Oh, okay. So you have a technique. Okay. So you can teach us that technique on Friday then. Good. All right. And regarding the, you know, next week's presentation, what about that? How do I approach that? So, so what I would do is plan to show a demonstration. So we can demonstrate this. And, and show, hey, look, here's what it's doing. It's running a garbage collection here. It's running a, a commit graph. It generated the commit graph here. And, and show now maybe, maybe for real life, we ought to have. Show it on my system rather than showing it on. So maybe we need to have me, me doing, me running the keyboard on the demonstration while you describe. So that, okay, share my screen, but you describe what the story is. Look, we're going to do this now and watch what happens here. These things that don't have these caches that don't have commit graphs. We're going to remove the commit graphs and now we're going to run commit graph and it will recreate them all. Or do we, you know, record a video only for, you know, running the maintenance task and, you know, go through it rather than, you know, having it done live there because I think that would be chaotic. You know, I wouldn't have things up. Yeah. I'm not the least bit scared of doing a live demonstration. That's, there is no fear in me because we know this is a, we know this is still under development. If nothing breaks, clearly we've done something really wrong because we've hidden all the breaks. So, so I think we should do a live demo because I worry about trying to have them display a video of us. And I worry about us spending the time and energy to create the recorded video. Okay. So if so, go ahead. So what I will do, I will try making a presentation first, you know, you know, containing like information about what this plug, what this, you know, project is about and what it's going to solve. And yeah, that that's what I would do, you know, to get myself ready. Good. Well, and let's, and let's plan to review that on, on Friday. We've, we can have a, at least let's talk about your outline, as to which topics you're going to discuss and what thing, what that means and why you're, why you've chosen this path versus that path, that kind of thing. Okay. And one, one final thing, can we run a maintenance ask you, you know, can you configure something like hourly something on your machine and you know, check after an hour, can we check after an hour, whatever those maintenance tasks are running or not, because I tried doing something like that and I couldn't get any logs. Ah, okay. Well, so yeah, let's, let's ask it to choose. Are you okay, which one would you like to do to run hourly and see. Oh, you know, a comment, not prefetch, prefetch is giving a lot of issues. Right. You could do a comment graph or a GC. Okay, so let's do hourly. Yeah, should I terminate the one that is running now. Yeah, yeah. Okay, so let's terminate that. So it says it's terminated. Now if we say hourly for commit graph and we do our and will there be there will be log entries for garbage collection as well. Right. Yes. Okay. And are those two enough or would you like more. Yeah, I think that's really sort enough for now. Okay. All right. Wait a sec. So you say those two are enough or do you want more? Yeah, it's fine. It's fine. Okay, so I'm going to say execute. And whoops, why did it say. Oh, you didn't save it. I guess. Oh, right. Right. Yeah, I love data UI. Right. I you are exactly correct. I did not. I did not. I did not. I did not. I did not need to terminate first. Yeah. And then I say hourly. Yeah. And then save it. And I set this to hourly. And then I say save. I see it. And then now. Yeah. Now I can say execute. And now if I go into the log, I should have cleared the log at that. Judicious point. I'm going to clear it now. I'm going to say. You cleared it, right? I cleared it, but there's a, there's a task running. It's just hourly. So I know. Oh, so I, right. So it'll have, it probably has not happened yet. Good. Yeah. Okay. So that, that means tomorrow, 12 hours from now, when I get up and I'm functional again, I'll check this to see what the log says. You'll have to see at least 12 logs. So I hope so. Right. And I added this thing, you know, a hash thing. And so, you know, so that it, uh, schedule is the task, you know, randomly to not generate load. So we'll have to check that also. Great. I tried doing it. I couldn't see it. So if you could see it. Yeah. I will, I will look at it further. Thanks very much. Thank you. Yeah. And I think let's see there was, I was in the, we were working on, or I was working somewhat on a fix for my mistake, right? Did we ever push that? Not sure that I ever pushed it just a minute. Maybe I didn't even fix it. Okay. I didn't. So, so who should cash. I apologize for leaving things with debris that there, there, there's a failing test in there. I don't know where I put the work I was doing before we switched our focus. Because. Hmm. Oh, oh, I know where it is. It's actually in this window. It's right here. There, that's the fix. All right. Before, before I go to bed, I, I want to know if this fixed it. I think it would depend on your version, right? It does, right? So my version does not have a dot zero in any of those. Yeah. I think I'm okay with that, right? So get commit. Fix. Test that I broke earlier. Two dot zero dot zero is a valid yet version. Okay, so at least one of the failures that I introduced is resolved. I'll try fixing those, you know, other ones. I'm just worried, you know, because I've tried doing all the, and then, you know, try doing all it off it and, you know, still there are issues which doesn't, you know, make the code run. That's something which is worried. Right. Well, and, and now I'm curious what, which of these I was expecting to see one of them. Oh, there it is one, two, seven, seven. Okay, good. So the changes that I just pushed have been. Detected. And the job has started. Yeah. Okay. Yeah. So it's still doing the checkout, but at least it has. Wait a sec. Why is it not showing the changes? That's interesting because it's already doing the checkout and yet. The UI is not showing the change that I made. Oh well. It will eventually. I'm going to call it a night because, because I need some sleep. Sure. I'll see you on Friday. Thanks. Good night everybody.