 Hi everyone. Welcome to the second Cloud Native SIG meeting. It's a regular SIG Cup where we discuss ongoing projects in Cloud Native Jenkins. Then we SIG Cup on these projects and make some decisions about them. So if you're interested to participate in the discussion, you can find all the links to the today's meeting on the Cloud Native SIG page. This is Jenkins IOS 6 Cloud Native and here you can find YouTube video. We broadcast on YouTube and there are the results, a document with meeting agenda. So you can find this document and if you want you can propose your topics there. If you have any questions during the call, please ask them in the Cloud Native SIG Gitter channel. We will be monitoring this chat and answering the questions. It's Jenkins CI slash Cloud Native SIG. We also have a participant link. So if you want to join the call, you can follow this link. But now we have a limitation of participants. So if you cannot join and if you want to ask something just ping in the chat and we will try to do something with that. So, okay. I lost the focus on the window. Yeah, today we have several participants on the call. My name is Alec Nashev. I'm the host today. We also have Carlos Sanchez on the call, Jesse Bielik, Rick Alex Nolont, Irina Tracey Miranda and I guess this is right. We have somebody else. We also have Jesse on the call. Okay. Yep. I just missed him in the list. So we have several topics to discuss today. I arranged the topics according to the feedback and Cloud Native SIG and discussions. So we will start from a Jenkins configuration code just to understand whether it is a Cloud Native SIG or probably another SIG. Then we will sync up on the status of external configuration storage. This is a new job which has been submitted something like 10 days ago. And then together with Jesse, we will provide a status update on external build login, which we were discussing during the previous meeting. If you want to get more information about previous meetings, all the links are also available here. And last but not least, we will try to set up a regular term slot for meetings so that we do not start doodle each time we want to meet. And yeah, probably we will have some other topics. For example, I propose to discuss some feedback from Jeff Pierce, which we have got during the last meeting. So yeah, this is the agenda I proposed for today. Does it look good to you or would you like to add something else? Okay. I guess no. So for meeting notes, we will be putting them to the document. So let's start from the first agenda item. It's configuration as code initiatives. So Evelina, maybe you would like to provide some context about this topic. Okay. So first of all, I am sorry, I moved into a new apartment and there is a very, very not good internet. So if I can't hear me well, Oleg can cover, but that's right. When I don't remember who was that suggested, we can have the link with the tools, Oleg or someone else. But when the proposal came up, there was configuration is called as a separate seek. And I was mentioned as a potential chairman. And I liked the idea. But then when it started, again, I can't recall who was that suggested that maybe we should try to be a part of cloud native seek. And I liked the idea because I wasn't sure how much we would have to talk about configuration as code. But I thought about configuration as code in the context of the plugin only. And Oleg pointed out a couple of days ago that configuration as code are also groovy scripts or maybe other things. So maybe we will have enough topics to talk about. And the configuration is called deserve. And I'm okay with that, but I would like to hear from you cloud native seek participants. What's your opinion if you feel like configuration is called is a is a part of cloud native or anyone anyone wants to give you to agree with the idea of having a separate special interest for configuration is called. So why I was thinking about having a separate seek? Firstly, jcask project has its own regular meetings every two weeks on Wednesdays. And as I really said, there are many other activities in Jenkins, which technically qualify as configuration as code. So it's, of course, groovy hooks, the entire pipeline and job they sell, etc. They are also configuration as code, whether it's the same sequel, etc. Another it's a separate topic and the configuration management tool. So they are a lot of them. And my point was that all of these topics are not really cloud native. They kind of prerequisite for Jenkins being cloud native, but they exist for years and they can be really used in common installations. So cloud native seek make confused particular contributors. Any other opinions? I think it depends on how much talks or you have or topics you have. I mean, if you don't have enough, it's maybe fine to add it as part of the planning. If you already have like, be by weekly meetings, enough traction, then maybe separate. Yeah, I believe that jcask itself has a pretty good traction. So we have about, yeah, maybe three or four participants at every call plus a number of watches after that. So for me, it already looks like an established seat. Yeah, I actually have not thought about what you have just mentioned, or like that. Well, Jenkins configuration is code is a tool that can kind of help with making Jenkins cloud native, but you doesn't have to have your Jenkins cloud native to his configuration is code or just DSL or anything. Right, but so yeah, it was my point, but I have to admit that these stories, and yeah, actually, whatever other story we do for plugable storage also can be used to outside or native. So maybe this point is not exactly valid. It's just a matter of placement for this story. Sorry for interrupting you. If you have enough, if you want to start with as part of the cloud native and then branch if there's a lot of interest, then that's fine, I guess. So Evelina, what would be your preference? I think they would kind of maybe transform our office hours towards a seek. Of course, we have things to discuss. But we for sure can find some time to have other things. And maybe it's also an opportunity for us to get more participants and more feedback. I yesterday had some about some UI issue, Jenkins configuration is code, and I'd love to have more people to present at some meetings. So and I'm planning to talk about it on Wednesday. But anyway, I shouldn't go into details. I think I prefer to under some of the objects. Okay, so what we could do, for example, if you want to keep the existing meeting time slot, we could just edit somehow here. So we had configuration as code into the list of seek interests, reference the existing JCASC subproject. And in the meetings section, we also reference meetings, which are already announced here. So in such case, you still have your own schedule, but it's referenced from cloud native seek. And in such case, you can just join the meeting sometimes and provide update to other participants. I think it would be useful for everybody. Yeah. Is everybody fine with it? There's no particular reason to have it not be part of this. No, I don't think so. It's just I sometimes I feel that topics maybe like very configuration is code specific. And that's or has I agree with the point where Olek presents his vision of this code. Yeah, cool. Okay. So can I say that we agreed on that? No. Okay. So yeah, let's keep it as is. Yep. Would you be able to do that? Yes. Okay. Okay. Once you create all requests, please ping me. You'll review and merge them as soon as possible. Sure. Thank you. Okay. Anything else on this topic? I guess no. So the next one is about configuration storage. So just to provide some background, I have created a JEP two weeks ago. So the idea of this JEP is to document how external configuration storage would look like in Jenkins. There are some pull requests from Alex who's also on the call. So yeah, there is a pending pull request to Jenkins core, which poses to move XML file in the hierarchy of XML storage, file storage, storage, so that yeah, it becomes pluggable. As you may see, I have looked at this pull request mostly due to binary compatibility issues, but I definitely agree with the idea that it's something important for Jenkins, especially if we talk about pluggable storages because external configuration storage allows to have some disaster recovery and generally you can get rid of many small XML files with configurations on the disk. So that stories like multi-tenancy also become easier. So I have drafted a JEP and in this JEP, I have tried to document my vision and actually I sent a message to the SIG mailing list, yeah, also two weeks ago. And what I wanted to do this meeting to just answer any questions, if any, and then maybe to sync up with Alex on reference implementations and on how we proceed with this JEP. So Alex, are you around? Oops, wrong button. Yeah, hi, I started looking at it and I'm trying to get some more hours from my employer to work on this. But now that the weather has improved drastically, I'll be able to do more work on my spare time. Yeah, that's good. Yeah, so regarding the discussion, I have started it in the SIG mailing list. So I've got some feedback from James, JC and Baptiste, but generally it's rather about what would be the priorities. I have asked several questions, which I propose to briefly discuss. Firstly, should it be a plugin or a library? And then, yeah, some implementation specifics. But I think that this one is a real key question to the current design, because if we move configuration storage to a plugin now, we get into all kinds of issues with configuration loading, because configuration loading obviously has to happen very early before the most of the configurations are loaded, because we need to load Jenkins config XML and other such files. So in my design, I assumed that whatever configuration storage implementation will be rather a library, not a plugin. So it means that you will not be able to install it from Update Center, but there are other ways. So for example, packaging, it or just adding a library to the cloud class pass. So I consider it as a feasible workaround until we have a pluggable core components story implemented. So this is an old story started by TK, which has been hacked during the Jenkins world hackathon one year ago. So generally there is a standard pull request, which needs review and a lot of testing to be integrated. But without this pull request, we cannot make this logic as a plugin. At least it's my understanding. Okay, so Jason, maybe you would like to just comment? Yeah, I'm not sure that's true. Well, I think at least we should be experimented with. I think you actually can have a plugin. You must if not all of the file replacement, there are probably some special cases, but yeah, definitely, I mean, definitely it's safer to do it as a Jenkins module. Yeah, so we already have Jenkins model packaging logic. So in the core, we have several components. Check code models. So yeah, that is a Cgd model windows agent installer, etc, etc. The most of these models could be actually converted to plugins at the current state. But yeah, we have an abstraction layer, which just gets packaged to the JAR file and which can be managed then in Jenkins. But yeah, generally you can just build a library. Okay, so Alex, since you proposed the original pull request, what do you think about the models? Is it reasonable to you or do you need a plugin? It seems pretty reasonable, especially since pluggable core component isn't available yet. Yeah, so yeah, I'm not sure what the current status of pluggable core components. Yeah, that wouldn't help here anyway. That would be for just updating the version of something that's bundled in Jenkins to begin with. To use a Jenkins module, you have to have a little step to create a custom R file. It's not hard, but it means you need to produce a special distribution. Yeah, with custom word packages, it becomes pretty easy. So just to show maybe not that new example, but yeah, what you can do, that you can just define the patches in custom word packages in location and it will add libraries, which will automatically become available in the class pass. Here I just replace remote, but you can add new libraries. And we already have one example of implementation for library-like thing. So what we have in the JEP is... Okay, so we have a QBify, it's an old implementation for fabricate. I tried to invite James to this call, but yeah, it seems he's busy. But yeah, what this thing does, so it's better to just take a look at commits actually. So if I understand correctly, this is an implementation which has been originally created for fabricate. So it's before Jenkins X started, and what they did here, that they just modified Jenkins to directly bundle Kubernetes client. And yeah, they replaced all logic in a pretty similar way like Alex did in his pull request. So there is already a reference implementation, which uses Kubernetes resources as a storage. So we could just decouple all these Kubernetes custom logic to a model and to get one reference implementation. And this is what I tend to edit to this JEP. So I assume there will be two reference implementations. Okay, what is it? Yeah, file-based storage, yeah, sorry, I'm lost a little bit. That's why we have table of contents. Okay, yeah, we have Kubernetes configuration storage implementation. I see it as just a fork of QBify and generalization it as a library. And Alex said he's about working on SQL implementation. So I edited here as a separate reference implementation which we could use. And the rest of the documentation is pretty aligned with the current pull request to Jenkins Core, though there are some compatibility requirements set. So for example, we need a file system XML storage, which is obviously legacy mode. And we also need some logic to ensure that people can migrate the instance, even if we do not provide common data immigration flow, at least in the current design. So yeah, three implementations, two for external storage, one for internal. And this is what I would like to see to merge these changes into the Jenkins Core. Does it look right to you or would you expect something else? I mean, I think in practice, most of the difficulty is not going to be just replacing, for example, the behavior of XML file, but testing all of the different corner cases about things that are not actually intended to be configuration or are written at the wrong times and things like that. That's going to be a lot of testing of particular scenarios and basically classifying all of the files in Jenkins home and trying to figure out what they're for. And probably doing a bunch of other core cleanup patches. Right. So yeah, this is probably the biggest problem of this defined design, because configuration storage EPA, effectively it presumes that we update logic for XML file and probably for other storage types, TBD. And this XML file is being used everywhere. So it's built XML, it's configured XML, it's whatever descriptor XML files being created. But generally, there are lots of use cases where plugin developers adjust cache data in XMLs on the disk and they use this thing as well. The advantage is that if we implement file storage in a way that it can decide on its own, whether the file should be persisted or not, that probably we could not care about this for now. And as an added value, you get an opportunity to offload any files. So for example, Kubernetes configuration storage could probably offload all XMLs, even built XML, why not? But yeah, for SQL configurations storage, we could offload only the configuration files, which we could track down because there is an extension point called saveable listener. So we can understand which files are saveable or not. Yeah, probably we can. So that files are saveable or not necessarily purely or at all configuration, they can be other kinds of runtimes. There are also, I think, some things that are saved that do not currently trigger saveable listener. Yeah, right. When I was experimenting with some bits, actually, I discovered that saveable doesn't really behave as you would expect because saveable listener gets involved in many cases, not when you actually say, not only when you save the file. So I was experimenting with some multi-tenant Jenkins setups and I hit that. So yeah, but whatever we do, we can introduce as many market interfaces as we wish. So it's also a subject for feedback. And if you want to just comment in the mailing list so that I reflect this comment in the design. So far, I just presumed that we provide XML file storage and extension point, not an extension point, but just obstruction layer. And then these implementations decide what to save on their own. So this is the current plan. Okay, so any other comments? Yeah, I will watch the recording and notes here. Alex, a question for you. Is the current information not for you to take over this activity for now and maybe to experiment with this public request? So just to see whether it matches your vision and to see whether it can be implemented? I think so, yeah. It looks pretty extensive. And if I can think of anything else, I'll bring it up in the Gitter Chat. Okay. If anything needed, just let me know and I'll try to process this full request. If you want, feel free to just push it on your own due to sponsors as well. So you are more than welcome to join this activity because you proposed the original public request anyway, and here I reflect many stuff from your original proposal. Yeah, so if you need to make some edits in the JEP, etc., feel free to do that. Okay. Any questions about this topic? If no, I propose to move on to the next topic on the agenda. It's about external build logging. So yeah, last week we had an update and the video recording is there. So we have presented the work, the design. Jesse has presented a demo of CloudWatch and Fluent Day implementation. I also demoed some core API changes. So there was an action item for everybody to take a look at the designs and to provide feedback. Maybe we could briefly update the status and then, yeah, we can discuss any comments regarding the implementation. So, Jesse, would you like to start? Sure. I can just give a brief overview of where things stand. So JEP 2.0.6 was sort of a preliminary change to pipeline logging. It doesn't touch freestyle or anything like that, specific to pipeline, which switches all of the log storage to use UTF-8 universally. All pipeline log storage? All pipeline log storage. Yeah, both for pipeline steps and the overall log is always stored in UTF-8 regardless of the platform. And then the main tricky parts there are dealing with shell steps and the like that run processes on particular agents that may use other encodings. So if you're running a or if you're running a batch step on a Windows server, it might be using code page 12.52 or whatever it is. So after JEP 2.0.6, any output gets transcoded to UTF-8 as needed. So all of that stuff has been merged and released. So I hope it's working. I guess the JEP actually needs to be switched to accepted or finalized or something like that since it's out there. Who's the JEP's Delegate Sam? I am not sure that was ever actually set. I know I was asking Chris K to try to pick a Delegate. I don't remember if the Delegate is set for it. At this point, it's sort of moved because it's merged. Yeah, so we definitely delegate TBD. So maybe it's something to lame to check. There will be a JEP status update. It was in cup tomorrow. So maybe we could take a look at this JEP and see what we can do to get it landed. It's actually, that was going to be today if anyone had any topics, but so far I haven't heard any things from folks. So I will, and Oleg will not, Koski will not be there. Well, you can add that. Yeah, you can add that topic to the office hours, I guess. So I tried to grant the right permissions to this document to every SIP member. So you could try. I'm not sure what it does because I'm a favorite to find Carlos in my address book. Okay. Yeah, Delegate and other JEPs. So just you also work on a JEP 210, right? Yeah, so JEP 210 that uses the treats 2 or 6 as a prerequisite. JEP 210 is basically a complete overhaul of how logs are actually, how build logs are actually managed for pipeline. So we'll go into the details of the historical decisions, but if you go into the specification, basically it covers four kinds of changes that build on one another. And the first change here, the invert control of durable tasks pooling. So that is prepared as a set of independently releaseable pull requests, which are undergoing tests right now. And basically, basically the purpose of that is to make sure that if you're running a shell step on an agent that any kind of cloud native build log storage that gets implemented is able to stream that log output directly from the agent without involving the master for every, you know, to handle every single line. So in other words, for allowing an implementation of logging to bypass the remoting channel between master and agent and have the cut out the middleman and just have all of the log text sent straight off to some other source. And then the other three phases are done as another set of pull requests that's undergoing review. And that basically in total allows or adds an extension point that allows the log sync for pipeline builds to be swapped out by a plugin. So we have a reference implementation that assumes that outgoing log messages are being streamed to Fluent D. And then you have to configure Fluent D to mirror all of those messages on to CloudWatch logs. And then if and when somebody browsers the build logs via Jenkins, then it uses CloudWatch logs API is to retrieve the content later. And I gave a, I guess it was last meeting I gave a demo of that running. Yes, it was the last meeting. Yeah, maybe the link needs to be updated because now we have another baseline. Yes, no, it's, it's in the Jenkins CI work now. Yeah. So probably I should also summarize my activities. So I'm working on a job towards seven and 12. So job towards seven is an ongoing patch to the Jenkins core. It's introducing API's for external logging directly in the Jenkins core. So as I said, the idea is to make storage reporters pluggable. So they do not go through so that logs don't go through the master, et cetera. JC does it on the pipeline level. And I'm doing it on the Jenkins core level. So we split to two pull requests and to two jabs external build log and support just minimal set of API changes needed to support that in the core for any job type. So it's freestyle jobs, which we mostly target and, of course, matrix jobs and all other types accepting pipeline. So there are significant changes in the implementations since the last meeting. So as we were discussing, there was a question whether we need a single abstraction layer class in the core or multiple ones. So finally, I implemented the thing. So now it has only one class, log storage. So there is no log in committed and log browsers in a separate way anymore. Then there will be also some changes in implementation to simplify that. And yeah, currently there is an updated pull request to the Jenkins core. After simplification, it became a bit bigger because I've also had some test automation and other stuff. But generally, APIs became more simple. I've got some another feedback iteration from JC, which I need to address. But yeah, generally, it's going forward. So I hope we get it landed towards the next LCS baseline. It's my plan. And regarding the downstream change, we have another job, which is 212. So it's external login PPI. It actually implements core APIs and also implements reach methods so that it transparent integrates with changes being implemented by JC in a job to 10. So here I have an external login PPI plugin and there is also a pending change. I also have outdated link. And there is a pending change, which will align it with core patches. So currently, all patches are passing for the tests. And logging works correctly. For my implementation, I have elastic search reference implementation. So now it shows logs. There are some glitches there, which I want to address. But before I will do that, my plan is to generalize some APIs, because we need to Json submission handling between implementations. And also, we need an implementation for eventual consensus and storage, so that we don't need to implement the engine multiple times. So these are the things which block me from alpha radius state of this external login PPI, as well as core patches release. But yeah, I'm pretty confident that I will be able to deliver them soon. So this is the current state for patches in the core. We still have an open question how we integrate JAP-07 and JAP-210 between each other, because there is some overlap in APIs. But yeah, I think that we don't need to integrate them in the initial implementation. And then we can do it in the follow-up. At least it's my understanding of the current approach. Yeah, so I wish we could get all these JAPs over the fence. So assuming that 206 is going to be accepted. So we have three JAPs. So this one is under review. And I guess there will be no major changes there, right? And yeah, these ones, yeah, they are also in flux, but yeah, I hope that JAPs themselves include all documentation required. So yeah, this is the current status. So I guess no cool demos today. There is still a lot of work to do, but yeah, we are moving forwards in all implementations. Any questions? I guess no. So should we move on to the next topic? Or does anybody want to spend more time discussing external logging? Okay, let's move on. So last meeting, Jeff Pierce proposed several bits to discuss. So he added several stories. Unfortunately, Jeff Pierce is not on the call. But what I would propose to do is to just to understand whether it would be in the scope of this EEC, or it wouldn't be another EEC. So maybe Carlos, would you be interested to review that so that we adjust the scope of this EEC and maybe clarify it a bit? So the first one, I'm interested in easy deployment via Kubernetes to AWS and support for Canaries. Yeah, for me, it's hard to comment whether it should be Jenkins Evergreen or Jenkins 6 or just Cloud Native. The deployment to a Kubernetes and AWS, yes, but that is part of what we are working on. So we have one abstract item here, replication. Canaries of Jenkins, yeah. Yeah, Canary upgrades, zero downtime. So yeah, this guy definitely regarding the rest of the topics, honestly, I have some doubts. So for support for Mac agents, I think that it's actually platform sick. So we have another special interest group which has been also created a while ago and we had a meeting yesterday. So this group focusing on all kinds of platforms, including JVM support, operating systems, architectures. So I guess Mac falls in here easily. So if there are specific topics for Mac support or whatever, I think it's rather goes to there. Unless it's something like Mac stadium or whatever is being used in the cloud. Okay, and this one, what would it be? Yeah, my proposal would be to ask pipeline team whether they want to create their own SIG. Yeah, because for me, it looks a bit weird that pipeline is one of the most active projects now, but yeah, there is no sub-project, no special interest group. So maybe it would be a way to go there because it doesn't look like a cloud native SIG to me. Or maybe just developer, at least. Yeah, I would imagine pipeline sort of touches so broad a section of Jenkins at this point that having a SIG for it might be a bit like inviting everyone and everyone to be part of it. Like, do you want to be part of Jenkins? Well, you're doing pipeline. Yeah, developer might be preferable. Yeah, that's all I think. Yeah, maybe when Jeff comes back, he will communicate it to him. So I guess it should address questions from Jeff. If you want to discuss something, we still have 10 minutes left, so just drop your items to the agenda and we will discuss them afterwards. And yeah, just to complete the agenda, what about regular meetings? Is this slot comfortable for everybody or should we consider another one? It feels that we have enough content for big weekly meetings at least. I'm still trying to get used to 7am and earlier meetings, but I think I'm not sure if I'm alone on that one. Yeah, in configuration as called plugin, we have meetings at 7am, maybe you see? Yeah, midnight. Yeah, yeah. Yeah, so yeah, this is a problem of all distributed communities. Actually, if I can comment, I was I was thinking if we have weekly meetings for cloud native SIG and we find a time slot that works both for Europe and states or other people shift and have the same time slot, but other every second week or configuration is called. Yeah, if we find a time slot to be just suitable for the entire planet, yeah, probably it's a noble price. Make the money off of that. We'll just establish that it'll be worth millions. Yeah, right. Usually we do such meetings that's something like 9pm VTC because in such case probably Europeans are still around and people from Asian and Pacific. Yeah, they probably can wake up by this time, but it's a bulk assumption as well. So what we could do, I could create a doodle so that we discuss the regular time slot. Oh, okay. I'm not a native speaker, sorry. Native speakers do worse, I'm afraid. Yeah, exactly. When you say bi-weekly half the time, are you twice a week? Okay, every two weeks. Yeah, exactly. So what I propose, I create a doodle? I won't. I'm fine with this slide. I'm just I'm just I'm just winging. All right, let me winch. Yeah, right. Oh, yeah, I will create it. Regarding the next meetings, any topics to discuss, by the way? So one thing that we will definitely have a meeting on September 17th in San Francisco. So yeah, since many people go there to Jenkins world, we could spend some time and have a face-to-face meeting of the SIG. It's something work and progress. But yeah, maybe you want to meet before. So today is 16th. What about a meeting around August 30th? Well, if it's afternoon, I have to run five cases. So I would prefer to be at the meeting, but I will create a doodle anyway. So I cannot guarantee what will be the final date. It's just approximately August 30th. So we will definitely have something to discuss for external login again, maybe for log storage, maybe for some other stories. Then Jesse and Carlos, we're working on an artifact manager. So maybe it would be interesting if we could present something in the SIG. What do you think? Have we not presented artifact manager already? I guess not. Was it recorded? So regarding that, what we could do, we could have Jenkins online meetup and just present it to everyone. It would be one approach. But yeah, I do not recall that it has been presented in any way in the Jenkins community. Obviously, there were a lot of presentations in between us, but yeah, I do not recall any public presentations. So would it make sense to do that? Or maybe not? Sure, anyone's interested. So what we could do is to just do it in a job. Yeah, we can do jobs, but you have to schedule. You generally want to schedule those two weeks out. So now would be the time to schedule it, I guess. Okay. Yeah, then I will take it offline with Carlos, Jesse, and maybe we schedule something. Right. Do you want to, I guess will it be, will artifact manager be far enough long at that point to do a job for it? Yes. Already released. Okay. So it's staged. Why have I thought about that? Because yeah, we have it in the cloud native SIG scope. Sure. Or maybe it would make sense to spend some time on that. It would be also interesting to talk about credentials actually, because yeah, there were at least two questions in the cloud native SIG room over the last month supported. But yeah, I will talk to people, maybe we could get it done. So if I get something like that, then we could do artifact manager and credentials storage in a single job. Yeah, that could be interesting. So then I will start to doodle. Then according to doodle, we schedule this meeting. And yeah, then we will try to have something more less regular. So any other topics? I have a question about configuration storage. If we sell configuration into Kbass, do we need storage in local file system or we don't need it? So the idea of external configuration storage and actually of all portable storage stories that when we implement them, we move configs outside. So it means that there will be no configurations on the disk anymore. Because otherwise you could just use plugins like a same sync configuration, which actually also push configuration externally. But yeah, they still rely on the disk. So they don't address external storage problem in general. And the idea here is that all configurations are moved outside. Obviously it causes some issues. I just second time I think now. Yeah, so in the JEP there is a backward compatibility section. And this backward compatibility section explicitly says that for new storage implementations, some things may be incompatible because the implementation will change. So some things on the list is disk usage plugin, job configuration, history, same sync configuration, also backup plugins, especially Google Cloud backup. So these plugins may be incompatible with the new storage implementation. It's TBD because all APIs will be compatible. But yeah, I'm not completely sure that we will be able to retain compatibility there in the same way. So for example, job configuration history, how it would work now, it will be still receiving saveable listener notification, then it will be retrieving XML file. So in order to retrieve XML file, it will be using the compatibility layer here. So yeah, there will be a beta file method, which actually presents a configuration file on the disk, even if it's externalized. So maybe config history will be working, but it will be definitely non-efficient. So yeah, these plugins may just break. Does it answer your question? Yeah, so what we require that for existing legacy storage, we retain full compatibility. So effectively, nothing changes in all use cases, including some not that common use cases. For example, there are plugins which make both the assumption about location of built XML or config XML on the disk, which is similar to what we hit with artifact storage. So yeah, these plugins may be affected by a file system-based storage. Nothing changes so that it means that any upgrading instance will behave as before. But yeah, for new storage implementations, there is still some research and testing to be done because it's just hard to predict how these plugins will behave with all this stuff. So disk usage plugin, yeah, it will be reporting accurate disk numbers because it will be pulling data and creating the data it needs, which yeah, it sounds completely weird, but it's how it would work in this case. Okay. Cool. Yeah, so any other questions? Okay, it seems there is no other questions. So then thanks everybody for participating in the call and thanks to everybody who was watching this call in YouTube. So this video will be processed and published maybe in 30 minutes. And yeah, if you have questions after the call, just join our GitHub chat or mailing please and ask there. So yeah, that's it. Thank you again and I'm stopping the broadcast.