 Cool. So recording and let me share my screen here. How's everyone doing? This is the get credentials binding project for August 4, 738 and India standard time. So I asked a question and I forgot I didn't introduce the meeting for the YouTube. And I'm doing great, although I have to admit I did not complete the things that I had envisioned doing in terms of preparing to test. So my weekend was spent on other Jenkins things not on, not on our project. I'm not sure why it was not connecting. It will give me a time out exception, but it was not working for me. The library is able to read the keys, past is collected or not past is collected, but I was not able to make establish a such connection. I believe has found another library. Would you like to talk about the library or shit. If you're speaking. No, we haven't. And oddly enough, I don't have the thing that would allow me to ask him to unmute. I thought in the past, I had had an option that would allow me to invite someone to unmute their there, but I don't see it here so maybe I'm just remembering badly or maybe it was only from a webinar. Oh, I got it. Oh, looks like he joined as another account. Yeah, maybe audio problems. So you should ask something. No, no, I was just saying that you found another library to do the work right. Oh, yeah. Yeah, I was successful to create. I was able to read the RSA key but not able to establish a such connection. Yeah, so I found out that the fingerprint of the private key that is generated using the decrypted private key that is generated using the SSJ library is different from the fingerprint of the private key. That is, you know, encrypted. So I think due to that, we are not able to connect. So the keys are different. So the server won't able to recognize a match for that private key with the public key. Yeah, I just wanted to ask before you introduce the new library. Did you ever try to read the keys and use the client called SSH client provided by SSJ to connect to a remote server using any of your SSJ keeper. You are saying to that I should connect to the SS server that is good to use using the SSJ library rather than commanding. I, yeah, I mean, I thought that the fastest way to know if SSJ will be able to decrypt the keys correctly and then, you know, it is going to provide a key which I'm going to work is to use their own client to establish a connection with any server, any, you know, any place which has 22 port over and you can establish a connection with it. If that is possible, then those keys, you can be sure that SSJ is not creating an issue for us. I mean, but you've found a new library, right? And, yeah, and you were, yeah, please move forward. The new library is, you know, the Apache SSHD. So even Jenkins has a plugin for that. It's called SSH server. So currently I'm facing a dependency issue in the grid plugin due to this, because the SSH server is an implied dependency. And this, I don't think that there is any direct, you know, define, you know, explicitly defined dependency or SSJ of the SSSD. So, and it's using a version that is below 3.10. So, and the code is very much different from the one that is that we want to use. So, have you used it? Have you used this library to use RSA pastries, connected keys and then work with them? Were you able to do that? Yeah, it worked with them. And I was able to establish the connection as well. But the issue I'm facing is, you know, the dependency issue. So I think I mentioned it on the GitHub chat. So what should we do next, right? Should we define this dependency explicitly in the plugin? And if we do that, then we have to, you know, increase the Jenkins version. More than 2.0 above. I think it's 2.8. But currently, the Jenkins version is 2.2, something like that, or 2.6. Not sure about that. Are you saying that a newer version of Jenkins Core has the version of SSHD that you need? Yeah. Oh, good. Okay. So, so that's a great excuse for us to, and what, what version of Jenkins Core has that? It's 2.0. Let me check on the GitHub chat, I mentioned that. Wait, 9.1. Yeah. 2.289.1 or 2.282. Okay, so, so if we upgrade, upgrade our dependency to require Jenkins 2.289.1, then we get the SSHD that you need. Yeah. And if we explicitly define, yeah, sorry. No, no worries. Yeah, and I think, I think you're saying that either one of these SSH server plugins would work? Yeah. So these have the, the version of the Apache SSHD library that we need, and that's supposed to decryption of the OpenSSH piece. Yeah, so then it seems like a simple answer to me, we upgrade the minimum Jenkins version to 2.289.1. I like that. I mean, preference mark on whether they pick, we pick, so it looks like there's two plugins, there's 310 and 304. One of them is a higher Jenkins version and one of them is a slightly lower one. The other ones 2.282. Not sure if you have any. I've biased in the past towards choosing an LTS version. And in this case, I don't think we've got a lot of users that would be on 283 through 289, but not on LTS, right? So for me, the preference is LTS. Yeah, that makes sense. I mean, weekly, you should upgrade weekly actually, because for instance, 2.300 has a security fix in it. And if you're, if you're choosing to run weekly, but you're still on 2.282, you're in danger. So, but if you're on 2.289.2, you're not in danger because the security fix is backported to it. So making it 2.289.1 seems very reasonable to me. And I knew we knew we were going to have to eventually upgrade. Yeah, there will be people who say, but I want to run the credentials binding without having to upgrade to 289. 289.1 and the answer is sorry, you can't. So, so harsh should it feel it sounds like then what your technique would be is you'll set the Jenkins version in the in the palm to be 2.289.1 and then you can use that that library that we've that's available. Certainly there will be people who say, Hey, I cannot upgrade to 2.289.1 and if, if we had for instance to do a security fix, we may have to deliver multiple versions of that security fix so for people who can't go all the way to 2.289.x, but I'm fine with that. We've done that before. So the get plugin already rely on SSH server plugin. I don't, I didn't think it did directly but let's check I mean that may be a circular dependency I'm not sure what that will mean, because I would expect the SSH server plugin probably relies on the get plugin. Yeah. So if we look at just look at get plugin. It's not in there. And I looked at get client plugin and it's not in there. Oh, I'm not sharing that screen. Sorry. Okay, get get server does not depend on the get plugin, at least not directly declaring it. Oh, that's fun and the get server plugin requires Jenkins 2.289.1 as well. So we're in good company harsh it. So one question I have is, do we want to take a dependency on the SSH server plugin. So, so when I look at, I'm not sure it's SSH server plugin. So you take a dependency on org dot Jenkins dash CI dot modules SSHD and use that dependency so I think now that's the SSHD plugin that you're looking at. Okay, yeah. Okay, what's what's the what's the body called the address of that thing the coordinates of that thing you're looking at is it or dot Jenkins dash CI dot modules. Yeah, it is okay so that's the thing that that SSH server or that get server plugin depends on. And I think is so harsh it is, is SSHD from Apache or from this code the thing that provides the method you need, or do you need something that's actually in the get server plugin. Actually has the, yeah, so we see that the online 53. It actually has the latest version of the library that we need, so either we can depend on it directly or we could depend on the. Oh, okay, so all, all you need is org Apache SSHD. Yeah. Oh good then we just should depend directly on that. And if that means in order to but in order to make that load then we have to increase the, the Jenkins version number to 2.289.1. Yes. Okay. Why do we need to increase the Jenkins number to this is this is an external dependency right when external and if you're choosing to use SSG cool. No, we have to increase if you can see online 34. But also if the, you know, define external dependency that is SSHD Apache, then in the mavin dependency tree, it will be used rather than the. It will be used in for the SSHD server or in a plugin Jenkins plugin. Instead of because currently that plugin is using the library version 1.7 so after we only define it, it will be updated to 2.7 so in, you know, implicitly we are changing the library dependency of that plugin only. No, but you can exclude the depend since you're going to directly use this library this dependency then you can exclude that from the plugin, which requires a newer Jenkins version and then you can use directly this I don't understand why would we need a new Jenkins version for that. I mean, if you're excluding it from that plugin and using this instead. I mean, or the library that the plugin has. Yeah, SHD code. SSHD code. This is the exact. This has the code. The functionality you need to. So, back to the shop's question but to ask it in the form of a question is or dot Apache dot SSHD dot SSHD dash core version 2.7.0 enough for you to have the API you need harsher, or do you need something that is it is. So then I'm prone to agree with, with Rashaab that we should at least try to just depend on that library if that still requires us to upgrade Jenkins dot version that's fine. It's acceptable to do that absolutely. But I would rather directly depend on that Apache library, then depend on a plugin that includes that Apache library. Yeah, because what'll happen is, if you depend on the plugin directly. When someone wants to install SSH, or sorry, when they, they will need to, they will basically get SSHD plugin installed to, along with the get plugin. Well, and I think SSHD plugin is an unavoidable plugin because it was bundled in core. Oh, is it? Oh, okay. I think so. I think there's, I think SSHD plugin is one that was discussed just recently on the mailing list with. Wow, why is this there I don't need SSHD and the answer as well because it was bundled in core and years ago and thus it's still its API still has to be available in core. Okay, well, ignore me. Even if you see the effective form of the plugin, you can find that plugin. The SSD plugin is defined, but the version is changed. Sorry, you're saying that the Apache SSHD core library is already in the, the get plugins dependency tree. SSD plugin Jenkins SSD plugin. I was talking about the Jenkins SSD plugin on the actual Apache library. Okay, so, so I apologize I must have missed a word so the Jenkins SSHD plugin already includes this and I think that's what we're looking at right now right yeah that's what what Justin's showing us is that that Apache library is included inside the SSHD plugin agreed, but my thought was, if all you need is that Apache library, then all we should depend on is that Apache library with its specific version number. Yeah, the plugin, you know, implicitly depends on that SSHD plugin. So, it, oh, it does. I was saying that in the effective form of the plugin, you will see the SS Jenkins SSD plugin dependency available there. The version is, you know, low, lower. We can overwrite that right. We can say that we don't want that we don't want SSD core from that plug in instead we'll use our. Yeah, but but I think I think, oh, go ahead, Harsha. That would mean that we are upgrading the version of the core only like we are not replacing we are upgrading the version of core for the Jenkins SSD plugin. How that could mean that it was the SSG plugin comprises of and it's not just SSD core right. It's it's a collection of multiple libraries. Choose to use a new version of another library. How would it increase the version of SSG plugin? Because in the dependency tree, if we look at the library that you know the dependency that is on the top will be used rather than dependency on the bottom. So the plugin that is using a dependency of the lower version will be upgraded to higher version. So explicit, so I can say implicitly we are, you know, using a lab core SSD core of version 2.7.0 only instead of the lower version that is 1.7.0. Ah, look at that. Okay, I think I understand what you're telling us, Arshad. Okay, so I did what you what you described which is Iran maven dependency colon tree. And in the output of that, there is a dependency on org dot Jenkins to CI dash mod dot modules SSHD version 2.7, which then itself has a dependency on as on the Apache library core jar or core SSHD core 1.7.0. So if we upgrade, I'm not sure if this actually works from maven. If I force it to Jenkins 2.289.1. Does it then already tell me hey, we have a newer version of SSHD core. Just a minute. I tell it to depend on to require Jenkins 2.289.1. It doesn't even have a dependency on SSHD core. Interesting. You've done the experiment, Arshad, and you said that by upgrading the by forcing the Jenkins Jenkins that version to 2.289.1 you get the you get access to the SSHD core library version you need. I was saying that I, I was defining the dependency explicitly. And as I also pointed out, I was defining the Apache SSHD dependency or explicitly both. I mean, I was trying to experiment that. So that will indirectly tell us to upgrade the Jenkins version. I think, I think that's the right approach. I think we should accept that in order to get this functionality. They have to at least be running Jenkins 2.289.1 because we need the version of SSHD core that Apache provides, and that is, that is available only beginning with 2.289.1. That's that's great. Arshad, with this library is was there a specific algorithm even you face an issue with or this library solves the problem we faced with SSJ. If we are using this library, so the working keys are actually in open SSJ format, they will be, you know, RSA, EECDSA, ED25519, and yes. So actually all the algorithms, encryption algorithms that are supported by the SSJ keys in utility and they were working also. That's great. We don't even have to use another library to do that. Yeah, that's great. Yeah, I was thinking of, you know, using the there's a PEM format support for PEM formatted keys or PKCS8 format keys in the lab in that library as well but the there's an issue opened on the, you know, mailing list. That says that the PKCS8 PEM encoded key is not yet supported. So as I have to switch back to BOMC Castle plug-in. Okay, you're talking about PKCS8 encrypted keys. PEM encrypted, yeah. PEM encoded keys. Yes. But they are not supported in the library SSHD library. But so my question is that do we want to first focus on the algorithms on the use cases which will be 90% of what we're going to serve or I'm not sure how many in how many cases we're going to see the PKCS8 encrypted PEM encoded format keys. I was just doing an experiment with that because SSHD library has a lot of functionality and a lot of support also so I was trying to figure out if I could mostly use all of that and instead of depending on the BOMC Castle API but it was not supported in that. Yeah, I mean, I think like, like we talked about the last time and you probably could start with, you know, just loading in support for the top ones. And if we find that, you know, PEM supports not working, we can document that as unsupported and that's a geo ticket or something like that for later. Okay, so I mean, should I then use the SSHD plugin or well actually we use both then we have all the keys working on all formats. When you say both you mean SSD and BOMC Castle? Yeah, both the library. If there's no additional cost of using BOMC Castle then we should. I mean, if everything is working then why not? Yeah, I guess from a process and work perspective, like if it's easy for you to get them all in one PR, like, maybe that's okay. One thing I could see is like, if it's hard to start getting the bouncy castle stuff, then maybe it's a good idea to start with the most critical algorithms to cover and then do those in a PR. But then I also realized there's some testing and stuff like that involved too. So, those are the things that I would think about for that, if that makes sense. Yes. So, one doubt I have is that should I, you know, explicitly define the SSD plugin that the Jenkins module SSD plugin right in the form of the plugin. Sorry, ask that question again it was should you define explicitly define the version of SSHD dash core in the palm file for the get plugin? On the SSD plugin should I explicitly make a dependency in the get plugin or should I use the Apache SSD core? For me, I would use the Apache SSHD core to make the dependency as narrow as possible so as narrow only as wide as you need. Yeah, and also explicit on what you exactly need to. It's kind of a nice practice. Right now, now no guarantee that that still that that will work because of the nature of that SSHD plugin being a formerly bundled plugin, you may still be forced to set the Jenkins minimum version to 2.289.1. Yeah, I will have to. Okay, good and and and just to reiterate that's that's perfectly okay. We want people to upgrade and this is a good motivation for them to upgrade. Cool. So I'm going to put that in our notes here to switch to minimum Jenkins. I'm going to go to 2.289.1 and then depend directly on SSHD core 1.7.0 and you're going to do that in the get plugin, right? No, it will depend directly on SSHD core 2.7.0 I think it is the current version is 1.7.0 and that's in 2.7. And then that'd be in the get plugin, right? Not in the get client. Yeah, and it's SSHD, Justin, just that extra H. You got me. Thank you. We can do it. I have a question. I was also looking at the dependency graph and I can see as I mentioned that in the Jenkins core job we're getting SSHD core SSHD plugin as a test scope. Can we not remove this altogether? Do we need the SSHD plugin for anything? Do we need it? Yes, in order to run tests. Is that what you're saying is that it's a get plugin test dependency? Yeah, okay, we use this. I just thought that this is being bundled but we don't use it. No, well, so the get plugin definitely does not bundle the SSHD plugin. So I'm not sure maybe you need to explain to me more clearly what you're meaning by bundling. Do you mean the Jenkins core bundling it or the get plugin bundling it? No, Jenkins. We're getting this from Jenkins core, right? Right. And is the get plugin using this library? As far as I know it is not today. Yes, my question was that can we not just remove this, exclude this and use SSHD core directly? The plugin altogether. We may be able to but I don't know what that will do on a Jenkins 2.263.1 when the get plugin wants to load SSHD core 2.7.0 but Jenkins has already loaded 1.7.0 of the SSHD core plugin because of the built in SSHD plugin that's using SSHD core 1.7.0. So I would expect Jenkins will say I've already loaded SSHD core. I refuse to load another one. And then we'll be stuck because get plugin won't be able to find the API's it needs that are only available with newer than 1.7.0. Harshit, I think I was describing it correctly. Do you need to correct something I said there and tell me no mark you made a mistake. Mark you are right. I actually have faced the error myself. So I know the error will be no class no class found exception. That was coming because the library that was loaded was one SSD core 1.7 and I need the libraries of SSD core 2.7. So there was I was stuck for, you know, for one day whole one day speaking on what why why is it this happening. So then I figured out that this for dependency issues. Great. Thank you. Well, and welcome welcome to Maven's dependency tree. So that sounds really positive. Does that kind of help you with all the questions you have for for that specific one harsh it. Yeah. I mean, I am currently focused on this issue. So, I have not even booked on the issue. I'm just focusing on this issue. Sure. But this issue is, I think this is a major victory that you've you've reached as far as you have, keep going. That's great. And did you see harsh it that we have one more person who's interested in in the work and did some experiments we can ignore their results, if we need to but I was delighted it's great that we got people who want to use your code. I mean, I missed it. Who is it. So I had closed a bug report from someone who asked for the ability to add a special case to the Jenkins get publisher that would allow it to push push multiple tags. And the person actually submitted a poll request to the get plug in saying hey here's the implementation, and it was a good implementation they did a good job with it. And I closed that poll request and said nope we're going to do this with with this Google summer of code project and so the person then went and took the latest build from the poll request and tried it. So you'll find a comment on the poll request where the person says, Hey, it didn't work for me here's this but given what you're working on with SSHD. I'm not at all surprised that it didn't work for the for the for this person. I just want to see that comment and the progress that he has made. Yeah, yeah, I will make sure that I will make a reply to that. Yes, yes, that's it. That's the logo. Yeah, and I pointed the user and said hey use username password it works now. And he said no I have to have private key. And this is his effort trying to work with private keys so that's great. Nice. It didn't work. That's, that's not a problem. The fact that we've got one more person trying it is really great. Mark, did you have any opinions on the PEM format and including bouncy castle in the in the next PR or do you think that should be like a separate PR what's your. Yeah, I'd rather. I've never once in all the time I've dealt with private keys never once used a PEM format private key with with SSH. I don't, I think that can be separate let's get the other things working and be confident we're tested and we've checked them in all sorts of configurations. So at least for me I'm not worried about PEM format right now. That's on okay her shit. Yeah, great. You get to deliver more value faster. I will try to make the PR mean I will try to make the PR buy today for the open SSH format support. Let's hope for the best. Oh, are we also are we also covering the cases or thinking about the cases when SSH agent did not produce open SSH keys before this format was adopted. I think, I think we'll get that as a result we'll get test cases for that, based on the, the, the tests that I intend to run with generating keys on sento seven, and on Debbie and nine, because those are relatively older SSH implementations. So I think we'll get test data. And even what I hope the test data will then tell us is, yes, it works with the SSHD core and oh it does not and if it does not we will probably just initially say, known to not work, you must use a modern key format. Does that does that seem okay to you, Richard. Yes, I think. Actually, I think I'd rather look at what her sheets code is I look at his implementation. How it depends on these algorithms how it functions with these algorithms visit. Do you have to write different code for each of them. Like, is there is different. I'll just look at the code. And I think mark that should be fine for us that we run those tests and then we figure out. Any other doubts or topics we should talk about. Yeah, so the grant here that I have opened for the SSH binding I should change that for open specific open SSH format. Arshad, I'm not sure I understood the, all the all the details with the audio so were you asking should you alter the current pull request to only support open SSH formats. Yeah. And I make a PR for the ones you got. For me, I think that that seems reasonable. If that's, if that's easy enough for you, I, that'd be the way I would go. If, if you find it easier to combine them into a single PR, and still allow us to do allow me and others to do testing. Yeah, I agree too, but for me a separate PR with the bouncy castle work feels better, just because I want to be sure that RSA and ED 25519 keys work, because those are the keys that are important to me. If I remove the bouncy castle, they are like four or five lines of code that will be affected. I mean, that's not even a factor. Okay. Minimum there for four lines of. Well, then, then it may not be worth your spending the time to remove it that that's entirely up to you. I just don't want to be blocked testing RSA or ED 25519, because of a format that I don't use now other people may have a very different opinion and say hey I use PEM format all the time but I truly do not recall ever using up. No, maybe I guess I did use one once because some of the cloud vendors give you a PEM format key. So I guess maybe I have used one. But it's very rare for me. Yeah, I have a reference. I'm sorry. Do you have a preference on which way to go like whether to include it or not or. I think I want to go with the included one. Great. Yeah, and now that I look at my config file, I was just wrong. When I connect to the system 390 mainframe that IBM has lent to us, lent to the Jenkins project I'm using a PEM format key. And it works great. So yeah, I think if we include it we can always just declare that, you know, something's not supportive. You know, even if it goes in with the PR, right. Sounds like we've got a path forward. Anything else we want to talk about, or any other doubts. Good. Your shot mark. I'm good. Great for me as well. Thanks harsher. Thanks very much and congrats on finding the solution to the SSHD core challenge. Yeah. This is awesome. Yeah, it is. Great. Well, we'll close it down early then. Thanks everybody. Thanks all. Have a good ones. Bye.