 Welcome, everyone. I'm Mark Wait. I'm here with Jeff Thompson. He's going to present the fine sec bugs and how we can use it to identify and help us prevent security issues in Jenkins core and in Jenkins plugin development. This meeting is being recorded. It will be posted to the YouTube Jenkins channel in the online meetups playlist. Jeff, go ahead. Okay, thank you. So yes, I'm Jeff Thompson. I've been using Jenkins for many years currently work for cloud bees. As you can see here, I've been contributing to as part of this to Jenkins core Jenkins security for a few years now and I'm also the remoting maintainer. Make sure the remoting and the agent stuff all works. So that's some of my background. What I'm going to be talking about here today is another project I'm working on in Jenkins community in general to try and improve our quality and our security stuff about adding Jeff. Sorry, you want to jump to next slide. One of the challenges will be how people ask questions. Yes, could you show that while Jeff will be right onto his intro in just a minute but we're going to use the platform special interest group Gitter channel. Okay, Jeff, go ahead with your intro now. This is the channel for question and answer during this session. If you'd like to type them. You're also welcome to unmute yourself and ask questions verbally we don't mind if you if you interrupt. This is we expect it to be a rather smaller group and we want to be to be conversational. Go ahead Jeff sorry for the interruption. That's fine. I've been working on this project to add the fine sec bugs tool into our existing spot bugs tool and to spread this throughout the community to catch problems or potential problems earlier and help improve code quality. So we chose this platform sick just because it seemed like a good place to dump this conversation. There's a little bit of background here spot bugs, many people are probably familiar with it but if you're relatively new to Jenkins or Java, or other things you might not be. It's a static code analysis tool for Java that you can hook into the build process. And it analyzes class files and has pattern matching rules to try and detect certain classes of problems that are fairly common and may cause issues in your Java code. Now, in a way it's been around for a long time, because it originally started as fine bugs which is almost as old as Jenkins and fine bugs then kind of collapsed a few years ago and relaunch the spot bugs. And I mentioned that because you see references to both in the Jenkins palms and all sorts of stuff. Current usage references spot bugs, but fine bug settings in Jenkins apply the same way it was, it was hooked in to move forward cleanly. So that's standard spot bugs is standard in many or all Jenkins parent palms, like plug in core the libraries many other places. It's been used there for a long time and builds fail. Now if you've ever submitted something contributed something build fail because of a spot bugs error, you go clean it up and get it. And that gives us a little better gives us some better quality and some cleaner code moving forward and may help us eliminate some actual errors. One of the things to know on this as a developer is that it's configurable and the easiest way to look at this is if you're working in a project. If you look at the effective palm and look in the properties or looking things related to either fine bugs and spot bugs, you can see some configurable elements of this. There's how how hard it tries, how the threshold for whether or not a report it's a reporting threshold. So it says, you know, I'm only going to tell you about things that I'm fairly confident in, or I'm going to tell you about everything whether or not I'm confident in it. There's some other configurations on that that you can see. So, find security bugs or find sec bugs is a plugin for spot bugs. So it's actually integrated into spot bugs. And what it does is I adds another 135 rules or vulnerability types. And these are all target around security issues, unlike the spot bugs rule set which is just more general some of those could be security. But this is a set of ones that are specifically about things that come up as security concerns. One of the things to be aware of it with it is that it's, it's by design the rules are written to be about Java web applications. So a lot of its rules are things that might be okay if you were running a local console application of some sort, but in a security application, maybe a problem. So, just as an example here quickly, it fits quite well with Java or sorry Jenkins, because it's a security application or web application in native security. It does a few things differently. So there's some oddities and some places where we need to kind of tune find sec bugs maybe. But like with the remoting side and the agent, there's some things where it's like, yeah, that doesn't matter because it's agent and the administrator is launching it and these sort of things. So you do see that that focus on Java web applications in it, which is a reasonably good fit. It's also focused on a lot of the things in the security community. Oh wasp is one of the big repositories of information about it they publish a top 10 list of what's the biggest vulnerabilities. And so find sec bugs tries to fit into that includes references to oh wasp in their descriptions and to the CW the common weakness enumeration in in the descriptions and the explanations. So there's really a lot of a lot of stuff to help you figure out what the issues are, although some of them are still quite complicated to understand. And there's the URL for find sec bugs project. It's also on. It's on GitHub you can see what it all is but basically there's the page for it. Let's go. So this is the proposal that I've been working on now for several months here. Well, yeah quite a while it's it's back into like November I think that I first started trying to get this going. So what I want to get to here coming up, not, not immediately and it's kind of piece by piece is to add the find sec bugs configuration into the parent palms and Jenkins. I've had some stages coming along for that, which I'll show you later and preparation for this, but I want to put the configuration into the parent palm for plugins into the parent palm for Jenkins into the stapler module parent palm. Any other places that we find it like that. This means that when this is goes into there that all projects that update to that mess to that new version of the palm. All of them will then automatically ran find sec bugs, in addition to to spot bugs. And way before I forget about it. Let me mention that there's an oddity. I think it's wrong in the Jenkins plugin parent palm that if you do skip tests. It also skip spot bugs. And this one has caught me multiple times. So, yeah, you have to, you know, really kind of fiddle around with that, or else just run the whole thing in order to see the spot bugs and, you know, also the find sec bugs actually run. So just be aware of that I've sometimes, I've seen people sometimes do well, this didn't just work fine on my machine and then I pushed it into the Jenkins CI and now it's failing. Well, that's the spot bugs that you're probably skipping tests one of the last times you ran it to save yourself a little time. So just be aware of that one. I will point out here also that on item number two, the Jenkins parent palm there is already a poll request for that one. And that one actually preceded my work. It's, it was put up there by somebody else in the community who started fixing a few things in Jenkins based upon the results that he found there. And trying to get that one moving forward actually maybe that'll be in the number one spot because I'd want to get that one moving forward soon. Let's see here. So, when I first posted about this, and wanting to move forward on this idea in the Jenkins developer email list. We got some some good, good healthy discussion, and some concerns were raised. And I've distilled some of those concerns along with some other concerns other people have had and other things. Under this page I'm calling them analysis conundrums they're kind of paradoxes or conflicts. There, there's things where there's a reason to do x and there's also a reason to do why. And so, I'm going to go through and talk about some of them and what, what we need to do with them. So one that comes up a lot is what is the best time to add new analysis capabilities like fine sec books. Always that a long time ago is the best time to do it from project start is always the best time. But at any point where it's now, the longer ago that it was was always better. And doing it now is always more difficult because then you have legacy code, you're going to have findings that you can't change or that are difficult to change. And maybe then may not be an issue. But you could have fixed them very easily long ago and now introducing it at this point. It's more things that you have to suppress and deal with. That's, that's not, that's not an issue with fine sec books. Specifically, that's an issue with these sort of tools is sort of analysis is sort of detections. So, the other alternative when you get to this point which is kind of the conundrum of it, as you say oh it's too difficult to add now we won't do it. So we'll just delay. But of course, a year from now, the best time to have added them. Well, a much better time to have added them would have been right now than a year from now. So it's, it's, you always kind of pay this price unless you did it right from the beginning, but the more that you wait, the worst it is. So proceeding forth and doing it is still better than saying well yeah it's too much work we can't do it now. The next one that I want to talk about that always comes up with these sort of things as issue detection. And this largely comes into in the two categories that I've listed here. Tools like this don't find everything. And as a matter of fact, in Jenkins, we have some specific types of problems that crop up a lot that this tool doesn't find. And so it's like okay yeah it's not worth it because it doesn't find everything. And the other side of that is that it finds things which are false findings they're not really an issue. And so you spend time trying to do something with that so it doesn't find everything but it finds too much incorrect I mean you playing both side of those. On the other hand it does find some things. And so there's value in the things that it finds. And there's value in it, forcing us or helping us to look at the types of things that it's reporting about to try and make improvements as we're moving forward. So as I'm pointing out in this next one it's always difficult with legacy outdated code to introduce these sort of things. And you always get to this kind of, well it's not finding anything on my legacy outdated code that I can do about so what's what's the value of it. In reality, most of the value from these sort of things isn't necessarily in finding things that's already there like a spot bugs. When I've added spot bugs, or find bugs predecessor before to projects I've done it several times. But what really finds things that are value I mean the percentage of actual interesting hits is is low. But what it does is it starts pointing out areas where you can improve the code. So you don't introduce regressions. And then as you're introducing a new thing. It says okay MD five is a really outdated algorithm don't use it. You go, I don't have to use MD five right now, I can use something better, and you don't add even more of that, that, you know, technical debt into it. So that's kind of that issue of existing code. There's another one which is separate from that which comes up particularly in security domain. Which is about this openness. So if we run fine sec bugs. It's possible that by running that on Jenkins, we will find real issues. And by doing that. People will become more aware, you know, attacker could learn something about how to attack Jenkins from running fine sec books. And so that's a concern with, especially the security side of it. But there's also this other side of it in that being open source. There is nothing that stops an attacker from running fine sec bugs on any part of Jenkins right now. And so it's a, it's a security through obscurity argument which always doesn't work, especially when you're publishing your source code on GitHub for everybody to work out with. So, and so quite on that one, even since fine sec bugs is doing bite code level analysis right. It's not even so much that we're open source is the fact that our bite codes available because I assume fine sec bugs could analyze bite code from any that's fed to any Java virtual machine. Therefore, we have not increased the threat profile, simply by by admitting that fine sec bugs could be run on our bite code. Of course it can. I would imagine that would be true. I've never tried it except as it was in a palm file as part of a build step. And since all of our source code and all of our palm files and everything are there. It's not hard to stick in the half dozen or something lines, run, you know, Jenkins build and see oh yeah this is an interesting area. So, yeah, it's it's not really a terribly meaningful argument. It again is the argument that we need to learn as developers more about this, and make it more part of our stuff. And then work through it and add, add the improvements everywhere that we can, and if we find any real security issues to address those. So, yeah, those are some of the comments that have come up in pull requests and discussions and such and other times that I've done this. So, now we're on to the part of it. Okay, of where I'm a developer. What do I need to do about this. I'm a Jenkins developer and a core developer and a plug Jenkins plugin developer. And this even applies, you know, if you develop in other spheres, but this is focused towards Jenkins. So, like I said my plan is to update to update the palm version of each of these areas at some point to add the fine sec bugs as a plugin. And when you do that, then when you next update to that palm in your plug in in the Jenkins library whatever it will then include fine sec bugs analysis when it does the spot bugs analysis. That's all you really have to do to get this. Once I can get those those palm plug it palm changes in is update. If I wanted to do it before then. I would recommend backing it back out after it go after you upgrade to a parent palm that does that. And if you if you want to go ahead and do it now I don't have any, any reason kind of not to So, one of the way palm inheritance works, you can. It's a little hard sometimes to add something like this in right in a child palm. And sometimes you get like two instances of spot bugs that runs one with this and one without. It's a lot easier to just add it where the thing is defined in the top parent palm, and to get that to line up correctly now is it a problem if you don't get that to line up correctly, not really I mean it might add a few seconds to your run unless you got to your run in Jenkins core or something like that. So yeah, it's you can go either way. But the plan is to make it available. Push it to everybody. The item that we've seen that you, you should know about you may see this this message, the following classes needed for analysis were missing. You can see an instance example of it from the platform label or plug in in the other text below as to what it looks like test apply and accept missing classes three. Ignore this. There's a spot bugs issue on it where they're, it doesn't mean that things are not running find sec bugs isn't running that spot bugs really has a problem. It's the spot bugs engine plug in something that is logging stuff at a higher level than it should be. So it's it's providing information that you can't make any use of here where it's kind of something where maybe the spot bugs developers could improve things at some point on their area. At most it means that it may not be able to test some things as well as it could but I mean it's doing what it can. So, this is what you really need to do is to include it in the palm, get it via the upcoming future version or get it by adding it in yourself I know mark merged it into the platform label or plug in directly and I've done that in the other places where I've already done it. And like when I did that in Jenkins, once I get it up to the Jenkins parent palm and tend to remove it back out of the Jenkins one itself and just use that one. So, this is what you may need to do as a developer maybe maybe not. I've seen cases where find sec bugs found nothing. Sometimes in plugins. So it's like, Oh, I know this plug in has a lot to do with security, and it's a fairly big plug in, and it must have some issues and it runs it doesn't it's fairly clean. And other ones that are pretty small and you go yeah that's one. So, you never quite really know how to predict that from what I've seen, of course, something as big as Jenkins has more, just because it's bigger. So, you any findings that it has, you need to examine them and figure out what to do with them. So here's some of the things that you may need to do with them. I'm going to point out that these two in item number one, CRLF injection logs and information exposure through an error message aren't really ones that Jenkins cares about you're welcome to try and fix them if you want. But in Jenkins core and remoting. I just set those to be ignored to be excluded. And it's it's never something that the Jenkins project has made an effort to. So that's a very reasonable thing to do on those ones. I wouldn't do it unless you need to. I see it in some some plugins where it's like yeah we get some of those, but a lot of them are some of the areas I've seen we don't. In item number two, you can disable all of fine sec bugs by putting in the exclude file this this directive as part of you can look up the full path of it I spoke the one line in there. But you can disable fine sec bugs in this exclude file by excluding all security bugs labeled all of all of theirs under this category. This means you won't get any use of fine sec bugs. My recommendation is to only do this temporarily when you really need to find sec bugs provides value that helps us create better code, better, better product for our users. So I really encourage using it. But if you have some thing you've really got to do or you're doing some testing or something. This is one way to turn it off it's going to be opt out because with the way maven works is very, very, very difficult if at all possible to make it opt in. And we do want to spread this throughout the community. But if you find that you do need to do it. There is this, I would urge you to try and get it working again as soon as possible. More commonly what you need to do is item number three. So this it uses the old find bugs, warnings annotation spot bugs still uses that the same it hasn't redefined any of that part. You add this suppress fine bugs warnings with the value being the, the error name, like the CRLF injection logs, etc. And there's an optional parameter that justification equals. I strongly encourage you to do the justification one also to leave traces as to why this is not a security issue. And I also encourage you to do narrowly. So, if, if it finds something don't put the suppress warning on the whole class. I found some classes where there isn't, there isn't an alternative you have to put it on the whole class because it's testing things at a class level. But try and put it very narrowly put it on only the method that needs it. So what I did a number of cases in Jenkins itself is I did extract method refactoring to pull out a method that just has the piece that has that specific problem and put it on that to try and really localize it and not not suppress something more than that so that we can keep using this value moving forward. So that's the technique of admitting that we want to narrow this thing as much as part of the use of this as much as possible, and extracting a relatively few lines into a separate dedicated private method, just so it can be annotated is perfectly acceptable. That's worked well for me as well I've had that with spot bugs warnings where it's just cleaner to be able to say I want this thing excluded. Just exactly this segment I'll extract it to its own private method and off it goes. And most of what I'm saying here in this is not different for sign find sec bugs than for spot bugs in general. There's a couple of pieces where you know it is a little bit different. But yeah it's it's a large it's largely the same thing, extract that private method, and, or put it you know if it's a small method, go ahead and use that and you have to usually add a number of these. Like I said it's the downside to put using this in legacy code. A lot of times if we're writing the new code, we can do number four. When we hit this sort of a fine bugs, a fine sec bugs warning, we can improve the code. And that's this balancing that we have to try and do to move forward. So number four is when possible, don't suppress just improve the code so that it does something better. And the one example I gave her I mentioned before MD five is used a lot of places in Jenkins. It's okay. It's used for security. So like it's used for fingerprinting in some places, but even there it's an outdated protocol. It's known problematic. And then it's hard, you know as you get looking through stuff like this. Oh MD five is used here. Is that really a security case or is that not, you kind of have to reconsider that in the future. And you have to deal with those sort of things. So if we can improve these things, we're much better off going forward sometimes there's just no option because we do have legacy code. Sometimes there are, and in. Well that's number six. So, number five is report security vulnerabilities. If you do through this, find a security vulnerability an actual one, or something you, you think might reasonably be a security vulnerability. Please follow the security reporting processes and report them to the Jenkins security team. And then we will work with you to do this through a Jenkins security advisory, let everybody know about it. It's better to do it that way, and to be open and upfront about it at the appropriate time when we can get over, give everybody the warning, then to kind of just quietly fix it or something like that. Just quietly fix it people may not know how much they need to upgrade things along those lines so follow the process. The security team will work with you and sometimes, you know, we find that there's a new class of issue that people have discovered that we need to look in other areas to and so that's another aspect of reporting that's very helpful. Number six is is a follow on basically the number four. Sometimes it's like, yeah, we can improve something here, but I don't, it's beyond the scope of what I'm trying to get done right now. And so you can create your tickets or tasks and whatever your task tracking mechanism is for improvement. And then look at the changes that I did in Jenkins when I introduced find sec bugs. Some of mine have comments relating to a Jenkins dash, you know, number Jira ticket of something where yeah we can improve this. Let's work towards doing this I put one in for MD five let's improve decrease or eliminate usage of MD five. That's going to be a big effort. There's no way we could do it in that same that same ticket. Okay, let me then go on. I'm going to show some examples of three places that will there's there are there are examples I'm not necessarily going to show all of these but we'll show some in a minute here. Some places where it's already integrated some places where I put in some demo PRs of what it would look like because it's integrated. And then I just wanted to come to this one to a lot of this and some other comments, and not some are also in a blog post that I wrote here recently. So you can go see some of that more to to see some of these, these details. I integrated this into remoting. I integrated this into Jenkins mark integrated this into platform labeler. And I'm just going to take a few minutes and show some interesting things about this. So here, this is in Jenkins. And you can see where this this is an example of where I did the extract method to get us a smaller method in order to put the annotation there now here was a little tiny method. And I could just add the suppression there. But in this other place extracted the get file from arguments method. And if we look down here, I've got these and I've got my justification is that this is run from the CLI by the user who's running it. And so this path traversal would be a bad thing if it's a server side action, but as a client side action where the user is running it, they're running it to do these type of operations and it's fine. So there's like, what was the 21 files that I had to update here to get this to work. Here's another example of doing a narrow one. Justifications and other things. So this one has been merged. It's been in Jenkins for a month or so. Now I think at this point, you're welcome to go take a look at it. You can see how I did it there and what it looks like. Which one is this one. This is the remoting one. And so, you know, you can see this one to like I said, remoting since it's not a web application. It's the piece that runs between them and a significant portion of it runs on the client side on the agent launched by something that the administrator sets up possibly a number of these are not, there's more issues in here. Remoting also runs at a level where it spews out all findings, whereas Jenkins runs at a level where it only spews out the most prominent findings. And so that was part of the difference while there's more files here in remoting. So, yeah, there's just some different ones on here. This path traversal in, you know, this is loaded on the agent side or from the server and that sort of a thing. Let's see. The platform labeler one I just wanted to point out two things on it as an example mark used actually a little bit different one when it did it but this was my demo. Here's an example of using an exclude file. And in order to prove that it was actually working. I introduced this line which creates a fight fake finding it's one of those two that I recommended reasonable to exclude. So, let's see another one here. One is is the credentials plugin and this one is interesting, because it really did find a meaningful catch now we'd already fixed it by the time it found it. But if I had run find sec bugs on the credentials plugin a year ago, it would have found this issue before anybody else had reported it. So I introduced one example of it really finding an issue, and it was arbitrary file right issue I think is what we ended up calling it it was included in the May 21 advisory of last year. This shows that it can find some. Now I just suppress them because this deprecated mechanism can't be used from the user it doesn't take user input. It can only be used on the server for migrating existing data. And so we'll keep that in for a little while longer. Here's an example of where the MD five hash is just used in memory and is never stored. So the easiest solution on this was to change it just to compute hash and have it compute a shot 256 instead of an MD five. So that was, that's an example of actually you know improving the code, and we can fix the problem pretty readily there. So curious, you didn't worry about the increased compute cost of that or, or that was in this case is just not relevant. I don't think it's important in this case. You know I haven't gotten this approved yet so maybe somebody else will be concerned about that. But I don't think so what it's doing in this particular case is it's computing a hash of the agent jar to see whether or not it needs to update it. Very infrequent usage this isn't. This isn't hashing every message that's coming over over from the remote side or anything like that. Yeah. And so I don't know how frequent maybe once per job, maybe once per connection probably more likely. So the cost is probably pretty negligible. And this one is an example one from EC to where there are some things here which would be quite a bit of concern in a server side process, but it doesn't look like they're used. So just removing them is the cleanest way to do it. In this case, the MD five is actually stored so it's harder to actually to just change it. I think eventually the project Jenkins project needs to get to that point but that's not what we're doing at the moment. So those are just some examples. They're available to look at I pointed out some of the things that I found interesting in them. As we move this forward will have more cases, but these kind of can serve as some examples of how to how to deal with some of this. So they're emoting and they're emoting ones been in. Recent versions of remoting for a couple of I think since December, not positive on that. The Jenkins ones been in for a month or two platform label or market in the last few weeks. These ones are demo or draft PRs to demonstrate that they're available. And like I said, there's some more or some other capture of much of the stuff also in the blog post so people can go back and look at that. And see some of these comments and some of the things that I thought were interesting in the various examples. That's what I have if there's any further questions. Please ask them but I think that the fine sec pugs is a very valuable thing to include in Jenkins. We have more and more interest in security with regards to computer software these days, and the Jenkins project we certainly see that that increasing and so we want to get that out closer and closer to the developers the more that we can, so that we can get the things correct but more quickly, not introduced in the first place which is really the goal of a lot of this don't introduce security issues to begin with. By adding this will cost us some amount of time now, but it's better to add it now than to wait longer and longer to not out. That's what I'm working on trying to get this forward and let everybody know about what's coming up. Thank you Jeff thanks very much. I haven't received any questions from the channel. Thanks for this. The video will be archived and record has been recorded and will be archived on the Jenkins YouTube. That's the most famous for online meetups. Thanks very much Jeff. Thank you. All right.