 All right, well welcome everybody my name is Micah and I'm glad to be here with you all today I'll do a brief introduction about my background and give Eric My colleague an opportunity to do the same and then we'll dive right into it So I lead sneaks dev sec ops acceleration program Which on on its face may not tell you what it's all about but it's all about doing workshops like these to help people developers understand security vulnerabilities and how to address them and My background is as a software developer. I've been a Java developer since it was released in 1995 I've also been developing in Node.js since about 2010 and I know just enough to be dangerous and get myself in trouble on some front-end technologies like Vue.JF and Shot through my whole career has always been an undertone of security Which is now the main squeeze as it were. I got a TIFS-P in 2002 there about and I've been a security practitioner my whole career So I'm delighted to be here with you all today I'll ask Eric to go ahead and introduce yourself and then we'll jump right in awesome I'm Eric Smalling senior developer advocate here at Sneak. I also have a developer background Java Not quite as long as Micah, but close I For last decade have been mainly centered around consulting in the DevOps space CICD all that kind of good stuff Been a Docker user since 2013 when it was really raw. I am a Docker captain now and have all the CK Certifications except the new one. I haven't looked into that one yet, but Can't wait to get to talk to you today about how this impacted containers and Kubernetes Nice, I like Docker's navel nomenclature. Are you gonna be a Docker admiral one day? They haven't given me motions yet, but I just got back in this year. So I'm happy with the my current rank nice congrats All right, cool. So what we're gonna be talking about today is all about the log for shell vulnerability how it's composed what makes it tick we'll see a couple of examples in action and Significant of significant importance how to remediate it Then I'll turn it over to Eric. He's gonna give us a great view of log for shell in the world of containers So you go ahead and fix your code and maybe you think all as well in the world And then all of a sudden you're deploying that code to containers and maybe you're not as safe as you thought And then we'll wrap up with a log for shell deep dive Which is really a look under the hood of how did log for shell come to be an existence spoiler alert It has been lying dormant since 2013 and only came to light in 2021 but we'll see through some Some strategic break points in deep in the code stack. What makes log for shell tick So let's look at the TLDR here. You may be familiar like most Java developers are with log for J It's a very popular Logging package. It's built into a lot of third-party libraries open source libraries and frameworks As a side note, you may know if you use spring boot log for J is not actually the default Logging framework for spring boot. So spring the spring community kind of dodged a bullet there But you can override the defaults and use log for J We love our abstractions in the Java world So it's easy to drop log for J in spring and lots of people lots of developers and companies have done that so they were still Affected by this even though they were using spring so log for shell as it was dubbed is highly prevalent critical easily exploitable zero-day vulnerability and We'll see a little later that it got 10 on our Level of vulnerability is because it was it gave shell access essentially There's a lot of stuff you can do with it but the proof of concept and and the reason why it was such a high issue and in December late December people teams all over the world were scrambling around to address this Was because you could get shell access and kind of keys to the kingdom there Importantly you want to identify where you use log for J. That's not always straightforward because it may be transitive dependencies where log for J pops up and and the best most effective way to Address the issue is to upgrade to the latest Release version, which is 2.17.1 You may or may not know that that the initial vulnerability was discovered in version 2.14.1 and we went through a number of releases to their credit the Mostly volunteer maintainers of log for J jumped right on this and they made some initial releases Attempting to remediate the issue in a way that still left some of the underlying networking components in place And because additional vulnerabilities were discovered along the way Not as severe but still troubling like denial of service attack Ultimately the underlying Networking stack that enabled log for shell was pulled out completely. We'll see more of that later when we get into the deep dot For full remediation guide you can visit this URL We'll put it in the chat later, and I'm gonna go through some of the items on this remediation cheat sheet later But the the TLDR is Upgrade to the latest version if you can that's not always possible. So that's why we have A number of other approaches in the remediation cheat sheet So I touched on this briefly. What is log for J? It's kind of Interesting in the world of Java where we started out with system that out that print lines and that is pretty good and People may or may not know that there actually is Just above system that out that print line. There is built-in logging to Java But it's a little less sophisticated than some of these frameworks and if you're like me Over the years. I've kind of assumed that logging was built in in fact when this first dropped I had to go and look to remind myself of what logger spring was even using I spend a lot of time with spring and spring boot and You know the default is is a log back but For me over the years it's been kind of set it and forget it logging is just a given. It's really easy to control through Through through configuration, but over the years also It's gotten more complex and the reason for that is is the use cases for logging have gotten more complex So it used to just be we wanted to see what was going on on the console but in the production world where we have maybe multiple instances and clusters of Compute power on Amazon EC2 and everything's coordinated and sitting behind load balancers We wanted a more centralized way to manage the logs and the logs tell us a story Sometimes we want to know what that story is when things are going right We might want to know where our traffic is coming from or what pages are most visited or what apis are used most The logging reveals that to us, but also when things go wrong We want to capture and be notified of events like exceptions so maybe we want to have all of our logs go to a database that we can then query through normal querying languages and Discover information or mine that information out of our logs or maybe we want to use a service There's a whole industry that's grown up around logging like sumo logic is is one of them or data dog And we want to just publish our logs to a third-party cloud service and get reports and roll-ups and all kinds of Interesting data that that we might not otherwise have access to so logging became Much more important in our stack and yet It's also kind of ubiquitous and it's just there in the background if you do use a Service like sumo logic or data dog You you probably have had the experience of maybe you know a couple hours of getting things set up and Confirming that it's publishing and then you kind of forget about it And this is one of the reasons that this led to the scrambling that we had is because people really haven't It's a logging is not necessarily top of mind You're not most people are not building applications with logging in mind They're building applications and logging is kind of a useful feature to to monitor it and to get data out of it so log for shell is this vulnerability and This slide was originally when as I said this the situation evolved and the original fix was 2.17.0 and You'll see there in the background if you're in the Java community you may have heard of our colleague Brian Vermeer He's a Java champion and he Literally while he was on his way out to vacation wrote our first blog post. This is December 10th that this first blog post was published but in very short order within within days Well, maybe maybe within a week or two We were all the way at version 2.17.1 from where we started and this is kind of the double-edged sort of Open-source we can react as a community a lot quicker when things like these come up and then a closed source sort of software, but It may take a while to get it right it may take a couple of releases and that can lead to Frustration especially if you're on a DevOps team and you have to find some downtime for your application or you have Complicated the deploy Processes to have to do it multiple times 2.15 2.15.1 2.16 2.17 like four or five different deploys that can be frustrating but It's also a pretty great example of how the open-source community works and how quickly The patch the the patch of all and now we've been at 2.17.1 In the the end of December or early January So this latest release is has kind of held as it were to the kind of worldwide scrutiny people trying to find vulnerabilities so The other edge the positive edge of that sort is we have a pretty high degree of confidence now in this release Now what presentation wouldn't be complete without a poorly drawn diagram and here's mine now don't get overwhelmed by this We're gonna we're gonna dig into this a bunch more Especially when we get into the deep dive, but it's worth highlighting some it some of the components of log for shell and You like myself might be left scratching your head and thinking well, how did this even happen? How did our humble logger expose such a or lead to such a huge vulnerability and Part of the answer is just in features that were added over time like I said dating back to 2013 in fact Where there may have been a valid use case, but as it turns out led to this exploit problem So if we kind of navigate around this diagram We have our app and our app has some logging built in well great log is important But what happens if you send a an interpolated string like this directly to the logger? Well, there is actually a valid use case. It may look funny at first but there or I should say there was a valid use case for this sort of string and that's in the case where you want to use some external directory to be able to Do some some logging now as I said earlier this capability this Jndi Java naming and directory interface has been pulled out of the stack now. So if you have log for j2.17.1 This box isn't even in the stack anymore But it was and what it led to it was a it was a combination of being able to do Lookups if you're not familiar with Jndi Java naming directory interface It's one of these great abstractions in Java where you know DNS is a directory directory naming service LDAP is a directory We kind of like we've accomplished with databases We want to have an abstraction layer so that we can use the same Java Interfaces to interact with these different types very different types of directories So having Jndi support in log for j means that it'll take a string like this and do a lookup in this case. It's LDAP Well, it turns out that LDAP lightweight directory access protocol Has a response type that triggers the lookup of a remote Java object Now that's a lot to kind of take in so just put a pin in that for now but the the whole mechanism of this is Leveraging Jndi to make an LDAP call that results in another HTTP call that results in a serialized Java object sent along the wire that then Jndi happily deserializes and that this stuff that's outside of your app is Where the vulnerability lies because this can be a Poison class that's not in your class path that gets deserialized and then does something bad like maybe delete your whole file system Or maybe exposes a shell that I can then remote control So if you're left with the question like how did we get here? It's it's unintended side effect of good intentions and we'll see later When we look into the history a little bit that this was all this went through this inclusion of Jndi in log4j went through a normal open-source sort of process a use case was Identified a request was made a patch to satisfy that use case was put up It was reviewed it was integrated and it kind of stayed dormant there for many years until somebody figured out how to Be up to no good with it Now we're gonna come back to this diagram and and get into more of the details when we do the deep dive later on But for now, let's let's take a look at at this inaction this exploit And if we look at the code snippet on the screen, you can see it's almost in a way. It's almost like XSS a classic cross-site scripting attack in that we're dealing with an unsanitized string So if you imagine we have this checkout method that takes an ARG Well, imagine that ARG is this unsanitized string that gets interpolated If an exception is thrown that ARG is just gonna get logged out So unsanitized unfiltered unchanged is just gonna get logged out and because of the Jndi mechanism that's gonna trigger all of this network activity and The class that ultimately gets returned can do some some nasty things that's pictured down here. So To see this in action we have kind of a very humble application. This is available as Open source. I'll I'll throw it in the chat at the at the end along with the cheat sheet and maybe some other references it's you know available on GitHub for you to play around with if you like but the the crux of it here is a that we have the vulnerable version of log4j in the mix and All we're doing this is a servlet. We didn't want to have you know We we wanted to stay away from like big frameworks. So we wanted to keep it the example mean and lean and mean so it's just a tomcat servlet and It's we're gonna get a username and password coming in and As you can see, this is super secure if the username and password equals a very specific combination We're gonna say welcome back admin otherwise we're gonna log out the username again unfiltered unsanitized and We're gonna say that the password you entered is invalid. So of course not meant for any sort of production scenario This is entirely a demo But let's take a look at this in action so I'm gonna fire up this application and Not do anything else just yet Let me get to here Whoops Demo monster. Let's see what I messed up Right out of a gate with me a second. Well, let me fire up my other components here So there's a couple of other moving parts to this. There's a very simple Java servlet application, which is what I thought I just fired up But some of these other dependencies may feed into it Then there's also Netcat now if you're not familiar with netcat, it's gonna serve as a reverse proxy We'll see that in action in just a little bit and then I have a local HTTP server. That's just delivering a response Let me launch that and This is the thing that's gonna deliver the poison the bad class to us Now on this screen I triggered my my Siri. Thanks a lot Siri Oh, I said I said the word again. Okay Here's the directory of where this project lives and the only reason I'm showing it to you right now is this is the kind of Nothing up my sleeve section Because what we want to do is we want to have an evil class that's going to be delivered by HTTP ultimately If we take a look at this little Python script here and you may be left scratching your head. Well, what are we dealing with Python here? This is a nice little example of generating a Poison Java class We look at this Python script This just gets one run once and then runs a thin LDAP server waiting for a connection What it's gonna do is use Java to compile this Java class that's been Set up in line here and then it's gonna write it out. It's gonna generate the dot Java class and it's gonna compile it So Let me run that So we're gonna say the our LDAP server is gonna run on local host We've got port 80 and then netcat our reverse proxy is Listening on port 9001 now this pile of kind of technology here is not entirely clear It will become clear in just a moment So a couple of things just happened we generated a Java class we compiled it and now we have This gindy string that it generated and it's saying when we drop when we log this gindy string It's gonna trigger this whole network stack to to do something that we really don't want it to do I'm gonna run netcat just so that it's waiting for us Then I want to come back here and Just show you that now we have these two files exploit class and exploit Java and And we can take a look at the generated exploit Java class. It's a little easier to To look at then in line in Python. So this was generated by our Python script and What's going on here is it this is where the reverse proxy comes in it's just gonna be using Java's process builder facility to create a process and then it's gonna ferry on a socket incoming Fights and send them to the output to our reverse proxy. So this is where we get shell access This is the the nuts and bolts of the shell access now, let's see Let me go back and restart my Server here I know why it's being belligerent This is what we call the demo monster because I did test this beforehand There we go. All right, thanks demo monster, I know why I didn't run the first time. All right a couple things to look at here First I'm gonna log in with the right Credentials and it says welcome back admin great so far so good now. What happens if I put in the wrong credentials Or or wrong username or wrong password. So they don't explicitly match It says the password you enter was invalid will log your information Now if I go and look at the output here, I can see that our humble logger has logged My username that was the wrong username and it's logged it. This is a perfectly valid use case. I want to capture Bad authentications in case somebody's trying to act my system Well, here's where the problem comes in and that is if I now instead log in with that Jindy string What this is going to do is hook into this Jindy layer make an LDAP request that LDAP request is going to return a redirect to an HTTP endpoint to deliver a Poison class that's going to give us shell access through our reverse proxy Let's see if the demo monster gets us again Okay, this is good. So now the application is hung. Why is it hung? It's hung because if we look at If we look at our various Interactions here, let's start over at LDAP We can see that we got an LDAP request and it sent a redirect to local host 80 exploit class well, that's where I'm running my Java class and Or sorry, that's where I'm running my HTTP server the HTTP server happily return this exploit class and now back on my netcat I have a Shell that I can start executing commands on I can look and see what's in the folder I can Output the password file. So this is why this whole exploit got dubbed log for shell Because through this kind of what may seem like complicated Architecture here using this reverse proxy using LDAP and this evil class We now have direct access to a shell. That is really not the kind of situation you want to have happened Now, let me just kill all this stuff so that later on When I do the deep dive, it's available to us All right, cool Everything's dead. So that was a kind of a whirlwind tour of the effect of log for shell. We'll look at The network dive in just a little bit But before I turn it over to Eric, I want to leave you with some good Remediations in case this is something that you're encountering I hope all of this flipping around screens is not making you dizzy if it is I apologize So bear with me a second. I just have to get the zoom thing out of the way. Okay So first of all Here's the original Vulnerability as reported on sneak and and elsewhere and and even on sneak we reference the other the CBE But it was given a 10 because of this arbitrary code that you could execute We're gonna see another example in the deep dive where I just deposit a file on the file system just kind of approving that log for shell has that kind of control and If you go to If you go to the front door of sneak right now We still have this link section with all with a whole bunch of resources for log for shell It was an important enough vulnerability that we put together a whole bunch of resources I want to highlight two resources and then show you Some mitigation that that you can do on your own code and and then I'll turn it over to Eric So first of all we have this remediation cheat sheet. This is a really handy resource. It's a PDF First and foremost the first two are really the most important which is Identify where log for J is being used in your application. And as I said, that's not always readily Apparent like you may say well, I'm not using log for J But maybe a dependency that you're using is using log for J And then the best most complete fix is to upgrade to the latest version But then You may not be able to do that or not be able to do that right away So you can take some other mitigation steps and and we've outlined those Through these other numbers and one of the things that I know Eric is going to touch on is this idea of restricting egress if your application can't make an External network call to an LDAP endpoint for instance Even if you're using the vulnerable version of log for J. You're going to kind of work that whole process Now Eric will show us that this is not an ideal approach There's not the best approach because there are ways around it But that at least may be a stopgap if you can't just stop and redeploy your actively running service maybe you can adjust some Firewall rules that would that would prevent this external LDAP call from being made in the first place The other resource I want to show you which I forgot to put up here is Let's see. I think it's in sneak labs and I want to find our awesome. Yeah, there we go Awesome logs for shell. This is a great resource for developers. It follows this awesome Format, but it's basically just these different sections with bullet lists of resources to understand more about log for J and to Remediate log for J All right, cool. Let me briefly show you a remediation or or how to address Vulnerabilities in general and log for J in particular and then I'll Turn it back over to Eric So, let's just say I'm going to go into Let me go over to here Here's another Project and we're going to be looking at this in more detail a little later But we have these goof repos that you may or may not be familiar with but they are Vulnerable by design and we have this is a multi module maven project called Java goof and within it We have a specific log for shell module that itself has two parts a client and server This is what we're going to be looking at in more detail in the deep dive in a little bit what I want to show you for now is One of the tools that you can use to find and remediate vulnerabilities is sneak code It's a plugin that you can run inside of IntelliJ and it does we have a number of tools This is one of them. We also have the sneak CLI. It does very similar activity to what the sneak CLI does But it does it in your IDE So I can run a scan here and it's going to give me a lot of information Demo monster Let's see how we do So it's going to give me a lot of information about the code itself and It should also it looks like this piece of it was not successful So that's a demo monster there, but it will also show you vulnerabilities within your open source dependencies Let me try that once more and If that looks like I might not have refreshed my authentication. So that might be my fault let's Skin the cat a different way because we give a lot of different ways to do that and that is through the web interface I can add a project right out of github So I have a lot of projects here. Let's let's go take a look at this goof repo and this is Java goof So it might take a moment to show up here. Here it is Java goof. I'm going to select that one and Using the command line tool is really handy to see what's happening as you're going Using sneak monitor, which you can also do from the command line or adding it through the web interface the way I just have Gives you the benefit of not only seeing what's currently at issue with your project, but also it Continually monitors it. I believe it's once an hour, but don't quote me on that But it's continually rescanning. So even if you address vulnerabilities and a new one is discovered It will show up here in the report So I'm going to let that run for now just in the interest of time I'm going to turn it back over to Eric and then I'll come revisit this and we'll see How it's picked up on the actual log for shell vulnerability when when we come back to it later So with that, I will stop sharing my screen and turn it back over to you, Eric Thanks. While I'm getting my screen share going there is one question for you from Aziz. Yeah, asking does log for J only Work or excuse me log for J works only used with Java applications true or false That is a truthful statement log for J. The J is for Java and And and I there are there are all kinds of different frameworks for different languages And there may be some, you know cross Contribution, I may not be aware of it, but log for J is definitely for your Java apps Okay So let's talk a little bit about this exploit in a container I especially in a kubernetes Context I'm going to show you a demonstration quickly here and this diagram kind of shows what I've deployed We have a kubernetes cluster. I'm running mini cube locally just to make sure I don't deal with any weird network issues But I've done this demo in EKS kind Docker kubernetes doesn't matter as long as you got a kubernetes cluster The in the default namespace. I'm deploying the same Java goof app. We were just talking about And we're going to be hitting that as an HTTP user and we're gonna at one point in my demo Send our favorite little agenda version of our agenda string into that and see what happens So let's go ahead and get over to that I have my application running already and this is a as I said It's just a simple pod with a load balancer service in front of it hosted on mini cube The black terminal section you can see kind of hanging out beside me is the tail log of the application Let me refresh make sure everything's up to date and I'm going to sign into our demo app now This app we use for a lot of different if you've ever seen our presentations before the goof apps are kind of our thing They have a lot of vulnerabilities in them this one specifically I've added a log line to this search. So if I search my to-do list For car I'm gonna get all the list items that come back for car and I'm logging that so searching for car I search for the word the I'm gonna get those back and search for the if I search for my name I get nothing back, but I still log that so we can see car the an Eric coming out in the logs If however, let me so I've still got this Let me type our favorite string, so we're gonna do G Gen D LDAP now I have an LDAP server running at LDAP dark web Port 80 I'm gonna pass van Dole Then the lies if I could spell Spelling goes up the door when you're doing a live demo and hit go So you can see the log line happened. You can see that same string. I just typed went in there I'm gonna copy this for later use And now you can see this this vandalize Vandalism that has happened to the header of this page if I go around my app Every page in this application now has that So what happened as you know as Micah showed you the Java application when the Gen D string was interpolated It went out to that LDAP server and in fact, I've got a slide. Let me show this I have another namespace in this Kubernetes cluster. It doesn't have to be running locally This could be running anywhere that has IP access from my application Happens to it's easy to do this for with a namespace because Kubernetes will give me DNS look up for free So we have dark web and I have a service called LDAP and inside there is a running the Simple Java application could be anything just as long as it returns an LDAP type response and it returns the redirect URL to this evil dot dark web on Port 9999 for whatever it's worth and that returned a class that sought out and changed the header JSPF in this application It really as as Micah said in his demonstration. It could do anything. I want it could have done an RMRF on on the entire web app It could have started, you know doing nmap Searches it could have done a certain tons of things it could have set up a remote shell thus log for shell Now the correct answer to fix this obviously is to upgrade your application to get off of the Vulnerable version of log for shell, but if you can't do that immediately there are mitigating things we talked briefly about firewalling and Both inbound and outbound so you could have perimeter firewalls that would keep The goof app in this case from accessing, you know, LDAP dot eric smalling calm or you know What whatever's getting out of your network to the public internet? That's good You could also set up WAF firewall rules to try to catch the gindy calls that are coming in And using them in a form field like this is only one of a million ways you can do this So a lot of time they the examples you'll see will use an agent string That is has the gendi string in it and that often gets logged by a lot of tools So We're just doing it on the form field because it's an easy one to demonstrate The perimeter firewalls will help you but what if let's say I'm working at a Starbucks and I my laptop gets some malware on it and I get back on my VPN or come home to my corporate network And I log into a bastion host and that malware sees it Stands up a reverse firewall back to my laptop and now that jet that my fake LDAP server running in that mall Where is available? There's all sorts of you can you can think of a million different ways an LDAP a fake LDAP server could get Stut up inside your DMZed area now kubernetes gives us a nice feature Called network policies that give you a micro segmented firewall capability That's tag based on top of all of your pods and the containers in them a lot of developers like myself when we first start Thinking about network policy It's it's kind of daunting because you start thinking oh my gosh Perimeter firewalls are so complicated I have to open a ticket with the team to open you know whenever we change the application or whenever we add to our cluster and Sometimes those things you know can get out of sync and I don't want to be responsible for that kubernetes network policy makes it a lot easier Let me show you what a network policy looks like if you've not seen one before let me jump over to my editor This is a kubernetes manifest That is to put the clearing a network policy happens to be named deny egress It is applying to all pods in whatever namespace this is deployed to and it has one rule type and that is egress and The only egress rule that is being allowed when this is implemented is DNS look up through port 53 and UDP and TCP Network policies all all they do is specify how pods can or can't talk to each other and they do it by selector So in this case, I'm using the empty selector, which is a wild card being any anybody in the namespace. I applied this to but So what I'm doing here is I'm basically saying hey pods in wherever I deploy this can't talk to anything But DNS they cannot send traffic out to anything you obviously would want to add things like if your application had a back end database to Talk to you would add and say oh this application that meets this, you know Labeling selector like app is front-end or whatever you might call it can talk to My sql dot my corporation comm on port whatever whatever and you could allow just that out You can also conversely on that wherever the database is running so the ingress, but let's go ahead and add this Let me jump back over to My demo area. I'm gonna go in I'm using octin because I like octin This is a cool tool. It's just a kind of a dashboard on top of my cluster and I've hit the wrong button I like the tool then I use it wrong. I'm just going to delete my pod Kubernetes will restart up a new goof pod that doesn't hasn't been exploited yet In fact to test that let's come back out here and As it's starting up, oops, hit the key correctly Make sure okay, it's up and running now And we don't see the header fun. It's not coming up. So we know we're back in a good state Let's apply this network policy. So I'm gonna go into Let's see we're in my log. Oh my sorry my to-do list goof I posted the link to this github repository in chat a little earlier So if you want to play with this you totally can In my KS directory There it is now I've K alias to cube CTL. So I'm too lazy to type it. I'm going to apply the no egress YAML So now I have deployed that network policy into the default namespace in this cluster So we're gonna come back and we're going to start logging Follow So this is the new pod that came up after I deleted the old one And we'll log back into my application And just to make sure it works. We'll do a quick good search. We get that Let me paste our JNDI string in here go So it's hanging. What's happening is if you remember back from my diagram It's trying to talk to this LDAP dot dark web Server and because the egress this little lock is my indication because we put an egress rule saying if it's not DNS you can't talk to it. It's it's timing out now This isn't perfect as as as Micah kind of animated This is not a perfect solution But it does give you a little breathing room to try to figure out what applications do we need to upgrade now It's not perfect because I am stealing a Tomcat thread and sitting on it So you still you know somebody could denial a service you by just hammering away at this But it's better than them getting shell on your on your in your container As as I said, this could have been an external doesn't matter where it is this egress policies blocking any Communications out of it now the other side of things you could do that isn't network related But that would mitigate this problem make it harder on a hacker our security context related things so jump back over to my editor and Show you this is the deployment we use deploy this application. It's very simple if you've done any Kubernetes 101 stuff It should make sense. It's a deployment. It has one replica the container spec is This is the variable I substitute when I deploy it Job a goof latest and it's just the default deployment One of the things you can do is Use security context to make life hard So if I were to say hey don't run as root by default Tomcat runs as the root user if you use the official Tomcat open source image most open source projects run that way, but if you read their docs, they'll tell you Hey, you could run this as another user. Don't remember if Tomcat explicitly says this But if you inspect their container, you can see that there is a nobody user at that you UID GID And I'm instructing the cooblet when it starts the pod if I were to run this don't let this run as root So it would just fail Otherwise, but I'm also saying but do run it as the nobody user And I know from looking at that container that nobody user only has read access It wouldn't be able to modify any JSPFs in the web app so people can't monkey around with it the other thing You can do and you can do you can pile these you can do these on top of each other security in depth is always good Our defense is up. They should say you can say hey only run with a read only root file system. This says Don't allow the When the container at runtime engine starts the container up don't start a read write file layer at the top Which is how Docker or container D or any of those work the the file system that the image had that started the container is Read only and it applies a read write layer at the top and it uses copy-unwrite type operations to make Mutations to the file system. Well, if you say don't do that then Effectively the container is running on a read only file system And it wouldn't be able to modify any of the files in there either now for Tomcat specifically and most JEE apps you probably have a lot of log and work and temp directories. You'd want to mount from other, you know Either empty dir or some other file system in that's what you're seeing as well here the volume mounts To make sure Tomcat can actually run but an immutable Container file system is another thing that stops hackers from easily being able to do things like hey install end map and start poking around their network or use You know curl down this other script that starts mining Bitcoin in in in this container So those two things are a couple mitigation efforts Before I hand back to Mike I'm running out of time here. We do have another Cheat sheet that is predates all this log for shell stuff that my buddy Matt And I put out talks about these security contexts that you can apply to your your Kubernetes pods and Containers and some of the heavy hitters like the easy ones that you should probably you know research and use I'll paste a link to this cheat sheet as well into chat in a moment I saw a question come in from Aziz that what is a pod? We don't have time to go to Kubernetes 101 but in pot in Kubernetes terminology a pod is the smallest deployable artifact or assets or object you can deploy it wraps a one or more containers and When you say apply a deployment, it's a a abstraction around pods It's saying hey make n number of pods that look like this and apply these you know data to it So pod is kind of the the smallest piece of Deployable thing you can do in Kubernetes and with that I'm gonna hand back to Michael. All right great So I I went over time at the beginning. We're just about up to the Q&A segment I'm gonna beg your indulgence and take five minutes of it No more just to kind of do a whirlwind tour through the deep dive I also put a reference in the chat of a video. We've recorded previously about The deep dive so you'll see those references there, but returning to our diagram What kicks all this off is this gindy part of the stack and that is what's been removed in the latest version and What I want to show you now is a little proof of concept here It's within the Java goof Repo, which I'll make sure we post a link to I don't think I did that one yet, but here's the meat of what the The client piece of this is doing It's going to do a simple logger output and then it's going to do logger output that includes a gindy reference now the server side of this is running a very lean LDAP and HTTP server and its only job in life is to resolve the request from LDAP and return a reference to the bad Java class the important thing I want to point out here before I actually run this is that Over here in the server module. We have this evil class That evil class does not exist in the client module and using a multi maven Module maven project these things are distinct from each other So this evil class is not in scope. It's not in the class path of the client piece Which only has this one class called main But when we run main, it's going to receive this evil class and deserialize it through the magic of Gindy and Java deserialization, and it'll be able to execute it Let's just run through this right really quick I'm going to launch the server side of things and all that's going to be doing is listening on a port one port for LDAP and one port for HTTP So that's all it's doing right now, and then I'm going to switch over to the main and I'm going to Debug through it and the main event so to speak will be when we Output this logger message and it kicks off this whole gindy interaction All right, so we're at the first Breakpoint and if you look at the file that we're in this file is Gindy lookup Java. This is the very file that's been yanked out of log4j now. It doesn't exist anymore but because it's doing Gindy we ended up here and It's at this point that it's going to make the LDAP request So you can see in our debug output it now has the LDAP string That it's going to it's going to make an LDAP request to resolve Let's continue on all right now, we're kind of deep in the stack here and I'm trying to there we go trying to make this bigger I want to check in with the server and if you notice That LDAP request has now been made and it's resulted in this redirect But there's a little more there's a little more to it than that and if we look we can see the the LDAP response Includes these attributes now ordinarily LDAP is used to look up people and organizations Sometimes is used to look up printers Microsoft active directory is based on LDAP but If we return this special response that includes these four attributes object class Java code base Java factory Java class name Then that triggers the the network stack to make an additional request to this code base This is remote Java Code reference and so it's going to make an HTTP request to this URL and it's going to retrieve this Java factory called evil Let's continue on Okay, now we're dealing with the the directory manager again We still have these these attributes here almost to the to the punchline now Deep inside the stack in our directory manager. We have get object factory from reference So it's using that attribute reference and the redirect to HTTP to deserialize a factory object and Now in fact if we look here our our factory object is This evil class which didn't exist in the class path IntelliJ is able to show it to us because IntelliJ is awesome and has the ability to kind of cross go across these different modules And we'll see now that when we get to our Final breakpoint It's now going to call the get object instance method on that evil factory class And if we trace into it now, we're in this evil Java code that only exists in this module And what it's going to do is output a file using Java's runtime exec and We'll let that finish up and Now if I go to my terminal and I look at my temp folder. I can see that there's a pawned file there As a result of this so the proof of concept was successful and it's all because of this networking stack starting with gindy that Is no longer is no longer there With the latest version so that was a real fast deep dive and I apologize for the timing on that But hopefully you found that useful And we have about five minutes left if there's any Q&A So far, I don't see anything in the queue, but feel free to Drop some questions in and I'll just thank you for your time and attention now and Happy to answer any questions You can always also look us up on Twitter and LinkedIn happy to respond to you there Great. Yeah, I don't any questions coming in so if we want to wrap up and Contact you like you said on Twitter or LinkedIn So that's good. Well, thank you so much to Micah and Eric for their time today and thank you to all the participants who joined us Looks like everybody looks like we have one question really quick. Oh All right, yeah a couple questions So first one is does this basically mean that even if I'm not using any LDAP we are not safe It could mean that in that the LDAP request being made is to some External domain resolvable Endpoint that you may not be in control of at all. That's kind of the nature of the problem here So upgrading will will remove the capability to hit the Jindy and LDAP stack And then like Eric said you could do some interim mitigations You may end up in a denial of service situation But you could at least disallow egress to LDAP endpoints if you don't need LDAP then you could easily just globally disallow LDAP requests and that at least would Mitigate in the short term until you could update to the latest version Anything else you want to add to that Eric? No, I mean, yeah The exploitable LDAP server doesn't even really need to be an L up server as we've demonstrated here It could just be an engine X server that has a page on it. That looks like an LDAP response. So yeah, it's not your LDAP That's that's vulnerable. It's the Jindy lookup part of log 4j. That is Yeah, very true And then the last question is does it apply to SLF 4j? and it doesn't just as a sidebar the log 4j one Author went on to work on log back and at the left 4j, which is what spring boot uses by default log 4j to which is where this vulnerability exists Was was taken over by the Apache Foundation. And so they're they're kind of distinct But this particular vulnerability doesn't exist in SLF 4j. All right. Well got some more questions. Awesome. Love it Let's see if all inputs from user get sanitized so without changing anything our problem will be resolved It's a it's a fair question Sanitization in this case may be a little tricky because it might not even It might it might be able to be worked around because we're talking about the context of Logging and not typical user input. So like the classic example is somehow there's an exception that gets thrown exceptions typically get logged and You know, maybe there's you found a tricky way to get a Jindy string in there So again the the best mitigation is to is to update but you know healthy sanitization of any sort of User input that could find its way into your system in any way whether it's Logging or sequel or anything else is is a good practice no matter what? Anything you want to add to that? Nope Cool. What else we got a couple more Actually mark that one fix that we have an inject server. So egress disallowed to Jindy. You can take care of from there Again anything being logged is Vulnerable so if they can get that string into something that gets logged whether or not it's Nginx or anything that's where the danger is so Hopefully that makes sense real quick for a sack for the yeah, what do we losing if we turn off Jindy? I mean, I know we're at time. So yeah, I would say in almost I would say a 90 plus percent of the case is nothing Very few people really need Jindy in conjunction with their logger Maybe other parts of your application need Jindy Logging isn't isn't often one of them. Even if you're using an external service like Sumo logic, it's usually not managed through Jindy strings. So Yeah, yep, and we are we're gonna call we got it. We got it and but yeah Basically any of the mitigations who said are just stop gaps. You need to upgrade that That's what it comes down to or get off of plug for J if that's a possibility But that's you know, your choice. So we have to go though. We're getting kicked. Thanks, everybody Thanks again for your time and as a reminder this recording will be on the Linux foundation YouTube page later today And we hope you're able to join us for future webinars. Have a wonderful day