 So my name is David David the dust. I'm with IBM research for many years now I'm also work security work group lead in Knative and member of the TOC and I'm also the owner of the guard code that we will also talk about today Yeah, my name is Roland host. I'm a software engineer working for red hat Where I dive into the world of open shift serverless, which is our Knative distribution on top of open shift and we make sure that it really works top-notch on this and I have to confess those security is not my main gig actually but security is on my mind when I'm working on software and developing software or on products and But luckily we have David here. So David is really the the true security expert and I would like to ask David now so for the next 30 minutes I hope I'm really eager to see what David has In the box for us how we can improve our security and that and I hope you enjoy this talk as well Okay, I think it's it's a fair to say that Everybody of us is really when dealing with CVEs. It's like weathering a storm and all our daily work so CVEs pop up out of nowhere and It's usually some of them are small and so easy to fix some of them are not and it's really Mostly in all hands on deck situation where you have to drop your work and start patching stuff like this And and of course that while we are really Fighting with the CVs and fixing them the bad guys do this as well Of course, but not for patching those but actually just to exploring them and really to break through our defenses and To be honest, this is really as a developer. It's really exhausting. It's something I'm I'm a creator at heart So I'm not a fixer. So I really want to be innovative innovative and not really responsive and just defend stuff like this and I think but I wonder really and this is my question out to David is really do you think that we can kind of really Stop this cat and mouse battle that we have break through this really cycle. Is there anything better than just having this? Reactive play with the attacker or not. So this would be really really curious about that okay, so to Really understand why situation in the defense side is so dire What we need to do is to see and understand the offenders ecosystem but not only that offenders of course have Access to all the vulnerabilities that we are gladly publishing. They also have available Exploits for many of them and these are available to offenders in a source code They can modify them. They can improve them more of them such as they can evade our security defenses now Having an exploit and having the ability to Modify it and improve it is like having a key to the vulnerability which is a door But this key the key It's important to know that this key doesn't open a single door It opens a door another door and another door another door is the same key So all I need to do as an offender is try to use that key against enough doors to get in Every door I open is an opportunity for a profit, right? I can steal data. I can sell it I can recruit to the node to my botnet. I can rent or sell that botnet I can do some extortion by using rental ransomware. I can make money out of open doors So for me, it's really a very lucrative business. It's a great return on investment. I Start a script and then go to the beach Spend many hours on the beach then when I come back I can see how much I made today in my hard work day on Defense side situation is a bit harder because a defender needs to protect against all keys He needs to be 100% secure and there is no way you're gonna have a description of all keys in your system to stop them You create signatures or you add signatures Maybe thanks to your security vendors to your system to protect you for all kind of exploits But it's never a hundred percent. It's 99.9 But you have that list and your neighbor has a different list and His neighbor has a different list because each one is using a different set of signatures Now for an offender, that's great That just means that when I'm going to use that key I'm gonna find who is who doesn't have the right signature for that key and I will go in Right and this is why defenders Sweat while offenders Spend their time on the beach and this is exactly what we need to change in order to make our life easier Yeah, I told you understand that David So this is really something and I wonder how we can go ahead in this game So actually use you said, okay We have to usually you have to track all the all the signatures that are available the unique signatures Like there, but isn't there a way how you can maybe? Prevent or detect exploit and then block them in advance without knowing the actual exploit I think this would be really a good Advantage of course, maybe then we we can go to the beach as well We all hope so so Put yourself in in the seat of border control officer Are you using a list of signatures to decide who gets in or who doesn't Do you have a long list of everyone who is a suspect and then if he's a suspect go send him for interrogation And if he's not let him in I Don't think how that that works So how would you decide whether someone is a friend or a foe? You have three minutes you can ask some questions You can look at the history at what what it said about that that person but previous entries to the US or whatever You have three minutes to decide how would you do that? Maybe through extensive training you would be able to identify that he's edgy or Sweating or or in some way Confused it doesn't fit what he says just doesn't make sense So you say okay, let's flag that one Send him for some more time that he will spend us so that people will either would ask him question before we let him into the US So you're looking for some tells you're looking for some Indicators that would tell you that this person should not just let in before you look further into that to that person to that alien Um with cloud native What what we need to do is is quite the same if you have a request coming in into a service We should ask ourselves. What are the tells? How can we we identify that this request is not should not just let in? This indicator that we are looking for this is what will help us identify that this request is not the normal Situation that's not what we expect So it involves Anomaly detection, but a very specific one one that is tuned to what we should be looking and what we should be looking is what we We need to further investigate This can help us identify a suspicious request But it can also help us identify as a suspicious service because if one of our services gets hacked Then it starts misbehaving not behaving as the way it used to behave before So it can also help us detect vulnerable hacked services The beauty thing about cloud native is that cloud native already did half the work for us Because we are already taking this monolith this big workload and dividing it into small pieces a microservice a serverless service a function and each such service has a Very correct well characterized behavior. It does typically one thing and therefore it also expect to receive one type of request and If that's the case there is a good pattern Also in the request and in the behavior of this small set of code that process that request So if there is good pattern, then we are good to go with trying to identify What in that pattern should be monitored in order to identify that it is an exploit being sent or a service being hacked Okay One more thing is that that note that signatures are a denialist They say all of these should not let be let in Security behavior analytics is Looking for an allow list. It would say okay That's the okay behavior of a request. This is how we request this this request for this service should look like and Now when you do something Off which is in security way. It is off. It is not correctly Not doesn't seem right from a security perspective. Then we should block it Okay, sorry. Yeah, okay, so I actually understand that so it's a SPA is about recording the good behaviors, right? And and of course, but then you have really also you need somebody who collects all the good and also the bad behaviors and have this list but this list is kind of It's also loopholes. Yes, so attackers might also Exploit those lists or find some ways around it. And is there any way how SPA can adapt to this behavior so that you Get how we can we analyze this behavior better or how can it work on this? Okay, so this goes to really to the secret source of of Security behavior analytics This is both the area is really important in that technology, which is what exactly do I am monitoring or what are the tails? And and I would suggest that what we should be looking are the atoms from which the exploits are built from We should be under we should instead of taking the fight to our end We should be taking the fight to the offenders And we should be asking what? Constitute an exploit that is being sent over the API What is it that is there which would let me as a person when looking at different requests identified? This is an exploit Curly brackets a dollar sign Quotes comma all these appear normally in the field, right? but they're also Essential for many exploits. So these are tails Well, if they appear normally in the field and these are tails, how can I do anything about that? Well, I need to look at each and every value and examine it for itself If there is a certain value and it's it never includes a dollar sign and suddenly I see a dollar sign I should be asking questions Because I already know that this value when sent to that service does not is not expecting to see a dollar sign Same goes for curly brackets and other Indicators and these could be unreadable characters that are being sent or unicode A very long length and many other indicators and we can go on and on make from simple indicators to a more complex and more complex indicators When I'm talking about each value what I mean is that I'm we are going ahead. We're gonna have to analyze each header HTTP header with its value. We're gonna need to analyze the query string We're gonna need to analyze the JSON body for each key and value We need to use that structure that they being sent to us and look at each each and every value inside separately such that we can evaluate its Content if it's content include those indicators. Yes or no and which of the indicators exactly This will help us identify when this value is now being used to carry an exploit Okay, so I understand so this sounds really very good and actually I understand okay You you really check the the pattern and the format of the input and have some some rules for that And but my big question is so this sounds. Yeah, Sean sounds nice, but it's just really a game changer This is really something that is disruptive in the sense that it really breaks the The game that this battle that we have described at the beginning. It's a really is it longer What what does SPA to actually to break the circuit? Okay, so to understand that we need to to look at those indicators It's if they are different dimensions in a space and think about a Certain value and in that value you expect to see certain Indicators go activated and Then if that's the case when these indicators will get activated It's like a subspace from the all possible indicators of the universe that we are monitoring of course all all indicators that we are monitoring Let's say that we cannot detect between an exploit and a good value in that In that specific value in that for in the header of the user agent for example Then this means that both of them if that was to the mention I'll look like a Long line so we cannot identify between the normal and the exploit So we need in this case to add another indicator We would identify that we are missing an indicator that will help us see now a three-dimensional space Identifying between normal and an exploit and every time we added such an Indicator it's not like a signature that only stops a very certain exploit. We are now making all values Allow us it allows in all values to see another aspect of an attack that we haven't seen before So by growing our number of indicators We can quickly reach a point where the Offender is losing any ability to send an exploit because everything that he's sending is being detected by the different indicators So we are reducing the space of the offender and by doing that we actually actually starting to eliminate all the exploit that he has in his In his bucket of exploits From being effective against our system No, oh, this is really a lot of theory They take David's actually and do you have actually is there anything that really is for real so that we can look at it Do you so what is the journey of SPA? Is there a tech tool that you can use for actually implementing these algorithms that you have demonstrated and maybe you have even a demo for us This would be super awesome Not surprisingly we have a demo Okay, so that's just a quick look at the history Let's see where are we in the time? Okay So it used to be an IBM research project and then we moved it to open source as a K native extension And then we made some more work on it such as we can use it in Kubernetes and in In vanilla Kubernetes is a sidecar And we can move and change this to run in any cloud native environment Oh, sorry Yeah, so the demo we're gonna have we're gonna use is is log for shell I mean, I would expect most of you to to be familiar with log for shell. It was very very well known two years ago And what we did is we took log for shell attack. We was first first see that attack and that attack would Install Wanna cry run somewhere a mimic. We would mimic wanna cry run somewhere being installed On a system and then we would see how we can stop it with security behavior analytics with guard Yeah, I do remember this look for share thing. This was really Disaster Yes, so the question is do we need to wait for the signatures for log for sure? Exactly, so can we stop it would would have been good if it would be prevented in advance, right? Yeah so So just just so we all understand the demo we have naive clients going sending to a vulnerable Application that vulnerable application logs the user agent Now our attacker what he needs to do he needs to set up an RMI server Somewhere and then send an exploit through a malicious client that will have in the user agent a certain string This is exploit which would tell the vulnerable app to go to the malicious server fetch some malware and Executed okay, so Let's start So on the upper screen on the yellow screen we're going to see the logs for the vulnerable app On the middle screen we're going to see just a log for the RMI server the malicious RMI server So gonna install wanna cry and this is our client on green and we sent a request the user agent as you can see his curl so All all nice and cozy we'll try another client. So we just see that it's working And yes, we now see that the Mozilla is the user agent So this is as an offender immediately send you to that spot of asking, okay Can I control the user agent? Can I control what is being logged? I will try xyz and its logs is xyz excellent now. I can try and send my exploit So I'm sending an exploit. This is how the exploit look like it just tell the vulnerable app to go to them the malicious server Stop just for a second Notice that now what is being logged is the object handle the object that was fetched from the malicious server And this is because look for chairs evaluated this expression, right? Yes It evaluated that expression that exploit and then that evaluation resulted in going to the malicious server RMI server and fetching whatever the result is Which then was executed in in the client and therefore when we refresh We find out that wanna cry was installed Well, it's a mimic one. It's just the page. Don't worry So next let's see what happens when we have security behavior analytics We we run the security behavior analytics to begin with so that it knows the normal traffic and We place it as a sidecar in front of the vulnerable app So first of all we run on top of the log for the vulnerable app and in the middle the log for the guard Which would save it? This is a sidecar and we send the first request We see there is no alert and a curl is being written up We send the xyz xyz is being printed out Guard is not has no concern about it Mozilla same same thing But then we would try to send out the exploit Now the important thing is that this request should not reach the vulnerable app, right? so as you can see it didn't and What we see in the debug mode is that it was blocked by guard and then the info why it was blocked by guard and It has some dollar sign and column sign and curly black heads and all kind of things that we don't expect to see in a user agent From our experience with this request This is really the crucial Moment where we say okay guard never saw log for shell God is not aware of this specific attack This could be a zero day as far as a guard is concerned everything is a zero day as far as guard is concerned But to be to be honest most of us will not encounter a zero day in our systems Most of the other export we will see our export that we just didn't cover us of well enough from Which is meaning it is for us a zero day it is something we are not prepared to to protect from Okay, so it's it's it's for technologically wise it's the same And one more thing is that that guard is keeping all those rules What should what is a normal situation in a Guardian which is just a crd on our system? I saw the CD then contains these rules to say okay No dollar signs are allowed with the braces around so that blocks and so these micro rules are then stored in a CD That's correct and just see this CD so we have some sense of what it looks like So it has some control so block or don't block learn or don't learn and so on and then a bunch of Keys and some numbers around there which are not that hard to understand and we will Maybe talk about that a little later Okay, oh this This really sounds sounds like a lot so actually you guess this year it gets really long if you have a lot of lists of what this thing and I wonder now do I have to manage this list on my own actually or is it I? Mean there's no general one Size fits all the situation so actually we need to adapt this for all of your service individually But if you do this manually, I think this is you still have a lot of work So be more less like I don't not sure whether this really makes my life much easier Is there a way how this can be improved or make might be easier? Yes So just just before I go to the other slide I think I think you raise a very important point of which you maybe wasn't emphasized before It's crucial for this defense that we will protect every service. We would build this You know a suit for every service on its own We are it's not one fit one size fits all We need a guardian first service and and the reason is because then we can really make sure that its value Allow list is only limited to what is really necessary for that service And this is a lot of work if me as a person needed to build one then maybe I can do that, but then The the service changes there is a new vision. There are other services just to maintain it becomes a nightmare It's it's hard And this is why the technology guard comes with a machine learning entity that would Make sure that we have a guardian being created automatically The way that this is done is that whenever a request come in is come is coming in We are profiling that request That profile is then used for two different things First it's being sent. It is being piled with other requests of the same service also from other pods and Then that that pile of requests is being processed in a machine learning in the guard service in To do some machine learning that would create all the micro roots for us Second thing is that profile is being used against the existing profile that we existing guardian that we have To see whether this request should go in or not should Bring up an alert or not So these these two things are being done with the same profile and both of them are done as we go along We don't I mean we can decide not to learn or to stop learning these are things that we can do We can decide whether to block or not, but both of them are done in one system Immediately at that sider sidecar and the machine learning guy. Yeah, so machine learning I think this is very very very helpful But actually I'm a little bit suspicious to be honest so because machine learning, you know putting all this rule into some model and I'm not sure how this can be reasoned about certainly who explains me actually who tell me that I'm really sure So if a CV comes in how do I know that this machine learning model is covering me and then of course I have my auditor where you know he comes by and he wants an order He wants some some proof that it's safe I I don't see now at the moment as how I speak and do that But maybe you could you have my idea how you how this can be improved Yes, so I agree machine learning is spooky in some ways Because if you can't understand what the machine learning is doing As a human then you don't know whether it will stop or it will not stop in attack And you cannot explain to your auditor and you don't you cannot change what is being decided You cannot control what is being decided, but it's not true for all machine learning In God what we have done is that we have targeted As a design goal that the machine learning that will be used is a fairly simple one actually We are using a lot of small and simple and well explainable learners a Big ensemble of them and then we are taking the decision based on the ensemble so that that single Decision each each value that we need to be decided and each Indicator that needs to be decided these are fairly simple to understand and to reason about So it's all human comprehensible and human controlled So human oversight over the machine learning is doable if you decide that this is your design goal If we would build a large neural network to see the request holistically and decide based on that Then we will have a problem Also, the beautiful thing about small learners is that they learn really quick So we don't need them many samples to figure out What the request look like? Okay. Yeah, this all sounds great. So actually this is okay, but you have more I see I forgot that also Yeah, so just to to show you how how easy it is to understand it We these same CRD that you see here is actually Translatable into a very simple UI. We just define what what are the? Special characters that are allowed how many characters How many numbers are allowed how many special characters are allowed in that field all all kind of small Indicators, which are all effective in order to stop An attack an expert from from coming in on on different Situations, okay, that's really great. I think this is very helpful also for my work But I have now kind of a random question actually So I heard something about a concept called zero trust and know a little bit what this means But I'm not on a person clear how the relationship of SPA to see your trust is active I know I heard some Rumors that there are some paper in the works from the tech security around zero trust Maybe you can shed some light on this how SPA relates to see about this would be all I'm really curious Okay, so Just just to make sure that everyone is online In line with as far as zero trust is concerned. So if traditional security is perimeter based Making sure that the bad guy will not come in through the perimeter Zero trust assume that the bad guy is already in Yeah, so it's like a terminal like a Airport terminal where you just treat everyone as a suspect if you walked in the terminal Recently you you know the feeling you're being questioned and you're being monitored in many different ways not only cameras And that's the attitude that's the kind of thinking that goes behind zero trust So is your trust? Everyone is a suspect meaning any request could be sending could be a way to deliver an exploit It could come from a beautiful person that we know very well and has great credentials and he proved his credentials But he may be already hacked So someone else is using his machine to send an exploit So we saw a lot of good stuff from him, but this one is an exploit meaning we need to actually monitor Everything that we see And everything will go wrong. So every request at some point may include an exploit and every service that we have At some point will get hacked With SBA Sorry with zero trust We must With zero trust we must In order to treat this situation where everyone is a suspect we need to evaluate Every request and every service for its confidence level how confidence we are that this request is okay that this Service is okay in order to evaluate that SBA is a great tool to do that Because SBA would tell us if we see if we see indicators for that Request being an exploit for it for example, then we go to access control In zero trust and there's in zero trust. We will have an access control a decision Well, if the risk is very low then do allow him to do all of that But if the risk is higher then maybe don't allow him to do all the sensitive stuff But do allowing me still go in and so on Okay, great. I think You're close to the end of the session. So maybe we wrap up now So there's one little thing which I'm still next me a little bit. So It's really so you said Having SBA is not an all or nothing general solution for everyone So you need to individualize it for every man a service. Do you have any recipes how you can run this? SBA in production so that it's for real worlds use cases okay, so so If I would to use this technology in a real environment I would probably make sure that as part of my CI system I would deploy the service into a test bed run a good set of Requests against it and use that opportunity to build the Guardian So I only already go into the field with the Guardian based on what I've done in the test bed Now that I have that on the field I would probably get some false positives in the beginning because maybe not all of my tests were Completed all the possibilities that are out there. Well, that's an opportunity to improve my tests but also an opportunity to Make sure that to check that what I see is okay What I can do further is that I can move from a learned bucket into a configured bucket copy my my Learned bucket into a configured bucket and then tweak there all the microservices manually one last point Yeah, so this is the last point. So actually I'm I'm from Reddit I'm an open source believer, but I have heard also people who say no Security is better hidden in proprietary software because they have the secret sauce. What is your take on that? Yes, so For me, this is an opportunity I see that as opportunity for the community Because that's that's a place where we can if we join forces to and we build the right indicators and share them We get to that point where we actually eliminate the tool set that the Offenders has Well, they can still use those exploit against others But when they are going to use them against microservices or cloud services cloud native services They're going to find out that those exploit are no longer Viable and in in the way at some time they would realize that going after those Those cloud services is is useless. It's just a waste of energy So this is one option the other option is to continue doing what we are already doing and sweat Hopefully we will choose the option of sharing our forces and Working together against the offenders Okay, thank you very much. We have run out of time So I hope you enjoyed the session and if there are questions come here and maybe you try out Security guard it's part of the native as mentioned and thanks a lot. Enjoy the rest of the good fun. Thanks you you can You can contact us on on slack Yeah, go for some on slack or on LinkedIn Okay, you have to ask a question there and here after and so on. Thank you. Thank you