 Good morning. Good afternoon. Good evening wherever you are hailing from. I am not for short but and neither is my guest today, but my name is Eric the IT guy Hendricks and You are joining us live for rel presents episode 23 overcoming security vulnerabilities with red hat insights Before I introduce our guests, I just want to apologize things may be a little bit off today as as you know Chris moved on to other things and So around the same time that he departed red hat. We decided to change restreaming platforms So we're kind of learning this as we go So if if you're if you're wanting to become a systems administrator and think that you have to learn all the things before you can do so Yeah, you you'll just if that's how you approach things. She'll just never be ready and John I see you smirking because I I I take it you know this to be true. So why don't why don't you introduce yourself and we'll get into it Yeah, figuring it out is pretty much the the mantra of any sis admin. Thank you Oh, hey, we're gonna get this brand new product in it's got to figure out how to use that Hi, everybody. My name is John Spinks. I am a technical marketing manager for red hat insights Which is one of the key things that we're gonna be talking about today My background is also as a sis admin. So even though I have marketing in the title. I spent Quite a bit of my career as a system administrator for Linux Windows, San hardware email a Variety of different things the smaller the shop the more hats you wear and I'm very familiar with that mantra of Here's new stuff figure out how it works now Finding his audio back for a second. It's fighting him. But again, this is new software So one of the things while he's struggling for the audio button I mentioned that I'm the marketing rep for redhead insights Insights is a product that we've got that is part of existing red hat subscriptions that you already have today It's included now as part of your redhead enterprise Linux subscription as well as your redhead open shift and your redhead Ansible automation platform subscription. So insights is a pretty Very product that supports all of the major platforms that we have here at red hat And we'll go into a little bit more detail here in a moment, but first Eric. I'm going to check up on you Yeah, I should be good now. I do hear you wonderful perfect Learning as we go that's that's kind of the theme of this show and and has been since it started so um So today John you're actually a Recurring guest on the show and we and you kind of did a quick overview there while I was reconnecting everything Of what insights is? Did did you happen to mention just how many different services are live with with insights right now? I did not because that's a it's a bit of a loaded question. I don't have an exact number. I Let's do it. Let's do it live because that's the way we like to do things Insights started out as the advisor service used to be that that's all that there was within insights So advisor and then we introduced the vulnerability the compliance and the system comparison which is now known as drift service. There's also policies patch subscriptions and We are coming out very shortly. We have in beta a resource optimization service. We also have a couple services that are in Private beta so we are somewhere between 10 and 12 depending on how you count them We also sometimes refer to our inventory service and our remediation service and then the entire platform has capabilities like our back notifications and these are all Part of what you get in the box as well and those are just on the rail side. I did mention that insight supports other platforms OpenShift has an advisor service as well. It also has cost management and it has subscriptions and then the Ansible platforms have a cost planner a savings planner a ROI calculator and Ansible analytics and it also has advisor drift and policy. So they're up to Seven if my rough counting is correct. So yeah, there's a large number I think we say across all of the hybrid cloud console, which is the actor formerly known as console.redhat.com There's somewhere around 2627 different services that we host not all of those are insights But all of those are out there and available as part of different redhead subscriptions And that's awesome. I mean when I was a systems administrator for geez At a decade, I would have loved to have had a tool like this because it's like having a systems administrator Who's always on call who never calls in sick? just constantly out there checking on on the health and and the performance of of of your your your entire fleet of Started off with rel servers and then you know as you mentioned expanded to Ansible and OpenShift So I mean this this platform has just grown exponentially just every single release It or every quarter or so it seems like there's there's new services or new features coming online So I would imagine John not we're not trying to sell anything But just for those out there going hmm you should look at this I imagine something as powerful as insights would be really expensive Again, it's included with part of your existing real subscriptions So it's something that you already have today All we want you to do is enable it turn it on and see some of the value because it is part of that value of Subscription and the reason that we we talk about it that way is Part of what you pay for a subscription for is support Insights was originally created with that advisor service and the whole goal of that advisor service is Let's tell you about things that might happen to your environment Before they happen. Let's give you some proactive guidance some analytics of your environment So you don't have to call red hat at three o'clock in the morning when your beeper goes off It's been a long time since I've had a beep or fortunately But you know email whatever you want to call it you get that ding or that call from your boss wakes you up in the middle What now? Rather than happen to deal with that if you can proactively know oh I have this version of the rail kernel and I have that version of a certain package and With that particular combination if it runs more than 120 days you have a kernel panic oops We notify you and that's a real thing that we actually look for I can't remember the specific name But that's a one of the real examples of things that insights checks for And we warn you at 30 days 60 days 90 days 120 days with raising Criticality so if you're about to encounter one of these issues we say hey, this is coming Hey, this is coming quick You might want to fix this really fast With the idea of we're gonna let you know about it before it impacts your systems giving you an opportunity to resolve the issue and it could be something you know, maybe not that dire but We had a one of the technical account managers or Tams Tell me a story that he was showing Insights to one of his customers and the customer what I think your thing is broken. It's telling me that Network bonding is incorrectly configured and we don't use network bonding. So that's not a real thing You know, I got a false positive here. Well, let's Let's dig a bit further because you know, that's it's essentially a you know an analysis engine That's looking at the configuration of your system and saying hey, here's things that we've discovered So it's it's trippus for a reason. What's that reason and come to find out one of the admins had Miss typed and in a CLI command and partially enabled network bonding at a box They never would have known about it. Otherwise, it's just like we we don't use this thing What happened but you know fat fingers are a real thing in the sys admin world right So we we kind of in a previous episode we you and Scott and Chris Kind of talked about insights in general But we want to focus in on The security vulnerability piece I mean every single day It seems like there's new new attacks new new very Public Attacks, I mean there was there's what the Facebook thing a couple of weeks ago There was twitch before that. I mean there's just There's been pipeline attacks. There's been upstream software Supply chain attacks. I mean, it's just one thing after another after another and So this is definitely something that was top of mind before 2021 But it seems like I mean, maybe you maybe your feeling has been different But it feels like in 2021 we've had more attacks more public very public attacks Things that have hit not just IT new spaces, but the mainstream news as well Then I think we've had since I started working in technology Yeah, it's Security is definitely one of the things that brings people to insights pretty frequently I think the most recent one that's top of mind for me is Microsoft recently had their Open management infrastructure issue that came up across and some of you may have heard of this already But it it's running systems in Azure. There was a oh my package that had a remote cold code vulnerability This is not a red hat thing. This was a Microsoft in Azure thing So this is something that popped up potentially very problematic There was packages being installed on a box that you know any Azure box that wasn't really Something that the admin installed or anybody purposely installed. It's just part of running your systems in Azure It was there and that was actually a really interesting one for the insights team because we Found out about this vulnerability pretty much at the same time as everybody else It's not something that's in red hat CVE database and we said I Wonder if we can detect this like can we tell our customers about this issue? In the insights team got together in super quickly Figured out how to detect if this particular issue was present. I mean, it's not super hard It say is do you have a certain package and you have a certain version of the package that is vulnerable? And then we created within the advisor service one of the services of insights that I was speaking about earlier It runs on recommendations and it's essentially a rules-based engine I think of it as an if this then that kind of engine just to make it really simple And we were able to say hey, well, you know, if this is present Then that's an issue like it's a It's a bird, you know potentially a vulnerable package and we should have an alert But just because it's there doesn't mean you're exposed Right, but is it running? Okay, if it's running that's a different level of criticality Is it communicating? over a port is That port internal or external if it's internal it's still critical Still super important, but if it's an external port that is communicating up. That's where the real risk is So we actually looked at these different factors and created recommendations inside of insights that says You know that looks at those different conditions. I think there's five total and we said hey, let's Go look for the presence of this package if it's not there. No biggie. We don't trip the rule If it is there we tell you about it if it's there and it's running We start raising the red flag if it's there and running and talking that red flag. It's really really high Let's let you know So this is a vulnerability that is not a red hat vulnerability. It is something that one of our partners had had an issue with and We wanted to let our customers know if you are running your red hat systems in Azure You could potentially be exposed and guess what insights can detect that exposure for you in this case, we couldn't fix it for you because we didn't have a At the time that we did this and it was all within roughly a day of finding out that this thing existed that we Created these rules and made them available to customers at that time. We didn't have a remediation in place But we were able to let you know about your exposure And that's one of the things that you get that call about at three o'clock in the morning Oh my god. I heard about this thing on the news. Are we exposed? Okay boss Let me check and without something like insights, you're taking the time you're logging on to each box individually That's running in this particular situation. So in this case it'd be a running on Azure and you're checking is this package installed Okay, it's not there. Okay. Oh, it is on this one. Let me write that down rather than happen to go through all of that You click a link and you Go into insights and you say oh am I exposed? No, I'm good or yeah I am and there's three boxes or 30 boxes or whatever the number is We have a public blog article on that as well. I think I've sent you the link. It's it's not with the rest of the show links Yeah, I'll highlight for you, but if that's one that you can grab you can throw it in there But I put about a blog about that A Vulnerability happen to anybody it is not I mean It's not something that was maliciously done on the part of Microsoft who is a partner And they're a great business partner in the creation of this blog. I actually talked to Azure product managers and marketing people From Microsoft and let them know that this blog was coming because like hey like we don't want you caught off guard Like we're not Microsoft going, you know, you did a bad thing and we want to let our customers know you did a bad thing We said hey, you know stuff happens security breaches are unfortunately a reality of life Let's help our joint customers know if they're impacted Well, and what was really impressive about that particular Situation was the partnership Kind of in in action. It's one thing to say yeah We're business partner But it's another thing for our engineering teams and their engineering teams and even the community to be working together to say Here's an issue. Let's fix it Let's get it in this tool that helps our customers are our mutual customers identified that this issue is there So that was the first that was the first thing that really shocked me was just seeing that partnership that that actual Communication back and forth. The second thing was the speed at which this was done I mean it was a couple of days if I remember the the incident correctly and the issue was identified a Those five criteria were defined Fixed was found and and it was put up in insights and and that took that that used to take weeks And it was it was roughly a day hours. It's roughly a day So I mean that was also a big win inside of red hat like as as a part of a product team Seeing people literally across the globe like, you know, we found out about this thing at, you know At 4 p.m. You know and then one of the product managers who has this thing He's is based out of a Mia like he he was on calls, you know, 9 10 o'clock at night his time Goes to bed wakes up the next morning. You know, like, all right, we got it. It's working. I Mean, right. That's that's awesome. That's amazing and it takes great teamwork to you know Within and without to make things happen Definitely so with with that kind of context in mind and that's that's at the develop at the at the vendor level But let's let's zoom in and what is what does that process look like for a company or for an individual team or an individual of Systems administrators or security admins working back and forth Let's kind of zoom way out here and talk at a high level like I mean, let's let's start with the very basics here We've been throwing out terms like vulnerabilities and CV ease So let's let's zoom out and and we'll kind of zoom back in with you using insights is kind of our It's kind of our exhibit, but what is a CVE? All right, so a CVE is Acronym for common vulnerabilities and exposures so essentially whenever we have a Security flaw flaw that is made public it gets assigned a CVE ID number of some sort and Sometimes if they're really major they also get fun names or not so fun names the Microsoft one that we were talking about because it had dealt with the Open management info that yeah open management infrastructure. Oh, am I it got the oh my god You know that that's what they ended up naming it and it's not part of the CVE Other ones you might be a lot more familiar with our boot hole or you know the old, you know Meltdown Spectre if you want to go back a couple years. Those are ones that everybody was you know Probably really familiar with if they had to deal with it You know those get a little bit more named. I'll call them celebrity status It's one those are the ones that you get the call about from your boss It's like what's going on because it shows up in a trade article or it shows up on the six o'clock news But most of them and they happen pretty frequently It's just hey, there's a flaw. We need to fix this it gets associated with a CVE ID And for red hat it also if it's something that we can fix because not every one of these can be fixed It's you know, maybe it's a super old Colonel version in the fix is to go from CPU type X to CPU type Y It's not fixed. I'd say change in architecture a change in hardware But if it's something that can be have a software fix it gets an errata associated with it So that's a you know a software patch or a package Well in the the scenario you're talking about happened to me literally with the celebrity status vulnerability called heart lead so I got the call in and It wasn't the middle of the night, but it was it was bright and early You could tell that someone high up on the chain logged into their into their news feed or over breakfast and went oh my gosh Are we vulnerable to this and of course I got the call and I got I think the company I worked for at the time had something like 400 Linux servers and there's three admins and So the 400 or the three of us got to divide up 400 servers at like six in the morning and go through them Just a couple at a time and go is this here? Nope. Is this here? Nope Okay out to the DMZ. Oh, we better fix us and so I've got to I got to go through that whole process and Now fortunately that was one of those situations where we found a fix fairly Fairly quickly once once it started going public, but I think the velocity of that has increased drastically since since the days of heart bleed Yeah, sure But in order to help make that experience better one of the things that we built into insights was our vulnerability service But before we we get into that you guys have been talking and you know listen to us talk for about 20 minutes I do want to go ahead and share my screen and let's start actually looking at some of this stuff because that's gonna be a lot more interesting alright, so sharing my screen here and A lot of times the ways you may get notified of security bulletins are through your email Maybe you get an email notification or maybe you just go out to Google and you type in the name of a Security bulletin or a CVE and in this case, I'm just on you know our main security vulnerability page You can go and browse these you can see the The ones that are resolved or some of them are in progress here So this one that's ongoing that one most likely doesn't have an errata associated with it yet And I'm a hundred percent guessing. I'm not clicking into it yet. I just see that it's ongoing Or you have some here that don't have any any notes yet. So we're still that's just release notes So that's simple But for the most part you might go into more of a name CVE like this one CVE 2021 that's the year and then the number is 33910 So you might be like hey, like what is what is the impact of this particular CVE and you can go through here We've got a lot more detail. We've got a summary of what it is the flaw itself So it may already be very familiar with these pages Part of this we tell you the impact at the top, but I also want to highlight a fairly recent change as we've added in This insights vulnerability analysis to pretty much all of our CVE's so if you are already a user of insights Somebody sends you to a link like this and wants to know if you're exposed. It's really as easy as Clicking view expose systems and that's going to redirect you to insights where you can say hey here I am I'm already logged in if I wasn't logged in you'd have to log in it uses your red hat account the same one that you would use to See a knowledge base entry or download software or file a support case It's the exact same account that you would use to access insights and in this case this particular kernel bug It happens to affect 15 systems in my environment. I can see it Let's do it right here turn away from my thing to behave and then I can actually see every single one of the systems that impacts listed below it and more information about it So this is one way that people might come in and find out about a Vulnerability, so I'm going to pause there and see if you got any questions about that for a move on Yeah So with the CVE database, I'm thinking that there's there's kind of a tiered system to it, especially if you're using insights So a question some of our some of our users may have is our CVE is exactly the same Globally or do certain vendors have certain changes that they make to CVE's? Like red hats now going to track a window CVE because we don't we don't have a windows operating system Does that make sense? Yeah, and same same thing kind of like that Microsoft example I gave is that Microsoft CVE it did have a number associated with it I'm not going to have that in the red hat database because we're browsing red hat CVE So these are ones that are specific just to us as a vendor So all I'm seeing in here is is the red hat stuff So it's definitely a red hat centric view of the world if you're looking inside of a a red hat product or a red hat website such as this one. This is just the plain Customer portal. It's the same thing if you go inside of insights I'm going to show you all the CVE's that are red hat specific but not Microsoft or any other vendor so So red hat takes all the CVE's that are out there narrows it down to just those that affect our products thinking rel or satellite or or JVOS or or something like that and kind of narrows it down to just things that affect us as a vendor and and those folks that use our our applications does insight Does insights give me the ability to narrow that down even further? So if I'm not if my systems don't use a certain a certain package Can I mark that can I set up sort of my own? CVE structure or if there's something that is critical to my business is there a way for me to sort of internally Say this is really really really important. It may only be a medium CVE But this is critical to our business. So I'm going to internally mark this as a high Yes, and I'll come back to that in a moment because there's one other thing I want to highlight about the higher red hat process before I delve a little bit more into insights We do have And the name has changed. We used to call it customer security and awareness. It's recently changed I do not remember the new name. I haven't Got it to memory yet, but there's actually a team that goes to some of these big security vulnerabilities like Boothole or meltdown inspector. In fact, I don't remember what boothole is. So give me a second. I'm gonna Yeah, that's fine, but anyway there there's a group that will actually go through and really dig deep into Some of these really impactful vulnerabilities And that's where this extra information comes from like these executive summaries if it's associated with any other CVE So, you know, this one is, you know, we're looking at this vulnerability But then there's also an associated vulnerability and so we're gonna give you just a little bit more information That's above and beyond just hey, there's a security flaw and you should fix it We're actually talking in some depth here about the different capabilities and this is just on the You know right within the customer portal site. So this isn't even within insights Insights will redirect you to this. So if you happen to find out the system is Affected by a certain vulnerability, we're gonna bounce you out here because they've put in so much work already I mean, I'm still scrolling down this page and I just got down here, you know, kind of towards the end We do have a diagnosis script a ton of references But the really really easy way to detect if you have something like this Like I said before is you just go and click the button and you can see it within insights This does require you to register your systems to insights and that is a super easy process There's Ansible scripts built in for it. There's a rail system roll. There's a satellite Roll set up and if you want to do it manually if you're on rail eight It's a single command insights dash client dash dash register and you're done If you're running an older system, it's gonna double your work You first have to install that package. So it's two commands. You have to you know install The insights client and then you have to register and we actually walk you through how to do that There's a register systems tab insight of insights will walk you through how to do it and give you both give you the Ansible playbooks If you need it But to your point earlier you you asked me If I have a vulnerability like this and it says it's important, but you're like no no no for us It's critical or conversely if you're like, yeah, you don't care Uh, we do give you the flexibility if you look here kind of towards the right There's a business risk and a status That allows some additional capabilities and if you're already familiar. Well, I got it highlighted If you're already familiar with a common vulnerability scoring system or CVSS We show you what this is base score is in here. This one's, you know, it's a 3.0. I'm sorry It's a 7.0 is your base score and that's Compiled from a lot of different vectors and we summarize them here for you But if you're new to this, you might not know that AV is a tack vector If you need to know that stuff you click the little question mark and it's going to give you all that information and it's essentially Summarizing from our database what the metrics are that make up the common vulnerability scoring system and why it's been scored the way It's nice to your point if you don't want that score we go in here I'm going to edit my business risk. I'm going to say, you know what this is high based on my business criteria This is a high business risk and the reason why is we need to fix this That's always a great reason to say that business risk is high That's actually going to show up in reports and stuff going forward So you might want to watch that maybe you don't put something too snarky in there But you know, we've got a business risk that's associated with it now And you know, like I said, I said it is high conversely you could put it as low The other part of that is the status. You actually have to walk this generally through a remediation status So once you know this vulnerability is here Have you reviewed it? Maybe if not you move it to in review and you say, okay We're going to start taking a look at us to see if we need to do it Your other options are on hold like hey, we we need more information Or we've got to go to another internal stakeholder and get more information about you know, is what's the criticality of this All right, we have decided we need to fix it then we schedule it for a patch Once we scheduled it for a patch we can go ahead and say it's resolved. We're done We we fixed this issue. It's over and insights will also be updated when there's no more Systems that are hitting it And then the other options are no action. We've just accepted that this is the risk we're going to take We're not worried about this particular CVE or you've resolved it via some other mitigation So you've got these different Options and you can set this either at the CVE level or you could set it at individual system level So if you've got a set of systems that are Maybe their development systems, so you're not as worried about certain vulnerabilities You can say hey for these systems. It's okay. We're we're going to just going to accept the risk on these But for the production systems, we got to fix it So once you set your status, it's now updated It's in review and we can go ahead to the process of reviewing all of this so I've kind of Jumped the gun here a little bit when we were talking about because we went in the way that a You know if you're going from the customer portal you might go in but I actually want to take a step back And I want to show Insights at a higher level. You cool with that? Yeah, let's do it. All right. So I just switched tabs if you notice I'm here at console dot red hat dot com You can also get there from typing cloud dot red hat dot com and logging in and this is that home page So this is our hybrid cloud console and this is the thing that I said earlier has like 27 different services on it At any point within this if you want to see what's coming you can actually use the beta release you can see stuff That's coming soon. It's not quite released yet. We Plugged this in for everybody. It might not have all the features It might look a little off because it is in beta But you should have noticed when I made that switch gave me a little bit of a visual look and feel I've now got a summary here of rail where I can see some incidents I can see a Remediation playbooks and I have those options for open shift in anspaul as well So this is a ui change that's coming and you can flip that on and off anytime you want right here inside the ui That settings gear also takes you to different settings inside the tool If you want like email notifications of when things happen You actually need to go into settings and opt in for your user Or if you want to use the rural based access control what's available from there Since we're focusing specifically on rel today I'm going to go right into the red head enterprise linux section And that's going to take me to the dashboard of insights From the perspective of a rel user And this is my my lab environment my playground So I've got 17 systems that are registered If I wanted to register more I click that register systems button It's going to just redirect me inside the ui and it's going to give me a little bit of a choose your own adventure type menu and say Hey, I'm on uh, I'm on rel 8-2. I want to register using subscription manager I want to use anspaul. There's a playbook Or I want to type my commands There's my command All there is to it And if you want to use other mechanisms, we'll tell you how to do those as well So we try to keep all of that really simple to use right inside insights itself If I go back to this dashboard There's a lot of information here if you're new to this it can almost be overwhelming because I've got All these different services the names of the services are on the left. I mentioned some of them earlier They all have different capabilities and they bring you different value and every one of them is included again as part of your Redhead enterprise linux subscription. So there's nothing to buy here. You just got to turn it on The one we're focusing on is the vulnerability system so I can see Summary here of vulnerability. I've got 33 cvs with the security rules. That's that more in-depth Evaluation of a cve I also have nine cvs that have known exploits I want to chat about this one for a second because this is a fairly new feature That does not mean that you yourself Has a system that has an active exploit like you haven't been exploited at this point But the cve itself has a known exploit associated with it So this is again, we're informing you of another level of criticality that you might want to be aware of So in other words with a known exploit that means somebody has actually executed this in the wild Someone has reported. Yes. This has happened to me as opposed to a lot of cvs are I don't want to say experimental, but I guess theoretically possible So with a with a known exploit this means it's it's actually happened. Someone has compromised my system using this this particular cve It's not under a harvest moon in the third month of the year on the day that ends with q you know, this is actually something that There's known exploits out there in the wild that people could potentially be taking advantage of so but it's very important You don't confuse that with you yourself have been exposed, but it is important to know that these potentially have a higher level of criticality because people have You know take take an advantage of these somewhere in the world And we can also see by cvs score. You can see your summary. So, you know, I've got some 123 between the eight and ten range, you know, 800 between the four and eight range, you know So I can see all that information here I can also go on the vulnerability side I'm in the left hand tab now just to make sure you're following along with my mouse. I've expanded this Uh cvs reports or system so I can view this by cve I can view this by system I can also view, you know, create my own report if I want to out of this vulnerability service So if I start by going into that cve view This is the boss calling you saying hey, I heard about this thing on the news. I heard about boot hole Uh type in boot hole. Okay. There it is It's cve 2020 107 13. It has a security rule associated. I have four systems exposed to this. Oh crap Maybe I need to fix this So I can go into the vulnerability itself I can see again some of the information from these security rules is a link to my database and if I Want to know that these are my four systems that are exposed Click the box. There they are I can export this out to a csv or a json. So, you know Boss calls up with that phone call saying I need to know every system that's affected by this particular vulnerability now And json file done, you know, here you go boss You know, I have that in your email in two minutes. I mean it doesn't take any time at all to do and it really Helps you know What's your security posture is inside your environment? But john isn't it just so much more fun to sit there and copy and paste all that information Out of your monitoring tool into a into an excel spreadsheet. I mean, I know eric. It's not Actually somewhere I'm gonna have to I'm gonna have to share another link. Um, I have to dig it out We had a third party company a principal technology that went through and did an analysis for us And we said, you know, your your challenge is thus Take a hundred systems spin them up And then put an experienced linux admin on the system and then tell us how long it takes to do a series of tasks And one of those tasks was you know, like find what system is impacted by the cve Uh, and then do the same thing with insights and let's calculate Time differences step differences and this wasn't I mean, that's the kind of thing that could totally be loaded But we were like no put treat it like That sys admin would because you and I've we've been in that space We're not going box by box. We're writing ourselves a script or we're we're somehow Automating or simplifying our lives because we work smarter not harder Uh, so we wanted to do it that way and that that's what they did and in almost every case You know insights came in far ahead and I think the only place it lost was our policy service Uh, and that included it lost only a number of steps and that included steps like Build a new policy from scratch and opt into email alerts So I could potentially argue that that one should have a lesser number of steps, but like I I take I'll take one We went on the other team, you know, I'll take one But it's fine Well having been on the on the receiving end of a security vulnerability report you bring in some third party They they do a multi-day audit process There's usually some black box script that they're like here log in to all of your systems as routes and run this thing Uh I think you not yeah And then they go well, we have a signed agreement that says you have to do this thing And so you do the thing and you export the you export their their findings to to a spreadsheet and it's literally Dozens of columns and thousands of rows And you just look at it and go I have no idea what any of this means And so then you just you have to manually start grouping grouping systems together You get really good with the whole filtering system in in your spreadsheet spreadsheet tool It's like sorted by environment sorted by system sorted by severity And whatever shows up on the top screen of this spreadsheet That's what we'll focus on this quarter because By then they'll come back and there'll be a whole new set of vulnerabilities and And then we get to look at you know another million cell spreadsheet that actually has crashed I actually had a laptop crash because I opened a spreadsheet I need to reboot and try it again I'm not sure I had that same spreadsheet, but I definitely had one similar That's a real thing Or I could log into insights. That's it one of the things I want to highlight I think I showed when I was on the dashboard. I had I think the number was 17 systems Registered so I mean it's not a ton of systems and I purposefully have old outdated systems in this environment because part of my job is to show you this stuff So for anybody watching yeah, I'm not just a horrible system admin I'm doing this on purpose Because I want to show some of these issues so across these 17 systems, I've got over a thousand CVEs that impact that entire pool of systems Um The default filter that we show inside of insights, you'll notice it right here above my mouse Systems exposed one or more we we have this turned on by default because we want to show you systems that are impacted by a CVE Um, if I clear this out, I just want to highlight What this is going to show for us is our entire pool of The database that we're checking against so there's over 11 000 CVEs that we're currently checking against in this database I'm only showing you the thousand or so that impact me and I can turn that filter right back on There's actually a bunch of other filters in here as well. So if I want to know only about those systems With known exploits, it's very easy for me to turn that filter on I've got nine of them. I can see what they all are right here again I can export that out to a list The lists are uh, they're whizzy wig You know what you see is what you get So if I've got this filtered in a certain way And I export it out. That's what it's going to show me. It's exactly what I have here on the screen Um, and I can also do it by you know, certain Base scores if I really only want to know about a range between uh, I only care about things between seven and ten There you go done. You know, it's it's pretty easy to go ahead. So I didn't click enter That helps sometimes it helps sometimes. Yeah Thank you. Oh, if that's the worst thing that happened in a demo, I'll be happy Yeah, but yeah, all of this information is here and it's filterable. It's just not a it's not a major. Oh, it is there So there's cvss base scores down at the bottom. So I got you know clear clearing the Exposed I've got over 2000 So yeah, it's a pretty it's a pretty neat little tool But one of the the next steps is once you identify that you have a vulnerability What then? so if I've got 37 3 7 7 5 0 Care be 5 here. I've got 10 systems exposed to it And I'm so i'm picking this one because it's a you know a decent amount of my estate that's impacted by this I can go into this cve again. I have this list of the systems that are exposed I can actually go through the remediation process right here So if your boss wants to know not only what systems are impacted, but what does it take to fix them? You can actually go through the process and I'm going to pick a couple of my My systems that are connected here to a satellite this 801 802 I'm going to click remediate and what this is going to do for me is uh Fixed stuff playbook. I'm going to create a playbook It's just got these two systems and I can add other actions to this so I can fix multiple things in a playbook I can have every system involved. All I'm doing right now is generating a playbook to handle this particular action Which is just upgrade the associative packages Associate, you know with this cve. It does tell me that it needs a reboot. So reboot required is yes I can turn off that auto reboot in the playbook if I want to I'm not actually taking any action right now Other than creating the playbook. So I click submit I'm going to click my open playbook fix stuff And that's going to redirect me to the remediation service. So if I look at my left hand now if we're there I'm now in the remediation section And this has got again. There's my cve. That's the only thing I have in here right now It's associated with Two different systems rel 801 and rel 802 and I'm able to download this playbook So if I click the download button in the bottom, it's going to show me a A zip file that it's pulled down. I'm going to on the other screen. I'll end that up And I'm going to bring it over here in a second And what we should see here is here is the playbook itself my thick stuff playbook that I just created And what it does so technical so my two hosts And we actually have uh, there's a signature here. So if we are actually going to run this playbook I'll talk more about that in a moment We've got a signature to you know letting us know that there's been no tampering with this particular playbook We're going to check for our update. We're going to update our packages Reboot the system if it's needed and then we're going to Rerun insights which will make sure that it's clear So at the end of this, you know, this is everything that's in the playbook I can see anything that insights wants to do right here If I want to manipulate this playbook and run it through An automation controller or formerly known as ansible tower You can do so Fully supported or you know, whatever method you use to run ansible totally cool We've got that playbook We generate that for you and we make it super simple To fix the stuff that insights finds just by downloading the playbook I'm going to pause there before I talk about the next step of it That seems super handy So what you're telling me is not only will insights with the vulnerability service bring all the cvs to the surface So I can prioritize and and see exactly what What piece of my enterprise is is affected? But now you're telling me that from the same console I can fix all of those issues in one swoop if I were crazy Yep Well, that that point are you're doing is you're downloading the playbook and you take that playbook And you do whatever you want with that playbook once you download it It's it's not any of my business what you're doing with that playbook But the part that you just said is the next step that just execute playbook That's a little bit different thing. This is the one part of What i'm talking about today that does require something more than just a rel subscription This particular feature also requires satellite So if you've got a satellite, you're good to use this You've already got everything you need if you don't Then what we're talking about is a smart management subscription which includes satellite So if you've got that you're good. If not, this is an advanced feature And the button would actually be great out for you. It's detecting in my environment that I have it if you Don't have that Particular subscription environment. You can't click this button If you have it, but you haven't set it up You'll get a message kind of similar to the one I haven't closed yet, which is do more uh Configure your systems with cloud connector and that's a job you do inside your satellite itself It's just an ansible playbook that you run and it creates a connection between insights and cloud.redhat.insights and satellite. So it just creates that connection for you What that allows us to do is click that execute playbook button This is telling me that my two systems are connected via my satellite that connection status is ready And if I want to all I got to do is click that execute playbook on two systems button It actually takes that playbook It transfers it over to the satellite the satellite if you've got any capsules is going to transfer it over to the appropriate capsule And then it's going to run up with ansible remote execution. So it's going to run this playbook for you It's running right here. It takes you know, this particular one probably takes a couple minutes to run and we could actually even see inside the UI Click on to 801. There's my playbook log and I can see what it's doing within this insights UI itself So that gives it uh, you know the ability as you were saying earlier to find it And fix it all in one place without having to bounce out to a bunch of different consoles So a use case that that struck me as you were talking was if I have satellite if I have my red hat enterprise linux boxes my red hat satellite and my vulnerability service and insights all interconnected. I don't have to leave this UI Otherwise if we're if we're doing uh downloading playbooks, I could see using those playbooks and calling them from Kind of bootstrapping them with within maybe a maintenance window playbook that hey on saturday I want to run we're going to patch these servers using an ansible playbook I'm going to remediate these issues by calling another playbook and I can actually just pre-stage all of this using ansible and then Oh, it's it's 10 o'clock on saturday. I can kick this off kick off a a playbook that calls other playbooks or or Run them kind of one at a time. Uh, I the the best day in my sysadmin career was when I When I discovered screen and later teamux because I could have multiple things going on in different places But I could I could take these these playbooks and pre-stage them So that when when a maintenance window does come around especially in the case of like your fixed stuff playbook that requires a reboot I can have all that stuff ready to go so that I don't I'm not sitting there the night of of a maintenance window going All right, it's it's it's now one after now. I've got to log into this box Let me look at my my spreadsheet here. Okay, so I've got to run this command and this command I just say nope kick off this playbook on the system and just wait for the system to come back up and How long would it take me to be able to go back into Into into the vulnerability screen and see that that that issue has been remediated But pretty much as soon as it completes running the last step of that job is for it to run insights again And that run of insights it just reanalyzes and it takes a couple seconds at most And it will update that hey, this is no longer a problem. So won't show you anymore I think kind of to your point. I went to the remediation section here So you can see these are all different playbooks that I've created through over the last couple months um It's in the ansible documentation for ansible tower automation controller Um that actually walks you through how to set up a job on the controller side that will sync with this Remediation so you can actually sync all of these playbooks you create inside of insights Over to your tower your automation controller And that's that's where that gets really handled so from a high level overview we can we can We can do more research on what cv es are we can we can look at specific cv es We can edit how they look to our own dashboards. So like this is this is we can change the the business priority So that they get so they get kind of surfaced we look at things Like which systems are But systems are affected We can either remediate live We can we can download playbooks to remediate the the situation to remediate those particular vulnerabilities later And then we can come back within less than a minute really and see see the updates to our our dashboards So if I mean if we took a an evening long outage we could fairly easily and quickly go through and remediate any of those top top 10 percent of vulnerabilities affecting our our If affecting our environment use the use the technical term there took me a moment Yeah, we've got um I went back into the playbook that we started a a moment ago And I only did two systems. I mean, this is my lab environment. It's smaller like for the purposes of You know this conversation like I want something that's pretty nimble happens pretty fast so I didn't want to run like 20 systems, but uh, yeah, I mean insights it's Can handle upwards. I think we're I think our key way testing goal last I heard is we're in the quarter million range for like concurrent operations and things like that like we like we This is built for scale even though i'm showing you two don't be like, oh, this is we can only handle two systems Just demo purposes. We're trying to keep it simple and easy but really the the complexity of this is It's there's nothing to it. I mean if you want to set this up inside a satellite You go into the satellite you run a playbook and it configures everything for you and you know Satellite any kind of technology. We know that it's sometimes it's not always that easy It should be that easy like assuming that you're running an environment out of the box like it should be just run the playbook creates creates the connection between cloud.redhat.com and the satellite and you're done If you're also using a satellite anything that insights is doing for its analysis that collection It's all proxy to the satellite by default So you don't have to worry about another web proxy or a bunch of individual nodes reaching out to the internet It's all proxied through satellite and we we should talk a little bit about that collection before we end Well, we've got about Five minutes and and I do have one other question about about roadmap But yeah, if you want to if you want to tackle that first, let's yeah, you put a link already in the chat I see no actually that's the insights one We've got one on the insight security page if you can drop that one in there We've got a trust page out there already. It's publicly available that talks about All of the information that we do with insights and I've got it open here on my screen as well If you want to know anything about what? Information that insights collects. It's all here for you that collection itself is really small It's uh, I think last time I ran the average the average was like 384 kilobytes of data that we collect and upload We do not target any personally identifiable information. We don't want your user names or your passwords I don't want any of that stuff All we're looking for is the base information that we need to detect If the situations exist so something like that. Oh my god vulnerability Like we need to know if that particular package is installed What version is that package is that package running and is it listening on certain ports? So for that particular vulnerability, we would need to know certain things for one like How long is the system been running before a kernel crash? We would need to know like the uptime. So, you know, it's basic system stats. I refer to it as metadata All of that information is right here on this page And if you want to see what insights collects, we actually have a One article on system administrator system information collected for insights There's another one further down that talks about how you can obfuscate your ip address and host names You don't want to give them to me. That's okay. It didn't bother me a bit. I don't need them It does make your life a little bit easier if you want to do the remediation steps Because if you do the data upload all we're going to see is a uid if you've obfuscated your your host names We do let you set a ansible host name or a short name that makes it easier for you But if you just hide everything At some point you make it a little harder on yourself to do the the remediation inside the ui So it's just something to be aware of But all the granular controls of data redaction are in here You can create a yaml file or a local file on the box if you want to that Hides information by pattern by keyword by command uh so if you've got Certain information you want to make sure never gets collected like you have full 100 percent control of that And you can generate a collection and you can look at it there locally So you can do it without sending any data to red hat and be like, what does this thing collect? uh Most of the people that go and inspect this have absolutely no concerns with the information that we're collecting because it all makes sense And the benefits far outweigh The risk to most people Well, and what's great is the the services opt in to begin with. Yes So it's not like we're collecting this data in the background at the saying Oh, you turned it on. Oh, by the way, we've been no, it's it's it's completely opt-in We did have one customer call us to the mat and say you've collected something that we consider personally identifiable information and we were like Uh, okay. Tell us more like like what's going on like you know, we have this We have this uh this sensitive name And we see it in the collection and this is very concerning to us and we're like, all right. Well, let's let's dig in. Let's find out what happened right Turns out that a sys admin had taken this project name or company name I don't remember 100 what it was, but they had named a a daemon inside the box With that name And it was running and one of the things that we do look at is what are running processes because we need to You know, we can test and there's other cool things we can do with the facts within our drift service and things like that So we did collect the name of a running service And it happened to be it was named this sensitive name and they're like Our bad Okay, we see And then but we did show them like hey, you have full control You can you can cut this out if you want to make sure it's never collected like do it It put any name in there that you're worried about be it your company name or a since you know, super secret project name Whatever it is, you know, put it in that Put it in that yaml file or in that deny file and you know, we'll never collect it We don't want anything to do with it But it is important to note if you say I don't want you to collect anything having to do with the networking stack Then we potentially couldn't tell you something about like that that early example of a Of a bonding issue that we we wouldn't know because we're not collecting that stuff But overall like I I think the information that's collected is is it's fairly innocuous. It's all metadata and you You have the control over it Take a look for yourself and we hope you uh, you like what you see and you decide to use insights Yeah, in insights. I said this at the top of the show, but it's worth repeating I wish I had this as a sysadmin. It would have saved so many hours of I mean, let's face it a lot of these fixes are update this package or disable this service unless absolutely necessary Things that are literally one command, but you'd have to repeat that command 400 times But the the worst part of it was the the research that went ahead of it What cvs are out there? What's affecting me and then manually going into doing that research? I mean that part took a hundred times what it takes to just fix the actual issue and with insights It's it's all collected for me. It's brought up in the background I mean, it's it's an amazing tool and whether you've got 10 servers or 100 servers or 10 000 servers It's I mean it's this this saves so much time And we only dug into the vulnerability and we glanced at the remediation Pieces there's there's depending on your account. There's anywhere from like seven to 70 other services But yeah, I I can't get enough of of insights And I I love how much Insights has grown and how much it's it's tying into all of our other other products Whether that's rel or open shift or even some of the add-on products like like satellite or I forgot what what other ones I was going to mention or Ansible was was the other one And just how much all the stuff interacts now It makes it so easy to be a systems administrator, especially in an age where security is becoming more and more critical And the the infrastructure is becoming more complex It's it's not everything is built behind castle walls in a moat now Now we've got multiple castles and then we've got we've got stuff in towns So, you know, of course, I'm alluding to this whole hybrid multi cloud Change I mean zero trust is is becoming more and more prevalent And all of these are things that we should we should definitely talk about more on on this show And john, obviously you've you've written up about about another 16 or so episodes that we could do But we'll we'll definitely have you back Right, right. Yeah, there is one. We'll definitely have to have you back on in the future Go ahead one last thing I want to reiterate as we've talked through this again It's included with your existing rel subscription And that is true if you're a developer if you're in, you know, full out On-premise server user if you're using aws Or azure, you know, you're using rel in the cloud And even if you're buying that rel in the cloud from the marketplace to buy it or you're buying that rel from aws You're buying it from azure. You still get insights. It's there for you Check it out Yeah, by all means play with it So that's that's actually a perfect jumping off point to mention that we have a discord server So if you if you look in the show notes of this episode, you can get a link out to our discord server I'm there a few of the other hosts are there. It's a growing community I think 10 new people have joined in the last like three days So if you haven't joined our discord server by all means jump out there ask questions Folks like myself and some of the other hosts can reach out to folks like john to answer questions to kind of help Talk talk you through this because this this may be this may be a live stream a show but The the reality is without our community without those of you that jump on every couple of weeks and join us live You know, we're just john and I sitting here having a conversation. So Those conversations are so much better when you join in You can reach me on twitter at it guy eric And uh, I'll I'll drag john kicking and screaming on to twitter a little bit more often But you can reach him at jb spinks And we're we're more than happy to to answer those questions I know i'm not speaking for you when I say that we we enjoy talking about this kind of stuff all the time Which is kind of why we why I i'm pretty sure I had to twist your arm pretty hard to get you to come on Oh, yeah, super hard. You want to do it? Yeah When No, I love talking about all of technology. I mean, it's just it's been uh Passion and part of my life for so long. So something like insights makes it really easy because we're We're doing cool things. We're helping people out and didn't get a whole lot better than that Definitely, so we'll have to have you back on real soon. Uh, meantime If you haven't noticed we are streaming live every other wednesday at I believe 2 p.m. Central. What time is it now? Yeah So we started about 2 p.m. Central or 2 p.m. Eastern every other wednesday and you can now catch us on Uh, you can catch the live chat on discord. You can join us over twitch opens open shift YouTube and as of this episode you can catch us live on red hat enterprise linux youtube channel All those links will be in the show notes john On on behalf of of all of us that watch today and we'll watch in the coming days. Thank you so much for joining us I really appreciate it Always a pleasure until then. I think that does it for uh for red hat live streaming today But join in tomorrow. I think we've got three or four shows Ranging from get ups to open shifts. So until then y'all take care. Thank you so much