 Good morning. Good afternoon. Good evening wherever you're hailing from welcome to a special edition cube con office hour Today we are talking about why service mesh and I'm joined by a bevy of red-haters And let's just go around the horn and introduce ourselves I'll go to Langdon first since he's a usual suspect on this channel this time of day Sometimes well, maybe not this time of day, but little no, this is a little early You know as I was just saying to my wife You know these if these Europeans could think about you know doing things on a more reasonable time That'd be much more useful Obviously, it's a little early We did want to kind of come and be a little bit more in you know the time zone of the of the actual conference You know, but that means that you know Chris and I are as often as the case suffering for coffee So if you're a fan of the level up hour, I'm Langdon White and I host that show It's usually on in about an hour and we'll be back next week But currently what I do at red hat is what's called technical marketing management Which means some weird term for you know, I kind of do things like dev advocacy for service mesh among other things and So here today, I'm you know talking about service mesh with all of our friends here Chris to hand it back to you or should I hand it to pick a person and I mean just round the Robin here All right, thanks lining. Yeah, likewise. Just getting getting the caffeine flowing this morning I'm so I'm Jamie Long here. I'm a product manager here at red hat for open shift service mesh I've been I've been here for I guess I'll six six eight months now. So it's been kind of a wild ride of talking to a lot of customers who've been You know getting on their service mesh journey So I think this is kind of a good time to start diving into a little bit about, you know Why they're why they're picking it up and some of the challenges they've run into and and why they look to the red hat So absolutely awesome Maltron or Simon flip a coin Yeah, I jump in always helps when I turn my mic on. Hey, good morning. You've got I'm Simon Seagrave I'm the product marketing manager for service mesh here at At red hats. Yeah, pleased to be on board. This is my first ever open shift TV session. So great to be here President team and yeah, look forward to these this discussion coming up Good morning, everybody. My name is Mauricio, but everybody all my friends is coming maltron and before anybody asks No, I'm not the younger brother of Megatron. He's my uncle. He gets great Christmas grifts to me I'm the product manager for Service mesh most particular for Kiali and Yeager and I'll be working for red head for almost eight years now and then be quite a journey Awesome. So please ask questions or share your feedback here in the chat around the CNCS slack number six cubecon red hats and You might get a t-shirt So, yeah, it'll be one of those things where If you ask a good question, we will be happy to give you a t-shirt So speaking of t-shirts, I mean Simon you are in marketing and maybe we should open there It's like where are our t-shirts because I don't have a service in that t-shirt yet And I'm a little I'm a little disappointed and I'm sure the audience is as well You're a hard man to track down Landon. I think those people chasing you so you address when I You're constantly moving so you're hard to send something to but there is something in the post. I'm sure I am flying the banner this morning. I have my red hat t-shirt on. Oh, yeah Well, I mean, I have I actually have a red hat for no t-shirt on Oh, no, that's very nice. I have the new open shift shirt that the logo is at the bottom of it Which doesn't help us on air here. Oh, right. So yeah, it looks great though. I'm standing up, right? I've got the the retro one. So that's a little easier to say Yeah All right, so I'm gonna stop this slide for now and we're just gonna talk folks Have a little conversation about service mesh here. So why service mesh everybody? Yeah, well, actually, let's kick off with a question here. I mean, you know, we're seeing a number of scenarios Out there in the world also with our conversations with customers on a daily basis here Around you know, what they ask, you know, why service mesh? Why would we use service mesh in the enterprise? As we know, there's a number of compelling reasons for that, but let's let's use that as a starting starting point to kick things off Yeah, yeah, I'll jump in I guess It's it's interesting to I think because service mesh it's it has a you know It's I think it's a technology that's kind of came on the scene with a lot of height, but some Like a lot of people, you know, they're then it kind of kind of died down I think because I think it was sort of an overarching solution that solved a number of different problems for people And and it wasn't really, you know, maybe it wasn't clear what what the the one that people would be be be looking to to grapple onto I mean when I came in here from a developer perspective, I was mostly thinking around the lines of You know, it's going to save me all this it's going to save me all this code for things like encryption and tls and instrumentation and Error handling and whatnot and so those are the scenarios that I that I was thinking of but I think as I've come in Here it's been a little bit different with regards to how how enterprises are are picking it up It's uh, you know, they may have had It's kind of at a stage and it's an option where they may have been been playing with it in a few different teams A few two teams might have got started with it, but now they're looking to go more serious. So you have like these um more centralized operations or infrastructure teams. They're trying to standardize how they uh, um, how they roll it out across the You know across across multiple teams um, and I guess I sort of sum it up as to I guess really why why I see them adopting it and then we can kind of expand on this but it's it's trying to bring some order to the Uh to the to the to the chaos that you get when you have a lot of different teams developing microservices Here and there it's uh to trying to bring some standardization some high-level security policies control, uh, some Some level of uh overarching control around, you know, things like who can talk to who and Sort of need to need to know and what not around around inter-service communication So this is more this is more important than ever. I guess, you know With a lot of companies out there adopting a cloud native approach to things and particularly, you know architectures where you have you're breaking these huge monoliths or legacy application into smaller microservices um That's great, but there's that involves a lot of communication between these microservices and Right. Yeah, but we talk about it. I was just gonna say it like we talk about it like it's new though, right? I mean, you know, if you go back to like service oriented architectures, uh, people are constantly looking for You know mesh uh deployments, right to be able to keep track of those services You know, you look at something like corba You look at something like, you know, a sun application server or sorry a job application server You know, all those they're all really trying to accomplish the same goal, right? Is like you have all of these different pieces, you know, kind of happening in Your deployment whatever that means, right? and you need to make sure that you're keeping track of them all and The the upside to kind of microservices, right is we're kind of going grassroots up You know the downside is we're going grad streets up And so I think and and what I think is interesting about this conversation like about service mesh is In particular the enterprise aspect of it, right? It's like why Why does the enter like why does an enterprise care in a sense about why about all these services In perhaps a different way than you know, a different kind of uh, environment might you know, like a startup or You know, like an academic environment or I mean, you know, sometimes those are enterprises as well. I mean But the point is like one of those, you know, what's the difference between kind of doing it at an enterprise level instead of like You know for lack of a better term a toy Um, but that's mostly because lack of coffee I don't really mean like a toy per se as much as one where you're less regulated say I think you want key to understand exactly Why service mesh is so important? You have to understand what was the situation before the technology came along As we know it because even before that like you have like several libraries There is a computer specific language that you have to add to to your workloads and specific to microservice like simon mentioned And you have to maintain those libraries. You have to be able to communicate among other services And one thing people just don't realize is Once you you split the monolith into a microservice You got you got a whole range of benefits from it, but also there's a whole Level challenges that you have to manage If service a talk service b and c and c fails You have to be able to pinpoint where the problem is you have to be able to understand exactly how to cope with this particular problem And you want to avoid disruption at all times in uh before even we talk about this this Technologies like you have different kind of libraries that trying to solve that particular problem and also Managed like changes over time always be incredible challenges So I think this is one particular Key information that's important like language said like once you got like Just basic layer what we call service mesh It eases out the the experience for the developer and also to manage that workloads in a seamlessly easy way So just drilling down on that a little bit more them ultra on how would have that worked in the old days? You know i pre pre service mesh for example Would uh, I mean Well, especially in large large enterprises, right? You're going to have these different development teams potentially working on different aspects of of an application Whether that's a monolith or whether that's a microservice based application But I'd imagine they're going to have their own ways of wanting to communicate between each other and and you know Definitely with a service mission it helps standardize Those communications and helps abstract that from from the concerns or worries of of the development team if you're standardizing You're right on spot simon. I think one thing is like people do really jump into the bad way to go microservice um In people writing embrace and especially we have to ask ourselves exactly Why microservice is now because distributed workloads is not something new We've been talking about like like landon said we talk about corba We talk about all the other kinds of distributed workloads in the past. So why is that so important now? And if I have to guess I would say like all the mobile applications that out there Even apis that you have to interconnect it It posed like a huge challenge like how to like manage like a bottleneck workload to a single service So you have to be able to distribute it and you have to be able to focus On those single services that address one single customers or one single applications or even mobile applications um, so This this kind of architecture is has come to to stay here in our daily lives And you have to be able not only to deliver those applications and develop like applications like jamie just mentioned But also I think one key question that addressed to corporations and enterprises. It's like once we deploy it How we manage them how we manage changes How we make more secure and how you make more resilience to anything that happens over time I think I think one one thing you hit on there with changes in particular is something uh, nothing relates to Where we see this land at large large enterprises because usually service mesh is like service mesh is not coming in as It's it's not new as you know, it's like it's not sort of the new thing that's driving it I would say it's a lot of these companies are going through digital transformations where they're trying to Change how you know, they're they're they're trying to become from maybe a business that was centered around being say You know a bank or a retailer before and become a primarily a technology company and really have You know technology first and so they you know, they you know, they maybe re-architected they brought in Uh new development frameworks, uh, you know, they've had sort of pilot products here and there to get started with new microservices you might have had different teams adopting different, uh Development frameworks different places. I used to You know before red hat I worked for a vendor which was an application development uh company and and we uh, you know, we would actually Add a lot of the features of service mesh into the framework themselves because we were you know, we were into these higher more complex distributed systems and so we were thinking about a lot of these things but Uh, it's not necessarily going to be the case that at a large company You're going to have teams that are that are doing things in different ways And so you get kind of this dimensionality problem that you know, you have it infrastructure teams that may have Adopted, you know, they may have adopted open shifts. They may be a few years into their journey But now they're kind of looking around the landscape and they you know, they see different teams kind of doing You're doing different services in different ways and it kind of looks like uh, you know, that the wheel be the wheel of how to um Prevents how to handle service failures might be being invented in one place here and and in another place you've got someone who's uh You know, bringing in their own security libraries and and and and then those services may have to talk to each other and so there's there's sort of this big mass of Experimentation which they you know, they want to encourage but at the same time they want to you know, not have teams be Uh trying to try to do the same same thing everywhere and Start to bring some uh, some some order to that such that uh, you know As in any company if one person moves around they don't have to reinvent the wheel or rely on uh, you know each team to to know how to sort of Resolve the challenges of distributed systems that uh are you know, usually we we end up learning the hard way, you know So Yeah, like I think that's an important point that gets lost a lot of the time with something like service mesh is um you know When you're you know, kind of doing development or application deployment or whatever at like kind of enterprise level um You you don't not everyone is a guru, right? Uh, it's very hard to hire all those, you know, really high-end high level people So what you want to be able to do is uh Kind of can you move the complexity out of the standard developers hands and into some shared place, right? So it sounds like jamie at you know at prior company, right prior life They did that with kind of a centralized framework Which is fine problems. It's not very standardized across organizations, right? You can't take advantage of that kind of knowledge in other places And one of the things I think that service mesh brings to the table is that You can actually do some of those cross-cutting concerns or what's also referred to as like aspect oriented programming things like You know securing all the services to be able to talk to each other At one place and inject it into each of your services You know with the advent of like web as web assembly. I mean it's web assembly, but I can never I can't remember what the cool name is web asm wasm wasm. That's it. Um But uh What you can do there, right is you can actually Create your own, you know, whatever, you know, they call them sidecars and kubernetes But the idea is like you can create your own little cross-cutting concern And apply that across your entire infrastructure of applications or services Without requiring, you know, every single person to be able to write that code And usually those cross-cutting concerns are the hardest things to write, you know, like error handling Security, you know, the things that you don't want your most junior programmers to be taking a you know, a swing at So I think that's one of the other aspects of that of the service mesh concept that You know We tend to gloss over it But it's really really important in an enterprise deployment scenario in particular Because you can take they take some of that hard hard complicated development out of the hands of your junior developers Put it into a couple experts who then uh write it for kind of everyone, right without also having to build your own You know software framework that everybody has to use and you know When some people like rust and some people like go and some people like python and some people like php Um, you know writing a framework is not trivial So Langdon actually you touched on a couple of things there. So the first thing that that stood out from me Uh, I mean the nice thing is if you if you learn the skill set, right? And then the standardized You know skill set around service mesh It's something that you can pick up and apply to another position or another role somewhere else With another company if they if they're also leveraging the same framework Right services framework. So so there's something in there personally, you know with my developer or my it ops hat on It's worth learning this because it's a skill. It's a valuable skill And I can see it only increasing in the future that it's something that's highly transportable with you as you move between roles And as enterprises continue to adopt a service mesh approach Well, and it goes both ways right even even, you know, there's the selfish side right for myself You know, then I have more transferable skills, right? But also for the enterprise, right? You know, if you're if you're thinking about you're hiring new developers If everything that you're using internally is standardized, you're you're barrier to entry for those new people is much much lower You know, which is nice So, yeah, so I think that's a very good point and I think it goes both directions, right? Actually, I notice we we had a question come through on on youtube there from uh, kelly Sorry, I apologies my friend Kelly and narrow man. Sorry my that's my but my best Best attempt there. Yeah, we just take a swing in it Yeah, yeah, but They've got a question they're saying look, you know, what does this look like from a support perspective? You know, you've implemented service mesh in the enterprise there Um, you know, once it's up and running, uh, what are the support considerations? Yeah, that's that's a that's a good good question. Um, I think it actually plays to I guess another another bay angle where this is a bit different from the enterprise versus say a small small start-up for company Which is the the people around how they use the service much the person is um And the individuals who might you know might support it might management versus people who are going to use it um We see that uh, and it's something that I've seen a number of companies that we're we're working with So we have a couple of a couple of really large banks are kind of rolling things out like Yes, where you have a you might have a centralized operations team That's looking to roll out service mesh across an organization. Uh, and they might be you know, there there might already be some usage there um, but they're the ones who are um You know their their role is is largely to enable development teams and to to give them the uh, you know the the tools that they need to Get their job, you know done as easily as possible without having to worry about things like, you know containers or ip addresses or You know the kind of the the nuts and bolts of how an application gets deployed And so uh, at least with a lot of the customers we work with it It does tend to be more of a central infrastructure ops team that is Looking at how to to roll it out um However service mesh users tend to be much closer to the applications. Uh, so like developers or you know, it could be You know, it could be a cross-functional team that has You know developers and and you know people who are more focused on the operations side who You know who will deploy and and manage those applications But I think just having this sort of separation of roles between the person Who's going to roll out the service mesh and make sure that people have what they need from it First people who are who are going to use it is uh, it's something that we uh And that we see and that's why with with with open shift service mesh We we have a strong distinction between the the cluster administrator who is a person who might Again roll out and administer that service mesh and they're the one who's going to be supporting it So if uh, you know if a development team has an issue or they even want a service mesh They might reach out to this, you know, this this infrastructure team who's going to Deploy a service mesh instance for them That they they can then start to use and so that it might actually be the team itself that's administering that service mesh And so that's why one of the unique features of open shift service mesh is the fact that it has this multi-tenant model Where you can deploy multiple service meshes that are Completely isolated within an individual open shift cluster And so that's that's that's entirely for that model So the administrator is the one who's gonna gonna have a view in some level of control over all the mesh meshes But for each individual mesh, you've got a you might have a team or a group of people who Development team for example, that's responsible for a set of services And and they're gonna have full control over their their area of the mesh without being able to interfere with with any of the others So that's kind of the rough distinction of personas that we That you know, we we support and see with a lot of our enterprise customers Um, what one thing one thing is important like most of our customers is particularly in financial institutions like large banks like insurance companies And they want to rely on something that really is enterprise grade level that you can start using And it's very secure and this is something that we take very seriously Especially in in our product It's trying to offer something that the customers can rely on and if they have any facial problems They can like see red hat as a partner to support their in in their particular journey we also help them out of the customers trying to Get the solutions, but sometimes one thing that we see as one big example in 2021 Is the 5g rollout that's happening all over the world So a lot of partners is trying to develop like solutions on top of service Service mesh So they can actually cope with the high load and high workload demand that comes From these particular solutions And also, um, they have to be able to trust on whoever's providing these service mesh to be a good example of that Well, I think that also goes back to kind of my earlier point of it too Is one of the other advantages of being able to have a place to stick cross cutting concerns is You can you can then take help from vendors, right? You can take help from third parties because somebody else who's an expert in, you know You know auditing let's say, um, you can you can take a widget from them and have a place to deploy it across your enterprise or across your set of applications or across your services And do that in a really consistent way without The you know, the at least for me when I played an ops person on tv as I used to say the I was terrified of ever introducing any new software into my actual environment, right? You know, because that's it puts the the thing that I care about at risk, right? So even if I would need to you know Institute sorry something like auditing Putting that actually on the servers makes me very very nervous But if I can put it in, you know in the concept of whether it's in kubernetes or something else the concept of the sidecar I feel a lot better because I know that if it goes down or it breaks or it has problems It's not going to break everything around it um, at least In in ways that I can't quickly quickly get out of right, um Yeah, I think that's a good point actually that That also raises another point that we see a lot with customers who are adopting service meshes Is basically reducing the blast radius of if something goes wrong in there In their services, right? Like if you and you know, I mentioned that that the multi tenant architecture as well Well, okay, you know, you have one open shift cluster Well, at least if you know something gets compromised and it's in it's in one area It's contained to one mesh, you know, even within that one mesh you can specify You know exactly who can who can who can talk to who so it's uh You can you can really contain what happens um A lot of uh, a lot of our customers ask for for zero trust networking and that's a bit of a buzzword across the Um across the internet, but I think it gets it gets lumped in with with service mesh. Um I think the way the way I prefer to think about it is less about sort of zero trust It's more like like need to know networking where it's you know, it's not so much, you know Let's let's be honest if it's absolutely zero trust nobody's talking to anybody and nothing's happening. So It's it's more about just minimizing Awareness of data and you know minimizing of the level of permissions that you're giving across Across the mesh and across you know for individuals and services. So it's similar to that You know, I hate to constantly quote it, but I think it's a really good example You know the famous Bezos memo inside of amazon about um, you know Every service that you create should be externally accessible, right? And so what does that mean? Well, that means that every single service that you have You know needs to have some sort of mechanism To do a login to provide like attestation, you know, all that kind of stuff Which is not something you think about when you do things kind of internally facing But I think the this advent of you know, buzzwords, but there's you know, like a lot of buzzwords There's some truth to it. You know zero trust networking. There's probably others that I can't think of right this minute but basically the idea that You know, if you if you consider your services self-contained you will have, you know less trouble later, right? and One of the ways to to deliver that without having to make every single person write every single piece Is to do things, you know, kind of like cross cutting concerns or whatever. I think I think there There's a lot of really good advantages I actually wanted to quickly mention Just because there was a question here in the audience about and we have some other questions teed up But I think this one's quick, which is basically like What is sre and service mesh have to do kind of with each other? And one of the things that I think is really it's like for me, it's like Uh service mesh is an sre's dream right like like that's where you want to be doing You know being a site reliability engineer if you're unfamiliar with the term There was a couple a famous book about it a few years ago, but basically the idea is instead of, you know You know, basically putting out the fire of a given application in production You figure out you not only do that, but you also figure out what you know What lit the fire and keep that from happening in the future It's something like a service mesh, you know, and I'm gonna definitely give it to maltron here But you know with with something like service mesh you not only get You know the ability to put out the fire, but also a lot of information about what caused the fire That's one thing I wanted to say lingdon is Uh, I remember back in the day when netflix was all about talking about microservices And they come up with these technologies called chaos monkey It was just like yeah It was a way to test it out with your microservices before going into productions So you want to Get as much information you you could Doing development before going into production. So when it's something fails in productions, you know, exactly what to do Uh, not only from a development perspective not only from a coding perspective, but also from a administration In operation capabilities This is also something that service meshes already provides right there from start Even when we start the development on top of service mesh, you can simulate like failures. You can simulate like Services breaking you can Understand exactly what happens when networking is failing between your each service And with that information, you know how to handle Once you go remove that service mesh into productions sre is like Can actually pinpoint the problems They are more aware of the things that's happening. And one thing that I think we fairly to mention is Frameworks, like you said lingdon is pretty much like language specific Right, uh, and that that was really good like back in the days But today like you have the freedom to to choose any language you wanted And with service mesh, it also doesn't rely any particular language You can expect using this particular language and that boosts your innovations in your delivery So once you have that information like sre is this doesn't they don't have to be knowledge about Any particular computer language they use on service mesh because all the service that pretty much concise how they put your work And you can actually all you have to do is to change configurations Or monitoring capabilities or being able exactly to pinpoint what the the problems are actually failing So one of the things I want to make a quick point here was just that We are in in this part of this segment right in this answer to this question We're kind of specifically talking about the open shift service mesh product because If you go and look up a definition of service mesh It doesn't often include kind of the analytics and the insight and that kind of stuff of kind of what's happening underneath particularly in a graphical way um You know the the thing that we ship and we call open shift service mesh is actually made up of A few different upstream projects, right, which is fairly typical, you know look at rel for example It's got a couple of other things in there besides linux. Um, you know, so In that right what we actually deliver is the upstream is is similar to istio, right? And then but then there's also keali and yeager And we kind of we feel like that set of tools is really what you need in the concept of the enterprise service mesh You know, maybe you need any individual piece, uh, you know in some other scenarios But that's what we really think you need to deliver to get a true kind of service mesh experience So I just wanted to clarify that, you know, we're technically using a slightly incorrect term when we say We want to understand what's going on from a diagnostic and systems level Actually, that's why it's nicely, uh, linked it into a question that we had, uh, come through on youtube there Around exactly that actually application profiling and performance monitoring, you know, what what's on offer with regard to service mesh With regard that so I think that that answers that question nicely So I think I can answer that that's one thing is happens like, uh, Based on the architecture of the service mesh There's one particular component that goes to get the which each service we call the sidecar And the sidecar is actually communicated between all the meshes and also the people actually control the configuration to call the The control plane And that information is come across also. There is a lot of telemetry And from that telemetry we have tooling tooling especially in the issue offering and also in in our product Uh, we call the keali and yeager and with that information we can actually see graphically And and that especially for sres to be able to see exactly what this What is the service available to them? They can actually track the traces that goes from one service to another And trying to understand exactly if there's any proper particular problem or any bottlenecks Any things that not supposed to go in through You can act on it Sometimes if the configuration is not properly you can change the configuration right on the spot And most important you're not creating any disruptions for your end user It's just to add to that I think I think a key point for all this information with with service meshes that You get you get all this out of the box without having to instrument your your application code or Or the way open shift service meshes is set up, you know, you install it and it's it's there automatically pretty much So you just you know once you once you add your sidecar is your application and you deploy it You're going to have you know metrics and and traces that are are going to immediately allow you to start Debugging your or doing doing you know debugging your the performance of your application seeing where some bottlenecks are um and so sometimes I mean sometimes people have questions on you know, what is what is the performance impact of Of having these proxies and that's a concern for for some some people and sometimes sometimes that is a valid concern for most applications the additional The additional latency that's going to be added there is going to be so small that it's it's not going to be You know a part particularly if it's an end user application. It's not going to be so significant that you're going to Um, you know that it's that it's going to have a major impact. There are there are obviously there are all our exceptions that sometimes we we have You know, we have customers that are that are using service mesh at a very very large scale and are really pushing the boundaries of performance on it But I think the the value they see of it is that overall if you have a system of microservices where you Where you have no idea where the bottleneck in that system is just having that data and be being able to See where you have uh, you know where you have a bottleneck provides sort of far more Far more value as far as your overall system performance than than you lose with the individual uh proxy You know a little bit of lag you might get from some of the proxies So it's kind of like a and that changes over time as well So you might you might profile it maybe it's things are things are running, you know, really perform it now, but then, you know You know a week later a developer pushes out a new a new patch to a certain service and then suddenly the performance changes and Well, you know that this is where service mesh is going to help you Uh capture that or detect that bottleneck immediately even if even if the team responsible for that service Had hadn't done any telemetry or hadn't really thought about how the uh, How to instrument that service? So it's it's getting that that uniformity across a system that I think really helps with Not just profiling on the individual service, but for your your overall system of services, so yeah That's great. Jamie says it says those larger enterprise customers. We've got there They're seeing negligible just to confirm their negligible performance hits On the applications by running. Well, it's it's all it's all it's all relative for the most part for There's uh, I mean, there's a there's a page on the we're we're based on Istio And so if people are curious if you think we will istio performance There's a there's a page in the upstream istio that that talks about the relative performance based on different load request loads and And obviously the you know, the the proxies themselves The the amount of resource you should will will need to change depending on the the load that they're under but uh For most applications, it's it's negligible So yeah for the benefits you get from from yeah, there is miss and I should I should say I maybe negligible is the wrong word I mean not not not impacting, you know, like if you if your overall response time is a hundred milliseconds and you're adding You know two milliseconds that that's not going to be a big A big impact if it's if you're expecting one millisecond and it's true that it is so it's It's like this comes back to the like this is this is one of those types of questions which always, you know we're often kind of frustrates me because Um, okay, you know, so let's say there's a two millisecond overhead or whatever Um, yeah, but think about it in terms of a different way. Let's say all of your engineers had written that piece by hand Um, well now you have somewhere between, you know, maybe maybe some of them are are really brilliant And so they get it down to one millisecond, you know But other ones have added now 10 20 or bugs or you know, whatever else and now you have all this code You have to maintain and you know, like there's all these other considerations Where, you know Yes, it can't be an unreasonable amount of overhead, but at the same time Recognize the trade-off you're making right is that it's not it's not just about the numbers, you know And so I think it's really important to kind of not don't gloss over The fact that you're saving lots of development time. You're you know, lowering your risk of application bugs, right? You have all these other advantages that you're getting That are unrelated to you know a slight performance impact Yeah, yeah, I mean that's you have to ask yourself How much does it cost like a service not running or you're not serving your users? I've worked on multiple projects where service not running is a million dollars a minute. Um, yeah, for example So, yeah, uh, some of the banks I worked with Not not cool. So same thing here, you know back into my it ops or it had been day Honestly, I would have given my left leg for this because I used to work for a um, a large multinational oil oil company and I used to be responsible for the service in the North Sea Uh as as administrator I was the salt buck stopped with me basically The all the applications running on there and basically, you know, I was always told that if if One of those servers went down one of those services went down We started losing a million pounds because I was in the uk at the time a million pounds per minute Yeah, yeah I mean another another example was actually regulation side to is I did a lot of work for pharma And there's a there's a thing what they do is every time there's a negative drug interaction That has to be reported to the feds right about, uh, you know the fact that it happened You drop one of those on the floor like just a single transaction that you just lost That's a million dollar fine. Um, you know, so so it's it's not only the kind of straight up kind of losing money But it's also there's whole regulatory spaces where it's a big deal if you you know If you make these kinds of mistakes So yeah, I just always like to kind of give there's other kinds of things that you should also be thinking about that isn't you know The obvious, you know, uh, how many dollar transactions are you making? Yeah, definitely You know, one of the topics we touched on earlier on of the piece and I see it's come up as a question in the chat There from a from a viewer as uh, you know, can we provide sort of some more? You know, let's let's delve into the service mesh security side of things a little more, you know, what what you know What does service mesh come with? What's what's the you know, the value add with implementing service mesh and security? Yeah, I think this this is where a lot of people start coming to service mesh and I think that the topic on You know talking about the the impact of of you know Of service being down for a short time. I think is that as a kind of a good segue into this because because of where We're seeing the biggest adoption is uh, large financial institutions who are you know implementing workloads where they they really care a lot about this so I think uh certainly you know encryption and mutual Mutual TLS is uh, it's probably the big big reason why people start come to service mesh in the first place So, you know, if you if you look at a wide service mesh, there's sort of the huge menu of of features Well, the the encryptions the biggest one that that people jump on first and that's uh, I mean that's um I mean with with us it's something with open shift service mesh. I should say we uh Uh, think a lot about as well. So like what's providing the encryption? Uh, in our case we uh, we use uh, we're able to to leverage Open SSL the open SSL libraries that are that are created by the uh, the redhead enterprise linux team Uh, for example, and so we're able to dynamically link in those open SSL libraries, which are uh, Uh, dips which are actually dips validated So they you know, they go through the the lengthiest process of getting uh, of getting getting validated and we're just able to to take advantage of those so that means Without going into the sort of the more regular regulatory side is that those those um those uh libraries those cryptographic libraries have been validated by an independent government organization with their their own testing It's people know they're getting a reliable Uh cryptographic library for for all the encryption within within service mesh um I guess other other ways that we really consider security a lot is just again thinking in terms of need to know with regards to access around around service mesh, uh, so we you know make sure that um, you know anything that requires elevated privileges won't be handled by the development teams that'll be Either a cluster administrator or in the case of open ship service, we have our open ship operator, uh, which is a Is able to handle a lot of the administrative side of a of Of deploying and and managing and upgrading a service mesh. Uh, and so that can take care of all the uh sort of lower level, uh lower level things like, uh configuring network. Um, configuring networking for the, uh um pods and containers between between the application container and the proxy so that that doesn't require an elevated level of commission So a lot of things like that that we really kind of go go deep on to to make sure that we're you know Our boat doesn't have any leaks in it. So to speak with regards to to uh security. Um Yes, uh, I guess another that's okay Just recently myself and jamie we talked to a customer who has actually deployed services like in virtual machines And each virtual machine leak needs to uh, uh, a brand new certificate So it's like extra money. They have to pay in for each certificate And they are moving to a service mesh exactly wanted way exactly to get to make sure you have security in place But also from from a cost perspective that you have to add all the certificates all by yourself like manually say so I also as anybody who watches my show or knows me knows, um being kind of a hardcore Linux guy You know, I like to add to jamie's point is like, you know, all these bits that you see in service mesh Most of them came from rel right? Uh rel's been doing this whole security thing for a bit now. Um, you know, 20 years or something so, you know We have a we have a lot of the same interests, right? We you know, the open ssl that we're using is the one that's coming from rel, right? So, um, I think it's important to remember that You know a lot, you know, obviously not everything that is shipped in open shift is also shipped in rel But a lot of it is, um, you know, it's like building an application on rel. Um, you know, we we know how to do security pretty well And we like to make it first enterprise ready, but also consumable, right? So, um, you know So we want to give you, uh, you know, this kind of product or set of products that is something that you can go and Use out of the box as an enterprise, uh, without having to, you know Validate every individual thing and we actually go after the certifications that prove that what we say is true like phibs So I went Yeah, I went super deep on that question into the into the sort of the low level. Um I guess the the other angle on security, which is maybe a little sort of more general General service mesh at a high level, but I know it just because a lot a lot of customers Talk about this or ask for this is just having that uh, high level controller view of of your, um Of you know of everything but in but in terms of services. So, you know, if you just have have kubernetes Um, kubernetes does have the concept of the service, but I think traditional, um network security is revolves around things like ip addresses and ports and then you get into Uh kubernetes security and you you've got pods and containers and you know, you do have, you know Service accounts and whatnot, but when you coming up to the service level, it's you're thinking more in terms of services themselves uh, and so in terms of Being able to write policies about who's allowed to talk to do to who when you get to the service mesh level That's that's that's more in terms of services, which is closer to business domain, which is uh, so you're you're kind of Get you know, you're you're being able to apply security policies and rules Less less so at the level of you know, ip addresses and infrastructure and more at the level of business domains and business business pieces And so I think this is another sort of angle of that service mesh brings to to enterprises, but uh might be a little bit That's a I really want to kind of hit that as a use case to you Which is you know, somebody asked about kind of one of the use cases for service mesh Besides like blue green deployments. I mean, I think that's a huge one and I think It's something where you know, kind of your typical operator, right your typical sysadmin can't Can't manipulate their environment in such a way that they can offer security at that level, right or you know authentication authorization all those kinds of things, um, you know, uh, one of my other regular examples, right is, um You know at a company I worked for that managed pensions The person who can change the amount of the pension and the person who can Change the address of the receipt the recipient of the pension are two different people um That's very very difficult to offer to to Cause security around or authentication or whatever you want whichever piece you want to call it Um, when you think about it in terms of network security, right? Or you think about it in terms of you know, moving bits around, right? Um, but it's very easy right to do when you know something about the business service that's involved But if you have two microservices one that changes the pension amount and one that changes the address Well, now it's with something like a service mesh. It gets much easier Yeah, definitely. I mean we've touched on a few areas here like we're talking about, you know security particularly that's that's that's one area that we sort of delve deep into on on on the call here You know, what other areas, you know, with with their conversations with customers, um You know, how are we seeing how they adopting service mesh because there's obviously a lot of areas You know, you've got the a you know the blue green deployments You've got the the security side of things the application performance Monitoring do do most enterprises out there. Do they just try and implement it all at once or do they take it? Cause there's a lot there, right? I mean, it's a great, you know, it's a great great product But what will do they take a staged approach? Um, depending on their use case or requirements at the time Well, uh, I guess I guess what I see is some some do try and do that Like, you know, I call it eat the whole elephants and I don't think that tends to work well for any any technology You know yet open shift kubernetes kubernetes as well. I think that that also can be uh, uh, you know A problem. Um, which I guess also why when when we by the time we come in oftentimes There's been a pilot here or there of some some team that's that's a little bit of experimentation in a corner or something and it's usually when You know, I say we're we're most companies now is they have that are as they have that poc that are that You know that one team that's that's tried it out and now they're looking to uh to expand But I mean, I think that the really critical thing and uh service like if you already spent, you know A year or or or more, you know, just getting up to speed with kubernetes and learning about things like, you know pods and deployments and and services and and you know Our back and all these these fun things and and then you throw in service mesh and you've got like, you know destination rules virtual virtual services now. Oh my god, you know, and uh, uh, you know authorization policies and a whole whole other set of resources and configurations that you have to learn it can be extremely overwhelming um And so I think I think first of all it's key to identify why you're Why you're adopting service mesh in the first place because I mean quite honestly you not everyone needs a service mesh You know, if you have a relatively simple system a couple of services, uh, you know, you might you might not need one And and if you and if you can uh, you know, if you don't if all you need is like an api gateway or you know Or an ingress then great. Just just go with that um, but uh, and then once once you once you really know why I think then it becomes easier to to adopt say, you know If what you want is the and the mutual tls. Well, then you can uh, you know, you can just start with the few policies that you that you need to adopt that and and you still will even let you do it in an incremental fashion so if you have a couple of services that you want to bring in and um and start, you know set up use use istio to to um uh Manage the mtls and in particular for those services you can and then if you have others that are you know, maybe they're they're they're still not you know, they're They're not ready for that and then they can uh, you know, that there's a there's a more permissive mode where it can can just uh You know allow all types of traffic Um, and you know, I guess yeah just in general where I'm going with this is just just to try and take a Take an incremental approach and you know, uh sort of adopt new resources and new configurations and as you you know as as the need comes and um, you know, just just rolling out a service mesh in the first place is is a is a big step and so um, you know once you get over that just to take it to you know, don't don't try and buy off more than you More than you need Yeah, another scenario why we see our customers is Most of the time they want to deploy the applications on top of service mesh But they also they want they want to be sure that these existing services are able to communicate to other existing services So cross communications between things that's not necessarily service mesh. It's another area of concern Uh, and also maintain the security maintain the stability So, uh, I think james is right on the spot. You have to be easy You have to be able to learn all the new concepts The service mesh and I think we we have to be honest with everybody here on the call is um, take it easy and just bear in mind that Non-necessary service mesh is a solution for everything Sometimes you just use like a simple, uh, api management can actually do for you And it actually can involve to more complex architectures That maybe service mesh can actually help you in that particular Service mesh is also something that either developers and also SREs have to be able to understand all the concepts Before they actually move into production well before and and Kind of understanding kind of what it's doing before it becomes beneficial to them instead of just another headache another infrastructure piece they have to worry about um And I think I think this was kind of the point of your answer a little bit But I would also kind of say like you don't necessarily need to adopt a service mesh as the as your first rollout, right? It can be something that gets incrementally Not really incrementally added, but more like something that can be added a little later You know as you see how your environment performs You know one of the things that is a big differentiator of kind of the microservice or you know cloud native or whatever you want to call it architecture push is that we can do things that kind of you know I talked about this in the beginning about like grassroots level rather than top down But that kind of means like we don't have to make every decision upfront And this is a huge change in the software development industry, right? Like you know going from waterfall methods to agile methods You know all these concepts is about incremental change to your environment to actually do just what you need it to do Um a service mesh deployment in some ways you kind of have to do a bunch of stuff at once But you don't have to do it immediately. You don't have to have and jamey talks about this I know a lot, but like you don't have to have one service mesh that covers your entire environment In fact, that may not even be a good idea You you can think about it in terms of problem domains or application domains or things like that And you can adopt them as you want to right Yeah, I think in general I think of the service mesh as being better if it's scoped to a team or an or an area you know not not being too too big or But I guess it's another I don't think I just want to quickly throw it there on You know how how to adopt service mesh one thing that I think is really helpful is the key alley tool which Which gives you a visual representation of of service mesh and The way that the way that opens your service mesh is set up, you know immediately it's just there when you when you run it and so You know, even if you just have a couple of services it can be really helpful for for debugging You know if a sidecar is not there or why one service can't talk to another or you know We mentioned performance that can can visualize your your mesh and show you the relative traffic between services as well as giving you interfaces for for for configuring and debugging the service mesh resources which um as I mentioned are a whole other kettle of fish to learn if you've just come through crew bin IDs and so um, I guess I'd kind of giving a You know plug to the key alley tool as being a good way to sort of honor amp to service mesh We have reports of customers using key alley as the main dashboard to look at the service at a given time So they can actually spot any particular problem that's happening. They can act really fast And it it also looks really cool Um, so there's that huge advantage speaking especially is like if you're a developer of services one of the things that's actually Sometimes really difficult to to visualize is like how they actually all hook together Especially if you're a more senior developer where you kind of just get pulled in to kind of do one little piece or whatever Getting a sense of how everything is actually working is Way more challenging than you might think it is Without kind of graphical representations these days when you look at a monolithic application Part of the the appeal of them is that they're relatively easy to understand When you talk about microservices, it gets really hairy really fast And any mechanisms that you can use to visualize them as graphs or whatever Really does make a big difference More than I think most software developers like to admit Yeah, even again with my ex-assessment hat on, you know, that's the one one thing that really stood out for me with the You know with the um open shift service mesh offering was the key alley dashboard You know again that that was brilliant the insights that provides and super clear as well And I like the ability how you can drill down to troubleshoot and everything Again, that would have been super useful about 10 years ago for me A little bit longer Yeah, literally 22 years ago Ish We wrote a visualizer and a tracer in visual basic to work on com across the hdp Because of this exact same problem You know, like we've been you know durendev. Uh, who's a common Audience member on on the channel asked earlier is like, you know, have we you know, and I kind of mentioned at the beginning Have we been looking at this problem for a while and I'm like, yeah, just me Like just me myself have wanted an answer to this problem for a long time And I'm not sure that you know like open shift service mesh or istio or any of the other You know choices are a complete answer yet But it's definitely down the right track It's definitely a good step and it's definitely got the ability to grow in the right directions to cover the Any other use cases that we come up with all right Well, we got a minute left and there was a ton of questions and y'all answered a lot of them So I'd like to remind everybody that we are in the cncf slack. Please come join us The channel name is six dash cube con dash red hat and We can continue the discussion there And we can all go about our merry little day and no more about service mesh But if you have more questions feel feel free to join us And definitely let us know There or elsewhere, you know, if if this was useful to you, we are happy to do more You know, we could do it on the level up hour. We could do a one-off show, you know And and talk about this more You know, I think we were all nervous that there wouldn't be any questions at all So we're pretty excited that there are so many questions So, you know, we we're happy to talk about this stuff as much as you would like probably more than you would like So please let us know if if you think we should do it again jpdade says yes, so Well, there you must now All right, so thank you all this was a great session. Thank you everybody in attendance. Thank you all for Being on air today. I know for some of us. It's a very early morning Thing lane and do you have a link to the cncf slack handy? Uh Cloud native. How do I get dash native that's yeah, it's cloud dash native dot slack dot com I got it. There we go. All right I thought you meant the like deeper invite link, but I think you can get from there. Yeah I think you can get to it from there. If not, uh, It's definitely hit me up on twitter at chris short or email me short at redhead.com and I will get you in somehow some way Um So with that, uh, I would like to show you you're taller than me chris. Yes, I am. Um He's taller than most people. I am a walking oxymoron as I like to call myself I am the human embodiment of don't take yourself too seriously. My last name is short and i'm six four so if in meters that's About two but it's two and change. Yeah, like two and change. I think yeah, so i'm pretty tall last name short It's awesome. Like it's an amazing part of my life. So I like it So thank you all It's been like my surname there chris. I never go sailing for obvious Because you'd go down with the ship. Oh man don't go sailing with me. Yeah Oh, wow, okay fair enough. This is gone. Very hard And with that have a great day everybody Bye. Bye. Cheers