 All right, good morning everyone and thank you for joining us today for another open shift coffee break episode. So today we will have a great conversation with my dear friend Tero. He's been our co-host for quite some time now and he's our I would say guest for as long as he wishes. And today we will speak about all things DevOps. So that's quite a lot to handle Tero. But I hope we're going to have a nice conversation today. So yeah, Tero, can you please introduce yourself? Yeah, so thanks. I have a proper answer. Thanks for inviting me here as I've been as a permanent guest and permanent host. Yeah, actually, yeah, I worked for Red Hat when I went this coffee, opposite coffee break was started and then I moved to Vanguil where I do DevOps now. So I, let's say I bring an outside view to the, to support these guys, you have all Red Hat eyeglasses. So they see everything from the Red Hat and opposite point of view. I bring some sanity into their thoughts and how things look outside and it's always good. I think it's, it's, it's sane and it's refreshing to have an outside view because one wants to live in your own world, you know, things happen outside and you need to be aware of what happens outside too. Yeah. All right. And why we are in this situation, Natalie is enjoying his wonderful vacation, probably in South Italy. So more bother goes swimming. So, and we don't have contacts like he has. So we just didn't have any guests to invite. So we are doing this like this. We have, we have more than enough to do. I hope. Yeah, yeah, that's true. And if you are waiting for slides or demos, there will be none. It's will be discussion only. So yeah, if that's boring. If anyone on the chat wants to ask questions as always please go ahead and ask whatever questions you have regarding the topic or even off topic if you want. If you want to know how Carol has always this nice haircut, you can ask him what he, what he uses. It's a barber as a service. It's a barber as a service subscription based nice. All right. So for, for those who are just joining for the first time, my name is the first driving and I work as a tech marketing manager for open shift prior to that I've been an open shift consultant for for about five years and I'm very happy to know to help people adopt container technologies and get better with with open shift. So, okay, so Carol, our main topic today is DevOps, right? It's good topics is you can talk whatever you want. Yeah, exactly. I feel like it's a it's a it's a big container where you can put anything you want. It can mean a lot of things to two different people so let's let's try to first set the ground and you know, get a better understanding of what lies behind DevOps. So can you tell us about, you know, your, your perception of what DevOps is? What is it? Why did people start adopting it? What are the goals, etc. Yeah, it's a, this is a nice question. The answer is always different because it varies and the short answer is DevOps is a culture. It's a culture of way of doing things like DevOps way. So whatever you do in your organization that helps you to like be more efficient, better quality, whatever there is between a feature and go in production and then running production. Everything that you do there between can be called DevOps and when you like not let's say not you can call everything DevOps. If you call way back, we have this old waterfall projects. It was not DevOps, but still those old projects, old enterprises can use DevOps tooling do like, increase the, how would I say, the increase the DevOps culture in the organization so that they can have small parts of the development process, automated or use some specific tooling, automated building testing, whatever. So DevOps is basically whatever happens and you define your DevOps. There is no, like Siamac said it when we have a last, we had a kid ops and DevOps episode that DevOps is a culture and kid ops is a way to like implement that culture. So kid ops is a tool, DevOps is a culture. And somebody said to me, actually just before I read, left red hat that he asked me that what is the most important tool for development in a DevOps culture. And the answer is pretty, pretty simple and really old tool. It's kid. So then you might think about it's, it's Kubernetes, it's containers, it's your development ID. But actually, if you think about it's version control system, probably it's kid, there's no SVN or CVS anymore. But, but that's the like, lengthy answer about what is DevOps. You define what is your DevOps. Yeah, yeah, sure. So let's, so thanks for, I would say, we are trying to set the ground so even people that don't have much of an understanding of that topic can maybe catch up. Of course, we see that the word topics is a contraction or the word DevOps is a contraction of two words. So we see Dev and ops. Again, just to know for the main viewers who are just, you know, kicking tires here. So why, why do we call it DevOps like what's the main thing that we are trying to accomplish with this, that maybe was not that successful prior to this culture change. So, yeah, why, why do we call it DevOps like obviously there seems for developers ops for operations. And can you just explain a few words what we are trying to achieve. I lost audio. So back. Okay, but I heard, but you all have seen the, the comic that there is a level that throws a binary over the world jobs. Yeah, and that's basically that's not DevOps. And that's the, that's the old way of doing that developers create a binary. And then they deploy it. So they send it somewhere. And then operators do whatever they like to get it running to maintain it. And if there is a bug, there is probably an Gira ticket or something that, hey, there's a bug, can you fix it. And then developers ask, can you send me some log files so that I know what is their message. So this is the old fashioned way of doing that there is no visibility whatsoever between the development process and the operations and running. And everything was insanely slow. I remember working with a customer way before red hat that you always had to create a ticket to get the locks from production. And that might take days. So how can you be like agile and fast and fix issues if you don't know what is the problem and then you have to wait for the actually know the problem. There might be interlocks null pointer exception. And then you see only that and then okay, where is it. When it happens. And not only that, but you also have to reproduce it like in an environment that that's going to be similar to what they have like in production. So, and also just that can take days if it's not, you know, automated and already I would say in a DevOps mindset. And how many organizations like copy of the production with production data and production load. So it might happen that when you have 10,000 request per second, this single bug happens. So you cannot simulate it. And what we can say that what we just described that is not DevOps. It's very easy to say what is DevOps, but then it's hard to say what actually is DevOps because it varies. But how we actually ended up to have this topic that all things DevOps is that it actually comes from my background that at red hat. Let's say, from June this year, five years back, I worked in pre sales, which was everything related to containers, Kubernetes, Kubernetes application development. Since I've been doing on development, I've been developing for more than 20 years. So the development part of the pre sales support was really close to my heart. And the DevOps, let's say five years ago, DevOps was was a term. And there were really good companies that tried to boost the term. There's a company in Finland that arranges DevOps days. And there were DevOps days globally to introduce the concept of DevOps duty, duty audience. But no one really understood what the DevOps was then. And there was this unicorn companies that did DevOps. And the problem was that there was not enough tool just because even the DevOps culture, you need those that tooling to implement that culture to do, or you have to implement those tools yourself. And when the DevOps got going, and when I work in the pre sales and try to sell the idea in double code, sell the idea of how you can like transform your organization to be more a child using DevOps, container native, cloud native, all these passwords. During that time, I was, let's say I got pretty good at selling the idea, but that was really opinionated view of only one part of the DevOps. So how get code easily and securely running. That's it. Because in pre sales, we don't see what happens once the code is running. That is, yeah, that is DevOps. Yes, but only tiny part of it. Yes, exactly. Yeah, I totally agree with that with you. And I've been sharing the same experience where mostly, you know, when we were speaking about DevOps, it's basically how do you get your code as fast and secure as possible from dev to prod. And most of the focus was on like CI CD. It was like there was a confusion between CI CD and DevOps. Whereas, of course DevOps is much bigger than that. Automation is a big pillar of DevOps for sure. And automation can touch many areas. It can touch like testing, as he said, can touch provisioning of infrastructure, it can touch deployment of your applications. So automation can be at the heart is at the heart of DevOps, of course, because you're trying to avoid all those manual tasks as much as possible to have repeatable processes. But yes, I agree that DevOps is not just like, okay, let me commit my, my change. And then, oh, there's a container running with the new version in production. That's, that's, I would say, a very simplistic way. But it helps basically, as you said, sell the ID when I'm saying sell, not in the meaning of like really contraction with the customer. Of course, that's an end goal at some point, but like to have the conversation with the customers about how they are doing things, how they are pushing code into the different environments, how they are provisioning the environments, how they are testing the applications, how they are getting feedback from production, when they want to, you know, improve maybe the performance of their applications, fix bugs, as he said, get logs. They don't necessarily, you know, want to wait three days before they get the logs and stuff like that. So all of those things, you see that we are starting to go into different, I would say, subsets of what DevOps is. You have to start somewhere. You have to start somewhere. And like having automated CI CD or building your containers automatically and deploying in that's a good start, because you mentioned how important automation is that some say that you cannot do DevOps without automation. And I kind of agree, because the DevOps way is to, in the cultural, it's kind of, you use DevOps as a culture and tooling behind that to make your organization better in software development. And you have certain limits that you can reach with manual tasks. You just cannot be like efficient enough to match the like the cloud native and Kubernetes and container native stuff and the base, if you don't do automation. So it helps a lot. And now it, if you like, want to put your code to the container and Kubernetes, that is insanely easy nowadays. There is a, you can build containers on Git, in GitLab, in GitHub, in BitBugget, you can put them locally, you can deploy them with Maven. There is a trillion tools to deploy those containers. And for developer, that is awesome, because the developer, usually the developers task end when they do a commit and it's merged and it's built and deployed. Feature done, next feature. But then it comes to the, what I actually, once I had done the five years of pre-sales, when I moved from pre-sales, let's say I moved to the Vungal and now doing a senior stuff engineer in DevOps at Vungal, and I basically moved to another side of the date table. So from sales to the engineering, my role is help engineering, help as a redeems to do their job better. And what you see then is that there is a lot of other stuff in DevOps than just shipping your code. Yeah. And then what developer, like, we touched this, like observability points, that one important task is to developers should know what's happening in the production, like if there is a bug that the happening production. It's a, the developer should have access to the locks and see the production locks so that they're masked that there is no, like, secret information in there, just because the customer data. But there shouldn't be a process to get the locks. It should be just a tool that I can read the locks so that this, you said that this is like you said that DevOps term comes from development operations. This is something that developers do something related to operations. They actually check the production locks and that was earlier the operation people locks. You created the ticket, you got the locks, you read the locks, did some testing and so on and so forth. But now developers go check the locks. And then they, okay, I know where the bug is, I fix it. Good bug fixed. And you all have seen the like DevOps having this number eight sideways, infinite number. There is a big arrow coming back from the developer loop, the operations loop. There is a feedback, continuous feedback. The continuous feedback. Yeah, so, and I think that that is the last part of the organizers in DevOps they implement properly. Because you can do like a lot of automation, you can have a really fast pace when you features. But if you don't have a like feedback loop. Do you know that these like these DevOps tooling that you use that they actually move you to a correct direction. You might be releasing really often. But every second release has to be rolled back because of the bug. So let's say that you released once per week earlier, and then you had a tooling you released twice per day advice per week. But every second release you have to roll back because there's a bug. So basically you are not any effective you still have a complete like success for release once per week. Yeah, so yeah that so that that means that so yeah I let's let's pause for a second and try to make that more I would say visual. So I love music so I'm going to try to have a visual image that depicts what you are talking about so imagine that you are like you know playing at a huge concert. And you're playing guitar but you don't have the feedback like you don't know what you are playing. So everybody's doing their thing. You are strumming or playing your solos but you don't hear what you are doing. So basically you can get pretty pretty lost because although you feel like you are doing your thing. You don't know if it fits well with the other guys you don't know if the the audience is hearing anything because you basically don't have the feedback you don't hear what you are doing. And you know when you're playing concert you have those feedback amps basically that give you monitoring monitoring what you the other guys are doing and also what you are doing. Yeah I'd say that that feedback from production is basically the same thing is okay we've we've done some stuff we've deployed the application faster maybe but is it better is it working. Is it improving that we have less bugs etc etc so that's that's the like the importance of that continuous feedback loop but now it takes it takes us to another correlated question which is so what do we basically track. So how do we okay we have a continuous feedback but of what like what should be in there. What do we need to to monitor to to have the the correct information to basically be able to diagnose the application troubleshoot and compare and know if we have improved performance and stuff like that. So do you have any hints like what types of metrics any KPIs. That's a really good question. And usually the answer is you get what you measure. And yeah, yeah, so you the problem usually is that in organizations you have. And those goals have been linked to an API's. Let's say you have to do 10 deployments this quarter. That's your goal. So the idea is that the team will do the 10 deployments and. Okay fine 10 deployments check we got our target we got our bonus. That is what I mean when you get what you measure. But then, if eight of those 10 deployments were all packed, you actually did two deployments. Yeah, so that is overhead and you even have overhead compared to what. Yeah, yeah, so you should be doing 18 deployments in the in the next quarter. And then that that is actually the hardest part of measurement the velocity of the DevOps team. So, like, there is pace and quality. And the quality can be a box security less costs so that it's kind of the pace is easy. Acts amount of deployments but but then you have to kind of. Because you have to match the quality of the base so is for deployments better than to. Successful deployments with good quality. So that if you have a deployment that. Like raises five new bucks, it must be like. Not that good deployment and deployment that doesn't raise any bucks and actually adds new features. So, this is really hard to measure but you have to kind of. You have to everything needs to be like started in the metrics from the business. What brings money to that. Yeah, but let's pause on that because it's very interesting and it shows how tricky it can be. So, yeah, we're saying okay is it better to have four deployments that raise no bugs than one that raises five books for instance but. What is like, because you have released earlier that one deployment. It gave you or your users enough time to, you know, to go through some new features and then you get but and then you are able to fix those bugs. So, then it gets better than the four deployments where you didn't catch the box. So, you know, it can be tricky, as he said to say, Oh, this is like the V KPI and is going to fix all our problems because it's basically always relative relative to do something. There are quantitative KPIs and qualitative KPIs. So, yeah, it's always hard to like define those sets of key metrics. And I think that the like it is for organization that hasn't like doesn't have a long DevOps history. They don't know the KPIs, they don't know the velocity, they don't know how many deployments they do. So it's very hard to actually do these measurements. But one easy measurement for development teams is that DevOps teams is that how many deployments and how many rolebacks. So if you had to do a role back that wasn't a good deployment. So, so if you are maintained to success for deployments, that's a good and easy KPI. And but then it's a good point to start because then we can add the quality in there that how many new features were in this in this deployment. And then you can have a how many deployments and how what's the lead time for new feature. So that's also one that how because that usually there's if you have a multiple deployments you have a ton of deployments you probably if you have a good quality. You probably add a lot of new features. But if you like have a bad quality, you have a lot of deployments but not much new features because they're fixing bugs and hot fixes and everything. So it's kind of balanced that could deployments such as for deployments and how what's your feature throughput. Yeah, can you pause there for one second again? Yeah, sorry if I'm interrupting. No, so try to keep your ideas, you know, fixed so you can go back to them afterwards but this reminds me of a conversation that we had with one of my former customers. It was a very big e-commerce shop so one of the biggest e-commerce shops in France and we were working on their DevOps transformation basically. And one of the key topics that we wanted to address at some point so everything was automated infrastructure provisioning was automated application deployment was automated. So basically they were doing many many deployments per week. So one of the topics that we at some point we wanted to address was basically what you said like the lead time to have a feature released into production but it's not easy to have that because you need to have, we spoke about tooling and know what you need to have across the whole DevOps chain. And what usually lacks is like traceability. So you start with the business with a business requirement. And then that business requirement turns into a desired feature in your application. And then this feature is implemented in some sprint. It gets tested, it gets rolled into release. It's deployed. And it's actually, you know, as you move to the different steps, you are going through maybe 10 different tools. You know, somewhere where you capture the business requirements can be a spreadsheet can be a tool like JIRA or, you know, this type of more agile ways where you define your user stories, etc. The question was how do you keep track like how do you know what feature has been deployed in what binary, actually, or if we speak about containers, what container images like what tags what releases implement what feature. So we had to think about, you know, all the links that you need to get from the business requirements to the user stories to the features to the code, like to the commit to the comments to the test sets that go with it to the release tags that implement that specific feature. So you can have the whole trace, you know, all the way up from development into production. And then you can say, okay, so now for this release, we have implemented these five releases. We have implemented these five features. Okay. So, yeah, it was not easy to set up all that, you know, traceability along the chain. Because you need to have the proper approach, like the right people need to speak together to understand that you as a developer you are working on this specific business capability, etc. But you need to have the tuning, then to stitch together all the pieces and be able to report on these these type of information. So you can then have your, your metrics, like you said, we have implemented these five features, etc. But the next question is, okay, so we have implemented five features. Okay. But is it better to implement five features that do this? Or is it better to implement this feature, one feature that does this in terms of business value. So again, you have to introduce some new KPIs towards like what value does each feature bring to your application in terms of whatever you want. It can be in terms of money. It can be in terms of quality perception from your customers. It can be, you know, customer satisfaction, etc, etc. So, you know, and then your KPIs start to be linked together. And then you can say, okay, we have delivered that value because you have the whole chain to go back. And that's the life. If I had a magic wand and unicorns, I would. That's like the, the best solution that you have a business feature that provides that brings money to the, to the, into the house. And then you have features and you know that I implemented this feature and that increased already have a new five kills so that you have accept money. And actually what we, in the mobile games, game refinery is one part of Bungal. And in there we provide analytics that because we know what the market is in mobile games. So what feeds what games are good, how much money they are making per day per market. So we can actually tell the do our customers that if you implement this feature, it will bring you X amount of revenue. So that's like, because that's, that's then easy. And because then if you have a development team and of course development team has X amount of velocity, let's say that they can do three, three features in a quarter. So then if you have 10 features in the backlog, it's easy to check that we take those three because those bring the 80% percent of the, of the revenue from all those 10, 10 features. So it's really easy to then prioritize what we do because those features bring money. And that's like in the software development, it's really hard to map that since one business feature might split to five different development teams to 10 different mic services and 100 different comments. And then how do you map that? Like the feature going to be implement on Twitter outindication. Yeah, or implement fit outindication, which is a very simple you implement outindication and you start seeing users coming from Twitter. It's easy to map but implement something X, Y, Z with has like data team needs to do something and then so it's insanely hard. So it is kind of, I don't know many, many organizers that can do this fine to KPI for development teams that they can actually see that how much each implemented feature brings money to the business. But that is the end goal. That's where, because that's the target. Yeah, yeah, because also they, there's if you have a bug that users cannot log in, we know that how much the company is losing money. That's really easy to like measure because there are metrics from the history source that okay we lose 100,000 per day so we should fix this pretty fast. So there's at the end of the day it's always money. But if you start your DevOps, like road, trying to map business feature money to development teams, you will never get there. It's just too hard. So speaking about features that you know you can map to direct value or money, just a funny example but this morning. My wife was doing some online, you know, grocery shopping. And after like 45 minutes of doing her e-commerce stuff. She checked out, but everything was lost and we she couldn't like go to the other and she basically you know has to go back to okay doesn't work. All these you know e-commerce stuff. Let's go back to real life and then. So that's, you know, that's a pure loss. And then you can easily say that the shopping cart that clears itself is directly your money, your revenue. Plus you waste the customers time. Yeah, it's just priceless. Exactly. So that happened to me also. I know, you know, those types of features. You need to make sure that they work. That's true. But we actually now that we went to the KPIs and how you defined actually that this was a spin off like you asked about KPIs, but we got off the track. But the most important KPIs for the application is revenue. How much of course revenue or money can be different companies. It can be a different kind of it can be. It's might not be concrete money, but it might be some so and so forth. Do you do you agree if we instead of money we say like value like identify value as a generic, you know, everything. Yeah, something that we want to get that basically drives value and how you define that value basically depends on on each context. Yeah, but yes, exactly. That always has to be the whole organization and let's say, as a reteam DevOps team development teams that needs to be the driving force. Because if you try to have something different than the actual value, then you are basically fooling yourself if you have a like how many requests per seconds my services can take. It might be that okay, we are taking a lot of requests. But those actual requests are coming from monitoring agent. They are not actually really users. So there has to be like a simple automated way of collecting the information and see that if you don't have like, if you are a web shop like the grocery store, you know how much your cell is how you know how much you sell. That's a concrete value. It's easy to measure. But then if you don't have that concrete value, you have to somewhere from other metrics, you have to campaign, like, collect the enough metrics to know that with this amount of requests, this amount of unique users via generality, in average, creating this amount of value. And if then you know that if we do 10,000 unique users per hour, we know that each user brings value 10x. So if we then lose, like, suddenly our services run only 500, 50,000 requests per second, we know that we are losing half of the value based on the metrics that we have. So those kind of, this is more like two as a redeems that should work close with at one of the as a redeems work really close with DevOps that the each service should have an KPI or service level as a lay so that the teams know when the service is working and medicine and then because if you don't have this visibility, you just don't know why you're doing the correct things and then it goes back to the development teams. So if you don't know how the services should perform, you never know are your changes going to correct direction. So if you make a change that decreases the throughput, you can then calculate that okay, this change actually we're losing money. That's a simple calculation. It's not always that simple. And also the like this kind of dashboarding monitoring alerts is now that I have learned it, I think that it's the most important thing in DevOps, the observability, so that you have a feedback. In Scrum, it was very well defined the feedback loop. What happens so you get feedback, what you're doing and then you assess your doing based on the feedback. But in DevOps, it is continuous. Yeah, continuous. You always see what's happening and you see how your changes affect the running system. And it can be whatever. It can be just, there's a huge amount of nice tools collecting metrics from Kubernetes clusters. You can have a fine dashboard, you can see the throughput, you can see the errors. All those, let's say, better to have like too many than too few. Because that is one thing that has like grown really well in the last years that how you can monitor your workload. If you think about pre Kubernetes era, it wasn't this easy to like collect everything from the running environment and create a dashboard. Now it's usually, yeah, now it's given it's out of the box. Yeah, because because I think there has been a lot of standardization about like, this is the expected format. And so then implemented in your application and whatever tooling that can interpret that format, and then digest the metrics and then show it in a nice graph or whatever. I think about like Prometheus metrics or stuff like that, where you can, you know, there's basically sort of agreed standard that people started to adopt and then with that you can scrape the metrics and then show them in your in your nice and so speaking of that, one of our customers, it's one of the biggest do it yourself shops again in France so that's a different e e commerce partner and those ones were happy open shift and they are happy open shift customer so that's that's cool to see you know that they are implementing those type of, I would say, new e commerce, microservices based shops, etc. And one of the nicest things that I have seen when I went to their DevOps plateau, as we say like the DevOps room, where you have different types of people like you have networking guys have storage guys to have. So it's a mix of different skills, but within the same team and they had their, their monitoring screens are all over the place. So it's a nice matrix that they had so they, they have, I think, more than five, five different brands, like an inch brand is a big brand in itself. And basically they had the dashboard like all the technical stuff, like the request, you know the number of errors, although those sRE technical metrics that they were displaying over their their their dashboards, but they had also one very interesting one which was the total revenue per act. Like, like I was seeing live how much revenue they were making each, like, maybe 30 seconds or whatever, and you can compare to the previous day, like, were they doing better or less than the previous day for each application for each like e-commerce brand. And I thought that it was like, you know, great. I would say DevOps adoption and mindset because they are, you know, they are tying their technical stuff to the business value to the revenue and they are continuously monitoring the two topics together, you know, they are monitoring and within the same team. So it was, it's of course, supposed to be confidential, but I was, you know, we have, we had some agreements and stuff like that but basically I was in the room and I was monitoring with them how much revenue they were making alongside the technical KPIs. Yeah, and when you have that, that like key value in there, then if you see a change, you can see a drop. Yeah, then you see, okay, what happened, there was a deployment, okay, what happened that deployment. So it's very easy to see the effect of what's happening in the environment. Yeah. Yeah, and they were also implementing things like, you know, cannery releases and AB testing and stuff like that, like trying new features and seeing the impact on their revenue line. They release a new feature or change the website design or maybe shorten the checkout process and stuff like that. And they check the impact and they compare to the previous version or previous day. And they can instantly know, okay, so now we, we know that we have improved whatever KPI they are monitoring, or no, okay, guys, you see there's, there was a huge drop of whatever because as he said customers can't log in or the shopping cart clears itself, because some bug was introduced, et cetera, and they can instantly correlate, they can say, okay, we did a deployment there. And here's the impact on that API. Yeah, and that that's, that's also one thing that I learned. When moving from pre sales to actual engineering that when doing deployments in demos and pre sales, it's a different thing. It's you deploy you have a rolling deployment. Yeah, yeah, everything works. But when you run hundreds of millions of requests per hour, you have hundreds of replicas, when you do a deployment then you have to do what you just said that you have to do a cannery. You have to see a rolling cannery so that you run, let's say 1%, 5% of the load. And then you check the metrics that did this new version increase or decrease the, let's say, let's say checkout time. And then you have instant feedback, okay, it was bad, we roll back and try again. And then or try to have like minimal load and try to figure out why it's a bad release. And this is just this is the observability part that is so important that you have to know what happens when you do a release you and actually that brings to another cool topic that I love. Yeah, I'm gonna see if you're thinking about the same thing. No, this is new one. This is my favorite chat ops. Oh, yeah, because you can. That's not the same that I have. No, yeah, I know what you're thinking about. But then you cannot force developers to go to the metrics and follow the metrics, follow the dashboards, because they have code to write. So then, like what Prometheus provides out of the box, and a lot of companies use Slack that when you have, when you know your team's KPI, you can create an alert based on the KPI. If your 25% tile is lower than X, Y chat and have an alert so that it's a proactive way of knowing that something is wrong. And it is very easy to implement and it is key for for the productivity for the DevOps team, the developers, so that they don't have to use time to follow the metrics, but they are, they are being told when something is wrong, when they can see the metrics. And this is one thing that is to implement and you should implement in your DevOps environment. In the in the first phases. When it can be when release is successful, when there is an auto scaling that you can have a ton of different different alerts, maybe too much, then you are dead or spice like. Yeah, exactly. But then you can you can set different channels and have alerts and subscribe to. Oh, okay, so that brings us to another topic that we wanted to talk about, which is basically, okay, yes. For sure, it's, it's good to catch stuff in production. It's, it's good to know what happened. It's good to know, know to have that feedback, but is there isn't there a way to to detect these things earlier. Like, we hear a lot about that concept of shift left, shift left, everything. Can you tell us a bit more about that? Like what's what's the, what's the, the principle of shift left or left shift? Yeah, left shift, shift left. It's basically, yeah, it is why it's left shift, because in all the diagrams, development is always in the left side and operators in the right side. Yeah. So what, what it means is that you more move tasks that have been done by someone else than developers, you move those tasks for developers. Let's say that the first shift left topic was unit testing, I would say, that you actually developers write the code. You don't ship it to the tester where testers tested do some basic testing and then it goes forward in the pipeline. So actually unit tested, you, in that, let's say test different development, you write the test cases, then you write the code to match the test cases. Yeah, it shifted so much that it comes now before the code itself. Yeah. Yeah. Two farm left. That's a good example of shifting left that the test team responsibly went in some level to cover so that the test driven development. And also then we saw tooling like Selenium or these so that you can actually do user interface testing as part of the build. You can like record a script that you do so that when you have a build you actually do some basically in the user interface. But now. And there's a good tooling or such a unit. And so forth. And also with new rest interfaces, you can actually with Quarkus, you can run a containerized test environment, just in the command line that you can test everything. And unit tests are always run when you have a code test. So it is actually not that much. It's part of development. And you don't think about testing anymore. But of course, some people don't like this development because you. It's some for some people, it's too slow. And if you do testing when development wrong, you create the code and then you write a test that always works. Yeah. But but the observability part is also good in here, even that you do unit testing. And so on so forth in the development phase, you still have to know that how many bucks you generate, because you will always have bucks even that you test all the set and get methods from your entities. So you have to know what to produce. And, but even more important is that what I have seen now, which is really good that the next let's say that what is currently going on is shifting left security. So earlier, developers coded, they wrote something and then the binary was deployed. And the security was meant that there was a security officer who worked with the operations they created an insanely secure environment with SSL and TLS and everything. Now, like, let's, I've wrote some Quarkus code today with TLS. So now you define in the development phase that where is the certificate you can generate the certificate with less and so developers can actually create the TLS certificates. So that's one example of shifting left the security but also when you do node or some other development and I use VS code. You get an instant feedback that these packages have vulnerabilities. So this is really important. And this is like the security part done right. It is supported the DevOps and let's say DevSecOps so that you have a security as part of your development. So shifting left security and you have the feedback loop continuous feedback integrated your development environment. So you have instant feedback so you don't have to go to the pipeline and then pipeline build and then you have security scanning and everything. And after a couple of hours you get back that okay this library has a vulnerability you need to update then you start over. So this is really good. And now, of course, there are also tooling that the Snook, Stackrocks and a lot of companies are building these tools that I would say that they try to make the security part invisible for the developers. So that the code that is produced is secure by default. That is really good. And it's like done right. And I think that next might be that when enterprises go to public cloud. There has been actually in the cloudcast.net there was a good topic in about public cloud costs. I can't remember it was this year, but the idea is that now it would be nice if we could shift left the costs. So that developer teams know that when they do the code when they deploy something this is how much their project is consuming money from the public cloud. So it is kind of one thing that if you have 10 replicas running and each uses eight gigs of memory and every you have a unit price for memory in the public cloud you know how much it costs. So then you have a non-functional requirement saying that we need to lower the costs so that we can scale more. So it's very easy to then like okay maybe we should have more performance code so that we can run less memory so that you actually start seeing the full infra costs on the features. And I think that that's next thing. But I like that still the shifting left the security dev sec ops. It's coming. That's good. I think next will be that the shifting left costs and whatever next shifting left ops. So there is only dev then no ops. It's going to be the dev dev. That's the future. Yeah, yeah. Yeah, yeah. Dev sec dev. So just one one thing we are one minute close to the end but I think we we can we can spend a bit more because we are you know, free here it's early. In EMEA, they are not awake yet in the other time zones, or especially in the US, where we have other shows. So, yeah, so what you mentioned about shift left shifting the costs like reducing the costs. And actually, just having a better control over the first like an understanding of if you run your application that much time that much time on that type of infrastructure that's what you what what it's going to cost you. So, public clubs do provide these type of dashboards but I have to admit that they are very hard to, you know, to correlate, you don't always understand why you you get that very big bill. Until you spend like a lot of time going into the details of that stuff so that's one of the areas where I see customers trying to, you know, implement that automated billing, which is tied to to to consumption which is again tied to matrix. How much CPU, how much memory, how much networking, how much storage do you consume over a period of time how much it costs you for each unit of all of all of those things to be able to define your your costing models, models, etc. But I think one of the, yeah. Yeah, yeah, I was just, you know, get into one point but I believe with you know the evolution of containers and things like Kubernetes compared to how things were done before with traditional VMs. It was so hard to get a VM that once you have it, you grab it and you hide it so no one knows that it's running you keep it in your closet and then when you need it, you deploy something on it, and of course it costs a lot of a lot of money, because you are wasting resources, but now we are shifting more towards a on demand infrastructure consumption, where for instance, as you said, okay you hit your merge button you have a feature branch development maybe where basically you provision automatically all the infrastructure that you need to build your application, run your tests for that specific release and check that everything is good, you get your results back, and then you decommission the ephemeral infrastructure containers or whatever you have provisioned. I think it's, it's much easier to do it today without spending too much time into automating stuff than it was before. So, I believe that this will definitely help reduce you know the costing side of things. So yeah, that's also part of the you know, part of the DevOps is like how you can automate provisioning and deep provisioning, because if you're not decommissioning your environments, it's good to automate the provisioning but if you don't automate the deep commissioning. And then there's a whole issue that you're not addressing which is like the ongoing and always up costs of your IT infrastructure. The costs is like, you have to think about the cost of the, let's say that you're a startup you start building something you have a couple of users, you don't see costs as they're just cloud cost. But if you don't take the costs as like KPI, when you start actually scaling and you actually, you break into the market, if you are not like, if the developers are not aware of the costs or the SRE team, it's, it might be like impossible to estimate the costs per user when you scale. And then like good example of the to have effective environment as a cost is that with a Kubernetes with request and limits you can easily set workload so that you have a full Kubernetes cluster. So every node is 100% full. But then when you go inside under the Kubernetes, you see that you only actually because each continent is not using the amount memory you're giving you actually running 20% of the of the actual hardware utilization. And then you are wasting money, any person's best money. This is one of the KPIs of how, how much money you spend to run this. Yeah, how much money you spend to run this amount of a month. So if I have 10 times more customers, will the cost be 10 times more, or will it be twice. So this kind of estimation because not many startups, let's say can handle like from 10 users to 100 users and at the same time costs went to 100,000 to 1 million. So you have to be aware from the feature. Yeah, from the beginning, how much it costs to run the stack. All right, so Tero, we are five minutes over time, but it's really interesting. So thanks a lot for being here. We have one question from Jean in the chat. He says, so yeah, it would be good to enter that. But from your perspective, like, so how developers debug apps, is it directly on production communities or open chief clusters, or do they need to reproduce it locally or in development cluster. So what's what's your take on that. You've been a developer. How did you do it before? How do you do it today? It depends. My favorite answer. But it has been made way easier to develop locally. Personally, I don't, when I do something, I write something, I don't run on my local environment. I run it in a command line or container. I don't run it in a Kubernetes environment. So it's easier to debug. But then you have these, you have these problems that, okay, it only happens in the production. It only happens with production data. Let's say at game refinery, we have a staging environment that can have a copy of production data so that we can then do testing without interrupting users. But also one, one thing that I have seen is that having your staging environment in a production having taking only small amount of the data so that you can actually do the debugging in the production without breaking the whole production. So let's say one of the replicas is actually running the staging version. If you have 100 replicas, you have one of those is running. But it varies. And also, it depends what you need to test. If you need the whole stack, you need 10 mic services, you need a couple of databases. It's very hard to build the environment. So you can run everything in containers in local environment, but it's hard to maintain. So if you're developing one mic service, then you usually use your development or staging environment to map the rest of the environment from there. So you run one mic service locally. Yeah, exactly. But some problems are Kubernetes native so that they only happen in Kubernetes. Then you have to go there in the Kubernetes environment and remote debugging if it's depending on the, in the text that you're using Java, it is pretty well working that you do remote debugging in actual container. So that's a good thing. But yeah, back to my favorite. Yeah, it depends. So yeah, again, yeah, thanks. Thanks very much, Tero. So good question. And again, there's not like one simple answer for that. It depends on the criticality of the bug. If it's impacting data or it's just visuals. What's what's sure is that whatever you do, you need to make sure that you can push it to production as fast as you can and as securely as you can, meaning that if you develop locally, then you make sure that you have maybe an automated pipeline that can push your fix and deliver it to production. Of course, never go to your container and and make changes within your production application. That's a no go. But yeah, you can you can do stuff in your pre production environment you can you can check logs and stuff in production to understand what's wrong. Maybe you can make some fixes in your staging, you know, preprod environment to make sure that you know you have a copy of the data as Tero mentioned. The one thing you said that it's a key part in the DevOps and proper CI CD is always to have rebuilt, do not repair. Don't do any of the fixes in the don't have to say it's still container or no, but fix it. Don't put tape on your on your leaking pipes. Yeah, yeah, change the pipe and fix it properly. All right. Thank you very much. It was very nice conversation. Thank you. You know, I think we're going to have a follow up on that because it's a very broad topic. There are many things that we haven't discussed, but at least you know we have covered some some some key areas of what we call DevOps and what we need to address. So thanks a lot. It was nice to have you and have your, your, your view as an outside, you know, of the outside world. It's always great to have that. And hope in the in the future I can bring like more depth to concrete, like metering a metrics, what we use at one. Yeah, that that might be in the future and there is some confidentiality stuff, but I hope that I can show you something concrete that how we do things. That would be nice. That would be cool. All right. Thank you very much. Thanks for everyone who has been with us today. Don't forget that they are always great shows coming up on the Twitch channel. So you can go on red hat. Twitch.tv red hat. Open shift channel. And of course everything will be available on replay also. So thank you very much. We show a great day and we'll see you again in two weeks. But thanks. Thanks a lot. Have a great day. Bye bye. Bye everyone.