 from a company called CollabNet, we are a product development company. So you have heard from the title, developers want change, ops want control. So you guys can guess, you know, it's mostly about DevOps. I want the session to be more interactive. I want folks to, you know, ask questions in the middle and, you know, it's mostly knowledge sharing, your experiences, things like that. So how many of you, you know, have faced this challenge where, you know, your developer code, it works in your environment, but it fails miserably in production, okay? That's good. I see a few hands, okay. And some of the things I am going to discuss is about our experience, how we are able to solve few things in our DevOps environment internally as part of a product development company. And I will also quickly run through a small case study that we are able to solve for our customers. So I am not going to sell any product and stuff. There are some guidelines, things like that, best practices I will share. It's up to you guys to go and implement how you want in your company or in your products. So first I am going to discuss about, you know, what are the challenges operation teams have and what are the challenges, you know, developers have. And we talk about, you know, DevOps value proposition and what are the building blocks in DevOps space, how you can build your own DevOps. So these are like general things and then best practices. And how we solve some of the DevOps challenges within our environment, in our product, we host our product in cloud, things like that. So you can look at that as a takeaway for you guys. And then one of our customers case study. So let's start here. So as I started earlier, we always have a tug of war between developers and ops. So developers always want change and they are always in a fast-paced environment. They want to develop more. They want to get to the market faster, learn technology. And ops typically wants control. They don't want change in their environment because, you know, they are always tightly network and they don't want to, their environment die or, you know, fail, things like that. So dev wants agility, ops wants stability. So the conflict between the two, they will always fight the conflict between the two is what DevOps is all about. So let's see some of the developers' challenges. So on the left, you see all the developers' challenges. On the right, you see ops challenges. There's a wall in between. So typically everybody knows, we are not going to talk about application lifecycle here. But typically everybody knows if you follow practice, any lifecycle you have like a plan, code, build, test, deploy. And then you also have release automation. So developers nowadays, you know, you have challenges where you build a product, even our product, it has a desktop application, the same product runs on web, we also have a mobile and the same product has to be tweaked to deploy in a cloud. So all the developers have these challenges and then increasing complexity of the code where nobody, you know, talks to ops until it goes to staging and then before going to production, people talk to ops. Within that time, people develop, they get new open source things, tweak their environment, tweak their code, nobody talks to ops. The third one is, you know, more agile. As we go through agile, we have continuous integration, continuous delivery, things happening faster, more iterations. But that same momentum doesn't happen beyond continuous delivery. It takes a lot of time to go from staging to production. Let's see that in future slides. And in case of ops, they have their own challenges. There's so many compliance regulations from the government or so many security vulnerability things they have to take care of. They have to take care of SLAs. They have their own challenges. And ops doesn't know internals of each and every app they are deploying in production or to their customer because they are in their own space. And ops also want to try to reduce any deployment changes in their infrastructure or upgrading in their environment. So let's see the challenges in industry. I got two things from Gartner and Forrester. So out of all mission critical application, what Gartner talks about is 80% or failures are due to configuration or release integration, handoff issues between ops, between dev and ops. Those are 80%. Forrester talks about overall failures where 40% is due to configuration or release management related. So these are some of the industry challenges that exist today. So people can ask me a question. So how do I know whether my dev and ops are working together? How do I recognize this? There's a problem within dev and ops. So there are a few pointers. You can have your own checklist. You can come up with your own things. But I just threw a few things, a few points here. What happens is in a typical development and lifecycle where people deploy code into DevTask QA. And once you go to QA, it takes a while to go to staging and production. That means that's one of the signs where you have a problem. And the next one is failure and deployment. You went all the way through, and then once you go to staging and once you go to production, it fails. And ops says it doesn't work. Developers say, it works in all my environment. So things like this, a few other things, non-compliance risk. And some of the non-standard process where you tell ops to manually tweak something. And once it goes from staging to production, you don't do that all the way through. You have to open up some port in production, which you don't do in other environments. And there is no documentation procedure. So there are a lot of ad hoc things happen in enterprise application. So all my experience I'm going to talk about is mostly for enterprise, big customers theme, which will be continuing on throughout my presentation as well. So some of the top benefits, let me talk about top benefits, and then we'll get into how the DevOps have been incorporated, or how we solved our challenges. Some of the top benefits we have is cost improvements, where when the application fails in production, you have cost. You miss your SLA, and your productivity is down. And you also have, once you have DevOps in place, you can decrease your defects in production. And you can also improve your agility. And some of the other things, business things we can talk about is traceability, where you trace all the way from requirements. You have stories. You have epics. Then you have test cases tied to them. Then you have continuous integration. And then you go all the way to build. After that, it falls down through your software lifecycle, where you should try to improve your traceability. That will also help have a solid solution. And then last one is, align all your stakeholders between DevOps, even your business people, where it will be easy for someone to go to time to market faster. So let's see how it works. This is a simple workflow. It's not any product or a tool. And when I talked about traceability, and from your workflow, you can see developers. They develop a code. And then somebody does a build. And then it goes to QA. QA does the code. And then it goes to ops. And then somebody manually deploys or you can automate it. Let's see how we improve our DevOps. So when these things happen, there's more frequent sprints. And then even this happens in our workplace, where we have very frequent sprints. We do continuous integration. And we deploy. And then there's a bottleneck. Because QA does the same turnaround. Because Dev and QA are embedded. It's in agile space. They're all same. And they do things faster between sprints. End of a sprint, you have some testable code that is around. So all these things builds get backlogged in QA. It takes some time. It's not in the same time frame where it goes to staging or production. So third challenge is when you go to different environments, in our case, we have a product where it has to run on Windows. It has to run on CentOS. It has to run on Suzy. It has to work in mobile. And it has to work in all private cloud or internal cloud platform, as well as public cloud that are available in the industry. So if you look at it, when you build, we have to make sure it works. Any feature works in all these platforms. So the way we implemented DevOps, for this discussion, you can think about having a small robo, where you tell the robo, I have finished all the continuous integration delivery. And then we tell the robo that these are the deliverables that are here. And we take this build, and it goes to this environment. And this is the workflow. It has to go through this many approval process. And QA approves it. Then it goes to release manager who approves it. It goes to ops who approves it. So you can build this kind of workflow within your system. And it can be homegrown or you can use open source. You can use tools that is available in the industry. And then the biggest challenge we have is when you go through multiple platforms, there are a lot of configuration things that happens. Where if it is a Linux or a Linux kind of thing, you have direct structures that are different. If it is a Windows, your direct structures are different. And when you build mobile apps, it has to work in Android and iPhone, things like that. So a lot of configuration things that has to be tweaked before going to deployment. So those things, you have to somehow externalize. There are other tools available or you can do homegrown. So in our case, many things we have done homegrown stuff. And you can also come look at some of the case studies in our website. So as I mentioned earlier, like a robot where it tracks, you record all these things. And then ask it to play. So when you go for a next release, again, configuration changes. So your whole operation steps that's been manually done, it's kind of codified in our case, where it's all written in code kind of. So when next release something changes, we change our workflow, we change our model, and then again play it for the next release. So this happens release after release. So I'll show you how this will be done from a story perspective how our apps are doing. So let me explain to you the building blocks for DevOps. When you start going and building your own DevOps space for your own environment or your product, first thing is packaging, which I said you have to make sure. For example, it's a web app. It's a year file. Or if something else, you zip them and then unzip. So all these things you have to decide. For from our perspective, I'll say we have four different kinds of packaging because we are having four different products. So this packaging strategy has been defined after continuous delivery. You can again do that through however you want. So you decide for web app, it goes as a year file. For my windows, it's an EXE how we build it and package them together, which is slightly different. So this is for first building blocks you need. And the next one is a workflow. I a little bit touched on the workflow where when you're deploying your package, you want to define first, shut down my server and then take a backup of all the logs and then take a backup of the old version and then take the backup of the data and then deploy a new server, stop the server, things like that. So those kind of workflows you can define again per platform, per environment, things like that, which has to be repeatable. So all the things are externalized. And nothing is kind of hard coded in the workflow space. The last one is model. This is where your workflow is in place. You have the package. You have the workflow. You have multiple workflows you have defined. Those workflows will be reused in your model, where in model where it will say it knows which environment it's deployed, let this server. And you have to have your space such a way from your workflow, whatever you're externalized, they're all embedded in your model. So you have different models for the same. Let's say you have a mobile app you want to deploy. So typically we have two, three models, one for iPhone and Android. But the basic workflow is the same. It goes through the same approval process. It goes to QA. It goes to ops. It goes to release managers. So approval process, they're all embedded in your workflow. That doesn't change regardless of your model. That is just a general concept. These are some of the things we also looked at. And we embedded into our application because we want our product to be latest and greatest. And from our customers, we got some of these DevOps requirements, which we embedded into our DevOps as well. Some of the things are scalable. They want our DevOps space to be scalable. And it has to be distributed because we are working in a distributed agile teams. And you have ALM space where you use distributed version control, distributed requirement, gathering things like that. But why not distributed for DevOps? That's one thing. Because you may have several data centers, one in East Coast and one in West Coast and one in Europe. So we typically have, we ourselves have three, four data centers, Europe, East Coast and West Coast. So for us, it's very helpful having a distributed environment. And many customers want the solution for them. So what they requested, we need to have some automated engine. I mentioned earlier you can buy a product or build something on your own. We have a combination of both in our case. They want the space to be also adaptable, where they want role-based access. Once you have Dev and Ops tied together, you don't want a developer going into your DevOps tool, clicking on a workflow and trying to push it to production without your knowledge. So we need to have some security, role-based access. So things like that are adaptable. And then ALM integration. This is traceability, which I covered earlier, where from your continuous integration delivery, you want to have traceability. Who picked this request? And who approved it? Who deployed it? The last one is compliant, where I covered again approval process. There need to be a documentation of flow. Now, earlier you used to have pages of documents, 50 pages of documents per environment, per product. It is too tiresome for an Ops person. And Ops are also in a rotation. They are not in the same Ops person. It doesn't do the same job every day. They have day shift, night shift. Different guy comes. One guy does half work. Other guy comes and picks it up. So there are a lot of errors that happen. And it's too many failures in your production. Last one is snapshot, where whatever you're trying to build using DevOps, try to have a snapshot so that you can recover, you can roll back, wherever you left, your configuration, your data, or your version of your product, the next one. So this is a standard DevOps process. So earlier I showed you the workflow and model. So here, again, it's very similar, where you have planned code build test. Once you have done your code and build, you try to have configuration manager as a DevOps role, where it's not a real human, but it does all this work for you. And then you can deploy. In our case, if you see, it's already covered there. We have Client Server app. We have Mobile app. We have desktop app and cloud. So all these are built from one app. And you can push it through a release automation. And the key thing for DevOps is it has to be repeatable. It's not like you make it work today. And then it works for this version. When you go for the next story, next release, we have to make sure all those requirements are embedded all the way into your next release and stories. So what we try typically practice is when you have a user story, when we have a release, we have a roadmap, we have backlog. What we do is we also have a DevOps stories. And those stories have a lot of security things. It has security vulnerability requirements, things like that. And then DevOps also participate in some of the agile practices you follow, like sprint reviews. And they also take care of looking at our environment and documentation from a customer perspective. I'll show that in a little while. This is just overall DevOps. What are things that can be streamlined? So I just showed you how out of the packaging, workflow and model, how we are able to do three or four platform deployments using our product. Now, this is just about how you can make this successful. You need to have a DevOps culture within your organization. And it's not a title or anything where DevOps to think like ops, ops have to think like dev. And then you also have to streamline your SDLC, where you need to have standardized process. You have to get all your stakeholders together. They need to have an agreement. And you need to also take care of a lot of things right from coordination when you're trying to do this DevOps, because once you automate this, when you click off a button, anything again can happen. So there need to be a coordination effort from all the stakeholders, all the way from business, not just dev and ops. And third point is I already covered where you can try to do automation of workflows and tools. I cannot talk about tools and things, but you can stop by or look at our case studies later. And the last one is traceability. This is very key, where in alignment of metrics in traceability, I really love this because after having all this thing in place, you need to have, again, KPI, you need to have some metrics. So we track it in a multiple ways, where we have feedback from our customers. Then we also look at our downtime. What was the reason for downtime? And then we have a feedback loop, which I participate every week. We have a call with, we call it as triangular meeting. It's between operations, engineering, and our support organization. So we look at the issues that happen in production, and then we also look at root cause analysis, RCA, we do an RCA, and then we provide a feedback. So we incorporate whatever we can from that. Certain things cannot be incorporated because there may be like a disaster, hard drive crashed. But certain things we try to collect them, and then before the beginning of the next release, we try to incorporate them. So we call that as a triangular meeting. That's one thing we have been doing that last six months, which is release metrics between DevOps and business. And you can also, you have to establish modes of communication, because when things are in papers, now it's all in a flow. It's all kind of automated. You need to have modes of communication because ops have to be very much comfortable. Whatever they are deploying in their space, they are aware of what's going in. So that's another key aspect. We used to miss in the past. So in top of that, not only having the DevOps in space that helped us to some extent, but we had other problems where once you have this automation, every day QA comes in, they want to start testing our application. But what do you do? Now you have to have a box set up for all these kinds of environments, and then you need to have data loaded for all these kinds of environments. It's a tedious process. And then the way our QA works is every day, they come up, set up new boxes. They start from scratch because they have a new build that would have last continuous integration. They take that build and deploy. So I'm talking about boundaries. This is not really DevOps. So how we were able to solve this is we have our own product, which is a provisioning engine. So we have a profile set up in our product where by click of a button from a QA across the globe, our QA teams are in many places. They are in Germany, Argentina, Brisbane, California, and then in India. So whoever comes on board today morning, by click on a button, they get that box. And it also has a profile, which is very much mapped to production profile. So after a few hours, the guy coming in Argentina, he starts at 6 PM Indian time. So he creates another box. And he doesn't have to struggle. If there is any change, it's already in our profile set up. So this is one of the key success to make our things work faster, even though that's why I mentioned as boundaries of DevOps. Not only that, it also, you can pick option like, I want CentOS, I want SUSE, or I want Windows. And we also have option to pick 32 bit versus 64 bit, things like that. So it's a very big win for our QA to provision the box on the fly. And same thing for our dev as well. Developers, when they come on board, once they test, they want to test, they want their own play area sandbox. Click of a button in five minutes, they can provision a box from our own internal cloud. And the other automation that we did was once DevOps things are working, it's deployed to one of our customers. We deployed our customer only, they host with us. So in those cases, we deploy. And our ops have gone one level ahead. This is not real DevOps. But once you deploy, they also have a code to go and make sure whatever has been deployed. Let's say it's a year file. They have automated program. It goes and checks the year file, makes sure it's the latest version. And if there's any new configuration that has been added, go and make sure those configurations have on and off. All those, they have automated to some extent where they can play it and get the results back. Even though there's a manual thing we used to do, but on a large scale, it's very hard and tedious. If you try to, some of our customers have 80,000 users and our servers are deployed vast and they have several terabytes of data, those have to be backed up. So some of these things are really helping us. So once they test the deployed installed app in production, they have their own automated tool to run. That also varies by release because they have to tweak. There may be new features coming on board. So that's an exit criteria before we hand it over to our customer. And as I mentioned, we have our own internal cloud provisioning. We also sometimes some people take the box every day. We also have vapor cloud where it expires every week. So if somebody doesn't return their box next day, we have a vapor cloud. It automatically expires the box and puts it into the system. So here is our sum of our ops environment. I took a smaller example where we have SDLC, we have ALM product using which we manage our requirements, our defects all the way in a build, continuous integration and whole distributed team. They collaborate, they use our environment. You can pick any ALM tool, doesn't matter. So that's how they collaborate. And then we also have a data center. So I touched a little bit where in a first QA guy who comes on board, he provisions his box. And then in 15 minutes, he gets the whole environment set up and a new app is also deployed into his space. And similarly, the next day he comes on board in a different time zone. He gets the same build, same profile and the third guy. And then I also showed you about the developer who does the same thing. And we also have developers distributed across four different countries, geographic location. And they're up to 70 people who again get the same thing. And this really benefits and saves a lot of time. The last but least, we also have production operations where once we have a new release coming out, we put it in our outside firewall for some of our customers to go and play with. There they have a fine ops profile where it's kind of almost mimics the same as QA profile. And we provide an early release. And this is, again talks about Agile where before our real release, we get feedback from our customers. Again, incorporate it back into our system. These are some of the best practices I would have touched upon most of them where our ops have their own requirement stories. So they come for sprint reviews, it's operated and then they also do some of the admin doc qualification where we provide a documentation for our customers, how to install. So in our ALM space, we have workflow where ops have to approve before it goes to continuous integration delivery for some of the ops related stories. So that we have tweaked in our ALM workflow. The next one is ops also tests the way our customer installs. Some of the customers don't host in our system, they host it themselves. So whatever operations do is, they again take our product, install it, they try it on different databases using our documentation. These are some of the practices that helped us. And then last one is the flags. We have many flags to turn on or features based on what customer wants, based on the licenses that they bought. Those things are also tested automatically as well as manual. Ops have the way to test both automatic and manual. And then they have ops have an environment to build a snapshot. They can go back to snapshot, like three, four versions before, after that includes the data and configuration. And last one is very critical where they have spent their heavy investment ops where whoever touches the environment in between in DevOps space, they have like a very good audit homegrown build thing where there's only one guy finally who has authority to go and see all the traceability. Everybody has their own. Somebody can build only workflow. Somebody can build only model. And there's only one guy in the whole company who can go and look at the traceability. He's like a security officer and he can check. So there is very tight security involved. So we have spent a lot of time building that as well. So any questions so far? I know I'm going pretty fast. Have any of you incorporated these in your organization? Good question. So the way our backlog works, right? In Agile, when we start a release, we have a backlog. So we get requirement from customers. So we call it as CRUS, customer requested user stories. We get requirement from customer. So we create a separate backlog for that. And then we get requirement from our customer issues. We call it a CRIS customer requested issues. And then we want to get some new features into our product based on the market things like that. So we create our own new feature. And ops typically have their issues because there may be downtime because the application perform miserably and there may be a lot of issues. So they have their own requirements. And recently I got one of the ops requirements where they want us to run cross-site scripting because the cross-site scripting we are running today, it's not good enough. They want us to invest. So we shortlisted a product which is very expensive. It's like $30,000 a year. So because what happened is one of our customer, they're using best product which is like $30,000. We are using $5,000 product. And they came up with a list of things. Oh, we have hosted your product in our premises. We ran this tool and we are seeing all this vulnerability. How come you guys didn't see it? So those things feed into as a requirement. So we have ops backlog. And then we also have performance backlog. So beginning of a release, what we do is we take all these backlogs, everybody have owner for their backlogs and then we feed it to the product manager and he creates a priority. So in our ALM tool, you can rank them and you can prioritize them. So that's how it ranked and gets into a release. Then we pick between split. That's true. But see, I gave you a good example of the security vulnerability where your product will still work. Only thing is somebody can tweak through and publish, put their JavaScript on the form and submit. Yes. Yes. Yeah, I take both of your points where you're right. When we do the, we have security vulnerability story. We have actually nine stories for security vulnerability because different tools have to have some differences because some of them are using JavaScript, some of them are using jQuery and we have like eight stories. So some of the ops requirement has to be developed by dev with collaboration with ops. So before we go into continuous integration, we also promote dev box testing where once a developer completes this, he has a provision, he has his own cloud box. He puts it in there. QA goes and plays there. QA doesn't open a defect. So they play there. Before, officially QA starts testing on a day-to-day basis. We also promote, ask ops for their stories. Now the thing is, it's also culture. When you have ops backlog, give their own stories. They're also part of the system. They are collaborating well with us. Yes. That's correct. That's true. That's true. Right. Right. I give you an example. Another example is one of our customer wants us to upgrade our product currently in a qualified for Red Hat 6.2. One of the customer said, I find some vulnerability on the operating system itself. Why don't you upgrade to 6.3? So things like that. In those cases, there is no dev involvement where it's just an ops story and it just ops involvement where they upgrade the profile in the cloud and it automatically provisions for dev QA and everybody and then they test it. I'm sure there are white people inside. Yes. There are white people inside. Yes. There are white people outside. That's right. That's correct. Yeah, I gave you the rail upgrade. Red Hat upgrade is only ops stuff. Even though dev is aware, so that's in their backlog. And the security vulnerability on both sides. I'll give you another story we are working on currently. We are fixing some of the clustering. Our app can be clustered, failover and things like that or make it active, active. That's another story. We have developers and ops have to be working together because we have some caching in our system. We have some caching mechanism where when you go into distributed server, cache always stays. Currently, the cache stays in one server. We are working on a distributed cache where dev team is looking at using an open source. They are developing a distributed cache. In that case, dev works on those stories. They develop it and then ops comes into picture where when you have to configure those, how do you configure that common cache thing, whether it can be built. I talked about packaging. Then ops comes into picture there. And there may be cases where you may have to put those jobs or whatever the distributed cache open source product may be into your class path. Apps comes into picture there too. They keep modifying our profile. So automatically it reflects into our dev profile. Dev gets QA, gets all in next day. They get it once they update the profile before provisioning the box. And even yesterday we had another issue where somebody wants to add beyond JDK 1.7. They want to add one more thing. It came from one of the customer. They want to add on. So last night I saw an email through our system that all developers, there's an email going around. All developers QA, when you provision new box tomorrow, you get to use new profile. Your JDK has been upgraded. So things like that happens. So you had another question, right? I covered. They are in four countries, as I mentioned. Seven or eight sprints teams. They are in four countries. And ops are only in two locations. So ops themselves are distributed. So when we have a separate backlog for the ops, then we bring them only those time because they don't have time at all. They work three shifts out of five guys. So it's kind of hard. So I understand. It will be top of the list. And we don't bombard them with many stories within a release. So I think you all made a good point because it varies by situation. Again, it varies by scenarios, things like that. Let's look at one of the case studies. So far, whatever I showed was how we are doing it within our organization. We have seen how we are doing it in cloud and who we are building it. So one of our customers, they use our product and then they build on top of it. So that's how our products are. Our product is. So they take our product, they build on top of it. So from our product, they get deliverable out of our product and they want to continue doing DevOps. So we helped many of our customers. And I'll just give you a case study. I cannot talk about the customer or the product, but how we solve them. We can read our case study. So this is a utility company in US where they had a lot of regulation. They have to get things done and they have some time frame. And there were too many people involved, as I mentioned earlier, like same operation person will not be doing the job every day. They have shipped things like that. They have no tolerance for errors, things like that. So we have done a fully automated DevOps space with workflow, with your models that I explained to you earlier with different packaging. They have a web app as well as clients over app. And then it has reduced their defects and downtime in production a lot. Like there are statistics in the industry you can Google and check, but people have typically said 20%, 30% of defects, downtime, SLAs have improved. So in our case, it's I think six time reduction deployment time. And the staff have been gone down to one. So this is one of the simplest case I took for this presentation. We have more complex. I take a simple one. We have a customer who has a Windows app. And I left other ones like app. They also have a SAP and Linux, three different apps. But I just took .NET. It runs on Windows. It has MySQL. And today, they were before implementing DevOps. They had five servers and five QA and then five production. So it typically takes 30 minutes for deploying by an ops person to do each one of this. And if you look at a simple workflow diagram below, you can see through there are a lot of conditions where you have to decide whether you have to go continue deploying or you don't have to deploy. So in this customer case, if there's any failure happens, we also automated the rollback where all the data that's been deployed, it goes back to the previous stage. Your build goes to the previous stage. There's automatic rollback as well. You can also program. That's the extreme case. You want to typically stop and check as some manual intervention, but they want automatic rollback. That's also implemented here. The workflow is a little bit complex. So after DevOps, what happened? They had their own homegrown lot of scripts and shell scripts and Python and so many combinations. So it's all consolidated into one simple workflow. That's what we did. And there's a centralized automation engine. And then now, after your continuous integration delivery, once you go through all the approval process, approved by QA, then the workflow goes to release manager, then production. So after it goes to release manager, click off a button, five minutes, they get all the five production servers deployed. That includes dynamically configuring on different five servers as different workflow. The model is same, but the workflow is different, sorry. Workflow is same, model is different, sorry. So it takes only five minutes for them to deploy. So this is a good case study. You can come to our booth, you can grab this case study as well. So the final summary, do you guys have any other questions? You know, eight years ago, right? I have small, I think eight years ago, we never had all these things. So I used to work for insurance company in Philadelphia, where we used to have so many servers and you know, like 40 servers. And then we didn't have all these things. What I used to tell is, if there's any failure happens in production, it's because of the document given by Dev. So the Dev person has to take the whole team out for lunch. So it's $20, 70 people in a team. It's like very expensive. So same thing with ops. Dev gave right requirement documents and op mistyped some configuration and fails in production. We do the same reverse where we ask our ops guys to take everybody in the team for lunch. So it's 70 by $20, you can imagine. So that's how we were able to control eight years ago. Now, you know, you have all these. So in a summary, right? Why we are, why need a Dev ops? So what is the thing we are looking at is productivity and agility. So Dev wants agility at the same time, you know, we also have productivity in place because we are automating a lot of things and we are also having control. And there are tools available where, you know, once you automate your workflow, you can also create a document out of it. Those are, again, extreme cases. Some customers follow that as well, creating a document out of your workflows or models. And the next one is compliance and governance, which I covered a little bit, where in your workflow, you can have security, you can have, you can have end to end tracking and things like that. And the last one is efficiency and cost. You can save cost because of downtime. You reduce defects in your production environment. And then other thing is, you know, technology is not the only thing that can solve. You have to choose the right one that fits for you. Technology is not the only solution. Last one is, you know, best engineering practices. Again, you know, a lot of engineering practices available, which has to be incorporated into your Dev. This is evolving space. Four months ago, when I did the same presentation, there were only half of these slides, which was totally different. And this conference itself changed a lot. So, best engineering practices have to be adopted. So, any other questions? Yes. I couldn't hear you. Yes. Good, I can give you another recent case. One of our customer who's operating out of, you know, 20 countries, they use our product. And our product, right, you can load up stories. So, our product has a tool where you can create stories. So, one of our customer had 100 projects. They are going, running 100 projects in parallel. They have 80,000 users. And each project had like 1,000 stories. They kept all the old stories and new stories, 1,000 stories. So, what we used to do in the past is, in our DevOps space, as you have seen in the staging, we used to take one of our customer data. And then, you know, we load it up in our staging and we used to test. That's the last phase of our DevOps testing. We used to do that. And last, more than a year and a half ago, we stopped that practice because we want to get more tight into, we don't want to expose our customer data even within our company. So, we got rid of those practices. And in this situation where, you know, we built our own data, we have automation tool. So, it goes and runs, click off a button, somebody recorded it. Keep on creating, you know, story, A, B, C, D and A1, A2, A3, it goes on. It creates hundreds. So, we never had 100 into 1000 stories in our automation. So, it worked fine here and customer went there. Customer kept all the stories for last four years. So, when you go to the, our application, click on it. It took forever and you know, you have web timeout, right? It times out in 20, some companies set it up for three minutes, some have set it up for 20 minutes. So, it timed out. We have those situations. So, that's where our triangular meeting comes in place, where we have engineering ops support. So, it's, we do a weekly thing. We do look at an RCA. So, last week I attended a meeting where I said, you know, we need to have automation to mimic this and have environment always available, okay? So, performance testing and, you know, it is a live like environment where we do this performance testing. That's how we are making DevOps people, you know, people job much easier. Okay. We have the same. We picked one of the largest customer. Again, give you the same example, 80,000, 20 different countries. We took that customer hardware and their data and then we have that always available in our performance lab. So, performance lab, after our continuous integration, right? We have our own QA environments where we all do this and we have CI, CD and then a performance lab also runs their testing. It's a daily, like when somebody commits, they parallely run their testing with that environment. So, we'll never know. We have to keep on contacting the customer because within three months, our customer acquires some other company and then they merge systems and they pull their code and stuff. And then, you know, within, without our knowledge, you know, it explodes. So, our test data goes out of sync in like, you know, three months. This is one of the example where one of our customer, I think he acquired other company and they merged, they had their own product and they merged and moved all the stories here. We explored, we never knew about this. So, some cases we have issues, yeah. How often we release our product? Our product has two relief cycles, two release cycles a year. So, our major releases are in June and just middle of before Christmas, December. And we also have a hot fixes, hot fixes where customer is really dead in a water. So, that goes on a parallel track. And we also have patches that goes in a parallel track. That can happen anytime. Patches are like three months. Your staging environment. Yes. Does it happen at every sprint or only pre-release? Our staging environment happens pre-release, but I talked about PSR, we call it as PSR. It's a performance lab. They have an environment that happens daily. So, that's where we have the whole staging is just a pre-production and performance is the one happens daily. And we all get an automated email. How is the performance? Last night we had an email saying degraded by 30 percent. So, okay. Any other questions? Mostly collaboration. There's always that tight thing. Now we have given them ownership. They have their own backlog. That has solved some of our issues, where DevOps, they don't want any change. That is our biggest challenge. So, you know. You mean the office won't change? They don't want us to change their environment because we keep on. Our product is based out of all open source products underneath our whole product. We have up to 25 to 30 open source products underneath. Any one of them are upgraded. Let's say that it's a struts or ticker or anything we upgrade, review board. Those are all open source products we upgrade. Then we may have some changes in our operation, but there's always a pushback because they are scared to move that in. That's why we're trying to get them in into the sprint and review. Whenever we think in our artifacts, also we have some flags where we customize some of our artifacts where in a story level we have a flag. Does this require an automation? Does this require a doc chain? Does it require a lease? So, we also have a note for ops as well so that they can be part of the sprint review. Thank you. Did I answer your question or no? I went round and round. Partly, yes. Partly. You can talk to me later. Sure. Thank you. I think I'm almost done. Any other questions? I think I got a cue that we ran out of time. So, if you can come to our booth and sign up, we are giving away two iPads. So, it will be announced tomorrow evening. That's all I think I'm done. Any questions? Thank you, guys.