 good evening everyone thanks a lot for coming to the presentation for me this is the first time at scale and i've been having a blast here the organizers as well as the community has made this event such a huge event and a successful one thanks again for coming all right as you can tell from my accent and just by looking at me i'm actually from sweden just just kidding i'm actually from india and let me take you back to a time when i uh a long time back actually when i did my undergrad in india i was um i i used to remember i still remember actually that i used to love going to the computer labs that was my passion but unfortunately in my university for my class we were allowed to go to the lab only on tuesdays between four p.m and six p.m things a lot better now but back then it was in the case and i had to find a way to make my way into the lab at that time and get a computer and do whatever i want to do and then come out it was very constraining but at the same time i kind of learned how to make it work in that environment then i moved on to u.s where i wanted to do my masters and here the first thing i did is uh i at the registration desk i asked about okay what is the lab schedule the registration day was coincidentally on a tuesday and they said um okay here here you go here the keys to the lab and i was like thinking i'm shocked really so no not only do i have to come on tuesdays between four and six i have to come here get the keys every time and then return it after i used the lab like that's ridiculous that's worse than the public restrooms and gas stations this is how i was thinking and then the lady clarified to me and said no no no you get to keep the keys you can use the lab anytime you want this is just like your facility was like wow this is awesome this is dope y'all peace actually i said it more like uh thank you madam really appreciate it that's but but but you get the point you get the point um so i was i collected the keys i was super happy i was uh doing my happy walk to the class and i was thinking in india the safest place to sit in a classroom is the last row next to the back wall so that it's it's like you don't see the prof a professor and the professor doesn't see you it's a good setup that way it's like the lines marking their territories if you will um where you don't ask the professor any question professor doesn't bother you you just get your degree done get the heck out all done all right but i i came i stepped into the us classroom and i was looking for the last room and it happened to be the highest row in the entire room the room's like kind of elevated and i was like whoa what's going on here this is like a zoo animal a lion from the zoo getting exposed to the forest animals now so i was like i remember thinking the very first day i'm screwed i'm screwed so basically i had to i had figured out a way to learn about learning in india and when i came to the us i had to unlearn everything i had learned about learning and then relearn the ways to learn here given that i used the word learn like 50 times in the last 10 seconds you guys probably figured out i'm going to talk something about lessons learned yes my name is victor gajendra and i'm from the ticketmaster team and i'm here to talk about lessons learned by doing devops at scale at scale all right that was complete coincidence anyways so before as soon as i mentioned ticketmaster to people there's a popular question that comes up let me throw it out there clear up the air i'm sorry i cannot get you free tickets terribly sorry about that all right so now that we got that out of the way let's talk about what i'm going to talk about in this in this presentation obviously i'm going to set the stage give you some idea about about give you some idea about i don't know why this lights automatically moving i give you some idea about why why ticketmaster is special what's our environment where we come from and all that and then i'm going to take you through our devops performance and then finally i'll also take you backstage to show some lessons learned things that we picked up along the way all right let me pause at this moment and tell about tell a little bit about what this talk is not about actually let me show a little bit of code i'm showing the code just to show that i'm not going to show the code in the presentation it's because trust me i've looked for programmatic ways to solving devops in your enterprise scale but there isn't any so if you're looking for starting a great open source project to address a market segment that has not been fulfilled yet untapped opportunity go for this programmatic solution for devops for enterprise and this is a project that will not get you anywhere sorry give me one second slides moving automatically let me fix this all right should be back now okay so um so let's get going so who are we we are ticketmaster as as soon as you think of the name you think of tickets and uh these are people that sell tickets right but what we actually do is we connect fans to great experiences that we kind of bring them together and that's the specialty and we basically what it takes is a deeper understanding of what the fans needs are and what their preferences are profiles are and they're matching them to the right opportunity so uh let's let's take the experience like even 10 years back like how was the ticketing ticket buying experience uh back then so some friend in your neighborhood probably told you hey there's a big show coming up and then you wake up that morning tickets box office opens around 10 o'clock so you go stand in the line as you as you walk into that place where they have all these counters your mind makes splits like a decision just to okay there are 50 people in each line that this line moves at this rate so i'm going to get my ticket faster if i go to line number three so your mind makes these complicated decisions very quickly and then you pick a line using probability expect expectation that you're going to get the ticket faster than the other guy right as you are in this line um and the line progresses slowly you start to wonder okay am i going to get the ticket when i reach the friend of the line you're anxious you're just waiting but the guy in front of you as it as he reaches as you reach closer to the counter guy in front of you picks up the phone and start talking on his cell phone and you wonder what just get out already get out of my way right and then he goes to the counter he buys his ticket at at some point he picks up his big bag and then pulls out his checkbook and starts asking who should i address this check to and then can you spell this for me oh it's like come on just get out already please don't judge me i think all of you have been in that spot where you have somebody in front of the line just blocking the traffic so just just compare that experience to what it is right now with ticketmaster now you don't even have to worry about which shows to go to in fact ticketmaster makes those recommendations for you and and then when you could buy it at the convenience of your home and then it is also verified tickets meaning if you buy a ticket from ticketmaster it's guaranteed that you will enter the venue and once you're in the venue we also sell technology solutions to enable you to order in venue solutions solutions like you could order food and beverages from your seat right to be delivered to your seat so things like that so we have a wide range of technology and products that we offer and and that's what makes our company unique now we've been doing this for over 40 years so you could think of us as not just ticketmaster but it's more fair name i would call as the experience masters all right so our technology view so we have a wide range of technologies this is a company that's been around for 40 years like i said so we even have wax computers actually not the physical ones looks like they don't sell it anymore but we have wax emulators running on lnx machines on which we run some a small piece of our software and that's on one end of the spectrum on the other side we have modern containers docker and we use configuration management chef and we use kubernetes we've started looking into that so there's a wide range of technologies we have pretty much everything in the middle as well so you got a note one thing ticketmaster has been the leader in the ticketing industry primarily because all this technological innovations have driven its business innovations all right so we do have we are a big fans of technology and there is a lot of investment that goes into building great technical products at ticketmaster we have about 20 000 plus virtual machines and we have a whole sale warehouse type massive data center that hosts all these virtual machines and to to so one might wonder like how do we how do we operate these this type of a massive infrastructure at scale without automation so obviously we did build our own automation this is back in 2002 before chef and all existed our engineers got together and they built an internal configuration management solution that is that is still being used in some cases in the company some of our engineers went to the chef training a couple days back and i saw a tweet on our internal slack channel a ticketmaster's internal slack channel that hey the way chef does configuration is very similar to our internal tool and that that that speaks a little bit to the type of the engineers we had back then we also had another set of engineers that tuned to the kernel of the operating system primarily linux now you you might ask why why in the right mind would somebody go and tune an open source code well it breaks the it folks you out so you won't get any support and you can't like and because this is a special use case you can't give it back to the community and get it supported henceforth but there is a unique reason for there's a specific reason for why this happened and that has to do something with our tick traffic pattern ticketmaster the the the the website ticketmaster.com it has a heavy demand at times and we go from zero to 40 million transactions in less than minutes this is a very unique property to ticketmaster because there are lots of people out the companies out there that have much broader load on on average but at the given time this type of spiky traffic pattern is kind of unique to us if you were in the auto industry we'd be like beating gaudy veron which is like zero to 16 to 0.5 seconds you'd be faster than any of that right sometimes our traffic pattern so spiky that we thought we are being d dosed by by some hackers from some university right but then it turned out to be our customer traffic so there are two factors that explain why we have such a traffic pattern well obviously the number one is we have 54 million unique monthly visitors to coming to our website and these are fans the loyal fans that they keep coming back to us for buying tickets and then we have something called on sale events what's an on sale event a little bit of a refresher on the technology sorry ticketing industry the way the ticket ticket sales works as an artist announces a show and the inventory is made available it's made available typically at 10 o'clock eastern time and then it is made available to that that people in that region and then they open it up at 10 o'clock central time for that that group of people and then mountain time and pacific time at 10 so from the west coast perspective you you start seeing peak loads at 7 a.m. our time 8 9 and 10 a.m. so that's why you see huge traffic patterns so basically think of it this way on sale events are events where the tickets go on sale for a broad broader group of population that usually the venues are smaller I mean relatively smaller for the demand that that they some of the artists see so a lot of people competing for a small set of tickets so that's what causes this kind of demand spike and and crazy all at the same time kind of a influx traffic so and and also all most of these key on sale events have happened on fridays in other words every friday is pretty much like a bank thanksgiving black friday for us where you just have to don't have to fight over a $50 tv but you can do it at the convenience of your couch but at the same time from our perspective it's like lots of people coming to us to buy our inventory all at the same time and that results in a traffic pattern that looks something like this this is an op-net graph from one of our real on sale events that happened recently a couple of weeks back and as you can see here at 7 a.m. up until 7 6 58 or so there's not much traffic but 7 a.m. you see a massive spike in traffic it rises up like it's nobody's business and then comes down a little more gradually as we start serving so this is this is one of the reasons why our engineers in the past had to go down to various to low levels of OSI and start tuning so that it's this faster processing of the incoming traffic tickets all right so at ticketmaster we have around 150 products these are products of different kinds and and there's like 400 plus api or so these groupings and this is pretty much the tech stack i couldn't include everything it's already crowded hopefully people in the back and see all the letters so let me let me start reading one by one actually i'm just kidding we're not going to go through this one by one the goal of this slide is just to show that hey we we have a lot of technology it's a complicated environment and we are primarily an open source tech shop our we encourage our universe i mean our staff to use open source technologies primarily and we believe that a corporate can cannot solve the problem like the broader community can solve and and also keep pace with it our resources are limited so obviously we use the open source technologies first we try them out see if it works and then we also encourage our engineers to contribute back in fact uh that we have engineers on our team who participated in developing chef and there's an engineer from the java spring team and so forth hopefully this gives you an idea about what ticketmaster is what are some of the technical um environment care attributes all right now that the stage is set let's let's get into the actual show so the the DevOps performance that i wanted to walk you through is actually a journey that ticketmaster took it's i'm going to describe it as starting at the inception point a and then i'm going to walk you through a few pivots bc and d that kind of drastically changed how we progress along in this DevOps journey it's a journey of seeking true north bc and d are called pivots pivots is actually a lean startup term we wouldn't get into too much detail on the lean startup principles but i'm happy to take questions pivot is basically a structured course correction designed to test business hypothesis or a business assumption so that this journey is not drawn to scale it's it's actually just to show how we progress along in our DevOps maturity model and and the point a is roughly 10 years back and i'll be bringing you back to where we stand today all right so the journey started in a it was pretty flat and then we went through it it's soap opera style uh johnny there's lots of ups and downs and happiness and sadness along the path and then we went to the peak of confusion we thought things worked and then turned out it didn't work for very well and then obviously i'm from bollywood so i'll definitely end with happy smile and dance all right so so in this presentation you obviously kind of figured out that it's not about the journey or how it started sad and ended happy it's more about what we picked along the way that is going to help you take it to your environment and then do something with it so hopefully i could share something meaningful with you today okay so let's break this journey down into four components uh four phases uh so first one is pre-developed and then i have given names for that as well and we go from red to green and getting back to good right okay so now the like i said journey the point a started roughly ten years back and as i described we have a chef like internal configuration management solution we have kernel tuning we have smart engineers that could really solve any complicated problem well we we started building all these internal pools and kind of started falling in love with what we built rather than the problem we were trying to solve the product that we created happened to be our babies and we started caring for them more so we we started stagnating meaning we fell victim to our own success like in many industries even in technology field if you are not rising you're falling right so we started not investing in modernizing our technology and therefore we started falling and and that explains the direction of the of the era that you see here and then pre-develops era and there are several attributes this particular to this particular phase i'm sure you most of you know this so let me quickly script to them like any dysfunctional organization that does not build products collaboratively we also had dev and ops that was building a tall wall of confusion in between dev and ops right and the developers would finish their product they would throw it across over to the ops with some labeling on it and some runbook and then the ops guys would look at it and go that doesn't make any sense let me throw it back to them because the runbook doesn't have the proper punctuation marks right so they they they send it back to the devs and the devs and look at it and then they keep point finger pointing at each other this results in a lot of wasted cycles and lots of rework and therefore eventually you don't end up producing quality products quickly at a reasonable cost so this was one then we had tech debt i mean it's a decades old technology company we've all kinds of versions of technology so we obviously have tech debt like any other debt it either grows or it falls right so if you don't invest actively in cleaning up your tech debt then it's going to build up and it's going to exponentially impact your productivity down the line that's what happened to us as well obviously we had long cycle times meaning the time from court commit or even the requirements definition to all the way to a customer seeing that feature in production was was getting longer and longer so if an environment is characterized by these three things then obviously what's going to happen is we're going to get our butts kicked by somebody right miraculously for ticket master that's exactly what happened we did get our butts kicked but this time it is by a bunch of nimble agile small startups so these companies started coming our way and then they started eating our market share away from us and ticket master couldn't compete with the with the pace at which the features were getting released by these small companies and and that we were starting to get concerned about that from a business perspective and that's when we reached the point b other pivot b so this in this deep bit of sorrow we had to make a decision it's a sinker swim moment right so you either change your ways of operating and get software out there quickly that meets customer needs or you kind of lose your continue to lose your market share ticket master has been the leader in the industry so that that's something very tough to swallow obviously okay so we we attempted devops and a few examples of our first attempt are shown here obviously we all know that multiple silos is a bad thing great so we said let's go and build a cross-functional team where the developers and the testers and the operations folks and they just get together and build they have a common goal and all they need to do is quickly get the product out to production as well as keep it running it in production just it just works it's expected to work well that's great and the next one is we we I talked about cycle times one of the deeper problems of cycle times was that we were not taking any feedback from our internal customers our development teams it's not like we did not have any bill pipeline or automation that will reduce cycle time it's just that the automation that we put in place did not meet the development or the development or the delivery team needs so they they when they came to us we'd be like okay we already know what you guys want because we've done this so many times in previous companies and whatnot so we know exactly what you know and I'm gonna build it obviously they did not adopt the solution and it continued to have poor cycle time okay and some of the folks even quoted Steve Johnson as the customers don't really understand what they want okay so as I present these things I I'm I also want to caution you that these solutions even though it worked for us may not work exactly for you so you might want to take these these findings and and then understand your environment and make it your own so there's gotta be some customization required here all right so we made this feedback problem disappear and we started listening to the development and the delivery teams and and we got their perspectives in so this way we were able to drive adoption of the bill pipeline automation that we had put in place and we started to reduce cycle time well the results are shown in the developed journey tracker that I have here we started making good progress in cycle time and and even the customer's nps course this is what I would call the functional domain meaning we are able to operate in a sustainable manner and we were able to deliver products as planned well almost as planned but kind of functional a lot of companies are stuck in this place this is like if it's okay but it's not it's not quite good but then on b-dash or the point that I've marked in the middle something started to happen our performance started to degrade we were not that productive anymore and we were confused as to why is that why is that happening so we dug a lot deeper we conducted some root cause analysis and figured that all right things that we thought were good solutions turned out to be bad like how is that like well we talked about cross functional teams right if you know ticketmaster it's a very siloed organization so we have a massive teams that are broken down into specializations meaning we used to have let me let me list all the teams in one breath see if that's possible so we used to have dc ops team data center team cloud team infrastructure development team a platform team a development team qa team business analyst team and so on so forth right so we have all these teams and they are specialized in their areas and they don't know about the technical difficulties of the other domains so now you have to form a cross functional team how do you go about doing that so we picked one from each team and then said okay you guys work together you're the cross functional team and then for redundancy we're gonna some guy get sick or takes a vacation or whatever we just want to double the number just for high availability right and the team size became massive jeff vestibus point was like two pizza size team ours was more like 10 pizza size team and it's also extra large pizzas so so this wasn't going to work the team was so big that the team members would not know each of this name it took a couple of weeks to just to get familiar with the people's names the funny thing is the card wall was for so long we used to call it the great wall of cards um so anyways it was chaotic it was it was dysfunctional it started to get dysfunctional so we had to break the teams down into um into into like smaller teams so we experimented with a couple of models we tried scrum of scrums model and we also tried independent parallel teams and we kind of settled for something in the middle wherein uh we we had the teams broken up into infrastructure as service and then that'll serve the next team which is platform as a service so they add their component and then it it goes on to the product delivery teams and between these teams there were unwritten contracts that okay i promised to deliver this feature to you or i i'll give you this code base with this functionality so you can build against this interface so we we looked at the microservices architecture and the best design patterns out there to enable um to learn from some of those things for example we have tolerant reader we have consumer driven contracts and those type of design patterns and microservices domain which is adopted the same thing for team designs so there's a sort of course corrected and it started to get a little better then the next one is listening to perspective like what's wrong with listening to different perspectives right uh everybody should do that but we started listening so much that we kind of became devops psychologists if you will so we started listening too much and if we wanted to get everybody's perspectives we want to solve the problem for everybody and we soon we figured out that a single solution is not going to meet every need people are going to have different needs so we said okay let's launch an initiative to really understand our customers better you cannot have 10 different customers and all of them are exactly equally important to you it's just not realistic a deeper understanding of the customer segments and what their needs are helps you understand and prioritize how important each customer is and then walk walk the list down within each customer again there are different requirements with different priorities things that solve bigger problems for the customer and things that are they just like to have so we had to break it down so that we produce incremental product feature releases that will really help and and have a better deeper customer understanding helped us with this all right so at this point we reached the point C of the pivot C pivot C is characterized by I would say we are devopsy in the sense that yeah most of the companies try this by reading quick books like devops for dummies or get devops in 72 hours so we bought all those books read all those quick tricks we leveraged from benefits of all the low-hanging fruits out there we've gotten cross-functional teams we got we started using CIs basic things right we saw results but at the same time we were kind of not feeling well at the core we were feeling like how's this going to turn around a 40 year old company how's this going to make it nimble such that we can deliver products quickly like a startup would right so this this concern started to raise we knew that this there's got to be more change required for us to launch into this contributing phase what is that how's that different from functional phase it's a contributing phase is somewhere where you you you not only operate and deliver the products as per plan or within budget but you also understand and think from customer's value perspective how much value you're adding to the customer and therefore you're delivering it so this type of value oriented thinking and needed a much deeper broader change okay so change is required but how much change is the question right well it is new year's time we all make resolutions so let's talk about a quick resolution apparently in a recent survey they say that health and fitness are top of the list that's the category where most of us make our resolutions followed by a career growth obviously with this picture up there i'm not going to talk about career growth right so i'm going to talk about health and fitness let's say we all we all know that eating an apple a day is is going to give us great health it's going to keep the doctor or whatever but is it going to get a physical transformation like you expect is if that's what you're looking for i was talking to my friend he said okay if i let go of my other six pack every friday will i get this six pack well not quite not quite you have to change your eating habits your sleeping habits you have to exercise regularly you have to start counting calories and macros and whatnot the entire lifestyle has to be changed it has to be a much deeper and broader change all right so the point here is minor tweaks don't produce major results they really don't if you want something big then you have to go big all right so we want we said that we wanted a more cultural organizational change and that's when we discovered the culture of promises okay what is a culture of promise let's say you have a company vision i'll break it down first and then i'll give you an example let's say there is a company every company has a vision so you break that vision down into strategic promises and each promise then gets further broken down into functional promise promises and then when it goes to the team it gets broken down into tactical promises let's look at an example um let's say our our our vision is to connect a lot of fans to great experiences and and if if the strategic promise that maps to that is something like okay i promise to make ticketmaster.com the favorite ticketing site for the fans you could break that down into smaller promises like to provide i promise to provide the best online and mobile experience user experience for our customers which could further be broken down into tactical ones like okay let's i promise to develop interactive site maps basically it's a feature that allows you to view the the auditorium or the stage from each seat so it's a feature and and it could be other operational promises like i let i promise to build the site that always works just simply works high up time all right so so now you see that we have taken this one top level vision and a kind of broken down that into multiple hierarchical smaller more actionable promises what this does is there is several benefits to this first one it aligns the entire organization towards with a common goal at the top so if the CEO and the senior leadership team decides to change the direction in the next quarter or next year then it's easier to make change at the top and then it kind of trickles down to the bottom everybody is now thinking about value and customer more importantly than they are thinking about their products or the code or their process everybody's metrics driven in this in this framework and more importantly the you you've seen this in studies the best way to improve team morale is to make people feel like they're part of a bigger problem right so this helps the team members feel that way everybody even though the person who's setting up a virtual machine is just responding to the tickets for virtual machines on his queue even he now is able to think of the virtual machine as okay this is going to help me deliver the interactive seat map feature which is going to tie to the best online experience which is going to tie to the making ticketmaster.com the favorite site so the promise is kind of rolled up every task is now more meaningful all right so we tried this and then we got excellent results definitely excellent results or turnaround times were faster we not only were able to deliver the products as promised but we also understood whether it really helped the customers or not so business business value delivery was happening on a continuous basis and then we reached pointy the pointy is not necessarily a pivot it's it's just an accelerator that brought us into the next phase is transformational meaning in this phase we were so tuned to thinking with customers in our center so that and not with our product or code or process as the center of our ecosystem that we are able to quickly pivot if we need to based on company strategic decision on time yet yet deliver to the diverse custom customer needs we were able to own the customer's problem and and always be keen on solving customers problem rather than the products that we built the term product manager is a misnomer as some of us know if if you call somebody a product owner then after a while they start loving their product and what they created right but if you call them a problem owner of the customer and the goal is to solve the problem for the customer then that the focus is different now they are putting the customer in the center of the equation and and they produce whatever products whatever it may be version one two three whatever to solve the problem for the customer so this is a this is a mind ship so because our organization was already set up for this structure by this time because we have been practicing the culture of promises for a while we started seeing some cool transformations that that were organic for example let me give you a few examples so we made a promise to say okay let's free developers from the constraints of on-prem infrastructure well they have and they have a need they submit a ticket they they wait for X number of days depending on the SLA and and and they get a server but at that time they don't have it properly connected no they have to restart the whole thing it it causes too much delays so we don't want to constraint the developers so that the team now comes up with this idea of okay why bother with this massive infrastructure and how the constraints here why not just go to AWS or web services right so that's exactly what the recommendation that came out of the team it is groundbreaking for team master because like I mentioned we have a massive infrastructure and we have a big team that runs this and for them to like accept the fact that okay we're going to go to the cloud because it delivers better value to the customer that is significant that that is a that is a huge shift the next promise I wanted to find out is reducing cycle time of the software releases so we said we'll believe we'll build a continuous delivery pipeline now as you know continuous delivery pipeline has several stages you check in the code and then it runs the unit tests and then integration tests and then spins up an environment using scripted automation and then it runs the user acceptance tests and then it takes on to further performance testing or security scanning or whatever and the code ends up in production all in a fully automated fashion but it's the same series of steps that takes your code through all these steps right now we built a framework actually we are in the process of building a building a framework that lets people plug in their own custom build a script in there and their own unit test frameworks in there but we still have that high level unified build pipeline or the continuous delivery pipeline for the entire set of 150 products now what does this offer now you have a common scale using which you could drive tech maturity or how's that now you could say let's say all the teams are using the same build pipeline continuous delivery pipeline sorry now team a might have 95 person coverage and the unit test step well you could measure that because it's the same standard pipeline right but team B has only 45 person and you could do the measurement and use metrics for the various stages and throw up a dashboard that says okay team A has higher level of maturity in adopting or practicing continuous delivery because their coverage is better they have more automation their environment creation is scripted and the cycle time is highest and so forth so you could use that as a model to motivate the rest of the teams and drive everybody towards a higher level of tech maturity the next one is we promise to eliminate environmental inconsistencies obviously docker came up so we wanted to contain containerize our applications so besides the obvious benefit okay eliminates environment consistent inconsistencies we have we have this other unique benefit that that comes out of this remember we talked about migration migrating our applications to AWS the team not only proposed that they they came up with this idea of creating a cloud enablement team that will go ahead and build an on sorry a self-service AWS onboarding kit that includes things like cloud formation templates the VPC layout and the connectivity all established and proper recommendations for technology choices within AWS product let's say it's a package that's easily consumable for the product teams now each one of the teams can just go through that and get their products up in the cloud so this enables us to move faster to AWS on top of that if we dockerize the applications on the on-prem infrastructure while it's running in our private cloud then it's much more easier to migrate to the cloud so docker was again a beautiful transformation that we recently figured out so anyways so this is pretty this pretty much sums up the our journey in this direction and as you can see we have come from all the way being pre-developed in the pre-developed domain to now in a transformational phase where we are willing just to see past the technologies that we'll use and look at what's more important at the end of the day which is customers and customer value so that's it that's that's pretty much what our journey is if it were a Bollywood movie this is the place where they roll the credits and sing the happy song and dance but I'm going to summarize the key principles for you to take take with you the two things mainly first one is automation drives throughput right so you don't want manual operations in your day-to-day operation things that repeat most often obviously you want them automated because you get the best bang for the buck so you drive for more automation as much as possible and then automation if you produce crap then your automation just produces more crap so you need the feedback loop right the feedback improves the quality of your throughput now there are a couple of examples different examples I want to give the first one is seeking feedback improves empathy for the dev and the ops team to understand each other's perspectives if dev teams understood how difficult it is to maintain and operate a software after it goes to production if they understood then had that empathy then they would definitely build applications that are HA easier to operate easier to backup and whatnot easier to monitor right so asking for feedback improves empathy the other one is we should always seek to use operations as an input in to close the loop into the strategy for example product managers when they create a feature and make a business case for building a feature or a sub product or even a big product they make a lot of assumptions they say okay if I build this feature the customers will come to this website and click this button so many times and therefore expect so many dollars to come out of this every quarter so they make they make these assumptions in the beginning of the project and then we go ahead get fundings approved and then we build the product product goes to production but very few companies actually take actual measurements are valid that will validate the assumption that are made in the ROI phase and feed them back to the product management product management pretty much blind that they they rely on quarterly finance numbers to actually find out much later on whether their assumptions that they made in the first place were right or not so operations teams can help enable that to close the loop okay how many of you know this guy is alvin toffler he's one of the famous american writer and he's also a futuristic thinker he's written several things about like revolution in terms of technology and communications he's the guy that said change is not a mere necessity of life he said change is life but the thing that caught my attention was this one he says that the illiterate of the 21st century will not be those who cannot read and write but it is those who cannot learn and learn and relearn what it is is this is an age where the smartest guy or the loudest guy in the room is not going to win it is those guys that can quickly learn and adapt to the changes that they see that's going to win to sum it up i just want to say that when you go back please encourage and try to create a culture of learning where every team kind of feels like it's safe to make some honest new mistakes as they try to push the boundaries of technology because when the environment is not safe or it is not suitable for learning what happens is then engineers start trying repeating the same thing that they did yesterday because it's safer and you don't get much learning in that process i hope i answered some of your questions thank you and i'd be happy to take more questions go ahead yeah it's a great question let me repeat it for the recording the question is so you you showed a wide range of technologies in the tech stack and as you do automation are you trying to work towards converging that tech set obviously yes because we do find that it is very wasteful to have repeat like duplicate technologies for example you don't want chef and puppet both running in the environment right we have an internal config management tool like i mentioned and now we are starting to adopt chef so one is going to replace the other obviously we're going to use open source tools so um so that's the direction we're going but it takes a lot of commitment and because we have 150 products or so it takes a while to get rid of legacy technologies and replace them with modern ones but that's the direction yeah these these cross-functional teams usually are self-organizing and they usually drive the technological decision making there is we don't practice the the chief architect that reviews every single design and approves the design we don't practice that anymore instead we have communities of practices meaning the architects who are interested in a particular topic they get together on a regular basis and they discuss what's the best practice what's the modern ways to do this and they they talk peer-to-peer and make decisions most of the time all the technical evaluation as well as selection is done by the corresponding teams yeah please so so there are several different frameworks that enable um one to implement the cultural promises one of the frameworks that we adopted is ThoughtWorks lean value tree framework you can look it up online it it basically goes around by by breaking down the vision into strategic promises and exactly the way I showed but they have constructs and proper artifacts in place that helps you and they have a step by step of rolling out such a program into your company we had to train a lot of people on what lean value tree is and we got a lot of pushback to be honest with you not everybody's gonna you're not gonna have the same hundred percent team passed this cultural rollout there's gonna be some resistance there's gonna be some churn you got a plan for that and and be more realistic yeah yeah so so there are two questions that you asked the first one is did you just go across and like instead of one team did you go deeper into a product or did you just say everybody just go ahead and use this as one the second one is a question actually the second question let me ask you again once I answer the first one it depends on the solution we were trying to solve like for AWS migration as one example we want all the products to migrate to the cloud and we have an aggressive timeline to make that happen so we created this cloud enablement team that creates this onboarding self service package so how do we know this package is going to work so we partnered with one of development team whose technology is suitable for to be the first ones and we might we help them migrate we are helping them migrate to the cloud as we are migrating them we're creating this self service package which is our product now we're going to go take another product which is on the other side of the spectrum and test the same self service tool kit right at that point we know that okay two different use cases have worked so I'm going to now now open the gates and say okay product teams just go it helps you just self migrate your own product see you in the cloud so it depends on what actually we're trying to roll out we've tried different things and this is what seemed to work for the this occasion okay do you please repeat your second question if you remember the the page where a slide where I showed the hierarchy of promises note that I didn't start it off on the bottom I went from the top and that is that is how the model works like you cannot have a bunch of teams trying to drive this cultural change upward it's it's going to be extremely painful and and won't even go anywhere and we have 80 80 plus development teams so it's not going to go much further our change fortunately happened from the top so it was the team of senior leadership executives sitting in the room wondering okay these minor changes minor tweaks that we are doing is not going to give us that much results or that much cultural change let's do something big let's go all in and that's when we brought in the lean value tree framework so fortunately it's from top down yeah exactly precisely yeah yeah it needed a bunch of these little school school of fishes out there trying to eat the shark we needed that kick in the back to like get something good going please yeah so very good question how did you define the kpis and how do you go about like tracking and making sure it's in the process you're constantly looking at it like I mentioned the lean value tree framework does exactly that in fact we have several artifacts that we identify the the goal leads when I say these are different the strategic promise leads the functional promise leads and the tackling promise leads they are supposed to fill a template like this is a framework on how do you how you roll out right that template has one of the entries in there is kpis success factors right so the the team leads they have to think through okay how am I going to deliver this value and then they have to come up with ways to measure it every cycle that we review that that particular goal or the project we will review the actual metrics and we have to show the teams are supposed to show the trend or the numerical change and progression it's okay if I mean it's not possible to expect the graph to be always like a hockey curve graph let everything is improving it's not the goal of looking at the metrics is mainly to find out whether the methodology is working or not it's not really it's not really to make sure that it's working so if it doesn't work totally fine you just want to the next idea and see if that works so that's the culture of running I was talking about currently we don't have any special tools that captures this in fact this is not this is a new framework so there's not many out of the box tools out there that helps you do it we are currently just using excel at this point it just works sorry oh absolutely yeah so take a is like 10 years back right and then be started roughly three years back and and then you could you could think of it as one year one year one year right so the the a to b is is the one that we had the like I showed the graph like dropping so fast it it was not that bad it was more gradual in fact that those are the tough ones to track because every day you're looking at the mirror you didn't lose that much weight but then if you take your picture from two years back you were like 10 it's like so that's what happened you don't realize the gradual changes as obviously as you do the drastic changes all right thanks a lot for coming