 But before we start, we'd like to know how many of you are from the BI world? Target, okay. And then, you're from Phoenix, Arizona. Anybody else from BI? Yourself. Trump? Fidelity Investments. All the rest are going to be from BI. Then, alright. So, how many of you have been practicing adults for the past one year? More than a year? More than a year. That's a cue to say that the presentation is going to start. Thank you everybody. We'll just start with Raghu. Can you guys hear me? Okay. Thanks, Anu. And apologies for the delay. One lesson learned is not to use 16 by 9. 4 by 3 for your presentations lessons learned. So, the way we kind of structured this talk is we want to kind of talk about a little bit about BI data. Data analytics. Data science. What challenges we've had. And then talk about how we actually leveraged agile and continuous integration and continuous development for delivery for that matter. And then finally come back and talk a little bit about some of the use cases where we actually used it. And what our experience says, how it worked, how it didn't work. And kind of give some insight to people what really worked and what didn't work. So hopefully at the end of the session you will see some use cases where we actually made it work. And some guidelines on to what to do and probably what not to do. So firstly, how many of you have heard about orbits? One, two, three. Cool. Awesome. I don't have to talk too much about orbits then. Personally, I've been with orbits for a little over 12 years now, mostly in the Chicago office. And then the last three years I've been in Bangalore helping start up our office in Bangalore. And as part of this office, we work on mostly half of our offices data, which includes BI, advanced analytics. And then the other half is product development, which is mostly building up our websites and stuff. So again, I think we're not that old of a company in Bangalore at least like two, three years. But in US, we were one of the top three travel agencies for the longest time. We started in 2001, five airlines kind of started orbits. I have a gift if anybody can tell me what are those five airlines? Wild guess. North American Airlines. United American Delta? No. Big airlines. Continental and Northwest basically. So in fact, our first machine names or rather our internal server names were Dunk LLC. Basically, it stood for Delta United Northwest Continental American LLC. So that's kind of how we started. We call ourselves a tech company rather than a product company or anything because we have really good people who have been involved in open source technologies, contributed a lot of books to the community. And with that, I will actually move on to the next part of our story. How many of you actually heard about Expedia.com? Okay, a little bit more. So last year, we actually got acquired by Expedia. So Expedia is basically number two right now in terms of travel agency and price line kind of neck in neck where we are. So Expedia started in 96, I think, right after Travelocity. And Expedia actually owns Travelocity, orbits, hot wire, hotels.com. Anybody knows where Expedia started off? Actually, no, it's an expensive garage called Microsoft. So yeah, so eventually they spun off from there and had their own stuff. In fact, we have offices in Gurgaon, Pune, Mumbai, Bangalore, Bangalore being us, obviously. Gurgaon is our bigger office, around 800 people right now. So I actually have two questions here. Let's see if you can take a guess. How many room nights do you think Expedia books in a year? I'll give you some clue. It actually fills up an entire city for X number of days. I don't need the number. You can tell me which city and how many days. US. Manhattan, actually. It can fill, yeah, probably at the end of the day or end of the session. It actually can do 30 days of room nights of all the hotels in Manhattan. So that's the amount of hotel rooms that we as Expedia book there. And that includes not just Expedia, the family of Expedia, pretty much everybody, right? And the other thing is airlines, right? I mean, they all go neck and neck, air, hotel, car, blah, blah, blah. So how many tickets do you think we book in a year? Okay, it fills up N number of planes, a type of plane, 737, 600 planes. So with this, right, you can imagine the sheer number of data that we collect. It's not just air and hotel. Like it's a slew of travel products. So with this being the foundation, we want to kind of talk about a little bit about how we worked at Orbez and how Expedia kind of deals with data. What are the challenges we have? Basically, how we actually use agile and continuous integration, especially to the world. So I'm going to skip this slide. Basically, I just introduced myself. So at this point, what I want to do is I want to bring Anu in here, who's going to talk more on what BI data and the challenges that we face. And then we're going to go to the next section of it. I think Shruti told I have. And when I lost what he said over there, I'm still figuring out what I'm doing. So 17 years back, when I started BI, my first job, it was very simple. Just a data warehouse, build a cube on top of it and give reports. That's what it was. Mostly waterfall. You have time-based people based. These are the problems. But over the last few years, we have all seen how BI has evolved. It has become a very small part in this big plethora of landscape. Sentiment analysis, social analysis, infrastructure, architecture, cell service platform, data platform. So you can name it technology initiators, business initiators. So all have gone into the landscape of BI, big data, big data analytics. So it's all more about unstructured data, real-time analytics, batch, right? So when this place is evolving so much, so much in an unstructured way, I think it becomes more and more important to bring process on the floor. So how have you actually delivered these projects? How have you delivered these different types of solutions, business or technology-driven? It becomes more important and how your teams are. What are you processing? What is your philosophy? How soon your feedback is? How are you knowing that you are reaching what you want to reach, right? Are you on the right track? Are you doing that square thing in the round thing? Right? So over the last few years, we've done agile in like 20 different ways or 2,000 different ways, right? It's never been a right way to say what agile is. Since my joining orbits probably 2 years ago, I can say that we have learned and I saw some hands go up when we said plus 5 years. So I'm just 2-year-old in agile, a good practice agile and we've seen how, what has worked for us in the BI world, specifically in the BI world, where we have different kinds of projects. We have web businesses. This is what I want very clearly, structured data. We have done agile in ways where they are migration projects like move DB2 to Taradita, move Green Plum to Taradita, right? How do you go about doing it? Because business are not going to give their time over there because they already have their solutions. It's a technology-driven initiative. It's your CTO sitting over there and saying, I want this to be done in this period of time. Then you have the other cool projects, right? The semantic layer. You want to build a self-service model for your business. So how do you go agile in that way? So these are small things. We are not in the next few slides. We're not going necessarily into the tools of how to do it. That's basically a choice. But the process, the philosophy is not a choice. What you need to do, what you need to keep in mind is what we have learned and that is what we will be going through in the slides, in this BI landscape. So how many of you can actually relate to this picture out here? In your world of BI, right? So I think if you look at any organization where they've done practice of BI, I can speak for orbits. We did BI right from, I don't know, 2002 when we kicked off our site, 2001 when we kicked off our site, and we had a BI practice. But it's evolved over the last decade into what it is today. And every time there is a learning opportunity in terms of what works, what doesn't work, what do you do, how do you do everything, right? So the first challenge, so I come from a product development background. That's where most of my experience has. But last five years, I've been working on the data side of the world. So the first time I actually moved over and I looked at the problems and said that why can't we use Agile here, right? And we've done Agile since 2006 in our organization. Why can't we actually do it here? Then actually came out the problem saying that there are different set of skillset that they're there within an organization and within BI especially, right? How do you actually put them together and really look at solving a problem rather than basically saying that, hey, my job is gonna be taking six months of whatever you need. You come back after six months, I'll tell you what it is, right? So this involves your ETL, your data modeling, your visualization, whatever the analyst-related work. So time-to-market was very important for us. In an e-commerce company, right, you have a hypothesis or let's say you have something that you wanna do from a product perspective. You have to act quickly because the market changes so quickly for us that we can't expect to think of something and say, hey, we'll do it after three months. Our release cycle in product development is actually weekly with at least 10 to 20 deployments every day. That's the kind of speed we move in the non-data world of product development. Now how do you take that and bring it to the BI world, right? So that was one of the big challenges that we had. The second challenge we had was real-time analytics, right? And everybody talks about real-time analytics. And I don't know how many of you guys actually do real-time analytics within your organization, but you have to question the people who are asking for real-time analytics. Do you really need real-time analytics, right? And those are the questions we started asking our business people, right? They cannot change the strategy near real-time. Impossible, right? If you have a strategy of growing room nights in near real-time, then they're lying to you. So you need to figure out where real-time makes sense. So we found out and carved out business areas where real-time makes sense. Campaign management, right? Let's say you launch a campaign on TripAdvisor, which we call the travel research area. You want to see how the campaign is happening right now, not tomorrow. You want to see, okay, are people really coming to our site based on this? What are they doing? Great case for us in terms of real-time analytics. Personalization, right? We do a lot of personalization on our site. You cannot have stale data to do personalization. So you need to funnel the data back in real-time or near real-time so that we can actually do personalization. Simple example I'll give you is anybody's heard about the controversy of orbits with Mac and PC? Nobody. All right, so this was in 2010 where we actually did personalization based on the browser settings. And we said that, hey, based on our analysis, we noticed that people with Mac tend to book higher rating hotels than with people with PC. That might be true, might not be true, but according to our data, that was true, right? So we wanted to do a test and see, okay, is it really true? So we had to funnel that data in real-time for our personalization and saying, okay, show them better hotels first in the sort order, right? Don't touch the pricing, just the better hotels on the top. And we actually proved the hypothesis. And that's one of the examples where for us, real-time made a lot more sense because it was not just the browser. We connected a lot of other things like, for example, is this person a new visitor or a repeat visitor? Is he coming from which marketing channel? So there was a mix of variables that we actually collected. So that's the second example of where real-time analytics made sense. And the third one, which was really big for us, was buy versus build, right? If you go back in 2009 when we started using Hadoop, the ecosystem was not as great as it is today. So we had to make a choice or decision to build a lot of things in-house. And we still use some of those things. And actually right now, you look at the slides further down, we're actually moving some of those things to cloud right now. So that's traditional BI, right? What we were before. So where does agile come into picture, right? Again, like I said, my background is mostly product development. We've been doing agile since 2006. 2008, we transitioned agile in the large. Do you guys know what agile in the large? So anyway, so agile in the large is basically what we did was the entire organization, right? They work in smaller agile teams, either a feature team or a component team. And it includes everybody who can actually get things into production and make decisions. So obviously a product owner and UI and data modelers and stuff like that, right? But we got our business teams and marketing teams ingrained into that model. Because now you have a part or a feature team which can actually execute end to end with making decisions, right? So that's what we did in 2008. Now when we come back to BI, that was not as so simple for us. I will tell you our journey was very tough getting BI people to start using agile in 2010. One, people usually come from a waterfall background. So they are always looking at the way of designing the model, doing the ETL, building the visualization, and then putting it in front of the customer. So for us to change that mindset was the trickiest thing and the most difficult thing. So if you think you're using agile, I think I want you guys to question yourself. Is it the mindset that we have changed with people or the behavior of using two-week sprint stand-ups retrospective? Is that what we are doing? I think that's the key difference out there. So for us, it was a rough journey and I won't say we are there. Perfect, we are perfect. We're not. It's an iterative approach. We learn, we change. Further down, we'll talk about a few examples where we started playing around with the sprint size in terms of two weeks, three weeks, four weeks. What is the ideal? So that's kind of one of the things we challenged. Again, so that is kind of where we really introduced agile to BI, right? So getting in front of the customers in a shorter span of time even though it's a tiny bit of information that they need from the big picture. So that was our first step in terms of getting BI transition to agile. Again, I think the question is scrum, can ban, what do you want to do, right? Eventually you need to figure out what's the right rhythm for you and then stick to that. Anu's going to talk a little bit more of some of our use cases where we actually use scrum and some of the use cases where we actually use can ban. So that is kind of, again, we don't enforce our teams to stick to either of the ways so they can pick what it is and based on that they execute. Actually, has anybody read this book called Phoenix Project? Okay, fantastic book. If you guys haven't read it, it's about IT and DevOps. So in this book, the common theme is called whirlwind. So what whirlwind is, you will relate in BI, right? So you're working on some changes. Somebody from marketing team is going to come over and say, hey, we are bleeding. Our sales are low. I need your help to pull this data out. That's basically whirlwind. You cannot control it no matter what, right? And in those situations, what do you do when you're in BI? You drop your existing work and pick on that or you say no to that. It's a very tricky situation. I'm sure each one of you have been in this situation if you're in BI. That's really tricky. So you guys have to be cautious of that in terms of how you balance those things when you're working in the agile fashion. So this is something we actually call disruptive BI. Not very common in a lot of the organization. But when you talk about continuous integration, automated deployments, continuous delivery, automated QE framework, outside BI, this is beaten to death. So if you talk to anybody from product development side or like a regular developer, they will just say, hey, what's the big deal about continuous integration? But I would want you guys to go back to your people from different organization in BI and ask them this question. What do you guys use for version control? So I'm asking this question to you guys. What do you guys use for version control? Anybody? Get? Get? Okay, good. When did you guys start using it? A year ago? Exactly. For us, five years ago, there was basically, BI didn't use any version control, but they used version controls from the proprietary software that would allow them to do. Like simple examples, Microsoft gives you a version control, Informatica gives you a version control. So they would still use it, but when you really want to do continuous integration, version control is the fundamental thing out there. So we use Git prior to Git, we used to use something called Acura and CVS, CSV, and so we had a long history of different version controls. So that was the first step, right? Changing the mindset of people in BI and say, hey, can you go use Git? And they're like, what the hell is Git? Does it have a UI? Can I use it? So that's where the mindset and adapting to change comes into picture. So for us, that one was a big deal. We worked through, in terms of how we integrate this with Jenkins, how do we integrate with Stash? By the way, Stash is called Bitbucket now. I don't know if you've heard. So, and then how do we really enable these people to think in the fashion of CI and CD? So that is where we spent a lot of time last year, actually. And I wouldn't say we are done, but we're all almost done. We are there. We know how to use it, what to do. So that's pretty much what it is, right? Cloud, again, it's basically, everybody uses cloud now, right? There are different reasons to why you want to use cloud, but I think it's a very touchy topic when it comes to BI and cloud, because of data privacy and all sorts of things. But eventually, everybody is moving towards this. So that's inevitable, I would say. So I wanted to kind of give you guys the glimpse of how we actually use CI and QE automation framework. So for visualization, we use MicroStrategy, and then obviously we use Informatica, we use ClickView, we use a lot of things. So the gist of this is basically, we have a Jenkins server that basically executes our automated tests every time somebody checks in. So we use Stash, Jenkins, Git. The one thing we actually tried, I know the bottom portion of the thing is it's not visible that well. Obviously we use Selenium for all the MicroStrategy tests. So that is actually pretty cool. So if you use MicroStrategy, invest in Selenium, it's actually very good. It's for automated UI testing. And then we evaluated this thing called SQLI. I don't know if you guys have used it, but it's a paid tool, but we weren't really, not sorry, query search. It's not, I don't know, we weren't really convinced of that, but anyway. So at this point what I'll do is I will transition over to Anu. So you kind of heard where we started with BI and where we are right now with both agile and continuous integration. We want to dive in a little bit more in terms of how we are going to do the use cases or what we did, what are the use cases we solved. There are a lot of technologies, tools, processes that we are following. What we also realized is we need to have, to my point earlier, a lot of discipline on the floor. How we are doing around the team. How we are looking at your project. When we say Envision, when we say Execution, Development, what it actually means. A retrospective, is it the same for every project? An Envision, is it the same for every project? What we realized with BI is a dialogue like a live in relationship. You get in when you want, you get out when you don't want. You just keep hovering around. So, yeah, I know. So it makes it more complicated because under the tour of BI projects, as I talked earlier, as I talked earlier you have the migration projects, you have business driven projects where they know what they want and then you have these data scientists and ad hoc power users, advanced analytics users saying just give me the data, I will know what I want to do. But then you just can't take and give the data. You need to build pipelines, power platform, et cetera. So how do you go about doing that in Agile? And then the cell service. There's pretty little tools which has just to begin with drag and drop, but then it becomes, can you give me a scheduler? Can you give me this? Can you give me that? So there are all these layers that happen in the BI. So exactly how do you fit in each of these into Agile world? And then not to just stop that. You have, sorry, I'm just going loud sometimes. I don't want to say this is important or less important, but, sorry. And then you have global locations, right? Like Expedia and the orbits, we were saying we have Gurgaon, we have Bangalore, we have Chicago, we have Bellview, and I'm sure Target has the same thing and several Dell has the same thing. So you have people located everywhere from Brazil, India, China, US, Europe, and all kinds of places, different kinds of roles. So how do you actually, you know, make sure all are going in the right direction? And, okay, I missed my thing, but we know we are Agile, we say make your decisions as you go, make your decisions this way, that way, but we all know that we are time bound. You can make your decisions, but I really know that I want this in two months. I really know I want this in one month, right? So you can't beat it, you are time bound. So how do we go about being Agile? So I'm going to quickly go through three projects that we have done in the last one year. I'm going to slightly touch upon, because of time, what were the challenges and why we decided what we wanted to do. So the first is the most favorite catered project with any team, that is the platform migration, which I talked about, right? Move from one platform to another platform. Move this report from MicroStrategy for some decision, move everything to Cognos or move to business objects. We're all very familiar with this. So what happens with that? What are the challenges? They are time bound because you want to move there. The license is expiring, blah, blah, blah. They're quite lengthy projects sometimes, but they are quite like, for example, Teradata or you have columnar storage, no SQL database, SQL database, you want to be moving across to say that this is most efficient or that is most efficient. You have a strong user mindset that why do you want me to move? I'm happy where I am. You are pushing them outside of their comfort zone, right? That's a big challenge because you're not going to get that time. Okay, I think half of it is not visible. But anyway, so what we did for these challenges is the first thing is we learned to break down into components, right? For example, in the travel industry, you know that we have products like Air, 48 car, cruise, et cetera. So Air, so okay, what is the 16 data of Air and the transactional booking data of Air? Within the booking data, what is blah, blah, blah? Customer data, channel related data, that is your SEO, SEM, and you know how people come in the acquisition of traffic. So we started channelizing the data. So you break, basically, I wouldn't call it vertical slicing, it's also a very controversial subject, but just break down into components that you can relate into. You can say that yes, this is something that I can relate it as an epic, build stories and start working on it. The other thing that you would have seen is the team composition, which I said is very important. You know, whether you're going to have a small team, you're going to have product in the same team, are you going to have QA in the same team, how are you going to do team allocation, whether it's 70% of your bandwidth in the sprint for your daily job, for your job that is envisioned, and you know, the other 30% for admin support, and your personal goals that you might have that I want to learn Python this year, right? Whatever it is. So we were very clear that we have to attribute only 70% of our sprint. Make sure you put in your vacation or unless it's an emergency leave. So such discipline was very, very important. So the team composition, and also to the challenge that we had teams half-split in Chicago and then in Bangalore, we went through a very big exercise in the beginning of 2015 to say we are going to have co-located teams unless otherwise, which will be in the same or similar time zone as our business, as much as possible. So that, you know, you have the product and the QE and that helped really well. So the iterations, I am not able to see that. Okay, there are certain other things like early feedback, iterations, where it's so key that we started, we kept talking to the business, to the stakeholders, the feedback. This, when I was putting this slide and Raghur and I was just talking, isn't this common sense, but we struggled. We struggled to put this on the floor. To make sure people log in efforts every day to make sure we have our sprint metrics at the end of the sprint. So we tracked velocity and over a period of three months we could really see the velocity, the burn down charts that improved drastically. It was a struggle to put it down and though it seems very simple, those were the things. And last year, the same Agile conference, the two people you see on the slide, Anushan Pradeep attended and you see the Scrum Master pointing our head on the developer who didn't log in the efforts. So the Scrum Master has to be harsh. We took it that the Scrum Master will be on a rotation basis and it started off with a lead so that they can set an example and then there were all, Anushan is just a college recruit but she did it. So the team discipline and the way we are doing the backlog rooming, the sprint metrics measurement, all this was very critical for us to finish the migration project. Similarly, there was another project example which is a business project, business driven project, these are user generated content. The previous project we knew that we needed a product and a QE but here we decided we don't need a product and a QE because it's a new venture, it's uncertain requirements, businesses don't know what they really want. We are going to figure it out with the business. So do you really need the same team model? Do you sprint envision, whereas in the other stuff was about technical feasibility, about doing a requirement for one week and then doing a technical feasibility. Here the sprint was more like data analysis, doing a wireframe, getting again an early feedback and getting into the system. So we actually called out what your envision should be, when should your execution actually start. So that played a very critical role in seeing that we progressed with feedback and the iteration. So here you see the scrum master starting up water bottles because the job of the scrum master is to have no block, to remove all the blocks for the developers. So similarly we had the technology initiated project that I talked about where it's a moving goal post, it's technology challenges, you have features starting from little bit to a lot more. So again, here I have not given what we did but essentially it is about again defining the challenges we are going to face. Keep that in mind and then say my sprint is going to be like this or I'm going to divide my work like this, I'm going to target it like this and then go about in that discipline. And then the fireworks which Ragu was talking about, you're going to have some SLT tell you I need this report in the middle of the sprint. I need this, there is one P1 bug, some Informatica failed or something failed and you need to look at that. That is why the 30% allocation is just a lot to not get our sprint work affected when you have a certain time written off. If there are no bugs good, you have your personal goals and you can always go ahead and help them. There is kind of a cheat sheet, not necessarily just two in all cases but for us to at least take us quick because there's always a question on the floor. Do you want to go scrum? Do you want to go crumb? So both the projects that I talked about other than the semantic layer we opted for scrum because of the reasons that it was short iterations we knew what we needed to deliver we were self-managed teams whereas on the other side like what Ragu was talking about it's continuous delivery consistent randomization, what do you do? So this was a quick graph to say whether we want to go scrum or whether we want to go crumb. And what we realized at the end of the year is agile works with consistent constant feedback, within the team members to your business stakeholders, again sounds very simple but the feedback was the most critical part to ensure that you are reaching what you defined for. So finally this is the last slide and what not last, my last slide what work for us, migration project any guess I already told what's your take, product roadmap driven projects, either you already know or you've been listening to me at home, I was hoping okay good, so somebody said Kanban though, so yeah he gets the t-shirt and you don't he spilled the beans, anyway self-service was Kanban and ad hoc's integration like he said we just got acquired by Expedia, there's a lot of integration happening around the world and we choose Kanban because we have to work with a lot of teams and there's a lot of randomization lessons learned. Yeah, so we're not going to hold you guys from your tea break so just another minute you can bear with us, one thing I want to reiterate from the previous slide is that's what work for us doesn't mean it's going to work for you, so you will have to really figure out what works for you so that's one thing, the other thing is again I cannot stress enough of this mindset stuff because if people are not committed and they don't understand the philosophy of agile without executing it then I think that's going to be a challenge and for us last one and a half years continuous integration and continuous delivery has been very critical within the BI framework and then the last one is test and learn and you can call it search and discovery test and learn whatever it is but you need to figure out what works for you so that's kind of the bottom line, attrition what attrition number for us people so so attrition rate it's different for us in different locations obviously I don't know how it's connected but yeah fine okay so nobody quit because we were agile so bottom line so we do exit interviews nobody came and complained you forced us to do scrum versus scan ban but it definitely leads to a lot of heated debates definitely yeah like I said right people skill set people mindset in my mind those were the two biggest challenges we had so if people don't believe in agile no matter what you do they are not going to give you the best and you cannot extract the best out of them first thing is change them if they can't change them figure out somebody else right that's the first one the second one is skill set with BI you have different skill set right a ETL person might not know how to do visualization so we did a lot of cross train again not everybody were interested in cross train we figured out who was interested we asked them to cross train across the different areas of expertise so that was another area for us in terms of working and the third one bottom line is holding people accountable right we come to standups and we upfront to their face ask what you did and if they don't have an answer people are going to hold them accountable so these were some of the things people were not used to yes one last question by the way before you ask a question right if you guys want to know more about how we track metrics what are the metrics we track feel free to reach out to us later we'll be more than happy to share some of our burn down charts or metrics that we track challenges and stuff see I would probably I had the advantage because our product development went in first with agile so we had almost four years of experience in agile within our organization and I was part of that four years and obviously you don't question a product development team whether it's successful or not for them right so you bring the same concept to be I and try it out it will work it has worked for us in way too many places not to discount it where it didn't work out so that's so I'll give you a simple example I know this was the last question simple example I'll give you have you ever built a data mart okay so anybody who's built a data mart what's the simple example I'll give you a site site analytics you guys know site analytics right so if you want to build a data mart for site analytics what's the time estimation in BI just a rough estimate before you give something to the customers when I say customers or internal customers six months three months eight that's fair enough it is nine months is the estimate we got when we started off this in 2010 they said it will take nine months to build this data mart alright we vertically slice this based on each of the use cases so that we can actually get one metric to the customer okay in three weeks so we actually took eight months to build the data mart but guess what time to market that makes a huge difference with agile if you're giving a metric to a customer there is a huge difference between giving in three weeks versus at the end of the nine months that is where you're going to win so thank you again thank you