 Our next panel, though, is a group of folks who are consuming all this open source. So we'll probably have a good perspective from, hey, we're bringing this open source in. We're using it to build real systems and use it in mission-critical production work, and really can provide that different perspective, not from a supplier of open source but really from large consumers of open source. I'd like to invite the panelists up on stage. I'll bring you all up and then introduce you one by one, but please come on up. We're adjusting. Oh, there he is. All right. Go ahead and have a seat. So we have Sabu Alamaraju, VP of Technology at Expedia. So you lead Expedia's cloud migration and enterprise data center, all the public cloud work that you're doing. That's a humongous job. Niti Gupa is with the SVP of Technology at Hired. How many people here know Hired? Oh, we got to change that. Yeah. It's like the change health care guy yesterday. He was like, we do like two-thirds of the health care financial transactions in the US, but we have a terrible market. I think Hired has better marketing on the front. And then Justin Dean, who works at a company I believe most people know, I end up hitting that refresh button constantly on Ticketmaster to get good tickets to my favorite concerts, but welcome all of you. So you all clearly are using open source, as Patrick mentioned. More and more, our systems are based on open source, it's sort of one. But tell us about what are some of the key open source technologies that you're seeing used in your stacks? What are you bringing in these days to move towards this sort of DevOps movement and so forth? And maybe we'll start on that end just because we always start on this side and we'll come back. Yeah. I mean, Ticketmaster at its core is, we've been open sourced since the beginning. Well, I say the beginning. We're a 41-year-old company, which a lot of people don't realize. But the culture has been very, very DIY. Our founders built the nucleus of the ticketing engine in 1978. Some of that software we still run, fortunately or unfortunately. But the culture has been very DIY. So that led into open source naturally. So we use probably almost all of the projects out there in some way, shape, or form. But the core of it in my world and the technical operations in our cloud and infrastructure spaces, Kubernetes and containers is all the rage for the last two years-ish. And it's really changing the landscape for us because we're using our container movement as a catalyst to improve software across the company in all aspects of it. So all of the, we're using these technologies that not just from a cloud computing perspective, but from a let's deep dive into every one of our applications and do the work of cracking open the code and making it more modern and really up-leveling on all fronts in order to get it to run into a container and then to get it to run into Kubernetes in a public cloud space. It has to be pretty good. It has to be instrumented. So we're sort of using it as the catalyst for an entire movement on top of the open source projects that are improving and becoming enterprise-grade. Yeah, Nity, it seems like trying to figure out what the right abstraction layers are to build these platforms is so different. Is it VMs, is it PaaS, is it containers? Which application framework are we coming in? It seems like that is such a tricky thing to figure out. And you want it to be maintained long-term. You want it to be sustainable. What are you seeing? What are those abstractions? What are those platforms? What is that that you're deploying? Yeah, I mean, I think our tactics. So let me just begin by introducing Hired, since most of you don't know Hired. Actually, even before that, happy International Women's Day, everybody. So Hired is an application that's purely built on open source. We're not just a web application, a mobile application. But in addition to that, we have a fairly sophisticated machine learning algorithm that presents job opportunities for highly skilled tech workers. As we've grown our infrastructure, and as we've grown our solution, our tactic is pretty much like what Justin said. It's about finding the right tool for the right job. We are based in San Francisco, highest. It's a very, very costly market for a startup that's getting off the ground, especially in this environment, which is so highly competitive. We are very, very cognizant of our human cost, and we're also very cognizant of our time to market. So because of that, we don't have the luxury. We were not born 40 years ago. And the ecosystem was very different. I started my career at Bell Labs, where we had an NIH syndrome where we had to invent everything within the confines of Bell Labs. And that's completely and utterly changed. So for us, our approach is not about figuring out the different layers. Even as we invested in machine learning and figured out what algorithms we should be using, we first go seek those solutions if somebody solved them. Only in, and then we enhance and build on top of them. There are very, very few cases where we have to sort of start from scratch, if you will. So our primary motivation and driver in making these decisions is around cost and time to market. Interesting. How about at Expedia? What are you seeing? What's behind staff? I think I actually echo some of the Nithi's comments on do-it-yourself versus open source. I worked at, before joining Expedia to lead cloud migration, I was at eBay building eBay's infrastructure, private cloud infrastructure for four years. And we embraced open source. In fact, we were, at the time, one of the largest successful stories of OpenStack, adopters of OpenStack at eBay. And subsequently, they moved on to Kubernetes and its open source, Docker, Meizos. They were all in the culture. And we do have similar mentality of embracing open source, being open, sharing all that stuff. But what I see happening is that as cloud has, the service-ification of problems has been growing very rapidly in the last few years, I think, from 2011 to 2012 onwards. Until now, we have massive warehouse-scale services offering very, solving real interesting problems that we were not able to solve as quickly in the data center. So that has changed from open source to a bit close source because I don't know how this service called Kinesis works, but it's solving that problem. So that is influencing how we are thinking about this open source in quotes because we have to take advantage of what exists there so we can invest in our time and money in where our business goes. So that's the change I'm seeing over the last several years. I could actually work with you, supporting you. Yeah, and we saw that yesterday in the serverless conversation, this avoid lock-in solves a super fast problem. I'd be curious to hear what the three of you, where do you land on this pendulum? Because I think those are tough decisions because it's so tempting to just say, I gotta get to market and just take that. And then five years later, you're going, wow, this is getting expensive. From the Ticketmaster's perspective, there's an interesting dynamic that happens when at a large company with lots of diversity of our product space and different teams and different cycle, like where they're at in the maturity of their particular product that they're working on, we have a lot of aspirations like, hey, we're gonna go and do whatever the thing is. We're gonna be all Kubernetes by next year. And then reality hits, and then you realize there's an entire spectrum of these different teams. So some of them take all the advantage of something new on serverless technology, and if it fits within, if it helps them either by speed, agility, or some reason, deliver some business function, and it's worth doing that that's not in the vanilla suite of sort of corporate offering type of a thing, then we encourage it, go do that, because there's some ROI that's important, but we try to be eyes wide open across the company that says by doing that, you may not get centralized support on all these things that are fairly, we can't support all Earth's technologies, right? So, but we've learned that standardize on the things that make the core bread and butter things, but definitely allow teams to operate like many micro-businesses, and they have their own drivers and motivations, and having an open mind to that works really well for us. On the flip side, it's challenging a little bit because it's easier when everything is one thing, right? Use this cloud, and here's the rules, and here's the menu in an out-program menu, right? It's just not super practical for large-scaled-out companies that have probably a diverse business portfolio. Yeah, yeah. I think to the point that specifically the lock-in point, lock-in can happen regardless of whether you're using open source or closed source, right? So, at some point, if you hold back on avoiding lock-in, you're going to sacrifice efficiency. You're going to sacrifice, there's a cost component to it. And no matter how much you try to avoid that lock-in, even if you sacrifice efficiency and you incur additional cost, you still will not be able to. So, our personal point of view is go all-in, and if you need to change down the road, you change down the road, right? Yeah. Can I make one comment real quick? Sure. A lock-in comment? Since we did have a DIY spirit, so we built our own cloud 10 years ago, and it's really, really useful for developers, it's easy to use and all that, I would argue that we have built the most lock-in machine in our private cloud that is really challenging. You can lock-in yourself. You can literally lock-in, and then you have to staff it, so it's challenging. So, it doesn't just go on the bender side of things, it's you can lock-in to your own stuff. Right, absolutely. Sure. Where are you? I couldn't agree more. I think the moment you pick something, build some tools around it, build some operating procedures around it, and culture around it, you're locked into it. Whether that is an open-source software or a closed-source appliance or a cloud service, lock-in is here to stay, it's part of our, it's actually a cultural artifact, because I do believe that the process of maintain, the act of maintaining change, having the culture to, yes, we can change anytime we need to change, I think that is much more important than, but then the fear-driven mindset of, okay, I don't want to embrace X product because the X is closed-source, I don't like that vendor, he might mess my things in future, that's fear-driven approach, I would definitely not take that fear-driven approach, instead I would look at the opportunities that exist for me, for my business, for my customers, and go after those, and the lock-in is a process of I have to deal with it. Yeah, it's funny because on Tuesday, I kept saying these black-and-white, lock-in versus not lock-in, it's, we're past this, now it's this degree of subtlety, of managing risk somewhere on that spectrum, which is what you're all talking about, and that subtlety truly matters. What's interesting to me though, in terms of you three being here, is we have a lot of folks in the audience that work at companies who build commercial implementations of a lot of open-source, and that could be intentional as a pure kind of open-source-based startup, or it could be just because, hey, we gotta get to market fast and our products just so happen to use a lot of open-source components to do that. But, and so they have program offices, they have people inside of the company who manage the intellectual property sharing, that work the tool, the development process, to pull code in, modify, share back rule, so on and so forth. And we hadn't seen that as much in what you all represent, which is not necessarily, you obviously have a service you're providing, but we think much more of this end user, these type of groups being formed. What, in each of your companies, do you have like these policies? Are you building competencies? Is it something to build right into the engineering team? Like, how do you approach open-source in your firms? Yeah, like I'll start over there. So, it's interesting because one of the first things I noticed being here and talking to people, and on day one, I sort of had a little bit of open-source FOMO, maybe? I was like, you know, everyone has these open-source program offices and we don't have that. And part of me is, you know, going and rationalizing like, well, because in our DNA, but then part of me is like, we're probably missing the boat a little bit on like doing it in a structured way. But, you know, we definitely encourage in just the fabric of our engineering culture, obviously using open-source, but contributing as well. And part of it is we want the highest caliber talent working at the company, and we want them to be happy and excited about what they do, and we want them to be part of the open-source community. And normally that's where we find them and we recruit them, so they're naturally inclined to want to participate in that, and then we support it in every level that we can possibly do. Everything from, you know, actually committing to upstream projects, you know, as well as supporting, sending people around and talking about stuff, which, you know, if you go back a few years ago, you wouldn't see a ticket master standing on stage talking about things that they're doing, right? Because we're super secretive and keep it all locked down, right? So we're changing how we think about the value we provide as a company, and it's not our secrets, right? It's what we provide in the platforms that we offer, and there's almost an interesting corollary. As a company, we're opening up our platform, meaning not the people who own all the ticketing inventory, but now we're becoming a platform for a marketplace, essentially. So there's almost this company level open-source mentality that's happening and permeating, so it's, yeah, so the takeaway that I have from this week is to really start investigating, you know, do we need an open-source office to take advantage of some of the work that I'm seeing a lot of folks in the audience do, and put a little more structure and rigor behind it, because we probably can increase our horsepower and velocity and contribute more. Yeah, interesting. How about at heart? So some of the things we do in terms of adoption, so our application is built using Ruby and Rails, so there's a fairly large community of gems, community support around gems that's available. So for anything new that we're looking at, we first hour to look for any gems that we can just adopt. In terms of contributing back into the community, because so much, I mean, literally, I could count on my fingertips of things that are not open-source in our infrastructure. So given that, we sort of make a conscious effort to contribute back into the pool, some of the rigor and process that we've introduced, which has been very interesting, is as part of an engineer's growth ladder, one of the, as they become, as they grow in their career, they need to, there's a leadership aspect, regardless of whether in the management track or not, as an IC, there's a leadership aspect and there's a community aspect, and that community aspect is giving back to the community that you've gained so much benefit from. So that's been helping. Yeah, I think we take a similar approach. In fact, I was thinking, Justin was thinking like, we don't have an office, program office, but we are practicing learning, we are practicing sharing, we encourage, in fact, one of the norms we set for our engineers, like you had talked about, is that learning and sharing are part of your norms. We expect each engineer to adhere to. So you cannot say, I won't share, I won't talk, we expect you to do that as part of exhibiting growth mindset, taking feedback from others and leveraging. So we have this question like, yes, you're doing this problem, but are you leveraging something that exists in a different team? We try to look for that to an extent, as long as it doesn't impede that person's drive to do some things on his own, because there are teams that want to do it and we don't stop that. But then more important, the leadership aspect is are you able to, as you grow as an engineer or individual, are you able to promote these behaviors, learning and sharing and demonstrating? Will you encourage contributions in open source, but we don't have a program office as such, and actually learning from others and how they approach that problem? Yeah, I mean, because the question I have is, what I hear a lot, you heard this from Home Depot yesterday, right? Very traditional company. And I think they said four or five times on stage in their keynote. A, we're hiring. B, we love open source developers. Did we mention we're hiring? We love open source. We're hiring, open source developers are awesome. I mean, clearly that is a huge motivation to let your employees go work in these communities because that's what they want. That motivates them, that helps you and so forth. What are some of the, but then I talked to, let's say, a financial services company that'll say, whoa, we're not, we're regulated, we're not sharing. What are some of the road, can you give some examples of road bumps that you've had in terms of, we're not quite sure how to share. We have white and black lists of licenses or we're not, we don't know how to pick projects. What are things that you struggle with that maybe this community can help you with? So I can speak for us exactly the example that you gave. As we get larger enterprises, especially banks on our platform, they get scared, they get scared. They ask for a disclosure on, when we get the InfoSec request, they ask about open source software and sometimes they ask for us to share the list of open source and that list is really, really massive. Do you have a good list? I won't put you into it. We actually do have a good list. Yes, we use this to, we use this gem thanks to the Ruby ecosystem that actually discovers it and also actually very easily picks up as to what the licenses are as well. But yeah, it's a big challenge in terms of not just educating our customers, but also their legal teams and their security consultants that is to the process with which we've gone through in terms of, through which we've gone through in terms of evaluation. We have this constant conversation with our legal. Thankfully, over the last several years, legal is sort of catching up. So several years ago, it was a huge struggle in working with our legal department, but I think the industry is catching up. Yeah, I would say we're in a similar boat. The thing that I think, I guess the biggest unlock for us in that space was starting to educate the folks in the company who manage those risks like our legal, our PR teams like those are the two big ones to get anything done or approved. And the closer they are to understanding the value of being a company that is embracing and adopting and contributing back in an open source community, the more they can help enable us to do things and approve things versus just look at the risk side. Cause there's not a lot of upside when you send an email and say, hey, we're thinking about doing this thing. Can you approve this? And it's just some legal doc, their default is no. Why would I do this? Hey, this is why we're doing this and encouraging this. And then if you get them connected to the financials of it, like, hey, we're trying to recruit just as a basic example, recruiting costs a lot of money. We need top talent to do blah, blah, blah. So if you can somehow tie it to, this is super beneficial and important for us and we need you to help us be an advocate to sign the papers or whatever it needs to happen so that it doesn't get stopped in sort of like email risk and or I don't really know. And definitely in the, I don't know. I've had a lot of those. It's something that should take two days, takes two months because no one's comfortable just putting their name on it because they're confused. But you've been through generations of sort of, OpenStack, the Mesa, to Kubernetes and sort of, what were the, did you experience similar roadbalics? We're like, whoa, we're going to rebase our platform on this thing and legal risk or like, what challenges do you see going through that problem? We actually were fortunate. We did not go, I did not go through the roadblocks in terms of using open source, adopting open source and operating at a larger scale because we actually benefited a lot from that community and we were able to scale, introduce tremendous financial and agility gains for the companies. So that was positive. I think the roadblocks are sort of subtle roadblocks that I have seen in my career in every company I work for because not every engineer is willing to stand up and say, yes, I have this small contribution. I'm going to make it, tell me how to do it. Oftentimes the, there is a reference, oh, now I have to do this. Why do I do this? So that culture, mine shift is still happening, has to, teams are going through that. I think there's a lot more mentoring and this practice of growth mindset needs to continue so that you're sharing and you're encouraged to share, you become more comfortable sharing. A lot of us still have the discomfort, oh, I need to commit. Now it's part of my resume. And is it that they're too busy to say like, ah, you know, you're already, I'm already working so much. Now I got to go commit this thing to a project and they're going to give me a hard time and they might not like my code. Do you experience this a lot? There is that. There is certainly some of that because teams are, in every company I work for, there's a lot more work to do than a number of team, the number of people that have to do the work. I think that is by playing a pit of four. So I've seen, unless you have a dedicated effort to reward teams, to encourage the practice, to show them how to do it, I don't think it's naturally going to happen. I'd love to get your, all three of your advice on, you know, one of the things we said early on this week is how do you, how can you be a really good upstream for this kind of downstream consumption? And one thing I just heard is like, man, if you make it hard for me to go submit patches and you're raking us over the coals all the time or it's an unclear process, probably not going to get that code back and starts to kind of deteriorate. Well, what are things you would advise upstreams to do? So one of the things that's worked really well is, you know, when we started down our Kubernetes journey, one of the things that we did immediately was realized, hey, we do not have the, you know, talent internally to really say that we can operate it at scale production grade. So we partnered with CoreOS and through that partnership, we've jointly built things that were very specific to an enterprise, things like detailed cost and charge back inside of Kubernetes itself, right? So things that probably the upstream doesn't really deal with, but for me, I have to build this stuff back, you know, a million internal users. So we worked with the product teams and we came up with, hey, you know, sketched out what to build and then we jointly build it and then they helped us push it back upstream, those sort of things. And it, they know better, you know, how to get something back upstream than we do or one of an engineer who doesn't do that professionally all the time, but it was just like the support harness that you partner with companies that have a vested interest and it makes it easier for us to jointly solve these problems and they take some of the friction out of it for us. So I thought it's a really good example of how to take a company who's, we have all the problems that, you know, a lot of the upstreams don't deal with. So we have a lot of knowledge and we can offer that and some hands on keyboard horsepower and then they have the knowledge of how to get it upstream. Yeah, I mean, it's interesting because you, you know, in any user driven innovation cycle, the end user requirements are obviously critical to this and you know, a developer, you know, maybe working in isolation isn't going to see the scaling problems and the, you know, that you gave a billing example that, you know, you would, but what you're saying, and I think we all talk about is that having a upstream that has a support ecosystem of organizations that can act as essentially commercial intermediaries to that project is highly valid. Do you look for that media? We don't necessarily look for commercial intermediaries, but we do look for open source software that's well supported and has a community support behind it. But is that supported in a commercial sense and that like others, 10 vendors there or is it like? May or may not. Okay. It depends. So for really large pieces of infrastructure, yes. So that's a backup option, right? We may or may not call upon that commercial, that support infrastructure, but we know that that option exists. But as it pertains to smaller pieces of software like Gems, we don't look for commercial support. We primarily look for, look to make sure that, you know, again, like tying to this previous conversation around security and making sure malicious software doesn't get into our infrastructure is making sure there's a history of changes. There's a good, a fair bit of community support around there, around those Gems and making sure that this is not just something one developer setting somewhere has just shipped. You know, there's like a, there's a few people who have been supporting it for a period of time. So it's hybrid for us. Yeah. I think going back to the question you asked, I think what I look for in off stream based on my experience in infrastructure open source projects like Kubernetes, OpenStack and Mesos, I think in the community, we tend to underestimate what it takes to run any software at scale. There is a service aspect that we sort of miss in the way we architect and build systems. I remember when I was doing an OpenStack, the number of, we had about 30 engineers. I remember, I think the number of contributions we made to upstream was handful. Where our time was spent is in actually operating the software at a certain scale and we were increasing the scale systematically month after month after month and that took a lot of work. And I think we tend to underestimate what it takes to run software. And I think operational operability is a fairly complicated task. And that's where I feel that the cloud is winning and open source when you compete on the same terms like the same feature open source versus a cloud provider. The cloud provider is winning because they're able to focus on automation and operability much more than the community is able to and there is that, we have to consider that a lot. I do feel that it's time coming, we're coming to a point where the community probably needs to move up a bit on the stack. Don't try to solve the same problems. Don't try to solve Kubernetes problem. Don't decide to solve infrastructure problems, move up. Then you can create more innovation and let the providers operate the service and they can add more value to us. Yeah, I think one thing that's always been interesting is that there's ethos of scratch your own itch, right? Like if you come to me with a requirement as an open source project, you better come with code, right? And I think Greg, Carl Hartman from the Linux community and Linus and I have had these conversations where years ago, I would say to them, hey, these banks, they're all using Linux and we're not getting to come and give code and scratch their own itch. And surprisingly, these guys saying back to me, we'll just take the requirements. We're just not operating those environments, right? We really want to understand. And so it sounds like just projects that have a healthier way of at least just even hearing you, even if it's not directly with code, it is something that you value there. Absolutely, absolutely. Yeah, I get it. I want to give a last word on the security topic that you just heard. This is something that I think we're just all going to have to deal with and live with and improve. And we're at the beginning of a long journey of how to fix that. What is your perspective on some of the concerns that Patrick raised? Which is, oh my gosh, is this stuff more or less secure and whatnot? What were your perspectives? I think I should touch upon one point. I do agree with the Patrick that I think it's time to move on from whether it's multiplied from the source versus proprietary. I think what I'm seeing as a bigger challenge in corporations is that in the last five, 10 years, we have sort of broken the walls between dev and ops. There's more feedback. There's more awareness of what happens in the site when things go wrong. So that feedback loop helped engineers to build better systems. That's happening very healthy trend, I see that. But when it comes to security, it's not so because teams don't understand what security is doing and security sort of has this glasshouse mentality. We are in ivory tower. We tell you what not to do, but we don't tell you why. We don't educate you. So there is that culture gap a lot and consequently teams don't make mistakes because they don't understand what the risk vectors are. I think that DevSecOps model where it's not just about tools, it's about the culture. How do you incorporate this mentality into your way of thinking and your way of doing. So you're aware of, you make better choices. You teach each other and that's neat. That needs to happen. Yeah, go ahead. So it's an interesting time for security for everyone, right? Because it's now in vogue, it's now a topic, it's now on the daily news. It's a thing that more than it used to be, even though the landscape hasn't changed, we're talking about it now. We are constantly reevaluating our practices, procedures, the culture around security within the company. I'm finding now it's a little easier to reevaluate because usually reevaluate probably means spend more money on it to some level, but now the company puts it as a number one priority. Clearly, at the C suite is securities number one, whatever we need to do to do it correctly, which makes it a lot easier than it was say two years ago. So that's giving us the bandwidth and the resources to really invest in the tools, the teams, put the right policies and procedures in place. So it's a constant evolution, I would say, and we border the different aspects of, securities, everyone's job, but then do you have people that are security in their title and balancing that, right? So you have to have the security team that has security in their title to enable and teach and get the requirements that you expect out of all of your engineers or all of your developers so that they know what to do and they know if they're winning or not. So I would say for us, it's just a constant focus of ensuring that we're continuing to double down. I think similar to what Subu said, I 100% agree. We've sort of broken down the walls between Dev and DevOps, but performance and security are two categories that are still an afterthought, right? So we had hired to have a very sophisticated CICD engine. We don't have, we hardly have any tech op staff, we hardly have, we don't have any QA. We've automated the heck out of our pipeline. We're now starting conversations about not just functionality, but also plugging in tools in the right places to make sure that in our CICD pipeline, we're measuring for performance and security. Today we evaluated in production that our third party software that's running, that's evaluating it, but to what Patrick said, we need to put that in place in our CICD pipeline. What I hear is unanimous consensus here on DevSecOps. I thought we were just talking about DevOps the other day. That was a new thing. There's already it's ruined. One of the dynamics around that is, so I like to say that we DevOps hard over the last like three years or so and when we're delivering product to market at crazy fast speeds compared to what we used to be. But part of that evolution is putting control and autonomy out to the edges, right? To thousands of developers. And then when you start the security conversation, it's usually the mindset is, well, we need to centralize the control. So it's like, where is the control? And when you give autonomy, you have to give accountability responsibility with it. You don't get autonomy without those. And that's where the DevOps stuff got a little off the rails, I think for most companies is here's the autonomy without the rigor behind it. All right, so next year, we are at all week, I've been hearing security is everybody's responsibility. DevSecOps, we have to integrate this in the process. We have to have a culture around this. I mean, next year, let's check in on how much progress we've made on actually doing that. I think that would be an interesting thing in retrospect if we look back on. So I wanna thank all of you for coming and sharing your insights and let's give them a quick round of applause. Thank you. Thank you. Thank you. Thank you.