 From around the globe, it's theCUBE with coverage of KubeCon and CloudNativeCon North America 2020 virtual, brought to you by Red Hat, the CloudNative Computing Foundation and Ecosystem Partners. Hey, welcome back, everybody. Jeff Frick here with theCUBE, coming to you from our Palo Alto studios with our ongoing coverage of KubeCon, CloudNativeCon North America 2020 virtual. It's virtual like everything else that we're doing in 2020, but we're really excited by our next guest. We're going to dive into a company that you probably know a little bit on the surface, but probably don't know a lot of the stuff that's going on behind the surface. So we're really excited to have our next guest. He is Jay Chakrabarty. He is the Director of Engineering for Core Infrastructure at Spotify. Jay, great to see you. Great to be here with you today. So as a longstanding Spotify fan and customer and premium customer and family playing customer, just so there's no question, I'm a big fan. The infrastructure to deliver what I want to hear, basically any song from the entire world it seems like. I don't know what the actual percentage of every published song you guys have. You know, kind of at my fingertips, searchable, available now to listen to is an amazing accomplishment. I can't imagine how big and significant and complicated the infrastructure you guys must be managing. And not only that, but kind of the New York growth over the last several years. So first off, just talk a little bit about Spotify scale. How do you guys think about it? Is there some things that you can share to help people really understand? You know, some of the big iron that's behind giving me the songs I want to hear. Absolutely. And thank you for the opportunity to let me talk about this. So it's a, as you say, it's a pretty mammoth project to be able to deliver just about any song that's in the world or now any podcast that you might want to listen to hundreds of millions of fans and also enable creators to be able to share their content with the consumers who are interested in consuming that content. So some of the metrics that go behind us are we have thousands of microservices running in production. We were one of the early adopters of microservices at scale and continue to build on that foundation with early entrance to dopper eyes. This is, and now of course, largely on Kubernetes. We also have thousands of data pipelines, hundreds of websites as well as micro app features. And we're doing about 20,000 deployments a day to give you kind of a scale of how fast things are changing. And for us, speed is a great virtue as we're testing out features doing ABE tests and trying to roll out the next best thing for the audio network. It's amazing. And I'm curious in terms of execution on the business side. I mean, clearly you're in many, many countries, you know, you're global. Are all the licensing agreements for the music different by country? Are you just like super micromanaging, you know, kind of the revenue streams and the licensing by GEO? Or is that just as complex as it feels like it might be? Or is there some simplicity or some scale that you can bring to bring a little bit of clarification there? Yeah, so that is an area of complexity as well. So, you know, licensing across the broad set of content that we have as well as the number of publishers and creators that we have to make sure that everything is well accounted for is also kind of a source of complexity in our organizational makeup. And then the piece that I don't think a lot of people know is you guys are huge consumers and contributors back to open source. And clearly we're here at KubeCon, CloudNativeCon. You've talked already about Kubernetes and containers. But I wonder before we get into some of the specifics, if you can talk about philosophically the role of open source and why, you know, you guys are such a big open source company versus kind of back in the old days when you would have a lot of proprietary technology that you would try to develop and keep in-house as part of the secret sauce. Yeah, thank you for that question. So philosophically, we are big proponents of open source. We believe in giving back to the community. We believe that when we as a community come together to solve these problems at scale, the end result is much better than if we were to try it alone. If any one company were to try it alone. So some of the projects that we've contributed or invested a lot of time in are Envoy, for example, which we use to power our perimeter at Spotify or Kubernetes, which we use for deployment purposes as many companies do. But there are also a number of other open source projects that we're committing to. So for example, with Cloud Bigtable, we have produced an auto scaler that's now fairly widely used to be able to manage costs better with Cloud Bigtable. We've also invested in a open source time series database called Heroic to manage millions of data points for a metrics platform at scale. So those are just a few examples, but philosophically, we believe this isn't something that we wanna do alone and we wanna leverage and do this together with the community. Right. Another one that you did mention there, what you've talked about and wanna dig into is backstage. And as you mentioned, you have a lot of developer teams working on a lot of projects. Like I saw a statistic maybe in GitHub of the number of GitHub projects you guys are working on. It's a lot. So what is backstage all about? Give us a story there. Yeah, so it's Spotify. We have almost somewhere around 500 engineering teams. And so you can think about backstage as kind of like a central nervous system to be able to help engineers interface across the wide landscape that is Spotify's engineering ecosystem. So if you're an engineer, you can go into backstage and you can manage your services, your data pipelines, your micro features. You can see what other teams are doing, what the organizational structure is. You can get recommendations and insights on your tech health. So you can see where you might need to invest more time and get some recommendations on how to get back to the blessed stack. So it's really a one-stop developer portal that engineers spend the bulk of their time in today. We open sourced it earlier this year and we've been absolutely thrilled with the response we've gotten thus far, a number of companies have already started using it and contributing back. So we've seen a lot of contributions coming back to backstage, which is of course one of the ideas to be able to get some of the great ideas on backstage. So we're really excited about that. And specifically within backstage, something that my team has just released into the open is a product called cost insights. So one of the problems that we were dealing with at Spotify is how do we sustainably look at cloud costs that do it in a way that isn't like a compliance exercise, isn't a focus on traditional top-down cost controls, but really taps into developers innate desire to work on optimization. Because all of us who come from an engineering background know that optimization is fun. At the same time, premature optimization is the root of all evil as the saying goes. And so what we've done within our cost insights product and backstage is really tried to find a good balance between engineering love for optimization and letting people know what are the areas where cloud spend really matters. So if making an investment here isn't gonna move the needle for us, we let people know that this isn't worth your time to worry about. So let me unpack, you touched on a couple of things. First off, you talked about it gives you an assessment of your engineering health. So does that mean that it's kind of a compliance within a standard? Is that looking for, I guess not quite red flags yet, but yellow flags of things that are known potential issues down the road? Is it tapping into maybe higher cost services or microservices versus less that maybe there's a less expensive way? So how do you define health and how do you keep track of people getting away from health and then steering them back to being more healthy? Yeah, that's a great question. So we have this concept at Spotify called Golden State, which is a reflection of how far away are you from all of the blessed frameworks, libraries that we recommend to engineers. And the way we think about Golden State is there ought to be clear value ads to going to a new service, a new library version. And so the way do we try to express it is, unless of course there's a kind of a direct security concern and there aren't really too many ways to get around that, but we really try to preserve engineering autonomy and say, if you go to this new framework, for example, you're gonna save this much time on average. So the recommendations that you'll find there are gonna be highly specific. So for example, if you adopt an auto scaler for Bigtable, you're gonna save this much time and spend this much less. That's in general how we phrase these things. Okay, and then on the cost insights, I mean clearly when a dev is working on a new feature or new experimenting maybe with a bunch of new features and you're setting up multiple A, B testing, this and that, are they or are they not really worrying about cost at the front end of that or is really kind of the cost optimization and you mentioned, don't optimize too early. Does that come kind of after the fact and after you've moved some new things into production they have potential and now we do maybe a second order kind of analysis of the appropriateness of that feature because I imagine if you're just trying to come up with new features and exploring and trying new things, not really worrying about the cloud bill, right? You're just trying to get some new feature functionality and make sure you don't have too many bugs and make sure you're gonna get some good client value and some new customer experience. Yeah, yeah, and we agree with that perspective. So we think about the world in terms of startups, scale-ups and mature businesses as Spotify. So there are a lot of teams who are experimenting with new ideas that fall into the startup category and by and large they are not gonna be worrying about costs. That being said, we as infrastructure teams have the notice on us to think about how do we provide shared services and frameworks that abstract away a lot of these questions around how do you properly manage your costs. So that is on us as infrastructure teams but really our perspective is for startups to move as quickly as they can and really if that's an idea that's viable and you get to what we call the scale-up stage or you get to the mature business stage where it really is a core part of our business, then that's where you might start to get some nudges or recommendations and cost insights. So interesting. So I'd love to, your background, you came from finance services and trading. We're clearly speed matters, accuracy matters. You know, that's, I mean basically financial services is a software game at this stage of the game and it's a speed game. And I saw another interesting video getting ready for this. I think it was with Gustav Soderstrom talking about the competitive advantage of the early days really being speed and speed to return a result and speed to start that stream and it just struck me very much like, you know, the early days of Google which was that was their whole speed thing and they even told you how fast you got to return on your search. When you're thinking about optimizing now with the huge suite of features and functionalities that you have, how do you think about speed? Is it still speed number one? How has kind of the priority changed and what are some of the design priorities that when things go from experiment to start to be into the scale realm and hopefully be successful in production that need to be thought about and potentially rank ordered in the proper way? Yeah. Yeah, it's a great question. And so, you know, I'll just refer to Daniel X quote around this which is we aim to fail faster than anyone else and so for us as a company and with our growth trajectory and investing in the areas that we are looking to invest into, it's still absolutely critical that we move fast, that we get the ideas of the startup phase out to be vetted and validated if we can go to the next phase to the scale up phase. So I see that just as important today, if not more than when I first joined Spotify, you know over four years ago at this point and regarding financial services, there are certainly, you know, touch points in terms of the amount of data that we're processing and the scale of technology that it requires to process that kind of data. But one of the things that I really love about Spotify, of course is that we get to move fast which is sometimes of course going to be a lot more difficult when you're talking about the financial service arena and various compliance bodies that are overseeing any changes that you might make. Yeah, you guys are running a little bit ahead of the regs I think which is pretty typical in the music business, Napster was running a little bit ahead of the regs and then we saw the evolution with the iTunes and then you guys really nailing the streaming service really for the first time and opening up this new consumption model. And I wonder if you could talk about kind of keeping the customer experience first and making sure that that's a positive thing. I can't help but think of the Netflix experience where they spend so much time on people's interaction with the application to get them to try new things, the recommendation engine, such an important piece of the puzzle. And I think what you guys have really nailed is the discovery piece because it's one thing to be able to quickly access a favorite song and be able to listen to it but everyone loves discovery, right? And discovery is kind of an interesting process and you guys have taken a really scientific approach in terms of cataloging music and different attributes of music and then using those to help drive the recommendation engine. I wonder if you can share kind of your thoughts in terms of being ultimately driven by the customer experience and their interaction with the application and these things called in a music or podcast which is such a very personal thing to interact with. Yeah, so from the perspective of core infrastructure and Spotify, our goal is to really enable the scale in which we are processing the amount of audio content that goes through our system. And so podcast of course is a new category that wasn't there when I originally joined Spotify but it's really to provide a platform so these experiments can be done seamlessly so we can have different ways of looking at discovery, looking at user segmentation and being able to come up with new ways that are gonna be compelling to our customers. So that's very exciting and fulfilling for us to be able to provide that platform by which our sister teams can iterate very quickly knowing that they have the guardrails which in our on-premise days at times was a struggle and where we're in a very different place now. Yeah, so last question before I let you go, we're at KubeCon, CloudNativeCon. And this is an interesting thing that I always think about when you're managing engineering teams that are heavily open source participants. And it's such a big piece now of a lot of engineers' motivation to be active participants in open source and to show their work to others outside the company but at the same time they have to get company work done. So I just wonder if you could share your perspective of how do you manage open source contributions? How do you keep them working on company projects but also make sure that you allocate time and priorities to open source contributions because that is a really important piece of the motivation for a lot of engineers. It's not just working for the company and getting paid at the end of every two weeks. Yeah, it's a key motivation as you say and it's key to our recruiting strategy and also how we think about retaining engineers that's Spotify. So there are different mechanisms that we use and there's a lot of focus that's Spotify on coming up with development plans for engineers that actually make sense. So I would say that all the way from the oft quoted 20% time is something that you might hear at Spotify where you have engineers who are working on open source 20% of the time or you might see a variety of customized options depending on who the engineer is, where they want to grow. And really I think the key here is providing the right support structures. So even if you have the time, are you getting the mentorship? Are you getting the right kind of support system so you know how to connect with the community? And so you have other like-minded people who are bouncing ideas and you don't feel like you're doing it yourself. So that's something that I feel really excited about that we've grown those support structures over the last few years as we've also been very intentional about giving engineers time to work on open source. And you give them as much as 20% I'd never heard that before. Yeah, and some cases some, I mean, if that is what, where an engineer really wants to focus and grow, there are a number of folks at Spotify who are spending up to 20% of their time on open source. Wow, that's amazing. That is a, that's a, it's just, it's such a great commitment for the company to the engineer if that's their priority and then everyone's going to benefit from it, both the engineer, the company, as well as the community. So really a forward-looking, you know, point of view to take that long-term view versus the, you know, maybe we should only give them 10%. We're losing 10% of their time working on a project. So that is super, super progressive and I'm sure you must be seeing great ROI on it or you wouldn't continue to be such huge proponents of open source and such huge contributors back. So that's a great story. Yeah, terrific. I mean, you know, we, we want those contributions to be in line with where we're growing as a company and we see a lot of opportunities where that is happening. So like Envoy or Kubernetes, just to name a couple of examples where folks have devoted time in those areas. Well, Jay, thanks for, thanks for sharing some of the, the story behind the scenes, you know, again, household name, what, what a tremendous success story and, and, you know, I'm going to be a customer. So definitely a customer though, no, no doubt about it. So thank you for your contributions. Congrats to the team and, and really love the story of how you guys are contributing back and doing a lot more than just making great music available to us all and a great channel for, for creators to get their stuff out there. So thanks again. Thanks so much for your time. I really appreciate it. All right. He's Jay, I'm Jeff. You're watching theCUBE's continuing coverage of KubeCon CloudNativeCon North America 2020. Thanks for watching. We'll see you next time.