 Live from Boston, Massachusetts, it's theCUBE, covering OpenStack Summit 2017, brought to you by the OpenStack Foundation, Red Hat, and additional ecosystem support. Welcome back, I'm Stu Miniman joined by my co-host for the week, John Troyer. We've been talking this week about how OpenStack, there's real clouds, there's real deployments, and happy to welcome to the program two people that have done this with Adobe Advertising Cloud. We have Joseph Sandoval, who's the engineering manager at Adobe Advertising Cloud, and Nicholas Brose, who's director of operations engineering. Gentlemen, thanks so much for joining us. Thank you for letting us join. Nicholas, I'm sorry, Joseph, we actually had you on the program at the Silicon Valley OpenStack days a little while ago. Fresh from our audience though, a little bit, your background, how OpenStack fits in with your role and what you do. Sure, now I've been in a long time in the OpenStack community. At that time when I was at the Silicon Valley event, I was with Lithium Technologies, so we also were an OpenStack user, but we were also kind of going through some transformation. I think I would say we kind of really pushed the Kubernetes button for the community at that time, so I think it kind of got a little rep about being kind of like an agitator in this community to try to make the product really work for people who are actually consuming it. Right, so not only have you deployed OpenStack, you've done it at two different jobs now already. Yes. But people think we're still so early there, but we're already seeing that progression. All right, Joseph, a little bit of background yourself, what brought you to the current role in interaction with OpenStack? Yeah, so it's Nicolas. I'm sorry, yeah, Nicolas, I'm sorry, yeah. It's okay. So I come from this startup company called Thermovals that got acquired by Adobe last year. And one of our challenges as a startup is to be able to scale our cloud infrastructure and our infrastructure in general. And we were an e-view user of public cloud at that time. But over the year, we faced like multiple challenge, not only the cost challenge, where it gets like easily out of control with public cloud, but also technical challenge. We were like in an hyper-gross environment with a very lean team operation. So we had to figure out where we can scale some of our technology and some of our platform and some of our technical prime while still having a reasonable cost control. And so we started to look at different cloud solution at the time that was like a Caliptus cloud stack, Open Nebula, and we tried, we investigated all these things and took some time to figure out what was the solution. And we moved quickly to OpenStack and start to implement and get like for a longer couple of year journey to implement that and scale our infrastructure. Stu, I want to point out something in that story, at least what I took away from it. Usually when you have a problem state of a lean team and you're trying to hyper-growth. It's scaling. It's scaling. The answer is public cloud. Oh, we'll just go to the public cloud that'll solve all that problem. You chose a different way and chose a different architecture. Correct. Anything that brought you to that decision? Yes, so there was a few factors. The first one was like, or cause growth on public cloud was going faster than revenue in some way. So that doesn't line up. You need to have a story that make more sense. And the second one was really like technical. We had like some very specific challenge where we're in the real time bidding advertising. So we have huge amount of traffic. We do 100 of billions of HTTP requests a day on our platform. All of those need to be answered in a few milliseconds. So the proximity of our partner, you can only see that as a smaller stock exchange for advertising. So we need to be close to our partner. So all this auction process is happening very quickly. And we have to store a huge amount of data. Any of the solution you will find on the public cloud will end up having like 50 millisecond latency, 100 millisecond latency. That doesn't necessarily fit our use case. Yeah. Joseph, maybe you can bring us inside the architecture a little bit. Sure. You know, talk about, look, public cloud isn't simple. Obviously costs people, you know, we understand that. And there's the debate as to where those pieces fit. But you know, open stack, speak a little bit to, you know, how it is to put that together. Simplicity is not usually what we hear when we talk, but you know, what worked, what didn't work. What did you have to kind of customize to kind of get things working? You know, I think the one thing as just, you know, coming through like, you know, two different implementations is that, I mean, yeah, there is complexity. And what I really got out of this was that, you know, you really just have to really consume the things that you need. So we've been very, you know, lean about the APIs that we consume. What services that we think are meaningful to our business instead of like really taking all as a service type parts of this framework, we really narrowed it down to what matched our business requirements. I think as well, it's kind of like how you're consuming. And I think if you notice the keynote on Monday, we all of a sudden we're seeing this new pivot of like, let us manage your cloud. And it still kind of speaks to some of the challenges that, you know, the end users of open stack have. And I think the part that's really important for anyone that's really going on this journey is that, you know, it's how you decide to consume it. Like, can you start really running it within like a CI CD model so that you're really getting into that really, that DevOps aspect of it? Even within Amazon, I think in my journey, that's one thing that I think a lot of people miss is when they try to lift and shift, like they want to race to the public cloud. You're going to still be challenged because you haven't really fundamentally changed how you're consuming the cloud product. You're not making yourself cloud native. And I think in my journey, I've made those same mistakes. I've learned from it. And now I'm actually really realizing that it's almost bigger than open stack. It's almost like how as a business you operate and how your teams fundamentally, you know, build their tools and how they kind of like make open source a true strategy. I'd love to hear about the applications that you're using in this environment. We heard in some of the keynotes, some of the users, you know, rapid move from where they started to adding applications. You mentioned cloud native. What are the class of applications? What percentage of your business runs on that? Sure. Yeah, and so the code name what we've given our, you know, our platform is cloud mogul. And really it comprises bare metal and primarily open stack. And yet we still also use Amazon. So, you know, we have all different frameworks in there depending on the type of, you know, workload that's there. As far as like open stack specifically, we really just consume the core. It's compute, storage and network. Storage is probably a little bit secondary force. The way we have designed our platform. Network is the really key thing. And as Nicholas mentioned earlier, I mean, that's the thing that in Amazon you'll see great choices for compute, great choices for memory, but go try to find affordable network, you know, intensive like, you know, instance. And that's what, you know, we have decided why we were in the data center. So we really have stuck in really with just kind of like the core open stack services. Currently our developers are looking at now, you know, rolling out Kubernetes. They're kind of doing it in more of a, you know, dev, POC. And as well as we're trying to balance out like the broader Adobe strategies are like, they want to move to multi-cloud. They want to use Azure. So there's quite a bit that we're trying to, you know, consume, but with the lean team, we have to really be judicious about what we decide to roll in. Nicholas, can you comment maybe on the application? Do you mention some of the clots? They, you know, cost compliance capabilities. Does that resonate with you? And how do you, how do you choose between the public and? I think it's where like, to get back to this lean operation, it really drive like summer for effort. We are not like a technology company in some way. I mean, we are building software. We are building certain solution. You know, our goal is to deliver like an advertising solution and to end solution that serve our customer. So we are not here to be like a storage solution for OpenStack or a compute solution for OpenStack or a public cloud. So we really had to focus on what is serving or best use case or solve our own problem. That's where we had like to really look at like, cutting the fat in some way on OpenStack and really just looking at what is going to be the best use case for us. So we leverage like OpenStack for most of our bidding system and manage all the scheduling of all the VMs. But we also like integrate that very easily with like a flat network design with our bare metal. So we are able to really like get the best of both worlds between like bare metal, OpenStack and virtualization. And now we are also like implementing like cloud bursting and be able to offload summer for workload back to Amazon or any other. We are starting to look at other like cloud provider but already trying to leverage like what's the best and consider like all the challenge we have. Can you give us a little insight that cloud bursting is a term that gets attention because data is tough to move, where the application lives. Is that container Kubernetes stuff that you're doing? Expand that a little for us. So it is definitely challenging. It's not something that, at the end we got a very quick iteration and we have been able to leverage it easily for us because we had like a very simple design on the way we were managing or cloud environment and OpenStack deployment. So it allows us to very easily integrate, have a direct connect to a VPC on Amazon and just offload some of the compute resources on this VPC. The challenge we had to learn is really to understand the workload and that was an iteration we already did when we moved to back to in-house, understanding like the network traffic you are getting, understanding like the back and forth between your back and your front and that's something you don't really see or understand easily on public cloud. When you move back in-house, then you start to see the bottlenecks and you start to learn about what is really your workload and we had to do this again with like cloud bursting, like okay, what kind of back and forth are going between our compute services and all the back end services that it needs to access and latency being very critical for us. We had to really measure all that. Yeah, you never know until you try it, right? You crawl, walk, run. Hey Joseph, you talked about CICD and rate of change. I'm kind of curious how you're seeing the rate of change of your infrastructure stack, so open stack versus the, you said you're now kind of experimenting with Kubernetes and containers and a lot of talk about containers here at the show. For me, it's becoming a little more clear where that in the architectural pie, layer cake that, top pie, layer cake that that fits in. Can you talk about rate of change? Are you looking at does your infrastructure need to change at the same rate as the application on top of it or how are you all looking at it? You know, just being in this journey, the one thing that I've really taken away and I was one guy on my team when I was at Lithium where he would always talk about like really meeting your developers where they're at. And yes, there's so much change and you have to really kind of balance it. And you know, some of these companies we've been with, we have some, you know, some software stacks that are almost a decade old. They're just not made with cloud nativeness in mind. And that's where, you know, I've always been a really like, let's move forward. Like, and I was one of the early individuals saying, you know, I was at open stack prog and we were doing Kubernetes under the control plane. In hindsight, I was like, well, it was a little kind of premature. It was almost a little reckless. But I think the thing that, you know, I'm trying to do now is really just try to leverage like where our products at. Can I help evolve the platform so that, yes, are we 12 factor? Can we get there? You know, we have big data kind of workloads. Like, how do we, how do we like start taking, you know, frameworks that allow us so that, you know, we can be in this multi cloud world. So I think there is a challenge. You know, you're hearing all these new great things that are happening. You know, you're coming to these summits and you're getting all this hype, but then you really got to walk away and I just kind of do that sniff test of testing something out to see like, is it really ready? And especially with, you know, where we're at in an enterprise, you know, we really have to map to like security compliance. And so I think those are some of the gates that we kind of were challenged with, as well as like, you know, is the workload that we're bringing in, have we, you know, adapted it enough so that we can really kind of push where we're doing. Cause I'd love to see us get to the point where we have the, you know, the frameworks of, you know, containers and Kubernetes, but not everything for us can get there. You know, so like on the edge, we're doing billions of requests per second. Bare metal is the key thing for us and we're running HA proxy on the edge. So the key thing for us is like, run it as code. Let's count how much can we do to get this so that we can fully automate this and make it repeatable. And I think that's kind of the core ethos for the team. You talked about coming to different summit, summits over the years and kind of the sniff test. What's the mood of the attendees here at the OpenStack Summit here in Boston this year in 2017? And is that different from previous years? You know, I think we've seen kind of like some interesting ebb and flows. I think when I was in Barcelona, it was a little, it was definitely different. I was kind of like surprised that some of these just felt like there was a little bit less energy. You know, Austin, I thought was tremendous. It was a great event and I kind of feel like I think there's a little bit more pragmatism that's set in, which I think is really healthy and a sign of maturity that, you know, people are really kind of like understanding instead of getting caught up in like the cloud hype, you know, public versus private and all these things. I think now we're starting to see a more mature audience. I think OpenStack Foundation and the community has also kind of adapted as well. I know they try to be everything for everybody under the cloud and the data center. And I think now we're actually seeing a more healthy approach. So for me, I think there's still a lot of energy there. Maybe it's getting a little boring, which to me in my world, that's a good thing. Yeah, yeah. Nicholas, I'm curious, do you either at the show or at other events, how are you working with your peers in the industry to understand that kind of hybrid, multi-cloud model and sort that out, you know, resources you go to, conversations you have, you know, how do you, you know, create that, you know, learning? So for us, it's pretty come from the culture that came from the startup to Mughal that got a prayer where we really focus on the customer and what the end goal of what we are trying to build we're not necessarily driven by the technology itself. We really try to leverage technology to solve our problem. We have a lot of geek in our team and that's really drive some of our discussion for sure. But we're really more trying to look at how we drive the product forward. And that's really like most of the discussion even with our vendor, like we started a year ago to use Fastly, for example, to solve some specific problem we had where we can't have like a global footprint as much as a CDN provider and they were able to address like some of a specific use case where they can do like a synchronous logging for us. And that was solving like a specific business case for us. And every time we go like to a vendor or technology we are trying to see like, what are we trying to solve? And that's what drive most of our discussion. Joseph, sounds like you've given feedback and been on some of the leading edge of some of the activities. Is there anything you look at where you're hoping for a little bit more maturity, either open stack in general, the vendor community out there, what are you hoping to see as we mature this even further? Sure, I would say one thing about the open stack community and this was always kind of one of my early beats about it. It felt so vendor centric and vendor influence that it just didn't really for me feel like the actual consumers, the individuals who really are using these platform are really being heard. So I think they need to kind of still kind of really force, I really listen, get that feedback from the community that what's working, what's not working. As far as what I'd love to see is, I think there's been a little bit more of like a correction I guess in a sense of like all the kind of like services that were out there, these side projects. I think that there was a little bit of messaging about like, let's all work together. Which I think it's kind of, I just kind of like kind of went a little bit, but I'm like, it's good. I'm glad that they kind of come to this recognition. I'd love to see more and more of that. But I also want to make sure that the opposite community, like stay distinct. I'm not sure if I 100% think like leveraging off the Kubernetes community is like, yes, work together. Let's make these things coexist and stuff. But when I do hear some things, we're like, hey, we should just make this service, be the back end for Kubernetes. I'm like, I don't think you've really looked at the framework of some of these APIs and how they're going to integrate in that environment. And I actually would like to see them develop distinctly, but you know, find some really friendly integration points so that, you know, me as a consumer, I can like easily use these, as we evolve and our platform evolves, I can easily kind of like start road mapping of these into our platform. Nicholas and Joseph, really appreciate you giving us the update. And we'd love to get that real practitioner viewpoint. John and I will be back with more coverage here from OpenStack 2017 in Boston. You're watching theCUBE.