 Live from Vancouver, Canada, it's theCUBE. Covering OpenStack Summit North America 2018. Brought to you by Red Hat, the OpenStack Foundation, and its ecosystem partners. Welcome back to theCUBE's coverage of OpenStack Summit 2018 in Vancouver. I'm Stu Miniman with a co-host, John Troyer. Happy to welcome back a company we've spoken to a few times at events, Patty Power Betfair. First time guests coming to us from across the pond. Dave Buckley, who is the automation engineer with Patty Power Betfair, thanks for joining us. Thank you for having me. All right, so first of all, you've been to a couple of summits and we've talked to Patty Power about OpenStack. Before we get into your, you know, specific implementation, tell us about, you know, your experience here this week and any kind of compare contrast to previous years? Yeah, so I've been very lucky and I've got to come to the previous two summits in North America. I guess what I've enjoyed this week is kind of that slight tilt towards, so away from being purely OpenStack, kind of towards this open infrastructure kind of thing, because I guess over, especially last year in Boston, like Q and Eddys was becoming a big thing. Yeah, and kind of the OpenStack Foundation becoming kind of more, not that that wasn't before, but more community based and kind of being part of the ecosystem. So yeah, I think it's been quite interesting seeing that. Yeah, not to put words in your mouth, but it was even the last year or two. It's more aware of some of the complimentary things. Adding pieces, you know, we had one of the interviews we did this week was the person who's the, you know, sick lead for the Kubernetes stuff and that sits under another foundation, things like that. Yeah, exactly. It's been quite interesting this week, I guess. So the cat containers project, which wasn't something I've been aware of before Monday morning, basically. I remember like sitting in the keynotes and they were like, you can have this container-like thing, which has all the speed of a container, but it's as secure as a VM and you're thinking like, how is that even possible? So I really enjoyed, I got to go to one of the sessions yesterday, like one of the technical introductions on that. Yeah, I always love, there's certain things where, okay, this is what I'm going to do with my schedule and turns it up, this got announced or I didn't know about this and you know, blow up my schedule, let me change everything else. Yeah, exactly. I think you always, you just can't, you have to be flexible, right? Adaptable then as the week goes on, you just go to what you think is interesting. So Dave, you and your company been working with OpenStack for quite a while. And you obviously run a system that needs to be stable, it needs to, you take care of, you know, betting and people's money, right? So that needs to be solid. But I understand you recently went through an upgrade and had some experiences talking about that. Can you talk a little bit about where you are with your OpenStack implementation and that sort of migration? Sure, so I guess it was about three years ago that it was Betfair at the time. So this was before the merger of the two companies. So Betfair starting using OpenStack. I think it was actually the last time it was here in Vancouver. So a couple of my colleagues who are kind of the technical leads at the time, Steve Armstrong and Steve Pereira, they flew out here to kind of get a feel for OpenStack, what it was, talk to people who'd had experiences with it. And I actually think that conference back then kind of was very informative of what the platform today now looks like. So some of the conversations they had there with people like, like Nuage Networks and like Arista, which were used for the switching, like the conversations they had there kind of, yeah, ended up being what we're now using today in production. I guess, yeah, over the past couple of years. So the big thing that happened obviously was this merger between Paddy Power and Betfair. And following that, they had an exercise which they called the single customer platform, which is annoyingly for like a cis-havening guy kind of like me, they was always been abbreviated to SCP, but you have to ignore that. So that was to kind of consolidate and integrate the Paddy Power and Betfair code bases and put it on a single platform, which was our OpenStack and Nuage platform. So that kind of completed in January this year. So that's live. So basically the Paddy Power Sportsbook has an entirely new website all running on OpenStack. A lot quicker and more efficient than the previous version. So that's been a real success. And as part of that, as you say, like stability is like really vital. So kind of in our business, if the site is down, we don't make any money. And if that happens during a big sporting event, you have a big problem. Yeah, do you have a metric around that? What a minute or an hour worth of downtime would do? So I guess it always depends. So the nature of our traffic is very spiky. So obviously when you have like a big, it's on a Saturday in Europe, the football, the soccer, maybe I should say, is like a very big deal. We have a global audience, football's okay. Okay, I'll stick with football, I'll stick with football. I don't want to talk about that. So things like the, yeah, the football, if you, that's, we just get peak traffic on that day. And even within a year, there's a thing called the Grand National, which is a big event in the UK, big horse racing, I guess, like the Kentucky Derby. It's kind of when we get our absolute maximum traffic in the year. So yeah, you kind of, you always need to be prepared for that. So one of the things, as you mentioned, we kind of looking to upgrade OpenStack from Kilo to Newton. So we've been on Kilo from the start. We're using Red Hat's distribution of OpenStack. So what Red Hat offer is this, they have like every three releases, I think it is, they have this long release life cycle. So that's the reason kind of we're going to Newton because we have kind of the, I think the support will go to like 2021. But if I remember it, it's Red Hat, the OpenStack platform 10. Yeah. And 13 is going to be Queens as their next one. Exactly, so I think they just, they just announced that this week, right? So I think we, at some point of the next, who knows when year or two we'd be going to Queens. Yeah, how do you determine when you make that jump and anything around the upgrade process, good and bad that you could share? Yeah, so I guess going from, we were overdue an upgrade in this case, Kilo is pretty old now. What we're lucky that we can do is because we have Newarch as like an external SDM provider. So the entire data plane is controlled by Newarch and you can kind of plug as many OpenStack as you like really into Newarch and you offload all the networking to Newarch. So what's that allowed us to do is, basically we'd have had a lot of trouble if we'd had to do an in-place upgrade. So I've actually been to one of the groups this week where people were talking about upgrades and just like all the nightmares it's caused. I know it's getting better as like, as you know the releases come out. But what we were able to do is kind of build a new, an entirely new OpenStack cloud on the side of, so we, yeah, we're not, we've turned it kind of immutable OpenStack, so your OSP7 cloud is there, we've got this new OSP10, but they're both hooking to the same networking, to the same Newarch SDM. And we have, the way our developers deploy their applications, I guess you want to see this in more detail, we've done presentations at these summits in the past, but kind of ensure every deployment we do immutable deployments as well. So every deployment will create a new subnet within Newarch and kind of do a rolling update of your VMs that are on that new subnet into like a VIP, which is kind of where the, that's where the constant is. So all the traffic's coming to that VIP and you just flip things in and out below it when you do a deployment. So what that basically means is from a developer's point of view, when they're migrating from OSP7 to OSP10, they'll essentially spin up new networks and new VMs in OSP10, and that deployment pipeline will kind of just seamlessly, everything else will stay the same because the networking doesn't change. So that kind of takes, so we don't have to have any downtime on the data plane or the control plane, which is really beneficial for us, because the way, I guess as I've just described, the way we kind of do, the developers do deployments, like we rely heavily on the OpenStack API being available. So yeah, you pay a cost and that you, so you need extra hardware to be able to do that, I guess. But yeah, we found it, it's something that's worked for us. Great. Anything else with the networking, specifically that you got all the running in terms of the load balancing or resiliency that you need to have for your apps? Yeah, so one of the things was, so it's kind of another problem that we're trying to solve with this whole project, this new OpenStack platform, is that historically that fair, as it was at the time, it would always run out of a single data center. But we kind of had, we had another site, but it was mainly kind of development environments running there. It's kind of the company thought, well, why don't we just have, we should have both DCs for resiliency, try and run things in an active, active configuration. Which is kind of fine for kind of external kind of customer-facing applications, where we kind of had an external load balancer that can point traffic between the two DCs. But then the question is, what do you do with internal apps? So this is what led us to use, so AVI networks, which is kind of a, I guess I call it a cloud-native load balancing technology. So we've been using that for kind of, to provide like GSLB for internal apps. So basically, we'll load balance traffic between the two data centers. So you kind of have it, it gets deployed within your OpenStack environment, has a really neat integration with Newarge, the Newarge SDN layer. And we'll resolve you to, whichever data center is appropriate at that time. So if you had a full data center outage, AVI should just be able to go, okay, points over there. So guys, it makes you and the networking team or the IT team into the heroes, not the villains. You're usually the people saying no, or we can't do that. I guess so, I guess so, yeah, you're fully right. It's cool technology though. I guess we're very lucky in that we kind of, we're given the opportunity by the people at the company to kind of experiment with new things. So even though we're always, we're about stability, but we're also about kind of trying to push things forward in terms of what technologies we're using. Dave, I'm curious how kind of the hybrid or multi-cloud type of environments fit into what you're doing today. Give us the update there. Yeah, so that's something very much on our radar at the moment. I guess it's, yeah, it's kind of what everybody's doing, looking to how you can have this kind of hybrid cloud model. So I think going back three years again, at that time, being like an online betting company, it's a highly regulated business, and I don't think at that point it was really possible to kind of put some of this stuff into the public cloud. It seems like things have come a long way. So it's something we're kind of looking at in the minute, at the moment, is kind of evaluating different solutions, different vendors like the Googles, AWS's and seeing, or even like some OpenStat public clouds and seeing maybe how could we migrate some workloads out into the public cloud? How do you want to do that to give us kind of bigger, more resiliency? And also, as I was saying about kind of our spiky traffic, it just makes a lot of sense to be able to say, burst out into whichever public cloud vendor, one on a Saturday when the football's on, seek to kind of deal with that peak load. So it's something that we're, yeah, we're very much looking at at the moment, but yeah, no formal decisions as of yet, unless they've done something while I've been away. With containers here at the show, right? Lots of different threads, right? Containers, edge, the open dev track and things like that. Anything else, we talked about Cata, anything else that came up that was interesting here or that you just watch in the Kubernetes and container track as well? So I guess in terms of containers, it's sitting in the key notes on Monday, you would, if you weren't watching, if you were just listening, you probably wouldn't know you were an OpenStat summit, right, because there's as much kind of Kubernetes container stuff as there is OpenStack. It's interesting, so we've kind of been doing, again, similar to kind of the public cloud conversation, it's something that's very relevant to us at the moment. We've done kind of a few proof of concept ideas, evaluating different solutions. So we have like running kind of a cube cluster ourself. Obviously we have a strong relationship with Red Hat, so that we've kind of explored using OpenShift maybe. And they've kind of the networking layer, you can kind of integrate with Nuage, which would be really cool for us. So it allows to do kind of the, all the networking and like the access control mechanisms, like as we do for virtual machines. And again, this is also something in the whole public cloud conversation is, well, if we wanted to do containers in the public cloud as well, like you have all the different offerings, would we want to like run our own in like an AWS or something, or maybe go to someone like Google where you kind of have that supported self-service model, I suppose. But yeah, at the moment it's kind of, it's at those stages. So I think Steve did a presentation on the Kubernetes stuff, like a POC we've done at the last summit. But yeah, it's still at the moment, still want to make some firm decisions about like which direction we're going to go. But a lot of the developers are very keen for this naturally and obviously like in for guys like us, also we all know the value of it. So I think in, I think at the moment, because we've had that focus on stability, we shouldn't have a period of time where we're able to kind of look at all this stuff a bit more, kind of hopefully, yeah, get some container solutions into production, which would be awesome. Dave Buckley, really appreciate you giving us the update. Love to be able to do some of those longitudinal case studies as to where you've been, where you're going, what you're thinking about. Be sure to check out theCUBE.net. You can actually search for Patty Power Betfair, see some of those previous interviews from Dave's ears, loads more interviews there, as well as all the shows we're going to be at in the future where hope you come by and say hi. For John Troyer, I'm Stu Miniman. Thanks so much for watching theCUBE.