 Live from Austin, Texas, it's theCUBE. Covering OpenStack Summit 2016, brought to you by the OpenStack Foundation and headline sponsors Red Hat and Cisco. Now, your host, Stu Miniman. Welcome back to theCUBE. I'm Stu Miniman with wikibond.com. Happy to have on one of the keynote speakers from the sessions here at the summit. It's Chris Hemmonds, who's the Director of Network Infrastructure Planning with Verizon. Thanks for joining us. And part of the solution that you've deployed is with Big Switch, and we have Kyle Forrest, who is the founder of Big Switch on the program. Kyle, thank you for joining us. Thanks, Stu. All right, guys. So first of all, Chris, I mean, give us some of the highlights, the keynote, what brought you to OpenStack and what you're doing in your solution. Sure, so we wanted to come discuss what Verizon's been doing for the past couple years, but the last 18 months in general. Deploying a large-scale NFV infrastructure for our next generation network. And we coupled that with a press release on Monday morning that's out there for everybody to see. So we're pretty excited. We're hurtling down the NFV track now, and OpenStack's key to that is Big Switch and Dell, our other partners in Red Hat from an OpenStack perspective. All right, great. Maybe you step back a second. Tell us a little bit about your role, what you do at Verizon. Verizon's a big company. There's telco, there's cloud aspects, of course, mobile, everything else. I get my monthly bill for a variety of services that I use, so what's your piece of the puzzle? Sure, so I work in our corporate technology group, so doing planning. I'm responsible for the architecture and deployment plans for the cloud for the network side of our business. So the cloud that we're going to use it to virtualize our infrastructure. So not a public-facing cloud, but an infrastructure private cloud. So like I said, we've been heads down, just executing here the last couple years, and we came out here to kind of show the world what we've been doing and spread the word about some of the good work we've done and the collaboration with our partners. All right, great. So Kyle, we've been talking since Big Switch came into creation. Big buzzword, acronym I guess I would say, is NFV. When I first started talking to Big Switch, I mean, it was all about SDN, so maybe you could help to code for our audience a little bit, that SDN, NFV distinction, Big Switch's role, and then we'll follow up some discussion of the Verizon use case. Sure. Think of NFV workloads that are running in a modern data center. Stress the network in ways that you wouldn't believe. It's not only throughput, but it's a series of optimizations that are required for different parts of the operational life cycle, and just the intense complexity of the network topology that they require underneath. You know, to take a really simple example, let's say you have two VNFs and a router in, router out, just to get 10 gigs of throughput for an NFV system, you need 60 gigs of routing capacity, and you need that routing capacity to show up in the right place at the right time. There's no way that you can do this with traditional L2, L3 networks. So while people say SDN and NFV, they're independent as technologies, but NFV really requires, it pushes the network so hard that it requires you to think very differently than traditional L2, L3 designs will let you. I think one thing that we're extremely proud of, look, the work under Chris's leadership and under his team, took a set of technologies last year that were not specific to NFV, last year there really was nothing out there, there was NFV specific on the routing switching side. And in their build, in nine months, really went from first conversation to full production, 50 racks in production network. I mean, that's a land speed record. That's a land speed record, I think, in the telco community, that's a land speed record in the NFV community, that's a land speed record in the large-scale open-stack community. And these guys delivered it. So the technical accomplishment that they did here is really something else. So Chris, I mean, this is not some solution that was pre-packaged, press a button, it drops in and everything there. You had a number of partners that you worked with on this, can you walk us through a little bit about how you kind of spec'd it out, how you chose the partners that are involved and maybe help us walk the stack a little bit about what makes up that solution. Sure, so if we rewind even a little bit further, Kyle mentioned the nine-month deployment. There was a lot of foundational work that happened before that. Verizon's been involved in the NFV since its inception. We've been doing a lot of work on virtualization, POCs, that kind of thing. When we referenced back to that kind of nine-month window, that was the, okay, we think we really, we need to go now. So we've done enough of that foundational work, we really just need to start executing now. So we started the final round of selection for the technology. And one of the things that was really key to us was we wanted to stick to the white box, the commodity hardware model, both the server and the switching layer. So when we looked around the landscape, we had a few choices in the switching capability and did a bake-off and Big Switch was there with just a spot on integration with OpenStack and really hit the mark where we wanted to be with that software hardware disaggregation and the functionality with OpenStack. So as Kyle said, we engaged, we did our initial kind of testing in the lab and then away we went. So Dell's been a great partner from a hardware perspective in both switching and compute. Red Hat's been a great partner at the OpenStack layer. We were kind of running on the bleeding edge with them the whole time, pushing, we were pushing OSP7 really hard. That's a year old now, but pushing OSP7 really hard, we needed some functionality in that. They worked really well with us to turn that around, working again with Big Switch and with Dell at the bios and firmware levels and everything that we need to do there. And then just walked it up from there and pushed it to where we had a deployable model that we then rolled out domestically into our first five data centers. So one of the shifts that we're seeing there is what does the customer do themselves versus what can I just get in the software and the platforms and everything to do? You say I want kind of white box on the server and on the networking. You talk a little bit about the skill set and the team that you have, what familiar are they have that you were kind of willing to do that, maybe give us a little bit of color as to how that went because I've talked to some people that were like, oh well this is trivial, I just take out what I had forever and I throw in new stuff and oh wait, I don't know what I'm doing and I shot myself in the foot. So what was your experience like and how? So we drew experience from around our business, server experts and networking experts and put them on a concentrated team to get started on this. One of the decisions we made on this initial deployment was we didn't go full white box for some of the concerns you just listed that you have to build an organization to support a true white box deployment. So what we did is we chose the Dell servers to give us white box like pricing and performance but we also still have the backstop of support through Dell there. So it allows us to build that white box support internally while we still have the kind of more traditional method there to use as long as we need it. Similarly with the OpenStack side, we partnered with Red Hat to get a supported release of OpenStack instead of trying to do it all ourselves because as everyone knows that takes a huge commitment in developers and internal capability that quite frankly is still being built up around the world. So we want to follow up question for you. Some concerns we hear out there is scale and performance. Can you talk, the network, I'm sorry, in the storage side my understanding is you're using with the Red Hat OpenStack, you're using Ceph and then of course you've got Big Switch. Can you speak to kind of your experiences, kind of meet expectations, any kind of adjustments you need to do along the way? Yeah, I mean as with any project where you're moving as fast as we are, there's adjustments along the way. So we've been pleased with our initial selections. As Kyle mentioned earlier, as you really push the envelope with some of the NFV applications, some of the Ceph's not going to satisfy all of our needs. We chose it because it's a very good solution for a broad swath down the middle of our needs for storage. There's going to be edge cases where we're going to have to do other things to address some extreme workloads, but for now we're pretty pleased with the performance we've gotten so far from what we've selected. And Kyle, how much is Verizon kind of typical? How much are they kind of bleeding edge is what they're doing? What are you seeing across this use case in general? Well look, these guys are absolutely industry leaders. And I think they're the first to real production at this scale. For us it's, as a company, especially as a startup, it's absolutely critical for us to work with the leaders in the industry. They're the ones who are going to consistently be six months, 12 months, 18 months ahead of the pack. They're the ones who are going to take best of read technologies and form them and meld them to exactly what they need. We started this kind of, only work with the best type of strategy that we presented to our board. I remember five years ago when we were starting a company said in every vertical and every geography we'd go into we really want to be working with the top engineering teams. And that's exactly what this was. The top engineering teams push us very, very hard. Guys, Chris knows well, in January, right before we started the engagement, we had identified six key areas that NFE requires that traditional L2, L3 can't deliver. Because of this engagement, by last summer we had 25. And I think by November we had delivered like 23 of the 25 and got through the rest in January. So it was just one of these very, very intense experiences. But this is what you sign up for when you work with the top engineering teams out there. And this is what we were looking for. So I think this, no, this to me is what success looks like. So Chris, one of the things that gets discussed a lot here at the OpenStack Summit, not just this year but previous summits is technology. There's hard things but sometimes it's the easiest part. It's one part's technology, nine parts kind of the operations and people. What's your experience been with the solution? Yeah, I would say you could slice and dice the percentages any way you want. But yeah, I would say the technology was challenging, the bleeding edge, fixing things. Couple of times we were on late night calls and had a patch the next day from the guys at Big Switch or whatever to fix some critical thing to keep us moving towards our goals. But to speak to the people part, it really has been a challenge, a good challenge because we're changing fundamentally the way we run our telco networks. So it's a change at every level. It's a change at the kind of hardware we're running, the kind of software we're running, the kind of skill sets that are needed. So it's been a challenge that Verizon's met head on. We've done a lot of training, we've done a lot of socialization. Even so, it's still when you're changing a big organization with a pretty significant technology change like this, it's challenging, but we're working through. I mean, we've got a history of doing this. We've done it with analog to digital and 3G to 4G and just, you know, Vulti and everything else we've done over the years. So this is just another step in the technology evolution for Verizon, but it's definitely been a neat ride. All right. How far up the management track a chain was involved in this project and anything you could speak to about what this will mean to your business going forward? Sure. So at the highest levels, our CTO was intimately involved in what we were doing here. So it was sponsored from the highest levels. I mean, the only way we could really do a change across the entire business. This isn't just like Verizon Wireless or Verizon Telecom. This was Verizon as a whole, opting to change our infrastructure. So that was sponsored at the highest levels. The challenges or the changes that we're hoping to realize are the benefits we're hoping to realize are kind of the cliche, you know, capex and OPEX reductions. We really do think we're going to see capex reductions. The OPEX reductions will come with kind of the simplification of our network at the hardware level, the generic deployment of hardware, simplification of spares, just a lot of operational efficiencies we get from it. And then the thing probably we're most excited about is that making our entire network programmable will allow us to launch things faster, quicker time to market, faster reaction to demands on the network, like Super Bowl, the Pope visits the city, something like that where you need capacity instantaneously, things like that. We're really excited about the opportunities that brings us to our network. So Kyle, this solution was done, as Chris mentioned, partnership. Can you just give us some of the learnings you had on that and how, you know, I think you're at OpenStack. You know, the word collaboration is thrown around a lot, but give some insight into that. Sure. There are a few. I mean, first I think for NFV, the tie between the orchestration system and the network system, to get the performance that you need there, that has to be incredibly tight. To give you a sense through most of the project, we were doing weekly calls and then near releases. We're actually doing daily calls back and forth with the Red Hat team, right? To coordinate the features back and forth. We're doing very, very regular calls with the Dell team to make sure that hardware underneath was kind of exactly what the software was expecting. So the coordination there was critically important. I think there are a whole series of, I think Verizon organizational things, as Chris mentioned, where these guys have a template for success. It takes a real specific work chart. That's something that we look for in the end users that we work with. And these guys showed the successful path there that works. So I think the vendor alignment's really important. I think that the end user organizational alignment is really important. And then I think the really, really smart thing that in my mind, that this engagement did, that we're starting to see now other folks who are six, 12 months behind Verizon, sort of copy a bit, is the selection of which workloads are first, second, and third. In my mind, really drives success. Putting the IMS core as the first virtualized thing on to me is a real dangerous path compared to taking other things that you can put on first, second, third, and start layering up the infrastructure. It's a recipe for success here. So Chris, I want to give you the final word. In hindsight, you guys broke a lot of glass. You chewed some glass along the way. The benefit of hindsight, what would you tell your peers? If you had to do it all again over again, what did you wish you knew at the beginning or might have done a little different? Sure, I think you got to have, you touched on it earlier, in a large organization, you have to have the sponsorship, you have to have the leadership, the vision to tackle something like this, I think you got to pick partners that are aligned with the way you want to do things. We selected our partners based on their technical capabilities, but also based on the cultures of the companies. We wanted to know that they were going to jump in this boat and row as hard as we were rowing in the same direction, and I think that was key. And then just preparing yourself and the organization around you for the kind of change that's going to happen. Like no matter how prepared you think you are for the change, it's bigger than you think it's going to be. Just making sure that you know, prepare yourself and your people and your management. I told my boss every day, I said, look, this thing's going to fail and we're going to fix it quick. So just get used to that right now because the no failures thing doesn't apply in this realm. We're going fast, we're designing for failure and we're going to fix those failures as quickly as we can. So just kind of, it's a reorientation for everybody but it's a lot of fun once you get it moving. Yeah, well, Chris and Kyle, thank you so much for joining. Absolutely, moving faster, understanding, you know, how failure fits in and into the whole environment is something that everybody's going through at this point. So we'll be back with lots more coverage here from OpenStack Summit 2016 in Austin, Texas. You're watching The Cube.