 Thanks everybody for coming This is our second breakout session of the day. I'm going to take 10 seconds just to say welcome. Thanks for coming And introduce our panel moderator the legendary Nikki Acosta Acosta, Acosta, excuse me. You think after working with her for a year. I'd figure this out by now Who's gonna who's got a great panel? I'm gonna let her introduce her panelists and And Just a reminder we are doing a drawing at the end of the each session So be sure to get your drawing cards and we'll do be doing a drawing for an iPad mini at the end of the session and With that Nikki I take I give it to you. Thank you And I'm gonna tell my teammates if you're gonna post a picture of me on Twitter or something make sure I don't do the Double chin thing. Okay. Hi everybody. I'm Nikki Acosta. Are we recording in the back? We're good solid awesome, so Yes, it's been an interesting couple of years. I'm so excited to have some panelists today that are Customers of Cisco OpenStack private cloud Which interchangeably I have a slide that says this but you may hear the term metacloud and Cisco OpenStack private cloud a little bit During the presentation so before we jump into the panel. I just wanted to take a couple minutes and kind of explain What Cisco OpenStack private cloud is There's that slide about the interchangeability of terms so I Kind of think of metacloud is kind of having two parts and one part is our distribution of OpenStack that we deliver in a platform as a service and Basically, we take OpenStack. We curate it We test it and then we add some extra features in there and I'd like to say that Metacloud is really a platform that is delivered To it teams but really for developers So if we we have some extra features and functionality that are really geared towards the operators To give them extra visibility and control within their cloud infrastructure. We do that on an extensibility model So everything we do uses OpenStack APIs, but we have some extra features For operators to be able to troubleshoot applications and kind of oversee the health of their cloud as a whole The other part of that is what we call our advanced operational support So this is the as-a-service part and the best way to explain this I've found is to actually show some of the tickets that we generate so here's an example of a ticket where a hypervisor had a critical memory error and so not only did we we have about 10,000 checks happening per minute across all of our customers availability zones, but we'll say hey You know your hypervisor, you know might be oversubscribed or there's an error We're gonna go ahead and move the VMs for you and we work with our customers to set up these processes and Actually, not only let them know there's a problem but resolve the problem for them and our operators love this and our developers probably don't even know that it happens But it's really part of the magic behind what we do Here's another example where we did an upgrade and so we basically said hey We've got some upgrades for you there might be an interruption of orchestration But we're not gonna have any issues on the running instances if you want us to reschedule Let us know and so we actually do in place upgrades in addition to all the patching and all the remote Operations and engineering and we release about every six to eight weeks So every six to eight weeks we're taking that open stack code. We're testing or making sure it's absolutely ready for production use And then we go ahead and roll that into the environment so Paris we had tapjoy on stage and that was a big use case Story that was covered by the super user organization If you haven't seen that check it out. It's a big data workload. That was Highly publicized giga ohm covered it and a few other folks. They're still with us and they're still growing like crazy And we expect to to hear more from tapjoy at Cisco live if any of you are going to be there And now for the fun part so I'm really excited to have these panelists I think their stories couldn't be any more different from each other And they're kind of all in various stages of deployment at this stage But I'm going to go ahead and let y'all introduce yourself Rob will start with you I'm Rob Lindros. I work for sprint telecom design engineer in SME I'm Ilan Rabinovich. I work with the Viola Manage our infrastructure and site reliability engineering teams Mattel Patel. I manage the technical operations at Shutterfly Rafiq Ardion. I was CTO at Metacloud before coming in and now Basically handling the same role for Cisco. So I run engineering and operation Awesome, and so let's start with our first question, which is probably the most obvious What are you doing with Cisco OpenStack private cloud? I guess I'll go first so Shutterfly has a brand called thislife.com that's hosted in AWS and We were seeing very ever-increasing OPEX costs that our business really wanted to control so We proposed the internal cloud which would allow us to kind of use our existing data center architecture that that we've invested heavily in And allow engineering to basically move the AWS for prints in-house And then in the long term over three or five years We're hoping that we could still utilize public cloud as a burst option for our seasonal workload So yeah, I mean currently we just finished installing our kind of proof of concept We're moving over some of the Dev and test environments that we have in AWS to kind of do a Validation that you know this the clouds actually gonna work and then early next year We're hoping to move the bulk of our AWS footprint in-house And you if I remember correctly you want to basically put a Heroku style type Architecture on top, correct? That's that's our long-term goal. I mean there's a lot of different options out there That's that's kind of a very Very fluid area right now So, you know, we're we're looking at mesos and Cloud Foundry and a lot of these different vendors that are building on top of open stack to kind of get us to that level where Developers can just turn up environments on demand as needed to pipeline the streamline the workflow We haven't really made a decision on which right we want to go yet But having that you know open stack in-house kind of gives us the flexibility to move in whatever direction we want And how did you start with open stack? Like what was the initial driver? It's it's really just research and figuring out if we want to go VMWare open stack Those are kind of the two big options We have a VMWare footprint in-house and we know how much that costs so that immediately kind of Shed us away from that that route and so Open stack just seemed to have the biggest community and the most traction So we just went that direction pretty early on. It wasn't too difficult decision That was earlier last year. I would say that we kind of decided to do that And did you install it yourselves and go that route? We tried yeah My team is pretty small and we're responsible for a lot of other things and so we weren't given a lot of Resources to try and you know play around with it. So, you know, we tried the canonical red hat some of the other cloud options out there and We just did not have the time to really maintain it and manage ourselves and that's why Cloud was a very attractive option for us Awesome. How about you Elon? Yeah, so we run We run an open stack for a couple different workloads You know or in our data centers both both here in the US as well as down in Australia and it's primarily we're using it to power a lot of our internal sort of back-end systems for our for our analytics platforms we all it does Entend video solutions everything from the delivering the video that you watch when you're you're probably on any of your favorite websites All the way to the other end of monitor that helping people monetize it and actually reporting on how you know How many times you click pause and did you watch it all the way through and all of the other interesting? analytics that you might want and so that You know, we're using we that's where we rely on meta cloud and Cisco private. Yeah, what have you to? to power the You know to power a lot of those systems and help us get that automation We've been an easy we were in the clouds We've been an easy to since 2007 our founders won the first startup challenge So I think at the time you could get an m1 large and s3 bucket and that's it sort of like any car You call any color you want on that car as long as it's black And so for us as we started to move into our data center a bit more for for some of that cost control We were we were sort of spoiled by the automation of AP you know the automation we can do around API's Something that we couldn't do as well if we were doing the KBM and Zen management ourselves And so we started to look at OpenStack and you know similar situation to Tim and tall we were a very small team at the time and We could we found we could invest in you know things like Cassandra and Hadoop and spark things that actually helped our application teams Be successful with analytics platform where we could invest in OpenStack and we think we made the right call Relying on some fine folks at Cisco and Meta Club Yes, just go OpenStack perfect COPC That's shorter. I'll go with that so You moved from AWS and you you are doing quite a bit of automation at this point in four availability zones Yeah, so I mean I wouldn't say moved. We're still very active in both AWS and in Azure We're using and we're using OpenStack as well But yeah, we have for for availability zones today running on OpenStack and Rob You know obviously Shutterfly we all are kind of companies born on the web Sprint's been around for a while. Tell us what you're doing as much as you can We are deploying a COPC to availability zone cloud in the core of our network to support messaging modernization so the next gen version of your text messaging and Has your team or other teams that sprint is this their first foray into OpenStack? We've been playing with it for a while But nothing in production as of yet Did you find paints in operating at yourself? Yes, very much so you need a certain level of skill a lot of knowledge steep learning curve Do the nature the size and the scale of what we're deploying we needed the skill set immediately and Found that Cisco could provide that course so one thing that's that I think is kind of interesting is you know in the model that we Deliver it's very much a a SAS type offering and so there while there are benefits to that there can also be Some downsides, you know, it's it's not as easy to just have any old feature out of OpenStack Added at any time Rafi. How are we handling that? So this is something we've had to to consider Over many years of running manacod So we've had deployments that were launched back in 2012 at this point And there are environments that have been up and running since that point in time and of course every six months There's a new OpenStack release. I think anyone who's been accustomed to and familiar with OpenStack over the years knows That features are introduced at a rapid pace, but they're also at varying levels of maturity so we're continuously evaluating the state of those features and Making an assessment and we go one of two directions if there's a gap We aim to fill that gap with what our clients need or we'll defer introducing that functionality into a later point in time so the point is we're We're Providing a layer of filtering to say this is suitable This is consumable and usable by our customers before we're just making it available to them and incurring Let's just say technical debt on their behalf So where have you found the most? Challenges in terms of projects That are maybe not ready for prime time in your eyes So it's probably anyone who's you know who's worked with us for a while knows that we've taken a pretty conservative stance on networking So we're just now making the transition for example from Nova Network to Neutron And it's interesting because the conversation as we have in a lot of ways starts off with Not I need features X or Y on network on my network But I need Neutron and as we distill down what the actual needs are we can fill them with a more tried-and-true Nova Network now we're reaching the limits and extents of where we can take Nova Network But that's strategy for us Is is somewhat reflective of the approach that we've taken in general with all of our features Which is we're not going to put our clients at risk. We're gonna favor stability and reliability Over features because it would have been really easy for us to say put Neutron out there before it was ready and Subject our clients to a lot of pain So we're incorporating it now We feel it's ready now and we feel that we've got the right tools be able to incorporate it and integrate it into our product in a Way that our clients can feel comfortable with the same level of assurance. We've been able to provide to date Which one of you which of you are using VMware currently? Using a little bit. Have you found that transition between maybe it's not even a transition But you know, how hard is it to get folks who are you know, well-versed in VMware to make that transition to start thinking In a more cloudy way. It's certainly been a challenge. It really has. It's just a different way to operate Having the tools and the ability to automate and orchestrate certainly much better Certainly you can buy tools from VMware, but there's certainly a price tag for that What about you Mattel? We haven't really tried to transition we kind of took a dual dual strategy where we'll have both available for our legacy code base Which really isn't Possible to kind of re-engineered to make it, you know cloud ready We tend to just throw it on VMware because they did Allows for kind of a more traditional data center But just that flexibility to operations to move stuff around and have high reliability And we're making open stack available and the direction I'm going is to have open stack be the the primary cloud And make sure engineering kind of writes to that and engineering's on board Obviously, they want to be able to use, you know self-service on man and automated provisioning and It just they don't have the resources to go back and rewrite all the old code So that's kind of direction we're taking it. We're kind of treating them as separate clouds and trying to find Areas where we can maybe bridge networking for example And maybe compute and see if we can down the road integrate them later on Seems like you know, especially for you Mattel and Elon that your teams have been pretty well-versed in AWS for some time It's probably a loaded question, but how much Importance or weight do you put on a consistent set of API's? That was that was key to us actually we we have a ton of engineering time invested in automating AWS scaling up and down pools and and automating Workflows and that kind of thing so having an easy to compatible API was important because we didn't want to have to redo all That for a different API over time We will definitely start using some of the more native Nova and neutron API is because they're more feature-rich But it kind of gave us that that you know a time frame to make that transition without having to immediately invest in it So you're using the open stack API's for EC2 for some of your workloads Yeah, I mean our provisioning and release and Deployment scripts are all kind of written towards the EC2 API so With very little offer we were able to kind of transition to work with the open stack cloud and Obviously, there's some gaps there, but over time we'll fix those by migrating to the Nova API What about you Elon? Um, I mean it's It's definitely a nice to have I think for us what we ended up doing on a lot of our tools just extending them You know extending the libraries we had written around interacting with clouds to cover open stack in some cases That happens to hit the C2 API's in some cases that happens to hit the Nova API's directly And so most people is they're developing against Open stack in our environment are not developing against the open stack API as they're developing as a tool We call improv that happens to talk EC2 Azure and open stack without having to worry about all that in between And I hope someday like we'll see things like chef provisioning and terraforms take this away from me And I never have to think about it again But that's you know today we've sort of we for us what we did is we built an abstraction layer for everybody And so that you know, would it be nice to have a consistent API everywhere? Yeah, I think at the same time there are things that you worry about in open stack that I will never have to worry about in EC2 I I don't care how much disk is available on hypervisors in EC2 because that's Amazon's problem in our data center If we're out of disk and I need to go order another rack from somebody I need to know that and so there's gonna be different APIs and tools and reports and I think I'm okay with that Any thoughts on API's prop Yes, we will be delivering a product at some point to allow us to make use those API's We have a different need though In the carrier world It's all about availability So a lot of the tools that we will be putting in place will be leveraging the API for deployment management Auto-scale things of that nature So let's switch gears a little bit and talk about Automation and configuration management. Have you each chosen a tool of choice and Rafi? This might apply to you on the back end of the platform as well Yeah, I mean we've we've gone through a multitude of different tools I think you know many of many of the folks here have heard about what we've done on our back end with config management We use sort of a proprietary config management system and I say proprietary loosely because it's actually open source But it's something that we've built But as we've gone along here, we've evaluated a couple of different options We started looking at salt stack we looked at chef and we've ultimately landed on Ansible And we landed on Ansible because of the simplicity that Ansible delivers So it's not quite as structured say as a chef But for the use case that we're covering in in effect centrally managing a network of clouds We found or we knew that nothing out of the box was really going to suit our needs So we needed the configuration management tool that was going to be most adaptable without having to get too much in the guts of The CMS itself, so we've gone the direction of Ansible We're actually transitioning our back end over the next couple of months to be completely Ansible managed We actually use Ansible as well for provisioning orchestration The the problem is because we have several brands acquired through different time frames some are using puppet some are using chef one is still using CF engine so We didn't want to take on the task of rewriting all of these Back end so we made decision to have Ansible be part of the provisioning orchestration layer and then just do a handoff to whatever system the engineering team wants to use for config management, so And that's a pretty common to we're seeing that quite a bit Yeah, we're we're big into chef. We've been using chef for a number of years now and so You know started off as just a way to manage the plumbing like you mentioned You know get the systems up make sure that everything from raid to mail are configured the way you want them configured and all of our different clouds in the same way And we're starting to see Significant adoption from our application engineering teams to use chef as well Although the second path that we're seeing a lot of a lot of uptake on is internally we've built a You know you mentioned wanting to go down sort of this Heroku path There's you know the internal version of that at all of something we call Atlantis that we've open sourced that's a Docker sort of scheduler and provisioner And so the other way that teams aren't deploying applications and provisioning nodes is via be a Docker Be this Atlantis service and that that also will run on top of runs on top of open stack So that's just about two VMs using chef and you know at that point doctors get scheduled on Have you all made a decision yet, Rob? Well, we like Ansible, but I think we're moving down the puppet path at this point Interesting Such a mix of everything there's lots of good options. Yeah, that wasn't always the case I mean Honestly, if you as long as you have a system for managing your configuration and you And you you know it well and use it well great You're better off than somebody that doesn't and then between that like that's after that It's personal preference code versus Pepsi and what does your team need? I think that's the important point here the recognition that everything needs to be automated and that your entire Infrastructure should be managed by a CMS is the key More than the actual tool itself because all of the tools largely solve the problems I mean, there's there's intricacies in how they're solving them different ways But by and large the getting the organization to embrace and adopt this as the style and method for managing infrastructure is the key You mentioned containers. So you all are using containers in production. You are and and Mattal I know you guys are kind of going down the path of just kind of exploring at this point Well, we have a couple of small services that We are deploying with Docker They're gonna be in production pretty soon. So they're more, you know, just getting our feet wet in terms of using Docker and We're definitely excited to see how Docker becomes or containers become part of opens that In terms of being able to use the unified API to do a VM and a container That's really great for us because we can give engineering either option Right now. We're kind of treating that the Docker pipeline is a separate entity than the open stack and a cloud Rob no containers as of yet, but I believe it's coming quickly excellent so but tall and Elon you both are running sort of in a You book on down the hybrid path, you know, you've got some public cloud You've probably got you know VMware other things. How are you connecting all of that? Is it at the application layer? Are you doing at the network layer? Is it all of the above? Oh Right now for us, it's it's actually not connected. That's that's kind of the the next challenge for us is to figure out how and when to connect the clouds and You know, we may make the decision to not connect them and just keep them as separate You know pipelines for for code and services either divided by the the brand and the engineering team that that's using it or just in terms of You know cost the cost centers just separating them out that way for us the in general the goal is to limit the amount of interconnectivity between these things as much as possible we treat We try to teach each of our regions or data centers or what have you as you know separate failure domains So you don't want you know if something happens if another you know something happens on the east coast again You know hurricane Sandy or what have you and maybe there's some issues on you know, US East one I don't want that to impact my metacloud deployment in Australia or in You know or in Sunnyvale But that's you know at the same time There's there's there's always going to be some need for there is some limited need of connectivity So like it may save the data layer, you know if you want to watch people want to be able to watch the same movies in in In in you know a pack as they can here in North America You need to be able to get that data to propagate backwards and forward so most of our most of that happens at the application layer at that sort Of at the data replication level, but in general we try to keep them as separate as possible And how are you overcoming any sort of latency or perceived latency that that might occur? I mean, that's the So you kind of want to work that this is this is where I think you know people use the term cloud native I don't know that I'm a fan of that term, but that's I think really a synonym for writing good software Like you're you're you're writing your software with the idea that there is going to be things like latency and failure introduced there And so that the VM fails or a network link goes down You're sort of failing in an open fashion rather than failing than rather than failing in a way that impacts your availability And so we've we've taken a lot of that into account as we're building our systems So one thing that I did a couple talks on this in the past few summits But one thing that is probably evident to everyone is that there is a sort of a lack of cloud expertise Available cloud expertise in the market. So, you know, folks are being paid a lot more if they have cloud expertise And companies certainly are recruiting to try to find good cloud expertise Are you finding it challenging to to find that kind of expertise? Or are you looking kind of at it differently and bringing your existing staff up to speed with with training and new skills? I believe the answer for sprint is to train our folks up Ultimately, but that was part of the decision to go with Cisco for the support model there From a application deployment perspective, we really have a vendor accountability model So we don't grow our own applications in-house So it's there's going to have to be an education with our traditional carrier vendors on how to move down this path There's most of them today are not using any virtualization in any way So it's a significant challenge. We may have to put on a workshop for Yeah, I think hiring is always a challenge, especially in you know, heart markets like like the Bay Area I don't think it's either or we're trying to do both and I don't even know that it's like cloud specific I mean, I think the reality is Whether it's software software developers or software developers focused on infrastructure It's it's a hot market right now and good talent is it's good talent The cloud is something that I think any any strong any strong operations or software person is going to learn very quickly Just it's just a tough market in general. There's not not not as many SREs out there as I'd like to see Yeah, I mean I think I agree with with Elon here So I mean one of the reasons we won't met a cloud is to kind of buy us some time to Get the folks in-house up to speed on open stack and get them ramped up so that they take a much more hands-on approach to You know upcoming iterations of our cloud and figuring out exactly the best route to go At the same time, we're you know, obviously looking around for for a new headcount and and it is very difficult in the Bay Area The people that do know the cloud gets snatched up pretty quickly So it becomes difficult unless we can make the case and then sell the company to them as an interesting place to work But you Rafi, I know you've got a lot of wrecks open We have a stampede of open-stack town who just wants to get in the door I met a cop The the truth is a lot of the Open-stack expertise or a lot of what goes into operating Environments such as an open-stack cloud is experience such as large-scale distributed infrastructure and computing So is there a lot of open-stack talent out there and available to come in and on day one necessarily contribute to open-stack Not necessarily, but over the years, of course, we as we've grown our team We've on boarded folks We obviously have the process down in how to incorporate them and grow the team as needed A lot of us have come from shops who have run and have run large distributed Infrastructure so we've built large teams and this is much take two or take three for us utilizing a different application and platform So it's something we're a depth in and we've continued to Prove ourselves as being able to to grow and scale the team over the last couple years Anything in your way that you think will prevent you from moving as fast as you want to Um So there's a lot of moving parts I think I think the twists and turns the technology will take and the industry will take will Will gate some of how quickly we can move we're having to keep ourselves amenable and Adaptable so that we can turn so for example the recent interest and shift towards containers right that could change in six months So a lot is still being defined because organizations are still working themselves to understand What's the what's the hype and what's the what? What are the real solutions and what are the tools to be incorporated that makes sense that actually provide value? so I think as The path becomes clear as to what organizations and companies want we'll be able to focus our Attention in a couple of distinct areas as opposed to being a little bit wider as we have to be today What advice do y'all have for other open-stack adopters? Would be open-stack adopters I don't know. We're still learning ourselves. I don't know if I'm qualified to give advice at this point It's tough to say on this panel, but look at all your options We we chose metacloud for for obvious reasons and there are definitely other vendors out there and it's a quick moving market So, you know, you got to stay on top of all the new stuff coming out and even though, you know, we're a few Versions behind of of kilo. We definitely want to pay attention to what's coming up so that we can you know Give you guys feedback to to say, you know, we're really interested in this feature and make sure you guys make it part of your Roadmap, do you ever see members of your team actually contributing back to open-stack in any way or not yet? Maybe soon. There's some out there. So maybe they'll take the hint You know, we're like I said, we're we're pretty new to it So I think as as we get more and more experience with it. We'll definitely start doing that What about you Elon? What advice? Yeah, I'd say I mean you really want to make sure that you're you start interacting with it early as an actual cloud and not Effectively another VM where like the web the web portal in school get to the point where you're automated against the API is very quickly I think and you know, there's also just the general advice I give to anybody going to the cloud and you know We're doing distributed systems is architect for that failure that we were talking about earlier You know, I think when in the first meeting when we were talking with with Rafi and some of his colleagues at Viola They're like, well, how much do you care about? About making sure that you know the storage is you know the storage is reliable That if a VM, you know an individual VM dies that we need to recover it and on a 90% of the time I'd rather it just die and not have to worry about and spin up another one I know the support guys in the back there were grinning because they've had to deal with me at times, but But but yeah, I mean that's that's the thing is architect for that day I'd be ready for any individual VM to die But also be you know, but use those AP that was API is that automation to replace them as fast as you can Well, I'd say do your homework research read spend time analyzing A lot of options out there a lot of different configurations I mean, obviously sprint is a very large company, you know, you probably use all kinds of different technologies, you know How are you going to implement cloud without bringing your old sort of processes of Requesting tickets and having to go through the approval process and do all that I'd say we're dragging sprint kicking and screaming. Yeah. Yeah, it's a challenge It really is. I mean, we're a traditional old school telco and Folks are ready. They're excited about what this means But it has its challenges. I mean some processes have to be redefined I'm not quite sure that they're ready to change their organizations yet, but We're trying to fit new technology and process into a legacy world What percentage of Moving to cloud do you think is cultural versus technology? I'd say probably 90% cultural The technology is there. I mean, it's it's it's easy to use. It's easy to run But it's it's all about the organization and how you're structured. That's really where all your challenges come from Do you agree with that Elon? I think it depends on what you mean moving to cloud Like I know lots of organizations that they just take their enterprise apps You know, that's the same oracle based whatever it was that they ran in their data center And they try to run it in EC2 and they say oh look EC2 is not reliable, but but but I'm done I'm in the cloud or you know open stack or what have you You know, that's that's not the hard part like getting your stuff up and running is not is not it It's getting it There's all these other benefits of cloud that people talk about whether it's the self-service provisioning or the fact that you're You're auto scaling or all these other pieces. I think that's the that's sort of the culture slash engineering part you want to get to the point where you're That's what your teams are building for rather than just you know Let me pick up this thing and move it into a new data center like that doesn't That's not that's not what I would call the cloud even if it's not running in your four walls What about you Mattel? Yeah, I think it's definitely more of a cultural challenge than technology especially for an established company that that's used to doing business with you know The yearly capex cycles and capacity planning and that kind of model It's it's tough to get business and other groups to kind of buy in on a more OPEC space model where you're you know Relying on external vendors to kind of provide the capacity So yeah, I mean and it's also I mean our company is not as old as sprint and not as big But we also have similar challenges where We have established processes and it takes a lot of effort to try and change those What about you Rafi? So I think I'm gonna echo a lot of what's been said and I do think it's largely cultural I'll make the distinction here and say that there's two parts to it There's I as in the context that we're talking about and there's cloud I as is a technology and an approach to technology cloud is a mindset and I as has limited Value and usefulness if you're not also willing to embrace the methodologies and the line of thinking that you have with cloud And we see this pretty frequently different organizations are at look different points in embracing this shift Some can more readily embrace it because they're they're able to invest the engineering resources or they're building new Applications others. It's a little harder because they've got more proliferation of legacy apps So we get a pretty good indicator of what we're an organization is when we start Initially interfacing By the type of questions that are generally asked there can be questions along the spectrum of I'm Translating my enterprise my legacy enterprise Infrastructure to try to run on open stack and there's benefit to doing that Versus a lot of what Alon is talking about which is From the ground up from day one the application was born and raised on EC2 So you can see that and I think there's a middle ground Our goal is really to facilitate and help ease that transition by having a Reasonable set of features present in our platform without covering everything that you would need to encompass For enterprise to get people started to help usher them to the point where they can facilitate more more traditional Or what we'll call cloud Mindsets and workloads and it's interesting. I think in a lot of the customer conversations I've been and you can you can tell very quickly Where that mindset is especially if they start talking about hardware right first right and I think that's a good point You know Cisco open stack private cloud even though we're part of Cisco. We are hardware agnostic We like to see Cisco preferred just because it makes everyone's life easier on our team Right and there's other benefits that come faster delivery times other things But does the hardware matter at this point for us right now? So parts of the hardware do matter, but we're still for the most part. We're non prescriptive So any x86 hardware UCS has of course benefits But still any any x86 hardware and that'll continue to be the case because there's no expectation that we'll go into a client And let's just say from this point in time. They decide we're buying UCS from now on well There's probably still an investment in some other Hardware from some other vendor that would likely be incorporated into the cloud environment So we know we need to embrace that that's not something we plan to change so completely Hardware agnostic today does the hardware matter Mattel It matters when it costs more I guess We I mean we run Dell I mean that's that's just the route we've chosen over the years and and we're not married to Dell It's just every year we kind of decides and do the math and they always come out a little bit cheaper So we always just go to them and with open-second matters even less It doesn't matter where your instance is running What about you Elon? I think the hardware it matters that the hardware is good Whatever whatever your definition of good is and that it meets your needs It matters that it's not Horribly failure prone and that you're dealing with dead drives every week But other than that like we're I mean we're a multi. I mean, it's always saying we're we're a you know We're a multi hardware multi hardware vendor shop I mean lately we've been drinking a lot of the hive Kool-Aid and buying a lot of the OCP type gear But before that we've had HP and Dell both announced and you know that those all three of those vendors are in our In our open-stack environments, so What about you Bob well previously hardware was everything a Lot of different options what we deploy in our network For our new solution here. This is all gonna be Cisco UCS based Yeah, and the one thing I'll add to that again to build on a launch point here is that you know What we see in a lot of ways is there was a transition for a while everyone was you know Dang the drum of white box until they started to encounter a higher rate of failures than even they were comfortable with So the reliability as as much as it's been de-emphasized It doesn't mean that it's not an issue at all because that cost still continues to play a factor The humans got to go replace those drives whether it's you or the HP repair guy or the UCS repair guy Somebody's got to deal with that and so you want to you want you're hoping for reliable hardware there I mean that being said I think the brand name hardware is the place We've seen more failures in the white box sure I'm gonna take the audience questions here after this next question So if you have any burning questions that you absolutely want to ask don't be shy Like we got someone back here Gary you have a microphone for these folks Awesome So while we're getting that mic ready, I'm gonna ask you a question Rafi we we recently announced sort of a Bundle it's a partner bundle that has some very prescriptive hardware from Cisco Can you tell us about sort of that bundle and what you think it will help us achieve? Yeah, so the idea is and I don't know that we've got an official name for it Alan was a key part of it He's sitting up here in the front row. We have a hardware bundle that's that's coming out that basically helps with With getting up and running Utilizing our platform. It basically gives you what you need to get going with our controllers And our controllers are x86 boxes with a particular hardware spec It gives you the networking equipment So you basically got the foundational components you need to get up and running with your first availability zone And as you need more you can roll in additional stacks of this hardware the unofficial name We've been giving it is sort of a playoff of the Cisco flex pod and called metapod very originally But it'll probably have a more formal name as we get along but it's it's seen or the intent is to be an Accelerator and to build on our model Which is ease of use and ease of consumption of open stack to apply that all the way down to the hardware And there will be some gains in terms of SDN Capabilities and other things that come along there will be we're doing integration against some Cisco hardware to get some Accelerated benefits such as integration against the Cisco ASRs to do L3 integration in neutron There was some articles that Cisco published about the work that we've been doing in that space But it takes a very robust neutron L3 agent, which has been to date Accelerated or placed on x86 hardware and moves it to try to intrude ASR routers, which will give us not only the the general level of resilience But fail over and much higher throughput than you would otherwise be able to achieve on x86 hardware And it's ACI ready so that that's right rolls out So the bundle itself the the metapod bundle is based on nexus 9k infrastructure, so All of them the 9k's are ACI ready So at the point where organizations if the decisions made that yes, I want ACI The switching infrastructure is already in place the idea will be to retrofit an APIC controller and Make a transition in terms of the the platform to use the ACI plugins and yeah, well it should I Can't say how disruptive or seamless it will be but the idea is it certainly won't be a greenfield hardware deployment to make That transition you can utilize the existing investment We had a question in the back Here How important it is is it that the open stack distribution is certified on the hardware for support and Yeah shooting between both the software and the hardware vendor how important is that so for us It's obviously important to do hardware level integration testing a lot of folks Focus on the open stack part and I like to make the distinction that open stack is a sliver of a large stack of Software if you consider one of the key components of that stack, it's the Linux kernel and it is still vitally critical to be able to do testing against the hardware that you're running over top of so that the Linux kernel for example That's incorporated in the distro that you're shipping is reliable and can present all the features for example that need to be consumed Upstack or up the layers of the technology platform into open stack So for example how reliable is the vx-lan implementation in the kernel that you're shipping because we incorporate vx-lan Into our platform so the end-to-end validation is what's important. It's not just the Python bits You look like you had a comment about that. No, I just I think it makes sense I mean that's this is something where whenever we were bringing up a new rack And then we've definitely had these conversations with you know with the Cisco team around You know what what should we be looking at and and yeah, I mean that's The places where we've seen you know where we've run into I guess more painful experiences at times I mean in general it's been it's been great But where we run into issues at times like oh the the that piece of hardware with that driver on that kernel They're not they're not making friends and they're not going to get along for a while And that's that's where you spend some time troubleshooting things. I mean the open stack bits I mean these guys these guys have got it down solid. Yeah, I mean to sort of provide an example In the room here. We've seen better reliability with the driver interaction at the Linux level on network interfaces with Intel cards rather than Broadcom cards So that's one example of where it matters and some of that has less to do or a lot of that has less to do with What we're doing in particular at Metacloud in our platform versus the maturity of that driver set in the general community So we can make those recommendations because we're able to we have visibility across a network of clouds All of whom are utilizing the hardware in largely the same ways using you know It's being used for an IaaS platform so we can we can provide those recommendations as well in addition to Delivering a solid framework and all the components that are necessary for that Okay Hi So one of the tussles that we have when you go from a self-managed infrastructure to a managed service like you're providing Is the tussle between how much control you want? How much you are letting go for each of you as a customer to towards Metacloud? So how did you kind of balance that if you can comment on your own experiences about that? So yeah, that was obviously a big question for us when we decided to go to Metacloud We toyed with a few other vendors and The the main the main factors for us in terms of control our visibility into the stack so we can make decisions about capacity planning and Uptime reliability And I think Metacloud Focused on that a little bit more than than the other vendors. So it made the decision a little bit easier Obviously, we don't want to give up control But at the same time we don't want to be in the business of you know maintaining an open stack cloud We'd rather work on our application and work with engineering and work on that level So it's a fine balance and I think as long as we have visibility and we can get enough information out of the cloud to make decisions That that was enough for us to kind of make the jump in For us we wanted input on the design of how it was built And there are some pieces that we still manage ourselves So for example the underlying network that our clusters are built on In most cases we're managing that It wasn't there wasn't so much of a control thing It was we'd let's have a discussion and build something together rather than a black box that we don't understand how it works And yeah, I mean at the end of the day We I mean the thing that's interesting to us is the stuff that our customers use not necessarily All of the hardware bits underneath it or the open stack bits underneath it And that's that's where we wanted to focus if we wanted to focus on open stack. That's that's where we would be well As I mentioned with sprint we kind of have a vendor accountability model so this kind of fell right in Very easy very similar though. We are Managing our network itself underneath We're in the business of a network, right? So But yeah, moving to Cisco with this was was pretty easy for us Yeah, I heard that cost was a big reason for bringing stuff out of Amazon or public clouds to to your private open stack cloud Additionally, have you been able to realize any optimized performance of your applications in designing your private cloud over? public cloud consumption is it measurable and part two Although acceptable, what do you miss or any functions you you miss and in using Amazon APIs or plugs that you don't have So we're still kind of moving test environments over So we don't really have final numbers, but I mean our projections are you know, 15 20% a month over month Mainly because we have a much cheaper Costs for power and space then we're paying Amazon It's tough to compete with AWS in spot pricing They have that kind of nailed down Where it makes a lot more sense for us is on static workloads databases elastic search that kind of architecture that Requires provisioned IOPS and reserved instances That's where we will definitely get the most savings. So the stack we have in AWS is kind of half and half our dynamic workers are all spot and we would actually be Paying more for the equivalent hardware and internally, but that is more than offset by the cost for provisioned IOPS and Reserve instances for databases. So, you know, we we made the case the business that you know initial investment upfront This specifically the shadow fly, but maybe other companies as well, but they much prefer spending capex and op-ex So that actually was a easy sell for us in terms of we can you know buy all the hardware And then we don't have to worry about month-over-month costs are varying that much Yeah, I think for us the big thing was actually that that performance piece that you speak of we were the initial move for us into our and into growing our data center with and with OpenStack was Cost for you know IO operations like we could get the IO operations that we wanted in EC2 We just had to get more instances or use provisioned IOPS didn't exist at the time You know use instance types that didn't make sense in terms of the ratio of CPU and memory to IO performance And so what we were able to do with in our own data centers was kind of create the instance types that really met our needs for Those high IO operations Yeah, I mean at the end of the day in terms of cost like you're optimizing for capex versus op-ex and depending on where Where you are in your organization's life cycle, maybe you want one or the other What do I miss I miss not having to worry about whether there's not enough hardware in the four walls Like it's I mean it it doesn't I mean we usually we have we plan pretty far out And usually we've got a you know We've got a what we've got another rack coming long before it's really an issue But sometimes that's not the case and you get to the point where you used up all the rack space in your building It's like well That there's no there's no fairy that like brings you the new rack and it just shows up overnight You don't have to think about it somebody has to go place that order and you have to plan for an advance And I miss that elasticity elasticity, but then again, it's I can't miss it that much I'm still in EC2 and I'm still in Azure like I've got other places to do some of this work It's just you know, maybe not in the place where I want it to be One point to add to that Amazon's done a really good job of getting the industry to focus on instance pricing If you and this is a common point the instance pricing has of course been reduced But it's not commensurate with compute capabilities, but let's not even focus on that part Let's use one example and the lawn actually alluded to it with IOPS IOPS transit these are two of the areas that Amazon makes actually a lot of their money Consider this you put an SSD in one of your systems. What are you going to get conservatively? 2030 it's actually more in the realm of 50 to 80,000 IOPS go in provision 50 or 80,000 IOPS in Amazon and look at what that'll cost you and that's really the area to Focus the attention on rather than how much is a compute instance because those prices are Intended to attract folks in and get them to start consuming Amazon so that they can make money in the other other areas. I'm afraid we're gonna have to wrap up We have another session ready to go, but we're gonna have a drawing for the next iPad mini So before we give everybody a hand you want to hand in your cards. We'll be doing a drawing Where are my sales guys sales they were in the back There's one back there We do have a really awesome pricing calculator So if you're if you want to see where that price break is for you if it costs as a driver Our sales guys can absolutely help you with that And thank you all for attending and thank you to my panelists. You guys are great. Thank you