 Welcome everyone I'm so make bearer. I'm in that you're moderated for today's panel on a lot of esteemed folks from various sizes of companies and various geographies were deployed open stack in their companies You know we're gonna have we talked they're gonna talk about their experiences of how they brought open stack to their company the challenges they faced and What you can learn from it so to encourage I'm going to start with some feeder questions after introductions And I would encourage for you guys to think of questions You'd like to ask them as well around challenges they faced So you guys can avoid it or you guys can better structure and be more prepared to bring open stack to your companies Okay, with that Alleged is myself. I'm so make bearer I've been involved with open stack since the open stack Diablo release first as a part of the founding team of The quantum the now we're now known as the neutron project and then I have been involved with customer deployments and As well as other various open stack outreach activities and with that I would let our panelists introduce themselves It's gonna go ahead Everyone my name is Yuri Brodsky From paypal cloud engineering Kim JC Martin From eBay sister company of paypal But I think that the reason why we are both here is because we we started from a different point of view and we have different experience even if As of today, we are part of the same team Simon Anderson, I'm CEO of dream host and the co-founder of intake Everyone I'm Shin Taro Mizuno from NTT R&D senior research engineer Good day everyone. I'm Tristan good CEO of Aptira. We were made gold members of the foundation yesterday I'm very happy about that congratulations great, thanks guys so The first question, you know the panel would be why open stack? Why did he choose open stack? I don't know who wants to take it? okay, so two reasons really one of them was We were looking for alternatives We were using VMware and the VMware memory tax Was an issue at the time so I know that's controversial I know we talked about that before but we were looking for alternative technologies that could fill that void the other Reason as well was we just wanted to find some sort of disruptive technology that was cool and fun and And I think we've picked the right horse in that respect So for us we select the open stack goes on before open stack we were doing our in-house development To do the similar things, but this in-house software got so complicated and big and we had had a Cost of it so much to maintain that so we were looking for the open source solution, which was flexible and Extensible and so that was that was a good solution for us At dream host we we've been a hosting company for a long time almost 16 years now and had shared hosting and virtual private servers based on you know Linux and and our own code that we managed to essentially create these managed services that could scale up and scale down and A little over two years ago two two and a half years ago when open stack was in its early days We basically started to look at various cloud platforms that were open source because that's generally been our approach to developing infrastructure and We really felt at the time although it was quite a bit at the time That open stack would have the best community the most Participants which is really crucial to the scale of this this project and I think that's really proving itself out cool so in our case it's a bit similar to the other person that talked about that Existing software so we developed our cloud automation Since 2008 and at the time there was no solution So we built it our ourself and today this solution is still running almost like 90% of the eBay site But when we looked at adding new technologies like for example new supervisors and network visualization like two years ago We decided to go with open stack because there was already a good integration with network visualization through the quantum project and The support out of the box of various supervisors like KVM So we looked at the cost benefits of either adding those features to our existing platform or starting adopting an open source project and From a cost benefit. It's obvious that we would get much more leverage by using an open source project that is Maintained by a huge community and we saw that the community was ramping up very fast so we knew that there was going to be a lot of momentum behind it and An ecosystem that would allow us to find vendors that would have drivers for open stack and allow us to build this abstraction between our automation and the vendors so that we can have a multi vendor solution So for us, I think the experience was very similar as well with probably one difference Oh PayPal did not have any cloud per se. We had virtualization. So we start looking at the different cloud options We had a couple guiding principles in place such as no vendor login We decided to go with open source, which was new to PayPal as well from that perspective and When we start looking around at what are the options were it's The results were fairly similar to what you see just described is there was strong community The project itself allowed us to build what we wanted to build to comply with the requirements that we have from a availability perspective and as well as Marketplaces team already had that in place So we had the extra benefit of using those learnings in house as well Cool. Thanks guys So it sounds like most people touched upon, you know choice on ability to take it and make it your own ability to extend it ability to customize it for the key kind of factors anything I missed out in summary of why why open stack and The community there's a growing community that you can contribute to Great. So when you started on this journey What are the some of the challenges you face starting with like organizational challenges, you know Do technical challenges? I'm sure where do you start? I think the technical challenges aside. I think everybody have a lot of smart people and Everyone knows how to solve technical problems In my opinion the biggest challenge are organizational challenges Want to start bringing this type of technology into organization that is historically in a t-shop It's a lot of unknowns a lot of confusion Because you start crossing boundaries there were historically very well defined you have your compute teams You have your networking teams you have your storage teams and now You have this technology That combines it all enables you to automate It's somewhere uncomfortable. I guess for a lot of people and I don't know how many of you attended the previous talk that Jonathan picker did but we had to go as an organization for this transformation of We need to go from the it shop into understanding of cloud technologies and how do you Have people learning new skill sets and being comfortable with this technology that in my opinion the biggest challenge Technology is very solvable huge community a lot of smart people That's not a hard part. That's my personal opinion so to add to that I think that the The challenge that we we saw is that when we developed our first cloud software of cloud automation solution We didn't design it from the ground up to be a cloud. So there was a lot of legacy infrastructure that we carried over and when we We embarked on the open stack journey. We made the decision to really be like an internal open open cloud or public cloud for our developers and business units so we We try to design it from the ground up to be like a cloud provider cloud really a serious one not just a VM manager and When we looked at that we we had to redesign everything from network to storage and then we we looked at Which team would support it because? We started we were only two developers bringing up this new open stack infrastructure with network visualization and At some point we wanted to scale out So we we tried to find how we build the teams around this solution and it was clear that The traditional model is not working anymore because there's less separation between like the storage team the networking team and the compute team and Everybody has to be kind of proficient in all those domains in order to bring up a cloud. So We we are still struggling to find the right balance between separation of concern expertise in domains, but also having people that are Full stack developers or operators that are able to understand everything from storage all the way to Yeah, I think it's interesting that Not a lot of us are talking about the technical challenges which we know there when we Our current cloud dream compute is in beta at this point But you know we've held it in beta for a long time because we just got to the Havana release And sort of doing continuous integration and there's definitely a lot of technical challenges. However, there's far fewer than there were two years ago and so that's great, but I think the key thing is definitely It's a people transformation challenge in that and you know both Yuri and JC have talked about this in that At as a hosting company we had a network engineering team we had a systems administration team We had a development team and we were actually ahead of a lot of other companies in having you know Really a dev ops type of model so the development team and the systems engineering systems administration team were pretty tightly integrated however You know there was definitely a silo on networking and networking is a great example where a lot of our Systems administrators and engineers have really had to learn and understand networking To a much greater degree to be able to implement and operate Open stack as part of you know as part of our basis for our cloud So that's been a big people transformation challenge But I think the reason to do it and why that shouldn't deter a company or a group from pursuing it is It's exact. This is the future. This is the way that Technical teams are going to have to deal with software and implement software going forward I think there will need to be a greater skill set level across networking storage virtualization and We're seeing that in our company. I think you know we're hearing from eBay and PayPal They're seeing it in their company and so that's that's a big transformation that if you're not going to do it now You're going to have to do it in a few years So this is a great project to get involved in and start to train your teams in that way So I think we first have a having same issues right in our team as well We've started our project based based on the compute people and The good thing about open-style is it's modularity and you can integrate all different software as a single system and We were focusing on the network side. So we were we were joining We were working together with the network people and created created a single team With the compute people and the network people and the issue there was a similar one, but The network people didn't know about the cloud or cloud people didn't know about network And there were differences in this quality of service the network people required more quality Rather than the flexibility, but you know compute people like to have our flexibility and modularity So those are the requirements difference in requirements was the big issue for us and To integrate them was a Big issue here So we had a couple of changes one, I guess two and a half years ago when we got involved with this were a very small team and We had to certainly broaden our skills across the Across the the team to be more In touch with with some of the things you've talked about but one of the things for us particularly was a Cultural change in that we until that point had been entirely reliant on vendor supplied software so we went from being a pave pay for vendor software to Embracing open source and that's a kind of a big thing to manage in a in a company because You are reliant on the vendor. You have a problem you call vendor vendor fixes Whereas this it's really a lot about what you put into it and what you Work with the community to bring your problems into the community and help the people that are out there fix it And there's also that I guess that the thing with working with a Business that was into reliant on vendor software There's a there's I guess there's a lot of social aspect as well that you don't really Interact with the vendors other than you just buy their stuff And one of the cool things for us is becoming involved with OpenStack and an open source in general is coming to events like this and holding events and getting involved with the community and drinking beers with people and Getting to know their business and they get to know your business and making great contact. So there's a whole Social aspect of open source that I think it's forgotten about a lot. That's really I find really great Cool. It sounds like you guys had a similar theme of how to create a single team Different people started from different points in the journey. Some people had established teams Some people had cloud established within the compute team While there's some people who had smaller teams who who learned other skills But it sounds they give key takeaway is that we need people who kind of were like full-stack Dev ops people can build the whole thing or understand That's great. So before I proceed to more prepared questions I had want to see anybody in the audience have any You know, it's a good topic, but how they structured any questions on did we would ask like to ask any of the panelists Here so it sounds like your question is that Sometimes in some an organization the silos get merged, but they don't completely merge They just you still remain with at least two teams or you know, just it gets reduced and your question Specifically was how do you solve the challenge or how do you instigate instigate instigate the organization to solve the challenge? Would you guys have any ideas about how do you restructure instigate the organization to create a conversion? I think I think there's no easy way to do it other than just like communicate communicate communicate So for example in our case we had a senior network engineer that Was all about quality of service and he was like nope You know the network is here and no one's gonna touch it. I'm not gonna give anyone access to it I'm gonna configure it the way I want to configure it and so on and so but what we did is we just We explained you almost have to go back to basics and You know bring those people along to open-stack meet-ups Bring those people along to cloud Events where they can actually talk to peers who've started to cross You know and understand the difference of cloud that You know, I can have a rack go down in my cloud because of a network issue and it's fine You know because it's distributed so There's there's aspects like that that it's a lot of it's just about I think you know community communication and Patiently stepping them through and some people will make it and some people won't and we actually ourselves candidly we've had You know probably four people who didn't make it. They just didn't make it They couldn't make that leap and they went off to an enterprise that was not doing cloud and that was fine So yeah, I think that in our case. There's an additional thing that you have to take into account is that With the technologies that are coming up the lines between each of those domains are blurring so if you look at network virtualization you have Logical routers you have logical switches gateways that are in compute When you look at storage you have distributed storage like safe that are implemented on servers with large disks, right? Or many disks so when you have a problem How do you solve that problem like when you have an issue? you have to have someone that either has all those three skills or You have to bring the three teams together and ask them to figure out where the problem is and that's what is happening In our case, which is at the beginning. It was kind of I'm going to keep you at arms Distance so for example networking team told us or if it's a virtual networks I'm not going to Support it. I'm going to help you but that's it you manage it if it's a storage the same thing if it's distributed storage It's not like large Sun. I'm not going to manage it. So at the end of the day We had Interesting project so they started to be interested in those project people were asking for example the networking team Hey, how is network virtualization going and those guys were like? I don't know because I told those guys I'm not going to be involved So now they started to want to be involved because everybody was asking them Hey, you're doing great thing in networking at eBay. How is it going? And those guys were I don't know something for storage, right? So at the end of the day, they are seeing that there's an interesting thing to do And they are like I want to be part of it and that's happening with our storage team We have now one storage guy in our team and there's some in the assembly and the same thing for networking You had one question earlier Did it change the relationship I Think it did because in our case with introduction of cloud we Finally got introduced to DevOps Which didn't exist before Right and with that the capability that got provided by cloud and now Are available to the end user which is actually a developer right You expose in developers too much more than they used to be before Right if you look if we look back a year in our case It was very standard sort of procedure the code is done. Here you go You figure out how you're gonna get it there where now They're unable to do it and the communication channel is much more open What we see we don't see so much finger pointing anymore What we see is more of the open communication saying hey, I just tried it didn't work and you help me. What can I do and we We've seen that leap on both sides Last finger pointing more working together trying to solve the problem Because My belief is once you remove this wall between IT and developers They see the capabilities they're interested as well this technology and They're willing to learn they want to work with you. We're just saying it's not my problem if I can just add something once we had the all-end and We invited a developer that was one of our first users on the our cloud platform And he told us that he wanted to name his son open strategies Which is the name of our internal a project So and the reason why he was saying that is because before there's a it took like developers up to seven weeks to bring up a new application life including procuring hardware and deploying it and With cloud and open stack specifically because there's API's they're able to manage themselves their own pool of resource and Deploy their application directly on that pool of resource. It's kind of Enabling agility and at the same time they are feeling that they have they are much more in control of what is going to happen So we only see beneficial outcome of this change Cool get few of the questions and go first Very interesting question In our case, I can say that our experience is not that the younger people were more up to change than older people like But the thing is more than the age it's more like the enthusiasm in what they are doing and that if they they see the benefits I don't think that this this would be an issue and it works across all kind of profiles In our experience during our pre production testing We were doing trouble shooting and When some trouble happens We didn't know which system has it the core root or the travel. So we created a troubleshooting team from Few members from each group younger people to create a troubleshooting team to look the broader Project so that they can discuss internally between on different teams and To have a broader knowledge of other systems By younger people so so that this troubleshooting team could isolate issues for each teams so and Those people could leave They go on communication between the teams and I think that that was a good Express within our development Good perspectives. So they were very traditional enterprise. I hear these had a very different way of structuring and solving that problem And we have even it so the truth take away. It seems like as there is ages and definitely another barrier enthusiasm and that's a people you want to target we had a question over there. I see your challenge it sounds like as You know you have successfully the PO seed opens tag got it up and running How do you not sell it internally within your organization to get the management buy-in before you can go roll it out to production? What you do is you you do a meeting with management Where you ask for requirements, you know you sort of like business requirements, of course and and then you know you sort of be ahead of the game and plan it out where Three days later you have a prototype up and running on the cloud and you show it to them and they go Oh, is this running on some you know? demo server and you know There it is, you know, I mean to me that's that's the way to really Sort of shock and or management into understanding it and getting it gets back to what JC was talking about you know the fact that that That like right now we have a lot of our tech support team, you know self-provisioning Cloud resources for their own stuff that they're working on you know at night It's and it's just the fact that they can they can do it They don't need to call anyone they don't need to email anyone They don't need to ask permission. They just do it. So that's sort of agility. I think I think management gets that That's that's really to me the the secret To unlocking that yeah, I just want to second that actually For us when we started was very similar right we build the initial deployment of cloud and Now how do you sell it? How do you actually make it production and for us? It was very similar. We did exactly this right we came to our executives and was it look Here's the application Let us show how it gets done Right couple of clicks. It's life. It's running We didn't talk to anyone in that work within talk about it to anybody in without no tickets right no change management for It went right and I think that's the big selling point right the business understands that in my opinion because at the end of the day It's all about business how fast it gets your products out When you show this agility That's how you sell any of their perspectives Good Had one more question over there What's it's we actually come full circle now, too. It's kind of a funny story That was just really a catalyst Because it was kind of a shock and I think a shock to many people that the licensing model changed so rapidly and I think that was backed off and settled Very rapidly as well So it was kind of a shock to get to get us into this and what's what's happened over the last two years since we We took up OpenStack Over the last year, we've actually turned around and gone back and worked with VMware Quite closely with Dan Wendler's team on bringing OpenStack Into or wrapping OpenStack around VMware. We like to call it. We call it the wrapper And that's all to do with controlling vSphere and we've had a lot to do with those guys in Getting that Out to where they are now with NSX. So It's kind of weird that we're still running pretty much the same core vSphere based core with OpenStack. We've also got Hyper-V and KVM stuff running as well and looking at how we move workloads in and easily in and out across those platforms, but So I guess in terms of saving money It's it's a bit hard to quantify because we're still basically using the same stuff as we were then so We haven't really with where we have picked up here and there as we've moved Things off and we're using the OpenStack sort of front-end to Run Hyper-V and running our Microsoft stuff on Hyper-V rather than running it on vSphere clusters, so That's certainly been a saving But as far as That initial thing it was just really a catalyst. It wasn't really a Direct saving as such Probably in terms of support OpenStack is a multi-vendor system using a multiple different software So in terms of support to have a have multiple vendors supporting our your system So sometimes monolithic solution would be cheaper in terms of support, but It's kind of tried off, you know having a flexibility in your system and or the monolithic system and the support With the team of our vendors and how you can lower the cost of support is a big issue so I have to discuss with the vendors and Find out the best solution For your required Cloud system, so the concerning support is quite important Cool There's one more question here, right? Yeah, so I can give you a part of an answer and the following Answer can be tomorrow. We have a talk with one of my colleague Subu About why OpenStack is not a cloud and I think that When you get the software you might think that you are done and you you deploy it and you are up and running But 90% of our engineering efforts are spent into running OpenStack at scale with sufficient SLA which means like doing support on hardware building monitoring tools Building additional functionality that that we require to support our business because out of the box OpenStack is more geared for Service providers that have a single type of customers like a service provider is not going to care about what Customers are running on their cloud. Every customer is the same, right? But in our case, we had different requirements We wanted to have differentiation on top of our cloud so we are spending most of our engineering effort in operating the cloud from fulfillment like accelerating the the pipeline to get servers in All the way to getting retirement like a tech refresh retiring them managing all the hardware all the the configuration all the like upgrades things like that So still a lot of work you're not done Yeah, I would say this sort of speaks to both the questions that I Remember the days in like 10 years ago when the Mware was getting going and it was like wow Look at this technology is fantastic, you know, and and I think you know VMware brought this huge leap forward in virtualization and server consolidation and so on and to some extent I think one of the false truths of the cloud overall is that You know, it's just gonna get easier. It's not. I mean IT is always gonna be hard because but when you think about When you think about cost and really what you're doing is, you know, you're comparing How many processes can I run, you know for a particular unit of cost and you compare that 10 years ago to today to 10 years time? That's gonna be the real measure because if you're gonna be competing You know, you've got to be able to leverage this these new platforms and so on so And of course the fact that you know VMware drove the virtualization trend all of a sudden Open up this entirely new opportunity for network virtualization and then now storage software defined storage and so on so So that's all part of it. But on the support side of things. Yeah, I think it just depends on your talent pool at the company I mean we ourselves at dream host we have a lot of You know talented Engineers who work with Linux every day, you know all the way up have developed our own configuration management software and so on so So we we don't need to get outside support But it is a matter of choosing, you know, do you need to work with a distribution of open stack and therefore do you risk? that aspect of it where you're gonna have someone to go to It's gonna be able to help you with, you know troubleshooting Bugs or issues, you know, for example with the with the hypervisor and so on so Making some of those choices is just it really depends on where you are with your skill set And I think as as this moves forward We'll start to see that really start to layer itself out, you know, there'll be incredibly robust Open-stack distributions that take a lot of the the risk out of running it There be very, you know suited to enterprise and then there'll be sort of service provider I'm gonna go with the open source code and not really pay anyone but maybe I you know Maybe I get support for particular modules and So on network or storage and and that sort of thing. I think it's sort of a matter of making those choices So it sounds like you guys are saying Running opens tag doesn't take away the problem of getting good engineers and good teams who can actually Run at the IT talent pool. Those still stay the same or it becomes more complex I don't know if it's more complex. It's a different scale I think that you have the multiplayer is higher, right? So for every effort that you spend in the cloud, you have higher return. However that that initial Tax is still going to be there. I'm sure you can hire someone to run the cloud for you But at the end of the day, there's still out where to manage. There's still data centers to to provision to Lease manage like all that aspects is not going away unless you go with the public cloud, right? Great thing. We're coming close to the end of the session Before I close it, I want to make sure we everybody gets a chance to give you know Any kind of the top three things you wish you knew when you start on your journey like you would do differently or Yeah, let's get started there. I would probably say one thing that I Would have done differently is to In the West into operational components of open stack from the beginning Right. It was a natural Guess reaction is let's get the product out Right through the software engineers at it make it work. We'll deal with the operational aspect with once we get there right But once it balloons now you have to play a catch-up games and that stuff So that's probably the number one thing That I would have done different everything else We'll get there Yeah, I'm going to mention only one or so. I think that for me the top one is It's focusing on the team that that you are building and the the main reason is that About what we talked about before like the skill set that you need to have make sure that you have the right skill set when You start because it's going to be harder and harder over time to backfill those positions If you you are not planning to get the right people in order to operate the cloud at scale Yeah, I think I mean we we invested pretty early on in getting our people to you know open-stack summits to meet-ups and so on but I would I personally I would double down on that because Even though there's travel cost even though there's you know time away from Programming or running your operations and so on just you know even I see it from our team we here We have I think 10 dream host team members, you know, we have product managers Justin Found at Dallas. Hey Dallas and see you there You know, we have systems engineers and so on and they can they can make one connection at a summit or At a meet-up that happens to be in the city and and if they openly talk about their problems Like what is the problem you are having today with this? implementation They can find a solution to it and I think Once again, that's that's the difference. That's the world that we're in today Technical team members The technical team members that flourish are those that learn how to communicate in this new world How to engage at a summit how to how to acknowledge that? You know, I don't know everything. I got a bug or I got an issue How would you solve this that sort of thing? So and that's all all there of course, but that's that's probably the biggest thing just invest and and get to get as many team members as you can into the community and Make sure that they know that it's okay for them not to know everything that it's fine That's part of it We've started the development development of open stack three years ago and We put our resources on the development of Diablo and what we didn't do then Was to upstream our patches to the community? So because it was you know Faster to keep it in-house so that you can do it whatever you want. So We didn't put much resources effort to we did but we didn't put enough resources to upstream so eventually our branch became very different from the community and Now we're trying to move back to the community. So we're trying to upstream every patch that we have in-house so that Costing us very much. So for those of you trying to add something to open stack you should Do it upstream it as fast as possible and sync with the community so that it's much easier to sync up with the committee and That is a big issue that we should have done three years ago. Yes, one minute. We're running out of time. Okay. I'll make it quick Less a technical aspect, but a business aspect. Look at your business drivers. What are you going to build? Figure out what you're going to build and then go back and check the business drivers again And then go back and have another look at what you're going to build and then go back and check your business drivers again Over the last two and a half years. We've been through Swift deployments and compute deployments and all sorts of things. I mean, I don't know how many times we've ripped our stuff out and Built it again and and turned it over and thought we were chasing rabbits down holes here and shooting at rabbits over there But what it came back to in the end is we need to look for the low-hanging fruit. There are things that would really sell And build on that and then go back and check it again. So check your business drivers Great. Thanks guys. Let's give our esteemed palace around Thanks for coming