 Yeah, let's do it. Sounds good Great. So welcome everyone to our session where we talk about Joe and I we talk about taking open stack from zero to production My name is Amit tank. I am a senior principal cloud architect. I work with companies like Cisco EMC and lately with direct TV, which is now a proud family member of AT&T in working with really large-scale cloud products and infrastructure projects Also have led teams to take open stack into production Yeah, I'm Joe Sandoval. I am with lithium technologies. I'm the director of the cloud platform engineering team there I've been involved in open stack since the Portland summit at a previous job and at lithium It's where we actually had a chance to really get to production So I think I've kind of seen you know quite a bit and hopefully today in this It's talk that we can kind of share some of the things that we kind of learn and as well as like kind of like Give you some strategies and ideas of like, you know what you can take away to be able to kind of get yourself from zero to production Thank you, Joe. So we have a very interesting guest speaker lined up later in the session and While we're talking about this topic There are some really interesting insights that we wish we could have Learned earlier in the stages when we were probably running through these cycles ourselves And so we hope that people in the audience can really benefit and take home some Solid ways to mitigate and maximize your chances of success. So with that said, I just wanted to have a quick By show of hands as informal survey for how many people are in the process of introducing open stack within your Organization or are planning to introduce open stack for production purpose in next 12 months Wow, that's very good. That's very good Joe Yeah, look solid. Okay. So without further ado, let's dive in So when we talk about open stack there are ways to introduce open stack within your organization and there are ways to Strategize around your POC some options are gonna work better for you versus some other options may or may not work as optimal for your organization our focus is to Look look at it from a architects point of view if you are in a fortune 500 environment What are some of the different ways that you think Joe people could maximize their chance of success? Yeah, you know, I think you know There's definitely you're gonna be you know dealing with kind of like you know identifying like what that audience that really would be The best use case to kind of help you kind of prove it out You know, especially because you're going into some environments at that scale that you know There there is silos that you're gonna have to be challenged with and here You are gonna try to come in and you're gonna be sweeping across like storage network, you know compute and You know, so you're gonna have challenges with security So, you know what what I really found and like in my current job and as well as my previous role because when I try to introduce it there I mean it was a disaster. I mean I really I mean I was very wide-eyed I was very optimistic of like the the promise of open stack But I just really wasn't was ready for the the cultural resistance that I was gonna face so, you know, some of the really prerequisites that I would really like strongly encourage is that you know is your organization or is your automation like really dialed in that that can play like a Really big key in regards to like how that initial kind of like kick off that POC can be as well as like, you know Are you trying to really retrofit something or like trying to bring things that are kind of like a brownfield verse I you know, I like to look at like starting off with like a green field on that initial POC and then you know Worry about those complex use cases later. All right. Very interesting Thoughts there. So I think one of the key thing that I've learned in different projects in these are knowing as an architect Knowing what are some important problems that you can solve for your organization is really the key Open stack has lots of things to offer However, is that really what your company needs to solve your company could be like the best as rocks maker in the world Are you trying to say bring DevOps into your organization? Say CI CD which stands for continuous integration continuous development or say something like self-service or say more scalability So I knowing what problems you're trying to solve really goes a long way great a great point about basically Strategizing around how you approach it one great thing that we've seen in in our recent teams is that when we Do small like we start small and we create internal success stories our chances of success are actually maximized People you you build your own advocate so for example let's say if you had your own end-to-end testing team and you were trying to Introduce open stack in that contained environment That's a better way to approach if you're in early stages of open stack adoption rather than trying to Overcommit or kind of trying to project as open stack as something that does it all so the great thing about open stack is that it allows if you manage perceptions it allows you to Be applied in a very pointed way. Yeah, well, and I think you know at lithium I mean, you know, we kind of went through like really like three iterative cycles And I could say like that first one it kind of rhymed reminded or it rhymed with hit show It was just a disaster You know, we really didn't go into it with like a real strong strategy of like what we were trying to do You know the implementation like the you know We had like a vendor that really helped us kind of like get that implementation up But it wasn't really like the right order like some of the things that we we learned from like the virtualization journey We just didn't think about like are we using like the right hardware? We was at the right, you know infrastructure to kind of like, you know What was it gonna support what we were trying to do and and then basically, you know Our developers who experienced it just had a really negative opinion about open stack and you know and just to be transparent We're a strong consumer public cloud. So, you know, we're really being rated against that So, you know, we really had our work cut out for us when we kind of like moved into like our second and third iterations of You know getting these successful POCs. That's great points. Yeah One thing I would really like to emphasize on is that understand what resonates with your executive leadership They're probably trying to solve certain business objectives as long as you can align your open stack value prop within those objectives I think you'll be in a decent shape So moving on to the next topic and I think Joe and I we have discussed about this very passionately before You have distro versus upstream options and There are various ways that it that's gonna impact your cloud strategy So I'd like to share an interesting perspective on this. There are definitely a lot more healthy number of options today then they were say two years ago and I'm sure there are all of these are great options. Personally, we've had some great success with Mirantis The implications of going pure play open stack versus not has really very interesting Considerations that you you'll probably need to think about are you really gonna know how how important it is for your organization to avoid vendor lock-in if you are going with a custom distro Down the line if you need patches and you need to say adopt this very widely across your operations across your Different organizations you are you gonna need a support net like off a big technology partner to build in build plugins around your Automations or say allow you to consume open stack in a very Customized way and so those are some of the considerations that you'll probably as an architect you would want to think very early on Yeah, I mean Kind of to say I mean I I'm with you on a little bit of it You know part of me though is like when I when I think about like you know with these choices And I agree with you like when I really was trying to like you know position it to like executive leadership I'm like what we were trying to do there was a lot of spend and you know That's that's one thing when you you know It's never agile when you're like coming in there and asking for $500,000 or a million dollars I'd like try to get something done and you know It's very like and from that level seems unproven But yes, we kind of went the same route We like we wanted we knew there was going to be some levels of vendor lock And we also knew that we had some levels of support that we were going to need as an as an organization You know the great thing though is that we've seen this like a lot of the learnings that are like starting to trickle into like the Distribution so what what I'm really interested now is I'm seeing is that you can kind of pick really? Varying levels of like how far you want to commit you know if you really want to play a conservative You know we're seeing some great options where you know there's like blue box where you can be like Okay, I really don't want to dip too far in like I wish there would they would have been really quick ways that I could have You know enabled us to like really see is this the right thing for us You're seeing other distributions that are very like community driven like big that they've taken the learnings of previous like a TCP cloud You know where I've been very impressed with kind of like how these companies are going in there and packaging and really like Reflecting some of the best practices into the distribution because you know when you get into some of these vendors You'll realize like there is the benefits of hey, they're testing it They're making sure things are working But how committed of it if you find a bug and like we all know that we have to be conservative and like a fortune 500 Environment about like you know moving too quickly you don't want to like disrupt the business So if you find like there's a bug and it's a fix are they gonna backport it into your distribution You know so we've kind of been on like both sides of it So it has been something that I definitely would have like you know I wish I'd had that insight when I first went into it great point And I totally agree with you that you have to you're better off being conservative when it comes to being on the edge For ecosystems like open stack if you're on the edge It's great in terms of feature parity But you're also exposed to a lot of noise when it comes to bug fixes Neutron is a great example. Lots of great bug fixes have gone into kilo release and liberty release. So Okay, great. So moving on to Our next topic which I think is personal favorite of mine about integration with your legacy subsystems and why it matters and When we when we when we've talked about this in past. We've always seen open stack as how What are your chances if if you are introducing open stack as a graded gradual path of adoption within organization Or are you introducing? Open stack as a switch the flip approach where you are gonna demand your company to build an entirely parallel environment and then have all of the workload move to that migrate to that environment as a switching the flip kind of like one-shot thing and The the graded approach I think in my opinion is really an approach where you can like basically Integrate with key subsystems like LDAP activity active directory IPAM DHCP DNS and so and so forth and it also Loves you to watch out for caveats a quick great example when you take open stack in production Let's say your active directory has like 6,000 users or typically for a fortune 500 company may have thousands of users What we've run into is like the Kilo version of open stack the keystone algorithm. It tries to scan Anytime when you integrate with a 6k active directory user database It tries to scan through the users and anytime any operations that you try to do in the cloud it slows up significantly or simple things like your Security people are gonna have a very opinionated approach to the Nova creds that are Stored in plain text in your open RC file Your IPAM plug-in support or you're knitted. So those those are kind of the caveats that as an architect if you Anticipate them and mitigate them before they become an issue You're gonna have much better chance at your open stack adoption. Yeah, well, you know, I agree with you I think I would actually kind of extend, you know the integration like yes, you know You definitely should be really figuring out like at what point do we you know? Do we have to integrate certain features, but I think it also extends in regards to like even like in your design decisions We kind of wrestled this like with our like our third iteration where we kind of we had some really aggressive timelines You know a lot of interesting things happen along the way as well as we went through this journey But we were we have a number of softwares of service and we have like really extremely tight SLAs and It's ideal as I was and the team was about like really committing to like everything being open source being software defined You know, we hit some challenges, you know last year, you know lock in my team You know, I really challenge them We had two really big big things in regards to like, you know What we were gonna do for like SDN and then as well as kind of like we were having some like storage challenges and you know We really wanted to Make the storage really work, but our block use cases were very strict So, you know, we really kind of came back and realized that hey, you know what? There is still a place for some of our existing legacy environment such as like, you know What we were doing with like storage so we actually had to integrate, you know Some enterprise storage and that would help us to be able to really meet, you know those block use cases We also realized that, you know, yes, we wanted load balancing as a service initially, but we really had some time I'm so I'm like look, let's make sure that that 80% that that is like rock solid Let's like let's let's just nail it on the compute storage and network, right then, you know But we said you know what we have so much automation happening in our f5 It's so integrated into our application that I was like, you know what we'll grow into a will allow those those projects You know to mature right and then at a certain point we can start strategizing like, you know How do we introduce these things or even possibly like allow our vendor to kind of catch up? So we've been very vocal to you know our vendors who handle our load balancing saying we need you to integrate into open stack These are the type of things we need your apis to be unified, you know with neutron So these are things that I would say, you know pick and choose where you decide to you know You know commit to if you don't have those type requirements great But I would kind of like defer to be a little bit more conservative on that side great point I think I love the part where you basically emphasize on pick and choose a great example is as soon as you put open stack in Production you're gonna hit this you're gonna have this meeting with your ops guys where today open stack security group isolation is actually achieved by IP tables where every entry is an IP based entry whereas your Firewall security team is more than likely to be using DNS for their wild card fire Firewall rule base and so that's gonna be a big disconnect. Personally. We had to mitigate that as well Yeah, a few other examples that you already covered very nicely I quickly want to mention this that this is where open stack really shines I mean we've had the open stack foundation which is which has done an incredibly job incredible job like things like project Navigator that just went live two days ago Where you can actually take open stack and turn off certain capabilities so that you can allow your company to Gradually adopt it if you don't need horizon great You can just consume open stack apis sooth through some meta scaling like a cloud management platform and still Continue to use your existing infrastructure and that brings me to a great Depiction of what a typical cloud would look like and this is really where your Open stack use case shines that you could have a neutron plug-in where you would have an existing legacy cloud So to speak still work with your open stack as a control plane. Yeah, no I think that's you know, you know, definitely, you know a strong approach I know we you know we kind of skip the virtualization journey So we kind of went from like bare metal, you know into into cloud so interesting You know, you know, and so definitely, you know, we didn't face some of the same challenges You guys get a direct TV in regards to like that having such a big strong, you know, large ESX environment But there still was like, you know some challenges that we had to face in regards to like how we provisioned and how we like Enabled our, you know, our our internal customers to be able to like self-service our product Very interesting point when you say internal customers Do you also have like customers that are external customers using consuming your open stack cloud? So we yeah, we look at it two ways I mean we we obviously have like, you know, we have over a hundred engineers that, you know We're onboarding into into open stack and this really like powers like our software So, okay, you know, it is you know, 80% of like what our softwares of services is driven by that very interesting Okay, so let's move on to the next part. So what challenges to expect cultural and otherwise Joe you want to take a first shot at it? Well, I'd like to think that everybody's like on board of the open stack like I was when I first really just found out about it But you know when I first was at my previous company and brought it in there, you know, it definitely was challenging You know, and I think part of it was just, you know, we were we were trying to dig through We were trying to do too much at that time. So, you know, I kind of blame like, you know We we should have been a little bit more, you know, more focused in regards to like what we were trying to do But you know, I think like I said, you should really just try to be and anticipate like some of those challenges Like really try to onboard and bring people along with it, you know, it's a journey I think it's like the really big key that I've kind of got out of it You know, we had to kind of like we started out when I came to lithium Mm-hmm And you know, we had an approach where we're trying to like really bring in like our existing teams And it was tough because you know, you're fighting fires every day But here we're also trying to like get Opensack up and going and there's a little bit of learnings that were happening there So, you know, we eventually decided that we really kind of needed to bring in like a separate team to really get it to be production ready And then what we started doing is really started really, you know, communicating out really educating getting people really to consume the cloud Internally so that, you know, we could really get them all on board to, you know You know want to learn more about it want to be an enablers in the business and really would seek out like, you know Engineers that were really like Lighthouse customers that, you know They could be our best advocates for like what we're doing there and understood the benefits of like Hey, I'm no longer waiting for like two months to like get hardware provisioned or dealing with some of the challenges of like Of all the snowflakes that we were building with bare metal They were really like starting to see the benefits of like how we were automating a cloud and what it was offering to them Great. No, I resonate with so many of these points that you just shared about because we've traveled We've taken that same path and we've traveled through those milestones ourselves One of the Litmus test that I like to use is the toy maker test Having open stack introduced in your Fortune 500 environment. Let's say you are a toy maker company Do you really have to turn into a technology company in order to consume open stack? If the answer is yes, then maybe you are taking a suboptimal approach to it Ideally open stack adopting open stack should not necessarily be projected as something where you have to maintain an army of engineers You have some great options of partnering with vendors Really great technology vendors. So no matter how if your primary focus is say selling travel Itineraries across the world Your company should be able to focus on that primary objective while your technology partners allow you to Come up to speed with an open stack infrastructure A great example is if you have teams Sitting on let's say your app owners individual app owners within your company sitting on six to eight months of validation data on a legacy cloud Infrastructure if your approach as a cloud architect is to go and tell them that hey your validation test validation is invalid Because now you'll have to run on KVM. Then that's Probably gonna be like it's gonna hit resistance and it's gonna be suboptimal if you give them an option of still continuing to sit on that their choice of hypervisor while still Consuming open stack in some capacity That maximizes your chance and your biggest ally here and I'm very fortunate to have some very Phenomenal executive management and leadership Sunil from my team is sitting in the audience as well They've given us really like on a weekly basis. They ask us very tough and pointed questions from a technological point of view So when it comes to these kind of caveats How can we solve these challenges using innovation? So whether it's making two hypervisors work with open stack or Growing skills organically doing internal boot camps and so and so forth like basically the approach is that as an architect You'll have to turn into educator Yeah, no, I agree with you there Especially because like I mean we all know like the the talent challenges is apparent And you know, it's great to see the the foundation kind of create like these education paths But you know, it's still gonna take some time for a lot of that to start really trickling down into organization So, you know, we've done the same thing. We've you know, even as I built the team I look at kind of like the composition of like what we build we have some with strong networking background We have other guys that were very great at tooling, you know strong linus, you know Internals externals that you know, these are these are what the guys are we brought in and I really was very strategic about Finding individuals that you know, there always was gonna be an easy answer even with having having like, you know Support, you know, there still was gonna be some things we're gonna have to power through we had to bandage some things We had to kind of like find some hacks and workarounds and you know, definitely, you know We you know, we're we're woken up in the middle night a lot of times trying to figure things out But over time that domain knowledge started to spread and now now what we're doing is like we're able to like really recede and culturally Really kind of like build like our next generation of individuals who are like helping us to be able to support the production cloud Very interesting. Very interesting. Yes, organically growing skills Maximizes your chance of success. I totally agree with that. So moving on to the next part. So Oh Open stack when you put into production, it's it's a strange beast where you really your chances of success are gonna be Maximized if your operations team your lab admins your L1 L2 support All of them are somewhat at a fluent level when it comes to open stack ecosystem It's not gonna happen overnight But are there certain approaches that you can probably take and there are there are some really you mentioned earlier about the open stack certification and I totally agree with you that the foundation has done a phenomenal job in like promoting this awareness where education is seen as a primary channel of basically opening up the knowledge within open stack and so Like how Cisco did with in 90s they did with the certification like everybody who was network certified when they moved on to their Organization they would be much more welcoming to Incorporating those technologies and so down the line Hopefully we'll have lots of engineers and architects fluent with these kind of certifications Where when they go in their respective fortune 500s open stack is not seen something as an alien technology It's something seen as a very Approachable very very very friendly technology. Yeah, you know, I would I would probably posit like You know the one thing that I really have gotten out of like this whole journey is you know I've consumed open source projects, you know over the years of my career But I think like the open stack community has done a pretty amazing job in regards to just really Helping me to understand like what it is to be part of this community what it means to like contribute back and it was Baby steps. I mean that that really was you know, I really didn't get it and now now I've actually kind of grown my team to where we're like hey, let's help other operators How can you know we ran meetups like let's let's just share everything whether it's like good or bad Like we want to really to kind of get it out there Now it's really interesting to see that, you know, we're in the SF Bay area and now we see like these great operator meetups We're like we're like we're engaged with like, you know companies like visa workday, you know Walmart We're like you're hearing other other stories, you know, we're connecting on slack channels We know we're finding ways to kind of like strengthen each other's knowledge and help each other like Overcome like scaling challenges or like pointing out like hey, you know guys Have you seen the same things like that we're seeing so I mean that's the one part that I really appreciate about being Being on this journey and being part of the community We're interesting and I totally agree with you that yes It's the it's having that sense of ownership when you have those people and that a sense of community where what do you do? helps others and you you get to learn from others experiences and I think that's really the big value prop off open stack and I couldn't agree more on terms of when you you mentioned about like your experiences on how You you organically grown skills as well as like you found these like pools of talents to tap into that's how I met you Like it the Vancouver summit. Yeah, absolutely. I mean that's a great example. And so another great point is if you want to Help your company succeed in whatever business you are in While using open stack as your cloud infrastructure project these summits are a great way like the foundation as well as the community Does a great job in connecting individuals where you get to really? Collaborate with each other and and that's the real spirit and I think I'll Emphasize on this point about like looping in various stakeholders within your company is a great key point We've learned that getting ops guys involved in the stage in the at the appropriate stage Is a great way to make sure that everybody is on the same page in terms of I'd actually say you got to take it further You probably I mean I we've been very adamant about like not only among like our users But like even above me even when we had challenges like what we had to like make quick pivots You know because we had been out there and very transparent right that you know We were able to have that you know the you know the the leadership support in regards to like some of the decisions that we had To make so I can only emphasize the importance of being very transparent and communicating up But then also like you know set the milestones make them very clear so that everybody can see like you know Your market you're tracking towards your KPIs great. Absolutely great point. So are you excited about the next part? Yeah, sure It's okay. So when you talk about This session it's really taking open stack from zero to production, right? So we have somebody in our audience today a Great guest speaker who really personifies the topic of our session today If you were to think I mean based on so many different experience with fortune 500 in finance sector and various different things That Joe mentioned about as well earlier taking a big blue chip cornerstone institution Company from a state where there is no open source presence to it becoming a very very Fluent with open source open stack like kind of project and becoming really champions in this ecosystem is As significant of an undertaking as you can imagine and we have to be for today among us who has actually done this with Phenomenal success. So we are going to invite him to the podium to share his insights and wisdom with us Thank you so much Toby. Thank you. I met in Joseph Really appreciate the opportunity to talk about our our experience and how it reinforces what was just described I think they did a great job at calling out a lot of the issues that you have going from zero to having something in production You know at AT&T We started our journey with open stack in 2010 2011 We did a lot of testing compared a lot of the competitive offers of that time whether it was cloud stack or Open nebula and then obviously a lot of our own internal tools And we did we had to do a lot of work to bring people together that could actually help to make this happen And we brought a live production environment up In early January of 2012 and so since then we've been operating open stack and It's been a long journey with many many lessons learned and I wish I'd sat through this presentation at that time because I would have learned probably Would have had avoided many of the obstacles and mistakes that were made along the path, but clearly, you know Finding the right partners finding the right group of people that can you can work with to make your vision real is essential I can't reiterate that point more Finding the right talent and retaining that talent has been very difficult For us we've gone through a transition of two or three different time periods and teams Sometimes the folks that we we spent a lot of time bringing in They disappeared especially at a large company it's very difficult to keep people around when it's when there's a lot of bureaucracy and there's a lot of challenge a lot of brownfield a Lot of bureaucracy a lot of processes a lot of people politics, etc So it's something really to focus on is how do you how do you keep people engaged and interested? thankfully, I think the open stack project has been been very helpful this way because So much innovation happens in this area that it allows us to attract and keep people engaged Whether it's externally or internally in a lot of the the module around it How well it's grown the continual sort of evolution of the project from from its start to the now The big tent and a lot of the new initiatives we have going I think that's been very helpful to get and retain interest Now that said also You have to be able to scale and for us we've scaled from a small number of sites in 2012 to now We're in the midst of deploying 70 locations We expect to deploy much more than that in the next two or three years So that transition has really required us to be able to scale the teams And then we've ended up having to use a partner Mirantis to help us to do that But also we realize more than just paying somebody to help you It's about really help getting the community to help you to work your issues and solve your problems And participating so it's both a given and take there's enormous opportunity to get and learn lessons from Events like this from the documentation from the various sources IRC's and etc But it's also an opportunity for you know for us to take our expertise and apply it back And so we've been trying very hard to to do more contribution and you'll see a lot more of that coming here soon from AT&T a lot more of our Researchers PhD is giving back try to balance the equation there So again, I can't reiterate more the some of these points that Amit and Joseph were talking about as we've gone through this process You know the When you're building an environment like an open stack that involves Storage networking and compute you want to make sure that these things are solid and you want one of the biggest pieces of advice I would give anyone is to think about it in the future. What what does it mean to have an environment one? Getting it installed and set up is one thing, but then two years from now What do I have what is it going to look like how many how many different apps or vnfs are going to be running on this platform and what kind of Poles and pushes are you going to get out of the the customers you have or the business internal business units? What's going to be pulling you try to think out in front and try to be Proactive about what what that would look like for us We've had enormous amounts of struggle dealing with migrations and upgrades setting expectations and such So the biggest piece of advice I have is try to give yourself as much runway as possible to get something stable and Solid the basic componentry of networking compute and storage Working absolutely perfectly solid and expand from there because if you go into it with too many things fluid and not You know iteratively being developed and not really Rock solid then that can cause an enormous amount of pain downstream And so we've tried many times to try to inject new networking mechanisms in the middle of a production environment and try to make enhancements and try to produce really quick change, but that's Having the base part solid and stable is is really essential because at some point you start getting so many customers That they weigh heavily on you and they pull you in many many directions So you have to be ready for that and so having the stable solid base is one thing but also thinking about the processes about managing service assurance and managing the customers that you have is essential as well because at some point you will be overwhelmed by the number of requests coming in and you have to have a way to to break out of the sort of conundrum of of Solving people's day-to-day problems and always being in a fire drill So I would recommend really thinking ahead and trying to be proactive about what you're setting up and be ready for the onslaught Because it's very easy once it gets started for many many people to spin up many many different things And it can get you can get overwhelmed very quickly So I think that's pretty much the main things is just be ready be proactive And then have something solid to start with and iteratively build from there So thanks. Thank you a minute Thank you so much Toby. It really speaks to the the charisma and the promise that Toby lends to the entire ecosystem so that architects such as ours Joseph and myself like we can have somebody to look up to and Really follow that footstep to making your success story stand out when it comes to open stack in your company So thank you so much Toby again. I appreciate that great Joe. Sure. Um, so with that We were open for a couple questions. If there's any out there You're also planning to share some of the great tools Things that we've picked up here and there like things like connecting your IRC and Slack so that you can still attend your Sunday 2 a.m. Open stack meetings on your Slack on your phone So we'll post some of these guides on Super user blog post a great guide is like setting up an open stack cloud on a single bare metal Node when you do POC in your company, you'll probably have to do like many micro POCs So that like when you want to do a demo to your CTO about scaling up Noa Docker as a hypervisor as well as like your KVM hypervisor You'll probably need to try that out on your own And so there's some great guides that we've probably come up with to deploy open stack like production on a single bare metal Node using a boom tool. Yeah, and then one one one of the things like tomorrow We'll be talking with like I mentioned earlier like some of the operator meetups that we have in SF with some of the like large-scale implementations There'll be a few of us tomorrow with from workday, you know from Walmart labs from visa We'll all be like, you know talking a little bit more about like some real some real like real-world problems that a lot of us are Facing so that's tomorrow at I believe 440 great and I definitely recommend I've attended some of these kind of conversations and Operators meetups are really great at understanding some of the caveats Which you are never going to learn in any books any guidelines any open stack guides Unless somebody has actually gone through them and mitigated them technically from a technology point of view This is a great way to learn about those. So thank you for sharing that Joe. Thank you that Okay, great Well, if anybody has any questions, we are open and otherwise feel free to pull either one of us or anybody Offline if you want to pick our brains into diving more deeper into strategy or technology Whether it comes to neutron plug-in or it comes to multi hypervisor or unified open stack plane We're more than happy to help in any way. We can yes. We have a question Great question. I'll quickly repeat the question. The question is about working with different vendors Do you have you found a trend where people want a single vendor for your storage? Or they are okay with going with multiple different storage vendors We actually Kind of like you know because like I said we have We're a product of like we're a company that exists for about 15 years. We've had two acquisitions We have some engineers that are very familiar with with AWS So instead of like really focusing on like specific versions of like what the storage is, you know Because we didn't want to get into that We did found it was important for us to like do provision IOPS You know there was we obviously wanted to like be able to as I mentioned earlier deal with some of the challenges of like Noisy neighbor as well as like we needed like QoS as well So that was more important to them to know that they had that and actually we were really We actually anticipated it and that and we didn't even like wait for that to become a problem We were just I was just like guys This is not going to work like all our performance testing and benchmarking was showing that we were going to break SLA So we went away where I proactive about it So that's a great question and coming from a storage background myself from EMC days I can speak to this question where Today the the customers are being very savvy where they want to focus more on their software because as a very wise man once said software is eating everybody's lunch and so When they look at it is they have a legacy vendor from like XYZ storage box Whether or not it can complement on a software site to help you define your software defined storage If the answer is yes, then they are better off sticking with that single vendor but if there is a Different vendor say startup that gives you a better software strategy around your storage Then you will see customers going for that with an abstracted storage around it So that you can change keep changing your gear gear, but still your have your business layer Stay stable Thank you. All right. Thank you so much