 Hello, welcome everybody My name is gilbaros. I'm here with the can and see you and we're gonna talk to you a little bit about What we've learned from one of our most recent large-scale telco deployments where we did open stack plus SDM Just a little bit about what we're gonna talk about who we are The the timeline of what an OSP engagement like this looks like you know where we start the many steps in between And hopefully we're gonna be the challenges that we bumped into which I think is what most of you guys are interested in Figuring out where you're gonna bump into problems and hopefully how to solve them There was a lot of new functionality added to open stack with this engagement, so talk a little bit about How how we got there right? I mean how did we define what the new features are that were needed? And how to get that into open stack or how redhead was able to get them into open stack some of the interesting things that we worked on also creating some testing automation with DCI and we'll talk a little bit about DCI and the future of what some of these third-party integration efforts are gonna look like with the composable services and possible roles So hello everyone. I'm very happy to be there with you In Barcelona, nice city and you can hear sometimes it's pretty hard to sleep very hardly So I'm a senior consultant for Red Heart. I'm coming from as an advance acquisition I'm focusing on the cloud technology principally open stack deploy with a director and As I can I'm doing a commit upstream So hello everyone, my name is Dick and Christian and I am a red hat from In advance acquisition the same as serial but my role is more to to be a senior technical project manager involved and I'm leading and contributing to the delivery of cloud projects and we're using our Interactive delivery methodology mostly And as I said, I'm Gil Barros. I am But by title a product manager, but really I'm more of a partner person I've been at Red Hat for a pretty long time. I've had a few different roles I was in consulting on Wall Street. I did support and support strategy and most recently. I'm in the open-stack business unit Handling partner engagements, which is how I got involved in this one So really quickly on what the timeline of this engagement in particular looked like Which is actually pretty similar to some of these more complex engagements That we've been involved in so around the middle of 2015 we had An RFP that got published we put together an answer did some POCs And really investigated what the upstream is as the integrations looked like in the OSP 7 at the time There wasn't a lot there let's say so really it was Figuring out what we would need to do to Get to the point where our customer wanted us to be Deal signed pretty quickly after a POC interesting POCs that we did quite a few other open-stack I Guess competitors of ours We're also delivered POCs to You know give their take on how they could do these integrations and how they could do this Deployment, but we signed the deal in February and got services started pretty quickly in March with C. Hewland site to Get started with a customer understanding Sort of what their environment really looked like other outside the RFP right the RFP gives gives us sort of a Very a very sanitized look of what they'd like an engagement to or a deployment to look like But then there's you know, what does the customer? Environment really looked like right it's not not always quite the same At that point we're able to put together our ideas for integration. So we did a hackfest in June with OSP seven at the time Seven or eight. Okay. We did I think we did one with seven one with eight and With OSP eight we got a bunch of upstream Work done in in triple O to get the client side SDN plugins deployed That brings us to a few months back UAT user acceptance tests from the customer We're in sort of the pre-production platform phase right now where the customer is Doing some final tests and we're going live very shortly So in terms of project challenges, so it's here just some of the Main challenges that we are listing and There are really three Main entities so the customer we are at Fredat and the SDN partner involved what was really challenging is that the The agenda of and the priorities as well as the objectives of the three are I mean we are not really the same Even if the target of course was the same so you had the customer that had to go in production in Q4 this year and What is interesting is that red at the full? I mean the target platform with the full integration with the SDN Was scheduled for the same Period of time so it's already difficult to have the go live with the products at the same time and What is even more complex is that the certification on the same platform With a partner the SDN partner what is scheduled for next year? So we are not yet at that at that level so that's one of the main challenge in terms of Schedule the the second point is about the The the expectation of the customer in terms of the maturity of the product even if they understand quite well that Red Hat is delivering open source products But they are still expecting much more mature products documentation and everything and We we agreed from the beginning that we are building that with them So that's interesting points. They agreed on that but still expecting you know The the top documentation from from day one in terms of integration mostly because you have of course the documentation of OSP, but the integration something different So the customer is expecting that and after that we are focusing more on Product delivering product not services. We're not selling services. So that's one of the challenges Because integration is is about having services. It's not about a product itself To deliver what the customer is expecting So that's a big challenge on our side and and for me to deliver that and to coordinate that the the other thing that is Interesting and not easy to deal with is the culture thing cultural thing So the customer is really working with providers So he deals with providers not so much with the community it's not something that they are used to do and after that we do everything upstream first With daily we are delivering, you know open source codes and we are more a community Company than you know a provider So sometimes it's not so easy to to discuss to plan things because we there are some cultural shift or Mindsets that are difficult to do like a acceptance of features, right? Yes, they have to be accepted upstream even though this is exactly what the customer wants We still have to negotiate that upstream and make sure that it fits with the open stock community And the the main impact is more on on the you know delays and the agenda because you need first Acceptance from the community before they're doing something So it's not just you want that we can do that right now when it will be there You need to have the acceptance of the community to certify that to make the projects and so it takes time Absolutely and really quick on the Okay Really quick we talked about sort of lining up these roadmaps right and It's interesting because It's not that we don't want to have lined up roadmaps But it's because we've we've each got a call it a product in mind, right? I mean the customer has the product that they're delivering delivering to their their customers We have an open-stack product that's going to meet those needs and the SDN vendor for example They have you know an SDN product that is going to integrate with that open-stack product to create the deliverable So getting those to all line up together Has probably been the toughest thing for us because this gets released here This case released here this gets released here But this has to be worked on way back here because that's when it was initially Submitted upstream exactly. That's why I have to work a lot to to to feed the gap and to provide them a way to deploy it With the better life cycle the better experience with it and doing the job to refill the documentation Help them to understand what's the gold what the will what will be the next to have a really Take the right choice to don't break everything with the next version some some topic like that So, yeah Basically, we are working with director our product based on triple O community So as a girl say you have to do an acceptance from the community to implement feature We have our roadmap. We have many things to do because we start Triple O in production with usp 7.0. It was pretty hard at the beginning because many many stuff to do and sometimes Issue to solve it and right now we have a solid product and a lot of people are here in I don't know if you take time to go to a session technical session to discuss with an effort guys All body in the room Say, okay. Well now we are ready to deal with open stack. We know how to deploy it We know how to do the life cycle, but when a partner and I'm at mean when SDM Coming in the picture, it's start to go mean going more harder to understand Oh, oh, I won't oh, I will deal tomorrow with my life cycle or I manage it because it is not You know straight forward and the customer was expected that was expected a documentation straight forward to just follow the documentation go ahead and sometimes you have your partner shoes in a way and Is not fitting with you. So you have to deal and to take time to implement what is needed to go through to what the Customer need really at the end. So it's mean Basically our partner working more with With a pack stack doing the certification API of past act So it was an issue for us because we we don't want something, you know We want to expose An open stack for scaling. So we need specific nodes with For example, we need database outside. We need some change and As the partner and us we don't have take time previously to test it So we test online and it's not the best way to do that, you know when we are facing the customers We are doing together the job and we saw bugs So it's not when you have a as if you can say a timeline very short because it's deal by What the customer need for them customers? So it's not we are building a platform and when we are ready. We're launching a service either. We will launch a service Guys, let's go. We have to do the job. It's not the same timeline and it's pretty challenging So at the end Oh SP 10 is supposed to do the job fully, but He's not there for the customer So we take time on one sp8 of previously no sp7 to do the job and to improve step after step with with looks From a radar to mentality is now the agile methodology to improve step-by-step to add feature and To have the proper refactoring some parts to have the proper deployment at the end and To know there is some difficulty like a certification take time. You have to wait The right version of you has done partner So to be ready to test it and to implement it. So there is some of delay and was a So we've worked out the the integration into two parts so the immediate requirement for the customer was that we're going to go live at the end of this year and OSB 10 wasn't going to be ready. Well OSB 10 plus the SDN integration and the SDN partners software wasn't going to be ready so as you mentioned we had to put a significant amount of effort into OSB 8 and 9 to do Initial integration, which means we were deploying the client side SDN software So the the the partner the SDN partner tools were still deploying The the SDN backplane the SDN management and web UI and all that kind of stuff But we were making sure that our compute nodes were properly configured and talking back to that backplane and that's really what took a lot of time in OSB 8 and 9 because Honestly at the time OSB 8 wasn't ready for that kind of manipulation of what gets installed into Into the compute nodes. So a little bit of work there, but we're in a good spot now so I just wanted to highlight other challenges. So The I put that as people challenges. In fact One of the main objective of the customer is to be autonomous and to be able to deliver I mean to support internally the teams to have the knowledge as soon as possible on how to deploy and Be able to provide so internal services to the different Customers so because he wanted that from day one He didn't invest so much in services and it's that's what we do usually because when we have a large scale deployments We prefer to secure that we have and bring our experts and Do that and show them how it works and then they can reproduce it But it was not like that they wanted to do by themselves from day one and The one of the big challenges that we have only serial on sites So only one guy on site helping them to understand how to do it Document it and try to support their internal teams So that's really really difficult. We managed to do it Hopefully, but that's a real big challenge What we usually do is Typically to have a workshop at the beginning just to align everyone because you have many entities you know partners customers and our teams and We know that if you are not having an alignment from the beginning on where we are going why we are doing things What are the constraints the problems? It is very difficult and we didn't do that so it adds a lot of challenges because of and we'll see that Communication issues synchronization between the different entities and it's it's not so good So that that was one of the the main the main points So the other one is of course as they wanted to do everything by themselves They didn't have on their side all the knowledge from day one. So it takes time everything is more You know is more slow to do things to implement things and on On the I'm in the last point What is interesting yeah from from the sort of the the red hat and the sdn provider sdn by a partner It seems like as expected we have deep knowledge in our own products, but to very little knowledge on on our partners products so that cause Quite a bit of trouble when we were trying to do integration where say red hat engineering is working on the configuring the client side Compute module, but we don't really know how to configure that compute module. So we're working off of you know Partial documentation and you know lots of back-and-forth between us and then the partner to figure out how to do this or vice versa, right? when the the partner is Trying to do a new deployment with director, which they hadn't used before they were using pack sack and again just bumping into into problems And I think this is one of the big lessons We've learned is we need a much deeper integration Between us and the partner with a lot more people either on site or some sort of partner Engineer exchange or something like that that we're still trying to figure out But it was definitely one of the the biggest areas where we saw a significant slow down Yeah, and I think there's a French term for that the Yeah, I just wanted to put you know a picture to illustrate the thing is you know an open stack You need experts to do the I mean on large-scale deployments involving you know deep SDN Understanding and in the telco markets with the MAV and everything so you need experts in every you know Every aspect of the of the deployments. So in French we have we say that we are looking for a five Five-feet ship so It's in French it's a mouton a 5-pats so you will not few Few few things from Expression So, yeah, I like this picture because I'm When you're talking cloud you think oh, I will spin up a VM. I will delete it I will re-spin it my VM life is near one day one week And we have a difference there because sometimes with else looking for something there's a History when they put something in production for 10 years or go 10 years old or more Than the antenna so it's it's very challenging when you're coming from The web or other different place telco have a very Other point of view and you have to deal with that to understand security is Very important you can have a no-go just for security and We are looking at products. Well, okay while it looks secure from them point of view, but sometimes they don't We don't align on some topic and we deal with it especially to have a Real segregation between data plane and control plane basically we are doing that by network But it's not good enough for telco. They want different nodes with firewall between To allow only one one way Discussion and we are lucky because with open stack. We have a way to do that with a message booth rabbit MQ It's pretty simple to do it, but the implementation is not ready for that. So we have to deal with Oh, I can hack it to have something running and Oh the support will support it when the customer will call because as this topic is a the customer will deploy This infrastructure in many many country and I don't want someone call me at 1 a.m Because they have an issue in Singapore. So it's a very interesting point and we try to to address its best and It was really a challenge after Another point who a tech time to explain When you are deploying a product you're deploying a product. We are deploying an ecosystem. It's not the same deal It's not some ID and you have to keep in mind if you touch something you can break another thing without any apparent link So it's another topic with that to really provide the help because as we can say the customer would like have The knowledge of what is doing and it's it challenging but it's interesting because you all time You have to think about what you are doing why you are doing to explain it to the customer and to be sure It makes sense. So you challenge every step of your deployment. It's Sometimes stressful, but very cool at the end when you have a win when you have something working when all people around the table agreed with a solution another topic about support Why people want to use director today? because they figure out Life cycle could be a mess With every six months when you release and as I told you sometimes they put something for ten years It doesn't fit so we have to help them to have the better life cycle and basically when you are just using OSP director is working Almost straightforward. Even if you don't trick some stuff if you are using the right way to use the product But at the end when you integrate as we did another partner in the loop Who is not Ready to do that you have to ask some Templates to add the feature or you need all the time you have to keep in mind the life cycle You know the day after what we become because we are going through a version for May Three year is not the same timeline as I said I told you previously when they put some stuff for years and years So it was a big big big challenge for me to achieve that Yeah Seems like we're going a little slower than we expected I'm talking a lot. I go ahead So one of the really important Message and Lessons from these deployments and something that we we used to do at redact is a hack fest But this time we were doing an integration hack fest so that's something new and something very important as The objective is to align not only internally but with the partner So we have engineering working together in a room for a certain amount of time usually one week And share, you know the knowledge and the problems face to face and that's really important If you want to go faster and to test things validate things and have something that is documented by the right people so it's always difficult to have You know to put these kind of people in a you know at the same place for one week It's a kind of important investments But the return of investment of that kind of event is really Tremendous, so it's really important to do that so as I said the Deliverables are usually a full documentation on all the things that they tested on the product So usually the use cases that I expected from from the product and from the solution here with Because we are doing a kind of integration test that is not something that we usually do will find new bugs So it's important to list all the bugs and to follow them So that's really something important and that's one of the Good outputs from this was a key. I think yeah, you've success. Yeah, they were definitely the the largest peaks in in I guess forward movement toward during the hackfest sorry with Both us and the SDN partner in the room, you know the right engineers together, right? I mean as you mentioned it's expensive You have to fly people across the world. Yeah, but it's a week in which you make significant progress Yeah, because as we said before you don't have so many experts So you will have one or two guys that knows almost everything you need to Know for the for the work So you have to take this guy one or two guys that know you will have more in the future Of course, but to put them from each partner or each entities together and then out of this They will know each other so everything after that will be easier They will come with a documentation. That's I mean it's proof that it is working and everyone will Be able to reproduce it because we will have the right people To do this documentation. So that's very key All right, so why red hat right? What is red hat bringing to the table here? one of one of the big things that we talk about is sort of our capability drive change I've read had is one of the main contributors upstream and Definitely on the triple O project And has quite a bit of experience doing project management for these large deployments, right be it's in telco fsi healthcare or whatever so we're familiar with how to understand customer business needs turn those into specific technical deliverables deliverables and then push those in and sponsor them in upstream community I think one of the the the things that's he'll mentioned was that it's we're really not talking about a specific product here It's it's an ecosystem, right? I mean every even though Your compute nodes are over here. It could be rebooted that actually could be touching something else That's gonna have a problem or your SDN controller node, you know that you expect is has a touch point here Might actually have multiple touch points. So being that we have experts in all of these Fields and have access to partners with experts in all these fields. I think it's an important part of what we can bring to the table Really quickly on how we handle these these future future requests or really sort of Converting business needs into into future requests It's a lot of communicating with the customer communicating with Not just the technical People at the customer site, but also the business people at the customer site and turning those into specific features that we'll either be driving with our partner for them to submit upstream be that part of the the integration of the SDN and Work with sort of our peers upstream to get them accepted We've talked a lot about aligning life cycles and aligning roadmaps One of the the items that came up In this project was we wanted to make sure that an extended support release of Red Hat OpenStack and an extended support release of the SDN Software were lined up right so OSP 10 And the partners, you know long-term support or extended support version matching that And that was one of the things that pushes that has pushed us out into early next year is their version of Their product with long-term support is only going to be available there Sort of the long-term vision of how we can accelerate a lot of these things is through distributed continuous integration It's our work with other partners and other customers has shown that this is probably the single largest time saver available basically it automates testing of The customer software or the partners software and the SDN integration together with the OpenStack bits from the pre-release phase so basically the DCI infrastructure at the partner site or at the customer site would have early access to Early red hat builds we call them puddles But basically they're internal red hat builds that they get automatically pushed to the customer site and I've got us We can probably skip to the slide that has this and I can Yeah, I'm talking about that. So let's skip to the slide and I can make fun So basically the DCI repository just gets pushed to the customer site the DCI agent on the customer or partner site Deploys OpenStack and deploys the partner software or the customer software or the SDN integration at that point and runs Both regular rally tempest tests that we have for OpenStack, but also specific Partner and customer tests. So we'll we'll work with the partner or the SDN interpreter in this case and create specific tests to make sure that Their SDN deployment as part of OpenStack worked and we do like a quick shakedown and then we return results to red hat and we can say Yes, we're good or What happens of course with early releases is oh this didn't work at which case Instead of being in a situation where the product has already been released and we find a bug We're actually finding these bugs early and we can fix them before the GA date So I think this As I said is probably the biggest time-saver because we're not waiting for the OSP GA date and then the SDN software GA date And then we do integration testing between the two right we're doing integration testing as the products getting developed early and we're fixing bugs as The majority of engineers are actually focused on working on a release right? I mean if you think about engineering time slices Usually you have engineers very very busy working on a release as it's coming up to be released Right, so nine out of ten engineers are working on the upcoming release one out of ten engineers to working on maintenance For a prior release these numbers aren't accurate, but it gives you an idea right so soon as OSP 10 comes out Nine out of ten again making up this number a little bit nine out of ten engineers are gonna be working on OSP 11 one out of ten that doesn't work one out of ten is working on Maintenance of the previous release so you can fix bugs much faster here than you can hear yeah, so in terms of communication, so there are I mean we understand that there are many entities many different business units and There the communication is key and was a big challenges and it's still a big challenge, so Yeah, you have which functionality is due when so we need to make sure that we will have the feature that the customer is expected for the rights deadline Which test requirements for which version so everything is is important and everyone should be synchronized on the version the features and everything and because you have you know the Red Hat versions on the different Products you have a partner the same thing so it could be really A mess quickly if you are not synchronized on all that and The three-way communication here is really critical. Okay, so we need to find the best way to Have everyone aware and and in sync on the hot topics as well as the right people doing the actions To move to move forward quickly identified stakeholders, right? Yeah, it's key here So if I go back to what we didn't do initially is the that kind of workshop when we have all the people aligns That's really key if we want to have No more issue later on so we didn't do that and it was one of the big the big problem So it was difficult to say we have that person as the owner of these actions It so because too many people involved and yeah so really quickly on functionality in OSP 10 that's that's helping us here one of the Features that we championed the stream for ups OSP 10 was a composable roles and composable services it's the idea that Triple O has the idea in OSP 8 and 9 and early have the idea that there's a compute role a storage role They control a role and these are sort of well-defined locked-in roles right a controller has You know horizon keystone etc. Etc a compute role has Nova etc. You know and the idea behind composable roles and composable services is that you can break these down and create your own custom roles You can compose roles if you will So you know you can you can move neutron around to different controllers keystone etc but more interestingly at least for me is We can have a third-party software deployed in roles So our SDN partners for example could have a role for their web UI and a role for Cassandra Or rather services for these that are rolled up rolled up into a role. Well And then put them on nodes And I just put a quick blurb on on sort of how that configuration looks like it's actually surprisingly simple And we can thank Steve Hardy for that amongst many others Is the all-time matter of I don't want to get an issue when I'm in production So test test test access is your friend and it was very amazing to to to meet all people Who deploy something or who made something to talk to how are you doing that? Oh, oh, it's amazing how we never think do that with your with your product Oh, we will trick-or-try. Oh, you are doing that. Oh, yeah It was very amazing very intensive week and at the end you start with nothing and at the end you have something working and failure Good good. We are good. So Every time you have to test of course we prefer to test before facing customer to have something really working But when it's challenging where you are not ready We have to be honest and we try to do it with a customer to say, okay Well, we are not perfect But we will do our best to provide you the better experience to be sure on production you will be able to have a few issues at least and be able to reproduce it Your your full platform in our annual side to help you and provide you quickly as you can the backfist and everything And for sure, we will continue with the next version for West P 10 when we will be ready to start the job Previously may contact the customer to have someone from the customer on site to be sure We are aligned with what you want to do what is needed and to do the same access to provide him the right documentation all to deploy now with the next version our products and SDN product Okay, so yeah, so was just kind of some summary of the lessons learned and few things that are important to do From our point of view so the as I said the usually the discovery design workshop that we do at the beginning of the projects is key is really important and Helps to avoid all the I mean many of the communication synchronization Issues that and knowledge sharing so to have usually all teams. It's more about the different entities It's not everyone of course Involved to to have the same level of information To share the detailed architecture. I mean at least the high level architecture But to start to discuss about the detailed one and to have the right architects involved To have the list of all requirements At least the one that we have at that time because it changes it changes a lot and to identify the roadmap with the Implementation targets and the the ownership So the you know the who is the owner of what is quite quite important with how we are going to communicate together Because that's that's something very difficult as well the tools used to the communication. It could be something difficult so We need to align upstream so because we are working here as You know open source projects So read out the asian partner and the customer early on the road map In order to look yes feature architecture and timelines Otherwise we will miss the Target so that's really important. Yeah, I think one of our one of our pain points with the scope creep here has been Because there are four independent work streams effectively, right? I mean if we have scopes scope creep after the upstream release has gone out the chances of or The effort involved in getting something new back ported into osp-10 for example or osp-8 is Exceedingly larger than if we had just added it initially So we really need to try as much as possible to lock functionality and say okay No, this is what you need at this date and this is what you need at this date Because otherwise we're always going to be sort of chasing the backport right chasing the new feature Which in this case is is even more painful than if it was a you know a proprietary product that there's one person doing so Something that is quite difficult, but very important is to have a shared platform Should integration platform, so we didn't have one that was quite difficult But that's something that's definitely is important to have something to test on the right architecture with everyone involved and We discussed about the DCI the the continuous integration the distributed continuous integration platform That's as well key in order to match the deadlines and to avoid. I mean the backfix later That's to fix things early in the process so make more hard first as it was some really the The key elements in the timeline that change almost everything in terms of relationship with the customer and with the partner because after that's time the Customer were much more was much more confident in our capability to work together and to deliver documentation and something that is working Yeah, I think looking back. So our plan was to do one per open stack release Looking back. I would have done more than one. Yeah, definitely one per release And then one maybe mid-release to confirm You know, did we have bugs back here that we expected to fix? Did they get fixed? You know, can we validate that? But the Difficult part here is that it is an investment. Yeah, so you cannot do that, you know Every month it will be very difficult But as soon as you do that You need to do one. I mean early in the process to the limit to demonstrate the value of it And as soon as you have this then I believe you will have investment for more. That's the idea If we do it every month, we should cycle right Paris Honolulu. Yeah So I think that's it Any questions Anything you guys would like to ask? I know we have people here from both the customer and the SDN partner. So you may get to Get more insight than you were expecting with your questions All right Feel free, I know where Yeah, so basically a lot of architectures now support Lee Spain Apologies and the networking. I know that's a feature. That's not currently in open stack director Do you know any plans to put it in in the near future what version that would go into I'm looking at a person who may be able to answer the question over there and he's saying Do you have an answer Mike? Let me run over there. I couldn't hear anything. Yeah, we didn't make you gonna be better They feature requests out there. It's presently being worked on But we don't make commitment to which release it's going into but it is being worked on pretty intently You can look into a lot of the work that Dan's done is doing upstream and follow along with where that is Thank you, Mike Any others yes So for automated testing, what did you guys use? Is it just a tempest s? So the automated testing Projects that we have is an internal DCI project. It uses the DCI agents that Distributed continuous integration is what we call it DCI And it uses rally tempest et cetera and numerous other sort of hacked together But definitely if you're interested in understanding more about DCI we can put you in touch with But basically the way by the way is a good start to start with with a tempest test but for example with us as the end you have more different stop and basically we work on a rally Rally implementation with different who is calling the feature of SDN to test service chaining a lot of stuff Yeah, so it is we use it with our partners and our customers, right? So anybody who has a specific use case be it, you know, we're Checking that the the I keep almost saying the the vendor name and not I know So we're it's currently in process to develop the the tooling to test the specific functionality of the SDN partner But yes, that is that is the goal Nice try It looks we are late Okay, thank you. Thank you. Thank you for coming