 All right, we're gonna get started. It's 1205. My name is Richard Schellberg. I run OpenStack services at Mirantis And with me today on stage is Anna Dipanu here who's a project manager or program manager at Mirantis and Christian Humner who's a cloud architect. They'll help me today with the presentation before I launch in that be nice to know Who we have in the audience here. And so first of all, do we have any open stack developers here a Few. All right. How about users of OpenStack or want to use OpenStack? All right. Good How about vendors, product companies? All right. Okay. Good mix. It should be useful for everybody So we're here to talk about project cloud cloud projects and how they go down and when we look at projects and the customers that we Work with we try to put them into this pyramid and at the bottom of the pyramid We have companies that are primarily looking for services and this can be you know in the worst case easy to and Otherwise rack space or Mirantis OpenStack Express outsourced services primarily and as we move up Into the pyramid a little bit more. There are companies that are looking for product kind of black box Drop it in Integrate with some of my support systems and this or business and operation support systems and Further up we get to companies that are really into open source and they do like OpenStack and other open source products for that matter and they'll pick components put something together But they really like the community and they like staying close to the trunk, etc And then finally at the top you have the true custom developers these companies have large development organizations They'll pick and choose code pieces from here and there and really build their own solution if it's a we usually call if it's a Stack we call that a Franken stack usually in in the office, but anyway as you move up into the pyramid the complexity and customary involvement increases significantly Another way to look at it is you have the true ninjas at the top and you get the small medium business at the bottom Today we'll be talking about the upper part of this pyramid because this is where a lot of the interesting stuff happens and some of the complexities and challenges all right to Go through this we organized ourselves into three ease we start with engagement level already before the project gets started we have to do a few things on the engagement side and then Anna deep will cover execution and Christian will cover some of the engineering aspects of these highly complex projects All right, so the first thing we have to do on the engagement level is to level set with the leadership of the customer and There's an expectation from the leaders who are heavily Invested in the effort that we're about to embark on and they believe cloud can be transformative technology They'll have a more agile organization fast at time to market fast to turn around for the developers etc and This presents a number of challenges so for example The project within the organization will have interfaces to other parts of the organization These can be the worst case external companies that you have to work with But oftentimes they're just other departments within the company and that's a challenge because those other Organizations are not agile The experience in agile process is sometimes a challenge even though there's a willingness and a desire to do an agile project That is not a substitute for experience. So a lot of times the experience level needs to be brought up overall And of course you have to have buy-in from all stakeholders the primary stakeholders The ones who took the initiative. They're always they're bought in but there are other stakeholders Operations teams networks groups etc. Who are impacted by this and that we need to get buy-in from them as well So that takes us to the stakeholders so from the core stakeholders their expectation is that they'll be doing a Agile project and so we'll be operating iterations. There will be Incremental results that we can see and we can use this will of course drive efficiency and Overall they want to get a adopt a more modern approach Approach to project execution and there are challenges with this as well So again, the rest of organization operates in a waterfall model typically. That's a problem Because of that and because of the Increasing speed that you're gonna see over time when you go agile The the external Dependencies tend to lag and you run into problems on with timelines, etc There's often also confusion around planning. So in some cases we're running into a situation that well It's all agile now. We don't need to plan. It's all in the backlog, right? That's a big mistake. You do need to plan. So planning is really important. You need to plan often and Before you embark on the project now in the on the other side planning is done elsewhere So there's a planning team that is doing a lot of the planning and actually it doesn't translate into the project and the agile model That we're doing so anyway, so we'll get into how we can solve this on an organizational level The expectation is that we're gonna create self-sufficiency throughout the organization push button IT I want the server. I can push a button, right? There's gonna be a lot of automation now when we're done with this. That's awesome and You know, we really want agile to be pervasive throughout the whole company when we're done these are the expectations organizationally and Of course, this has a number of challenges. There are some existential questions. They rise some organization or some departments wonder if they have a role to play after we're all done So that's create some resistance at times and they can be related to Operations where they've traditionally been very hardware focused and they now have to be focused on user experience So that's a change Networking the network topologies for cloud may not be aligned with various security security policies or traditional networking, etc hardware as well, they may have had a Process of acquiring a certain type of hardware and we're now suggesting that you should go with some You know commodity hardware of some other kind etc. So All right, so step one is to manage the expectations. We have to organize to manage expectations throughout With the core stakeholders and elsewhere in the organization. So we have to get the executive level buying from day one This is a transformative evolution of your organization and it has to really be Visible all the way up to the top You have to identify a project owner from your customer side. This is the Agent of change if you like this person should be empowered and be able to really push others to to get things done Decide or agree on a minimum viable prod product This is your first major milestone. This is something that is it has a tangible Useful application for your users. It should have the minimum set of requirements and features to be Able to handle core workloads and create value quickly This is really important and then remind everybody involved of what that minimum viable product is every day or every week Plan for gradual organizational changes, you don't need to change your whole company on day one You can sort of draw different parts of organization in incrementally as you need to rate through the project and milestones for that matter Identify external stakeholders all the time they're gonna be in other stakeholders later in the project as opposed to in the beginning Pull them in get commitment And identify communicate risks always assess risks not just in the beginning but throughout the the project That helps manage expectations Step two. This is a fairly typical governance model. You layer it like this. You have project execution Let's see if this works at the bottom And you put project governance one step up This allows you to separate the execution part from other issues that encounter that you encounter throughout the project You may have Escalations you may have scope creep or you will have scope creep You will have a number of different issues that you should try to get out of the project execution path right away And you deal with it at a governance level and that's where the stakeholders meet etc And of course at the top this is where the executive sponsorship and of course from us the vendor side Our account management leadership sits as well pretty standard governance model step three Establish an agile model. So everybody's heard of scrum perhaps It's a great agile methodology, but it's has some assumptions It assumes that everybody's involved in the project is a universal soldier essentially so given that the customer will have Most of time in these kinds of project the customer will have their team members participate as well You're gonna have people that are not universal soldiers They may be good in certain things and not so good in others and you need to plan for that. So pure scrum might not be suitable You got a easing to agile so you do that by accepting that everybody has strengths and weaknesses Set the sprints the iterations at three weeks not two weeks or one week It gives you a little bit more time to do demos, etc Plan carefully like I said before plan often replan prune your backlog Etc. You got to do that. You got to communicate these things Within the team and with the customer all the time You should not share people a lot of times you do you execute these projects as a number sub for Projects that are all executing in parallel can be anywhere from three to 15 parallel tracks depending on the size of the project Don't share people between these different sub projects. You can move people but don't share them between you're gonna lack commit you're gonna lack commitment to What you sign up for when you do that Also identify an overall architect can come from the customer it can be from us And a program manager on that one person who is in charge overall for architecture and a person who's in charge for the program Okay in terms of organizing Try to look at how you organize a project in three different layers There's a business layer a technical layer and operational layer and this allows you to do Essentially two things a lot of times people are just focused on the technology But by focusing on the business layer and the workloads that you're gonna run in your cloud You can actually get the users engaged early and of course they don't have a cloud to run on however there are you know Solutions for that as well You can go to rack space you can go to Miranda's open stack express you can get the users familiar with how Open stack works. In fact On Miranda's express we You can actually have a whole cloud and you can actually manage the cloud from hardware and up It gives you a lot of really good experience on on open stack Early on in the project operations is all equally important. This is usually the biggest issue We have once the cloud is ready to start Going into production the operations team if they're not pulled in early, they're gonna be Having all kinds of issues in taking ownership of the cloud. So they should be there day one They should be part of all the planning and they should be Part of the development of the logging monitoring operations tool tools that they they'll be using once the cloud is up and running All right Fine, I think this is finally set up the interface model So you're all up or operating in in a agile model, but the rest of organization is not However, you can talk waterfall when you have to deal with other people So just look at where you are in your sprints or in your iteration and see what the focus is For example, it can be on the deployment side For a while. Well, then you are in the deployment layer in the waterfall. You can be doing testing while you're doing testing So when you talk to others you talk about it in a waterfall way knowing that you're in a particular iteration or whatever Very easy to actually translate between the two So we'll talk about execution now and I so liken that to being handed the baby And you have to ship them or her off to college, right? So in some sense, you're left holding the baby The first thing that I would say is well, you are now one team The organizations that you go into whether you're the vendor or you're on the on the side of the Customer who's who's hired the vendor it all becomes one team So when you do that the first thing you do is clarify and verify the team roles that Rickard was talking about that were would then identified so now they may have changed or The organization chart that you had was formal, but there's always an informal Organization chart who's the go-to guy for program management? Who's the go-to guy for testing? Who's the go-to guy for architecture? They're not sometimes. It's not the chief architect. It's someone else You have to get you have to know those Definitely really find out the next thing You'll find is when you have these big organization the ninjas. They are geographically dispersed. It's almost a given I have had and then you have the vendor and they'll be geographically dispersed so What you got to do is you know Make some Arrangements for that just keep that in your mind They also each of these teams that you have they will have some interest that is that is that needs the cloud but is not Exactly aligned with just deploying cloud. So again make make some arrangement for that Keep that in the back of your mind and they will have a different timeline than the cloud deployment So they want something deployed by X and you your you want something to apply by Y and then there will be tension between that and you've got to negotiate that So make sure you know that now Excuse me the the biggest problem you'll You'll face Is you know, you know the technology you know what to do but coordination And it's coordination of not just it's not just people certainly people but Also your your systems You know your user stories your Documentation your processes so make sure that you are you start with that The other thing is in modern management, you know everything is matrixed The deployment team is where the rubber meets the road. There is one deployment team however stakeholders are there's hundreds literally and You have and guess what you have some responsibility to the stakeholder You just don't report your management chain But you also have responsibility and that's the matrix and it's not it's not like the movie. It's not fun so so then One thing you will notice is Excel spreadsheets are are not Good coordination documentation mechanisms right there and got to have online tools The only thing worse than Excel spreadsheets is emails You shouldn't have to rely on only our email system for coordination Again, sorry So look at look at all the stuff that you have that you know ticketing user stories source code control unit test integration test smoke test documentation all of these there are tools and guess what Your customer and the enterprise or and the and the vendor they will Invariably use different tools. So when you start the project, you know, there's a lot of pushback That someone's tool must be used at least use one right start with one and apologize later and it and as the project goes on your your tools will align But you've got to make sure that you are using a ticketing system from day one. Otherwise, there's got to be a lot of pain All right, so let's look at some of the challenges and I'm some suggested solutions that we have The I wanted to use my my French since we're in Paris. We have Boku stakeholders. We talked about that There's just make one source of truth, right? This is the ticketing system and it's not enough just to make hey We are using JIRA, right or we're using bugzilla whatever, but you have to make sure that Nothing gets looked at unless it's already in ticketing and that every single developer They even touch that problem. It's it's reflected in the ticket and that's that is hard to do because developers want to develop right and And then there's a crisis and people want to Want to know what's going on and they call you so your answer should be look at the ticketing system Believe me, that'll save a lot of a lot of problem and people will appreciate what you've done later on the critical dependencies that you have and this was in Rickards diagram With the other teams You've got to make sure you have a connection with those teams You know what their plans are and that you have a sponsor lined up on your side either the champion of the project or Someone that you have a relationship within that Otherwise you're not going to get deployed If the network engineering team doesn't give you the switches in time the load balancing team doesn't give you the load balancer Stuff in time you can't deploy so it is critical the Don't do too much too soon minimal what Be don't be ambitious about the MVP Demo demonstrate stuff first make small things work take out the risk and Do this in phases and then you have you have an opportunity to replan every time And you can you can then see what your velocity is You you have a transformer to change you want to do everything all at once don't do it That's not good. Oh that is going to cause a lot of Expectation mismatch, I don't know if it's unique to the cloud But it seems to be magnified by it because there's so many people who are interested in this transformative change They are they they are expecting you to build different cloud. They think that they'll get Stuff that you haven't promised or vice versa. You promise something that they don't want so every time Every day My program and a job is that I am talking to some stakeholder Clarifying what we're going to be delivering and and making course changes if there are differences This probably is unique to the cloud You know we use whatever it's Tempest or you know other tests that are there in the community, but they tune for different things right? What you got to tune the cloud for in the multiple ways you can tune the cloud everybody, you know It's if someone like is storage heavy or network heavy There's the you tune it differently so use the real test cases that belong to the users of the cloud Not from the team right there so because eventually the users of the ones who are going to say oh This is good or this is bad Understanding is a problem everybody has a different definition for cloud everybody And not everybody's heard of open stack that might come as a surprise to you, but not everybody knows what? Solometer is right, so you've got to get that and then also about agile all of these things you should make sure that there is Some common understanding and you fill the gaps with training or mentoring opaque planning and this has got to do with The cloud is seen as a transformative IT, you know our business technology or movement and The the the entire enterprises involved not just a small department in in doing that kind of Transformation so they there are people who have who are making plans for that transformation and they may not be talking to your team If they're not go talk to them Find out what they're going to do And and absolutely be involved in the plan, you know be in that circle of planning that they're Then the last one is you know lots of jobs chasing open-stack engineers right so Do not rely on that one person who knows neutron or Or knows you know how to tune your your cloud Always make sure you you have multiple people and expect your right and plan for it so Those are some of the suggestions no question. Oh do you? Okay, so Management has decided you want they want cloud the manager comes oh Yeah, good the manager comes to your office and says well we are going to get into the cloud space and first of all, of course there's many ways to skin a cloud skin a cat and So you're sort of left in the cold usually only get the very small chunk of information that you are Going to have to work with and then finally you Get told by the by the same manager. Oh, and by the way, we've hired this company to install the cloud for you So this is what we oftentimes get seen as guy who comes in folks people with a sharp instrument and makes it Takes it over takes your job over Makes you look bad to management because you Are not doing in the eyes of management are not doing the job and that's not at all how it is in reality we have The challenges of course you can do it by yourselves open stack is open You can download it you can install it you can configure it If you have to ever try to do that manually you'll find out it's not it's not fun And it usually does not lead to what you wanted to lead to so To ease the pain of doing that you can use deployment tools They will take away some of the problems that they will experience But even more so if you don't have somebody who has already made those mistakes and has seen other companies Make those mistakes. You're going to run into those problems anyway even the deployment tool is not going to help you with that second thing is we come in and First of all we sit down. We have a stack of paperwork and we are talking to you about this and that and the infrastructure and your team and Skill sets and anything that you can imagine of architectural assessment and Of course Yes, it's boring. Yes, sometimes these meetings seem like they are leading nowhere, but we have the best Practices that have we have developed in a long time of actually doing this in the field we with people with other companies that are Seeing the same problems that you have and then It's a better. We understand your Infrastructure. Yeah, but the better we understand the business problems that you face the Users that you are working with the better. We can help you Build a cloud that matches the expectations of management of the users and of yourself, of course and then finally you Yes, I used to do to think the same way when I was working at the company of Somebody from a vendor came in. Yes, I can do that by myself But we rely on you being able to do your job I mean you wouldn't be in this room if you weren't good at what you are doing people management doesn't send you here to if you are not actually among the top people in your company to do that and What we really are doing or what we really expect and hope for every time we go into the company and every time we Look into the set of faces that sitting in that bottom is that we will have a team People who are all skilled at their jobs and doing what they did what they do best a Few things that we encounter every time or in many times that we are doing cloud project first of all preparation you Your management signed a statement of work with us. There's something Relatively opaque and that says okay, we all build a cloud for this purpose and this is what we what roughly is going to happen But the real decision of what needs to be done is done together with you and this is in the preparation phase oftentimes You know between the statement of work and the actual execution. There's a few weeks and you'll think okay I should be filling in this piece of paper that they sent me or I should be Working on this but in reality People somebody the music comes in I want this somebody comes in I want that and in the end The day sneaks up on you Wow, they're here coming in so Taking a little bit of time and taking the effort to do prep work specifically prep work in terms of How do we work? How are we going to work with these people? How are we going to give them access to our environment? Are they going to be online or are they going to be on site here working with us? communication with us if you have questions say so right away the better the earlier the better and then finally procrastination Everyone has done it if somebody says I've never procrastinated in my life. I'm pretty sure that they're lying but in this case it makes sense to Keep that at least a minimum Except change. Yes cloud is entirely different from a prey from a classic data center infrastructure for instance you have Service you have Traditional storage you have traditional networking and a lot of this is now instead of locking into a router making the iOS changes and saving them You have some software that does something similar but different and you have to adapt to that But on the plus side You are going to be the elite in this field This is going to be bigger and bigger over the years and you are the people who are on the front line of this and You are making yourself more valuable and you're making your company and us more valuable if you are Adapting to what we are doing well and Finally, this is a team sport We Everyone sees, you know, this is my group. This is the other group there's little fires little Tossels, but in reality the better everyone works together the better people communicate with better people are See each other as Cool workers as teammates playing towards the same goal the easiest is for every one of us now Patterns to adapt this is things these are things that we find useful when we are doing a cloud deployment First of all best practices Everyone talks about them. It's one of the buzzwords. Well, I have those best practices and I'm using them but We have a few things that we tell people to do and did they all come from long experience of not doing them and Failing and Or experience experiencing failures and difficulties because we have not been doing them Usually it's cheaper to learn from the mistakes of others Then to make the mistakes yourself become the genie pick and making it You know, you don't want to Chew on the truth and leave of lettuce Let the other guy chew on the leaf of lettuce and I know and find out whether it's really tasty Recycled within reason. This is another thing that I've personally come across a few times lately specifically companies that have that set aside something specific for a specific project and I have to tell them This is unfortunately not going to Provide you with the performance or reliability that you need for this specific project If I say so I'm not doing that because I want to sell you something We don't sell hardware. We don't sell any kind of Infrastructure if I also don't have personal reasons I don't like new and shiny any better than I like something that's classic and proven but if I Come to the conclusion to calculation and through experience that this is not Going to work or that we will have to augment this with some new purchase then I'm saying that because I want to Advance the project and improve the experience that you have from that and finally this is on a trap that I have fallen into myself quite a few times favor stability don't Think that you need the newest version of OpenStack to become to be happy if I deploy Juno OpenStack right now in a production environment. They're probably not going to be very happy about it and neither are you So having tried and true is good Every once in a while you come across a problem where you really need a specific version and In some cases it really makes sense to go to that newest version and experience the pain because there is a way There's no way around the version that For a specific item that you that you need for for something But normally I would recommend do not upgrade every six months and do not get the newest version of the get go I would for instance deploy ice house right now and then in a year upgrade to Whatever is the Experience of the stable version at that point and one thing that you certainly want to avoid. I at least want to avoid it Is this? expensive down times and sleepless nights And I've had enough of both to know that I don't want them If you keep to all that you are going to have happy cloud users, and this is exactly what we are here about Finally, this is the bad part Certain things need to be avoided and one of them the most important of them actually is shooting from the hip Yeah, I'm going to make that change here. I know how to do it. I can make it reliably Although something might go wrong But what does the poor guy do who gets gets woken up? six weeks down the road at night To troubleshoot the problem that my change inadvertently caused he doesn't know anything He doesn't have documentation. He's going to sit there for six hours and swear at me Maybe he doesn't know that it's me, but I don't want to live by that. I don't want to Be the one who just was lucky enough not to be discovered to make a make a bad change. So Even if you're not employing employing CICD I would recommend to at least find a certain modicum of control over your change process. Otherwise, you are going to be in for the sad sleeveless nights and unhappy users Second thing is losing direction Anand and Rickard were both talking about the minimum viable product and the minimum viable product is very important because in this case We are working towards a common goal And if not everyone is aware of what that common goal is and we have this defined minimum viable product Then people are going to diverge and then you are going to get the zigzag Towers the goal that takes Extra hours extra effort extra money and unhappy management And finally and this is another thing that I've seen all too many times and not only in small Small companies or not not even predominantly in small companies because small companies have this way They have a guru hackle oftentimes who knows how to get how to get keep that running more or less I've seen that in great in big companies. I mean 1,400 companies Experimenting in production they have a test environment They have a dev environment and they still make a change in production just to see whether it will work and then figure out Oops, I didn't what do you don't and and this yeah most nothing is more expensive than unhappy cloud users you lose customers you you lose revenue and People are going to be unhappy and this is you lose reputation and this is the important point So and this is what we are all about together. We will build awesome clouds. You can't do it alone I can't do it alone, but all of us can Thank you very much. I think we have time for questions You talked quite a bit about minimum viable product as you as you talk to ranges of companies Is there a prototypical minimum viable product? That lots of people could deploy Yes Yeah, so That's a really good question I think it in general You always want to start with Infrastructure as a service and you may add a few items to that for example some other network functions such as low balancer firewall Some companies will what look at the business layer and they want to be able to enable certain workloads at the push of a button so you will That gives you kind of a template of the first minimum viable product But there's usually minimum viable or an MVP for each major milestone So it evolves, but you start with your basic service. I should be able to Get a server. I should be able to create a little virtual private network. I should be able to deploy a low balancer That's kind of where it starts usually Other questions Hello, I'm very interested in the MVP approach and I was looking for information about tools tools that are are Useful in order to try to make the MVP approach I don't really find them like free mind mind mapping first all of the tools on the MVP approach so the the tools that we have in place like User story tools like rally is one you can You can you can you can look at an MVP from the user's perspective and say these 15 user stories have to be enabled So there it doesn't have to be you know, what you're doing with the MVP is defining scope not so much You're that you know, there's At the end of the day, that's the what you're delivering But the tools for delivery do exist and I believe we've been successfully Reusing like ticketing we use JIRA for example Rally and and for deployment we use The ticketing system and for user stories so to rephrase that for infrastructure, we use the ticketing system and for for the business User use cases we use the user stories and those are work successfully. That's not to say that there aren't other tools Elsewhere on internet you have a website for some somewhere where MVP can be a very explained or detailed We should have your experience and all the experience about MVP. I saw someone which is Eric Ries or something like that in 211 in the had some experiments on that and he says MVP sometimes can't work Yeah, there's some it was for a website. I saw the six so we need to improve. I think MVP or Find what is the experience about MVP? I Do think that's that's correct. Yes, and we are in the early stages, right? Thank you Thank you, it's maybe more remark than the question this is about how to build the cloud But I think then the challenge will start to to maintain it For example, you didn't explain about to give a predictable roadmap for for our company to use the cloud When to go to next generation of the product? Yeah, so you can tell a little bit about that. So Definitely, Rickard did touch upon it on the operations if you can see if we can Get back to that. Yeah, we we told we said that it was very important But we didn't get into we didn't do about it necessarily But it is a day one problem that has to be be addressed and you need to have Team members from the operational side involved Especially, you know in in these more complex projects where a lot of time operations is provided by the customer themselves So you got to have people involved with this already you got to look at logging monitoring and metering primarily those are three things and Yeah, it's yeah, it's a it's a cultural change a lot as well a lot of it is but and then tools Automation etc becomes part and one more thing that a trend that will soon start to emerge is managed services where The other vendors will now be asked to run the cloud On behalf of the customer and then there's a whole set of tooling that you have to do you have to set up processes We do have that it was not in scope in this talk, but thank you for the suggestion We'll we'll definitely maybe next next summit will address that Excuse me in one of the sliders Here In one of the slides we have seen a term CI CD. Yes, what does that stand for that is continuous integration continuous development? So the The idea behind CI CD is that instead of making changes manually and implementing changes manually is That you have a defined process to do that and to use tools to do that specifically tools like Garrett for code review and Jenkins drink Thank you. It's been a long day So and Jenkins Jenkins for the actual implementation these tools have the advantage that if you use those In a controlled fashion, then you will have a lot less risk of somebody who is manually doing something making a mistake and also have a Backlog to see well who did what at what time so you can Determine this is where the problem was introduced and this is how we are going to resolve that next time And this is part of agile as well You have to have an agile model to really get the full benefit of a continue CI CD so what's your recommendations around sort of Versions versus stability you made the point that you know customers more said ice house now It's a stable product, but we've heard this in the summit about boiling upgrades being a key one for some enterprises What are your thoughts around so I I'll let Christian address some of those as a program manager The decisions you have to make are the following right one is features Which features does the customer need and where have aware are the fullest and Versus stability right so you you've got to make that decision Based on what the customer needs the second one is is about Then you might have to make a decision about well Is there a way to we can backport these features where we take advantage of whatever stability we have for this current version? That the customers using Versus, you know going to a completely new one the third one is upgrades upgrades are Definitely a big priority for for our customers a lot of the customers know that open stack is evolving and have Some plan based on that if they don't you have to let the customer know that That in place upgrades that they're used to in regular data centers don't happen yet with open stack So I don't know if that completely answers your questions, but those are some of the concerns that we address right, so there's No real rule of thumb of how to do that what I would suggest is first of all Looking at an existing version Let's say we're looking at ice house and see whether the features that matches what you need out of it Backporting features has its advantages in some cases the problem is that is that you are getting something that is support-wise more difficult to manage than Straight release if you have a straight release and you talk to our technical support they can tell you okay This is a known bug. This is not an own bug. We need to file a bug about this This is something that we have we can fix this is a configuration problem If you have something where you backport for instance a newer version of keystone or something like that Then it's a lot more difficult because then they have to go back our people have to go back and look at the Infrastructure and see whether they can replicate the problem somehow So a backporting I would only do if there is a good reason to have if you have a Features that the largely matches your expectations, and you only have this one piece that has to be different so about versions I would Definitely not use the version that is leading up to an open stack summit that like everything has been Juno right now I would at least not at the beginning of the cycle because there are still so many bugs being found and you are not the What you should not be the one who wants to find all those bugs? I mean if you really are into into a bug chasing certainly is good idea But in general I would for instance use the summit but the kilo summit everything Leading up to it has been Juno now That is probably the time that Juno is gaining the stability way where you might want to implement it and In the end the other thing that I suggest is talking to other open stack users Visiting forums reading blog articles because you can clean a lot of the stability problems for stability Status that you are talking about in a specific release by seeing what other people experience that are using the same release Do we have time for one more question? Any other questions? Okay, one more So did you did you ever have to say no to a customer be it because the customer might be too small or because Every day the case doesn't suit like could you elaborate on every day? Okay, I can give you a concrete example. I'm not going to name the customer, but I A short while ago I wrote an architectural review that concluded with we should not engage in this specific case The reasoning behind it was we had the customer had specific performance and reliability Golds and and parameters that they had to be working within and They did not have or did they simply did not want to commit the necessary resources to reach those The resource that was so short that I have from from my experience I had to say sorry this is not going to work and we do not want to be Or we cannot be part of this because this is going to lead to something that is not going to be sustainable No, it's it was it was money for infrastructure In this case, I think they I know which case you talked about but there are sometimes Misalignments between what they ambition to do with hardware with there are vendors who want to put the hardware solution together And they may want to run particular Kind of Opstack configuration or Components of open stack on it and when we come in and we look at that architecturally we find out that the performance Objectives of that solution will not be met. So if the customer is not willing to change To the recommendation we have it's not much we can do, you know, so Yeah, so so and when you're within a project from the project execution framework There is always pressure for adding another a new feature or or Enabling another user And if you have done a plan before and you know that that's risky And it can't be met then you can't commit to it, right? And you have to say no at that point that doesn't mean that you don't use our ingenuity to try and find a solution But it it definitely means that if you want to deploy something that is Meets the MVP then you can't have additional features At the last moment, but this is also I'm sorry. Also, this is not a flat no in the sense of Okay, I don't the customer doesn't want to do what I want to do This is a no that says okay This is what I see as the minimum that you will have to have to get your stability and performance goals and So can we can we make it a deal there? Can we fix that? This is what Richard was going to say now I was just gonna say that by trying to always focus on the business objectives first Usually there are other solutions a lot of times even the customer tends to focus on the technology too much There are many ways to skin and cat as Chris Christian said All right, I think we're out of time. Yeah. All right. Thank you very much good questions