 Check we were better organized when we actually wrote the book So welcome to the open-stack architecture design Panel I'm gonna moderate So this panel is devoted to Open-stack architectures, so we're I know a lot of people have been at the at the Kilo summit sessions. I know I've been some of the design sessions, but we're really taking it up to the 50,000 50,000 foot level and really talking about architecture This was actually a passion of mine. I had the idea to write this book About two years ago. I think I suggested it to Anne and she was like, ah, we'll do the operations guide first So what I'd like to do is Do you want to take over? Okay I'd like to go through and do a quick introduction of my fellow authors And then we're gonna we have some sort of canned questions and then we're gonna open up to the audience to ask questions So this is your opportunity to get the experts to tell you why your open-stack is not working So I'll start very quickly my name is Beth Cohen and I work for Verizon and I actually am not working on open-stack right now But I did Architect several of the world's largest open-stack implementations that actually didn't ever get deployed So my name is Mayside location. I'm an architect for Cisco based out of Jerusalem in Israel At the moment most my focus is on open-stack and architecture Hi, I'm Kevin Jackson. I work for rock space in the UK I am an architect in the private cloud team. I do Professional services on open-stack deployments Good afternoon everyone Scott Lowe. I work for VMware In the network and security business unit with a focus on open-stack and other open source related projects Hello, my name is Sean Collins. I work for Comcast in the new Trump project. I work for rack space I'm the information developer so tech writer in this crew other than obviously Beth And I mostly work on rack space private cloud and I guess 50% open-stack 50% RPC Hi, my name is Sebastian Bajiras. I'm from Red Hat. I am a principal architect that specializes in large-scale data systems and parallel file systems and large-scale system architecture I'm currently my roles of specializing in seph and open-stack integration Hello, thanks for coming. My name is Vinnie Valdez. I also work for Red Hat I'm a principal enterprise architect focused on open-stack work with a large customers and partners and make sure that we architect solutions and Integrate open-stack with seph and our cloud management platform as well Hi, I'm Anthony Vega. I'm a network engineer for the Comcast open-stack cloud team based out of the Philadelphia region Steven Gordon working for Red Hat out of Toronto a senior technical product manager there focused on open-stack compute and network function virtualization One thing just to mention there's two people which are not here And one of them is Kevin Chase and Sean Wynne which weren't available to come to the conference Nick Nick Nick chase and Sean Wynne The question is what did you learn as part of being this process of trying to work together as a team to write a single book What did you learn what was surprising about the things you learned and also? Mountain Dew M&Ms and don't eat in the room except when the moderators aren't there and Put a double space after a period and I'll come get you in your sleep. I'll kill you I don't care what you think you were taught in high school No, especially after a can of Mountain Dew you dead So for me the thing was that we actually got booked out within five days Which was actually a very good document which nobody I think anybody actually thought we actually could do that Yeah, I was surprised You know given the fact that usually you know writing a book is a quite a long-winded process It's very you know structured old way of doing things to actually have it as a you know a five-day sprint is actually quite I was actually surprised at how much we actually got done and to actually come away with a you know Finished article is his testament to everybody on this this panel I'm very readable without double spacing. I was very impressed by a lot of these people here What they've done in their jobs the kind of solutions that these guys have architected is just amazing You know, we really put our heads together and I really hope some of you out there have read At least some of the chapters out there. We were really proud of what we did and I think at the time At least for me. I wanted to fix so many more things But now we have it upstream and we welcome of course any changes by any of you out there And we have that opportunity to do that now one of the key things for me in the whole book sprint process was learning how to divide and conquer an Extremely large project an extremely short period of time. I think it was pretty well executed We broke things down some things you can do concurrently some things that you need to do in a more serial fashion Sorry to the few people who had to choke down the entire book and and rephrase it at the end but in all seriousness being able to work together as a team and to break out each individual section and write out the things that you know the things that you're good at and then Get outside your comfort zone a little bit and try to proofread and read through the sections that maybe weren't your forte was a really interesting part of the process Don't do it. I would definitely say being a writer alone and I did a I had a creative writing background So I left my degree and started working for Red Hat and then eventually fell into rack space and Coming in being a part of this team was extremely terrifying for me Like my technical knowledge is still developing and I'm learning every day and there's so much to try and figure out and My words of wisdom essentially is it doesn't matter if you're not a developer and it doesn't matter if you don't know everything You never will and the best experience was having these guys I mean I got a especially think Anthony and Sean here on the first day when I thought I was going to smack my head and Crawl into a table and cry because I knew nothing that they both just sat there and said no you you know how to use cloud Did you use it every day? I mean come on girl. You have a hotmail account You know it was that was really powerful for me and everyone came together especially by day five We all really collaborated so that was my big thing. Yeah, I think we were really lucky because Beth and Nick and a couple others really led the charge in the beginning to actually determine the format of the book from The beginning where we were going to do a topic based And we had a really nice structure from the get-go that we could then fill in the details. There was never any There was really no Significant changes to the structure after we agreed on that there was no having to like go back tear it all up and start from scratch And I think that really was the reason why we had we had we didn't have to rush at the end of it Was that we had a really good structure to begin with that we could then break into our subject Matter expertise and then fill in Yeah, that was that that was actually that's a good point The collaborative Decision-making process as to what we were going to focus on really helped kind of dictate the structure that we were going for instead of like, you know, the old school method of writing a book where Sending messages by a carrier pigeon back and forth, you know or email or whatever you'll call it It's a much longer process But we were able to accelerate everything by being in the same room with a bunch of experts We all decided on which direction to go on which technologies to focus on which ones weren't applicable You know this time there was a lot of stuff We left out of the book that we wanted to work in but we didn't have enough real-world examples to kind of put into But you know, we I think I was really happy to see that everybody was able to kind of like you know make concessions and give up some stuff and The end the final product was a lot more I Guess powerful than than it would have been otherwise So as someone who has written books, and I know a lot of the folks up here have done that I Would say that if you if you have the opportunity to participate in a book sprint I mean it seems really really daunting and yeah, and you heard Kevin say run or don't right But it does seem really really daunting But I would I would really encourage you if you have the opportunity to do it And I don't know if any more are planned within the community or not, but if no I Guess and see it another planned. Okay. All right, but when the plans arise for the next one I and you have the opportunity. I would strongly encourage you to do it. I think that it's a great opportunity to kind of break out of Any previous ways of thinking about things the whole process of creating the book Collaboratively focusing on content generation up front and then editing and polishing on the backside It's a different way of looking at content and then the distribution model, you know Is very different as well compared traditional traditional public publishing model So it's it's a really great opportunity to kind of do something different So if you have the opportunity or you want to help create an opportunity to do that to generate Material that'll be helpful for the community. I would I would recommend it It seems daunting, but I think it'll be worth it in the end. Yeah One thing I wanted to mention answer to Ken's question around if you're gonna do this It's only was mentioned in passing was our ever-present captors or facilitators Faith and Adam from booksprints.net. I think around day three there things were probably gonna go off the rails If it was just us, I think you need that external facilitation to settle some of those disputes and ensure that Collaborative atmosphere continues throughout the entire week. People will go in nuts. Oh, hello microphone. Yeah, I'm sure you all heard me anyway I would say the most important thing is to start out with the requirements You know, I mean, I know it's it's a simple thing, right? But you really can't know what it is you need to do until you understand what use case what problem you're trying to solve and One of the interesting things about the way the book is structured is that we have you know This storage focused approach or compute focused approach or whatever and that really does underlie Changes that you would make architecturally to a cloud implementation Depending on what it is what use case it is you're trying to solve and I think that's I think that's the most important thing Is understanding the problem that you're trying to solve and the challenges that are around that problem And then you can you can head into an architecture discussion around how best to structure the technologies like abundance involved to do that Yes, and really just real quick what it comes down to is business requirements and business objectives That's really what everybody or every every enterprise is trying to solve You know what applications to use to solve those objectives and how are you gonna do that? And we had these discussions internally and Beth might expand on that but if you look in the book you'll see that we actually called it user requirements and the reason was we we sort of Discussed back and forth that if a system admin or an architect or an operator were reading a book and saw the term business Objectives they would probably skip over it, and we didn't want you guys to do that Yeah, I'll add to that I think I was the big proponent of the the user requirements business objectives because I Actually in my real life play roles on both sides So, you know, I'm the person who takes you know goes out and listens to the customers and the sales calls You know which are not technical and then I have to come back to the technical people and explain Oh, well, this is what the customer actually wants to do And then the next step from that is to say okay, and how are we actually going to do it? And that's what this book does. It says this is the type of requirements It's a cookbook in some ways, but it's also a guide In the sense that it says these are the types of things that you need to ask your customers to get that those Requirements, and this is how to translate those requirements into an architecture Yeah, I'd say that when we went and designed the structure of the book We debated a lot about whether we wanted to be very prescriptive or more advisory And we ended up going to the advisory role because in the end, you know To Ken's question when you're thinking about doing an architecture design for a cloud environment There's no one perfect solution You're never gonna get everything exactly the way that you you envision it What you have to do is figure out what the trade-offs are of all the different things that you want to accomplish with your environment and then make decisions based on where you actually want that to land and You know each cloud is different if anyone tells you that they've made ten of the same exact cloud They're lying to you or they're not designing it towards their actual use cases So the intent here is for the book to be a sort of a guiding hand to say well If you want to accomplish X here are a couple of the options you can use to get there and here's a couple of examples But you know there's always going to be more research involved for every particular use case because they're all different How soon would the book become absolute obsolete? Well, I think the key thing about the book being obsolete is that now that it's part of the overall documentation project It's kind of on the onus of the community in some aspect to continue to let that evolve much in the same way We evolve the open-sac software itself We are anticipating that having this book and this content available and And being able to be updated and patched if you will through standard processes and procedures Would allow the users the community the operators the developers to make this a living breathing Document and not be obsolete at least that's that's the idea Yeah, and I think the onus dance to the question is before we finished writing it and we knew that going in But the key to the sprint process is really it's five days of getting as much of that content created and then you know There is still that bigger maintenance piece of the puzzle to solve as well to keep that up to date And we have the same thing with the operations guide as well Where work has to be done constantly to keep that up to date and this will be similar Yeah, but I think there's I think I remember having the discussion of because we are actually quite high level You know, there's there's a huge element of it. That's obviously not obsolete We're high enough to actually just keep on going for of course There's some very specific features of certain, you know, that was you know, but the software that was available at the time They're the bits that would need Maintaining but because it's an architecture guide. It's a it has a general principle of being Quite it's got a long shelf life Did you guys have any heated debates while you're doing the books print about the books print or about technical stuff? So I'll go ahead and throw mine out there So and this kind of touches on one of the questions can't ask earlier about anybody who wanted to participate in the future one of the things that I had to learn through the week was that I I'm sort of obsessed with automation and efficiency and the tools we're using where we're have a lot of room for improvement, we'll say and You know, I immediately wanted to switch this over to using get an ASCII docking a bunch of other stuff And there's obviously not enough time to do that in that sort of week and and I had to let go and I had to say okay, they're more important things their higher-party, you know content was more important, but We definitely discussed some of those things Because we can we certainly had some technical debates But I think in general, you know, if you look at the book, we don't have prescriptive commands for you to enter So we weren't arguing over Implementation details it was it was a really more high-level discussions and then how do we want to approach things what examples? We want to use maybe that was more of where we had some of the debates Well, there was two people that were strong advocates of using neutron So we were we were able to make that decision three three, okay so Yeah, we were lucky that we had a lot of people where there wasn't much overlap in the subject matter and a lot of the stuff like so storage For networking for Nova. We really had subject matter experts in each field. So there wasn't I don't think as much Disagreement when it came down and also as others said it wasn't really deep in the weeds for Configuration so we were able to step us step aside many of the common debates. I Think that another part of the debate that we that I actually like I remember that I remember is that a Deciding which technologies we were gonna leave in like we really wanted to work in containers But we didn't have enough examples so we had to kind of leave that out But this that is something that you know, we knew that over the next year We were gonna have a quite a few more examples and maybe we'll actually be able to put that in at a later time Or maybe the community can develop a guide Hopefully the community will develop a chapter based on on Containers so those those we had I think that was the biggest one There may have been a couple more there was a few about the structure of the book and like which technologies to Were more important at which times But yeah No, there was there was actually a fair amount of debate around where Where the examples would would fit in in which of the categories? so we ended up having a general category which The way I phrase it is it's it's the Amazon model, which is that you are Writing you're architecting for the 80% The great unwashed Or three guys in the garage as I like to say to my co-workers and So a lot of examples ended up being in sort of that giant bucket But then then when we kind of dig a dig down and we kind of say wow, were they really going in other places? I think there was a fair amount of debate around that so Sorry can real quick so one of the things that you'll notice if you read the book is About when it comes down to the technologies that we selected and this is something that we actually did have some debate on Because sometimes the options were limited, but we tried to stick to you know Foundation included projects specifically wherever possible and not deviate from you know the actual core elements of open stack that are You know community-driven and part of the project what we made a couple of diversions Some places because there just weren't any better options, but we tried to keep that as limited as possible I'll also just say real quick that There's nothing like being confined to one area for an entire week for about 10 hours a day that'll force consensus 12 12 14 so I think one of the things that comes up pretty frequently is in terms of Not just multi-site, but when you're scaling out deployments There's a couple of different techniques Available with different levels of support across the different projects and it's actually a design session on this I'm going to immediately after this but the fact that sells for example is a no for only thing is a big problem and Regions for most people leave a lot to be desired as well So that's still a real challenge for people I think selecting which of those techniques to use for their cloud when they're trying to scale up and out and For me just to continue what Steve said is that open stack is a living and growing thing It changes the whole time specifically every six months and not necessarily what you're designing for now will be able to Work or work the same way as you did when you originally designed your architecture So that's something that you have to completely take into account of how you're going to adapt to that change Every six months or every year depends on how you make your upgrade cycle, but that's something you also have to take So if you really wanted to and you wanted to build for the 80% model and reduced your feature capacity Then you could potentially architect only to the least common denominator So you can take into account that there are features that are existing now that have no planned application That have no discussion in the mailing list or over IRC about whether or not they're going to go away And you can implement with those in mind That's functional it'll work for like we said 80% of the use cases if you have something specific in mind And you have actual real user requirements that deviate from that Then that goes out the window But you know if you really wanted to you can take the sort of long-term support approach to things and try and run a release that you know has the features you need None that will be deprecated in the near future at least for the next release cycle or two and That have at least most of the bugs fixed that you're concerned about I'd like to add that One of the biggest challenges and I know it's been a topic of conversation at several of the summits is upgrade paths And I know we did discuss that It's not touched on in the book. It's touched on but it's not really gone into detail But that's actually something that you really really have to think hard about because right now the upgrade paths really vary between the different modules And in some cases can't be I was just gonna add and just kind of flip that round a little bit and it goes back to like some of the kind of Corridor conversations that are happening about having like longer release cycle or having like a dev release cycle for open stack and a Long-term release kind of model. I mean, there's nothing Saying that you need to upgrade every six months unless you need to and it's just standard IT management now But yeah Try and you know if the community could come along and actually, you know Try and slow down this this idea that we need to upgrade every six months will actually help a great deal Yeah, I I just wonder though, you know from a customer-facing perspective I think I find a lot of the time that demand You know in the community that you see is driven by the fact that we don't really have that 80% of use case Is actually covered in the core project at the moment and that's still something where you know Even Juno may not meet those needs kilo we'll see and so on, you know, we're not quite at that tipping point where There's enough functionality for that 80% of users that they can sit on a release for a longer time Intentionally, I mean the upgrade issues that Beth mentioned means that some people get stuck there anyway, but I'll add on to that so There's this whole other option out there in that, you know If you're architecting for something you're usually not writing the code for it in all cases But that is an option because this is open stack and it's open source and anybody can go pull the code anybody can go Suggest changes so if there's something that you really need to match that 80% use case and it's not available now Write it. I mean it's it's kind of a difficult thing to swallow from an architect viewpoint But that doesn't mean it's not an option for you. It doesn't mean that you can't you know bring the resources together and and Make an effective change and it benefits you as well as the rest of the community if you you know upstream that code Or in the very least file a blueprint, you know of what your desired features were So it's actually a good question What I'd like to start with was our brainstorming sessions what we did on the ability the first day was we actually Everybody took poster notes and wrote down use cases that we thought were useful within open sec And then we mapped them on to a board and we categorized them And that's actually what how we came out with the different chapters that you see in the book today So we can probably go through some of the examples just quickly, but you know dev test very very simply we had My favorite was a security checking, you know Password cracking things like that which is not necessarily something that we're gonna do in public But you know, maybe if you want to do internal security audits or or penetration testing things like that That might be something that you could use a generic general cloud for I Was just gonna say I think that the 80% in my mind the 80% use case is what? From at least from people that I've talked to is that the general-purpose infrastructure as a service cloud They don't necessarily have a dedicated use case. They're not necessarily deploying HPC environment They're not necessarily deploying what is intended to be a storage cloud for object storage or anything of that nature They're not necessarily doing NFV or something of that nature. They're saying hey, I need to deploy Infrastructure that can meet a variety of, you know workload demands from my internal or external customers That to me is kind of the the 80% it's that general-purpose cloud that you don't necessarily know exactly what apps are gonna run on it You have a rough idea you have a rough idea of the kind of architecture that is best supported And how to address that but it's not you're not necessarily designing with that that sort of that one thing in Mind at least that that's that's my my perspective Okay, so what are the rest of your building? Just out of curiosity I'd love to know I mean feel free to come up to the mic and ask this question About a specific use case a little more storage intensive since you can run some database stuff on there Okay, one of the ways I thought of this is it's really a set of levers that you're that you're adjusting based on your And that's how I think of it in my head Based on your use so for example a database is gonna be a lot of storage But it's also could be if it's a transactional database could have a heavy CPU or compute focus as well so And then another one would be You know CPU have a you know CERN probably does mostly CPU heavy. Although probably has big database problem, too so My question is How far you can see for the future of the architecture? I mean For as a like a user normal user. I see just the current version So how far you can see about the updates? I mean two years three years So you're asking about opensack itself not necessarily the book, right? Is that is that correct? Architecture so I mean as those who are actually inside so can you see how it looks like after two years or three years? I mean I'll defer to some of the others on the panel who may have a little more experience But I would say that all of us are as much on the inside as any one of you could be just by looking at the specs That are submitted for where the product is going I mean, that's kind of the nature of the way opensack is being developed is that as users and operators and developers We submit specs to say this is what we think the product do or shouldn't do Depending on you know what it is you're trying to do and then all of us can get a feel For what that means to the product and what that means to our implementation and therefore our design of the architecture Moving forward Scott's got a good point. This is an entirely community oriented process and community means participation There's nothing preventing anyone from coming out and participating in it in any of these designs If there's something that you think is important to you Jump on the mailing lists jump on IRC join either the open-stack operators group or the dev group if you want to participate in development and Ask questions ask about you know if it's network oriented, right? Maybe you're looking for like DVR or maybe you want more advanced networking features like routing Which is in blueprint and being worked on So you can always get a feel for the future of open-stack by taking a look at what's Currently under development go look at the blueprints go look at the the general vibe in an IRC channel or check the mailing list for topics That you're interested in you can probably find Pretty good information about not just six months out, but maybe even a year out as well Like I was just gonna sort of reiterate the same point But like the community this is one of the biggest communities I've seen and I'm not I haven't been in the industry very long But this is enormous like the first project ever worked on basically Well, I don't know if any of you remember a product project called Elis essentially that died and just one day the mailing list stopped happening and The amount of people that are contributing to this is phenomenal like I've never seen anything like this you know creative writing wise as well and I think that's extremely important to note that this will keep going as Long as we keep going and there's still passion for this and it's clear that the 5,000 people here are passionate about this project So you've got to sort of look at that way I guess and that's from my rightist perspective I guess as well You know documentation is a really really good way to get into this kind of environment and see what's going on and see how things are Going and where this is going because documentation is important no matter what some of you may think you know like None of us would be here without the documentation of this book. So yeah, just what Anthony was saying I think no one can honest honestly give you a you know exactly what OpenStack will be in two years roadmap I think there are areas of contribution where they're we're getting better at longer running efforts like what the guys been working on with IPv6 Noven Network to Neutron parody efforts Just some examples that I can think of at the top of my head where there's both the community understanding and Understanding from the technical community technical community to back that up that these things are important long time long term efforts And those are areas where you start to see some of those longer range plans We cannot get a better feel for what's happening in 12 months 18 months But I don't think you can yet get that picture across the board with the speed things are moving Some of some of the future is going to be dictated by Everybody's use cases. So like our use cases what we need if we need high performance You know systems for our databases with we need, you know massively scalable data stores to just keep things forever Data epoch in data lakes, you know all of these things are going to dictate where we go and it all depends on what you need And so when we when we decide and like when there's enough need in the community to kind of develop those things You'll see those changes and those you see those things solidify and precipitate, right? but I mean I think that you know Being active in the community and kind of letting us know either by blueprints is a really important way of getting us to Where you need and where we need to be looking and focusing our energies, so I'll just add to your original question about architecture specifically two years is I mean that's four major releases out, right? I mean, I don't think we can really design an architecture today See something that's gonna happen four versions from now But what you what's nice is that there are vendors including obviously I work for Red Hat So I have to say this but you know, we will support current versions of right now three years out So that's that's something that we can design architecture and still support you for that that long and I know other vendors have What I would add to that is that What the community is being very good about throughout is always been you know enforcing there's some kind of migration path as well So there's if something's being pulled out There's a cycle of deprecation where you know that the next release that'll go go get removed And it'll also only actually get removed if there's a clear migration path for you to get from what you were on before To that next step, and I think that's the important safety and as long as that's being enforced By the PTL's and the technical committee. I think we're in a good place Just out of curiosity kind of peaked about your question. How many people in here think that it's easy enough or That you have the access you need to see where the project is headed by a via blueprints Just I mean right like raise your hands like I know how to go check it And I know where the project's going and I feel comfortable doing that so not very many people it looks like Okay, so that's an area of improvement for those of us who are active in the community to help Everyone else who didn't raise their hands Figure out how I can go and see what's being worked on and who's working on it and what state it's in and that kind of Thing so all right. Thank you Consumably yeah This particular book didn't have that particular Mandate, but there is actually another book which is published on the it's not actually printed But it's it's on the open stack org site And it's basically how to get involved in the community and where the resources are and and I have to admit that I'm not real good with launch pad and ether pad and github and all that stuff because I'm not a coder So I find them incredibly boring Should I answer that question or the gentleman's question? Okay, well One of the docs things they did is is all of the specs that we file as developers are now being published on a site So it's like spec dot open stock org Yeah specs dot open stock org and that shows you The pieces of work that people are working on for the current development cycle. I Yield my time so I Think something that would help architecture like architects all over the place make better decisions is Examples bait tied to actual metrics So if people have examples of you know bandwidth requirements and like how you solve those or or those kind of Specific details that people shared with the community. I think that would help architecturally as far as like Components of open stack that would help make better decisions something along those lines to help record that and and Report that in a more, you know Accurate way so that architects can make the right decisions on which hardware to buy which direction to go with whatever You know networking storage compute, you know, that's what I think would be I mentioned cells earlier But not necessarily the expansion of cells to say neutron or other projects, but a concept like it to enable multi sites or at least larger horizontal horizontal scale than it's possible in a single cell And some of the benefits there without needing a second endpoint necessarily So the other thing that that is really lacking the community and it's sort of expansion of what Sebastian said is We really don't have a whole lot of test metrics of stress Tests on these environments. It's very difficult to do stress tests on cloud environments for obvious reasons Or maybe they're not so obvious But that's we really need that that feedback There's a lot of moving parts in these systems and changing one element can have Sort of cascade effects on other pieces of the environment That you kind of don't find out about until it crashes majorly ask AWS about that Maybe they won't tell you but it has happened to them So that's that's something that would be would be great if you know if you've had catastrophic failures And you've done some root cause analysis contribute back to the community to tell us about it So we won't do it again So one of the challenges I see a lot is is not so much Picking out how you want open stack to look it's more molding it to get it there and One of the things that I think I'd like to see when it comes to deciding on an architecture and actually getting it functioning is deployment tools it's it's right now outside of you know Dev stack with vagrant or whatever you want to use to build it building an actual production open stack deployment can seem like a pretty daunting task the first time you go at it and Getting all the pieces set up where they belong and functioning together. It is it's a learning experience So I think it yeah, it still is so some some better tools around deployment would be great Can we turn this question around to the audience? Good news and bad news we actually Printed a hundred books to give you guys bad news is the previous conference took them because I got here early, so I apologize In all seriousness though This is up at docs.openstack.org And it is a community project and it's a rolling living book So if if you see things you want from it go pull it down if you see things you want changed go suggest bugs against it Or actually file some some commits, so go take a look doc so open stack.org. Thank you very much everybody