 Okay, welcome everyone. Thank you for coming out. We're here today to talk about a book we wrote in five days, not really all that long ago. It was about a month ago, a month and a half or two. And, you know, six of us plus Diane Fleming over here wrote this book in, well, realistically we wrote it in, you know, three and a half or four days and we spent another day, day and a half actually editing the thing down in an attempt to have it make some sort of sense. And there was also one more gentleman who was with us at the time, Adam Hyde. He was a facilitator for the book Sprint. This is kind of a methodology that he developed during his time with Floss Manuals. He actually helped found Floss Manuals. And today we're just going to talk about that whole process of writing the book, you know, the kinds of things we went through, the kinds of topics that came up. But I'd like to put a little, a little slight twist on this particular panel. You know, so many times during the process of writing the book we kept asking ourselves, you know, what do people really want to read about? What is it they're interested in? What matters for them day to day? This was really intended to be, you know, the kind of guide that you give to somebody on their, you know, day one of when they're going out to start operating an open-stack cloud, you know. This is becoming, this is a job role, you know, operate this open-stack cloud for me. Where do you start? Well, one of those things could be handing them this book. So that was one of the uses that we envisioned for this book. And without further ado, I'd like to introduce our fine panelist here to start off. We have Anne Gentel. And do you want to kind of take 30 seconds or so and introduce yourself? Sure. So I coordinate the documentation for open-stack and have since Rackspace hired me more than two years ago. So go write docs. Nice. Joe? Joe Topchin. I'm a systems administrator with Savera. And so basically I'm an operator for most of the clouds that we deploy. Where do you operate those clouds? Thank you. From Calgary, Alberta, Canada. Excellent. Thanks. Lauren? So I'm Lauren Hoxstein. I'm the lead architect for cloud services at Nimbus Services. We're a little company in the DC area that does cloud services for engineering applications. And that's sort of like traditional engineering engineering, like mechanical and electrical engineers. And so I sort of wear two hats. I do traditional web software development. And I also administer our little prototype open-stack cloud that we have. The data center out in Ashburn, Virginia. Then my copious spare time, I volunteer on the documentation. Nice. And thank you for that. John? I'm John Prue, systems administrator at the Computer Science and Artificial Intelligence Lab at MIT. We operate about a thousand core cloud internally. My users, it's very much like operating a small public cloud, because I provide the infrastructure as a service and then the research groups consume that through the API. So even though it's an internal cloud, I don't really have much visibility into what my users are doing, which is a bit of a challenge that you're more likely to see in a public cloud than most private cloud deployments. And when you say we, how big is that team? The team that's operating the open-stack cloud, that's about a third of me. So it's the royal way. It's the royal way. There is a larger group that I'm one of the senior members of. We have about a dozen people that operate the infrastructure overall. And one of the things that brought me into writing this book is as documentation for those team members so that they could understand how the cloud works. I had to write our internal docs. You're getting ahead of the game. I'll be quiet then. Hi, I'm Tom Fifield. I'm a cloud architect at Nectar. Nectar is an Australian government project to provide cloud computing to the research sector. So we run a cloud that's this year scaling from about 7,000 cores to 30,000 cores. And we have a whole gamut of the research sector. Everyone from archaeologists to winemakers using our cloud. Yeah, that's really cool. Okay, and I myself, my name is Everett Taves. I'm a developer advocate at Rackspace. Previously, I had run, built, designed, and deployed Cactus OpenStack cloud into production. And that was one of the reasons I was included in this book sprint. My job has changed somewhat since then. Okay, so let's get down to it. I'm going to start with just a few easy questions. I'll first ask, you know, just a couple quick questions to the panelists. And we're also going to start asking some questions of the audience, because, you know, we wrote this book for you, literally, like for you, the people who are actually cared enough to come out to this panel. You know, we want to know what you think as well, the kinds of things like, you know, what what crucial topics did we miss when we wrote the book that matter so much to you, but are completely absent? Or, you know, there's something really important missing from that topic in the book, we want to know that. So you know, you can include that in a question to us, you know, where's monitoring, you know, where's quantum? We want to hear those questions. Was there anything in the book that made you throw the book across the room, you know, into the drywall? We want to know what the airspeed velocity of that book was. Okay, so panelists, why? Why did you want to help write this book? Starting with John. Well, I'll start because I started answering that earlier. So like I said, I have a team of about a dozen people working on the infrastructure within the computer science lab. And we are looking to move more to OpenStack to provide some of the infrastructural services that we're currently providing on older platforms. In addition to this infrastructure as a service that we're providing as a new service. So I needed to train my internal people and document our procedures. And to the extent that this was generalizable to the larger community, I thought the right place to do that was in the official documentation, rather than replicating that on an internal site, which is kind of hidden away from the rest of the world. Obviously, there's bits that are unique to my site, but I try and keep that to minimum. Be shy. I guess so one of the parts about my job, our cloud is actually quite unique in that it's run by eight completely different organizations using Nova Compute Cells to put it under a single API endpoint. So one of the parts of my job is to get these detailed design documents or architecture documents from these other organizations and review them. And very often they come in and you just really have to either fly to their location and explain exactly how OpenStack networking works or go back and forth for about 20 email exchanges to try and get through to them that maybe you should consider using flat DTP multi-hosts, H.A. instead of this other thing. And that's really an effort saver. So from my point of view, so I'm probably the person with the least system administration experience here since I come from a software development background. And I basically wanted to be, so I like writing documentation. And I realize that's sort of a rare thing, apparently. And I wanted the opportunity to just basically be in a room of people who had done production deployments of OpenStack. And so this was an opportunity to do both those things to generate docs and actually be around people who really knew what they were doing. So by the time we got around to writing the book, I had deployed three different clouds for Cibera. And well, the documentation was was still pretty good. And it got, you know, a cloud up and running, but there was still a lot of day and operational. And this weird thing happened to me. And this is why it happened to me. A lot of those things were missing. And a lot of in-depth troubleshooting procedures were missing as well. And I thought that would be very valuable to have written and given out to the community. And that was my primary motivation. Shall I go too? So I am completely motivated by a very experimental way to write a whole book, 232 pages worth. And I had done a couple of book sprints in the past with other open source projects. So OpenStack seems like a perfect place for us to try this technique. So Everett actually did a great job of helping me recruit these. I didn't know that Everett had worked with Tom and Fudo Australia and helped him set a PIS code. And so this group was just like this amazing, you know, collaborative group where we could try the book sprint method in a safe way where arguing was allowed and not too harsh. And I just felt like this group of collaborators could actually pull it off and they did. So my motivation was like, right place, right time, let's do it. And the foundation gave us 10K to do it. So thank you, foundation. Yeah, and my motivation was, you know, pretty similar to Anne's and Lauren's really, you know, I was curious to see if this would actually work. If this was really possible and to be a part of it, you know, whatever that meant. And also, you know, I knew a number of people already who were applying on being on the team. And I think again, it speaks to I've been a huge community proponent of OpenStack since the very get go. And I just thought this is an extension of that, just the power of the OpenStack community of being able to bring people together to do these kinds of amazing things, developing in this case a book instead of developing code. And then on the technical side of it, you know, coming from development background, I kind of wanted to make sure that the dev was in DevOps here, and, you know, illustrate to people how they can actually extend the functionality of some of the pieces of OpenStack without actually having to change the core code. So many times, operators need the system to do that, that one little special thing in their environment, you know, they've got some weird policy on their network or who knows what. And, you know, being able to show them how to do tweet those things in OpenStack, I thought would be, hopefully, you know, really valuable to them. And I guess we'll see how that plays out. So we talked a bit earlier in another session this week, just yesterday, about, you know, how do we keep going with the book? You know, we want to keep this thing up to date. This is a living document. This isn't just, you know, a book we throw it out there, we throw it over the wall and forget about it. We want to keep this thing up to date. So one of our challenges that we discussed towards the end of the sprint, the end of the book sprint, was how do we get experienced operators to contribute documentation? There's so much knowledge in this community, just an incredible depth of knowledge. But, you know, dragging it out of the minds of some operators is not an easy thing to do. So this is the start of, you know, as much a question to the audience as it is to our panelists. Would anybody like to take a crack at this? Did the questions get too hard too fast? So we actually worked in a system that just did HTML edits. And that was great for the in-person collaboration for that five days. And it gave us a neat pump right out of it. But what we've done since is bring it into the OpenStack documentation workflow. So you have a Garrett review, you have the ability for me to work on a chapter, John works on a chapter, all merges together because it's doing that on Git. So to me, I think that's the way it's a living, breathing document is because of Git and GitHub and our Garrett system. We can merge this stuff together. And I think the output is fabulous. The other thing it enables that's really interesting to me is translations. So we already, I believe, have a Chinese translation of this book done. And that's how it becomes living and breathing. It keeps opening up to more and more groups. And the comment is, I note that there's quite a bit of colloquial language in the book. And whilst there's nothing wrong with colloquial language, if your language is English as a first language, in translation, that could become quite a difficult issue. Something to the effect of a final example is if a user is hammering cloud resources. Now, we all know what hammering something can mean from a network perspective, a disco perspective, and so on and so forth. But when you translate something like that, someone's going to scratch their head. So yeah, apart from that, though, I reckon it's an absolutely amazing job you've done. So I guess as far as the actual contribution process goes. So I didn't actually write anything for OpenStack documentation for the OpenStack documentation team before this book. But afterwards, I've started writing and it's actually quite an easy process. And it's almost a lot easier than subinit code for review for OpenStack. You still have to go through a Garrett review process, but you're just sort of writing words and paragraphs instead of, you know, code structures, and you have to learn a bit of XML. So in terms of barrier to contribute to the book, it's actually quite low once you read through the how to contribute doc. So from someone who is new to it as well, it really wasn't that hard. I think hammering is probably an Australianism from me. Apologies for that. Indeed. Yeah, exactly. Just getting back to the comments about gaining contributions to the book. And I remember when I was standing up my first production OpenStack cloud, at one point, got an email from a colleague of mine, basically saying something along the lines of whoever touches the puppet config again, I'm going to cut your hands off. And yes, it happens. But I think we can really capitalize on this frustration and anger that builds up when you are doing a sysadmin task that or devops task even that is taking a lot of time, especially if it's OpenStack's fault. Even if it's not a bug, even if it's something that is more difficult than it should be, I think we should be encouraging people to vent at the documentation team or just put their vent in a bug report, just type something up so we can take it and not only add it to the book in case the code takes six months to fix, but also use it to improve OpenStack. So I think there's sort of a this is sort of an open problem in documentation is how to get feedback from sysadmins. And how do you get them to even just complain through formal channels so we can take that in and use. So I don't know if there's a plan for it. Does that occur to me? People here actually have a system administration background. How many people have of those people have contributed to documentation and an open source project and any open source project? That's pretty good. Thank you. So let me ask you people. So for those who how many people have contributed to the OpenStack documentation project? That's that's good. Thank you. So let me ask people who've contributed sysadmins have contributed to other projects but not OpenStack. What's kept you from doing anything even if it's just like, you know, complaining about some, you know, usability issue or a bug or anything like that? What's that? I could say the thing that prevented me from being... I just remember getting to like the now it's time to sign the CLA and it's like GPG and I just like my eyes glazed over. I shut down the screen. It was like, seriously, I have stuff to do. But I know that now that's gone. Yeah, I think that's a fair comment. You hit the the Garrett workflow wallow words and it's and especially if you haven't worked with Garrett or Git before it's you just want to get going, right? But now it's just a button. Right. Yeah. So great. It's not that bad. Actually, once you get going, it's not that bad question. So I think one of the things that's possibly missing in this is an appreciation that it's a relatively young project. There probably aren't a lot of people who have the experience. There will be. I think that's that's the number of people around here that are interested already in this. We can see that users are growing. The numbers of users are growing. I think that's great. I think there will be. This is a great start. I think this book is useful. I think it's going to help build this community. So certainly for me anyway, I felt that I was at a point where I could not contribute usefully until now. Thanks a lot. So I have one thing to interject. Tom was mentioning complaining to the documentation team. The acu documentation team is pretty much Tom, Lauren and Ann. The rest of us contribute bits and pieces here and there, but it's a very small set of people. We do have Diane now who's putting serious time into documentation. I want to forget Diane. Even though you are blocked by the table. So we're not looking at hundreds of contributors. So if you have this one little thing that tripped you up and you can even submit a bug saying that you were tripped up by it. If you can make it over that little hurdle and actually enter the five lines that will keep someone else, that's fabulous. But if you can get as far as saying this tripped me up, just in a bug report, there's not a lot of people working on it, but at least then it's on the radar for someone to look into. And I know that a lot of times people don't know the answer a priori. They have to go and research it or post it to the dev list and get an answer for that to put it in the documentation. And for the three of you who do that, I really appreciate it. I also just like to add one other thing. And that's if you're if you're assistant men, don't be shy about filing bugs against any of the OpenStack projects, not even just manuals. I've gone through this cycle between some of the devs that are sort of horrified to find out that assistant men are doing shell script workarounds and poking at the database because the devs don't know about problems that the assistant men are having because there's no feedback coming back that X is too hard or this doesn't make any sense. And there's like a disconnect there. And so if you ever want to, if there's something you don't like about OpenStack, you can file a bug with the ops tag against any of the projects. And they're pretty receptive. They may not fix something right away, but they won't just say, you know, no, this is, you know, this is stupid. We're not going to change the way we do things. Yeah, if you're if you're gluing stuff together with scripts that, you know, could very well point to a deficiency within the system that the developers want to fix, but don't necessarily understand the use case. Hi, my name is Keith. First, I want to say thank you very much for your work. It's very inspiring, inspiring so much that working with the OpenStack security group, we want to try to do the same thing. So in that vein, how important was the facilitator in making all this come together? And if you guys had to do it again, would there be a facilitator involved? We have not lobbied for the 10k. We're going to try to do this on our own. So give me some feedback, some directions and guidance here on that. Having somebody to shepherd the process is very important. Very important. That's how I had tried to do like maybe one day Doc sprints in the past. And then I went to another one of Adam Hyde's book sprints and lobbied for the funding for Adam Hyde to facilitate. Yeah, it's just I have participated in enough that I knew I couldn't facilitate myself. It's a special skill that you have to practice enough that I mean he he has gone to books prints where he literally gets everybody in the room, but then he gets cornered in the kitchen on day one. And everyone was like, we can't do this and you can't make us. So he had to work his way around like with that and through that and you're going to get the book you want. You don't even know what it is yet that you're going to get but you're going to get the book you need by the end of five days. So I I knew I couldn't do it with Adam. So yeah. And I found you guys an author if you're interested. Yes. Yes. Okay. Good. We'll talk. Yeah because I'm collecting people. Actually, I guess my comment is about the earlier question. I think a lot of artists, admins don't feel like they that they have anything to contribute or that it's good enough. And so there's this fear factor that, you know, well, it's what I'm going to do good enough. And I must admit I'm sort of in this in that I've just I haven't done for almost not yet. I just did for fedora. But skating this habit of, you know, gosh, I've seen this problem. And I know it took me half an hour or two hours, three hours to fix it. I should contribute this. And so one of the things you might do is look at mailing lists. And if somebody writes up a good answer to a question, send them an email saying, Have you thought of it contributing this to the documentation? Suggestion? There's one comment I'd reply to that with, which is the review process. We obviously, when we're going through this book, we kind of broke it up into sections and we'd write a section and then market for review and then one of us would come along and review it. And, you know, sometimes we found syntax errors, sometimes the layout was terrible. Sometimes there was words like hammering in there. But this this same review process actually exists in the normal open stack documentation. So it's possible to just send in a patch and then someone else will have a look at it and kind of mentor you through that. And I found that very helpful when I'm writing code for open stack. I mean, documentation is something that you can, if you know what you're talking about, just, I don't know, more Australianism to smash out fairly quickly. Whereas code, you have to think of all the test case and this kind of saying having this review process, I think really helps. And just being aware that the people are there to help rather than actually put a big red minus two on it and say this is wrong and this and that and the other. I think that's something where we can improve the time of our views. But it's something that might assist. So I guess another comment on that on on fear of contributing. And if, you know, if I ran into an issue, I'm going to look like an idiot for trying to post and either ask for help or say, if anyone else runs into this problem, here's this. And that's it's understandable. And but it's something that people really shouldn't be afraid of. This was the first open stack operations manual. So in a way, all of us were putting out these problems that we had. And there was, you know, there was obviously a chance where it's like, well, everything you guys did was completely wrong. But it didn't turn out like that at all. It turned out where a lot of the things, everything that we documented was incredibly well received. And a lot of the goofy mistakes that are documented in the book have been like shared across, you know, all sorts of different, like there was, there was an IRC chat log that Lauren pointed me to where someone came in and said, I'm having an issue where my instance just randomly locks up. And if I log back in, then everything's fine. And someone goes, Oh, check page number, whatever, of the open stack manual. And it's it's the VLAN on VLAN issue. And that was incredible. Because, yeah, I felt like an idiot when I saw I did that. But I'm really glad that other people have now seen the solution to that. So there really shouldn't be a fear of saying, Hey, I ran into this issue. It's yet to share whatever you have. And it's going to make a great knowledge base of of issues and solutions. I got to attend a presentation pretty recently. And the speaker was talking about contributing to open source communities, whether it's documentation or code or what have you. And he himself said, you know, one of the things and this is going to get a little touchy feely. But I mean to effectively sometimes to effectively contribute to an open source project, you need to be vulnerable, because you're really putting yourself out there for peer review, right? And all of the things that goes with it. And it's intimidating. I personally find it, you know, pretty intimidating. But, you know, if you kind of work through that, and you kind of accept that, oh, you're putting yourself out there, you're, you know, being a bit vulnerable. And if you can distance yourself from your code or from your doc and say, it's not you that's being reviewed, but it's your documentation that's being reviewed, it's your code that's being reviewed. It's, it can be actually quite liberating. And I think you become a more defect, more, more effective developer or documentary. What a Freudian slip that was. Okay, moving on. Did we have any other questions or should I ask another one? Great. Just sort of humdrum question. What's the release cycle going to be? That was my question. That's not a humdrum question. It's, I think, an undecided question. We did have a session yesterday discussing, you know, what is the life cycle of this book? How, how do we contribute to it? How do we, how do we release it? I don't know if there was a decision. What there were opinions. Yeah, so I mean, this is the perfect question for the audience. I mean, like one of the things that came up in that session was, Oh, there's, there's a handful of groups doing continuous integration, continuous delivery. So, you know, the, the doc, the book just has to be updated constantly, simply not realistic. And in addition to that, not everyone is doing continuous integration and continuous delivery. Most people only have the resources to upgrade every six months or what have you. So what would be useful to you? You know, when, when do you upgrade your open stack clouds? And, you know, that might be presuming a bit much that everyone, or even most people here have a open stack cloud right now to answer that. Or that they've upgraded it. Or that they've upgraded it. Yeah. No, and, and intentionally so that wasn't one of the objectives of the book. But once you get past that point and you want to learn how to operate that new version, let's have an answer. So keeping in mind, if you look at things like the O'Reilly texts and that may or may not be a good example in the context of what we're trying to achieve here, but, and given there's only three core people actually on the team who are actively writing this stuff, I would suggest maybe a yearly update cycle would be possible. I think that would be maybe realistic to say that maybe every year the group should get together and, and, you know, pen, pen and update. Because within the space of a year, you've got those two sets of release gaps. And maybe that's worth it at that point in degrading. But yeah, beyond that, to say every six months, I think is too much to say any more than that is too much. So when you see, like the open stack operators fall some guide and now that Grizzlies out, does that actually put people off from reading the book knowing that it's, it's, you know, as the word say, it's a release behind. But even though a lot of the information is still very relevant, so that you don't find a problem with that at all. Do you think that tagging the book with the false and release is a good idea or it should just be left out completely? Maybe additions of the book are important. But I don't think you need to say it's false and release. It's a grizzly release and so on and so forth. But additions in a year number or something is probably appropriate. Maybe others have other opinions. I think that it's important in the book to say which release it covers. But you shouldn't have the book saying this is a grizzly release. A slightly different question. Grizzlies just come out. And if I'm not mistaken, the documentation is still fulsome. And so if you want to go see grizzly documentation, do you need to go to the Get Repository and check it out? So we always publish continuously to docs.opensack.org slash trunk. And I mean, we might even merge like five times a day some days. Tom's pretty prolific there in Australia some days. So you can always just substitute slash trunk for slash fulsome. But one of the things we talked about yesterday was we need to release the grizzly docs, but they're not done because the installation isn't tested. So as a doc team, we decided last week, let's release after the summit, after the install guide at least is tested. After that, we'll collect bugs just like our timed released code does. Be useful on the main page to say this is fulsome. And if you want to see the grizzly documentation that's in progress and not tested, go to this. OK, yeah, there might be useful for you guys to talk about how you chose the reference architecture, like how you made those decisions because that was a big part of the process. Oh, do you and I want me to mention that? Sorry. I was going to say example architecture. Yeah, example architecture. Reference is just such a loaded term, but but that was actually, I mean, that's that's a part of the discussion as well. John, do you want to hear? Yeah, so there's been a lot of talk of reference architectures in this room today. I don't know how many people have experienced that, but obviously we need something to talk about. You can't talk about it in a vacuum and you can't talk about all the different ways you could possibly permute all the different pieces together. So we kind of took a target of what we operate and what we've seen people operating in in mailing lists as sort of the best covered cases and the most popular pieces in use. So we use KVM as the hypervisor because that's what a lot of people seem to be using. Mostly made our choices with with that idea in mind. With the user survey, I guess we get to take a look at that and see if we were right. But that that's mostly, I think, how the discussion progressed. What did you think, Lauren? Yeah, I don't know if I have that much to add there. I mean, we had to write based on what we knew, right? So that's one of the reasons you don't see quantum in there. And so you have to and then this is a general problem with documentation, right? I mean, we, you know, if we haven't installed it ourselves, we can't. It's harder for us to write that documentation so we don't know if it's going to break somewhere. And so we this was like a subset of everyone's knowledge or maybe a super set of the example architecture was a super set of what everyone had a direct experience with. So they could actually write what they knew. I think you made a good set of choices. And I think you were also pretty clear that you had made these choices and that they weren't the right choice. They weren't the right choices. They were the choices you made. Yeah, that's good to hear. I have a general question actually. Do you plan to also incorporate questions or discussions that happen on other online public forums like like ask.openstack.org or other places where people ask questions and there are others who give them answers. I just had an example of this where I had a problem with some swift configuration. I asked the question and eventually I was able to debug it. I put the solution in there and I just later found out that one of the engineers who was active on OpenStack actually took that and put it into the documentation. This apparently has happened in the last one week. I didn't even think about actually adding to the documentation. But he already did that. But I'm sure there are many such, I guess, bits of information or maybe even longer discussions that are already there on other public forums that could make it into the ops guide and make it into the book and that will be useful for people. So I guess I can comment on this because this is something that I try and do. I read the mailing list every morning and if there's something in there that crops up as something that is a common case that people encounter or something that's a bug, try and add that to the documentation. The problem is there just aren't enough people to actually do things like go through all of our Start OpenStack.org or even the old Launchpad answers site and take all of the caveats and pointers and corral them into a document. If you'd like to start doing that, that would be fantastic. But yes, we do try and do that. It's just basically limited resources, which means that we haven't managed to get all of the community's knowledge together in one easy to replace. Yeah, I mean, I would say I like agree entirely. So I do something similar. You know, we opportunistically, you know, people who work on the docs, if we see something, we'll try to slurp it in. I try to do that with blog posts. So if anyone there's like plan at that OpenStack.org, which aggregates a bunch of different blogs that are out there, there's often a lot of good content on there. If I see some, if I have the time and I see something that's really good on there, we try to contact, we have to contact the author. We can't just copy and paste it in. Almost everyone, I know it's ever said no. I mean, sometimes people don't respond to me, but typically if they respond, they say yes. And then we slurp it into the docs and you get a lot from the mailing list. But it's really just a matter of resources. So we do it when we can, but there's just much more content out there than there are people who can make the edits and put it in the docs. OK, so it's six o'clock, I believe. That's our wind up time. Did anyone have any closing comments they'd like to throw out there? Besides pull requests accepted to the docs and standard open source throw it. I also want to thank Rackspace for donating these books. They actually purchased the print copies to give to all you. So I do have to send a shout out to Rackspace as well. They hosted in Austin, Texas. We brought in Yummy Tex Mix and then they turned around and bought books for all you guys. So thank you, Rackspace. Thank you, everyone.