 Hi, thank you for coming to my presentation. Don't just consume, participate. Talk about operators in the community. I'll be talking today about how as a community, we should reevaluate the relationship between the people who build OpenStack and the people who operate OpenStack. And this is important because how we build and think about servers is constantly evolving. Well, how do I know? I'm Jay, I'm a sysadmin. I've configured systems, monitored them, woke up at 3 a.m. to fix them and solve their problems, but actually, I'm not. I'm not a sysadmin anymore. That's what I used to be. I'm a Linux engineer now. So what really changed? Instead of working on a server level, you kind of start working on an environment level. You start thinking about whole systems and how they interact. But the basics are staying the same. You're deploying and managing infrastructure to help the people who count on you get things done. But even this is wrong. I'm not really a Linux engineer. I'm a DevOps engineer. So this is more like it. Who configures systems by hand anymore? I have my little robot chef who configures them all for me. But really, I'm doing the same thing. I'm providing infrastructure to people who need it. And only the tools I'm using have changed through this transition. But even this, I'm not really a DevOps engineer anymore. I'm sorry, I'm just, I keep getting, I'm a developer. That's it. So really, it's a little over the top, clearly. But the point is that we've all held different titles or we may all hold different titles and we have opinions about what that means about what we're supposed to do, but they're just words. We shouldn't let our titles constrain us because despite what I've been called throughout my career, whatever title was placed on me, I built infrastructure. I cared about customer problems, be they internal or external. And I wanted to bring them infrastructure to help them solve these problems. But actually, I messed up again. I recently got manager. So basically, now I just give presentations at the OpenStack Summit instead of, but that's even not true because as a manager, even though my tools have changed from technological ones to kind of personnel and organizational ones, I'm still doing everything within my power to improve the infrastructure for the internal and external customers even if the ways I do it have changed again. So what does this really mean? It means that labels don't matter. We all, if you're in this room, you at least care about cloud infrastructure as a baseline. It's actually likely that at this point, if you're in this room, you're an expert in some portion of cloud infrastructure even if you don't realize it yet. Because we all consume cloud infrastructure in some way. We all operate or build cloud infrastructure and most of us do both. It shouldn't matter that our tools are different and that our areas of focus or our titles are different. And I'll give a little bit of a personal story here. About a year and a half ago, I moved to another team in Rackspace which gave me the opportunity to work on OpenStack Ironic as part of a new project. At the time, I would have preached to everyone that I was a DevOps engineer, that I didn't write code, I didn't know how, I didn't want to, you know, that's not my job, that's what the other people do. But over time, I found out as I worked with my local coworkers and my upstream coworkers that really it's just a tooling difference and that I can use Python as my tool in the form of a software project to do a lot of the same things that I was doing with operator tools before. And we're all focused on the same thing. But if you're an operator in this room and like that, I've been there and I still am there, you don't have a lot of time. So the question at the top of almost anyone's mind when they saw the title of this has to be what's in it for you? So it may seem like you don't have the time that it's not something that's in scope in your job but I view it as an investment. Getting involved in the community is investment in yourself, in your career path, in expanding your horizons and developing professional contacts. But it's also an investment in your environment. And that's because there's an infrastructure revolution that's ongoing right now. OpenStack is moving faster than ever. New ideas and tools about infrastructure are emerging quickly. And that's proven because more brain power than ever is being put into how to run infrastructure. It shows an open stack. For the Kilo integrated release, over 1,500 different individuals had contributions. And OpenStack is moving that fast because the industry is moving fast. We saw a demo in the keynote of Magnum which spun up a cluster of containers at the push of a button on a machine that was booted with Ironic. Something that would have been impossible because Magnum didn't exist 10 years ago. Six months ago, it would have been impossible last summit and even 12 months before that, Ironic maybe wasn't mature enough to have a good, easily consumable product. But this revolution is being enabled by software because software allows us to develop and iterate quickly. And a lot of us have been taking kind of six month or larger bites of that. And I don't necessarily think that we have to change that on a code side as much as we do on a knowledge side. When you're kind of at the end of the fire hose of the release, there's a lot of information that you're trying to consume in a short period of time to upgrade your environment safely. But realistically, you have to implement continuous learning. And then later you can implement continuous integration, continuous deployment if that's what you choose. But the continuous learning is very important because being at the end of the fire hose, I can barely consume what's changing in Ironic and I work in it every day as my primary job. If you're trying to consume it all via a release notes or something at the end of the cycle, you're gonna be disappointed and there's gonna be things that are missed. Another big reason you should participate is to advocate for your use case in your environment. I've seen many different environments of OpenSec installations that use different parts, mix and match them in different ways, use maybe obscure features that other people maybe even didn't know exist. But decisions are made all the time on a mailing list or on IRC or in design summits about which of those things should stay, which should go, how they should evolve. And if you're not in those conversations, something that you depend on may get dropped or a feature that you may be looking forward to might not happen. A good example of this is actually in the Ironic Python agent, which is the upstream project I primarily run on. I work on. We use systemDNspawn containers to run Ironic Python agent in the RAM disk, but we thought that that was a fine use case. They allow you to express capabilities so we can say, allow us to do crazy things to provision hardware, but then we go to upgrade the CoreOS image and we find that systemDNspawn has broken our use case because upstream made the assumption that no one would ever do certain things in a container and no one was in that community to stop and speak up for our use case. And another good example of participation is how we solved it in the end was by participating in the community, getting on the mailing list, going back and forth and actually end up spent today writing a patch to fix it. This sort of stuff can and does happen in OpenStack. Our APIs are versioned, but there's often a time when there'll be a mailing list message or an IRC message about a minor feature to get removed and if you aren't reading that, you're gonna miss it. So that's a couple of reasons. There are some others that are kind of sprinkled in throughout, but I wanna talk about how you can participate. And there are lots of different ways to participate. Include you can do code contributions in review, that's kind of the obvious one, that's what a lot of people think of when they think you're contributing. The spec writing in review, writing good bug reports, supporting other operators, troubleshooting test failures, or even building OpenStack project infrastructure. But we're gonna dig down into each one of these items and talk about how operators bring a special knowledge to the table that can make it useful in doing this. So if you want to contribute actual code though, that's fine, that's great too. I would suggest looking for bugs tagged low hanging fruit if you haven't contributed before and you wanna get a start. And you can ask the IRC for good things to work on or maybe if you're not confident in your abilities yet get someone to help mentor you through it. And if you don't know Python, that's perfectly fine. OpenStack has many repositories in many languages. There are entire projects written in Bash, there are entire projects dedicated to documentation and install automation. And even as we're gonna talk about later, there's an entire set of repositories that people contribute code to that actually build the infrastructure that the developers of OpenStack use every day. But if you don't wanna actually contribute code, you can review code. The big piece here is don't be afraid of reviewing code. I've talked to many operators, many people who claim that they aren't qualified to review code. I don't know Python well enough or that was already reviewed by someone more senior. But as we established earlier at the beginning, you're an expert on some aspect of cloud infrastructure. Already, if you're in this room, you're running it in production, you're an expert on it. Use that expertise when you're reviewing the code. Oftentimes the people who build OpenStack every day don't also have the opportunity to run clouds especially at scale. And so that insight that you gain can be helpful when you're reviewing code. Coder view can help the reviewer as well. You could see changes in parts of the software that might impact something you particularly care about or impact the scaling issue you've encountered. And generally speaking, being more familiar with how the code is organized in OpenStack can help you when you're having that 3 a.m. call because your cluster is down and you have to fix something. You don't want that to be the first time you've looked at that piece of code if it's breaking you. But again, the biggest thing is to be brave though. Read the code, put your best effort, vote on it. If you look at that and you say, I don't really feel like I gave that a great review but you were able to see it, you should put your plus one or your minus one on it with a comment and say, I'm not sure I understand all of this but I looked at it for X, Y and Z and I didn't find any issues. And that extra plus one as a core on some projects in OpenStack, I find it helpful to know that an operator has looked at this and says it's nice. And to be honest, if you do that too much, people aren't gonna call you an operator very much anymore. You get to, because titles don't matter anyway. But you can also review specs. Now what is a spec? A spec is used in many OpenStack projects. Some of them use Blueprints instead but the same concepts apply as a form of requesting and also planning new features. And they're kind of varying levels of detail from project to project. But many specifications have a big impact on operators. So you should review them. Things that you can look for when you're reviewing specs as an operator include that if it's a new feature that you understand how you'd use it. That you actually think through how would I implement this in my cloud and make sure that the spec reflects that reasonably because oftentimes that gets missed. Look at potential performance bottlenecks. If you know there's a performance bottleneck in your cloud and that spec is going to touch that area of code, make sure it's not gonna make it worse or even better find a way that it can improve it. Things like does the client library need updating as well because there's nothing more frustrating than having your client tools broken after a server upgrade. And finally, is this internally consistent? Does this spec solve its own problem statement? Note that nowhere in there do you have to have knowledge of software architecture to make an impact. You don't have to have written a certain number of line of code to get in. There's no gate, it's open. The review process is open for the specs just like it is for the code. And there are even some projects. I'll give ironic as an example, I think Cender has a lot of the same problems where we have hardware specific drivers that are being written by maybe people at a hardware company and don't get anyone looking at them until they're actually used. So if you were using a piece of hardware and a driver is being written for it and the spec is up, please take the time and do a real good look at it because as we all know, what the hardware says it does is not always what the hardware does. How the protocol says it's supposed to work is not always how it works in the real world. And it can be helpful to use our experience in the real world to direct people away from that. So what about good bug reports? And there's a reason I put good here because there's a big difference between saying something is broken and equipping someone to fix the problem. So what makes a good bug report? Detailed reproduction instructions which must include how to set up the environment to reproduce the issue, which means your configs need to be there. Your full version and environment information need to be there, including what distribution you're using, the libraries installed, what pit freeze says your dependencies are if you're installing via that way. Literally anything that could be relevant. Sorting through the information in the bug later is a little bit easier than not having it to solve your problem in the first place. But if you want to go above and beyond, instead of having to provide that information, you can go boot DevStack. You can boot DevStack on the stable release that you're running if you are and reproduce it there so that your environment doesn't matter, or you can even try to reproduce it on master. Additionally, you can find the change that broke you. We're usually familiar enough, especially if you've been looking at the code to understand what pieces of the software could be impacting, and going in there and looking at recent changes or change log can sometimes help you pinpoint that. And having this information there means that your bug is more likely to get serviced because someone's gonna see it, the information's gonna be there for them to solve the problem. Another important way you can help is by supporting other operators. And yes, I do mean technical support, help desks sort of things. There are lots of questions about how to operate OpenStack. In fact, documentation is maybe one of those places that we don't do so well in that actually operators can contribute because documentation's code as well. But you can answer questions on IRC. And IRC's actually a good point that I'm gonna hit on a few times. This is the water core of the OpenStack in almost all OpenStack communities is what happens in IRC. You should join the channel. You should be conversational. Even if you never intend on contributing otherwise to OpenStack, being active in the IRC channel is a form of participation just by itself. You can also look at ask.openstack.org or use your operator mailing list and help people. And why? Well, I'm gonna sit here and tell you it frees up developer time to build OpenStack. But we all know that's not the real reason we should do this. The real reason we should do this is we've all been the person asking the question. We've all been there trying to do a new thing and not being sure how to do it and getting help from someone. And part of being in an open source community is taking on the responsibility of passing on that knowledge to others. And when we fail to pass it on in documentation, sometimes we have to pass it on in one-on-one help. So another way is by troubleshooting CI or test failures on patches that you care about. And this is actually a process that is gonna mimic what us as operators have done many times in the past in troubleshooting our own production environment. And basically here's how it works. Every patch that's submitted to OpenStack must pass a battery of tests. Some of them from just level of unit test all the way up to spinning up a mini cloud inside DevStack and functionally testing against it. But these tests fail. They fail for a variety of reasons other than the code being broken. And sometimes in the worst case, even the test on all changes in a project can fail and really clog up the pipeline. This is sort of the emergency nature of how these have to be fixed. Because every day that that's clogged or every hour it's clogged is an hour things aren't getting merged. And in OpenStack that can sometimes be a problem. So what do you do to troubleshoot? You read the logs, you look at what code or maybe dependencies changed in the environment. But the reality is you already know how to troubleshoot these problems. You have a production OpenStack cloud running if you're an operator and you can go in and apply the same knowledge you use to operate that cloud of troubleshooting these test failures. The logs are actually very well documented as well. And this is incredibly, incredibly high impact. Because through the work of one person on clogging that pipeline, you're unleashing dozens of developers to be able to get their code in again. So that's a good effort to value ratio for your input. And as a note here, troubleshooting doesn't necessarily mean you have to fix it. Fixing it can be a rabbit hole sometimes, I understand that. But sometimes just by looking at it and by pinpointing exactly what broke, you can accelerate that process. Because if you determined that every test that was running with version A of a library is passing but version B is failing, just pointing that out as often enough to enable very busy people in infra who fix these sort of things to fix them. Just research, comment on the bug or on the code review or even on IRC, and then someone can maybe help turn your diagnosis into a fix. Another kind of finally, a way that you can contribute is actually in the project infrastructure itself. OpenStack is a huge user of clouds as well as a builder of it. There are lots of tools that are used by OpenStack developers that are managed kind of in-house by an infrastructure group in OpenStack, the infrastructure project. Tools like Garrett, git.openstack.org, Storyboard, the various IRC bots and services that we use. The entire integrated gate that I was talking about before, Etherpad, the Wiki, the website hosting for the docs. All of this stuff that we consume as users and as developers of OpenStack are built by essentially people who are doing DevOps using Puppet. And these are all technologies you're familiar with. There's an ELK cluster with Elasticsearch, LogSash, and Kibana that I'm sure many of us have in our environments as well. And all this is built using Puppet. And what's really crazy about this is I've never seen a production environment documented as well as the open source production environment for OpenStack is. You can actually go to ci.openstack.org and see where everything is, see where the pieces are, try to fix them. And I'm sure if you have willingness and you don't know which direction to point it, if you talk to anyone in Pound OpenStack Infra, they would be happy to mentor you and point you in a direction of help because more hands are always useful. So now that we've talked about how you can help, let's give a few tips. If you're a first-time contributor and guest presenting with me for this will be Pixie Boots, the ironic mascot. And so Pixie Boots says we're gonna see if we can drum up some help for you. So first you have to get your bearings. Researching first is okay. Just join the IRC channels for any project you might be interested in. Read the mailing list or the archives if you haven't subscribed to the list. Look over the approved specifications, recently landed code, and give yourself an idea of how that community works. The different communities inside OpenStack behave differently and one of them may fit your taste more than another. You can also attend the OpenStack Summit. In fact, by being here in this session, by being here at the summit, you're already participating and doing that at at least a minimum. So you can do this research and then you can maybe decide what project should I contribute to. Now myself and Pixie Boots, we can't bear to build ironic without you. So it's really important to pick a project that's gonna be accepting and wanting to. I would advise, if you've never contributed to OpenStack before, maybe avoid some of the larger projects like Nova or Neutron, and Nova and Neutron is great software. It's excellent, it runs in my cloud. It's just a higher difficulty level. And so why make it more difficult and burden yourself with the difficulty level of trying to put into one of these bigger projects when they're smaller projects that your contributions can have a larger impact on. They're entire portions of OpenStack that are libraries we depend on that are managed by maybe one or two people. And by adding your help there, even a little bit of time can often make a big difference. The other big piece of advice is choose a project you care about and that runs in your production environment. Realistically, you're gonna bring the most value if you have experience running a project in production and if it's something you care about. Conflict is unbearable. So don't forget the human element. Treat people excellently. Your peers in the community are your coworkers. We're all working through the same goal. We all wanna make great infrastructure even if we have different customers or different approaches to it. So what does this mean? This means that if someone's short with you on IRC or a mailing list, remember it comes from a person and that a lot is lost in the translation from real life into text. Remember that if you do try to contribute for the first time and someone puts a minus one or maybe even a minus two on the patch, that that's not something against you personally but it's their way of pulling the rope and making OpenStack better and maybe in the process making you better by showing you something that you might can do better next time you contribute a patch or you can do better in this patch before it lands. And it's also important to open your mind to new ideas. We all have different ideas about how to run a production environment. You may choose to run stable branches. I may choose to deploy master often but that doesn't mean that one of us is right and one of us is wrong. It means that we should have our mind open to new ideas and determine why people are doing things the way they are in order to make it better for everyone. And a big piece is also discard your fear. You are likely your harshest critic. I know that if I had come to this talk two years ago and I was sitting in the audience, I would think, this is nice that there are operators around me who could contribute to OpenStack but I'm talking to you. I'm talking to every single person in this room. Like I established at the beginning, you're an expert in cloud infrastructure. If you're here to discover, you will be very, very soon. And the only real failure is a failure to try because if you're not doing things daily that are stretching your abilities, then you're gonna let yourself stagnate and that kind of goes back to what I was talking about about what's in it for you is if you wanna grow your career, you have to grow your mind. You have to grow the areas of expertise you have. And there's really nothing to lose but a little bit of your time and after all, bravery is very metal. So in conclusion, we need to embrace our common goal of providing high quality infrastructure to the people who count on us. Focus more on what we have in common rather than the tools we use or the titles we hold. Understand that contributions are a lot more than code. Contributions can be an encouraging word in IRC for someone who's dealing with a hard problem. It can be a good bug report. It can be documentation. It can even be evangelizing OpenStack to a fellow operator who maybe is wondering how to adopt a cloud. And OpenStack, we're all better off if you're bringing your perspective to the table as an operator. That's what's really important. So I think OpenStack's better when we're all builders and operators. And I think we've made great strides toward that but we need to keep going. And also, there's no such thing as enough ironic puns. So thank you very much. Before I ask for questions, I will point out that Rackspace is hiring. Expo Hallbooth P11 or bit.ly slash rackertalent with a capital R and a capital T. Or you can come up here to the stage and chat with me afterwards and I can talk to you and tell you how great of a place Rackspace is to work. But at this time, I'll take questions. If you do have a question, please come to the microphone. Wonderful, thanks everyone. As a manager now, how do you keep up with all the emails from the DevOps distribution list and keep up with what's going on in the community? Note how much I focused on IRC and didn't focus on the mailing list. I have found in my personal experience, especially with OpenStack Ironic, that real work happens in IRC and at the design summits and mailing list posts tend to be more on a philosophical or a higher level about project governance, about what our approaches to problems should be. And honestly, as someone who has a product, has an infrastructure to provide to customers, you know that's not the sort of thing I have time for. I'm very glad that Rackspace hires people to have those discussions and to think about those things. But for me, I try to stay in the trenches and keeping up on what's going on and keeping an eye on the code and the specs and being really active in IRC and at the conferences, which oftentimes is where more of the work happens. Thank you, I got another one. With the documentation, Rackspace probably has some documentation that they wanna keep internally to maybe how they customize some features. How do you separate then documentation upstream and documentation internally and direct maybe tenants of the cloud to both sites? Our tenants get confused very easily and you send them to the upstream documentation and they just get even more lost. That's actually interesting because when I mentioned documentation here, I was almost thinking more along the lines of developer and operator documentation rather than user documentation. With developer and operator documentation, you don't really have that much of a problem. It's a lot more declarative of here's config where if you wanna build an ironic cloud integrated with Nova, here's a wiki page on the 10 config values you should set and where the caveats are and things like that. For user documentation, I think that's a very hard problem. But to be frank, it's a very hard problem when you're running a product which has patches and a downstream and an upstream as well. So I don't know how that's handled at Rackspace with user documentation, but I know we do handle it because I know we have an internal set and an external set of docs. When I say that, I'm mainly thinking about operator documentation or developer documentation. The operator documentation especially, I think is really lacking in a lot of places whereas the user documentation maybe gets more love because it's seen more often. Thanks. Yeah, if you wanna come up and say hi to me afterward, I would gladly introduce you to someone who knows how to handle it, who knows how that works. You mentioned IRC, so I feel compelled to ask as a very prolific IRC user, I'm kind of curious, how do you manage your IRC space if you're that reliant upon it? So like myself, I'm probably a member of like probably at least 100 channels at a time. So what do you use as your configuration, as your platform, are you using IRC, you know, IR, SSI? And do you have your dot files published anywhere? My dot files are up there in a private repo, but I could maybe finagle some things out of it. I'm always looking for like better tweaks for my configuration, so you know. So I'll give you my answer, and then I might redirect to I think Jim maybe has a better setup for this sort of stuff than I do in his bouncer, but for me, I found, I try to cap the number of channels I'm in at like 40 or 50. Once you get over that amount, I'm honestly a little bit shiny thing over here, shiny thing over here, so if I have that many channels, I find that I spend all day reading IRC and not all day doing anything. So for me personally, I just run WeChat inside of a team up session on a Rackspace cloud server. So I don't wanna apply a lot of technology to it. I'm also a speed reader, which makes it a little bit easier. I know that Jim has mentioned once that he uses ZNC and has it capture any notifications to him and put it in a separate buffer, which can be useful. But realistically, I just tried to be opportunistic with it. If I have an extra five minutes or if I have a distract time, I go look and I read some scroll back in IRC and try to determine what's going on and you can often skim through and determine if a conversation is something you care about or not in a big hurry. But I think what you'll find is if you're active in OpenStack IRC channels often enough, people will start pointing questions that you're capable of answering and things at you so that you can kind of shift from maybe having more actively monitor the channel to being slightly more passive about it. Right, fair enough, thanks. We chat, WEE chat, it's basically IRSSI plus modern. A little bit, but it's still text because we like good interfaces. Question, excuse my croaky throat. I'm kind of at a different stage of devolution to code, assist, objects, et cetera. I'm in business development and sales, so I'm probably the filter of the earth. How can I contribute? I'm not hands-on, I'm not seeing the bugs, but I'm hearing what the customers want from an operator. That's a great question and a great word. Devolution's a good word, even though it maybe means different things, but the point still remains, you may do sales and biz dev, but you care about quality infrastructure. You have to have that quality infrastructure to sell to customers for it to exist. So I would say that you maybe even have an interesting perspective as almost a user or as a representative of many users in OpenStack. And so you can still take that knowledge and apply it in ways close to this and maybe discard some of the more technical pieces if you're not technical and you can't do that, that's fine, but there's still plenty of places to help. You can still be in the primary IRC channel and answer user questions because they come up a lot. You can look and become very familiar with the client libraries and how to use them and then help others with those or maybe even work on the user docs as well. There's plenty of ways to be involved and honestly by being here, you are being involved by selling OpenStack and by spreading the open source commercial word to the masses, you're already somewhat participating and I would actually be very interested to see if you tried to participate, if maybe you could identify better places than I could for someone like you to enter into the process because I maybe don't have great answers for you as a salesperson, but I'm saying that if you go into IRC and you say I'm not very technical but I'm willing to help, someone's gonna find a way for you to do it cause we always need more hands and I've even seen in the past people who maybe when they started out less technical, they start getting involved, they can become really important in the community for helping things happen. Thanks. No problem. So I'd like to thank you all for coming and participating if you have any further questions, feel free to come and see me up front afterward. Thank you all very much.