 Hey, good morning everyone. My name is Christian Yushevsky and this is Darren Sorrentino. First before I start, we're going to talk about the OpenStack Unify CLI, which is a little bit legacy name. OpenStack Client is the, to be more current, it's OpenStack Client. Darren Sorrentino is distributing a little cheat sheet that we have with the redhead that describes some of the new OpenStack client commands. We have only a few of them. We have a couple more left on the desk here, but if you guys want them, they're available at the redhead booth. So there's definitely more for you to grab. And I apologize. We didn't bring more. To be honest with you, we didn't expect such an awesome audience here. So thank you for coming. So before I start, I want to ask how many of you fly or flew American Airlines for the last year? Okay, so I'm going to make this quick announcement just like American Airlines does before they departure. So this is your last chance to deplane before the aircraft door is closed. I just want to make sure you're in a good place. Thanks for staying. So OpenStack CLI, so I don't want to disappoint you guys. We are not the OpenStack client developers. So what makes us an expert in this subject? Well, first we both worked with OpenStack, used OpenStack for a really long time. And our current role is OpenStack architects. And we not only on a daily basis, we operate, troubleshoot, and you know, deploy OpenStack for our customers, but we work with a lot of customers. And a bunch of them are newcomers to OpenStack. And we felt that the OpenStack client, OpenStack by itself is not the easiest project for the new customers to get into. And we felt OpenStack client is definitely a step in the right direction to bring these new guys on board. And that's why we picked this topic to talk to you guys. And it makes our lives much easier too. So I have another question. How many of you guys consider yourself OpenStack operators? So operate OpenStack on a regular basis. Okay, we have a good number of people. So one last follow-up question. For the guys who operate OpenStack, when you try to list all the instances that you have available, how many of you use Nova list instead of OpenStack server list still? Okay, all right. Good. So you guys definitely in the right place. So thanks for joining in. So who is this session designed for? Who did we prepare this for? And I would say mostly the operators, the guys like us who have been working with OpenStack for quite some time are used to, you know, from now on I will call it the legacy CLI. So your typical Nova, Keystone, and Glance commands. And you guys start seeing the error message that I listed here on my screen, that some of these commands will start getting deprecated or starting getting deprecated. And you need to move to this new way of executing CLI. And it's not just that, but folks who are trying to do more to take advantage of some new functionality of the OpenStack client. We're going to talk about those. We're going to talk a little bit about basics and principles. And then the second part of this presentation, Darren, my colleague Darren here, is going to give you a really, really awesome use case of troubleshooting and reverse engineering on how he learned the OpenStack client. All right, so these are a few pointers, principles that I want to go through. What can you expect from the new OpenStack client, some principles? So first, consistency. If you think about, again, as I would refer it as a legacy CLI, it was not always that consistent. For instance, the examples I brought in before, if you wanted to list instances, you would say NovaList. But if you wanted to see the list of images that you have in Glance, you would type Glance ImageList. So from the command perspective, it was not always that consistent. And for the new commerce, especially, that's not a good approach for the CLI. So OpenStack client simplifies that and makes it more consistent. So simplicity is the second biggest principle we have. And I already mentioned some of it. Another aspect of simplicity, we're trying to minimize the output that the OpenStack client commands are showing comparable to the legacy CLIs. Not only that, but the interface or the output and the commands it serves are designed around users more than how the back end of the system is created, which definitely creates a much better user experience. And transparency, so exposed current state of the infrastructure and its interactions. So these are some core principles. I pretty much copy them off the OpenStack guidebooks. All right, so here I'm going to try to do a little bit comparison between legacy CLI and OpenStack clients. I'm going to start with command structure, and then I'm going to try to ask some, or answer some questions that usually folks ask when they switch from one to the other. All right, so command structures, it is pretty similar with, I would say the major differences, the legacy CLI would start with the OpenStack module. You would have to call the OpenStack module first, then follow with some options like tenant name or user name, et cetera, and then command itself. And all of the commands had the user different command options. And most of you are probably familiar with that. And Novaboot is one of the examples. The new OpenStack CLI starts with its consistency, you can see it right away. It always starts with the word OpenStack. There's a whole list of global options after that that you can invoke, starting with the similar as we had before with the OS user name or tenant, et cetera. But there's some more, the new global commands that we're going to talk about a little bit later that they're quite exciting to, and will help you improve your everyday operation. Then command and some command arguments. So right from here, you can see some benefits and principles that are being applied to the OpenStack client. One of them, again, all of the commands will start with OpenStack, which is consistent, but also the user-designed approach. If you think about yourself not knowing about OpenStack anything at all and that's most like a lot of customers that we're working with, and now we're going to ask you to type Novaboot. Would you ever be able to connect this to actually creating the servers on the OpenStack? And for most of our customers or most of my customers that I work with, the answer is no, they don't really make this correlation. So OpenStack server create makes much more better sense and we really appreciate all the work that developers are doing here. All right, so I'm going to compare these two outputs. This is the, again, as I call it, legacy CLI, glance image list. So most of you who used to use OpenStack, you're pretty, or you're still using, you're pretty familiar with that. If we compare it to the OpenStack image list, you will see that most of these commands, we're going to lose that dash. And also the output is shortened significantly. And if you think about this command from the user perspective, from the OpenStack consumer, in most cases, he won't care about the disk format and container format and these additional information. So there's definitely another principle we see here that's being applied. But maybe some of us, including ourselves, maybe we do need all these additional columns. And so how can we get all of these columns back? So OpenStack Client introduced this new flag or new options, dash, dash, long, and it's available in, I think, any command of OpenStack Client. It gives you not only the information you used to see in a legacy CLI, but it extends that to, you know, to even more. And it can be messy, right? There's some of these commands will output much more information that you would expect. But there's a solution for that too. You can use the dash C flag, which stands for column, and only limit your output to the columns that you're interested in seeing. So if you have any scripts that are taking advantage of the old CLI and you just want to take advantage of them in the new one, you can definitely make that happen with these dash C parameters. So now the question is, where are all the commands? Do we have all the commands in an OpenStack Client? And the answer is yes and no. So no, because the OpenStack Client by itself, it only provides the supports for what I call the first-class citizens. So all the projects that are part of most of the installations of the OpenStack. And we certainly agree with this approach. So you see compute identity, image, object storage, block storage, and network. And with network, it's not all the new... Neutron is definitely one of the most growing project out there. And you're not going to be able to see these new commands like tap as a service or something like that right in the core. But no problem. There's also a concept of plug-in. So if you're using Ironic like we do a lot or triple or things like that, there's certainly plugins that will allow you or enable the ability to use OpenStack Client as well. All right, so a little bit about what format options do we have. I'm going to try to breeze through them because I have a little bit more exciting stuff later. So format, the default one is stable. I didn't have to specify that here. But if you just type OpenStack Image List, you will get exactly the same thing. I think everyone is familiar with that. OpenStack Image List with the format CSV for folks who are, you know, maybe want to export this to some sort of spreadsheet format. This is how it's going to look. There's a JSON format for, you know, JavaScript programmers, HTML. There's YAML for folks like us who use, you know, use the YAML type of format quite a bit. And then there's also Shell, which is a little bit more interesting. If you put format Shell against any list of the commands, it will most likely fail. What it does, if you put it against any type of show commands, it translates all the output into this, I would call it close to the bash variable format where you can export it and then use these parameters as a variable. So a little caution if you do that. And you want to just avoid some collisions with your existing environmental variables. You can add the prefix to it, and there's a way to do that as well. All right, so this is the more exciting part for me since I'm the operator, and I want to walk you through some of the things we discovered, some of the shortcuts and things that we use definitely quite a bit when working with OpenStack Client with our customers. So we're going to start with some help, and everyone needs to know how to use help. And it's very similar to the legacy CLI, but I want to show you a little bit difference and a little bit of a few improvements in a space. Then I'm going to go over some OpenStack Complete to show you how quickly enable auto-completion with the OpenStack Client, interactive mode, working with multiple clouds. We heard today at the keynotes that more and more deployments are done in the multiple cloud weights, and we certainly see that on our end, too, with Redhead, when we design the solutions for our customers, they more often try to add more regions to the mix instead of just creating one monolithic deployment. Logging and debugging, this is probably a feature that I use most when I troubleshoot. There's some improvement there, and there's a couple more helpful tips. So let me start with the help. The help parameter or help command works similarly as it used to before. If you just type OpenStack-help, you're going to get a very long listing of everything that OpenStack does in the version you have installed. You can, of course, filter it by the command you're interested in, and here's just one example of filtering it by looking for all the endpoints. And so for the folks that are transitioning and moving to the Keystone version 3, there's also a way to force our OpenStack command to display the commands for Keystone version 3, especially for folks who want to take advantage of some authentication advantages that Keystone version 3 are bringing in. But this is just a quick shortcut on how can you do this here from the command itself. Then there's, of course, the help for the particular command that you want to see. So you start, again, consistency. You start with the OpenStack, you type help or dash-help or dash-h. And then endpoint list is one of the examples, but it's going to be very detailed. It's going to show you everything you need to know to use this command. And I feel like help with OpenStack is very robust, and it's written very well. And it certainly helps our customers to go through all these commands quickly. All right, so this is one of my more favorite features that I kind of found when playing with the OpenStack client. OpenStack complete. If you just type this in your shell, in your OpenStack environment, you're going to get text similar to what you see on my screen here. It's going to be much longer, of course. It's going to have a lot of bash-looking lines. And what it does, actually, it provides you ability to use the autocomplete capability with the OpenStack client. So what you can do now, you can just type OpenStack complete and stream it to your local bash or C. After that, all you have to do is install bash-completion. And as soon as you do that, you can use the OpenStack tab-tab and be able to get some advantage of autocompletion of commands, but maybe also a quick peek of what's available in certain commands itself. So definitely a helpful thing for us. There's also interactive mode. To get into the interactive mode, you just simply type OpenStack, and then you can just start typing the commands without the OpenStack in front of it. So there's an example for the image list. So OpenStack, the interactive mode has an autocompletion enabled by default. So you can use the tab-tab to see all the commands. And then there's also... you can also script using the interactive mode. And I'm giving you a little example here where I execute two commands in a one-swing. So there's another tip or trick that you can use with this new OpenStack client. So this is probably my most favorite feature in the OpenStack client again, using OpenStack client with the multiple clouds. So OpenStack supports using this configuration files, clouds, YAML, and you can put it either in the current directory you're operating from or in a Home Directory config OpenStack or in the Etsy OpenStack. And the first one from the tab list, so the most local file is going to win whatever you have in this file. And there's also another file called Cloud Public YAML. And this file is designed more to store the public cloud information. So I'm going to go through some examples just to show you how cool this speak functionality is. So this is one of the files I created. So what we use mostly with Redhead, we use the triple O to deploy infrastructure for ourselves and for our customers. So the triple O concept for those of you guys who are not familiar, you're deploying OpenStack from the OpenStack. So automatically every single customer we work with have at least two clouds that you need to manage. There's an under cloud and there's an over cloud. So what I did over here, I created this file that specified my credentials for the under cloud. And I also separated my over cloud, so my actually production cloud into the tenant roles. So I have one over cloud for my admin and one over cloud for my tenant one that I used. And this is one way of doing it. You don't have to specify every tenant out there. I find this works a little bit better for me, but you could specify user on a line itself. And there's another example here on the right. It adds, if you have a public cloud that you want to add to the mix, you can do that too as I show on the right in the public cloud section. So now execution, all I have to do is add this one parameter, OS and the name of the cloud that you want to execute it against it. So now if I type server list against the under cloud, it's going to show me my instances running under under cloud. But if I do it with the over cloud, even if it's the same place, I'm going to have output for another cloud. So definitely a big improvement from the usability perspective. And this statement at the bottom, it's in Catalan means amazing. So I first learned this means amazing, but then I learned it can be good amazing or bad amazing. So I think it's more like good amazing. So logging and debugging. If you're an operator, you probably use that quite a bit. There's a dash dash debug and dash dash log file that we typically use. So with the clouds YAML file, you can include the two more parameters, log file and log level. And from there, you no longer have to put the dash dash debug and dash dash log file every time you execute the commands. I think that's quite helpful too. If you're in a stage of troubleshooting, if you have any problems and you just want to get things going quicker, I always forget to add dash dash debug to find all my outputs is usually a problem too. So I think this is another really great improvement. And the last one before I turn it to Darren, it's very simple. I kind of use that too. I just create a little alias for OpenStack, and I make it always directed to the bash. And then instead of typing OpenStack every time, I can just type always an image list, for instance. With that, I want to turn it to Darren who's going to have a more exciting part of the presentation. Thanks. All right. So Chris went over the syntax of the commands here. I'm going to talk a little bit about the framework and what I listed here is deep dive, which is probably more accurately would be reverse engineering. So when they went to go design the OpenStack client, they needed to come up with something that was rigid yet flexible. They wanted something to provide the consistency on the input and the output as seen on Chris's slides. But they also needed something flexible because, as you all know, OpenStack changes over time. The existing projects add additional commands, additional projects come underneath the BigTent infrastructure, and we even move from the integrated platform to the BigTent model. So there's constantly new things going, and they wanted some kind of client that would be adaptable so that you could learn it once, and it would be consistent regardless of what projects you added to it. So these goals here, when they came up with the client, they set us a list of goals that they wanted to accomplish. The consistent naming structure. It was Chris covered all that. Actually, he covered most of all this stuff. The one thing that he mentioned there was about the shell is that the single binary approach, it allows you to start up the shell. It'll authenticate the first time, and then after that you can issue multiple commands against your cloud, and you don't have to authenticate every time you're actually running the commands. So that was one of the key things, as well as they wanted to utilize the Python API libraries. They didn't want to reinvent the wheel or redo things. They wanted to use what we had and what didn't exist code for that. So they leveraged this thing called CLIF, the command line interface formulation framework. The whole design of the CLIF framework is to give you multiple layers. So you have the open stack command, which is your binary. All of the global options go immediately after that, such as the OS cloud that Chris showed in his previously slide, followed by a command. They needed something that was capable of dynamic loading of the code, so that allows them to add additional projects or change existing projects code. So they also utilized the setup tools in Python, which allows them to use an entry points text file, which is what we're going to dig down into in the next couple of slides. So this is my disclaimer slide. I am not a Python programmer. I basically... what you're going to see in the next couple of slides is me trying to adopt the unified CLI, because it's the new greatest thing and everyone says we should use it. And as you can see with Keystone, you start using all these other things. You start getting the deprecated messages and stuff. So the next couple of slides is my approach to actually learning the open stack tool. Hopefully it'll be somewhat exciting and help you also understand it. It's more from the perspective of an end user. And when I was younger, I used to do a lot of reverse engineering of assembly code, a lot of opcode changes from 74 to 75. So that's the kind of perspective I took on learning the open stack client. So the way I really learned it is I actually had a problem with the open stack client. It was broken. So the problem I ran into was with the open stack flavor delete command. So as you can see here, here I am all happy using my open stack unified CLI to list my flavors, create my flavors, list it, there it is. Now I want to get rid of it. And when I went to go get rid of it, now all of a sudden I get this odd message, public endpoint for messaging service not found. Fortunately, with the open stack unified CLI, even though you have that, the old command still works. So when I was on the customer site and ran into this, Nova flavor delete came to my rescue and I put this on the back burner and figured I'd come back later and try to figure out what's going on. So when I finally did come back to this later, I did look up a little bit online and read about Cliff and how that worked. And then I wanted to see, I'm the type of person that needs to see it and see work directly back through how things work. So the first thing I needed to do was figure out where the open stack binary was. So I figured out which one I was using, use yum to determine what package it was in. Then I started looking at the package and I noticed the package was under the site packages of the open stack client. So once I went into that directory, I listed the directory. So just, again, just trying to figure this whole thing out. I started to notice that there's directories here for all of the different aspects, the core projects within open stack. And then from what I read on Cliff, I knew that there was an entry points file somewhere. So this is me basically looking for the entry points file. So then I pulled up the entry points file and looked at it. And the entry points file has a number of sections that all begin with open stack dot and then one of the project names. Underneath that are all of these things, which I guess would be the entry points. So at this point I started to look at this and I wanted to take a simple command that I knew from that open stack and I knew worked. So what I wound up doing was I took the host list command here because the output of the host list command shows your hypervisors. I wanted something simple, again, not knowing Python or anything, just trying to figure out how this thing works. So I looked at the host list command and here's where I executed Nova host list as well as the host list in the open stack using the open stack unified CLI and the output is identical and it actually worked. So I figured I would use this as a basis to try to feel how the code works in regards to the unified CLI. So I went back over here and I was looking at it. The one thing I noticed is that when I did open stack host list, there was a space there. So I started to make the correlation here that host underscore list is the exact same thing. They basically took the space and replaced it with the underscore. So you got the host show, the host list. So I started to see a pattern here, but then I was trying to figure out what this was. But the directory I was in was actually called open stack client and they all had subdirectories in there for the various projects with an open stack. So I started to put two and two together and took a look and sure enough there was a host.python file which matched what was in the entry points file. So that left me with this part here. So which one is actually being the class. We'll take a look at that in a minute. So I took what I knew here on the host list and applied it back to the flavor delete. So here's the flavor delete pointing to the open stack view client flavor, delete flavor. So in this directory, there's three files that are called flavor. The first file is the Python code. The second file is the compiled Python code and the third flavor is the optimized Python code. Google explained that to me. So this is... So when I looked in this file, I took a look at the flavor.py file and this is the class for the delete flavor. Again, I have no idea what this is actually doing because I don't know Python. However, so being reverse engineer trying to get a feel for the code, I can tell that there's two functions defined here. There's this get parser and take action. So I started to think, okay, so if this is broken, maybe I can compare it to something else that actually works. So I went over, this is kind of an eye chart, but perhaps you can see this. The slides were made available after on the open stack.org. So... But I looked over here at the show flavor class and in the show flavor class, again, I see get parser take action and show flavor works. Flavor show works, which it's called flavor show, but the class is show flavor. So the consistency is on the input and output, not necessarily the code, I guess. So I knew this worked, so I compared the two and they both seem to have pretty much the same contents. So I started to think, it's like sometimes you have to take a step back from the problem in order to figure out what's actually going on here. So we know the open stack syntax, where it has open stack, the global commands and the commands. We know our problematic command is my flavor delete. The entry points file, it appears to have a section name that deals directly with the project that... The project that the commands are for has an entry point, which points to a file that has the code and the class that it needs to execute. So then, like I said, the entry point seems to be the command with the space replaced with the underscore. That is an eye chart. So when you've executed open stack help, I looked there to try to get a better idea, but one thing that stood out to me was, so I knew I had in the directory, I had commands for compute, identity, object, block, but what's interesting here is I started to see bare metal. I started to see all these other things in here that weren't pertaining to those specific projects. So that got me thinking. If there's additional commands that you can follow open stack with, there's got to be something else that it's using to get those command lists. So I went back to the site packages directory and I looked for all files that were entry points files and there's 81 of them. So I really didn't feel like I was going to sift through them, so I started thinking, okay, well, what I'll do is I'll just look for the ones that have open stack sections within them, because those will be relevant to what's going on. And sure enough, there's a set of six of them that come back. The open stack client, heat client, ironic inspector, triple O. So that started me thinking. So all of these other entry point files are extending the open stack client by adding additional entry points to it. So in thinking about this though, it's like, well, when we do open stack server list, there's nothing in there that says it's for the compute service. It's just open stack server list. So how does it know that server list, entry point, server underscore list, is part of just for compute? What if the names, the entry points have to be unique across all of them? And what if there was another entry point that was delete flavor? So with thinking that, I said, what if the entry points must be unique? So I looked for flavor delete entry point, and sure enough, it's not only an open stack client as a core, it's also, there's a car client. So I took a look at the open stack client entry point file, and you can see here open stack messaging, and it has all of these entry points here. And what's interesting is that most of them start with Q because it has to do with messaging, except for pool create and flavor delete. They kind of stand out there as odd oddities there. So now I had this theory that maybe the entry points need to be unique. So once you have a theory, you have to test it. What I did was, of course, backed up the text file because I wasn't sure if I was going to break my environment. So then what I did was I changed flavor delete to just exact flavor delete. I didn't know what that would break or what that would do. I just wanted to get that out of the picture when I actually ran my flavor delete. So once I did that, I actually was able to run the flavor delete Barcelona, and sure enough, it worked. So that told me that the entry points would be unique across all projects that are utilizing this. So before we go over what we learned, the other interesting thing that it's kind of ironic because they came up with the OpenStack client in order to simplify the end user's lives. So it's consistent and repeatable and stuff. But in doing that, one of the things using Clif is that now the developers need to be cognizant of the entry points of all these other packages and they need to work together. So in one way, it simplified things for the end users. It's another way it complicated things for the developers now. So they need to basically work together and ensure that no one steps on other people's codes as they develop for the OpenStack Unified CLI. So basically, this is a brief summary of what we went over. OpenStack client comes with the built-in, baked-in commands for the core project set. We covered the overall syntax of using the OpenStack client. Talked about the general syntax of the entry points TXD file and how to basically find what that code is and take a look at it. Talked about that any project can extend the OpenStack client by adding its own entry points into the mix. The end points appear to be just a direct correlation of the arguments you pass to the OpenStack client replacing the space with an underscore and all entry points definitions must be unique. Just a side note, I did go back and rename this a car thing and there is a bugzilla open on that actual bug. So hopefully that will be fixed sometime soon. So that wraps things up for my section. Do we have any questions? Sure. So the core projects, the gaps are, I would say the gaps are very, very small and if they do exist, they exist more as bugs such as this because this is a nova command. For the projects outside of the core set, they fall upon each of the projects to come up with the command sets for them. They're slowly moving away from... I mean, everybody wants to move away from using just the project names to the OpenStack client. But just like any of the projects within OpenStack, it takes time, even though they moved to the Big 10 model, the code still has to mature over time. So just like any other project within OpenStack, it's going to probably take a couple cycles for it to be one widely adopted into the direct equivalent of what you can do with the project commands. Yeah, and just to add to that, I think when I ask initially how many people, even in this room, still use NovaList instead of OpenStack ServerList and a lot of us still raise the hand and I think it's for the old school OpenStack guys, it's imprinted in our head. Just automatically, we still do that. So I think the way for the community and for the users to move forward the OpenStack client is to give them more benefits that will get the buy-in for them to actually move to this new way, right? And I think we're going in the right direction with some of the features we showed up. But if we can add more benefits from the operation perspective to motivate these guys to move on to it, it will be definitely better. So for Keystone, it is there. But for many others and OpenStack documentation, the switch is still not there. So many people, when they go through the dosage, don't zip into your client. That's preventing the migration. Right. Yeah, well documentation historically in OpenStack has always been somewhat, I mean, no one wants to do documentation. So it's not just that. It's not just that. I mean, if you look at some of the documentation stuff, it's not there. It's going through the same pain points that all the other projects go through. So it'll get there. I mean, when we're actively working on helping get that fixed. Good question. Thank you. Any other questions? All right. Well, thank you all for coming. Next slide. I just have a couple other sessions that are supported by Red Hat for today. We have that slide there and that slide, but you guys all have the mobile app, so it's probably better than my slides. Thank you so much. Thank you.