 So, welcome to our first talk on DEF CONF in this room. As a chairman here, I should tell you a few things, please, if you want to leave the room or come inside, do it as quietly as possible. As well, do not forget to vote and leave a feedback for this session through the DEF CONF website or through the DEF CONF mobile app. So, and now our first talk, our first talk is about shell scripting my way through the OpenStack. After a short introduction to OpenStack, this talk will illustrate how one may get to a reportable setup for testing by means of shell scripting. This makes no claim to be complete, but rather tries to show the presenter's struggles in a funny and thought-provoking way. Our presenter is Phil Sutter from Red Hat Networking Team. As has already been said, I'm Phil Sutter. I'm working in the networking team at Red Hat. Usually, I do some kernel development and maintaining IP routes, NF tables, user space packages. This talk was a bit of a surprise to me because I'm not really familiar with the topic and especially I was surprised that I'm the first speaker for the OpenStack track, but anyway, here I am. Since it's pretty early in the morning, I guess my goal would be to keep everyone alive awake. For me, that's pretty easy. I'm just talking all the time. The problem is with you guys, so I made something up. I'll be just lying from time to time. That's the deal. And if you notice, then tell me afterwards. And there will be a cake, I promise. Okay, so how come that I'm still talking here, although I have no idea what I'm talking about? A certain manager told me I should look out for possible improvements in Neutron, which is the networking component of OpenStack. And my reaction was pretty easy. Yeah, come on, just set it up, run some benchmarks, identify bottlenecks. How hard can it be? Yeah, let's find out. But I guess I'll start with first things first. What is OpenStack? Here's a nice quote from opensource.com I found. OpenStack set of software tools for building and managing cloud computing platforms for public and private clouds. So what does that mean? It runs cloud images, obviously, in a virtualization cluster. And it has support for multi-tenant virtual networking. Basically Neutron. And for me, the most important thing is its open source. What's included? OpenStack comes with a number of components which are blog and object storage. There is identity management. There is an image service for the cloud images, obviously. There is accounting. So you can build users, tenants in that nomenclature. You have orchestration. So you can set up complex scenarios in one go. There is a nice dashboard. And it even has bare metal management, which means it supports setting up the whole cluster hardware at once. I guess picture is better to show this, how everything of this works together. Here's one, which is pretty obvious, I guess. If you don't think this is clear and easy to understand, look at another one. I didn't even try. So who does something like this? The first release came in 2010. It was developed by Rexbase and Anzalab. And since 2012, there's the OpenStack Foundation, which is an independent body, tries to protect and power and promote the software and the community. Another interesting party is RDO, which is a community project, and declares itself as being an OpenStack distribution for REL, Fedora, and CentOS. RDO has many meanings. The most useful one is RPM distribution of OpenStack, but go to their project site and find out what the others are. It's pretty funny. The top contributors to OpenStack, Newton is the current release, as far as I remember. Looking at lines of code, there is Mirantis as the top company involved, with RETED on second place. I found also one which has RETED at first. We do the most commits with the least amount of code. But in any case, as we see others will. Okay, still setting it up, despite all what you have already seen. I'm going to show three ways to do it. There are more, but I'll just limit myself to this. There is the manual installation version. It's not so hard to do. It's all documented. There is docs.openstack.com, which has detailed instructions on how to install and configure each of those demons. The problem with this is you have to repeat this for every host that will be part of the virtualization cluster. Not every component needs to be installed everywhere, but most of them. There is a simpler method using DevStack, which looks pretty easy. I didn't know about this when I did my testing. It's just about creating a user, cloning the repository, create a simple config, and call stack as edge, which should set you up and get to running setup. Another alternative, which I used, is the pack stack command. In order to use it, you have to install this RPM, which basically creates a young repository and install the package and then start it. This is already pretty easy, but sadly there is some post-installed patching up required because OpenStack doesn't support network managers, so you have to disable this. You have to configure the external bridge, so virtual instances will get access to the outside world, and the default security policy doesn't allow even SSH or ICMP, so it's advisable to enable this in order to be able to connect to the host at all. I collected a bit of information on how to use it. This is far from complete, because I simply don't think I have time. First, the basics. I guess the dashboard is always a good point of reference. It's easy. The learning curve is Windows-like, so pretty self-explanatory, I guess. For doing more serious work, there are a number of command-line tools. Lens, Neutral, Nova, OpenStack Config for configuration, and they basically do this for a specific purpose. It's not always that clear, but the good thing is there are man pages and help texts available for each of them. Okay, so I have two examples for command-line usage. This is how to spawn an image. Basically it's just the first command, every instance should have a floating IP, so it can be addressed from the outside world. Otherwise, it will just get a private IP and it's not addressed. Sadly, not everything is as easy as this. Here's how to create the default network and router combination, which Packstack installs. So you have to create a net, then a subnet with IP address information. You have to create a router, set its gateway to point to the outside world, and add an interface to it so it will actually route between networks. The good thing is, again, about the dashboard, there is a pretty nice graphical illustration of this. Here's an example of two tenant networks which are connected to the outside world, routers, and each tenant network contains an instance. So the actual topic. I didn't spend much thought in beforehand, so I just got started, and the scripting idea was more or less an intuitive approach of getting things repeatable and act like a $hist file as a memory of what I did so I could recover or review. The scripts that I created in this process are most importantly this setup OpenStack shell script which installs OpenStack on two machines, which I got from Beaker. Those two are RAID 7 machines, and that's all the stuff. I have another slide explaining what it does in detail. There are two scripts for tunnel configuration, changing MTU or the tunnel type. There are two ways to get firewalling done in OpenStack. There is the hybrid firewall which uses IP tables and OVS in combination, and there is a native firewall new approach which has all the firewall rules in OVS implemented. I created a script to enable distributed virtual routing and I have this recreate tenant network which is helpful because config changes only affect newly created networks so after a reconfig change I had to recreate the tenant network anyway. The server list is just a customized list of servers. I have that in detail as well. Then for testing I created actually three simple scripts. What does setup OpenStack do? In order to get me a little bit faster to a new setup I added at the first point to just uninstall everything that's on the boxes so I didn't have to reprovision them which is always time consuming. It subscribes the machines to write that network. It configures a direct link between the machines so there is more which is there to provide more performance. It installs OpenStack obviously, applies post install fixes I talked about. That's my SSH public key and boots up some machines. Recreate tenant network. I showed you what's needed to create a network. This undoes it first and then redos it again and a nice goody is refreshes SSH host keys because I have to recreate the virtual machines along the path. Serverless SSH is a simple illustration of how to get custom data out of this. Basically it asks for every host, compute nodes in that terminology which runs virtual machines and then shows all the virtual machines on each of it. This fields variable holds the custom information for each VM I want to show. For testing I used NetPerf and this is, well, it grew a bit. NetPerf test is pretty simple. It just performs NetPerf for UDP, TCP and different buffer sizes. Tunnel test run then calls NetPerf test with different tunnel configurations and since this all generates really many numbers I have PlotterSH which just creates graphs out of it. Here's one example, which is throughput using hybrid firewall for GRE and VXLAN tunnels with normal frames and jumbo frames. In this case, I'm not sure if you can read it, the higher numbers are those of jumbo frames which clearly run in this situation. Same test with native OBS firewall looks a bit differently, performs drops with jumbo frames as soon as fragmentation occurs. I guess in this case we have an indication of a fragmentation issue or at least a bottleneck when it comes to fragmentation. Conclusion, you already made it. And me too. What did I learn? A little bit I did. Pitfall is despite DevStack so I set it up and it still wasn't working. I spent quite some time figuring out why, which is a bit sad. I guess using DevStack might be better in that case. Dashboard was always a good reference especially when I messed up and had to find out what went wrong. My biggest problem with scripting was that output parsing is not as easy as it might seem. And I guess Python would provide some API for that, which I didn't use because I'm lazy. The problem with parsing command line output is that ideally you have output selectors like the column and the field which says, okay, which value do I want to have and in what form. Sometimes those are not available for so I will have to go back to using awk, grab and whatnot. The first of the bad examples should already be resolved and the ugly ones are especially ugly because they're not portable at all. So I just have to... They're specifically tailored to my setup and won't work anywhere else. So things that could be improved is having an apply button for change global configs to apply to existing networks, which might be nice because you change the global config and you would imagine that it affects the whole system and not only a certain part. I would like to see this output selection for all commands, which might simplify my parsing a lot. And I didn't quite understand why the hosting node, the compute node is left out from the list of instances when you just type novelist. Maybe it's for security purposes but I don't see how that makes sense. Okay, here are some useful links. The above three are just for documentation and the lower two are specific to how to use dev stack or pack stack utilities. Okay, that's it from my side. Are there any questions? Yes? If I try to... No. In fact, pack stack uses puppet internally to set up the machines. But for me, it was just a matter of getting some testing setup running so I can run my benchmarks. The setup I got wasn't nearly productive or anything, so I didn't use anything. I didn't use this bare metal provisioning at all. So in my case, it was... I started with two running Red 7 machines and pack stack in that case allows you to just specify the IP address of both and then it will do everything needed. I'm not sure if that answers your question. What is next server? No experience, sorry. You could look up triple O which supports bare metal management. Maybe that would help you. It's also on RDOproject.org. Okay, what else? 15 minutes or so? For me, it was like fire it up and go get a fetch a coffee. Yeah. Ideally, it will run then. A bit louder, please. Now it's running out of the box. The Beaker machines I got were freshly installed with Red 7 and that's also why I had to register them with RHN so they would get updates at all. Yeah? Pack stack installs you the whole thing. You can add as many as you want. It's a matter of configuring pack stack accordingly. I can show you... Can you read this? It's better? Is that better? Yeah. So those are... It's a bit cut off, sadly, but those are the options I passed to pack stack and using install hosts you can just pass 10 or 20 IP addresses and it will set up all of them. The first one will be controller node and the remaining ones are compute nodes for hosting virtual machines. I just used Red 7. I played around with RDO and Retter at OpenStack platform, but... each had its own quirks, so... It's not one or the other. They both can be used. I think that's what your question was. My question was, why do you like pack stack in the normal? Yes. But the new method is to register and it's... Until the record was pretty mature, the pack stack would be used but now that the record is out, it will be 10. I think the record will be 10. Guys, could you please talk outside? We don't need your questions. Okay, then. Thanks a lot. Thank you, speaker. So we have 30 minutes till the next talk. From Pune certification, from Red 9. So I work on OpenStack, so we still have a product called RDSO, called the Tempest Test and generate results based on that. So I was pretty much interested that you work on automating director kind of stuff so that I can use your work to deploy at my end in just some pack stacks. I was just using pack stack. Okay, because I just want to understand that what is the customer when they use director deployment or the pack stack deployment? I guess customers would want to use something like Triple O to get basically bare metal machines running. They plug it in the rack and get it fired up and have it installed. I guess that's the idea behind all this. Unless you have some PXE environment which sets up the machine already and then I'm actually pretty far away from customer setups, in my case, since I'm concentrating on Neutron only and... You said that you are working on all this. No, I'm working remotely from Germany. Okay, that's pretty new. So I might get in touch with you with email? Yeah, sure. Go ahead. You saw my... You have it on YouTube. My name is Kamil. I'm interested in your presentation. The thing is that I was using OpenStack for my learning and some... The thing is that I'm doing these Packstack things, etc. But that's what you are... This show us, yeah? It was like your main show script using these Packstack things or it was default ones? Those are my show scripts. Your show... So I have to write them on? But you will be showing this stuff or it's like only for showing purposes that we can do this? I'm not sure if you've seen the code I showed. At the end it's not polished. Yeah. The thing is that it's not really hard to do this. No, it's not hard. I mean if you know what you're doing then you're pretty quickly at the point where you can repeat this. On the other hand I guess looking into Python API right from the start because all those tools are written in Python. It would be much better. It was just for me because you want to speed us very I hacked things together and then looked at what did you do and then copy pasted into a shell script and you'll only feel like you are using shell scripts to execute Packstack which is using So it's really It's crazy. You see with OpenStack you quickly get used to them, stuff like that. Yeah, exactly. Okay, see you then. Hi, I'm Martin. I'm from Satellite. I will use CI OpenStack instance and my question you don't understand me do you know what is Satellite? Yes, Satellite. We provisioning new machine and we tested this on Beaker or we can test it on OpenStack I need to set the new instance that core of Linux is there I don't know the HCP next server is option to set new instance after boot here is FTPF to do net boot Yes, net boot, yes. It's possible to setup for new instance or for IP range or subnet. Surely, but you know those cloud images you're booting? Yes but it can be yes, it's instance because it's not a generic virtualization environment OpenStack is tailored to starting cloud images which are pre-installed so you have a cloud image for Red7, for CentOS, for Fedora and they work by they boot up and the boot process contains this cloud setup script which then either asks this object storage for configuration data or you can even run it at home by having a custom ISO which will attach to the VM when it boots up so it finds the data and configures itself. But a clear empty machine it's not possible? It's not used in OpenStack I'm not sure if you can do stuff like that with Raph? I think OpenStack is the wrong approach for that. For me is one method use some image and pre-booting with network booting and now I saw that I take one rel image and in there I install KBM or something and there I can setup next server for but I don't know Those machines get their IP address via DHCP as well as far as I remember it runs DNS mask so you might do dirty tricks like instantiate a cloud image and instead of booting the cloud image it will use DHCP and then start doing other things. But connection between instance and private network is blocking for DHCP package because OpenStack don't want to destroy the infrastructure about DHCP definitely if I'm one tenant and I have my DHCP server running and you're the other one then it's not good if you get my IP address you shouldn't we create some own tool for create instance because we need for example create instance join floating IP and join volume for example and these steps are much more important. I don't know if you know script from base OS it's one minute script it's called one minute that one minute interface I have seen this already but I never used it it should be really quick I guess yes it's really quick but it's only for their requirements but we create generally it's called five minute we have to create volume 100 gigabytes volumes and it spent much time but we use it for these things it's very fast because Besh Nova and this tool has really big problem with people go from version to version because if you parse output now before it empty space and then it's known very fast change in this tool I think those tools are really meant for direct user interface and if you want to have something a bit more stable then use Python API but I have no idea how stable it is I didn't look at it at all so it might be that it's just a common open stack issue breaking backwards compatibility all the time I wouldn't be surprised to be honest thank you very much can you can you can you can you can you can you can you can you can you can you can you can you can you can you can you