 Okay, this is Michael. Okay, Mike is on Hello everyone Good morning. You can hear me. Okay, excellent So I would like to welcome you to our talk our presentation a short introduction into the training labs project it's a project that we started to support training groups people who try to do trainings in OpenStack environment our mission is to provide an OpenStack cluster for training for trainees What you're trying to achieve is an automated setup Something that is very easy to install. You just click something and it will give you an OpenStack cluster We're aiming for minimal hardware requirements for ISAs we were Able to do this with four gigabytes of RAM a small hard disk for Juno and Later, we might need a little bit more RAM, but eight gigs are really plenty We support all the major platforms So you can install that training cluster on Linux OS X or Windows We have minimal dependencies. You will see that all you really need is a Back-end a hypervisor. We're using virtual box usually And our software that you can just download It should also be easier to remove because many of the students They just want to check out OpenStack do a training and then remove the software without having a lot of software in their environment or in their registry or wherever Well, we also like to give you is a Reproducible environment because it's important that when you do training that you know What your environment is going to look like? It's going to be the same every time What we're using is the install guide official OpenStack install guide is What we are reproducing we're following the steps from those manuals to install the same cluster that you would get if You followed those instructions And last but not least We're providing an OpenStack cluster that you can use and that you can rebuild without access to the internet so if you're doing training in a place like here where everybody's using the internet and Wi-Fi is flaky you will still be able to provide Training and the cluster for training because we don't need internet once you get the initial software setup It will work automatically without internet access Hi, I'm Saili Lungar and I work for SUSE and I'll be taking over the rest of the next few slides. So like Roger said we mainly provide a training cluster and Since we follow the install guides very closely We support the main core services that are in OpenStack. That is keystone horizon glance No, one neutron cinder Heat is supported in Juno and Kilo Celiometer supported, but it's not It's not active by default so you can Activate it and then you just need to do some configuration to use Celiometer, but Yeah, we can We can use it So let me show you what we do on Mac OS X and Linux So Firstly, you need to get OS bash so We have a repository on gate.openstack.org. It's called training labs You can clone the repository since we recently moved to a new repository. We don't have The tar balls yet, but they will be available shortly So you can either clone the repository or you can get the tar balls Then next to run OS bash so we have Two options to run OS bash one is with the GUI and one is without the GUI so we mainly use watcher box currently to Deploy the cluster so if you do if you if you parse the minus G GUI option then you can see the VMs that are actually created and What's going on inside them, but if you you can run it silently as well? So once you run this it's going to start downloading the stuff. This is only for the first time So we it downloads the ISO image currently we were running on Ubuntu So it's only the first time that it will download the image of course and then once it's cached it will just use this So the base disk initially a base disk is created we will get into Deeper understanding of this later, but it's mainly just installing the operating system and Caching the packages that we need for the open stack packages and the other dependencies that are required so you can see when the script is running at what stage and Oh Which part of it is going on so the in it upgrades downloads and then which services getting installed so you can see it's installing my sequel and You can just follow and then once it's done. So it shows you how much time it takes roughly The first time it'll take longer because it's downloading the packages, but once You have it run the first time it takes belly 20 to 30 minutes to run the second time so Once you run it successfully you can log into the nodes. So if you have the GUI option enabled and you can just You can see the VMs here You can log into the VMs or you can we can use SSH To log into your controller computer network. So we have a three node architecture So we have the control computer network and the architecture will be explained more in details in the next few slides You can access the dashboard and You can create instances and do whatever you need to do. Yeah We also have a script that you can run to launch an instance. So it's just a test script. So You can you can find it in the tools folder and this is the command to run The launch instance script and it will just launch an instance and you can probably ping your instance SSH into it or anything else and If the test succeeds then this is the result you get And you can SSH into the VM that you create using the following IP. It's a seros image by default and There you go. So you can log into your guest Hey guys, I'm Pranav. I work for SUSE and Also help contribute to this project to make it more awesome so that more and more people can deploy OpenStack without having to Touch their machines or change anything on the host, but just install virtual box We should also do it for Windows because a lot of people use Windows in the universities A lot of students they usually like Windows for some reason and also in some big companies or corporations So not everyone is a Linux user. So that's why let's do something with Windows So as Saili mentioned, you can either clone the repos clone the training guides repo or you can get the tar ball Just expand it or the zip The zip file for Windows As you can see the bash the dot SH file is useless in Windows. It can't run it You could try with Cygwin, but it's It's not recommended. So we have some batch files which automatically generated It's generated by the by the Jenkins shop on the back end But right now we don't have the tar balls ready yet. The repo is quite new So here you can see that there are a few batch scripts and the first The first thing that you have to do is create the host net So this creates the virtual networks or host only networks on virtual box or On the on the Windows platform, which allows virtual box VMs to connect to connect to the host machine communicate with each other and That's that's quite important for a multi node cluster the second part is to create a Ubuntu cluster node and But before that we need the base disk as Siley mentioned and we'll get into more details on what exactly is the base disk and why do we need it? Why not just install a node and call it controller? but for now just understand that we need to create a base disk and So for that you run this script and Then the next one is to create the Ubuntu cluster Which means that it creates a controller computer network node cluster on Ubuntu or Using using Ubuntu as the guest operating system so you can just open double-click on it or open it via command prompt and You can see that This is installing the base disk which is installing the operating system creates the base disk and then You can install the control you can start the cluster which installs different nodes like the controller node here and After a few minutes you can find that all the three VMs are up and running These this is because this is a multi node cluster. We have We have the three three nodes controller network in compute node and This is the most awesome part is that you have step-by-step snapshots on every VM on every node This allows you to you know play around with something say hey, why not try Keystone API The new version, but I don't know if it will break my existing cluster so you can just revert back to this snapshot and Do what you want to do Continue if it doesn't work if it breaks you can just use the stable snapshot So I'll hand over To Roger who will explain how exactly we do this Okay, so I'm giving a brief architecture overview The OS bash workflow starts with a user and their PC or their laptop first thing they will do is That's what we showed in the previous slides is they will install virtual box or in Linux It could be KVM as well So they have to have a hypervisor and in addition they just have the the OS bash files It's just one directory that they already have somewhere They use OS bash calling it which will operate the hypervisor We're using the Vbox manage binary to remote control the hypervisor Always bash also transfers client side scripts To into the VMs that get created Those client side scripts are just bash scripts that mimic what the install guides do and With that you get also The KVM installed on the compute node. So you have a hypervisor inside your hypervisor and Then you can install Guest VMs you can run guest VMs on your virtualized cluster You can SSH into the controller node. You saw that you can SSH into the other nodes You can also directly SSH into the guest VMs and from there going out Okay, so let me just quickly explain to you what exactly happens in there So the first step is to install the operating system So what we do is we install the operating system once create a temporary disk and Then upgrade update the packages upgrade the kernel reboot it and Catch the packages. So this is very important because in in many training scenarios or Many places where there is community training or even in professional training or hands-on sessions the internet never works right and That's a very big problem. So we do cache all the packages here We do not use any any special magic or any Anything like a caching server, but we just use the download only option So apt gets download only and you can also do that with yam and zippers so what it does is it catches caches the package on the file system and You get the base disk. So this base base disk is ready with With the required operating system installed in it with all the package Packages already present but not installed So the next step is to deploy the controller node. So it creates a copy and what it does is it does not create a Clone of the disk but starts working on top of the base disk So if the base disk is around 8 gigs the controller node will would stick a few MBs or 200 or 300 MBs depending on the amount of storage space it takes or amount of things that That is unique to controller node So we we have a very compact We have a very compact cluster and the biggest disk is base disk Which is around seven or eight gigs depending on depending on the operating system version and what version of Open stack you're trying to install So during the deployment What it does it it takes a snapshot after every service as I showed you all the snapshots in virtual box QE so it takes a snapshot after every service it installs The the services and this is kind of like a loop till it finishes all the services on controller node It moves on to compute node. That's the same thing and the network nodes and after Installation you get the cluster and I will explain I'll explain how it is How exactly the speed is increased so the base disk is installed once and you need internet for it And you can take the base disk take it with you in a small flash drive go anywhere Plug it in and run the cluster it will install without internet So it has the updated operating system. It has all the packages cached in it and The internet requirement is only when you create the base disk and you can use it As many times as you want and in any kind of environment it will work without any hassles The other awesome thing is node snapshots. This is really really handy when you have limited training time or limited demo time so maybe you have around one hour to explain some concepts and Let's say you want to just showcase how to install heat or You are developing install guides or you are trying to test out some code that you wrote or your DevOps guy Who wants to understand how Cinder works? So why do you want? So that person or the person who just want to test out one single component open stack Does not need to install the entire cluster that he can just use the snapshots reward back to Cinder as I explained earlier and That is very handy and it's it's very fast. So you can actually give 15 minutes for one service and Give give a very nice amazing training session in one hour and that's could work really well You can also Restore restore to any point so Let's say one of the services fails to install for some reason So you can restore it back and you can start or continue from that point So this is another point of view or this is kind of like the architecture or whatever diagram you want to call it so the host operating system is on the laptop or the base machine or workstation and Please take a note here. This is not for production deployment, right? so usually people want to deploy open stack on a bunch of servers, but our project is is made only for home users and for students So you can have all of the three mentioned Operating system versions which with the processor that supports virtualization. This is important because we are spawning open stack or cloud instances on the second level of virtualization and That requires a little bit of power from your processor which means that you just need something like an i3 i5 or equivalent quad core for AMD and You actually don't need a very powerful processor and I think most of the modern laptops they they come with decent specs to run this So this is the repository the folder that you clone or you get from the tarball and This and this is the content of the repo So I'll just quickly explain what each folder means. So the config folder contains the configuration. So let's say your your home network the The network addresses or the subnet of your home network is conflicting with one of the host only networks and you want to use both of them So you can just change the network parameters You could change the amount of RAM given to the Controller or compute node and you can tweak a few configuration systems there. You can say hey, I don't want a snapshot for this For this service or hey, I don't want to install this service. I just want to you know Do something on my own? That's one of the one of the points the image folder caches the base disk and the operating system image The logs of course, it's very important if you want to debug or if you want to just Take a note or look at what is exactly happening later on The scripts is the is a folder where you can find all the required guest site scripts these guest site scripts are useful for It's it's the one-to-one or almost equivalent of the installation guides in the form of batch scripts And they are responsible for installing each and every service Inside in the VMs here So the compute node controller node and network node the services installed come from the scripts folder the tools folder has some handy tools like the test scripts or repeat tests so That's that's what we use for testing our cluster because we need to know if If we can deploy the cluster Properly or the deployed cluster does it that's actually spawn instances or does it? They'll freak out somewhere in between and the lip folder. It contains the library files or the files which are required for For helping out with installation for virtual box for injecting the scripts inside the VMs and running them So this is basically how How it it looks like when you when you want to take a deeper dive or when you want to start understanding how this Training labs projects work on the technical side So I'll hand over to Roger who will explain how to debug the clusters and okay I'll just say a few words about our debugging tools because when you try to When you try to port new releases of Open stack to our training cluster its handy if you can develop quickly and see why something breaks or doesn't work anymore just following the Installation guides often will result in a broken cluster because there are some issues that are not mentioned in the install guides So the first thing we have is lock files They're quite verbose and they tell you everything that the scripts are doing that they're happening on the high Provisors all the commands that we're giving out Then we have a test script that was already mentioned by Siley launch instance That's a script that tries really really hard to launch an instance as you know It's it's quite easy to launch an instance if it works, but if it doesn't work there are many failure modes. There are races So this script just tries to launch an instance and if it doesn't work Then it tries harder it tries to go back or it tries to do a different way until it succeeds This is also good as an instruction for students who try to find out how to make instances work somewhere else We have test scripts to repeat tests so we can do a thousand Test runs for example, which is important because there are quite a few race conditions in In the open stack code or in the way the servers interact and they're often hard to reproduce So we have tools to repeat Tests again and again and so to make it to speed it up even more you can just start from a Snapshot and just do like the last few steps of your installation and see if the race happens there or not This is a quick comparison between what the install guide May tell you and what we're actually doing in a script So this of course it doesn't look that different everywhere, but in some places We add quite a bit of code to make it work. So this is what the install guide says This is literally copied from the install guide and this is what the script looks like And as you can see we're checking for example that the VLAN tags are coming up and that they don't show Or indicate an error condition, which actually happens occasionally So these these three lines are the three lines Here the green lines are the green the three lines here and the fourth line didn't fit here because there is more testing code Underneath the DHCP agent restart Now we often get asked what is the difference between OS bash and dev stack and I hope you already know the answer by now because We try to show that in our Talk so far, but just to make it more explicit Dev stack is a tool that is aimed mainly at open stack developers So it uses the latest source code from the open stack repos because that's what the developers want to see That's what they want to test Whereas we install code from the official distro repos In addition our code it manages to VMs So the user just has to double click something or start a script and it will automatically do everything create The cluster create the VMs install the scripts do everything to set up the cluster and there's no configuration Needed it it works out of the box. It will install install guides cluster out of the box So that gives you a predictable environment for training that works Now for the roadmap what we have planned for the future We would like to finalize our ports to kilo and Liberty kilo is mostly finished, but we still have some rough edges and After we're done with that we're going to port the training labs to to Liberty Usually that's not a big problem because training doesn't happen on cutting edge distros anyway We already have a KVM backhand The main reason we have that is because it's easier to use for integration testing on the open stack CI Infrastructure, but we still like to make it more versatile and match all the features that we have with virtual box This is the CI system and that we're still in the process of building on the open stack infrastructure and We would like to port the host side code of our scripts to To Python because right now we're using bash for everything and we're going to stick with bash for the client side scripts Because we want to match the install guides as closely as possible But on the host side, it's just easier to real to use a real programming language But we will but we are committed to do our coding in a way that it will work I will be supported on all three major platforms as before What we would also like to have is support for additional distros right now Our coach should run host side and pretty much all latest distros, but client side within the VMs Right now. We're just doing Ubuntu So if somebody wants to help us out and add something like Santos for example or Suze, they would be welcome What we're also looking at is some form of bare metal deployment because there are just some things you can't really do training with If you have all your infrastructure virtualized, there are still some differences between a virtualized and the hardware environment So we've played around quite a bit with Raspberry Pi and we have some VMs running Already in a virtualized environment on the Raspberry Pi But we're not sure yet if the Raspberry Pi 2 is going to be powerful enough for this. So this is still something we're working on so since we Told y'all that this is mainly a training use case used for Training purposes. I had conducted a few trainings in India and I actually conducted around four trainings, but we have surveys from two trainings it was a two-day training and one of the days we were using the training labs for a multi-node this deployment we were around 60 people 60 students and I think we We got training clusters running for most of them except maybe two or three exceptions and on Linux and Windows so it was pretty successful and You can see the survey results there Just added a snapshot from the survey results and Yeah, it was a good session As we said the factor that we didn't need internet what really helped because we we cached all the packages and Replicated them on all the machines. So all they had to do was to run the scripts and they had Multinode cluster running and they could play around with open stack And that was very useful because that's generally the most difficult part when you get started with open stack that you don't you spend Good amount of time just trying to understand how to deploy the cluster before you can even get to know Understanding what open stack is so that was one of the feedbacks that I got from many of the students there So this was a smaller training and about 20 people and This was a bigger training with about 60 to 70 people. So Yeah, those are the results If you need to know how we went through the training in more details You can come back to me after the session and I can probably tell you all more about the hardware configuration and how we manage all of that Thank you for coming and You can contact us on IRC or our email So if you have any questions I Think we have some time left. Yeah. Yeah, and we have the meet up meetings every weekly meetings on You can find the link for the meeting here. It's on Friday This is the meeting that's at the summit and we also have a weekly meeting on Google hangouts, I think Yeah, there's a question over there Could you hear me? Yeah, even better. Okay, so So for those in here, okay, so I bootcamp instruction part of the delivery includes The actual, you know, typing the commands of you know going from one step to the next does this this is great because I was actually working on this Now I can stop and And I could hopefully see what you guys have going on there and maybe help out a little bit But is there a way of stripping out the the the process that actually pulls the packages? You know I'm saying or the process that actually configures the packages so that we can just simply have like a lab guide Where this sets up the base environment and then we can have you know all of like the snapshot Statch, you know, so if they're going through and they are having a trouble with Nova I could just revert them quickly to Nova and they can have like a working environment See the VM and then continue on from there So is it is it possible or is it easy to break it up a little bit? I Think that's the part where We use the screen the snapshots for at every stage So we have it literally at every stage even before we start building the open stack services So if you want to roll back to know why you can easily roll back to know what maybe do the manual Instructions from the install guide if you want to do it and then move on from there You have to remember that that they are cumulative snapshots. So if you revert back to a point Right here. So all the all the snapshots below it will not be You won't find those services installed and there is a reason because we need there are some dependencies like without keystone You cannot have the other services running. So that's the sequence of the snapshots Let me add to this that you can easily customize our scripts and the way It's set up right now. There's just a text file That is a list of all this client-side scripts that we run inside the cluster So you can just comment out those that you don't want to have installed or delete them or add new ones if you have some And it will just install that part of the cluster that you configured there It's very easy. So when you get that when you get the whole cluster up and running does it Would I be able to revert each of the nodes back to like that clean image? Yeah, so we actually revert them all so to clean so basically they have a cluster. Yeah, and then they can build it from there Yeah, okay. That's very easy. Yeah, and we're happy to support you if you have any problems just shoot us in the mail or talk to us Also, if you want to take a look at the cluster we have Running cluster on our laptop. So if you're one you can come there and just check out the code and Maybe take a look at the dashboard or something Do we have time for more questions? Yeah, we any more questions? Yeah, we could take one more questions if they're one and if not you can always catch us later or meet us at the documentation Meetup tomorrow will be there Thanks all for coming and listening to us Hope to see you later