 Hi guys, my name is Amol Chaubey, I work for Ericsson and here is my colleague Sudeel. My name is Sudeel Ketamaka, we both work for Ericsson specifically for cloud management solution and Amol is the infrastructure architect and I lead one of the development teams. It's going to be Amol's show today, I'll be helping him out but if you have any easy questions, I can help with any tough questions, I'll ask Amol to help out. So we are going to keep it simple, feel free, free to interrupt me and ask me any questions. So the way we are dividing this particular session into a short presentation about the good practices on the logs and monitoring aspect of it and then there will be a lab introduction, in fact everyone, the prerequest for this lab is to have a virtual box installed and yesterday I believe the email was sent out for everyone to download the OVA but if you have not downloaded it, we have the USB's in the room so just ask for folks and start copying the files because it will take quite a while to copy all the files. So what we'll do is we'll start with the short presentation and then we will go into the lab introduction where we'll set up your DevStack, it will hardly take 5 minutes to set up the DevStack if virtual box is installed. So as you mentioned, we both work for Ericsson and we have been working on OpenStack since Qualcomm release and since then like every release, all the new features are coming out and we have been playing with it. So when we started with OpenStack, at that time, especially me and my team, we struggled a lot because at that time the installation was not that mature, now we have a lot of vendors around who had like literally one click installation that we have for OpenStack and all that. But installation is one aspect of it, once we install, it's done, then what we need to worry about is maintaining that OpenStack, troubleshooting that particular OpenStack. So this particular session I would consider is more for the beginner to advance that level and that category falls down. In a sense like you got to be little familiar with OpenStack components and basic architecture and the labs are pretty simple for the advanced users but if you are a beginner, the labs will be kind of challenging for you. So OpenStack, like in the architecture we have a lot of moving parts, so if you look at it, if you want to create a VM, while creating a VM we got to go to Keystone for authentication, once you get authentication you go to a glance to download the image, then it is pushed to no compute and the VM is created. So there are a lot of moving parts and because of that troubleshooting become quite a challenging aspect because each component has their own logs and we have to then dig down to those logs and try to figure out where could be the problem. So like at high level, I just put a basic building block there with the core component where you have a controller in the center and then you have like a NOAA, Neutron, Keystone, Glance, Swift and the Cinder components surrounding it but apart from that you have a basic set of hardware, on top of that you have an operating system and then the hypervisor. So when we are troubleshooting OpenStack, at that time we got to worry about all these individual components. So what we are going to do is we will try to look at the basic aspect of how to troubleshoot. I am not going to go to each individual component troubleshooting but at least a high level where we should be looking at and what are the typical pain points for everyone. So starting with the very basic stuff like logs, in the OpenStack the logs are like for each component there are different log directory and you have to dig through it. So on the left hand side I have listed out for a typical distribution of OpenStack, what are the directories, not necessary all the distribution will have same sort of log location but the lab that we are providing you is DevStack environment. So in that environment the log is located into the OpStack log directory. So that is one thing I wanted to point out because if you try to look at the directories which are listed on the left hand side you may not find it on the DevStack VM that we gave you. We have been using, for the commercial version we have been using Mirantis OpenStack in Erikson. So those are the typical path for Mirantis and perhaps it is the same one for Aldio. I am not that familiar with that one but I think the path is still the same. So now the basic component that we have got to worry about here is the monitoring aspect of the OpenStack. So what we should do like I am giving an example of FCAP model which is ISO model basically. We define the network management task. So like FCAP as it is listed out stands for fault configuration, accounting, performance and security. So not necessary like one tool will be able to monitor all of these things. You have to figure out what is the best tool that will monitor all these components. So and we need to worry about checking at multiple levels right. We got to worry about at the host operating system level, at the hardware level, at individual OpenStack component level and also at the VM level which you have provisioned through the OpenStack. Typically the VM level could be the responsibility of the application side for which you have created the VM but these all monitoring we have to do that in order to ensure our OpenStack is working sound and good. Now in the best practices right there in last couple of days there were lot of sessions on Elk right which is elastic search, lockstack and Kibana and I am not sure how many of you got a chance to attend those but those were the good sessions and in the market there are lot of commercial tools available like Splunk or Stackify which are commercial tool but if you do not have those tools right from the open source point of view Elk is the best model I can think of where which will collect all these different components log in one dashboard okay. So in nutshell what Elk is Elk is elastic search, each stand for elastic search for deep search and data analytics, lockstack is for the centralized logging and you pass those log as per your need and the Kibana is the GUI which is UI interface to it. So if you have not look into Elk model I would strongly suggest look at that and as I said there was one session this morning yesterday also there was one session on that one go through those videos and try to do it yourself it will help you in the long run. Then there is something called intelligent alerting like using a tool like Nagios or something like you can periodically check and get the alerts like if something goes wrong or something is not right Nagios will keep on sending you alert and that is the reactive state you will react to a problem. Then there could be something called proactive state where through trending you can try to find what is going on right. So what is trending? Trending can give you a great insight into how cloud is performing day to day right and you can learn like for example right on a busy day is it like if there are lot of activities on NOAA happening there is some slowness is it a kind of a rare occurrence or is it a time for you to react to it and start adding more compute nodes so that you can ease out that performance issue. So trending overall is like little different than alerting model right so while alerting is based on the binary result 0 or 1 whether it is working or it is not working but trending it kind of is record the current state of something at a certain point and that will help you to analyze how your open stack is performing. Then there is something called guru meditation report that gives you so NOAA has done a good job of like integrating with that one and if you issue there is a command like kill-user1 if you issue that command kill-usr on the PID of the NOAA API the guru meditation report gets generated and it gives you the current state of what is going on in the NOAA it gives you the thread count it gives you what configuration your NOAA is running and all that so we do have a lab on that one at that time we'll discuss in more detail about that. Any questions so far okay I'll move on so now in troubleshooting right where to start we have like operating system level tools which if you are from the sysadmin background you are quite familiar with it what are the operating system level tools or there are some open stack specific tools right so using the open stack CLI you can troubleshoot then there are some open stack specific tools for administration aspect of it so if you want to create a user or if the VM is if you want to set up reset the state of the VM the so NOAA CLI has a very specific tool for the administrator so in terms of the troubleshooting tools right the tools used to investigate or fix a problem within your stack right so that will help you to narrow down the issue where the issue is okay and also these tools need to span virtualization networking and the normal system at administration operation so that also you need to touch base all the domain in order to figure out or troubleshoot the open stack problem something that we did in our lab which is like we wrote our own canary script like which will every five minutes it will run it will create a VM create a volume and delete it every five minutes it's gonna run and if the run is not successful we'll get alert message the advantage of that is I'm creating a VM right that means my keystone glance NOAA neutral on all these components are working fine if VM creation is successful if I'm creating a volume that means my cylinder component is working fine so this gives me like at least a assurance that this is working fine if you have a third-party commercial products like in Agios and other then you don't need it but if you don't have those start with this at least it will help you to get alert in this aspect so canary script again it is specific to your lab we are more like in my particular group we are more focused on these these core projects but it's really you have to decide for yourself typical commands I believe most of you will know like how to check the statuses of like components of neutron or up or components of NOAA is up and running so for each component there are different set of script so try to look for the NOAA manager sorry hyphen manage whatever you are using cylinder manage or you neutron manage a neutron doesn't have managed they do it through the agent list but most of these core projects right they have like a hyphen manage extension and that's where you can like monitor the current status of it one thing I would like to bring up here like NTP is very important here okay so when you are across the different nodes which is running different components of open stack if your time is not in sync right open stack things that it is down should ensure that you have NTP server setup for all these operations now troubleshooting hypervisor so again it varies by hypervisor but most of the common hypervisor use is the live world or KVM hypervisor okay so each one has their own tooling so when you try to troubleshoot a hypervisor right we need to kind of map a virtual machine to a hypervisor using a NOAA so command and then investigate a hypervisor through the worst tool so like typical example is I'm not sure how much clear you can read on the screen but you create a VM right doesn't matter whether you create it through the heat stack or normal horizon or NOAA command but you create a virtual machine then what you can do is you can do a NOAA show on that one and there is something called OS extension server attribute for host and hypervisor it will tell you where is your hypervisor running on what host it is running so what you can do is you can assess to that host and try like a KVM has a worse something like a worse command line utility you can do a worse list and try to find out whether my VM is running or not there now here keep it in mind DevStack that I gave is all in one DevStack that means it has the operating system hypervisor and all the components are running on the steam virtual machine but in reality you will not have this scenario reality it will be pan across different host so now you figure out right you are on the like you can go on NOAA compute host and go to instances folder so the VM that you have created right it is it has a UUID so NOAA compute host will have a directory called NOAA instances now in our case in case of DevStack it is an off stack data NOAA instances but typically there is a separate file system for NOAA on the compute and that's where this folder will be what this folder tells you about it has some files like console log disk libworld XML these are very important files because like I'll give you example of like libworld XML libworld XML contains the information about the network that you have created right so neutron create a tab device for that one so it has the name of the tab device you can actually go in there and try to figure out what is the tab device as allocated by a neutron for that so in one of the lab we are going to look at that the VM that you create you're going to look at the what is the tab device mapped to it so this help you a lot in terms of troubleshooting when you are kind of clueless what's really going on apart from NOAA API and scheduler log also what you can do is you can use the chemo tool and get a info on a disk which kind of tell you the what is the file format what is the virtual size of the disk and all that information so disk comes in very handy while troubleshooting now troubleshooting network now this is the network problem is like for the beginner or even for the some advanced user it's freaked out people once it's a network issue people are clueless and try to give up like I'm not sure what's really happening so again with experience it will come at the more you do a hands-on on the open stack it will come but the majorly right there are three areas that we got a look in troubleshooting networking one is at the Linux level one is at the open we switch level and then there are some open stack tools for it so for example right the open stack tools will show you the logical configuration of your network so there are something called neutron port list command what it does it it lifts out all the interfaces and the port allocation and then it isolate to a particular user or the netlist it shows you a network created by the cloud operator or a provider network or a tenant then router router shows a tenant or provider any router that has been created again it's only a logical view from open stack but then if that doesn't help you right you can narrow it down to open we switch level where there are some command like OVS VSTL and DPCTL what it does is it will kind of map the internal and the external bridges okay so there you can figure it out like what is the tab device what are the bridges created for my particular VM so we do have a lab exercise number three where you gonna run these commands and try to map it to the virtual machine that you have created and then there are very specific Linux tools that is very common but one is like we need to look inside the VLAN or Linux name spaces because Neutron heavily uses this for if you are using a GRE type network or VLAN or VXLAN type network so this IP net NS is your friend it will help you to narrow down the issues there okay so we have a lab and at that time we can talk more about this particular aspect so in nutshell right this if you do IP net NS so at that time it will look at the name space okay and the DHCP get created so whenever you create a network right you need to know all the flows it goes through so now sometimes what happened is you create a network right and when you boot up the VM you see that there is no IP address assigned again you get clueless like why there is no IP address assigned on open stack horizon or command line it shows there is IP address assigned but in reality when you started troubleshooting you realize that oh my IP net NS so inside a DHCP I'm not seeing any tap device for my particular VM so then the most common problem is there is something called DNS mask queue which is not up and running and you restart it easiest way if you are in the test environment is just to drop that port and recreate it that will solve the problem but again you should be able to narrow it down where the problem is okay so we do have a lab which is going to cover that one now then again we go to the operating system monitoring right so again at operating system level you got a monitor CPU memory the availability of the OS itself the space disk space couple of times I have seen issue where the host OS then you have VMs running sometime the host OS run out of the disk space and because of that VM is not up and running because VM try to read or write some bytes to that disk space but since there is a hundred percent disk space utilization you cannot really write to it so always never ignore the host OS monitoring because sometimes the problem is at that level and you will not able to figure it out from the open stack logs then process level information so all the open stack processes like we talked about the core component but then there are cues rabbit mq is very common there is a database post grace or my scale or if you are using kilometer MongoDB is very common for that so you need to have the process level monitoring as well for these processes okay and of course all the open stack component related processes okay and also you should depend on the host to send out a notification to monitoring server what do I mean by that is when in the cloud world right you will have n number of compute host so you should prepare a model where host is notifying back if something goes wrong host is notifying back through SNMP that this is wrong rather than a monitoring server is constantly probing the whole this is the right practice and the last part which I wanted to touch base is the debugger this is little bit for the advanced user where you are trying out everything okay and issue could be at the open stack itself at the API level itself and at that time what you can do is open stack for all the component except horizon is written in Java in Python so you can use a Python debugger okay so this will help you at least to figure out where the issue is but now here the challenging part is you have to have some knowledge of Python language and how the API has been implemented so it is going into a little bit advanced user category but I will we have a lab actually that will show you a very simple use case and you can create like set up the Python trace and figure it out where the issue is so this will help you once you get advanced with all the troubleshooting tool kits and all that debugger component will also help you to narrow down the issue and if you can do that you can even contribute back to community if you figure out this is the bug just you can contribute it back default ports as a open stack admin or open stack user you should get familiar with all the default ports because on all the logs you see this default ports information and you should be aware of what component is running on what port you should be aware of that no matter what so I have listed this out what I am going to do is after this session is over I am going to upload it on my Google Drive for which there is a link I will show you right now so these are again the ports so in means I am just concluding the presentation aspect of it so it is a complex topic open stack troubleshooting okay but it is very easy to navigate and troubleshoot if you have proper tools and proper mechanism to monitor it okay you definitely need to have a proper understanding of open stack architecture that is going to help you a lot at least how the flows are how when you create a virtual machine or when the neutron goes and create a port what really have happened in the background at least have a basic knowledge of that aspect and that will really help you to troubleshoot now hands-on is the key there is no alternative to hands-on you have to try it out play with it and for that day stack is the best tool like install it on your laptop and start playing with it virtual box has a feature called snapshot so if you have clean image take a snapshot of it and then play with it if it goes something goes wrong reward back to a snapshot you can but hands-on is the key here so let's start with the lab now so I am assuming that everyone has copied the files that we have provided for the lab anybody who still need to copy we have usb here yes I have slide for that yeah any question so far a more question here yeah so the question is if the image has stopped working that he is not able to import the image well it should be because I did it like yeah it could be because we had tried it out yeah yeah let's let me go to that screen any other question will go to lab yes it is possible to assess it to the image so I'll show you that so what you need to do just to make sure that you are using this 192 168 64 network if you want to automatically start up the image ensure that you are using that network how can you do that so in the open stack right sorry in virtual box go to file preferences and network so I can show on my screen it is before actually did you know imported good before you started so import is good if you import it you are good but then what you do is go to this file then go to preferences then go to network and ensure that you have a NAT network setup okay and in here in the host only network at least I have three here but if you go to to ensure that you have 192 168 64.1 define that because what happened is the image that we have provided you has 192 168 64.21 as a static IP address so if your virtual box has this network defined then when you boot up the Dev Stack and you start up the Dev Stack at that time everything will be up and running so ensure that you are doing that let me check so 4.3.26 and higher so I tried with 26 and so maybe that could be the reason that virtual box so in the USB we have provided a binary for virtual box as well yes there is yeah okay but just to assure you the OVA is good that good so when you are able to finish that you can try it out yeah yeah just just set up like that and that will take care of it what kind of IP address dot 21 yeah yeah and the credentials are Dev Stack Dev Stack one two three so if you are not able to assess that means at the virtual box level something must be wrong so yeah so this is the IP address that will be so how many of you are able to at least set up the Dev Stack so far okay okay so I what so you know what what we can do is I'll just run over the flow one more time of this one so okay okay let's let's go over the flow okay so see I'm gonna be I'm gonna stop mine and I'll go to over the flow so sorry let me duplicate this I cannot see how to duplicate it okay so I'm gonna stop my I'm gonna power off my Dev Stack so let assume you are at a state where you install the virtual box right and you imported the image that's all you did so far okay so virtual box is up and running and you imported the image which is in this state okay now first is go to the file and preferences okay and go to a network so file preferences network ensure that you have a NAT network defined there if not just click on this add button that's all you need to do there okay then you will go on host only network you may not have anything if you installed this virtual box from scratch so what you will do is you will go and do a plus here okay it will say yes yeah so it will create a new network interface for you but what you need to do is go and edit there and ensure that that you are using 192 168 64 dot 1 rain there that is the first thing that you need to make sure you do that okay and then you click okay what will happen is virtual box will create that interface for you you okay you okay you okay you guys okay here okay you guys okay good you guys okay so what I'm trying to say I tried it on Mac to follow so exactly go to network right host only network so now hold on no no no this was already defined for you just to let you know because so just to let you know this is the page link to the Google Drive you can do a QR scan as well that's where all the OVA and the labs are posted as well as only thing is remaining that I need to post there is a this presentation which I'll upload it but you can download it from there one note for the people who are starting the first lab real quick is what I noticed that the curtain paste doesn't work when you try to do the first lab the heat template heat template is YAML and it is quite sensitive about the format so this PDF the curtain paste is not it is messing up with the formatting of YAML so I actually added a line there in the lab I just updated it an hour ago on like a W gate command so if your VM is connected to internet you can directly download this file rather than trying to curtain paste because you will get a error if you try to cut and paste unless you type it by hand but I would suggest do not do that on my page here I have also put the YAML file you can directly download it from there and FTP so whatever you are comfortable with W gate or FTP do it from there yeah you can actually easy part is go to that page and download the YAML or you can do the W gate still not good good you got lucky or no okay and and and just oh no okay got it got it so actually I should have tell because the start DevStack script right you have to run that so if you are not able to connect to a network run the start DevStack script what it is doing it is bringing up the network I forgot about that yeah no you don't need to run stack.sys at all stack.sys will reinstall the DevStack for you you don't do that so if it failed perhaps I would suggest re-import OVA because you are you probably might be in the different stage did it try to do one field yeah so okay do an LS here do an LS here and there is a VIS start go go go one directory up go one directory up start DevStack no no so now it might fail for you because it kind of in in the stack.sys might have tried to recreate everything I doubt it you might have to re-import OVA one more time but try it out dot slash start DevStack.sys so this is starting the services and that is one of the reason we are sticking with that 10 sorry 192.168.64 network because it is pre-configured you just start the DevStack and start using it stack.sys what it will do it will try to redo the installation of DevStack so that's a different so let's see what I would suggest it try to once it come up try to go to the horizon go to browser go to your browser type 192.168.64 21 21 that's it that's it let's see if it it's you might have to re-import let's see so just a note this is pre-configured DevStack don't run stack.sys or something just start up the DevStack there is a start DevStack script just use that to start it up if they go to that Google Drive they can download or you can open another session or you can get out of screen yeah controller 2D so that this network you define right you log into DevStack okay and ensure that you are running the start underscore DevStack.sh that is going to start your services I should have put that in the lab page now now all the services will come up yeah that's good now leave the screen open or you can do it is still running it's on the screen so go to the lab folder and download the MyStack.yaml file either delete the old stack or create the new one with some hyphen one or some different name so what happened is just a background NOAA right when you create a virtual machine it really doesn't care it allow you to create with the same name again I can create 10 virtual machines with the name something but heat they have a unique constraint for the stack name so that what the error is NOAA completely work using a UUID which is unique for each name they all their call goes to the UUID so you change the flavor to M1 tiny I'm assuming no no no no no no but the file the ML file what did you update the flavor to this will fail now this will fail this is supposed to fail good hi there is a solution towards the end of this lab but what I expected if you notice you created you try to create a stack with the flavor of M1 extra large extra large flavor need 16 gig of RAM your day stack is running on 4 gig of RAM so you are going to bomb out saying that host not found from hits that host not found means what happened is call is go goes to NOAA scheduler and NOAA scheduler try to find out appropriate compute host which has that amount of memory and since we do not have that amount of memory it tells that expected RAM and what is what you are trying to utilize is not there so the work around for this lab is just change the flavor to M1 tiny which is M1 tiny is of 512 MB RAM so just try to change it and redo the stack the people who are moved to lap 2 just to note in the network try to use a predefined network which is a part of the image to create it and it is going to throw out the error and try to troubleshoot it according to a locks or anything is always run if it failed run NOAA show and the name of the VM and exactly exactly exactly look at the so I know for sure but so look at RAM I know okay look for the RAM try to at least cover the high level what are these lab for and what you need to watch out for in the in those lab so lab number two right we created we have given you the predefined subnet and we asked you to create a virtual machine with that subnet if you notice that subnet has a CIDR of slash 30 that means it doesn't have IP address to assign to your VM it's gonna bomb out so alternative is create a new subnet and then recreate your VM you'll able to create a VM so on purpose I gave you a subnet which really does not have any IP addresses and it tried to assign that IP address to a virtual machine it's gonna bomb out which so the first thing that you can do always is do a NOAA show okay NOAA show and your VM name that itself if you look so so NOAA show VM will show you this error basically fail to allocate a network with the error no fixed IP available dadada okay but now which log file you will go at okay now since you are running a NOAA command right so the first request is gonna go to NOAA API because you are running a NOAA command so look at the NOAA and API log NOAA API log first so most of the distribution that we have right now they sync up everything at the controller node so if you have five compute host right most of the distribution available kind of sync all the locks to the controller and you can view it from there you don't have to navigate down to the compute host you have a centralized view so I would start though troubleshooting wise the way I will start is I'll go with the NOAA show try to figure out what's wrong mostly the common error is no host found but now in this case it is very specific about it I can certainly say that based on my prior experience is this is the change in Juno sorry this is the change in kilo you are running a kilo version of Dev Stack right now till Juno it was always no no host find no valid host find but this is much improvement in that one so you start with NOAA show right if you don't find any information then go to the NOAA API log and navigate through that one you don't bother with neutron log at this point because NOAA make a call and that away that's where it is telling you okay so that was the problem and then basically the solution is you will create a subnet with the class 24 range and that's going to solve your problem that is lab 2 so you'll create a two VMs okay so at the end of the lab what's going to happen is you have certain something like this view the two VM connected on the same subnet okay with unique IP address okay now the third lab is the networking lab this lab is it is nothing destructive in this lab the commands are there but this will help you to narrow down to find the UUID so you use a neutron netlist which give you a logical view then you basically narrow down like something calls IP net NS I'm doing a pseudo here keep it in mind it is only root can run this command and you figure out your DHCP so if you notice this is the UUID of my network and this is the DHCP UUID it is matching so again in your case it might be different as you create a different UU networks and all that thing but just to highlight the point okay then what you can do is you can go into a net NS and do a if config on that net NS to find your DHCP server okay now whatever VM you created right you should be able to ping to those VM using the net NS so this is just showing example so you should be able to ping both the VM okay now you should try this is kind of a tricky part where you will try to do SHH to that particular network and it's gonna fail it's gonna fail because the firewall rules are not there so you will basically go ahead and add a security rule for port 28 which is a port for SSH and then here I actually covered like how to find out the tab device associated with your VM that you created and this is a subset of the picture but if you look at the big picture here is the big picture like how the neutron network works and it basically you have a VM it create a tab device then a bridge and then that get hooked up to the physical layer of this stuff so these commands it just gonna help you to narrow down if you want to investigate what are my each layers are okay nothing destructive in this lab then the lab is the fourth lab is about guru meditation report which I was telling with tell you the current state of the NOAA environment so it is just like you just gonna run kill minus USR you're gonna find a PID I'm looking for 8774 that is NOAA API so I'm going to do a kill minus USR on that and in the NOAA API it is going to generate this GMR report which is the current state of the NOAA so then in the exercise you're gonna do a trending exercise where you're just going to go and grip for space 200 in the NOAA API log what it is going to do is it is going to give you account that how many requests was fulfilled on that in that given log file this is helpful because if you are generating a log file on the daily basis or something you can at least see the count of the successful request and you can keep a track of it like if the pattern is you are creating 50 VMs a day or 30 VMs a day you know and suddenly you see today you created 200 VM that can help you to like proactively figure out what's really going on do I need a more compute host or not same thing for info how many calls NOAA is getting you can get a count on that one then NOAA users list that is the command in NOAA which said for each tenant how much servers are there how many ramps are there like how many RAM is used for how long RAM hours it is in RAM per MB in hours that's how it's called but it helps you to figure out who is my tenant and how much resources that particular tenant is using you can alternative go to your MySQL database of NOAA and do a quota uses so if your quota is defined how much you are using or utilizing out of that when you are managing multiple tenants this is very helpful for you to figure out what is going on there okay so this again it is a very informative lab in that sense and the debugger lab debugger lab it little bit tricky but the what we are doing essentially is here is we are trying to add a new key pair in the stuff and we are giving a name like using these characters I know that this doesn't support these characters okay so what we are doing is I'm showing an example like take a backup of your API.py which is a Python file and then you're gonna add go to the line number 365 it set this trace up and restart the NOAA API okay once you restart the NOAA API and if you rerun the same command again your screen which is a DevStack screen it's gonna stop at one point and you're gonna go to that screen and type this safe car which basically read the code and tell me that these are the only allow values for this particular component so and then you get out and then you restart the API so this is just to troubleshoot it but get there are so if you Google right you will get tons of stuff so again in OpenStack there is something called Tempest files okay and there is a project going on right now where they have implemented a different set of debugger okay this is very basic but if you look into there are some white papers which I'll put it in my link when I upload so those white paper will help you if you are interested in that aspect of it there are tons of tools available in OpenStack for debugging this is the most simplest one for the NOAA's but there are very good and powerful tools available I think only one minute is remaining any questions and I will also send my email address my own Sudeer email address if you have any suggestion for future lab please do so yeah not just that I mean if you for a few days will will support any questions that you have I'm sure at leisure if you try this thing in a much more relaxed environment you should be able to figure out all these labs they're not really yes yeah virtual box virtual box okay sure sure we'll do that okay yes so that's the one I was telling you about that try to use this elk okay and if you cannot do elk there are tools available like Splunk or Stackify that those are commercial tools again it's expensive but at least if you have a good Java developer in your company you can take his help and it's just a matter of in a day or two you can set this elk thing yeah you can set up it on the controller node because that's where you have direct access you can even set up it remotely but then you have to make a call to the controller node either way yes exactly yeah yeah so what it does is elastic search it has your own database actually it pull all the lock parse it and keep it you know no no it is it is very good if the person you tell and you can narrow down exactly where is the issue and you can even map the no longer neutron lock side by side you don't it is pretty cool it is very powerful so there were a couple of sessions yesterday and today about that so I would strongly suggest what the video of those no and there was one hands-on lab I couldn't attend it but if you can get a hold of that material that will be good he'll tell you I think we have to get out alright thank you guys thank you for coming