 Good day everyone So I've been lucky to use open stack and operate and maintain for a while and Yeah for a big service provider in Europe and and some enterprises actually a mix of environments Open stock and others and I'm gonna share a track on on operations and Kind of details about a tool that I developed for for the last year Yeah, if you have any question, I'll leave a few minutes at the end hey So I think it's not that easy to operate yet Maintaining and troubleshooting is still Some administration overhead and you you need some skill personnel To operate and I think the result is I mean the cause is mostly focusing on fulfillment and Provisioning and maybe a bit less on on on operations This this is the statement I took from Riranti's presentation yesterday Great guy explained run in five thousand nodes and he said that most of the Failing that happened was because of issues in networking So I'm using networking charter here to explain, but I think it fits for for others Trying to back up my statement with some some details, so that's the survey about it spending a lot of time troubleshooting networking and The lack of operational tooling coupled with with OpenStack today, so I think the problem lies on obstruction I Think by design the the product was developed with obstruction in mind. So the end user Or the end developer focus so we have controllers Talking to agents those agents are vendor specific The agent are actually doing More hooks into something we call walkers. So this is an example of OVS There is a neutron OVS agent He's doing the walk with with the net linux namespaces with the bridges with OVS itself Nova is using Libert. We all know that so that is the the design Obstruction here's an example. So the current API have an instance That instance is connected to a network There is a port attaching for example the instance to the network and There is a router and guess what when you look at the network the status is active So this is a nice view for the end user But I think operations when they face Issues on the network They need more details if you agree So let's say you have a VM Connected to a network and that VM cannot communicate So this is a stuff that that we faced in reality For several reasons now Good people that with knowledge about how things work Still needs to say, okay, what kind of distribution are we talking about so this is an assumption here The example that I'm going to take is from that distribution with that mechanism driver and that type driver That's the variance that We have over here. I'll talk about that later and also assuming you have a only one region So let's say you queried the Nova API and you know the map in between instance and a host That's several queries And let's say, you know the instance name mapped to the UU ID All right, so that's prerequisite Here's a look into the list of availability zones So let's say you know that the instance run on node 6 and you know that It is in the Nova zone You also know that the host is up. You know why there is a smiley here Try to parse that And then you look at hypervisors, right and you see that the hypervisor is up. So you say I'm I'm good there And also Nova is running over there Then I'm asking neutron Show me the list of networks and you have a UU ID for the network that you see from the API Okay Now Very recently the network also belonged to that availability zone. That's really important and that's a great change That happened in neutron recently You also have a way to say which agents are we talking about? So on node 6 we're talking about open v-switch agent So I took this assumption, but now I know Yeah, and this is available since Liberty Then you Need to know which type of overlay tunneling or type driver is used So here's a nice command that neutron gives you back. We're talking about this overlay So now I know for sure and also I know that this network and this instance have some services running So it's the dhcp service and layer 3 service Helping this network now where those dhcp service and Router service is running Note that I'm not saying open stack too much today This this stuff I'll show you why I'm saying that later So here is how I know where those services are running. So node one All right runs in this case the namespace for dhcp Meradada and layer 3 agent, which is the router and again, they are alive See the smiley's moving on I'm saying there are more details in Maria db than that are not exposed To the native open stack API Here is an example a short list. We see Cisco details if it's Cisco arista details We see bgp details if it's used We see DVR details HA information Mapping of ml2 Vxlans and gre or jenevo, whatever neighboring details Vendors of neutron are just pushing it into Maria db because they have no other means to expose it That's a fact, right? And there is no well Depending on the vendor the vendor might give you an API To get those details that specific vendor not open stack Now if you ask me this question Do you really need that my experience? Say yes It depends. So sometimes it's so simple that maybe you can get away without it What many times you do So you're sshing To node 6 where the instance is running And why you do that because from my experience? Database is not enough Not enough information Mostly real-time information So sshing to node 6 and asking libvert The list of instances by the way that name That you see over here Is the libvert name. It's not the nova name So let's say you know how to map it. That's what I said before And let's say That you look into the xml of that instance And you know that nova created it And the nova name is vm 200. That's the assumption that we said we're talking about this vm All right So looking further into that xml there is a section about the virtual nick So the virtual nick of that vm is virtio type I'm going to skip the details of virtio in I only have 40 minutes to Talk about this stuff And we have a mech address assigned. That's cool And I know we're talking about an interface name tap something Because that's the mechanism driver naming convention And I'll show you others Later And there is a bridge a linux bridge with that name Connected All right If you do if config on a node A hypervisor that is part of open stack you will find you'll find tons of those taps. Just tap there tap there Mapping the tap to the instance is inside libvert xml files All right Then You look at the bridge at the linux bridge Here's a command for the linux our mechanism and here's the The bridge name that you found in the xml and here's Two interfaces attached to this bridge. This interface Is the tap interface of the instance And this is the name for the bridge port Attached to that instance All right, so just like in a legacy Switching environment you have the the pc and the port. That's the tap and then you have the port on the switch. That's the Qvb in this case the bridge interface Now you ask yourself Okay, that's the port That open stack is talking about The port of that instance awesome. I found what it means in in operation in reality All right, and that's the port on the bridge. So actually the port consists of Two sides down and you ask yourself the bridge is up Is it up? Is it running? Leave that thought for now because i'm not going to talk about how you monitor this object Just about the fact that it's there and running right for now Then because you learned that we're talking about the ovs mechanism driver I'm learning about that and I know that there is some ovs commands that I that I can use Now i'm asking the ovs system show me The data path Inside ovs And here I see some interesting stuff. I see qvo See it ends with ff2c Just like over here. So I got it the bridge port Is connected to the ovs port Number six Port number six on ovs And here's some details about pockets going in What comes next? So that's another obstruction From the port that the ovs port number six is there Now Looking deeper into the controller of ovs This is a big list You see something called integration bridge br int. That's a common name by the way And you see that qvo port that I was just talking about Getting something important called called a tag So a tag is the way the ovs system Isolate one a network one customer from the other Inside the ovs system instances attached to same tag Can communicate with each other they are sitting on the same network different tags Different networks All right, but now I know it's tag one And I see some overlay tunneling going on From this node number six to some other places I'm asking myself where are those endpoints Going So there is a bridge running the tunneling there is a bridge um For vxlan there is a bridge called integration bridge for the instances themselves And that's another representation of what open stacks see as a port Looking at maria db you see a list that is important Which vxlan endpoint attached to which host To which node All right, so I know This ip is assigned to node number one this ip is assigned to node number six for the vxlan overlay All right I'm assuming there is no routing in between so you see the ip on the same network not several hops away Sometimes there are several hops away and you need to find out How those are routing to each other if you have a region That's why I said region one if you have other regions like one in new york One in san jose guess what there are more details here Now i'm looking into Who on the host is holding that vxlan endpoint? So I learned it's 192 168.2 that too But which system or which interface on the host holds that ip address For mirantis liberty case, it's a bridge called br mesh For canonical, it's a bridge called juju zero now That bridge sitting between my overlay tunneling and physical network How do I know that? another command Asking about the bridge For the overlay tunnel Telling me it's connected to some physical nick ens 160.103 What's that anybody knows all right This is a vlan connecting just the vxlan endpoint To the physical network repeating that Connecting just the endpoint of the vxlan to the physical network That's the implementation all right How do I know that? Very easy look here Vlan raw device en s 160 and that's the id now More details we left node six from that instance Now i'm i'm thinking maybe this instance is not uh getting an ip address For example Now where's the dhcp server the dhcp server is running on node one That's what I Saw on the on the db right so Now in the database, I know Looking closely In neutron dhcp binding table Not exposed in the open stack api Which network Have which dhcp uuid The dhcp uuid in Node one Is this Because if you go to node one the namespaces and you do ip net ns enter You'll find a bunch of them One pair network If the network has dhcp doesn't have to So i'm looking at one of them And i'm executing Well, i'm executing a command inside the namespace The correct namespace just doing if config. Guess what the dhcp server has a tap interface All right And that tap interface is not connected to a linux bridge Why because this is mirantis 8.0 That's why the dhcp Server is connected directly to ovs and not through a bridge And here's the proof. All right, that's the tap interface On port five on ovs that we saw on the Namespace the dhcp namespace Here is interfaces qr qg anyone here has a knowledge about what is qr qg Router correct So There is a router to get out of that network So anyone on that network that wants to get out to the other networks or to the internet or whatever Goes to that router that router has two interfaces. One is called qr is connected to the network internally One is qg. It is connected to the internet gateway. I'm not leaving that space yet Again looking at the bridge inside ovs. Here's the qg qr tap interfaces looking at the tag Awesome tag one was assigned Did I find places where the wrong tag was assigned by ovs? Yes And guess what you query the open stack api and it's going to say network is active because of obstruction All right Here's a list of the vx LANs tunneling in node number one I moved a bit quickly here And then again There is this br mesh interface connected to the correct vlan On that node number one To have that communication in place I'm not going into management networks Storage networks used by signer Or self or whatever that are used for Hosts communicating with their disks Okay, imagine the networking going on in Neutron also provides hosts To connect to their storage disks So do you think it's important? So here's What I call virtual services. So dhcp Router firewall as a service vpn as a service you name it as a service all the vnf stuff They are all Different types of they are not really instance So let's call them virtual service And the virtual service vnix Also have ports You see that on the router ports And here's the query for the router namespace And here's the qg and qr stuff that I refer to See here's the ip assigned to this router Guess what? It should be the gateway ip address of the instance If everything in Neutron was programmed correctly And here's the integration bridge that I showed you before with the tags and everything Okay So if you ask me what If we're talking about different distribution Like a red dot or an apex or whatever What if we're talking about different mechanism driver different type driver? Guess what the discovery Of how things are connected Gonna be different The logic of how a human being administrator will find out what's going on that that logic Will change So here's an example of a program dpdk fdio A virtual switch that is not ovs it's a vpp it's Running in user space In that case inside the instance xml file you will find an interface called vhost user Not a tap But a vhost user something With a mac address with a type driver virtio Maybe a couple of them depends what this device is And here's the command to map the virtual nick on that particular virtual switch to the instance Inside the the interface is named virtual internet something something Here's a command that shows you The bridge it's not a linux bridge internally it is called bridge domain 2 And here's the command to map To show the interface and the pockets coming in and out from the instance Here's the bridge And you see learning and some details about if that bridge is up in this mechanism driver And here's the mapping to some physical interfaces Here's 40 gig interfaces or whatever And in this particular mechanism driver You know where it is going out Of course, I skipped all the tunneling stuff and and villain stuff and all that just give the point here so Till now we were just talking about one vm Let's say your environment has more than that all right And let's say your environment has a feature called ha What happens there? And let's say your environment is walking with distributed virtual routing not the the the standalone router in neutron Guess what discovery time multiplies by number of vm multiplies by number of Well ha multiplied by two Because there is a couple of them a couple of dhcp servers backing up each other One is active. The other is standby. You need to find which one is active. You need to go there and find All right, so this is what I call discovery and discovery is important I think because it's the baseline for troubleshooting And post discovery comes actually fixing the problem And I'm coming back to why All this stuff Is affecting The time spent for troubleshooting and maintenance so yeah That's Beaching and now Do we have a cure? so I've been spending the last year Being part of a team developing something that we called calypso And in general, uh, well, of course, it's going to be open sourced soon in a few months Um, the point here is networking operations api So an api focused on Cloud operations not the cloud user And this api should expose Everything that I showed here and many many other details Per vendor in a model way Not bound At least start to If you agree just start looking at that All right Um, here is some projects that we might be attaching to We wanted to move faster So we spend some time alone But those projects are You can can totally be Integrated Now Inside this the the system you have discovery logic pair Pair environment, let's call it distribution mechanism driver type driver. That's an environment. So pair environment discovery logic and Visualization of that And then monitoring of each object in the system. That's a baseline for doing some more sophisticated analysis And I'm saying point is visibility Equals predictability equals Overall stability So for maintenance and troubleshooting Uh, allowing inventory discovery reporting visualizing that and monitoring each and every object all inside the same graph Here's the object model that we have so we added this to What we think operation api needs to have it needs to have some kind of a virtual service like dhcp and others virtual nicks Virtual connectors because it's general either a linux bridge or other type of bridge vmware might call it port groups whatever Virtual edge the device that actually connect the physical to the virtual Here's examples like mido net ovs others physical nicks Imagine physical nick is not in open stack api Overlay tunneling endpoint because it can be a vtap like a vxlana endpoint But there is a gre endpoint that the geneva endpoint. So in general, let's call it overlay tunneling endpoint And of course there is the virtual topology that is the fact what's going on And there is the kind of the policy topology, which upper layer application is actually seen all right Here is it in the in the open stack model. So regions zones host and everything You see where where those seats v services in particular plugin virtual nicks One-to-one relationship with the network port That is part of a project the host is running all those objects inside I think it's very adaptive. We found that Many many vendors actually we tried it against 12 Have the same concepts. So it maps one-to-one. There is always a bridge Call it something and it does what it needs to be done because we didn't invent the wheel here So calypso discovery has pair environments scanning logic With hooks into open stack api open stack database and command line. We don't want to do that Priority one we want api priority two we go to the database priority three We go to the command line if we get the data that we need from the first one. We're fine I think it will take time for things to move Into api right Then we have links analysis analyzing which Object is connected to which object based on the discovery And then we have something for the ui called clicks And it is focal point equal links. So which focal point are you interested in equals What would you see on the screen Then this environment is dynamic one big environment that I've worked on in our in a webx operation based on mirantis Because of cli parsing and database mostly cli parsing and because the data was missing today Initial scanning took us like 17 minutes We couldn't help it It's all about ssh parsing and stuff We want to get it from api and dream that to seconds, of course We need that operations api, but let's call it first install first discovery Then Changes, so how do I take changes we have listeners pair environment Believe it or not the messages in different environment varies So we have listener to the robbit mq message bus of open stack getting those Changes going on and based on those changes updating the inventory in real time or maybe doing an object scan So not scan everything Again, but just a particular object to find what changed like maybe just the instance name or things like that Then we integrated our monitoring module with sensu So we chose sensu as the framework and we have sensu client on each and every host We can configure that for you when I say configure Monitoring needs a lot of configuration. It's not just putting stuff there Like for example, you want to do a ping from a certain source ip address to certain destination ip address to see if it works The detail of which ip address to which ip address Is from the inventory So we take stuff from the inventory pushing it into sensu checks There's a lot of details there, all right We wrote 10 different Sensu checks for our 10 different virtual objects that we discover in the system Sensu is giving this back to us. We wrote 10 different sensu handlers The sensu handlers in calypso are pushing the result in real time to the calypso inventory So now you have not just details in topology and and and and And like dependency model but also stat Status, right? It's up. It's down Walking or not Okay We have an interface port into time series database giving you information about okay, how was my topology and held Last thursday And compare to today Finally visualization was a big part. We investigated. We found a really cool algorithm To illustrate the model of clicks and links that we have in the system and do it Intuitive and in real time Here all our modules are container based So last time I checked installing calypso is 10 minutes Pulling from docker hub and running Each and every container discovery monitoring our database based on mongo Yeah, we have an api. That's the operation api offer that we can start with The ui and the bus Once we have an agent for operations api you can simply take our Image over here and use it right now as an agent. We're going to save all the ssh and stuff Here is what we tested against all right a big list of these distributions and mechanism driver Some in qa phase if you don't see the variance is just mean we didn't test it This proved us that our model fits and uh, we can gather the data multi environment you might look at Other Systems that are not open stack. We also have a mapping for that to container based bare metal and and even VMware I want to move into a video to show you how the system looked like So you you're going to add a new environment. This is where you're going to tell the system Which distribution? This is the list that we guarantee supported which type driver And which mechanism driver you're going to tell us if you want us to to deploy monitoring That's the only right that we will do because everything else is a read We can create just the files and you take the files and you deploy them on sensu Here's the scanning going behind the scenes the scanning actually Fill the inventory and you see the details about the scanning and what's going on Here's the link types that you're interested about You want to see this object connected to that object Some people just want to see the horizon like Instance connected to a network awesome, but then you want to zoom in so you have a click saying, okay If this focal point is clicked on the on the ui Here's the list of links that i'm going to show you on the screen We think that this is really intuitive and help you kind of Make the ui fit your needs or network operating system System by the way it was built for big screens Here's the messages from the ruby temq and we have a link on the message Taking you exactly to the point in the graph where this stuff happened Here's one message We have the the body for you that you can take a look Here's a strato scale environment. It's not really open stack. It's um, it's a High convergence But it fits our model and it has the open stack api. So we use that To demonstrate and here's how fast We guarantee this graph generation saves 90 95 percent at least of administration overhead To find that out Guarantee So this clique took us to this bridge for example and here I'm sorry It was built for a big screen and that algorithm has a big line between objects I can make it smaller With our js algorithm Here's open v-switch running on that on that switch We're moving to a mirantis Version nine here running like 11 instances. Here's the spacex network Here's how the network looks like so you can zoom in and out the network is a big So the network is active because it's blue All right, not because open stack api is saying that it's it's it's active Here we say what the network is about all the tap interfaces all the bridges all the interconnection between those guys All the Ovs overlay tunneling All right Here here is the status. I'll show you next how the status change From error to an upstate. Here's the br mesh that we saw before Uh connected to The port and the physical nick all the way down to the network So the focal point of this particular graph is the network. This is why you see all those branches Going on in the environment Here's the other side like one vm on that side one vm on the other side Again, you don't have to zoom in and out and just browse around It just depends on the way we configure the algorithm I just did that a couple of days ago Um, so we find complex interconnection. You have a search engine in the in our Uh system just against our inventory. So you can in real time ask for I don't remember. What's the name of this router? Here's changes in real time. See the router name is router 04 now something in open stack change that name to something else see It's change in real time. No need to refresh the browser Um Here's a big environment. We call it thunder cloud. It's an internal project That runs a lot of um nodes and and instances This is canonical based by the way So in that environment We have that search demo. So in this example, I'm searching for Something that starts with this name filtering and then choosing that particular focal point Which is an instance for example And guess what you will get to the graph That shows you the pieces connected to that instance. Here's monitoring Look at that. It's error and suddenly it's okay It's a ping that we do over over on the overlay. We have also a trace route So you get information about like for example a round trip delays and things like that That's part of the sense to checks that rewrote in the system You've seen that before here is an example of analysis So let me stop here for really just a second here So just to prove a point We ran some statistic collection using collect d just like the other guy. I just explained before Um and and log stash as well sending packets from a certain mechanism driver to a big data environment at this environment Analyzed the packets for something that we call throughput pair session So one instance talking to another instance, of course can have multiple sessions those samples over here is um Kind of an example of traffic for like a backend database. So this is what was a proof of concept that we did and The result of this traffic like average throughput over time Come on, please continue anyway translates to Ability on the ui to choose a session from a source to a destination And here's the graph that we have to show The total throughput Over time from this source to this destination here. We're using MAC address, but we have the context of Mapping that to instance name, of course That's it. Thank you very much any questions I can take some questions Yeah, awesome Just a quick question on that last graph. Yeah, we're using Uh, we're using flow data to get the Yes, it's a script that that we wrote to get this flow data. Yes Okay, so it's like like net flow ip fix as well open flow Okay, thank you any more questions Thank you very much everyone Do you do you map bare metals? Oh So bare metal is pretty easy. Yes, absolutely. And there's no value there I mean, we have the networking management systems that can do that It's just the point between the end of the virtual side the physical nick and then Other systems handling like leaf and spine things giving us this data. Yes. No problem