 All right, I guess that's our cue. So thanks everyone for coming. I'm Adam Johnson from Mitochora, and wanted to invite a couple guys from Cerner. They'll give their background in a little bit to talk about their views when approaching network virtualization overlays. So this should be an interesting event. Oh, how do I? Okay, there we go. So we're gonna go over some quick introductions. Then these guys are gonna go through their story around implementing NVOs for their OpenStack Cloud. Some real practical experience. So I think it'll be really good. So just a little bit about myself. I'm from a company called Mitochora, and we created software called MitoNet, which we open sourced back at the Opusack Paris Summit. And MitoNet is a network virtualization overlay. So it's distributed software networking, plugs into Neutron, it's Apache license, we have an enterprise model, and Cerner is one of our users. And I wanted to invite them on the stage to talk about their journey with overlays because when implementing overlays, the cloud team is pretty interesting because usually there's someone or a team from the networking side coming in. You got Linux people coming in, and they both have to work together to actually deploy something like MitoNet. These will apply to pretty much any NVO that you go with because in the end, MitoNet and all the other solutions are just Linux software, but requires networking expertise because you do have to connect it to the outside world. And there are protocols and things you have to deal with. So that's what this is all about. I wanted to have real users talk about their experiences from the two different viewpoints, what they learned in going through that. And I think it's very applicable because we see something very similar with a lot of the users out there. So hopefully this is valuable to you guys. So I'm gonna hand it off to Ryan and Russ, and they'll go through their journey. Thanks, Adam. So my name is Ryan Baker. I'm a senior technology architect at Cerner Corporation. Today I'm gonna be representing the Linux slash system engineer perspective. So when I started at Cerner about eight years ago, I came in with a lot of experience around managing operating systems. And that kind of slowly started to evolve into managing and supporting solutions that monitored and managed our network and storage infrastructure. So I got a little bit of exposure outside of just the compute technology silo. I really enjoyed doing that. So I had an opportunity then to become an architect on our infrastructure automation team, which really focused on providing self-service automation for our traditional infrastructure that we were running our enterprise workloads on. And over a few years that kind of naturally progressed from traditional enterprise workloads to cloud workloads. And that's how I got to where I am today as an OpenStack architect. And I'm Russ Starr. So I kind of represent the network background. I've done a lot of data center networking, routing, switching, which is probably no surprise since I'm BGP here. And a lot of proxy and load balancing background. So I'm taking that skill set and that knowledge and applying it to our OpenStack cloud and kind of refreshing all those Linux and sysadmin skills. So this is kind of our journey. So a little bit about who Cerner is. So Cerner is a healthcare IT company that's based out of the Kansas City area. And we're best known for our electronic medical record solutions that are running over 18,000 client facilities worldwide, ranging in size from small local community clinics all the way up to some of the largest hospital systems in the world. In 2015, we did $4.4 billion in revenue and we had an R&D spend of $650 million. And as an associate at Cerner, it's that spend in R&D that I'm very appreciative of because that's what has allowed associates like Russ and myself to really explore and deploy OpenStack to really contribute to Cerner's mission of systemically improving healthcare delivery and improving the health of our communities. So a little bit about Cerner's OpenStack journey. So we really started in 2013, 2014, and this is what I always kind of call is our pet project phase. So there were several associates that were deploying it in our lab and kind of kicking the tires or on their laptops or whatever infrastructure they had at personal infrastructure running at home. And this is also when we sent our first associates to the summit in Portland and we've actually been able to go to every summit in North America since then. So then as we kind of moved forward in late 2014, we kicked off our first official proof of concept with OpenStack and this proof of concept was really around the functionality of OpenStack. So we weren't really focused on things like providing high availability, scalability, those things. We were really focused on getting OpenStack into the hands of our developers to see if it would meet the needs that they were giving us around providing a more agile and a more transparent infrastructure for them to develop their next generation platforms on. As they worked on it, they came back to us and they basically said, absolutely yes, this is what we need. This is what we need you guys to go do. So go do whatever it is you do in infrastructure to get us servers. So in early 2015, we started our production planning and this is when we really started to become concerned with things like providing high availability and scalability. And we had understood from the community at that point in time that Neutron was one of the things that we needed to be very concerned about scaling to the size that we were gonna be scaling to. So that's when we teamed up with Adam and the team at Metocura to implement Metocura as our network overlay and really address those scale points that we were concerned about. Mid 2015 is when we went live for the first time. So we've been running it in production ever since then. So now it's kinda when I wanna start to dive into our actually split brain and the first thing that we kinda wanted to illustrate was how we work together as a team and kinda what we see as how we split up the responsibilities as well as what our involvement is in the project and kinda how we split that. So when we first started looking at it, we broke it into three phases and we said, well really the first phase was really around planning our deployment. So as we kinda started this, it was very critical that we had both our system engineers and our network engineers involved and as Adam said earlier, working together we were in the same room or the same space and constantly working together to collaborate to come up with the way that we were gonna deploy. And as a system engineer, our responsibilities were really architecting the OpenStack platform, but it didn't just stop there. As a system engineer, we were also responsible for the architecture of the overlay because with something like MitoNet or any of the other network overlay technologies, it's really just software and packages that you're installing and configuring. So that just naturally fits into a system engineer role. Not only were we concerned about the architecture, but we also are trying to figure out how we were gonna deploy OpenStack. What type of configuration management tools we were gonna use. And that didn't only include the OpenStack projects but also the network overlay because again it's just running on a Linux platform so you get all these cool tools that you can use like Puppet, Chef, Ansible, those type of things. So it was critical that we understood how we were gonna manage the network with those configuration tools as well. So on the deployment planning phase, as a network engineer, we have to do a lot of planning for the underlay and one of the things that was really hard to get used to was keeping the underlay simple and not over-engineering it. So with MitoNet, all that complexity is higher in the stack and anything that has to be done from a routing, bridging, load balancing, that's done by MitoNet. So planning for that underlay, it was pretty easy. And then one of the harder tasks is kind of planning for the external network connectivity and how that talks to the home networks or the existing networks that have to consume the services that are coming out of OpenStack. So the next phase that we kind of wanted to dive in and illustrate was really around the deployment. And as we talked through this and we kind of discussed how we wanted to share how our involvements kind of changed throughout these phases, the one thing that we realized was the deployment phase actually had really two components to it. So there was the initial part of the deployment and then we kind of went into the back part. So the initial part of the deployment was really heavy on the system engineer involvement. So as a system engineer, our responsibilities were obviously deploying OpenStack and all the projects that we talked about. But as I mentioned earlier in the planning phase, we also took it a step further and our system engineers were also responsible for installing and configuring the overlay. The overlay, especially like with MitoNet, has databases and API endpoints and all of those type of things that you have to install and configure. And so naturally that just fell into the system engineer rule. Our network engineers weren't sitting by idle during this time though. They were continuing to flush out how we were gonna uplink our OpenStack infrastructure to our existing external network. But then also, and this is the really critical part, they were sitting there and they were observing the deployment with us. So we were in the same room again. He was watching over shoulders or we would be projecting on a screen and we were ensuring that the network engineers were engaged with us so that they would understand the environment and how it was getting deployed. Because after all, their network infrastructure is gonna be running on these Linux hosts. So, kind of the final steps of deployment, it kind of ramps up toward the network engineer taking on more of the responsibility. So we're gonna be configuring the overlay to support all the tenant networks and then integrating that routing. So it's actually doing the work, establishing the BGP sessions and making sure everything flows through creating some test tenants. And then during that time, the system engineers is pretty much tuning the platform and just getting things finalized and ready to go. So after deployment time, it kind of goes back to 50-50. So this is kind of like our ongoing support. System engineers are helping tenants through Neutron and any advanced overlay troubleshooting. So if there's anything with debugging or tracing things at a system level, they might come in to help. While the network engineer is really responsible for that overlay as a platform and the version of it, and kind of just maintaining it, watching it, watching the logs, any type of housekeeping that needs to be done if somebody trashed the system by creating a bunch of load balancers that aren't being used, we might go and look at that and check on it. And then any advanced Neutron tenant requests. So something beyond the basics of our router important network like LBAS or things of that nature. And then here is our practice during AmitoNet upgrade. So in this example, the network engineer does most of the work. So they're responsible for getting that target version, tracking the release notes, making sure it's where we wanna be, sending the notification and coordinating with any of the business units that need to know if there's any sort of downtime or anything like that. So, and then the system engineer, they will be on standby for help. So if we need a repo updated and we don't have the right software to do the upgrade or something that we're not quite familiar with, they're usually on the call and we're working together. And then they kind of peer review all the work and support. So here's the system engineer perspective. Yeah, so really kind of wanna dive in and give a little bit of my perspective and really mostly just kind of like the chips and the tricks and things that I found as we worked through not only to appointment but also running an overlay in production. And when I started thinking about it, it really boiled down to three things, everything I could categorize into three things. And the first thing was really becoming knowledgeable as a network engineer. And I don't have to be a network engineer. I don't have to understand everything that Rust does because I can always escalate things for us. But it is important that I understand key concepts in the network space. And so anytime we have like a new system engineer that starts and they always tell me, it's like, hi, I don't have that much experience in networks and I'm not that familiar with it. And I always tell them, it's really okay, it's not that complicated, but let's start somewhere small. Let's look at something like CIDR notation. And let's get you start speaking in CIDR notation versus saying my IP range is from zero to 255, say it's a slash 24. That may seem like something very small and very trivial, but to a network engineer, when you speak in terms like that, it automatically lets them know that you have an idea of what you're talking about and they're gonna wanna help you more than just a typical request. So the next part is anybody who has talked to a network engineer for more than 30 minutes knows that every other word out of their mouth is typically an acronym. So you don't have to know every single acronym, but you do have to know a lot of the common ones and especially the ones that you're gonna be using in your overlay environment. So these are some of the ones that I use almost on a daily basis now. And the first one is really VLAN. So you need to know things like how your VLANs are providing isolation in your underlight and then you need to know how your VXLAN is encapsulating traffic across the VLAN. And then you need to know things like how does that VXLAN encapsulation impact your MTU sizes. And then you need to know things like how you're gonna route traffic outside of your OpenStack network into your external network with routing protocols like BGP. You start doing that and all of a sudden you start sounding like a network engineer really fast. So the next one, and I struggle with this a lot, it's something that I still struggle every day with is understanding that it's okay to be lost in these network meetings. So as a system engineer, every one of us has probably woke up in the morning, looked at our cell phone before we had to work and realized that you have a meeting on your calendar that has something to do with the network. You get really excited, you drink five cups of coffee, you drive into the office in your car, you play your pump-up music, and as you're walking into the office you think to yourself, today's the day I'm gonna go to a network meeting, I'm gonna understand it, I'm gonna pay attention through the entire thing. Five minutes into that meeting there's been 30 acronyms that have been said already. They're talking about network technologies that are so deep that you have no idea what they're talking about. And your eyes kinda start to glaze over a little bit and you kinda start to think about Keystone configurations that you need to make or the latest vulnerability that you need to get a patch applied for. And this is where I challenge all the system engineers is to stop, refocus yourself back into that meeting because you have to remember at this point that the critical concepts that they're talking about in this meeting are soon gonna be running on your Linux platform. So it's something that you need to at least have a high level understanding of what they're talking about so that in one month, one year, however long it's gonna take to get that onto your platform you have an idea of what's running there so that you know how to operate it. Now as a system engineer, it shouldn't be a surprise to anybody that I'd give the advice to stand up OpenStack platform but I challenge everybody to not stop there, install the overlay, configure it yourself but then don't stop there either. So Meadonet has a really great Quickstart guide on their site where you can download and you can do a Quickstart that'll stand up OpenStack, it'll configure Meadonet, it'll configure the overlay but then it also starts to walk you through how do you configure it for external routing? So they start off with saying how do you do static routing from that all in one host into the gateway so that you can kinda start to see how the routing is working and then from there, once you get that working then you can take it to the next layer and you can work with some of your networking guys and see if they have infrastructure in the lab that you can try to do the uplink yourself and do all the routing configuration on and if they don't have the infrastructure or they won't let you on the networks because for some reason they don't ever let you on the network devices then sit with them, don't just say go do this configuration, sit with them, look over their shoulder, ask questions, make sure you're engaged with the entire process end to end so you have a good understanding of how the whole thing works. So moving on to the next key thing that I really found that was very critical for our success in deploying the overlay and managing it is all around communication. So one of the things that we tried to do at Cerner is really remove the phrase it's a network problem from our vocabulary because it's not really a network problem it's more our problem or it's more an open stack and by doing that we really realized that that was really kind of reducing some of the barriers or the silos or the walls in between the traditional network and compute silos. So we started to realize that as the network engineers or as the system engineers were seeing problems they weren't just saying oh it's a network problem and let the network guys figuring out we were starting to jump in ourselves and starting to debug the issue and starting to try to figure out what the issue was and if I couldn't figure it out that was okay I would escalate over to Russ and here's the critical part that we started doing we didn't just escalate it over to Russ throw it over the wall and say hey let the people who reported the problem know when it's completed we then went and we sat and we worked with Russ until the problem was resolved so that I could learn what the problem was I could understand how he troubleshot the issue and how he knew where to go to troubleshoot the issue and just knowing that and going through that process makes you a better system engineer trying to run the overlay because you have a much better understanding and appreciation of how the whole system works. The next advice around communication is really think about things like minor changes that you're making so before I was doing OpenStack I never had to worry about installing a package or doing an operating system upgrade and it causing a big network issue but now that you have this critical network infrastructure running on your Linux platforms it's a very real possibility that something simple like that could cause an issue on the network and somebody like Russ is not gonna be very happy with you. So how do we tackle this? So we do daily standups every day we all get together we have representation from our system engineers our network engineers or storage engineers our security groups and then our operations groups as well and we talk about what we're doing what we did yesterday what we're doing today and any roadblocks that we have so it really breaks down those barriers for communications that exist typically between your traditional network silos. And the final piece of advice is be aware of the words and the phrases that you're using because a lot of words and phrases have very different meanings between a Linux engineer and a networking engineer and I used to always talk with Russ and we always used to talk about a certain scenario in which a user could launch an instance and then create two ports to two isolated networks and I always used to describe this as they were bridging the networks and Russ used to always get extremely mad at me and say you're not, they're not bridging the networks that's not what that means. And so one night I was like, oh, I got it I'm gonna go in tomorrow and I'm gonna refer to it as being multi-homed and so I came in I was so excited I said it to Russ and Russ just shook his head and he looked at me and he said, well you can't really say that either because that has a completely different context than network engineers typically and they're gonna start thinking in BGP land. So it's very critical that when you find those you realize what's going on and you communicate and at least establish a common communication platform. Still to this day just for the record whenever I'm talking about it I just say that I'm gonna boot an instance I'm gonna create two ports I'm gonna put it on isolated networks. So if anybody can help me figure out how to shorten that up I would very much appreciate it. All right, so now kind of moving into as a system engineer it's our responsibility to help the network engineers. So we see this a lot and you really need to be aware of this when you have a network engineer who's just starting with OpenStack and they're coming from a traditional networking environment. And so when you start to talk about things like configuration management the first thing that may pop into their mind because they're from a traditional enterprise networking environment is things like connecting their serial console cable to the switch and running some commands or SSHing into the switch and running this commands but you need to help them realize that their overlay, whoops, their overlay is now running on the Linux platform and there's a lot of really cool tools that can help the network engineers now manage configuration on those devices. Things like puppet, chef, ansible, salt all of these really cool tools are now available for them to use but they're gonna need some help with it. One of the things that we do is there's a quick start puppet guide. So you can download this it's basically a vagrant image you download it and it has a guide that goes along with it that basically walks the users through a very small minor configuration all the way up to deploying a solution and making some configuration changes on it. And so we try to have a lot of the people that are starting on our team go through that exercise and I've never had either a network engineer or a system engineer do that full guide come back to me and say, you know, I just, I don't see the value in doing this I think it's creating a lot of overhead. They always come back and they're blown away and they're very excited because now they have the opportunity to do this. Now hand it over to Russ. Okay, so this is a network engineer perspective. So, come on clicker. So these are tips for network engineers that are working with overlays and working in an open stack environment. And the first one is keeping the underlay simple and low cost. I kind of said that earlier. Just go with the low cost, you know, try to base switches and run the overlay over that. So I know in a lot of places the demarcation point is different. We own our underlay network as an open stack team. So we pick the hardware. I know in a lot of organizations that's just whatever is in house you use and then the overlay rides on that. So, but if you have the choice keep it simple and low cost. And also like obviously you gotta know Linux you gotta know the bash environment really well. And then there's a few things you should probably level up on in Linux. So you wanna know your networking commands really well. You wanna know these without even thinking about it. So these would be like reflexes. And I'm sure most people here know that unless you're from a network background like I was and you have to brush up on them. But those are pretty self-explanatory. Netcat is kind of our in-house, well not in-house, but we love to have fun with Netcat and use it for testing and breaking things. And then knowing how to chain these utilities together like grep and set and awk and tr and dealing with files with tar balls and extracting them and creating them. And then knowing your way around GitHub and the Git commands associated with that. And last but not least know the file system. That's usually neglected if you're new to Linux and I'm guilty of this. If I wanna run, say a packet capture TCP dump, write it to a file or turn my network overlay agent into debug mode and forget to turn it off. You fill up a disk and then you break some other services. So be cognizant of where you're putting that stuff and clean up after yourself. And get comfortable with the JSON format. That's not a natural thing for network engineers. It looks ugly at first and then you, after a while you start to learn to embrace it and that comes with more benefit after you can string it together with a Python script and make sense of it, parse it and be more efficient. And like Ryan said, learn to use the configuration management tools, not an Excel config generator that you paste into a device and then hope that it takes everything you entered. Like, we gotta move past that. And I like to do this. I just make up three to five letter acronyms. They usually buy into it. And so here's how network engineers can help system engineers. So routing protocol basics. These are basic things for network engineers but system engineers usually just need a little update on. So just knowing the difference between an IGP and an EGP and why we might use one for our underlay and one for our overlay. I'm not gonna discuss that in detail but knowing the concept is important. And route selection. So I have a variety of different routes in my routing table. Which ones do I actually use for forwarding? And how does ECMP work? So equal cost multi-path. How do I load balance across all my links? And this is a good one. We ran into this when setting up EdoNet. We were looking at the quagga status for BGP peering and it said active. And I think a lot of the system engineers were looking over my shoulder thinking, well what's wrong? It says it's active, that's good, right? Well established is what we're really looking for. So that's kind of a network engineer thing. So, stateful devices with connection timers. They exist in our environment and probably everybody else's. But just understanding how they work and what their settings are and what they might be preventing from working normally. And so on our practices, here's like some organizational tips that I guess we formulated. So what we did is we started our OpenStack team with a small number of good people. So we just got, like myself as the network engineer and Ryan and a few other people that were just subject matter experts in their respective domains. We came together and formed an OpenStack dedicated team. So we have Linux, virtualization, networking, storage and we even have software engineers that are dedicated to work on projects that we need help on. And we operate cohesively. So Ryan kind of alluded to this, operating agile practices and having the daily stand-ups. We do those with everybody. Even, so like security for example, we don't have a dedicated security engineer on our team but they're involved, they stand in our daily stand-ups and they're up to date with what we're doing every day, what we're releasing and what that looks like from an iteration standpoint. So this is, we're gonna do some Q and A and the way this is gonna work, we're gonna have just a few, I guess practice questions to get your, get your minds thinking about what to ask and we'll answer them and then you guys can come to the microphone and ask some questions after that. All right, thanks a lot, that was great. So Ryan as a system engineer, what was the hardest thing for you to learn or accept about networking? Yeah, so I think really the hardest thing for me to really learn was how to filter all the information that's out there. So it's very difficult just to go out to Google and start to learn about networking concepts because there is so much information and so much of it that may or may not be applicable to your environment. So really, it was very difficult for me to know where to start, right? So that's where I kind of leaned on people like Russ to kind of help give me pointers, point me to the Networking for Dummies book, right, for Cerner. And so once I kind of got that, then I could go figure out things on my own but it took a while to really kind of know how to filter the information. Yeah, that's a good point. I came from a Linux background as well. Now I work at a networking company so I went through a lot of the same things where I didn't know any of the acronyms. I spent a lot of time on Wikipedia. I think that was a good resource for me. And Russ, how about you? What do you think? We've already touched on this but just the idea of configuration management is hard at first but it's something that you really, it can really help you in the long run. And then the other thing is accepting that the network can be simple on the underlay and just not having to worry about an expensive, complicated, high-touch network device. All right, and what were you the most surprised about when you started down your journey with an overlay like MitoNet? So I'd say the overlay itself really isn't the concern because it kind of just works. It's just a concept. So like we spend more time working with our tenants and all the guys that have projects out there trying to help them with Neutron, setting up the environment the way they need to help them, helping them understand the IP space and how it's tied in. And so what I did is I did some analytics on our Slack communication to figure out what, how important is the overlay? So we did some word count on a few key words. And so the word overlay, it only came up three times. And so we found some other words that have a similar word count just to put it into context. For whatever reason, VXLAN, we talked about it a lot more in comparison to beer or barbecue, important topics, especially in Austin, yeah, or Kansas City. And BGP, obviously, I talk about that a lot versus Keystone, it might have been a different Keystone, but you get the idea. So you guys can step up to the mic and ask questions. Yeah, so we've got t-shirts. So on the underlying networks, you mentioned that you guys were doing BGP. How are you scaling? If you are using provider networks, are you able to scale horizontally? So on the underlay, we're actually doing OSPF, and that's just connecting all the systems that have MitoNet agent on them. And so that's just a VXLAN tunnel. So it doesn't care if it takes a layer three hop. So it's just a simple clause layer three fabric. And then the overlay handles the scaling out. So we can grow that out with BGP. So the BGP comes into play at the gateway. So the gateway is a device that provides the physical external connectivity to the cloud. Any reason you chose OSPF or BGP for the underlay? So the underlay is just all OSPF, and then BGP is basically just a quagga instance running on a Linux server that talks to the outside world. Let me rephrase my question. So are you guys doing L3 from the top of the rack to your spines? Yes. So that's where OSPF is, right? Right. So is there a reason why you chose OSPF or IBGP? Just because it's simple, and it's, we don't have the scale requirements. OK, last question. Are you doing IPv6? No, not right now. Not in this environment anyway. I was going to ask you guys about health management or OSS. That's usually a contentious decision between the network and the system guys. I'm just curious how you addressed that. Yeah, so actually, that's a good question. It's something that we actually still fight about. So I mean, it kind of goes back to some of those earlier slides when we talked about, we really have, we try to remove some of the concepts of isolating the data center silos, right? So we kind of both take responsibility for it. And especially in the health monitoring aspect, if one of us sees a problem, it doesn't matter who it is. We try to find a root cause and then try to figure it out. It doesn't matter if it's a system engineer or a network engineer. So it really is transformed from it's a network problem to it's our problem. So is that answer? And just to add to that, as far as tools and things like that, are you looking at the same panes of glass, or do you have your network health monitor and your system health monitor? No, same panes of glass that we leverage. And that really kind of helps to build that unified one engineer concept. I think I'm going to coin that, actually. That's pretty good. Just for reference, I'm a network engineer, but I've been learning or re-learning systems engineering, which was I did in the beginning of my career. Sorry if I offended you. No, no, not at all. Not at all. I just, I find it funny when you talk about thinking it's hard to learn networking. I mean, yes, there's a lot of different languages you have to learn, if you will. But it's not the same as coding from scratch or looking at large-scale systems problems. Yes, it does if I create a loop, I tear down the entire network, but things like that. Do you guys have multiple data centers? What are you doing for inter-data center routing or using EVPN, or are you allowing your tunnels to cross the WAN? So right now, our open-stack clusters, they stay within the same site, but we do have them. We have different clusters in different locations. So we also have a huge hosting network that is our legacy that does electronic medical record hosting. So that's already in place, and we use that for transport right now. And then we have our own kind of multi-rack system that's autonomous from everything else. And then that BGP connection, eBGP, because it is autonomous. That's how it connects to the rest of the network. And the second question I kind of had is, are you guys doing or currently looking towards hybrid, like putting anything into AWS or any other longer-term storage kind of thing? Yeah, I mean, that's something that we're always challenged to do the math on, is does it make more sense financially for this workload? So yeah, that's something we're always looking at. How do you deal with HIPAA when it comes to that? It is a challenge. But there is the GovCloud, and there's some FedRAMP-supported services in AWS too. But I stick to more of the open-stack side myself. Got you. Thank you. Hi. Let's consider that network topology at three instances. For instance, configuring all these switches, which is like a flat network. Then moving towards the VXL and kind of overlay network. And then when we introduce some kind of SDN solution, such as Midonet into an open-second environment, that will introduce more network complexity. So where should a system engineer start bothering about the network complexity? How can we split these two aspects now? OK, this is the issue of network engineer and not of a system engineer. Yeah, so that's a good question. Usually, I try to not delineate the line. So I'll try to take it as far as I possibly can until it doesn't become feasible for me to spend more time. So if I'm researching something or doing something like that, I'll time box it. So I'll say, I'm going to work on this for two days. And then I'm going to kick it over to Russ. And I'm going to leverage Russ's expertise. Really, as far as lines of delineation, we try to maintain the system engineer as far as internal communication. So compute host to compute host, those type of things, is what we really try to focus on the system engineers. And it's our expectation that they understand those components and can troubleshoot them. And then anything that is making its way out of OpenStack Northbound, then we kind of start to kick it over to network engineers. That's their realm of expertise. Thank you. You're welcome. Anybody who asks questions, feel free to come and get a shirt. Or if you didn't ask questions, because I don't want to take these back to Kansas City. We just have mediums. All right, looks like we're done with the questions. Thanks, everyone, for sticking with us till the end of the day.