 Hello everyone, how are you doing excellent excellent? Hope you're enjoying the weather I'm Matt Reese Also joined by my partners in crime here Zach Bullard security extraordinaire Mike mescal mad scientist Mike And Justin Palmer just scripted these are my friends and partners and The open-stack implementation. I'm the lead architect for our open-stack project It's good to be here guys just honored to be here. We're lucky to be here actually We almost didn't make it, but we're here and glad to be here. So it's good to see all the smiling faces up here So we are with the United States Department of Agriculture National Information Technology Center We are located actually primarily in Kansas City. We also have data centers located in Missouri on the St. Louis side we have data centers actually all over the place and So I'm sure you'll if you haven't heard about us. I'm sure you will at some point So I Want to ask you have you read the essay or book by Eric Raymond The Cathedral in the Bazaar Any hands anybody read it? Oh good good. So you know what we'll be talking about here So real quick if you have not He talks about Two very interesting concepts the first one being the Cathedral style So there's basically a single rigid way to do things with this style He talks about writing code one way. He talks about one management style Primarily doing this in a silo, right? So basically Also implying that projects won't be as successful perhaps Also the so Linus came along and he shook things up So he brought on the bizarre style. So the idea being he would outsource his projects Getting more participation Which could also result in a better product? Was able to get more and better releases this way and so on so That's the style that we took in our open stack implementation actually I'm sure most companies today are still looking at the Cathedral style So trying to build in the Cathedral can be difficult and we'll touch on that later So if you haven't read it, I would recommend reading it For all your open-source junkies Definitely a good read. It's a little bit dry unless you're into open-source stuff. So go ahead and check it out So you're probably wondering who the heck are we so we are the enterprise data centers for the USDA Little over seven eight years ago It was mandated that We consolidate down to two data centers to primary data centers like I mentioned before So we're pretty much targeting all federal and state government entities We obviously are heavy into federal data center consolidation efforts. So that's been our primary goal for a long time. So trying to Consolidate the USDA as a whole and migrate them to two primary data centers So what we have been up to historically in slide transition here His historical note actually not sure if he knew this but Abraham Lincoln actually founded The USDA in the 1860s It's a very interesting Fact that I actually didn't realize until I looked it up recently Nitsi however came along a little later 1957 the primary focus for years was around supporting the farm services So I'm not sure if you heard of the farm bill, but that is the biggest application that we had hosted for a very very long time supporting those farmers Multi hundreds and millions of dollar project But over time the data center actually expanded it became a lot more popular So hosting for a lot of other agencies not just USDA and as I have up there Who we support so we support a lot of different people as you can tell Department of Homeland Security Coast Guard to name a few One of the biggest projects to mention actually some White House related activities. So we did Let's move gov. We've done choose my plate some some of the Websites that you've probably heard of Michelle Obama So over time we've developed and we actually were overwhelmed with business so we couldn't Handle all the business that was coming down the pipe, right? So total focus on the slow and steady For Nitsi has been the kind of the primary objective so we've Hosted primarily things on the mainframe. So we're a huge mainframe shop for years And then eventually grew into a traditional hosting Center So we upgraded to VMware platforms and did some really cool things. So Our primary focus has been Paz offerings and says offerings for a long time So we started talking and trying to figure out what is missing here Some of the things that we've been talking about for a while is this I as service. So how do we get there? So shifting customer needs Obviously We noticed that customers needs and wants started transitioning to other things Like I said, we've been a mainframe shop since the 90s and yes, it is still running We needed true I as And not just says not just past but I as we our ultimate goal has been to empower the customer so give them what they want and and Kind of differentiate ourselves. So what we've been for a very long time is a full service gas station What customers really want? things like self-service gas station, right, so They have two options basically at this point Us to help them through everything and then there's also The self-service option. So Paz Versailles. This is something that we talk about all the time Some of the biggest challenges are around budget. So we are not appropriated all our agencies are they got the money But their budgets have been reduced drastically so with Fisma and accreditations and all these things. They've been really looking for us to help Bring that all together. So they've been after us to do Paz mostly, but As we noticed things started shifting We're not able to keep up with what they want and what they want is Basically I'm sorry Amazon but in a government space. So what we do very well is Basically Switch to IS at this point. So we're looking at IS offerings. We're thinking about it But we're not quite there. So there's a long path to production. Everything's pretty manual We're trying to speed things up operationally We're having issues with with just keeping up with what customers need what what they demand. So we're Building new versions of VMware. We're implemented implementing the same things over and over just upgrading But the IT business process doesn't align with what our customers need. So we're basically a One-stop shop or one-size-fits-all model But we need to adjust Towards what the customer needs and wants are so today we would like to share our story of how we got to This open-stack implementation from the conceptual side of things to the production side of things so Zach would you kick us off buddy and Thanks, man. My name is Zach Bullard. I'm a security architect at the USDA's NITSY I found an open stack on the web late one night and brought it up on break with our deputy ACIO and our ACIO and And you know and one of the seminars earlier today that was something that somebody touched on was you know Don't be afraid to approach executive management. Try to you know pitch your idea for doing private cloud. I Had a 10-minute conversation with them. We talked about it and I explained basically how rackspace and Amazon worked and you know We could we could either slap cloud out in front of what we were doing with VMware We could you know put together a real team of guys that got it And and try to do something different So they put together a team that we called atmosphere and we chose that name Because an atmosphere could contain many clouds So We we got a room and we booked it for like 90 days and we called that the cloud cave and We just started research and everything we could get our hands on open stacks offer to find networking Vanity-free hardware. I mean you name it the POC had to be done in 90 days, so we had to build something that worked in 90 days We had no money not a lot of resources and it was it was really quite the challenge. So We scrounged together hardware. We you know, we did what we could and Within 90 days we got something working. So We based it on NYSERA open stack and CEPH and Getting all that to work in combination let alone individually was was quite the challenge We ended up demoing it for executive management and they really liked what they saw so In the meantime, you know, we had architecture to work on Justin had Linux to support as team lead and Mike was transitioning from VMware team lead over to architecture as well And you know, we all had full plates. So it was it was a little bit more of a challenge than Then I could even explain to you. So Wasn't until we got deep into open stack that the hope kind of turned into anger and frustration and You know, that's where angry stat came in at, right? So I'll let Mike explain the angry stack part Hi, I'm Mike mescom network architect for USDA and I get to be the bad guy part of the presentation, right? So get into trying to build this thing and You know, all we've got for resources is about five guys that kind of get it we Diverted a blade center that was heading for the trash and that was what we built on which actually turned out really useful because If all of your stuff in your development firemen is breaking all the time you get really good at troubleshooting and high availability stuff So if you've got some real junkie stuff put it in the lab, it actually helps So we started digging into this thing started building it and we for our POC we wanted to do a a fulsome build Get nice era MVP Which is now VMware NSX in there because we really wanted to kill the Vlan thing over in our You know managed hosting setup We call it death by Vlan every customer that comes in has to be provisioned a bunch of VLANs And there's all this confusion about who's got what and where stuff supposed to be deployed to we wanted to be able to put That in front of the customer and let them do it There are a bunch of other benefits to the SDN piece, but that was our our main thing So as we start to get into building this and start to look at what components do we want and How do we want to do sender? What do we want to put behind it? What do we want to put behind glance, you know We start seeing that Everything in the docks leads up to a certain point and then we see This next patch is coming that has this awesome feature and we see the next thing coming along We're like, oh we need that right and so it seems it's always the case that there's some business problem or some customer service problem that's it's always solved in the next release because everybody's got the same problems we have and Luckily, there's somebody coding against those problems But I always have a current problem. So this was pretty much the most fun. I've ever had at work was building this So start googling like crazy, right trying to make stuff work our initial strategy was that we knew since we all had day jobs basically in addition to doing this thing We had to cut down our communication time between people. So round everybody up stick them in a In a Conference room that used to be an office and then used to be a training room and then became a conference room again The I'll go in there during the summertime. Yeah, the ventilation is not right in there Apologize to anybody that ever goes in there. So while we're in here in the sticky room, you know, we Start learning agile have a really good scrum master to start out and it's stickies all over the walls, right? There's your stories. There's your backlog all that stuff There's whiteboards on all four walls of that room And then we bought one with wheels on it because we needed to have a lot of whiteboard time to figure out what we're doing But having the whole team in room in one room was critical I know everybody's really into distributed groups and you know people all over the country stuff But the communication time of a guy across the desk from you is very short So we started talking to some vendors and one of the vendors right on the phone flat out when we tell them what we want to do and how much time we have Literally told us we were crazy and it couldn't be done. So That really encouraged us now. We really had to make it happen. Nobody that's in here though. No So when we first set out we kind of try to divide and conquer like okay, Zach you take glance and Justin you take Nova I'll take Cinder You know, we'll give Matt quantum now neutron and That doesn't work you can't go and kind of hover over one thing and work on it and try to get it done And then sink back up later. So we found about 500 ways not to build open stack and Iterated real fast and when something didn't work. We just trashed it just start over do it right take some notes Try it again Eventually we kind of developed a method where we just Through the projector on the wall and at least two if not three people are looking at it at every command going in every config file Change and so it was kind of like extreme programming for configuration files All right, but mistakes were reduced if you make a mistake in Keystone game over, right? It's not fixable. You're gonna have to go fix and rebuild Keystone to get it straightened out At least back in the Folsom days probably easier now so We get to where some stuff's working some stuff isn't and then some stuff that was working stopped working and so the the wall there is the tech debt as Post it after post it starts flying to the wall and start to get kind of depressed But then you know you have the upside Everybody concentrates on what's in the tech debt to fix it and you wind up with Okay, glance is fixed now now it works and that's a huge victory And we got to the point where these victories were hard fought enough that we start clapping every time somebody comes to scrum It says this is fixed this works check out the log. It's not filling you a gig an hour anymore It's you know images are going in and out of glance as an example It's a little motivation to keep you right and so the atmost clap comes up so then we get to about two-thirds the way through and And We're finally got enough we've learned enough about OpenStack to get a cluster built and working and we're starting to do our NYSERA MVP to quantum integration and NYSERA was a huge support on this they had a an amazing engineer That was just he built an environment just like ours and started hacking through the exact same OS same Packages all this stuff and so we're about two days behind this guy You know we would call him up for the next day's meeting and say hey I got to this and security groups are jacked up and he says oh, yeah, I ran into that. Here's the code I wrote here's a little dock on how to get it in there. Okay. Well, I know what I'm doing for the rest of the day Right, and so talk to him the next day get the next piece Eventually got that working and we're starting to get pretty happy about this thing So I'm gonna let Justin take over for the for the actual the fun the fun ends and things are actually working now All right Awesome stack. So at this part in our project, you know, we actually have something working You know our main feature being a Deploying VMs without errors very exciting for us Through the horizon interface which you know, we had some challenges with As Mike said, you know, we finally got you know later plug-in from NYSERA And we start seeing the awesomeness of you know what SDN can provide us, you know, we you know, we're On the verge of getting rid of VLANs We could see customers can be empowered to create their own networks routers firewalls, etc. It was very exciting and Along the way, you know, we made you know some additions. So If you ever use any enterprise IT software or any software in general, you know There's always things that don't work as well as they should or there's a feature there that you don't have And so one of the reasons we chose OpenStack in the first place Was to be able to extend it in the you know, the way that you would expect to OpenStack being, you know, open source Written in an approachable language We were very excited about this and one of the challenges we took early on was our version of Horizon lacked the ability to boot from volume. These features were available in the CLI and API of course But we felt like in order to put on a true demo to the executive staff that we really needed to do it The way a customer would and that's through the web interface So we dove into the Django code and put a working piece in there We also Took a look at the instance deployment Workflow we felt it really wasn't as intuitive as we would like our customers To be so we tore all that out and you know put in a step by step ask you questions you put in answers at the end a VM just like you wanted it Additionally the monitoring Now we're all Either former system admins or current system admins and so we knew a ton about monitoring and could easily have punted here and Chose a traditional monitoring product like Nagios or something But since you know, we felt like we've already done all these extensions We wanted to bring it, you know closer into the environment and actually, you know make a dashboard That had host health on it Monitored the Ceph bit That turned out to be a lot more complicated as we wrote both, you know clients for the hypervisor as a server Exposed an API that you know horizon was able to get to So that was exciting for us This last piece Atmos monkey is a project. We started early on We wanted to be able to continuously test Open stack and so we wrote a project that would continuously do all the things that a user would do Create VMs create users Create volumes routers networks everything and test them in to end and notify us of any problems because one of the things we ran into as we Changed configurations extended horizon this things would break and we really wouldn't know about it because we're not constantly You know exercising the product and so, you know, we we wrote a thing Now with the you know the tempest project. I mean, we're totally going to Scrap our implementation and just use that, you know with all the scenario based tests I mean, that's those guys have done fantastic work there. So we're just going to take that and run All right, so Here we are end of our 90 days demo day we're invited up to the Executive conference room Now like any presentation you're gonna have some last-minute problems ours was a security group error Whenever we create a security group through horizon a scary Red message would flash across the screen Of course everything would would work, but you would still get this, you know ominous error So we we had to tell the whoever's driving the presentation to be ready and you know hit a quick f5 on that So you could hopefully avoid any embarrassment. So After we you know spun up a bunch of VMs to prove this thing was all working and you know what a customer would see You know, we wanted to give kind of a deep dive into a seph Because we really viewed that as you know a future platform that not only for VMs, but other parts of the organization So we wanted to impart on them, you know, how impressed we were with the with the product not only through our terrible blade-centered Hardware failures But configuration changes, you know us just not knowing what we're doing the thing what you know performed, you know fantastically So kudos those ink tank eyes. That's amazing product So all right Everything was a success beyond impressed and so What the organization kind of realizes this open-stack thing may be our future So we wanted to you know go out and socialize that with the group and Let's build a production thing But of course like any government organization. We actually have no money So there's no R&D and we need to find a way to pay for it before we can actually start So this begins the long process of you know taking this POC Demoing it to all of our largest customers who have problems that we think are solved with this new way of Deploying I ask and so luckily we found such a customer who was willing to put money in advance and kind of buy the dream So like any big change or any product rollout in an organization, you know Not everybody is you know open arms and pats on the back, you know, we had some concerns with some of the internal groups There were you know finance concerns whether this thing it was going to be so cheap To sell that was really you know potentially going to cause revenue problems. We had you know operational concerns with some groups that This thing was going to let customers do for themselves what they had previously paid us to do So we had a lot of meetings with groups and you know kind of talked about How we gonna do things, you know in the future, you know, how we split models up into you know Professional services and sell add-ons and not just have everything baked into, you know, the traditional thing we have right now so With that I'm gonna hand it back to Matt. He's gonna talk about what we ended up with Yeah, I'll just tell you want to flip this slide. Oh, okay, that works to you so yeah solution to stack so What we ended up doing was Writing on our whiteboard world-class cloud. So that was our goal every day. We would walk in and think about World-class and we would not settle for anything less than that So we've been doing all these crazy things, right? So we've been Googling things we've been building things we've been adding and destroying and all these things going back and forth and Destroying the the stack and rebuilding and all these things. So we're trying to figure out What do we need to do to become world-class stable sustainable? so We're not able to hire like a rack space so what? We meant there is we're not able to just go hire 1500 people and let them go nuts with it, right? So we basically had five guys and a stack of servers to do our development and build this thing. So Trying to figure out how are we gonna actually do this thing? And And so look for companies to understand our vision. So we were looking around we read some things We saw that there are some cool companies doing some cool things. We of course talked to some of the founding members of OpenStack And and trying to really dig in and see where we needed ahead some of the things that government entities of course are really Strict on is 24-7 support so we need to build a rely on other people outside of our organization What if we just you know picked up and left for something we run over by a bus or something like that? So they want somebody that's going to integrate with them Very very well had that level of stability behind them things that we are also looking to do is Plan for Fed ramp. So we are Fed ramp Well, we have the Fed ramp provisional Agency we are the first in the government to get Fed ramp. So yeah, so It's a risk authorization management program It's basically an off-ramp for IT to go to the cloud like Amazon rack space Yeah, so we are the first government entity to do that as you will see the list is growing a Lot have it now. So that is motivation for us to continue our development efforts So moving on here. Oh, there you go. Perfect. This is what this guy does every single night. You may want to Take take take the night off tonight. I don't know So these are 853 controls so this is something that Zach embeds himself in daily Unfortunately comes in and he gets angry because he has to read so many of these things and he I Don't know where where you left off right now. Did you? Submit or we have 95% of our controls written for for this So we'll be submitting here pretty soon. This is 853 rev4 so We're at rev3 now rev4 s be implemented here soon But I mean this is this is the building blocks for security in the government right now So this is what fizzle is based on He needs lots of hugs we give him hugs lawyers shouldn't write nerd descriptions right nerd should write this stuff not lawyers and That's what kills it for you Exactly. Yeah. Yeah So moving on We continue to think about operationalizing So we've been doing lots of research like I was mentioning some of the things we liked In our research is provisioning that piston was doing so molt every service on every single note Right so designed for failure. So if a host fails no big deal We are impressed by their operational philosophy. So Being able to integrate with them Would be a plus, right? So Being able to have that support method in place another into Partner in this is the mware NSX. So the maturity behind their software to find networking components Or bar none the best that we've seen so far There's a lot that are coming along We continue to do research on a variety of different ones But they are definitely the leading SDM providers at this point And we successfully implemented that And like we mentioned we are on the heels of turning ours live. So I think we're about 60 days out from actually lighten that sucker up and and officially declaring it production So also the troubleshooting capabilities within NSX are unbelievable I don't know if you had a chance to see any demos out there or go check out what they got going But I would definitely recommend doing that. Yeah, those troubleshooting capabilities really helped us get it built we When you have an interface and you're trying to figure out why this VM can't talk to this router port or why this VM Can't talk to this other VM you go into the NSX manager web page You go in there and you choose which two ports you're talking about and you hit go and it will go down through the entire Stack of all your SDN stuff and the controllers and the service nodes and which hypervisors and all this stuff and see which ones can talk to each other in which direction as we were building the thing out we Ran into problems where we had messed up something an open v-switch on one of our hypervisors and got it right on the other ones and You go and do that and you're trying to solve the question of how come the VM can ping its gateway when it's on this box but if I move it to this other one it can't and You hit the troubleshooting wizard there and you just see This hypervisor it's receiving packets this direction, but the packets aren't coming back this other direction You know exactly where to go troubleshoot and then you can you know blame your buddy who last touched the config file So we do have this setup and working and if anybody is interested in seeing some things behind the scenes We're more than willing to show so let us know The silicon mechanics piece is another vital part for us So the amazing fulfillment part is Whole rack increments. So their philosophy is is we build whole rack worth of servers whole rack worth of cabling Switch gear everything all wrapped up and bundled up They deploy piston And they ship it to us So it hits our dock and all we have to do is wheel that sucker in put it in place Powered up and away we go and we got a cloud built in a matter of hopefully days rather than months Obviously the first time around We work through all the issues. So hopefully the next bundle that we that we order will actually go smoother, but That's just internal Related issues that's not vendor related issues So that's us fighting with trying to get power in place I don't know if anybody is from the government around here with the silos. Yeah, exactly So the question that we get down to is how do we get this to happen? So get what we want what we ended up doing is talking to all these guys and saying hey come to Kansas City and have some barbecue so we can pick your brains and Even Josh McKinty came out and he tweeted lunching Kansas meat is not optional So if you come hang with us, you're gonna beat in a ton of barbecue So if you want to come out and take a look at what we got Please do we'll definitely stuff your faces full of Oklahoma Joe's or something like that But anyway, thank you very much Questions we're open for anything so throw anything at us. We're willing to answer anything I don't know if there's anything we have to keep off the table here Yeah, let's try not to talk about an Easter fisma Any questions we don't want to replace we Passes a in the government. It's a special thing because we have to we have to retain full control up through the operating system layer We wanted to provide an infrastructure as a service With with new abstraction layers that provided, you know a greater span of control for the customer and that's what we saw An open stack and that's really what the hope was so what we were having with our VMware managed hosting setup is The one-size-fits-all problem. There's some customers that go into that and they don't have a zack They don't want to do all the security controls and all that super tedious and boring Patching and certification and audits. They want to inherit that from us and for us We become really good at that because we're gonna do it for thousands of machines But you're also forcing customers in there that have a huge IT org and are really quite good at this already So they come in there and they're just super frustrated They're their first question is what's the root password and then it's a meeting schedule to explain to them How they can't inherit the controls if we give them root and so those guys really need an alternative They need a true I as where they can bring their huge very well-developed IT organization that can already fend for themselves and Satisfy their goal of going into a consolidated enterprise data center and get you know their boss cubed off their back about Why is this computer closet still running? Fed ramp really kind of puts a target on hosting centers like us in the government space because it's sole purpose is really to authorize Government to go to rack space to go to Amazon and we feel that you know There's just some systems that need to be behind government walls. So yes Very sure. Yes, absolutely. Welcome competition. It can be a compliment as well I mean there's gonna be hybrid scenarios in the future where you may want to have a portion of your multi-tier app That's there and the other the other tiers are in my data center. So we're looking for whatever our customers want so Hybrid scenario may be something that they want in the future Right now. I think it's just To to have more empowerment, you know with data center consolidation One of the things, you know, we've all migrated systems here and in support of that that program And one of the things you're doing is you're taking power away from ultimately your customers and you're forcing them into processes That they didn't write or necessarily consent to and so it's empowerment is what we wanted to give back to them That's what we kind of found with OpenStack Looks like we've got about five minutes left. Let's go on to the next question My question was similar like I was just wondering if you'd like described what sort of what your three main benefits Were overdoing OpenStack rather than like public cloud providers So there's one of those I hate to say this but government regulation, right? so if you've got some some application that has to do with a lot of personally identifiable information like social security numbers things related to the farm bill and Monetary authorizations related to farm bill and stuff like that. I Don't think we're at a point where that stuff's really appropriate for a public cloud provider now if it's the USDA.gov website Sure, there's nothing scary there and you know those guys that run that can make a decision So we wanted to have an alternative that's public cloud provider like but still within the government regulations still You know government employees and government contractors with all their background checks and all that stuff, you know still doing the work Another benefit for it was that When we moved away from trying to bolt something on the front of large vmware setup and Start to kind of mix this past thing and then put I as that self-service into there to move toward open stack The the pricing starts to align much better with what the public providers are able to offer Even when we're not at their gigantic scale and that's who we're being compared with too. So I mean Customers would come to us. Why can't you do this Amazon does it? Why can't you do that? Rackspace does it they'd quote the 510 into cloud and you know demand agility and elasticity and We can't do that with Cots. We have to we had to build it And I'd say probably the final large benefit is that we could further tailor open stack to our specific mission or our customers mission You know when we get out of the one-size-fits-all We can do a lot more for the customers if they want something different in horizon. We can get that for them So I don't whether you're seeing the latest or the most recent DHS RFI and they explain the ECS to how they're going to build that framework around a hybrid solution So what I'm seeing in the agencies or most agencies within the federal government is everyone's trying to sell services to everyone else It seems like a Grace this whoever can be there first with a hybrid solution to sell brokerage of services Is going to win the battle a DHS? described in their RFI that they want to sell services DOD federal agencies federal contractors and consumers I actually did some reference architecture for a partner that I'd worked with around Describing what that would look like Overall from a hybrid standpoint. So I guess my question is do you guys see that you have to position yourself in competition with the other agencies and is there some way that you're looking at Covering certain services versus what DHS would cover and obviously DHS from an intelligence agency standpoint So I would say that right now we're in pretty direct competition and trying to out develop each other Right, even we might wind up using the same products What I'd really love to see is the hybrid cloud model becomes an option of I put some of my stuff at USDA Maybe I put some of my stuff at DHS's cloud Maybe I put some of my stuff at Amazon and move them around as needed, you know by business needs There's a lot of room There's a lot of computers in the government and more of them show up every day Yeah, we've been we've been recruited also to build on the site So a lot of these guys that we're doing demos for now and asking for all the services you're talking about Actually want them on-prem as well. So I don't know if you've seen that DC1 DC2 they're already talking about Consolidating and also having a single sign one to get access to their conventional That was going to be my point was that data center consolidation, which is a mandate All right, that comes at a very high level. That's going to force cooperation at some point Once once the week have been weeded out that the redundant data centers and the closets and all that thing where it's going to force Cooperation at some point. Yeah, my last question is a quick question With all these mainframes to where you attempted to run open stack on their ZVM. No Good one. All right. I guess we're at the end of our time. So if you have any more questions We'll be out in the hall there We're welcome to talk or do whatever so come on over and talk to us. Thanks. Thank you very much for attending