 We have Nathaniel Burton from the NSA let's hear what they're working on. Hi everybody I'm Nate Burton here from the NSA and I'm gonna talk to you a bit today about what we're doing with OpenStack. I Wanted to start by saying thank you basically if the the only thing I did today That I said that you took home out of this was that the community is doing an amazing job with what they've created Thank you to Rackspace and NASA for starting OpenStack making an open source sharing it with community Building a system that kind of fostered development and creating a thriving community around OpenStack and Making making it pluggable in a way that you could still have a thriving ecosystem of Vendors open-source companies startups around it creating innovation and specialization in specific areas So I'm gonna talk to you a bit today about The NSA some of our IT challenges and how we have leveraged OpenStack to build a private cloud However given that given the nature of our work. There's a few there's a few things that I can't talk to you about So the number of users The number of systems we have running OpenStack The number of servers we have in our open-side clusters the amount of storage capacity we have The various applications. We're actually using our open-side system for and I'm pretty sure I can't tell you that So I work at the NSA. I've worked there for about 10 years. We're located in Fort Mead, Maryland We are part of the intelligence community. There are 16 agencies and organizations all under the management of the ODNI The NSA in particular does signals intelligence and information assurance protecting and securing US government systems We have a large workforce made up of civilians military folk and government contractors and One thing that's unique about NSA is that we have a very large technical civilian population So rather than lots of things being Run by various contracts and things we actually have a lot of hands-on technical folks doing development doing research And in particular we've got lots of people doing Innovation in computer science mathematics Crypt analysis we're actually the one of the largest employers of Linguists for doing foreign language analysis as you might be able to figure out We we use pretty much all the technology out there We use a mix of commercial open-source in-house develop software if there's any particular type of hardware Software applications development languages. We've probably got it in various pockets in our in our environment and every time we go to Build a system or try to solve a particular problem. We look at existing Commercial products out there to see if there's anything we can use open source And then pick the best thing to solve the problem because we don't want to reinvent the wheel So cloud the we got a good talk from the physicist before me Randall about some of the cloud stuff that they're doing and from That point of view but in media like Clouds been this buzzword that's been blown out to be be everything right it solves everybody's problems. It's a panacea And you've got like All the as-a-service type thing software as a service platform as a service things like that But particular at NSA. We have a slight Specific definition of we think cloud what cloud is cloud is cloud is big data Obviously, we probably have a lot of problems that have a big data focus so to to us at NSA big data is being able to take data and enrich it across other data sets And analyze it in a way that that goes beyond the limits of what you could do in a more traditional Ingest process index search type workflow So our big data systems are based on Hadoop and a cumulo cumulo is Was developed at the NSA. It was inspired by Google's big table paper And we added some things to do individual cell based security iterators performance enhancements to increase ingest rates into the the database and and two years ago We actually open source that and it's now a Thraving Apache Software Foundation project out there that anybody can go and use So on on cloud the guy from Hubspot yesterday mentioned how When they were trying to look for their private cloud solution, they were looking for that kind of mythical beast the the unicorn Unfortunately, our existing systems at the time were not at all unicorns. They were horses with a cone on their head So so our infrastructure is the service cloud in the past was manually intensive There's lots of stone pipes of excellence There was lots of different teams Who shop was to do DNS or to do IP addressing or to do network configuration or stores or whatever So all those individual teams actually had a lot of automation and and stuff that worked well within their little component But there was no kind of cross component orchestration to to do lots of Efficient service delivery So to give you to give you an example of what our previous system was before we got into OpenSAC So somebody has an idea and they have to go through this process The process is like fill out amount of paperwork wait in line while you're while your request gets processed Well, somebody looks at the the the request and then goes back to you and says oh you forgot to fill out This is in this and it goes to an approval board and then this process might iterate for every step of the way And you might find out. Oh, there's some other group that wants to insert their process into the past to actually getting anything done So obviously government it was large it was a large bureaucratic process, so we're trying to get past that so weeks or months later Somebody trying to go through this process to do something to try to develop a capability or an idea to see if it even worked Would get so frustrated because the process would take weeks or months to happen Before they can even try it out and by that time you're like Why bother what was I even trying to do? So the the problem we were trying to solve was was that it took too much time From idea to capability to actually deploy anything in our environment and we needed scale and we needed Agility to be able to do things quickly as the mission changed and demands came up So we looked at a lot of things and our solution was we wanted something to lower the barriers entry There was self-service on demand elastic with API access so that we could do things in a much more efficient agile way and We investigated a private open stack cloud And I'm gonna walk through kind of our story of how we how we rolled that out So open stacks are up and I I went to the Diablo summit in Santa Clara years ago, and I came back all invigorated about everything the community was doing and that racks face and NASA had done and The technology seemed really cool, so I came back and me and another co-worker mine too bad scientists We stole a rack because the process to actually get equipment went through that long process of paperwork and tickets and boards of people So we we stole a rack in a lab And our goal was to try out open stack offer flexible hosting and kind of just get a get a feel for what opens I could do for us to help solve some of our IT problems So within within about two weeks We actually had a system up up and running based on cactus like Mark said earlier at the time It was a little rough around the edges, but we actually got it working and we had API and CLI access We had tens of users at the time. We're just talking about a lab here But it we really started to get an idea of what it could do like Users no longer had to go Submit a request to the lab management team to get something they could actually do it on their own So we started to see that this might be able to help us out But we needed to go beyond that because this was just a lab environment people can only really create Toy applications or prototypes in there because they couldn't talk to anything out in the external Network or put it put their application in front of other users to see if it was beneficial So that was the pilot system was our patient zero So we started to see our first unicorn, but it was kind of kind of just a paper unicorn here So our next idea was to go bigger obviously as a as a techie you always want to try to go bigger So we wanted more hardware more users more use cases Basically just more so we wanted to go bigger. So we decided to co-locate our second system with one of our big data systems So we started with half Iraq and over time we tripled that in size This actually gave users access to Mission data, so no longer where they in a lab environment where they were far well-dough from ever from everything They could actually get access to data and develop things that other people could use and they could actually see the Benefit of so we started getting use cases such as relational databases web applications not Hadoop processing That they could now do in a much more flexible way because it was co-located with the big data system And we had open stack to be able to give us that elastic on-demand access to things So the results of that were really great. We had hundreds of users over about a six-month period We started seeing the benefit of the fail fast model where people are actually encouraged of trying out new ideas And whether or not it might succeed isn't really important. It's whether it's giving people the incentive to try things out So we were very generous with capacity in the system because we were still trying to get a feel of how the open sex system worked for us and we gave people large quotas and We kind of playfully shamed people who were Who were abusing capacity just spitting up a bunch of stuff and not doing anything valuable with it? But we really started to see the huge potential for kind of general applicability of Solving our problem. So this system had more unicorns twice as many That was patient one so what we really needed to do next was figure out how to make it real how to how to put it in production to the the entire Technical workforce at the agency to solve development problems and solve production hosting problems And so we had to look at some of our Organizational challenges to rolling out something that completely changed kind of that so fives of excellence model And We had to figure out how we're gonna kind of get to production So we had to look at things like automation Automating kind of the deployment installation configuration of of open stack So we did that with things like public and we got some point where now we can actually take Bare metal racks and servers and install our open stack load on it in about 20 minutes and have a functioning open stack cluster We had to look at how do we secure the system? We're actually kind of good at that so we Secured the operating system of all the the host in our clusters. We looked at how do we secure the API? How do we put SSL everywhere? And how do we how do we secure the guest OS's that are going to be running in our cloud? so we we did a lot of image instrumentation with kind of baked in security practices standard hardening based on some of the NIST guidelines and things like that and and So we hope to in the future take some of the stuff that we've learned from Securing the open stack system and kind of release that back out To the community as kind of secure hardening and configuration guides similar to how the NSA has done with Operating systems like Linux insularis and windows in the past So hopefully we'll be able to do that in the future We had to figure out how are we gonna actually give people accounts? How are you gonna track usage in the system in a in a public cloud environment? It's kind of easy you just build people based on the usage you charge their credit card So that's how all the public cloud providers do it, but and I say We didn't have that so what we what we ended up doing is we have PKI and PKI is awesome because It's we we have ubiquitous PKI for all our users So it gives us the ability to really easily add single sign-on authentication identity tracking and management Within the system and to actually track accountability of how the how the users are using the system How do how to tie that back to a particular project or organization that they're working in? to kind of track the usage to feed to tie into show back and metering systems so that the powers of the can Determine how to best take advantage of the resources we have at our disposal so along with that we we figured out how to We created a free tier since we have PKI and all our users have PKI It was really easy to we actually hooked into Keystone and horizon to do auto-account creation when a user hit the hit the horizon website And and we we created this free tier where users could get a deep default quota With ever without ever having to talk to somebody which I don't say is good because most people are introverts And they don't like talking to people if at all so So this free tier really allowed people to start using the system create a VM or two and Try some out try out some idea before they even go figure out if it's going to be worthwhile. So that way they spend Appropriate time of the things that are going to be valuable So so this free tier really led to an outbreak. We had lots of unicorns. We had some rainbows We had some zebras with a unicorn head like thing So we had everybody just without even advertising the system when we finally went live We had an epidemic We we didn't launch it. We didn't market it at all People just kind of found the URL for our system and with PKI in the free tier It we had viral growth. We had hundreds of users just within the first few weeks without ever Mentioning that we went live over time that grew to thousands of users. We started seeing people running production workloads on there and Over time we've we've had multiple open sex systems. We migrated from Diablo up to Folsom I'm actually really happy to say that we're no longer running Diablo because in the past three summits I've been at we've always been still running Diablo and and The interesting thing is this is all been managed by a really small team in the past when we have had the the previous infrastructure service system all the various so pipe teams were fairly large and This opens access and let us grow to a scale that a very small team of 12 or 15 people Could could manage Which is really great for us So we started to see now that we had the ability for developers to get access to resources in a really quick Time we started seeing a change in our development pattern We we had users start kind of crowdsourcing and sharing ideas for application sacks We came up with these in a box recipes where somebody could create kind of Django in a box or Tomcat in a box things like that And then share that with the rest of the users so that you didn't have to reinvent the wheel every time you wanted to Deploy kind of a standard application stack It changed the system lifecycle process In in in our previous systems. We had very rigid Development environments test environment production environment and an application had to flow through that entire system and they're all physically separate systems And now with the flexibility that we had in our open second environment It allowed each individual project to manage for themselves how they wanted to control their system lifecycle If they wanted to have separate projects for development test production They could certainly do that But if they wanted to do more of a DevOps model this actually put the power in their hands to be able to do that Which we hadn't had in the past We had a more common since we had a more common environment. It also helped us to be able to share Application configuration stuff so that people didn't Have to worry about I built this thing over here And now I want to run it over here in this other system Is it going to work the same in both environments and ultimately this led to better collaborative collaboration amongst our developers And ultimately better mission to be able to create better mission systems Along the way obviously this was this was fairly disruptive It it was a complete paradigm shift to the entire Existing IT community So we broke lots of things along the way we broke lots of external systems that weren't ready for kind of Large-scale automated request access So so we've been having to work with those various external service providers of ours to be able to figure out how do we Create an API between us and them for those things that aren't controlled with an open stack We had to look at how do we change or eliminate IT process? That may or may not be useful when you're now talking about this elastic agile cloud environment And we had to rethink a lot of our problems. So The certification and accreditation process for taking a application and making it production We've had to work very closely with our security folks To figure out how to change that So ultimately this led to changing the game using open stack gave us better agility better flexibility Better scalability to be to be able to create better mission systems and ultimately it was a it was a win-win So it To the security folks there it was kind of a nightmare for them But when when we really kind of talked them down They were able to see that it provided It actually lowered the risk since we had this kind of Consolidated and central system that they now had much deeper insight into rather than these separate stove pipes so we we had a Better working relationship with the security folks to be able to show prove accountability Prove that the system was secure in various ways through reporting logging metrics And following kind of this trust but verify model that we worked with them for So it's drastically changed the IT environment at the agency So we we've kind of transformed the NSA and over the next few months We're actually going to be working with the larger intelligence community to actually roll out and deliver our open-sack system across the entire IC community and over the the next few months Hopefully under OD and I will be pushing out and giving access to our open-sack systems to the rest of the IC so that they can leverage the same efficiency that we have over the past Year year and a half So our next steps is we're gonna continue our growth We're gonna continue scaling out and adding more systems Hopefully we'll be able to track upstream releases quicker than we did initially We've done some open-source contributions. I've contributed a vulnerability fix for Keystone and a few other things and over time we expect and hope to contribute more and contribute more and partner with the community things like the open-sack security group and Things like that Thank you