 We are part of the United States Digital Service which is a family of different digital services inside agencies and the AT&F group which we focus mostly in doing project work for all these agencies. We are an open source team so everything that we do is out in the open. And we are here to talk a bit about DevOps. DevOps is hard, I mean there's no way around it. DevOps in government is a pain. You might say well I mean I work at an enterprise and it's not easy either but trust me it is worse. We have laws, we have regulations. We have security teams that go in and take a look at what we do and procurement people is usually pretty interesting to try to deploy something in the government. So in the past if you wanted to deploy something the way that it would work is that you have your application, the application is there or it's mostly there and you say I want to polish it, I want the public to have access to this. So you document what you had. You present it to a change management board because of course we have those. You procure the servers, you know, get bare metal, you set them up, you configure them. If you're lucky there's some sort of virtualization in there that you can run. You apply your application code, you run a lot of security scans which can take a very long process and submit it again for some other people to view and finally you get an approval to launch. This process, this whole thing can take 14 months. So good luck being agile there. If you want to deploy a change to the application you still have to run through a lot of hoops and you still have to get that approval which still prevents you from doing agile work and be a charity and that kind of stuff. So a lot of agencies are following the trends today and saying there's this cloud thing going on. We should get some cloud. So we have this thing and it's just cloudier. We instead of procuring servers which takes a couple of weeks, we instant somewhere. But the whole process is still in touch, still takes a very long time. So at AT&F we like to change things, we like to break the mold and they hate us. So one of the things that we said, okay, we have these things that we do for every application that they look the same. I mean, when you're deploying an application pretty much it's the same thing every time. The only thing that changes is the application. So we said, okay, let's package things, there are only ones and then your application needs to, you know, can be deployed on top of that. So we created a common architecture, basically baked an EMI, documented everything that had that EMI. And when the developers wanted to deploy a new app, they just, there were some issues with this approach. One of the biggest problems was that we didn't know what was going on. So if you wanted to update the base image, who is using what, where, and there was no inventory service of what we were running. So why do they have to write that? Billing, we can't give away stuff for free, so it's a pretty big issue if we do. So we had to create some billing options within the infrastructure that we had. And that involved a lot of manual tagging and stuff that was pretty ugly. And developer productivity with all this took a pretty big hit because, again, they're writing manifests or recipes or whatever when they want to write their applications. So we took it back and said, this is not really working. This kind of sucks. Let's try to figure out what's the best way of doing that. And that brings us to the present. It was pretty obvious, at least for me and our group that pass is the price that helps if you have multiple, I don't know, interrogating projects you probably want to pass. So we said, okay, let's take a look at the passes that are out there and see what we can deploy. So we had a pretty big set of constraints, one was that it was open source that had multiple language support that was deaf friendly and Ops friendly and that it's supporting multiple infrastructures. We didn't want to have vendor lock-in. One of the big things in government is that we don't want to have our software be tied to one provider. So these are the ones that meet this criteria at least at the time that we were looking at them. And we had a pretty fun, I have guessed, which one. So Cloud Foundry for us, it was pretty clear that it was the platform that we wanted to go with. Basically, it's just being production ready. You would be amazed if you go talk to one of these other platforms, they're like, wow, I'm not sure if you should deploy this stuff with our platform, but Cloud Foundry was like, yeah, come on in. It is multi-tenant, it is enterprise friendly. So some people don't think that much about either UAA or spaces or organizations, but for us it's amazing, it's all baked in. It has the community and the foundation, those are fantastic. And there's one thing that as an offsperson, I got to love, and that's Bosch, I don't know what Bosch is, go check it out. Go Bosch, Bosch is the thing that is allowing us to have open source benefits, we have the description of all our stack in our GitHub repo. We have everything that is going on, even if it's small, it is there. It allows us to do that multi-deployment stuff that we wanted. So if we want to have a staging in the production environment, we can do that, but also it will let us do stuff like agency A can have its own cloud and agency B can have a separate one. It has its own audit logs so we know what everyone is doing. And there's this thing that if you use Bosch, you know, but it feels safe. When you're running Bosch, it tells you, hey, you're about to do this. And buying something in Amazon, it tells you, are you sure about that? And you click yes all the time, whatever. But when you're deploying a platform, you want to know exactly what you're doing. And Bosch is usually pretty safe and when something breaks, it tells you about it and you can fix it. And lastly, it lets us do this layer and thing that we were doing before. So we have the stem cell that we can review and we can review jobs on a case by case basis. So we decided, okay, let's have a real full-fledged pilot for Cloud Foundry and see how it works. It will be an internal platform. We would have to have multiple languages and multiple frameworks deployed in this platform. You have to leverage AWS because that's what we were using before. We had to create some small services inside it to see how the framework and it has to be able to deploy production apps. So within this pilot, even though we knew it was going to be a pilot, we wanted to have applications that are going to have real traffic going in. And lastly, we have a very small team for those, it's just three people. And we had to say to reduce the load on the team. We were hammered before we were doing this pilot and it didn't make sense to deploy something that it would have more load on the team. So it hasn't been, awesome. We have been running this platform for five months. We have about 100 app instances, which is not a lot, but it's something. About 50 users, and this is all internal stuff, I mean, only 18F. We have some production apps that I can't really talk that much about. But they're there. The user management that we have today is 800 times better than what we had before. Basically, when someone left the company or the company, the agency before, it was a pain to get rid of someone. But now it's just, you know, you can see the lead user. And we have, we are able to say, hey, what should we build this agency and it's just right there. Developers love Cloud Foundry. Our team of development just loves it. I asked around for quotes about it, but they were asking developers for quotes, it's tricky. There was one serious one, which is about how faster we're doing ATOs. Now, ATOs is the authority to operate. Basically, the approval stamp to get a project out in the open. Some projects before Cloud Foundry and a lot of other bureaucratic hacks, this is not just Cloud Foundry, would take about a year, and now they're taking about two or three days, and that's pretty cool. And then we have some random other quotes. But if you take the Cloud Foundry away from us, we will hurt you, or with Cloud Foundry, I went from punching myself in the face to punching other people in the face. So what did we learn so far? We learned that developer happiness and productivity can be pretty deploy stuff. So if their deployment environment is awful, and it takes a lot of work, then they're not gonna be happy about it. Having a common architecture is a big plus. Cloud Foundry has its limitations. You can only deploy 12 factor apps right now with Diego, with me. We're gonna be able to deploy other things. But this 12 factor limitation is great, because all our apps now look the same, they look pretty similar. We have a much better utilization rate on our servers. Before, we were spending redundant stacks or whatever that had no usage at all. And with Cloud Foundry, they are a lot better. One of the important things that when you're deploying Cloud Foundry is that you should spiff your manifest. We learned this, it was awful maintaining that huge YAML file. If you don't know what spiff is, it's basically a tool to wrangle YAML files and configuration for Cloud Foundry. It is what CF release uses. And it's just needed if you're running Cloud Foundry. And there are some other things that we learned. It's like, you need to document everything as you go. Because some stuff, you don't know how to do it again, if you don't. And it's also very important for your team, for your development team, for your operations team to see how to do things and how you are doing them. It's also very important to contribute back. I mean, whatever you're doing, we're in an open source community, it's very important to contribute back. And communicate, communicate with your team, communicate with the community. And don't be afraid. I mean, this is a very welcoming community. I made thousands of dumb questions about it. And the other thing is that, don't be afraid of going in the source code, taking a look at things. There's no magic going on. I mean, I found it again. The hard way, I'm like, what the hell is going on here? And it's all there. So this is all great and we do want to take it further from here. We want to see what goes on with Cloud Foundry and our platform, but we do want to expand on it. We have a couple of things in progress right now, which are a lot of CLI plugins for the ECF CLI. So to improve productivity for developers, they're pretty simple. We want to improve our monitoring right now. It kind of sucks, but it's gonna be better. One of the things that I mentioned before was that we are in Amazon and we want to leverage their services. So we are the Sbroker, so we can have databases created from the platform for our developers. And they can