 Hey guys, my name is Yvonne Nam and I'm the manager for the OpenShift Solution Architects for U.S. Public Sector. I'm really excited to be here today for two reasons. I guess three reasons. I've never been to Austin before. But the real reasons are, if you know me, you know that I really, really love OpenShift. I've been at Red Hat ten years now and I've never felt more passionate about any technology. And the second reason is in public sector, we don't often get a lot of opportunities for us to share the awesome things that we're doing. You guys know in federal, we've got pretty unique mission statements, pretty unique requirements. And I know maybe some of you guys have some preconceived notions about what IT is like in federal. But I am here to tell you we are doing some of the most innovative and cutting-edge stuff with OpenShift. We just can't talk about it that often. We're really fortunate to have a group of folks here that are thought leaders in their respective spaces, that are really blazing trails with their organizations on the adoption of OpenShift. And I'll give a chance for everybody to introduce yourselves starting with Ayesha. Hi, I'm Dave Connolly. I'm a computer scientist with the Internal Revenue Service in the application development area. And I'm leading the platform as a service initiative for our DevOps work. Yes, good afternoon. My name is Steve Grunch. I work for USCIS as a Customs and Immigration Services. And I'm the branch chief for Enterprise Cloud Services. So I represent the infrastructure and operations side of the DevOps. Cool. I'm Jason Kinsel. I work at Oak Ridge National Laboratory, which is a department of energy laboratory. And I help run the supercomputers at the National Center for Computational Sciences. Good afternoon. My name is Joel Turner. I kind of have a dual-headed role. I work for the Federal Judiciary. My main focus right now is my day job as I lead the Enterprise Architecture Program for the Administrative Office. And on the side, I'm also a Chief Deputy Clerk in the District Court in Madison, Wisconsin. Pretty impressive panel. So I wanted to start off by talking about some of the challenges that each of your organizations faced. And let's start with Joel, because Joel, at the U.S. courts, they have a pretty unique software delivery model. And that presented some unique challenges for them. Sure. So part of when you think about how we deliver software, a lot of companies will handle software delivery for their organization, all their users, use those particular applications that are kind of approved. And we have the same model in many respects. So the Administrative Office delivers national applications to the over 200 courts within the Federal Judiciary. However, local courts, because they work very closely with their local bar, basically their attorney practice, there are sometimes unique characteristics of those groups. And they also try and deliver solutions to their target audience. So think about it. You have a national service organization delivering service to a wide group of individuals. And those individuals, in turn, are service providers to the local community. So you have both national applications and local applications. And that creates unique challenges in terms of delivering applications to those unique groups and then sharing those applications across the federal space. Yeah. So you're almost like a software vendor within the federal government. Absolutely. And Jason actually runs the fourth largest supercomputer in the world. Yeah. So we're an open science facility. So all the science that we run on our machines gets published, although we do some work with industry. And they don't have to publish, but they pay for the use of the computer. But our scientists, because of that, our scientists are all over the world. We've got people using it all the way from the LHC at CERN taking data off of the particle accelerator all the way to other on-site labs. And other computational codes that run on the machine. And so what we were trying to solve is really bringing in, allowing these scientists to come in and use the computer, but then also develop a way that they can run their own workflows, databases, those kinds of things in our environment while still being able to meet their scientific needs. And Steve is with the USCIS, and they're going through a large modernization effort. So he has a unique challenge around the legacy systems that they're trying to modernize, but he was pretty fortunate in sort of the organizational support that he received in that modernization. Sure. So one of the things that we've talked about is that in order to get an open shift implementation in place or any kind of strategic enterprise architecture initiative in place, you have to have senior leadership. And for us, our CIO kind of mandated that we implement the solution. And one of the reasons that led us to this was one, we wanted to improve the customer experience with one of our applications. And we felt that in order to do that, the best way to start approaching that was to actually break the service apart. Creating the front end and part of this messaging queue. And we ended up putting the messaging portion of that application as one of our first microservices onto the open shift platform. And because of the leadership that we had and because of the way that we approached the problem, we were actually able to do that in a really short period of time. Yeah, I think they actually got something to production from zero to production in six weeks with open shifts. It was really exciting. Dave, now you're with the IRS and I imagine that you guys have pretty unique challenges around dynamic workloads, right? So I bet your workloads scale up and down pretty dramatically. We do, yeah. And that's one of the things that our filing season is January to April. So that's really the period that we build to. And right now we have a lot of applications. We build to the physical limit. So we have four or five servers on a web farm and it's targeting the high watermark, which really is those four months. So it's a shame that that infrastructure is sitting there those other months. From May through December, they're not nearly as busy. So when we look at open shift, it provides that opportunity for that scaling and for us to be able to spend the taxpayer dollars a little more efficiently. So that's one of the things that we're looking at with these applications, these legacy applications in particular. Just throwing an open question out there, feel free to answer. What was it about containers and open shifts that really appealed to you? What was the value that you saw in those technologies? Well, I'll take that one real quick. So from our perspective, it's breaking the dependency between a server and Jboss. So I have people waiting in line for months and months to get a server so they can get Jboss to start doing their work. And we have a Jboss 7 upgrade coming pretty soon. These people are waiting in line. They can't get to work because they're waiting for server deliveries through our current pipeline. And I don't know what happened over the last couple of years, but the volumes have just gone too high. There's not enough human beings in the building to be able to build these servers timely using these processes that we defined over years. It has to go automated. And so for us, it's breaking that dependency between the server and the Jboss container because we did a pilot test and one of those applications, the legacy applications, we put it on a container, took about six weeks to convert it from an Oracle application server on Solaris running to a Jboss 6.4 instance. We did it on a virtual server dedicated. We did it on a container. And to the developer, it was exactly the same. He had no concerns which one it was. He really said, I don't care because now I have a Jboss instance that I can actually use. So for us, that was sort of the trigger point. If we can get our developers productive, they need a good six months or nine months or a year to do these conversion efforts. We got to get that clock ticking as soon as possible. And the physical dependency right now between a virtual server and that Jboss instance is limiting our productivity. So that's one of the reasons we're looking at this technology. And for you, for a software delivery model, I bet it was really interesting. Sure. I think for us, there's a lot of things that are attractive about the technology but there are really two things that stood out. One was the portability of the containers. The ability for a developer to work anywhere and have an entire environment up and running and not be bound to either existing infrastructure or some of the services that were necessary, those could be basically on their laptop that could work anywhere they want. The other issue or the other attractive feature was the immutability. Change management is an issue that I'm sure a lot of people face. I'm not sure if we're unique in that regard. Nonetheless, the ability to have an immutable container and be assured that when you start the development process and as you move through testing and then staging that application finally putting in production, that you can assure that what you started with is actually what you end up with. And you don't get a lot of fat fingered configuration changes. So that was a very attractive feature for us. Yeah. For me, it was a good technology decision because in HPC over the past 30 or so years, as a community, they've really bought into POSIX. So POSIX users, POSIX file systems, we have a huge parallel file system that's POSIX compliant called Luster, GPFS is another one. And so these are all at the OS level. And so being able to kind of use containers since it's at the OS level and kind of use that as a... You can abstract as much as you need to, right? Since containers are just C groups and namespaces, they can appeal back what legacy applications don't understand, right, podpip namespaces or things like that. But it still gives us that control that allows a user to come in and run whatever application they need to run. So that was the big thing for containers. Sure. So for us, the portability issue was a problem also. We have different contract teams that develop software for us. So the containers address that issue. The other part of it is that when we did the containers and we built our pipeline to go into OpenShift, we kind of created a level playing field for everyone. And it also allowed us to put everything under standard code. We put everything in a source control repository so that we got consistent infrastructure and consistent builds across the board. We made a decision early on to automate everything. We ended up putting our OpenShift cluster into AWS and so we leveraged the cloud formation templates and building out the cluster so that we got consistency every time. And then also from the security aspect, because of the way that we did it and the decisions that we made, we were able to integrate security in from the beginning. So Steve, you hit on a really interesting point. So in federal, we have some unique challenges. We're not like an enterprise company where everybody's an employee, right? So we've got all different sorts of contractors with different terms around their contracts. So organizationally, we face a lot of barriers for adoption. So could some of you guys speak to the organizational challenges that you guys encountered maybe today? Yeah. I mean, I'll be happy to. Part of our challenge is like you said, we have a lot of contractors that come on board to support applications. One of the challenges we have is a lot of these systems are mission critical tax season filing systems. So the contractor comes in and says, I need 50 servers to be able to deliver that workload. You're kind of held to that number. So part of this technology provides me the ability to start judging that and seeing whether I need to or not. And like I mentioned with our build processes being slow, it's difficult. Other projects get delayed while primary systems get the treatment, that kind of thing. So that's one of the things that I think that we look at is hopefully being able to kind of shift out of this model where we're doing server directly, you know, based upon what someone tells us. You know, because we're kind of held to that contract versus the usage pattern. So we're kind of looking forward to that as we roll this forward. Now, Steve, we had mentioned earlier that organizationally you really had the support of your CIO, which meant a lot. But you guys also employed some really unique practices to help really break down some of those, those finger pointing barriers that existed. Sure. So when we went to go to implement the OpenShift platform, one of the techniques that we used was a war, what they call war room style. We invited pretty much everyone, all of the stakeholders that were involved in the initial rollout of our microservice. We invited the business unit, the operations side, the developers and security. And a couple other folks. But anyway, we got them all in a room. We talked about what we wanted to do, what the goals of the project were. And we got buy-in from everyone, you know, all at one time. This technique, what it allowed us to do was it allowed us to make decisions in real time so that, you know, if a contractor or, you know, someone from the development side had a question about one of the policies or about one of the decisions, you know, a federal lead was already there. We were already there to explain it. If we had to go back and forth with security, security was right there to make the decision. So, you know, we didn't have to call anybody or anything like that. As the project did progress, though, we found that we did have to reach outside of the core group, obviously. This came pretty apparent when we started to interface outward from the network. We started getting into, you know, security certificates and things of this nature. We did have to interface with outside groups, but we were able to pretty quickly get those folks on board and just get the job done. Yes, it seems like you made a deliberate effort to reset the culture to one of inclusion, collaboration, and communication. I think Joel also made a pretty deliberate effort to make sure that everybody in his organization was educated on OpenShift and containers and DevOps and these new practices. So to be fair, it's not just Joel. The whole team of people, some of them are here with me today. They do most of the work. I'm just the guy on the stage delivering this particular presentation. The reality is, though, is that I wouldn't characterize them as organizational challenges. I think it's more about, at least in our space, our groups or teams have been delivering applications on a virtualized environment for years. They know how to do that. And what you're now introducing is a new technology that has inherent questions about how do I do security? How is this going to impact my job? And so that's what you're really dealing with. The technology is never the problem. It's what is it in it for me and how do you get people to see the value proposition? What are you trying to solve? And then the technology kind of sells itself. So things similar to what Steve indicated, we spent a lot of time doing one-on-one reach out to individuals. We spent a lot of time having training sessions. We created reference implementations to demonstrate how the technology could work, pipelines delivering both internal and external to the cloud. But I think the most important thing was trying to dispel the notion that this is just for Greenfield. We have a lot of, you know, older applications that are mission critical. And so we took some of our main applications that one may construe as legacy and we started showing how those could be containerized. That got people's attention. Because where we started to gain more traction is that, okay, this isn't just for new stuff. You can apply it to, you know, what you already have and actually gain efficiencies as part of that process. Yeah, and we were joking about this earlier, but so much about this technology change is like relationship counseling, right? So you need to break down old barriers that existed before and get people to talk to each other in a way that they never did before. Now with Chasen, your constituency is remarkably different, right? You're working with academia and scientists, right? So I imagine sometimes they're very opinionated about their technology choices. Oh, yeah. Yeah, and the big thing with, I mean, it's two things, right? It's kind of like what Jill was saying about the operational side. And that takes a lot of presentations, things like that, to the ops members, ops team, you guys, and talking about the technology and then getting them in and letting them play with it, right? That helps dissuade a lot of those fears about operationally supporting the technology. But then, yes, from the customer side for us, it's the scientific customer who is probably not anything like a computational scientist or anything like that. They're a domain scientist who learned enough programming to do his computational fluid dynamics code and to get it to run on a computer. But he's, or she's, mainly designed around getting that code to run, right? So for us in OpenShift, it's getting that getting that framework so that we reduce that kind of, that friction for them being able to do their job, to get what they need out of the system and not get bogged down in MongoDB and things like that, being able to just kind of do their scientific work and move on. So you guys are all in various stages of adoption of OpenShift in your organizations, but could you guys share maybe some of the benefits you've already seen or some that you're really looking forward to as you put this technology in place? So obviously one of the first things that we, that we were able to bring value to is doing the blue-green deployments. We leverage the features in the OpenShift platform that allow us to, you know, deploy a new version of the application without burning the other one down, and OpenShift kind of does that automatically. So that obviously is a good, a valuable feature for us. And from a business perspective, what that allowed is for users to basically continue on with their input of data that didn't have to actually restart data entry if the service was unavailable. So reduce downtime or zero downtime even? So we're fairly early in our implementation. We were out of a pilot stage and now we're moving to operationalizing the service. I think the biggest benefit we've seen so far is we've changed the dynamic of the conversation. People have bought into containers as a strategic objective and how they want to deliver applications. So you're seeing more enthusiasm around the technology, more discussion about how to do it. And I think it's energized a number of people around the technologies at the point that they're becoming the evangelists for us rather than us always being the group that try and sells what the advantage is. Now you have the actual lines of business that are playing with these technologies and saying, hey, I've been able to do X, Y, and Z because of this. You really should look at it. I think for us it's, the conversation is changed, absolutely. And in a very positive way. With the scientific community, we don't have to, I don't know how many of you guys have watched Kelsey Hightower's intro to Kubernetes, I'm sure a bunch of us have. And he spends 20 minutes doing the Tetris scheduling talk, which is great. But what's cool in scientific space, we don't really have to have that conversation because the high-performance computing machines already live and die by a bad scheduler. And so, for us, it's been cool because we've been able to kind of focus in on what they're trying to do and empower them to do, to get their work done on the open-shift side just like they already can get their work done on the high-performance computing side. So that's been the biggest benefit so far. So we're just really starting our pilot work, exiting that and entering into, I guess, the phase where the CIO has identified this as being a major activities, assigned budget, as well as been executive in charge. So we have that really ramping up now. We're negotiating with our partners what the scope is, and obviously that's part of the fun work, trying to explain what this technology is. I will say this, though, that I found interesting from the conversations with these guys that are ahead of us. Some of the conversations are a little intimidating because it sounds like they're so far ahead, but a lot of these projects are only four, five, six, seven months ahead of you. So the speed at which you can take off is really kind of cool. So when you hear people talking about how far along they are, in my mind I'm thinking, oh, they're two years ahead of me, and now I'm finding out they're three or four months ahead of me, so I feel a lot better about where we're sitting right now. Yeah, I mean, it's amazing how quickly you can make an impact in your organization with a technology like OpenShift. I think we have about five minutes left. Do we want to open it up for Q&A? Does anyone have any questions for the government panel? Any other government folks want tips and tricks? It was either really, really good or really bad. We won't buy it, we promised. It's a real treat to get these guys out on the stage. It's sort of a coming out party in some way, so it's quite wonderful. Yeah, so we kind of make this just sort of gut instinct delineation between commercial and government. Have you guys seen the other side of the fence? And so what are the differences? My experience with government customers is they're pretty much enterprise customers. And there are different security standards, there are different data sets, but the problems really span both. What's the biggest difference? Like what's the biggest challenge that being in a government organization offers that doesn't exist or does exist in the commercial space? I was going to say that one of the big ones is just technology insertion. When you look at the graph you're showing up there, there's 1,050 new tools coming out every day, and for us it takes us a year and a half, two years to get it procured, cyber approved, somebody trained on how to install it and to maintain it. So some of the restrictions we have is trying to take advantage of investments we already have, like our investment with Red Hat. We use VMware products, so if we can leverage that, that helps us a lot. We don't have quite the flexibility that we'd like to do Greenfield at times, so that's one of the challenges. I think it is unique to the federal space. I would also, as David said, contracting procurement is a major issue in terms of speed of delivery for any solution that you're trying to do. The other factor which is probably also prevalent in non-profit is just that. There is no profit motive from a government agency. Our drivers are probably more cost containment and then delivery of value to our constituents as opposed to a line of business driving for revenue generation or profit motive for your shareholders. So there's different drivers to get you there and the speed in which you can potentially execute because of that. Is there another question? It's because you all need coffee, right? Between them and cookies. Maybe some parting words of advice and then wrap it up. Anyone want to jump in? I would just say from what we're finding is just keep muscling through. I think like the young lady said earlier, you're running into these barriers. I feel like a buzz saw. I love the Don Coyote picture that was up there. Every day feels like that as we're trying to get this sort of going, but it sounds like the ramp up after it takes off is just phenomenal. So we're really looking forward to that part of this. Yeah, I guess from a technical perspective, one, making sure that you have your pipeline set up, making sure that you can automate your implementations from the beginning were key from an agency perspective in terms of organization and people. Really what it came down to was the relationships between different groups of people and their ability to trust each other and to be able to represent their interests in any given situation. So those were I think the two key aspects along with obviously the leadership support. I would also say don't be afraid of integration. I think we found that integrating with existing tools and technology, storage systems, IDP, those kinds of things are massively simpler than I've ever seen before. So don't be afraid of that sort of integration with your existing tools and technologies. So I'll approach it from an EA perspective. Don't be a barrier to progress being an enabler and you will find that people will come with you on the journey. Thank you guys so much.