 I'm going to bring the gentleman from Cisco up. We're going to get started. And I'm really pleased to have them come and present. They presented before at other OpenShift gatherings. But this time it's a little special because they're my wonderful case study that brings together some of the CoreOS wonderfulness and the OpenShift wonderfulness, and they're going to tell you all about it. So thank you again for coming. Thanks, Diane. All right, are our mics working? You can hear us? Great. See if I can make them go forward. So since we're speed dating, I'll make the introductions quick. My name is Mike White. I'm an architect at Cisco. We work in the IT group called Enterprise Platform Services, where we deliver platforms and middleware technologies for all of the developers at Cisco. And with me today is Nara Tatapalli. He's a senior engineer, tech lead, and subject matter expert on our Enterprise Container Hub, which we deploy using CoreOS now, Red Hat's Quay Registry Service. So that's what we're going to talk about primarily today. But let me set some context first. I think I'll answer all Diane's questions as I go through this slide. Cisco's been on a multi-year journey. And we've called it different things over the years. We've called it our global data center strategy, our cloud-native transformation. But what we're really trying to do is deliver services to our developers in a programmable way. So back in 2015, when Docker and Kubernetes were getting popular and OpenShift rolled out OpenShift 3 that included those technologies, we started evaluating technologies and seeing if we could do container as a service. So go beyond just platform as a service. Could we run whatever technologies the developers wanted on a common platform? So in 2016 is when we really got into the initial delivery of some of these services. And while we've shared a lot of our OpenShift stories with you over the years at these forums, one of the things we didn't highlight was our Enterprise Container Registry story. We actually deployed Quay prior to OpenShift in 2016. We saw that developers were going to really pick up this container thing at that time. And we felt that there was a need to centralize that function within Cisco and have a good place to maintain all of the images that were being created. Last year at this forum, we shared with you the fact that we were in production with OpenShift 3, we had a lot of large customers deploying in a cloud-native fashion on this platform across multiple data centers. And as we go into the future, we're building out OpenShift on top of OpenStack and looking forward to a lot of the tectonic administrative features that will be brought into the product going forward. So with that, I'll turn it over to Nara and he'll share with you some of the considerations we had when looking at a container registry. Thanks, Mike. Hi, everyone. So when we started to decide that we will deploy our Enterprise Container Registry at Cisco, we came up with different requirements. Some of the requirements are highlighted here. Main requirements were it should be on-premise because of the security reasons. We don't want our images to go to outside public cloud. So it should be more secure and it should be deployed on our data centers. It should be high available. We wanted to deploy this Enterprise Container Registry in different data centers. So as of now at Cisco, we deployed at three data centers. And also, the images getting pushed to one data center should be automatically replicated to the other two data centers. Even though we give our developers full access, whatever they can do, we wanted to restrict where the images they can pull and push to. So as of now at Cisco, we allow the developers only to pull and push the images from our Enterprise Container Registry. It should also have a good integration with LDAP. So when our image is pushed to Enterprise Container Registry, Cisco is very big with security. So an image should be automatically scanned for vulnerabilities and report back any critical vulnerabilities we have at that image. And also, it should have good UI visibility and it should have good vendor support. So keeping those requirements in mind, we evaluated different products. Some of the products we evaluated were Docker Trusted Registry, OpenShifted Internal Registry, Artifactory, and Quay. So Docker Trusted Registry was just evolving at that time and didn't have all the features which we wanted from our previous requirements. So we didn't go with Docker Trusted Registry. OpenShifted Internal Registry was just for internal Red Hat and it was not useful for us because we have other products working for this. Artifactory, as of now at Cisco, we use Artifactory for our CI-CD stuff, but it didn't have the required UI features which we were looking for and also there was no vulnerability scanning stuff. Keeping all these requirements in mind, only Quay was fitting our requirement, so we went up and deployed Quay in our Cisco data centers. So as of now, this is our current deployment of Quay. So we deployed this Quay in three data centers. Data center one being the primary, data center two and data center three are the secondary data centers. Each data center has a load balancer in front of it. This data center can be individually accessed with the load balancer URL. All these three load balancers connect to a global site load balancer. Data center one being the primary. We deployed many components in it. We have Quay, we have build servers, we have Clare for security scanning. We are using Postgres for database and we are using Ceph for storage. As of now, the database is only active and passive. The primary database is in data center one. It's a manual failover. If the database goes bad in data center one, it's a manual failover to the other data center. So these Quay images are deployed on individual VMs, so they're not deployed on OpenShift as of now. After we deployed Quay, we had a lot of success stories. So these are few of the success stories we wanted to share with you. This is from our side, from the admin side. We had good support on architectural guidance. So when we wanted to deploy Quay on different multi-data centers, we got good support from the Quay team. And whenever we had some issues with bugs or anything, we can directly send out an email to Quay and they are very quick to fix the bugs. So whenever the developers, whenever they have new requirements, they will just ask us for these new requirements and we talk to the Quay team and they're very responsive to give the new required features. Can I jump in here, Nara? These first three bullets are really important to us. We've been really, really pleased with the relationship with Quay and CoreOS so far and I'm sure it'll continue with Red Hat. We've got a good partnership here, but being able to give you guidance on how to set this up, how to scale it, the response time has been really impressive. So those first three bullets are super important to us. So I think at Cisco, we upgrade Quay every two, three weeks because they give so many updates and the regular frequent upgrades. So we upgrade Quay every two, three weeks at Cisco and the vulnerability scanning is very good. So whenever we push an image, it tells us where the image is vulnerable at, which layer of the image is vulnerable and also it tells if there is a fix available for that image, for that vulnerability. And we have, we are using this API. So Quay comes with a lot of APIs in our automation. We have this CI CD pipeline automation stuff going on. So in this, we use these APIs to automate most of our stuff. So this is from the developer perspective. As of now in Cisco, we have more than 5,000 users on our Enterprise Container Hub. 420 organizations are being created and we have total close to 10,000 repository and each repository will have multiple tags. As of now, the volume at Cisco is around 12 terabytes. So we have around 12 terabytes of images deployed on Enterprise Container Hub. Initially, we thought the build system was not that popular. We didn't want to go with this build system because Quay comes with integrated build system. We use our own Jenkins stuff to build images and push to Quay, I mean push to ECH. But surprisingly, all of our developers, they just love the build system. So they just get the Docker file or they just get the Gita CM URL and they just deploy to ECH directly. I think the success there comes from the ease of use, right? I didn't think it was gonna be all that useful. I'm always on the command line doing Docker build, Docker tag, Docker push. And I think this just gives developers a really good way to start simply and get going. And so that's been a pleasant surprise. And also, we've got good feedback on UI. Even though if you're not familiar with any of the container registry, you can just log into the Quay UI and it's very useful. It tells you where the tags are, how to do stuff, and all the things are very easily done on the UI side. And also, we use a lot of notification by Book of Post. So whenever you have pushed an image or something to it, it can give, with the notification thing, it can give you, if that image has any vulnerabilities, you can get an email or a Slack chat message or anything from that. All right, my turn again. So putting this back into larger context again, what you see here is that we're trying to provide a programmable lifecycle for our application developers from end to end. I shared this slide last year, but I brought it back again so that we could talk about the middle bar there where we've got our Enterprise Container Hub. But our developers at Cisco are able to establish their tenancy, if you will, through a programmable API, provision their OpenShift projects, and also manage their continuous integration and continuous delivery pipelines, all programmatically in a cloud-native fashion. So Quay and our Enterprise Container Hub sit right there in the middle. Once you're provisioned and ready to start developing, you gotta start building containers, right? You gotta have a place to store them. And Quay is our spot for doing that. So let me dive down into that just in a little bit more detail. One of the things we're trying to accomplish with this programmable cloud-native model is the speed delivery of applications and functions to production. So in the old days, we had a lot of bureaucracy and process and gates in the way of developers deploying their code. And we're trying to stay out of that production flow from dev through stage and into production, but we're trying to maintain our security and our standards by giving visibility back with frequent reporting and auditing. We can catch vulnerabilities and get them remediated much more quickly. So pretty standard flow here. Developers are checking their source code in to get using Jenkins to build the container image itself and compile the code. And at that point in time, we're running different application scanning tools to determine whether the code is of good quality and has all its test cases, et cetera. Once the app image is checked into the Enterprise Container Registry, Quay does the clear image scan on it. And at that point in time, almost immediately after checking your image into the Container Registry, you can see if it has any vulnerabilities. And I don't know how many of you have seen the UI, it's very detailed and even provides which libraries are vulnerable and which versions you could upgrade to in order to remediate the vulnerability. It ranks them high, medium and low and it's really powerful UI for helping the developer manage the libraries inside of their containers. Now, as Nara mentioned, our Enterprise Container Registry is the only pipeline into our OpenShift platform. You can't pull direct from Docker Hub. You can't deploy from your local laptop. You've got to pull from the Enterprise Container Registry and run it in OpenShift. Now, we don't have any blocks between there. So you could technically go to Docker Hub, pull down an image and push it right in here and push it into production, which is a little bit scary, but we can audit frequently and report back and find out what's going on in those images and get the fixes in. About six months ago, I don't know if you guys remember, there was a Struts vulnerability and our InfoSec department was really concerned about it. And they wanted to know, do we have that vulnerability running in any production systems? And for us in OpenShift, it was relatively easy to answer that question. We went into OpenShift, got all pods, got the SHA ID of all the images that were running, and through those APIs that Quay provides, we were able to interrogate the vulnerability database and find out if that vulnerability existed in any of our containers and then take it back to the developers to remediate. Luckily, we didn't find it, but it was cool that we could pull the running images out of the system with Kubernetes APIs and then use Quay's APIs to check and see if that particular vulnerability existed. So as I mentioned earlier, as of now, Quay is running only on individual VM hosts. We wanted to deploy this Quay on OpenShift. There are many advantages of deploying Quay on OpenShift. As of now, we are maintaining, if you want to, depending on the load, if you want to increase our Quay running containers, we have to individually extend the VMs, deploy Docker on it, and run Quay on top of them. If you deployed Quay on OpenShift, we can just scale up and scale down, depending on our needs. So our future goal is to run Quay on OpenShift. And also, we wanted more tighter integration with OpenShift. As of now, we don't know what tags are running in OpenShift, because OpenShift is not only the platform which is used as of now at Cisco. We use different platforms to run images. So we wanted to know if we have... So we have so many tags in ECH, in our Enterprise Container Hub. We wanted to know if these tags are part of OpenShift or not. This will give us more idea on audit stuff, like if you have any vulnerability running in OpenShift, it can show up on Quay. So we wanted to have more tighter integration with OpenShift. And also, we need better admin visibility features. So as of now, if I'm a super admin in Quay, I don't have access to all of the repositories. I know it's a security thing, but if teams come and say that, okay, this tag is not visible, or this tag is not working properly for us, we have to manually go and take the ownership of that repository and do all that stuff. We don't want to do that for thousands and thousands of repository. So I think we need better visibility, better admin visibility features for this one. And also, as of now, as I told you, that we deployed Quay only in three data centers. So we are getting lots of complaints that people pulling from remote locations, like in Europe or somewhere, they're not able to pull and push the images that fast. So we want to deploy Quay in more data centers now. That's the long-term goal for us. Nara, before we move on this one, you say we want to run Quay on OpenShift. Don't we run one of the components on OpenShift already? Yeah, so we started moving one component now, components by components. As of now, we are running Clare on OpenShift. We're running like 80 parts of Clare on OpenShift because we pushed like 10,000 images now. So we wanted to increase that Clare to 80 parts. So as of now, we are running 18 stints of Clare on OpenShift. Yeah, that was a real easy scale-up operation, as you might expect, running on top of OpenShift. I think I was the one that complained when my images failed to get scanned and the guys just went and scaled it up. So I think there's a lot of potential there as we go forward to run more and more of the components of the container registry on top of OpenShift as well. So I think that might be the end of our slides. We've got just a minute or two left if there are any questions. If you have a question, just raise your hand. We can get the mic, all right. Hi, so I know that you're using Quay to scan for vulnerabilities. How do you measure the effectiveness of that scanning? How do you know that the vulnerabilities that you're scanning are catching everything? You know, I'm not a vulnerability expert. I'm hoping that we can trust Quay. I know that they are plugged into all of the different CVE databases. And anytime that our InfoSec has approached us about a particular vulnerability, that's already been updated in the registry. The other thing we didn't mention is that Quay's, it doesn't continuously scan, but it continually updates. So even after you've deployed and passed the first vulnerability scan, if a new vulnerability is brought into the database, it'll update that flag and send that notification that NARA was talking about. So while I can't attest and certify to the completeness of the vulnerability database, we've been very satisfied that we get the Info quickly. Any other questions? I think this is an important point too. We often get asked on the product management team, like can you talk to our security teams? Can you make them feel good about putting the system into production? Because it's new stuff, right? Containers and Kubernetes and OpenShift. What they start to realize is when you're running a fully immutable environment, you're managing all your deployed apps through these images, you're doing scanning and have all this automation, it might actually be more secure than what you're doing today, right? And it's a real eye-opener for the security teams. We've seen it over and over again on deals. And so again, if you need help having those conversations, we're happy to do that. And we have great stories like this to emphasize. So I guess if there's no more questions, we can, oh. Oh, there. We're gonna really make Joe run this time. Has there been any thought put into running this in like a disconnected environment or how you'd go about actually pulling in updates to CVEs and things like that into this system? The CVE updates are automatically pulled in. That's part of the Clare product. Right. So if you're in a disconnected environment where you don't have access to the internet. Oh, when you don't, I disconnected. I'm sorry. Yeah, sorry. You know, I'd ask the Quay guys if you can find them here this week, ask them that question. That's a good one that we haven't had a particular experience with. I bet they have a solution, though. I bet they should. Yeah, I think, yeah, we'll have to. That's not me, unfortunately, but we can follow up on that. All right. Well, thank you very much. These guys, this is your speed dating moment. Check these guys out. Thank you. Thank you very much. Thanks, guys.