 Are we using this last one? Yeah. Okay. Does this have an HDMI pole? Yeah, I need HDMI. I have it. Okay. Okay. Say something. Am I audible? I don't think it's coming out of my... Try again. Yeah. You just gotta project. Talk a little bit louder. Louder. Talk louder? Yeah. There you go. Okay. Let's flip it off and tell you where it needs to go. Yeah. There you go. Hello, everyone. If you're just joining us, welcome to the Collaboratory Track. This is a track of interns mostly from BU who have been working with us at Red Hat all summer, doing lots of interesting stuff and also Northeastern, Peter reminds me. Our next talk is on navigating the Kubernetes job framework or as they have apparently retitled it, using OpenShift to accelerate Chris development. This is a really exciting project we've been working on aspects of this project for about almost a year now, maybe even a bit more. And these folks are gonna talk specifically about what they had to do to get Kubernetes to do what they wanted it to do. So please welcome Ravi, Parul and Urvashi. Thank you all. Yes, I am. Okay. So hello, everyone. My name is Ravi. I work as a software engineer at Red Hat. Today I'm with Parul and Urvashi and we're going to talk about how we are going to accelerate Chris development using OpenShift. So how many of you are here for OpenShift? Okay. How many of you are here for Chris or some sort of medical image processing and radiology? Okay. All right. So let's get started with Chris. So Chris is a medical image processing platform built at Boston Children's Hospital and it consists of multiple open source projects. It consists of multiple open source projects. The problems that Chris is trying to solve are the first thing, medical image processing relatively happens in an isolated fashion. So say if I develop some image processing plugin and someone else wants to use it, it's kind of difficult because there is no community around the open source medical image processing tools or software. So Chris wants to be the store where if you write some medical image processing plugin, we could store it in that store and then it could be downloaded and used by other people. At the same time, Chris wants to abstract out the infrastructure from a developer perspective so that if I'm a medical image processing developer, I don't have to care about where Chris is running or where, how my software is scaling, et cetera. So why is Red Hat interested in Chris? The first thing is we want Chris to be using Red Hat Stack. That's the number one interest for us. The second thing is we want VCH to help build a community around Chris so that the innovation could happen at a rapid pace. Now let's look at the typical workflow how it happens using Chris. So say if there is an input image that our plugin needs to work on and Chris pulls that image and then pushes it to the compute environment where processing needs to happen. And Chris talks to store. The store is a place where the plugins are living, meaning every software, medical image processing software that you write should be registered in store so that Chris could query the store, find out the list of plugins and then instantiate one of them in the compute environment so that it could work on the input image and generate some output. The output generated would be copied back into Chris or pulled back into Chris and depending on the configuration, it could be stored in the data store again. So as we can see, Chris manages a single instance of Chris could manage multiple compute environments and it could act upon multiple input images as well as multiple plugins running in multiple compute environments. The thing that we are focusing on is a compute environment here managed by OpenShift and OpenStack. So if we want Chris to be successful, we want to have many people or many developers coming into Chris and then developing plugins on top of it. So how are you going to achieve that? If I'm a plugin developer, I don't want to think about the infrastructure where Chris is hosted or how Chris is running. At the same time, say if I'm a plugin developer and I want to see my plugin running at scale, there should be a facility and Chris provides that facility to run the plugins at scale. There is also another advantage, say some of the plugins as of now take from hours to days for image processing. We can actually cut them down to minutes or even seconds making them clinically relevant if they are paralyzable. So the scale, talking about the scale, we actually get the scale from Mass OpenCloud. It's a collaborative effort between five different research universities, government and other industry partners like Red Hat. Like Amazon, Microsoft or Google, it's an open ecosystem of cloud services and it's operating over a 15-megawatt data center. It's about the size of a Walmart. It's green data center. Around 70% of the power is coming from renewable energy and it's backed up by OpenStack. So why do we need OpenShift in the mix now or Kubernetes? So some of the components of Chris could be deployed and scaled easily if they are running on OpenShift. That's one advantage. And the second advantage is the job framework that Kubernetes provides. We can actually parallel some of the plugins using the job framework. And I'm going to talk about that in a bit. Let's look at an image processing plugin that acts upon the image that you see on the slide. If there is no parallelization involved, all the entire image is going to be processed. But if we can break down the image into multiple quadrants with each part of the image being worked on by some worker, we can actually reduce the computation time for the plugin. So we use Kubernetes job framework to create those worker parts which would take some part of the image and then process them at the end of the processing. We can find out that the execution time for an individual plugin is going to be reduced by some fraction. Ideally, we hope it to be reducing to one-fourth of it. But depending on the plugin and the way, how much of it is parallelizable, we may see that the entire plugin, depending on the software, it may not be parallelizable. Now this is not only advantage for the plugin developers, it is also advantageous for mass open cloud. So we use the concept of QS tiers which are available in Kubernetes to use the Slack resources on mass open cloud. So some of the resources in mass open cloud could be reserved, some of them are free, some of them are consumed, and some of them could be Slack consumed, meaning in the previous slide I was talking about an image that is broken and worked on by four different workers. We can actually increase the number of workers in the plugin and ask the Chris platform to run from four to ten. Like we would start at four if there are some Slack consumed resources, we can actually go till ten or whatever the number that we give it to be. So this actually ensures the utilization is also up in the cloud. And I would like to hand it over to Parul who is going to talk about the architecture and how we have brought Chris onto OpenShift. Hello everyone, my name is Parul Singh, I'm a graduate student at Northeastern and I'm currently a software engineer in town at Red Hat. We have used Kubernetes job framework to do most of the scheduling of how the workers are going to handle the computation. But before we dive more into it, let's understand the architecture of Chris. This is the current architecture of Chris. The front end is deployed at BCH which is Boston Children's Hospital. We have deployed two other components of Chris, the process manager and the IO handler on OpenShift at MOC. The reason for this segregation is we want to run jobs at scale. So the process manager and the IO handler need to interact with OpenShift, that's why they're deployed on MOC. This is Chris would send the medical data on which the image processing has to be applied by the IO handler. And the IO handler puts that on Swift, which is the open object store. The operations that need to be applied on the data is sent to the process manager. The process manager translates operations into jobs. And the job is run inside a pod and this is how the pod would look like. In Kubernetes land, a pod can have multiple containers running inside it. There are two kinds of containers. One can be an app container and the other is init containers. The difference between app containers and init container is that the app container is the first container to start and it always executes to completion. Only when it's done, the app containers begin to start. So in our project, we needed the medical data to be available to the image plugin. So for that, we have the init container download the data from the Swift and put it on the shared volume so that it can be accessible across multiple nodes. Once that is done, the image plugin takes the data from shared volume, apply whatever algorithm is necessary, and store the results back on the shared volume. The published container would take the results from the shared volume and put it back on Swift so that it is available to the analyst. Our end goal is to run job at scale. We don't want a single pod. We want it to be scaled to multiple pods. But say if you have scaled to a million pods, all the million of init containers would be downloading data from Swift and putting on the shared volume and one that is done, the plugin container and the published container which are the app container will start. How would the published container know that the image plugins are done and it's time to upload the data to Swift? There's no way of telling that. To handle that problem, we are using Kubernetes watch API and we let all the published containers watch all the image plugging across all the pods and once all of them are done, then only the published container collects the result from the shared volume and put it on Swift. But we can see that there's a lot of redundancy going on here. You have a million of init container downloading the same data and a million of published container waiting in an idle state for the image plugin to be done. And then all of them would be putting the data on the Swift. We don't want that. But since Kubernetes does not has an implicit order assigned to the worker, we had to tailor a solution using the file locking. So obviously, this is not a good design scheme. What we did is, let the init container of whichever pod start first, get the data from Swift and put it on the shared volume. This pod also writes a file on the shared volume to communicate to the rest of the job pods that, yeah, the file has been downloaded. You don't need to re-download again. It also writes a file within the pod to say that I'm the leader. The next init containers in the pods that start, they just refer to the file on the shared volume. And if it's downloaded, they exit. The published container sees am I part of the leader pod? If not, they exit. And the published container of the leader pod would be only monitoring or watching the image plugins. And when all of them are done, it would put the data back on the Swift. So we were able to achieve the architecture. We will have a single init container and a single published container uploading and downloading data. While the rest of the job pods will only have the image plugin running inside them. Kubernetes job framework seems to be a very apt framework for workflows like ours. It has the capacity to run jobs at scale by creating one or more pods. It can also, it also ensures that a specified number of it always completes. And it handles job and container failure by the restart policy mentioned in the template. But this, yeah, so there are two kind of restart policy. If it sets to on failure and a container inside a pod fails, then the pod stays on the node, only the container is restarted. But if it sets to never, then in both the case of the pod failure or the container failure, the entire pod is started. But neither of them works for us. And the reason is, Kubernetes inherently tries to run the job to completion. Say an application developer made a code error inside the plugin. In that scenario, we don't want Kubernetes to re-attempt to create, rerun the pods again and again. We just want it to exit. But not all plugins can handle restart. Also, the work done by the plugins can be mutually inclusive. Like, a one plugin doesn't know what the other plugin is doing or a plugin needs to interact with each other to communicate and gather the results. So the results from the previous runs like the locks or the temporary files or the output has to be handled outside the Kubernetes job framework. Now, I would be talking about the cleanup. Let's say the previous case I was talking about when the jobs fails. But what if all of the jobs completed successfully? You had million of job and all of them runs to completion. What happens to them? Well, they stay when the job is completed, the nomopods are created. But the job pods are not deleted either. It's good to have them in the system because you can always go back and see the results to get the diagnostic output. You can see the warnings or errors. But they stay in the system forever and consuming the resources. So what we had to come with a custom solution, which is called the Reaper module. What a Reaper does, it monitors the system and look for job pods who have completed and age beyond a certain time. In our case, we have said that time to a day. And once it find pods who are a day old, it just goes and kill them. In this way, we are releasing all the resources used by the jobs, like pods or volumes or share directory. So far in a nutshell, I would like to say that Kubernetes job framework is really powerful. It does most of the thing, but they have certain aspects that's lacking. One of them is the job termination and the cleanup. We want it to clean after itself. Once the job is done, we want the resources to be free. And it would also be good to have workers and orders pass to all the pods. Otherwise, people might need to declare the leader and they have to go through all the hassle of the file locking, which is not good. And it also would help to divide the work among the worker pods. My talk was only centered around the Kubernetes job framework. But we have used other aspects of Kubernetes, which Rishi is going to talk about. Hello, okay. Hi, so I'm Rishi Manani, and I am a software engineer at Red Hat, working on the OpenShift Runtime team. Now I'm going to talk about two features of Kubernetes that we leverage in the CRS project. So, CRS requires authorization and authentication to perform some of the actions that are vital to its functioning. These actions are, one, accessing confidential medical images on the script store, and two, interacting with the Kubernetes API to perform tasks such as job and PVC creation. The way we give CRS these privileges is by using secrets. So, what are secrets? Secrets are a Kubernetes object that holds sensitive information, without directly exposing said information. Pods can then utilize these secrets by mounting them as volumes. So this way, the processes running inside the pods can access the privileged information they need and get the necessary privileges they need. Sorry, yeah. So, as you can see here, for CRS, for the two actions I mentioned above, we need two secrets. One of them is the Swift Credential Secret so that the Iohander can talk to the Swift Store. The second is the Cube Config Secret so that the process manager can communicate with the Kubernetes API. When you create a secret in the Kubernetes API, the secret is stored in a Base64 encoded form. That's, as you can see in the second image here. Base64 encoding is not a form of encryption. However, the Kubernetes has encryption algorithms at the rest layer level. So users can go ahead and use some of these algorithms to encrypt the secret data before storing them in XED, which is the Kubernetes data store. The next feature I'm going to talk about is shared volumes. So, one of the factors that add to Chris's scalability is the fact that we don't have redundant downloads and uploads of medical images, as Parvall mentioned during her talk in her slides. So, since we have multiple pods and containers that are working on the same image, we need, the image needs to be saved in a location, in a common location that can be accessed by these pods. And the way we are doing, meeting these needs is by using shared volumes. So, for Chris, the kind of shared volume we're using is persistent volumes. Persistent volume is storage on the cluster that is provisioned by either the admin cluster or it can be dynamically provisioned. A given persistent volume can then be claimed by a user in the form of a persistent volume claim. Users can specify the amount of storage they require, as well as what kind of access modes they want, such as read write once, read write many, read only. The great thing about persistent volumes is that the content on it can be shared across multiple nodes, since the multiple job pods that Chris run are not necessarily guaranteed to be on the same physical node. Another type of storage we have is called Booster, which is a volume that can be shared between multiple pods, but only on a single host. So, this way, that doesn't meet Chris's scalability as we needed to be on multiple nodes. But it is actually very helpful for local development. Another kind of storage is the empty door. So, empty door, you can use that to share contents amongst multiple containers, but only in a single pod. This is actually very helpful in the Chris project for the leader election mechanism that I will mention before. Since the multiple containers in there need to access, need to look at a file to know if they're the leader and if they're the leader, they stay, otherwise they just die out. So, tying back to the architectural decisions, we have used Kubernetes concepts such as jobs, PVCs and secrets to face the challenges posed to us by the Chris project. Going forward from this, we hope to have many more plugins for the hospitals to use so that they can better their diagnostic processes. And one way we plan on doing this is by hosting more Chris centric hackathons where we can both raise the awareness of the Chris project as well as further the development of image processing plugins. Another goal we're working towards is to enhance the collaboration between hospitals so that they can share anonymized data sets together using Chris. Finally, we are glad to be, we're glad that the Chris project gave us the opportunity to solve real world problems the open source way. Thank you. Talking to anybody in the Kubernetes project to add those requirements that we need. Are we? Okay. As of now? Just gonna say. As of now, I think there is a project which is not even in Kubernetes incubator but developed by MADS which is kind of similar to what we are doing, but it's not yet available in the Kubernetes community as such. Thanks very much, you all.