 So, hello and welcome everybody again to yet another OpenShift Commons briefing. This time I'm really excited actually to finally get to see and learn more about Grid Engine from VINIVA. It's been something we've been looking forward to having them present. Today we have a whole crew from the Grid Engine team. Rob Vellon, Cameron Greener, Ian Lung, and we're going to let Rob kick it off and tell us a little bit about what Grid Engine is and how we can make it work for scaling on OpenShift and Kubernetes. So, Rob why don't you take it away and we'll open it up for Q&A after the presentation. If you have questions post them in the chat and we'll try and answer them as we go and then we'll read off the questions that aren't answered at the very end and open it up for a conversation. So, go ahead Rob. Excellent. Thanks, Diane. Give me a thumbs up if everybody can hear me okay. And yeah, we'll keep going. So, I'm Rob Vellon. I'm General Manager of the NABOPS product line. NABOPS is the product line within the Univa company that is centered around containers and Grid Engine is our core product from the core business. We'll talk about and clarify how all that works. So, I'm on the line. We've got Ian Lum here, our chief architect, I almost gave him a promotion there and we've got Cameron Bruner on the phone and he's our chief architect with the NABOPS product line. So, we'll let the smart guys do the talking and I'll do a little bit of an introduction on the solutions. So, just a bit of background so you know where we've come from and a lot of the core Red Hat customers might know us already because we've got literally thousands of customers using our product between the open source solution and the commercial solution and 95% of those customers are running on Red Hat. So, we're in the HPC technical computing business. We've been around for many years. We acquired the Grid Engine product from Sun by Oracle so people might know the Sun Grid Engine name more so than they even know the Univa name because it's got a real rich legacy. We're most of us on the phone here in Toronto jumping in just to our customers just a little bit to give you a sense of the scale we deal with. We deal with some of the largest companies in the world, the largest oil and gas company in the world. Saudi Aramco uses us, we're dealing with some of the biggest clusters in the world as well as the biggest companies, right people running 5 million jobs a day, 11 of the top 12 life sciences companies use us and these are real big Grids for everything from doing fluid dynamics analysis to check airflow on F1 cars to designing BMWs to doing molecular modeling and genome sequencing, new oil and gas exploration in the financial sector doing a risk analysis and machine learning type applications, all kinds of things. But wherever people need to spread load across a very large grid of computers and really achieve super computing with that grid, we're the guys that often are doing the scheduling and the distributed resource management behind that. What I'd like to do is just turn it over to Ian for a few minutes to talk about the core product grid engine and then I'll take it back and we'll talk a little bit about how grid engine is being used within the container world within the NAB OPS line and how that applies to Kubernetes and OpenShift. So over to Ian. Thanks a lot Rob. So Rob's given you a nice introduction and some context for high performance computing and the core sort of flagship technology that as Rob has mentioned has a really rich open source legacy is our grid engine product and really what it's all about is trying to handle the supply demand equation in the very complex technical computing environments that he described or at least gave you some examples of. So basically what it's all about is trying to match what applications require, might be software licenses, might be specific hardware like a GPU for example with where those resources are available in the enterprise in some sense and doing all of that subject to a set of stated policies for allocating those resources and those policies are very important for a discussion today because they turn out to be a central to what we're now introducing with NAB OPS command but more on that in a moment. So what this means is that this allows organizations to really get the most out of their infrastructure and as you can see from this slide here we really are in some of the largest most substantial enterprises. We have customers that are literally running millions of jobs per day. Some of their clustered environments are on the orders of multiple hundreds of thousands of cores and in some cases for example some of those financial use cases that Rob alluded to we're talking about people running jobs in very very tight succession. So maybe we could go to the next one Rob and just waiting for this to refresh here. Here we go. So very quickly we support a variety of different types of workload. It's not just batch it's low latency workloads as well as big data workloads whether you're using Spark or perhaps some kind of a Hadoop component for example. As I mentioned we are able to handle all kinds of very specialized devices including GPUs and stay tuned for more on the Intel front coming very soon. And we are supporting a variety of different use cases that include not only shared memory parallel applications but also distributed memory parallel applications like MPI. And so in the next slide what we're going to show you very quickly is the three-tier architecture that applies to grid engine. So there are a variety of different interface points for the software as you might expect there's a command line interface but there are also a number of APIs as access points as well. Some of these are standards based actually they're all standards based really just different standards. One of them is a RESTful API and one of them uses a standard and distributed resource management. The brains of the operation is something we call our master host or our Q master host and this host serves a number of different purposes but this is where the policies actually get put into place as workload comes in in real time and needs to be scheduled out to the back end execution agents. So that's just a very quick overview so I'll hand it back to Rob now to kind of take this and frame how we're using this expertise in the NABOPS context. Absolutely so what we've done is we've taken our core products and a couple years ago we started moving them into the container world so we don't really talk about grid engine within NABOPS although as Diane aptly titled the presentation it's about teaching grid engine to speak Kubernetes and OpenShift but we refer to the product as command and command is the product that is doing this within the Kubernetes context. Let's take a step back for a second and talk about why you do all this. So for us it's about improving server utilization, optimizing cloud expenditures. You can see at the bottom there Gartner says that the average industry server utilization rate is at 12% that varies on time of day by company by type of application obviously but 12% is just a kind of astonishingly low number and when you start looking at how cloud utilization is growing it's growing at almost 20% a year according to IDC up to 2019 and at 2019 it's supposed to be almost 50% of an IT organization's budget is cloud expenditures so it's different when you've already got sunk costs and you've got machines sitting in your data center if those aren't being utilized it's kind of money that's been spent anyway it's not optimal but it is what it is but when it's cloud it's spending money every day and if you can be utilizing that those cloud resources more efficiently and turning them off when they're not used and sizing them right you're going to save a lot of money and when you're talking about 50% of your IT budget it can be pretty significant so you look at the top issues for Amazon their top issues were in their top 10 were things like oversized instances people were too many instances and instances running idle and that's really what command is about it's about writing things more efficiently on top of that our world is getting more complex so while containers container based applications CID all these great things have a real place and we're obviously moving that direction for good reason it does get a little more complex applications decompose from monolithic applications singular applications down to many many microservices and containers and those containers could be in replicas so that there's there's a scale issue there when it comes to how to manage all these things and there's also software defined layers as well so your networking and storage is now defined in software you've got new constituents involved people in DevOps introducing new processes and new software to manage the deployment and management of the applications and that's where our policy management and advanced scheduling comes in so we have two solutions in our container line in our nav ops line if you will there's launch which allows you to very easily provision a cluster whether that's on-premise or in the cloud or in hybrid but today we're talking about command and its relevance to to Kubernetes and to OpenShift so it's again it's all about you know sizing those machines properly about getting the right workload on to those machines dynamically provision them and the policy engine that drives that and I'm excited to get the Cameron's portion of the presentation because he's going to he's going to show us this stuff in action we've kind of talked about this interestingly in the brownfield in brownfield companies which most of us are in you have existing architectures you have existing software running and you want to blend that workload with your container workloads so you don't have to isolate those on different servers it's different for dot-coms you're building solid applications you're running kind of one monolithic application if you're if you're an uber I recognize that they have multiple applications within their suite but it's very different than major bank a or b that has hundreds of different applications and mainframes and Linux systems and all kinds of things so managing those workloads in context and not just looking at your microservice and container workloads is important too and that's what navbox command is partially about so what taking disparate workloads putting those on the right machine so that containers that need to run on the same machine for latency reasons can run on the same machine or on the same subnet and containers that shouldn't be on the same machine for security reasons or network or what have you can be separated and really packing those machines as tightly as possible to optimize the use and I'll let Lee and Ian talk a little bit more about the use cases I told the guys here that we had to a demonstration of the installation of our software and here we go and it's installed just kidding but I did want to talk about the Kubernetes architecture a little bit so on the right hand side you've got the worker nodes the kubelet and the proxy server those handle the communications to the worker and the distribution of work at the worker node level we've got the API server which is very important component that handles all the communications to all the other master components to STD and to the worker nodes there's a controller manager that's handling replication so as you want to replicate say an nginx database and have 10 of those running the controller manager will make sure those are all running and if you lose a node or you lose a pod that the controller manager will handle that replication and then there's the scheduler and that's where we come in we talk at the Kubernetes API level and we return off the Kubernetes schedule and replace it with our scheduler which is more than a schedule it's also a very significant robust policy engine that can handle matching the organization's requirements with how things are going to be distributed on the on the hardware so command as I said it talks to Kubernetes we're seeing a lot of open shift out there and they're very significant for us in our customer base we run on top of any Kubernetes distribution as you can see there and within the open shift context you can see a command slots in on top of the scheduler and on top of Kubernetes within open shift I'd like to turn over to Ian to talk a little bit about the use cases and how these fit in from a business perspective and then we'll hand it to Cameron for a demonstration thanks Rob I do want to emphasize that we didn't just you know come up with these use cases from our creative imaginations this is everything that we have here has really been derived from this wealth of experience we've had in addressing the workload management requirements of our enterprise customers and high-performance computing for really quite a number of years and that's why some of them might strike you as being a bit esoteric or unusual if you're not familiar with this kind of thing but suffice it to say we've really you know battle tested these in in other contexts so one of the you know one of the most relevant things that we can add I think to the game here is the ability to introduce more in the way of access control lists in some ways you can think of this as perhaps giving you a little bit more in the way of flexibility and perhaps helping to to close off some of the concerns that organizations might have with respect to the security of containers in terms of something completely different we have the idea of being able to make a reservation for pools of resources if you like at some time in the future that might be for a specific application perhaps for a specific purpose and this ensures that you've got a portion of the infrastructure carved out in a fairly logical sense to ensure that applications get the resources that they require as we're all alluded to before we have ways of fitting and placing containers subject to some kind of of an algorithm depending on what the objective is because there could be a number of different objectives it could be driving towards utilization on the fewest number of resources or it could be some other objective a lot of our customers in HPC have contention based use cases where you have for example perhaps different kinds of applications and we've given you some examples there from big data analytics perhaps competing with some sort of a database infrastructure this is extremely common you we also see a lot in the way of projects that need to compete for the same finite pool of resources and so the infrastructure that we've been building is going to be able to help you split those rather effectively another policy that seems to be extremely appealing is that every so often something extremely important comes along and it may be necessary to preempt already executing containers pods etc what have you to to allow for that more important workload to to receive the appropriate degree of resource allocation so we can handle that as well and then finally just to kind of give you a little bit of a sense another key component is that we recognize there could be a need that there be dependencies between different components of the infrastructure so that's something we have a lot of experience with in our our past and we're bringing that to this this new realm through nav ops command in the context of orchestration so just to you know further amplify what we're going to talk about momentarily in particularly the demo that Cameron's going to give will actually give you a sense as to how some of these use cases are actually starting to take shape in the case of our nav ops command products so let me hand it back over to Robert this point yeah thanks Ian we're actually going to hand to camera now okay so if you could camera I guess if you could grab the screen and take over that would be great in a minute to do that okay can you see my screen at this point absolutely okay great so yeah so what I'm going to be showing here is kind of the current state of our interface to nav ops command which is the scheduler the container scheduler solution that both Ian and Rob have been discussing so we're going to go through some of the use cases too that really that Ian had just brought up on that previous slide so the first place that we're going to go in here is to kind of look at our pod placement so really here with with this this is kind of a high level kind of general scheduler placement strategy allows you to say whether or not you would like to spread or pack your containers so try to use a minimum amount of nodes or try to spread it out to use more nodes but keep the load down on each node so there could be reasons why you would want to do that or you can go by maximum utilization where it'll just try to maximize your other policy parameters to make sure that you get the best utilization the most amount of pods running in your given system we also support the ability to mix for different types of workloads so for certain users or for certain projects you could say I want to spread for them or I'd like to pack for them so what you'll find as I go through here is a lot of the things that we're going to be talking about are more control over where the actual work runs and how it runs as well as the order in which the work runs so this one this screen here is really mostly talking about the actual placement of work the next the next section I'd like to go to here is the is our access restrictions and runtime quotas so this is a namespace centric kind of policy configuration dialogue and it allows you to go through your namespaces and set different policies for each for each namespace so right here this namespace one is a Kubernetes namespace you can consider it like a default and here if we go into its access restrictions and runtime quotas we can see a whole bunch of parameters that we can set so first you can grant to specific users in group access so who can go and actually use this quote this this namespace you can explicitly deny if you'd rather go go that route you can also decide which hosts are going to participate in in a given namespace and this could be because you want to limit certain hosts to certain users or certain projects based on special capabilities that those hosts may have we also have the ability here to limit the ability to run privilege containers in a given namespace or also disable hosts or allow host networking additionally we support quotas and these are different than the normal kubernetes quotas which you know if you're familiar with kubernetes quotas they prevent the actual admission of resources into the system which we'll still we still believe is very valuable with command what these quotas do inside of command though is additionally allow you to limit based on the current running conditions in the in the system so this would allow you to say that I don't ever want a certain number of certain users in this namespace to use more than a total of three gigabytes of RAM or limit the instead of by user to projects or even through application profiles which are kind of templates for describing how you would like to be able to run work inside of a given namespace we'll get a little bit more to application profiles later on in the system in this demo you can also limit to to specific hosts so you can say that there's different hosts with have different limits inside of your given quota inside of your namespace and this is kind of the simple way of configuring quotas we support a much more advanced robust way of adding quotas if you need to do something a little more sophisticated going on to our next bit of policy we support let me slide this over a little bit we also proportional share so this allows you to try to balance the distribution of work inside of your cluster based on you know projects and users and let's say the importance of a given project or users work or just groups work in in your environment and what this will let you do is allow you to say that this work is more important so when you run into resource constraints so let's say you don't have enough hosts or you only have 10 GPU hosts but you have 20 pods that want to use GPUs you can use this to decide which hosts should get which users should be able to get those first and after they use it for a while you know it will automatically let additional work that might want to use those those GPU resources that may not be as important but the fact that they were not able to run for a while they'll eventually get their fair share and be able to get a little bit of work in as well we support a arbitrary level of depth here and grouping both on project and by user another thing that we can support with command is interleaving this will allow you to say as you're scaling up a given application and here we're referencing application profiles so let's say you have your WordPress application and as it scales up you know you need to scale some other application as well so this will adjust the priorities of work such that when you do scale up your your application scale up your replicas for both of these that they get placed in a certain ratio as opposed to a whole bunch of one getting placed in another not getting placed and then you're running out of resources and having an inefficient imbalance of of application instances we can also do interleaving between different types of of projects and and resources here as well so and here is an additional way where we can go into interleaving based on the actual nodes and their capabilities so we can say between pods that are requesting these labels place one first on the SSD and then one on the label one once again it's more flexibility over you know how you want your work placed and where the things should go and this really helps you cover situations where you may have you know some container have nodes or a rack let's say go down and you need to replace those those pods that were running there and you're running close to maximum capacity these types of rules help make sure that when those when the work does get replaced the right work gets get started first and you you run in a in a in a proportion that actually allows your application to keep running successfully additionally we allow for ranking of resources and this will allow you to prioritize work based solely on the types of resources that are being requested now this will work additionally with our other ranking policies but by doing this you'll be able to say I want to make sure that my GPU jobs schedule first when when we're looking at a resource constrained situation additionally by saying here mem as being a higher one that will allow you to schedule work that requires larger amounts of memory before smaller amounts of memory which that's in general a very good kind of strategy as it allows you to get better utilization of your resources if you've you know if you scatter around a whole bunch of small requesting jobs you could very well have on any given host not enough memory in order to service a very large requesting container and this goes kind of to the point that Rob was making about you know low server efficiency you know that 12 server utilization you know putting in policies like this allow you to achieve better utilization while still maintaining the ability to run the applications that you want to be able to run when you want to be able to run them so the next section I have here is application profile ranking and this is where you can define different types of classes of jobs and say which ones are more important so just this sheer fact that a workload matches one of these allows it to get a slightly higher priority in the system and be able to be scheduled first over workloads of another one since we've talked a bit about application profiles at this point I'll jump down there to that section give you kind of like an idea of what these are about so as you can see here they kind of are roll-ups of a whole bunch of attributes so you could think of this as being a grouping so pretty much saying like with WordPress in this example only publishers and the admin users can actually go and use this and the dev users cannot that means that if we attach this WordPress profile to any of our other resources only these users will be able to use it it will also make sure that certain things are met when when they're set so this would pretty much by having this SSD host tag here it would make sure that if somebody tried to submit a pod that was to the WordPress profile that it would no longer it would have to have this SSD host pretty much the admin saying that all WordPress applications need to run on machines with SSD and it'll make sure that that happens additionally we can have other labels that can be there that we would prefer to have when scheduling the pod but don't need to be there so this would be you would want to be on an SSD host it would have to have that but you would prefer one that had a already mounted good scratch file system but would be able to tolerate being placed on a node that does not have that additionally the profiles can be set to direct projects so that you can actually have your projects set automatically and have and be tied into those types of ACLs and quotas as well so we don't have much in the UI currently for configuring the projects but generally they're just the creation of a name and then you you are able to attach user objects so those would be either users or user groups which are also very much like projects which would just user groups are groups that hold a set of users or other user groups so I hope that this kind of overview gives you an idea of what we're thinking and what we're going at when it comes to NavOps command and the types of situations that we really want to address you know in summary it's making sure that the most important work is the work that runs make sure that you know you're you get a good utilization out of your infrastructure and really you bring you know the these these lessons that we've learned over many years of supporting our high performance computing environments where utilization is very important and and time and the placement of the work is of the of also of most importance with that I would like to return it back to Rob to wrap up the the rest of this presentation thanks Cameron that was excellent unfortunately we went extremely fast through all of our materials so we're we're finished well ahead of where we wanted but hopefully people kind of got the gist of what it is we're we're doing with command and how it fits in we didn't spend a lot of time talking about it in the context of of open shift but I think it's it's logical how this would sit up above open shift at the scheduling level and make sure that the production workloads are running in the way that you've designed them to run using our policies this product is just reaching early access availability as we said it's based on the very mature grid engine but we've containerized it we've put a web UI on it it does have a kube control type of command line interface as well that'll be available and you could even get real dirty and drop down to the grid engine interface if you wanted to and that exposes a lot more capabilities but I think what you've seen through the UI is is extremely rich and far richer than what's available for for scheduling now and we're real excited about it so it's going to be available right after docker con and we'll we'll send out an announcement of as to when people can download it and try it out if you're running open shift we'd love to work with you on showing you how it works how it all works and if you're at docker con certainly we welcome you to come by I think we're at s22 booth and we'd love to show you the product in action in a little more detail so with that I thank you and I'll turn it over to Diane I don't know if we've got time for q&a or have any questions so thanks a lot both Rob and Cameron too because it's really interesting to see a different perspective on the policy management aspects of kubernetes and open shift and taking it to that hpc world and the worldview of that people need in order to manage and orchestrate those workloads and so basically from from what I can tell um that you're making open shift and kubernetes hpc ready is that a correct statement I wouldn't say it's hpc ready I think what I think the way I would look at it is we're bringing those policy management and capabilities to open shift so people wanted to orchestrate microservice based applications within open shift in a manner like they did with hpc which just means far richer policy management I'd say that that's the approach we're taking so I think hpc people will still use the distributed resource management system and not necessarily use kubernetes they they could do it this way but I don't know that that's the approach that most people will take so I guess long story short is no we're we're doing policy management and scheduling for microservice based applications maybe I should have made that a little more clear at the top at the top of the hour okay so you're bringing to bear all of your experience with hpc and large-scale stuff to kubernetes workloads and and thus the open shift yeah so I think I think I got that I was just trying to figure out how to frame it in almost a tweetable capable you know how can all of what Cameron just said in 140 characters or less so no it was really good because it's it's one of those things is grid engine has been around for a while it's got a huge customer base and a huge user base and lots of good good folks doing very very interesting lots of red hat customers doing very interesting work with it so I'm really going to be interested to see how taking all of that experience and bringing it to microservices can help out some of the workloads that we have running on open shift now today and we're going to have to have you back once you launch again and see if we can't get a cluster access for you to do a demo against and compare and contrast doing it with shall I say raw open shift versus using the command piece I think that will be very an interesting comparison and I'm also going to be interested to see once people get their hands on the grid engine command nav ops way of doing things if they won't try and drive some of that advanced policy management back into the kubernetes open source project yeah that's that's definitely a possibility you know I think there's all kinds of engineering investment going on in kubernetes and the scheduler is one place where you know there could be a lot more work done so it could could start to move up closer to to what we're doing they've added batch workloads that's not really scheduler specific but they've added some batch capabilities and you know 1.2 I think it is and you know there's more and more happening there so you're right it'll start getting richer and richer over time and we'd love to we'd love to show the product on open shift next time around we'd also love to talk about some real life use cases as well by then yeah and if I could just add in Ian here again Diane just wanted to add in that you know there are there are many use cases that we'll be able to support but certainly our experience from high-performance computing suggests that there are some some policies the implementation of some policies that are extremely broadly implemented across the entire customer base almost regardless of which which vertical market is represented and that that really relates to some of the the share-based policies that we talked about on the use cases slide and the Cameron demonstrated so I could see that as being something people might want to provide some kind of an implementation for you know perhaps in the the open source implementation right I think we're a long way from that so and there's so many other pieces to Kubernetes I'm just like I think that what you what you're probably going to give people is the way to do it now and maybe they'll drive some of that back into Kubernetes the project itself or into open shift someday but I really think there's so many other pieces that people are working on that you've solved a good good chunk of the scheduling issue so this is a great opportunity for people to take advantage of your experience and for grid engine to to shine in a in a new world in a new world order right so thank you again thanks again day morning me today I will be back on next week and we will see you at Dr. Khan and I know we're going to see you at Red Hat Summit probably too so there'll be lots of time to to check it out and open shift online today just went live for the dev preview so maybe we can even get some demos going there but though I think you need your own cluster to install rather than using our dev preview so we'll we'll get we'll get you access to a cluster pretty quick here awesome thanks everyone all right thank you all