 Hello and welcome everybody to yet another OpenShift Commons Big Data SIG. Today, we're going to talk about Apache Spark running natively on Kubernetes on OpenShift. Eric Erlinson has been doing a lot of work on this behind the scenes, so we're real happy to have him to deliver this talk today. If you are going to be at KUKON next week or at the OpenShift Commons gathering, he'll be there as well. If you haven't heard about the OpenShift Commons gathering, time is coming up soon. It's next week, November 7th, the day before KUKON starts, which is November 8th and 9th. So can we in the chat and I'll make sure you have the details on how to get a hold of us and some connections to get there. And so I'm going to, without further ado, let Eric start. The format for this whole session, it's a SIG session, is that we're going to let Eric give his talk. You can ask questions in the chat, and then I'll open up the mics after the talk, and we can have a conversation. So without further ado, Eric, take it away. Thanks, Diane. Good morning. My name is Eric Erlinson, and I'll be talking about running Spark natively on Kubernetes and OpenShift. I'm a software engineer with the Emerging Technologies Group, and I'm working on the Dicon project of which this work is a component. And here's the various email and Twitter handles you can reach me out here. So I'm going to begin with some very brief review of some properties of Apache Spark. It's in the business of being a commodity scale out compute model, which is to say it runs compute on many cheap machines. It's based on a declarative computing formalism, so you tell it what to compute, and its job is to figure out the details of how and where. And it does all this based on a data structure called the resilient distributed data set, the RDD. So what does the distributed component mean? It means that you have a logical view where you have an application, and it's working with a single monolithic data set. But in reality, the physical situation is that your app is running on a master process, and your data is actually being shared across different executor processes. So anybody familiar with Hadoop will be pretty comfortable with this idea. What does the resilient component mean? So RDDs all contain their own compute lineage. They know how to recompute their values. Furthermore, they're immutable, which means essentially that when you compute them, they're referentially transparent. There are no compute side effects. So these two properties combine to make recovery of lost data very, very easy. So what does this look like in a concrete example? Suppose I have here some integer data residing on some executors, and I want to apply a function to it to double all those values. So the master ships that function off to all the executors, and then executes, and you can see that it doubled the data. So what happens if, say, an executor gets lost? The master knows how to recompute this. Basically, once it finds another executor, it produces the original data, reships the function, and reapplies. So the situation is a good fit for pods in a container management system. You can have your application running on a driver pod, and each executor can also occupy its pod. Now it's not perfect, but it is good. The reason it's not perfect is because executors are not quite pure cattle. Each executor, as you can see from the previous example, has different parts of the data. However, it's still a good fit because Spark is able to recompute the lost data. And so if you happen to lose an executor pod, Spark will recover that if you replace it. So it acts almost like cattle, just not quite. That is really all I'm going to talk about Spark. Anybody who knows will know that's a whole universe unto itself. So here are a couple of breadcrumbs for learning more about Apache Spark and also learning about my group's vision for Spark running on Kubernetes and OpenShift. So on a diagram, how does this submission work? You have a client, which might be running on, say, just a terminal outside of your cluster. And it spins up a driver pod from outside. And in that, it's running a scheduler backend, which is a component of Spark, which is semi-plugable. And its job is to spin up the actual executor pod. So the scheduler backend is essentially managing executor life cycles. Now, what does Spark on Kubernetes currently look like? It's a collaboration we're engaged with with some guys at Google. Upstream, the intent is for it to be a new Apache Spark subproject, kind of like a mesos support is now. The code is using the Fabricate Kubernetes client library. And architecturally, it's basically producing two new scheduling subclasses, the Kubernetes cluster scheduler, which is in the business of creating the driver pod from outside the cluster. And then the Kubernetes cluster scheduler backend is actually running on the driver pod from inside, and it's creating the executor pods. So the cluster submission parameters relating to OpenShift are, of course, the cluster URI, which is just your API endpoint. And of course, you have to give it a Spark image, which is used by both the scheduler and the backend. And those are for producing pods, having container images that have Apache Spark installed on them. You also have to give it a namespace, which you can default if you're just in like Kubernetes, but in OpenShift this really matters. And of course, it's used by the submission client in the scheduler, defines scope for authorizations and supports non-admin submissions. And lastly, you also want to give it a service account name, and that's used from the driver pod inside by the backend. And it authorizes that pod to actually create the executor pods. So at this point, I'm going to break out of this for a minute and give you a very short demo. I have here queued up a Spark submission and a running cluster. And you can see here, I am giving it the classic Spark Pi application, which is like Spark's Hello World. I'm giving it the namespace, which is just sort of the usual My Project. Here's the images, which are some custom images that I spun up for running on OpenShift. And it's synced up, of course, with the local Spark build on my machine. And then I also gave it a jar file, and somewhere in here should be the actual name of the Spark surface account that I gave it. So anyway, if I give it to kick it off, you can see it's producing a driver pod, and it's running. And the driver pod just kicked off an executor, and it's a very short process, so it's already done. And there's nothing but the driver pod. So if I go in here and look at the log, somewhere down here far enough, you should see the value of Pi being output. It's a font size, it's actually hard for me to find the value of Pi, but anyway, you can see that it actually completed successfully. And somewhere in here, the value of Pi got output here. So I'm going to kick out of this and go back to the presentation. So what's next on the roadmap? One thing I'm interested in is supporting persistent volumes. You know, it's an important feature from OpenShift's point of view, and you've got to give it some kind of persistent volume claim from the command line and some corresponding mount point. My particular vision for this is probably they're going to be read many and maybe write once because we don't want to be importing all kinds of output on them. That's not efficient. Hey, take care, man. So I think another byproduct of this is if you have your data mounted this way, you can actually push your volume secrets to the cluster and users don't have to get involved with that. Now, non-persistent volumes are of course also an option, but you have to give the secrets at submission time. There's a whole bunch of interesting options for executor pod management. You know, you can use a straight replication controller for just static executor scaling. If you want elastic executors, horizontal pod auto scaling ought to work really nicely provided we can get the right custom metrics going. If you want to support elasticity plus multi-tenancy and like fair sharing across applications, I think you're going to have to use a custom controller to do that because it needs a bit more global information. And also I think, you know, this is basically working purely from the command line currently, but, you know, you can obviously kick off a driver pod from a console running on OpenShift. You know, the current Oshinko project, which my teammates have been working real hard on is, you know, a great model for that kind of capability already. And lastly, the state of the code here, the base branch is by this fellow on Arud Ramonathan and he's working at a Google. And I've been posting my updates as a pull request and the actual images are on my Docker account there. And that's it. Thank you. Perfect. Can you tell me a little bit more about the Oshinko project and what that is? Yeah, Oshinko is a lot the team might stop me if I start saying things that are wrong, but it will, it will spin up a spark cluster from inside the console and I think you can also now run apps against it. It's not working exactly natively. What it's doing is producing a static standalone cluster using pods. It looks fairly similar to what I did, but it's not, it's not driving it from down inside the code. It's sort of working on it from exterior. Okay. Look forward to seeing that soon. So I'm going to open it up for questions and unmute folks and Michael you're unmuted and Trevor and will be unmuted you if you want to add some other stuff from your point of view on the team. And if anybody else would like to get unmuted, just raise your hand and chat and I'll do that. I agree with what Eric said about Oshinko that was pretty right on. This this project so it is still on a work in progress from what I from what I'm gathering. Are you looking for more people to test it and contribute to it? How are you looking to get feedback on it? That's a great question. I think you could actually use it. It's now, like I said, I got it working properly against OpenShift. And so if anybody is interested in actually, you know, dog food in this definitely get in touch with me. I think I should do that. And you'll be at the upcoming KubeCon event. So I'm sure we'll get a few people interested in that. And maybe we can even get you into the Red Hat booth to do a demo of it there if people want to check it out. Okay, there's any other questions for Eric at this point? Anyone's got some? I'm just curious if other people who are working on Apache Spark, if these images that you have created here are the same ones that they're using or what you've done to customize these images that you have here? The two chief customizations are they just happen to be based on builds with this new sub project in them. And I guess the other difference is I tweaked some of the kind of like startup logic a little bit to allow it to at runtime pull down the application jar file into something then without having to open up permissions like all over the place. So that sounds kind of interesting and templates for that might be where they would find that in special tweets if they wanted to create their own. I guess I didn't put this on here, but if people are interested, just reach out to me and I can show you. I also have a branch of the OpenShift Spark repo where, you know, my little customizations on the Docker file are. And I can show people where all it is and how to build this stuff. It's relatively standard except for the existence of the new sub project. It sounds like a good idea. So I think it sounds like good fodder for an actual blog post and to get something up there and writing in the doc set as well. So we'll have to work on getting that once it gets a little bit closer to production time. And do you expect this to be something that's in a production release? I think, well, based on the classic model, you know, we're going to have to basically put this into an official Apache Spark PR at some point and then they're going to have to, you know, get it onto their roadmap. And it's not clear to me how long that cycle is going to take. From my point of view, a lot of the stuff I really wanted was to get it working against OpenShift proper and so that's going. Some of these other features, I think we could work on prioritizing and try to get this as a PR, you know. And I'm not sure, you know, I'm not totally sure what Google wants to see over and above what's currently there. That would be interesting to get Arinda to come on and talk about his work as well. I was really hoping that these guys would be at Kubecom, but neither of them are going to be so. Well, then what we'll have to do is reach out to them and get them to come and do another talk sometime in the not too distant future on this and see where they're going. And then I'd love to get something that showcases the Oshinko work, often available for people to take a look at as well. If you can put up the slide with your contact information in it, again, maybe that was your very second or third site so people can find you. That would be great and we can end on that. And there you go. So the folks want to find Eric and work with him on this project. This is how to get a hold of him. We'll be at the Openshift Commons gathering November 7th in Seattle and he's also attending the Kubecon event that's coming up. So please look for him there and if you can't find him, find me and I will help you find the other members of the Openshift Commons big data. So Will Benton is giving one of the talks on the day. And they'll be Openshift big data SIG lunch meetup. So definitely bring your questions and come join us there. Hey Diane. Yeah, this is Mike McEwen. I'll put a link in chat here to a video I made that shows Oshinko and one of the applications we've been working on. This is for a talk I'm giving an Apache big data in a couple of weeks. So if people want to see like Oshinko working or something that's, you know, they can watch the video. There you go. That would be awesome. And I will add that into the blog post that goes out and the email to the big data state so people can find that. All right. Oh, and it's on Vimeo. All right, guys. So thanks again for Eric for taking the time today to do this talk and we will see you all in just a few days in Seattle. So I'm looking forward to that. Take care. Thanks. Thank you.