 Hi, I'm Eric Tune. I'm here with Daniel Smith. We're both long-time Kubernetes contributors since before it was public And we're gonna talk today about extensibility and Kubernetes Okay So before I start talking about extensibility, let me address Who I think cares about this so? Cluster operators are gonna want to understand the components that are deployed in their clusters even if they're you know Not writing extens- extensions and they will They may want to create site-specific policy distribution creators will want to know how to integrate with infrastructure as a service Cloud providers, especially will want to integrate with their own infrastructure as a service Maybe add new APIs that expose, you know unique or specific features of their cloud Paz authors these Kubernetes is a good platform to build platforms of service on top of May want to create add new APIs, but still be able to dive down into Kubernetes for debugging into the underlying implementation hardware and software vendors may want to add new APIs to Kubernetes or expose unique storage or network features most importantly if you're a core contributor or any kind of Kubernetes user you're gonna benefit from Stabilization to the core of the core of the Kubernetes project So right now we have a lot of pressure to add New features constantly and yet our users are asking us for a more stable product less changes Release notes are ridiculously long. So the answer there is to break that up and to so have software components with different maturity levels the core needs to become very mature and yet we still want a rich ecosystem of Extensions or plugins that can have different levels of maturity different release cadences I tried to come up with like a short definition of what it means to be be using extensibility in Kubernetes Or what's an extension and what's not an extension and actually couldn't come up with one that I could fit in a sentence So let me first start by talking about what Kubernetes is and and what it is not what isn't extending Kubernetes So Kubernetes is open source. So you always have the option of forking it And feel free. That's hard. It's projects moving really fast The code base is really big and it's hard to figure out where the point is you want to extend things There's constant refactoring going on and finally if you're a lot of Kubernetes users are using who Kubernetes in a hosted Environment, I think at least the top four clouds probably have hosted Kubernetes I'm not sure exactly you could argue over exactly what the ranking is But you there's a lot of hosted Kubernetes out there and I don't think any of those people want to support a fork So another way you can use Kubernetes is by what I call automating which means just taking this that what I call the normal APIs things that you would normally post resources to and then using our client to Automate that process once you have a you don't need to do things multiple times We have clients and I think seven different languages. So that's easy, but automation doesn't cover everything It's not good for things. It's a synchronous So it's not good when you need to hook you into every event make sure that you don't miss anything block things before they happen It can't add APIs and it can't change the behavior of existing APIs So when you need to be synchronous We need to add APIs or change them or add new features then you need an extension now Kubernetes I think if you maybe somebody went to Chen's keynote This morning she mentioned there are like 12 different extension points in Kubernetes. Guess what? I'm an exhaustively list those for you with the help of Daniel In fact, there's so many ways to extend that it can be a bit overwhelming. So part of this is gonna This talk is gonna give you a guide of maybe which ones you want how to combine them and How to think about our roadmap whether you should sort of trust this morass of extension possibilities Hopefully you'll come out believing that you know, there's at least some you want to use So I like to say that Kubernetes is really two separate things to me It's an abstraction over infrastructure and it but and it's also a framework for declarative APIs and distributed control There's actually pretty separate like a lot of people are talking coming to me and saying hey I want to use Kubernetes style APIs, but I actually don't need pods and services and nodes So I think in the year or two we're gonna start to see more of that. That's not what this is talk is about So we got a dozen extension mechanisms and And for me it's useful to categorize them into two rough groups One is what I call infrastructure extension mechanisms, which matches my first definition And the second one is API extension mechanisms. I'm Daniel. I'm gonna divide this up I'll talk about infrastructure extensions and then Daniel dive into the API So it's December for me. I'm getting excited. I'm thinking about skiing Maybe some of you are pretty excited too. So then you know what this is I didn't hear woods from everyone. So maybe I should explain this for a second These are like the difficulty codes you get when you go skiing and to tell you how hard a run is a Green circle means, you know, beginners can go there double black diamond is the hardest and sort of in this order And so trans I tried to translate this to Kubernetes green meant like I have high confidence that you can find examples documentation you have to write a small amount of code you get a good choice of languages and You're unlikely to bork your cluster by trying to write one of these extensions Or if you do you'll be able to understand what's going on. It doesn't affect everything in the cluster It only affects what you're extending And the project is likely to extend it So it's a support support this extension mechanism going forward or not plenty to deprecate or anything like that So the negation of all those would be a like a double black diamond. So we're gonna see all these ones as we go Some of these were some of these runs are gonna get groomed, you know over the next year So they'll get easier but for now this is the current state So let me go in infrastructure extensibility I put if you guys are the type of person who likes to actually like get way more data While I'm talking and ignore me and there's a URL up there for some docs that actually just merged They give a lot more detail about all the extension mechanisms in the kubernetes.io docs And I will I see a people taking pictures. That's fine. Please tweet whatever I promise that we'll upload the current PDS afterwards So you can if that's what you're doing you can relax your cameras So I'll start with storage extensions So the storage extensions allow you to add new kinds of volumes to your pods things to get mounted as a positive file system This is not intended to like handle API based storage like bucket storage It's meant to handle things that have a Unix file file API So there's actually two different mechanisms of flux volumes came first. These are super easy to write I think I saw someone write a volume plug-in and plug-in by the way means a binary that gets called out by the Kublet or another component of the kubernetes system Someone wrote one in bash with like a hundred lines including a really long copyright header. So I thought that was pretty amazing The plan for that I just talked to Saad Ali who's one of the leads in sick storage and He said expect flex volume support to stick around indefinitely, but we're not going to make it better But to me it's super easy. So I think that's great This new thing that just came out in 1.9 is called constainer storage interface. It's really open Docker MeSos and kubernetes all support this type of plug-in and it's easier to upgrade and deploy on top of kubernetes Instead of having to like go into your node file system and like upgrade a bash script You can actually deploy the CSI plug-ins as a Damon set, which I think is really cool I'd expect this one to stick around and grow its alpha in 1.9. It's gonna get better in the next year Infrastructure, let's see cloud container managers This is factoring out like right now you have a choice when you run your master to say like hey I'm an AWS or I'm on GCP or or whatever or I'm not any of those We're gonna move that code out so as and so it's actually a separate binary and that Almost got in 1.9, but it's good. It looks like it's gonna be alpha in 1.10 And I would expect to see a lot of a lot of changes there in 2018 Device plug-ins, that's a way to like allow discreet hardware resources to be surfaced from your cubelets So things I could when I Google for this I saw people talk about GPUs FPGAs Qrng I hadn't even heard of apparently that's like a random number generator I don't know if that's for like financial traders or or something. I don't know does anyone know that's for Okay, so I thought that was cool like it's being used to extend resources. I hadn't even heard of So so this is this is pretty basic, but it's cool It lets you surface them to the eight the scheduler so you can see which nodes have these and which ones don't This is alpha in 1.8. I expect to see that get fancier Let's see first network plug-ins This there's an open standard called CNI and there's like a couple dozen these plugins And they I think they started with Docker, but we had to tweak it a little bit to get it to work with everywhere with kates because we have Kubernetes because of host port so there's a couple that support Kubernetes fully and I expect to see more of those over time This is a little bit Trickier like I think this uses a gRPC instead of being a binary plug-in I Know the least about this so I should stop talking All right, oh wow that did not format so those diamonds are supposed to be like side-by-side, but so This one's one that scares me the most This is like and I like double black diamond runs, but this like I broke my leg snowboarding This is like this is the scary run for me is replacing the scheduler You can replace the scheduler with your own scheduler But like I think that'll work your cluster personally I would talk to the SIG they're doing a lot of refactoring I would I would think twice before doing that the easiest thing although it didn't render right on the slides This one's called scheduler extender is actually the easiest one to do you can add your own prioritization So if you're like oh, yeah those machines on like rack 23 I really want to use those because I can't wait for them to burn up You can like put in a plug-in model. This is always prefer to put the stuff on those machines You know if it fits And see secrets 1.9 like the secrets that are stored in that CD are now encrypted at rest So there's like a data encryption key and a key encryption key and like right now I think the key encryption key is like on the like in a file But the plan is from 1.10 you be able to actually put that key encryption key in Like a KMS like vault and that'll be an extension where it can like call out to your KMS And so that'll make people happier. It's that's been a long Asked for for a long time And I talked to one of the leads on that and said maybe by Ended 2018 lots to be some GA on that encryption at rest support All right, so those are all the Infrastructure extension mechanisms I want you to know about And I'm gonna hand off to Daniel. He's gonna talk about Let's talk about So work better to hold here let's talk about the API extensibility So I broke down a little okay, you can sort of see it I broke down a little spectrum of some various API level integration extension points Going from easiest to hardest We have CRD's how many of you know what a CRD is Okay, great most of you Next we have a CRD and you can put a schema in there or we have a CRD plus a schema plus a Validation web book that's Getting a little bit more difficult and then finally if you really want to challenge you can use an aggregated API server All of these are mechanisms for adding your own API object into the system So let's talk a little bit about those in more detail In 2018 I expect CRDs to go to GA Along the way I expect us to improve completeness. We already have a schema We're gonna make it easy to make validation web books We're gonna make it easy to set permissions Yeah, so so I should integrate with our back and the author of the CRD should be able to tell what permissions What roles does it make sense to operate on the CRD? We shouldn't force the cluster administrator to understand that and I expect to improve Aspects of the API wide consistency like we have a status pattern. We have a scale pattern We have a few common sub resources like that I expect to add mechanisms to CRD so that you can seamlessly integrate those So so that you can we can have HPA horizontal pod autoscaler work on your CRD so you can scale up your CRD instead of scaling up your deployment things like that and Yeah, there's lots of stuff that we can do if we have a common status Which I think is in a future slide so when people hear about CRDs and they hear about aggregated API servers The the question is invariably. How do I know which one to do what with or? If you look at a CRD, it's pretty easy. You submit a YAML file to the system It gives you a new resource But if you look at a making an aggregated API server, it doesn't look so easy because you have to compile stuff and import a library and there's a API server builder program, but You've got to do a bunch of extra steps. So why do we have both? I thought it might be useful to say why exactly you need you would ever need to use a aggregated API server instead of a CRD and the the Really there's actually the the number of things is shrinking over time So the the one thing that I expect to never change is with a with an aggregated API server You can completely change the storage mechanism so if you are doing like our Metrics API server if you are fabricating data for somebody who wants to watch or or pull An aggregated API server is really the only option And that's the the metrics is high-volume high-volume data. It's manufactured. It's it's aggregated together So yeah, it doesn't make sense for that to be in a CRD Something that is true today and will be less true in the future as soon as 1.9 goes Goes out live the admission chain You can if you have an aggregated API server you can put, you know, whatever random business logic you want in the admission chain And Something that is true today and might not be true at the end of 2018 is that the aggregated API server allows you to write arbitrary version transformation, so if you have The first version of your resource and you make a bunch of awesome changes and you have the second version of your resource With the aggregated API server framework. You can automatically upgrade people Who is this who is this for? We wrote this We had somewhat in mind extension developers, but we had in mind ourselves a lot The Kubernetes repo is very large it is twisty and tourney and full of passages all alike and It is getting to be difficult to change to make changes that don't have Effects that ripple out through the entire thing. So the idea is that We split we give API authors their own binaries and They have there's less of a chance of them interacting poorly with another component Yeah, so so the the idea is many Targeted API servers rather than one massive API server So I mentioned on the previous side that admission stack. What is what is the admission stack? it is everything between the authorization check the permissions check and Before we actually commit the request so everything between the permissions check in the storage layer And this is where we do all of the all the fun stuff like checking quota or making sure that you're Making sure that your pod image pull policy is is what the What the your system administrator wants? This is the ideal place for policy enforcement. Well, there's a problem It's all compiled in So if you wanted to use this before 1.9 your option was to fork API server and Write a compiled in admission control plugin If you've contemplated forking API server, you'll be very happy that We have Produced webbooks so you can now dynamically register a web book The API you have to implement is relatively simple You can get called whenever an object changes when it's posted when somebody tries to delete it Should double-check that one. I think I think I think you can get that It's beta in 1.9 and we intend to take it to GA in 2018 The dynamic configuration should be awesome So you can do this in a in a cluster like the reason the resource has to be on but you don't have to mess with Flags on API server. You don't have to let mess with config maps. You can just Add a configuration object to the system Another option for a similar purpose is initializers We've sort of been working on both initializers and webpokes initializers are in alpha and At some point in the course of 2018 we expect to make a decision about whether we will Go all in on webbooks or if there is a place for initializers also Yeah Yeah, so what else are we gonna do in 2018? I expect to be in the business of moving flags into config files for for cube API server and Ideally, I also expect to move config files into API is where it's appropriate I don't like the fact that to change things about API server. You have to Twiddle with the flags and restart it right now. I think that's very inconvenient for cluster operators and So as a concrete example of the sort of thing the Permissions or Aussie web hook right now. It's a command line flag and it's a config file and you can only have one I Think we need to that that's that's a great example of something else that I can see lots of people wanting wanting to add the difference between the permissions check web hook and the Admission check web hook is that the Admission check any one of them can fail the request and you can also modify a request with the admission web hook For permissions it only takes I guess you can now they've they've they fixed it. You can now deny a request, but It must pass everybody Yeah And my my goal my ultimate goal is fully portable extensions you write an extension on That runs on one cluster you should be able to move it to another cluster just like the the goal of Kubernetes is portable workloads I think your your your extensions are not Going to find a wide audience unless you can be confident that you can take them with you to whatever Cluster cloud environment you want to run on On and with that I think back to Eric on how can you combine? these various extension mechanisms Thanks, Daniel. Yeah, so there's a bunch of different choices of extension mechanisms, and I kind of categorize them That doesn't mean you can't mix and match them So I get put a couple examples up here I just talking to a gentleman here who works in the brook project and they actually use us I didn't know this they use a CRD plus the volume plug-ins so you can like Post a CRD that describes a certain type of volume and then have that show up that that was cool I don't know the details, but And then I found out that Calico canal I don't know much about it, but apparently uses CRD plus a network plug-in the basic combination There's a CRD plus controller automation. That's like we call the operator pattern. You've probably heard that especially if it's like running and applicate like a Database or something we call that operator pattern And then Daniel mentioned that when you can use the mission web books plus CRD so you get better validation on them I'm really curious to see what sort of interesting combinations people will come up with like combine like a storage plug-in with you know an Aussie plug-in or something. I don't know so Next year we'll tell So Daniel talked a little bit. We're heading I want to also share my aspirations for the next year I Want to see Automatic rich see like if you write a CRD or any kind of customer source And it's just automatically in the CLI in the GUI GUI give you a good experience that means like you know You should be able to say kubik could all get and have the right columns that you intended when you list it out You should be able to you know in your UI You should be able to see like a red dot if it has unhappy status to it So that's somewhat aspirational, but that's the direction we want ahead Also want like right now you can build Customer sources on top of the platform, but then you can't build take existing things and have them come on top of the customer Sources as well Daniel mentioned this adding a scale resource will allow you to use HPA also, let's use a pod disruption budgets, which I'm talking about tomorrow if you're gonna be there Another thing I've heard is that as people are seeing their Customer sources mature. They're missing that version conversion So I think personally that's a big priority for us to figure out in 2018. We don't know how we're gonna do it yet So please find me if you have ideas on how to do that And then Daniel suggested like a cluster introspection API so one thing we do in GUI's is when you create it So typical pattern with like a customer source is that it makes more resources and those resources make more resources But as a viewer you typically only want to see the top level thing that you created Except when it breaks then you want to debug and expand it So I want this to be a pervasive pattern in UI's where like you just see the top level resource But then if you'd want to drill down you can find the things and if you see an air message from a pod You can tie that back up to its parent And the API server actually has all that information in a graph used by the garbage collector But I think there's a lot more we could do with that. So that's That's one more thing up we do So I hope you I've convinced you that like the Kubernetes project has a strong commitment to being extensible Maintaining that extensibility across all platforms You can see that from the number extension points. We have the number talks about extensibility Keynotes that mentioned it Completing these is a large part of the work for several SIGs in the next year You can also see a commitment to open standards where they apply for example CSI and CNI are used by other orchestrators and CRDs use JSON schema and I'm very interested in more opportunities to Reuse open standards with Kubernetes, especially extensibility And I without I think Daniel I can take questions If we have to use Kubernetes for Orchestrating non container a services along with container service. Do we have to write our own or our own? Extensions is that a pattern at all that you've seen Kafka and others that are not contain raised, but there are the container services So the pattern I see is some people are starting to use Kubernetes style APIs Maybe even using CRDs And an operator to manage resources that actually don't have pods under them So it could be like serverless which might be on top of Kubernetes or might not I think red hat open shift has an extension allows you to manage your VMs Using a custom resource. So I would say it's fine to use a custom resource to manage infrastructure. That's not containers And happy to follow up on that and I've had multiple requests today already for Sort of API server without a Kubernetes cluster. So Yeah, I would look for something along those lines in the future I'd also have a call to people that are like less container people and more like API deep thinkers to come Join the conversation. We want to hear from you next question I just have first of all I want to put my vote on initializers yay initializers Second it'd be super helpful when I want to use an initializer I'm a bit concerned putting it on some of the core resources because like I want to put it on pods But it should be kind of cool in my mind to have initializers target based on labels So it's like oh, yeah, this is an initializer for pods like this and then don't stick it on the rest I don't care because right now you have to process every single one or they won't get initialized I have I have two thoughts on that one is we're a little scared if you put them on pods, too It's I'm doing it. So the other thought is we for the admission webbooks to make them safe We added a namespace Selector so that it's possible to have namespaces in which the plug-in doesn't apply. So when's that where's that at in? That's that's the configuration object. Yeah, when's it is available in one eight though 1.9 Okay, all right We want you to try webbooks in 1.9 and then come back to us in like a quarter and say how that went Can you modify objects in webbooks? I thought it was just you can 1.9. Okay. All right Yeah, give it a try you So related to that Can you modify objects before storing using custom admission? Yeah, the the admission webbooks come in two flavors. There's a validating and a mutating one and the mutating one runs on both posts and Puts so both creates and updates and you can make modifications to the object. Okay, and you can Install new admissions without restarting the API server. That's correct. There's a dynamic configuration API. Awesome So can I ask a few more questions? Okay, so the other question is I know pod presets were something that allowed injection and Haven't read about initializers. So what's the main difference between initializers and pod presets both allow injection? I thought Do they both allow injection of containers when we let Eric answer that one? So there wasn't complete agreement on exactly what the pod preset API should be so instead of putting it in core the decision for now is to Put it in the service catalogs Resource API groups and I think the intent is to use In a future version to use the mutating webhooks as what service catalog builds the pod preset on top of and Then they're gonna run with that for a while and get feedback on whether the API is good And then that can be a conversation whether that needs to pull in a core or what? But it also shouldn't be too hard to build something similar if you have a special use case And we'd be happy to chat afterwards about that Yeah, so what's the can we contrast the cloud provider API and the extension APIs? Yeah, what do you think about that? So the cloud provider manages like IP address ranges for services I think it might help with ingresses it manages like the life cycle of nodes and detecting when like the Backing instance of a node is disappeared. It probably doesn't other stuff. I don't know anymore So it's possible that might get broken up into more Like individual webhooks if there's a demand for that But right now the low-hanging fruit is just getting it to be a separate release train And what I think we'll just let that soak for a while before we decide it's definitely not overlapping like the it's it's about No, it's more about nodes whereas an IP in physical resources Whereas the other all the other infrastructures I talked about or about like bringing those physical resources into the pod abstraction So they mostly don't overlap. Yeah, I see I see the I see the cloud provider as like another downward-facing Extension sort of like the the infrastructure extensions We're running out of time here We got time for one more. Yeah, you had a bullet point about how you don't want people to replace schedulers And I just want to just want to understand where you're coming from or trying to understand What do you run into what issues you run into based on your experience? So those are my personal opinions and not just that run looks scary to me Maybe you're like an expert here. You've already been down and it's not that bad But I do think that there's I Think with dual schedulers it will If you go too far down the dual scheduler path It'll limit the ability in the future for us to add better like resource sharing Primitives that we would like to add like a year from now So I think at some point that might get broken, but maybe it solves your immediate need We have Bobby from the scheduler team here. Let's let Bobby talk then. Sorry No, I know I actually share Everything that Eric said so as we are going forward it seems harder and harder to Support other schedulers, especially competing schedulers with Which is sometimes the case in some of the clusters So people run multiple the schedulers at the same time But as scheduler is advancing and adding more features It becomes harder and harder to keep that promise that we are completely a scheduler agnostic Feel free to replace our schedule and that was that was sort of like the promise initially But it becomes harder and harder. So split so basically Nobody prevents you from replacing the scheduler, but you're sort of like giving you a heads up I don't want to use warning, but giving you a heads up that it may become harder and harder and Maintaining a customer scheduler, which completely replaces the schedule may become very hard So what we encourage people is to keep running a default to schedule a default the schedule is very Comfortable you can pretty much disable everything that it does almost and by everything I mean all the predicates and priorities that it supports and It's pluggable currently we are also improving the pluggability of the architecture and the performance of extensions So hopefully it should be enough for almost everybody Yeah I might summarize it as like like a go package might export an interface and they're committing to like not adding new functions or change the signatures that interface and Like web hooks when that becomes GA like that'll be we're committed to not changing that and we already don't think we're gonna Change it a lot the scheduler does not have a well-defined interface It has a hundred responsibilities, and we might add 50 more in the next year So you you'll be running to keep up. Is that yeah Basically the same reasons why we Prefer you to just use at CD as this backing storage and we didn't want a bunch of different storage backends And people also replace kubelet, which is to me even crazier than replacing the scheduler But like I mean, you know if you've got a big enough team then then go for it You know the Kubernetes is the thesis ship Let's let's stop there and Daniel, and I'll be sure to hang out in the hallway afterwards. Thanks everyone