 All right, good morning and welcome to our first and to your first session at KubeCon North America 2021 in Los Angeles My name is Archie. I'm CNCF ambassador from Canada. I will be a host track Please welcome our speakers Katrina Veray SIG lead and track chair for 6cli He's also software engineer at Apple And we have Jeff Reagan software staff engineer at Tesla Though they are customized contributors, so please welcome them at our first talk Okay Quick housekeeping items at the end of the session you may raise your hand and ask your question I'll be running around with microphone and helping to answer the questions if you're watching us online You can also submit your questions In the Q&A box and we'll make sure your question is answered. Thank you. Thank you. Thanks for the introduction Okay, so I'm Jeff and this Katrina. We're going to talk about customizing customize with custom Resources Thanks to the CNCF for bringing us all here today. I'm really glad to be at a live event this time around The talks gonna have three sections. We're gonna talk about what is customized at least in the context of extending it It's not going to be an introduction Then we're gonna go into talking about how you would use cluster and client side customer resources to do this extension And then go into the nuts and bolts of that So first, let me review what customize is So you can talk about extending it It's three things It's a command line utility freestanding YAML editor that knows about Kubernetes It is a set of code modules Right, and it is also a command in kubectl. So back in late 2009 or early 2019 late 2018 Brian Grant fill with truck and others noticed that there were many missing features outstanding issues in kubectl That kind of centered around doing edits to the client side data either edits or creating data as opposed to what kubectl normally does which is edit the live cluster so We wanted to Fix or provide a fix of these features, but it was complicated. We felt it was kind of big So we decided to do it outside of the contents of kubectl Because we figured we'd probably throw away a few versions. So this resulted in some modules which are now called by Customizers is relatively in CLI and the same modules are called by a kubectl So customizes the configuration stream editor works with Kubernetes YAML as I just said It's also very good with the working with variance. Now everybody has multiple environments Multiple clusters multiple namespaces maybe multiple namespaces and clusters and customized is really good at expressing the differences between these different environments, so if you've got production staging and Development and they all share like 90% of an uncommon Customize allows you to succinctly describe the differences between these different environments And a place next to get so it can read information from get and You're encouraged to write the information back into a get repository and it can be extended Which is what the core of this talk is going to be So what are the guiding principles when you're thinking about customize and thinking about writing extensions? first Configuration is at rest. It's just data. It should be directly usable by Kubernetes. So we issue a templating We issue domain specific languages. We just want to see Kubernetes YAML second edits or Generating Yeah, well should be done that that operation should be described itself using Kubernetes YAML So no matter where you look you see Kubernetes YAML Whoops, let's get this slide there This seems to be aggressive Not too close Okay, so this is an example of a layout for a very simple customized situation where you've got three You got a three normal Kubernetes files you want to make edits to them you you can edit them with vi or emacs or whatever Or you can drop a customization file in there to describe the edits that you'd like to perform And this makes sense in many different contexts, which we're not going to go into today The customization file is the center file on this screen. It's saying add a name prefix add some labels Do it to the service.yaml file so that the service.yaml file on the left Comes out standard out looking like the on the left your left comes out looking like the file on the right So you invoke this by writing customized build and then the path to the folder that has a customization file in it There's no there's other flags you can apply here The flags don't affect the output The only thing that does affect the output is the you can have a flag that says Put the output into a bunch of individual files one per resource with names that are generated from the resources kind and Normally though you can just pipe it to standard out And when you're doing development you can pipe it directly into your Kind cluster your communities in doctor cluster that's running on your laptop. You never do that in production though So what we have is customized Orchestrating transformers it reads yaml from disk Runch generators to create more yaml feeds all that yaml into a pipeline or transformers and some of those transformers are listed up on the Right there you've got a patch transformer for you can do jason patching you can do strategic merge patching You can change the namespace modify the names add labels annotations and whatnot so Getting down into how we're going to extend customize here are two customization files that do exactly the same thing on The left is a short customization file with one field in it that actually does stuff It's the name prefix field and it says add a name prefix Bob hyphen And that's gonna add that name prefix to all of the resources in the view of the customization in its in its purview on the right There's no name prefix field. There's a transformers field and that has a list of Transformers specifications. There's only one of them there. There's only one transformer there. It's the prefix suffix transformer And there's a prefix field that says Bob hyphen and there's a field spec that says Modify the data that's in the field. That's at the jason path metadata name Now who is it whoever owns this customization file could add more fields there and thus Have Bob added to the value in those fields They could also add kind specifications to limit the transformation to only apply to say deployments or cron jobs and leave Ingress and services alone So you get a full power by using the specification on the right. It's just More of a vote so at 90% of the time people are gonna use the thing on the left But this is the pathway to expansion so On the left you see the same thing again. That's the prefix suffix transformer It represents there's a chunk of code specified there and there's how you configure that chunk of code to run and on the right There's a proposed custom resource that's doing the same thing It's specifying a chunk of code in this case a container as image and there's gonna be data in that custom resource spec Telling that image telling that a code what to do So it's helpful for us. It's helpful for me anyway to think about how What problems we're gonna solve and we with customary sources or extensions and we put those into three different categories? One is generation and then transformation of validation the generation you might take a you might want to create a config map From a basic in V file. You might want to create a community secret by exchanging your single sign-on token for some passwords that you're pulling from hash a quote vault or You might want to do something really fancy generate a bunch of things ingress service deployments for a Java set of Java microservices And it's being spring with environment if you're customizing your Transformations and you're working you're you're developing your own say server-side customer resources and let's say you're You haven't debug your reconciliation very well And you need to make sure that you apply things to the cluster in the order that you like you might write a client-side Reordering function to just actually not modify the transfer the the individual resources But actually change the order in which they appear in the animal stream And then finally you might want to do a validator This is like a transformer that doesn't do any transformation It just simply fails with good error messages telling you what you've done This might be a way to say constrain the number of replicas to some bound Okay, so this slide is meant to trigger just ideas about the kinds of extensions you might want to write There's lots of good things you could do you could generate a pound you could write sidecar generator is etc and Now Katrina is going to talk about the wise and where for us Thanks Jeff So hopefully now we have a good idea of how customized extensions work and the many use cases that we can have for them But you might still have some questions about why you should build a formal extension this way After all the fact that customized deals strictly in Kubernetes data Means that it is easy to pipeline it with other tools and in some cases it might be easier to write a script than to build a formal extension and Maybe that makes sense of not as an option sometimes but building the functionality as a customized extension has many benefits Most of these come from the fact that extensions follow so exactly in the footsteps of core customized features as we're just seeing This means that the users of your extension will retain customized properties that they know and love that is They retain a purely declarative configuration that is Fully encapsulated by their customization directly directory, and they can use the familiar customized resource model exclusively in making their edits They also are not exposed to any templating and have no new language to learn on their part Another awesome property of customized extensions is that they're based on an open standard Which means that if you build an extension for customized It's also going to work in some other client side configuration tools such as kept We're going to talk about that more later Since we've been describing this extension format as client side custom resources another thing that might come to mind for you is the question of When you use the client side customer pattern versus the server side traditional server side custom resources How do you decide? Well, there are many benefits to moving more of your configuration management to the client side and therefore earlier in your software development lifecycle A really obvious one is that client side custom resources naturally do not require you to install anything in your clusters Which can be a huge benefit and especially in some multi-tenant situations where CRD installation might be not desired or not even possible For an abstraction designer perspective, there's also an advantage to the fact that the end user retains control over the final result Specifically the client side version of this pattern is perfect for situations where you have an abstraction that your concerns might become leaky over time For example, imagine you're designing an app abstraction that wraps a deployment and some best practices for your organization If you do it on the client side Your end users are going to be able to use the familiar language of customized to modify the deployment arbitrary fields of that deployment even after you generate it and You don't have to require you don't have to expose any additional fields to let them make those arbitrary modifications So your abstraction stays tight and yet they get the power to modify whatever they need related to that Once you have the results of your extension and the modifications that your end users make All of that is going to still end up in as gamble in your source control if you're using that get offs best practices Of course, this provides greater auditability and change major change management capabilities and then the server side implementation of the same thing would And then finally the fact that we're fully finalizing our configuration on the client side also means that any problems with it can Be surfaced earlier in the development lifecycle that is to say they can be surfaced even before commit or maybe NCI So that sounds great, right? Let's shift everything to the client Well, no, there are many use cases for server side custom resources that are really not suitable for me being ported to the client side Notably server side custom resources should be completely side-effect free Which means that they can only be used for abstractions that simply resolve to a set of resources For example, let's say you're designing a custom resource that's going to cause a database to be configured in a third-party cloud somewhere Well, that is an enormous side effect, and it should definitely not be happening from a client side CR Also, although customized extensions are a really great way to validate Kubernetes resources They're generally not suitable for doing any sort of Enforcement of either policies or standards because as we were just mentioning in the benefits section the flip side Giving your users a control over the final modifications is that they're always able to do that. So You don't actually have full control in typical workflows again over the final result after your extension runs Then finally and you need to keep in mind when you're building an extension that using extensions is going to add Dependencies to your customized build and that's really important to keep in mind Especially if you're not already committing the results of the build to get So now we understand how extensions work and what they're good for so let's have a look at what it's like to build one First an important note about extensions and customize The type of extension that we're talking about today isn't actually the only one that customized currently supports It's the one that we recommend though Customize actually has five different types of extensions all of which have been in alpha for quite some time First we had a format for extensions that could be written as binaries Are as go plugins and they had to be installed imperatively by each end user Then came a newer format that follows something called the Kram function specification This is actually the open specification that I was mentioning earlier It's shared with other configuration management tools such as cats and this gives your extensions written in this format more reach The newer format currently supports authoring as binaries in Starlark or as containers So this year we had a look at all of this We decided that Extensions are really important and we really want to graduate them out of alpha But having these various ways of doing things having to maintain all of this It's really not sustainable and it's pretty confusing So we have a proposal open that deprecates the legacy plug plugin styles and promotes the style that follows that newer specification For the time being we recommend that you build containerized care and function specifically Because Starlark is also proposed for deprecation and exec might undergo more small changes as part of the progression to stable Whereas containerized functions were expecting to graduate In fairly consistent state to the way there are now If you're curious about the details of these plans, please check out the caps that we have open We're going to include these links again at the end of the slides and our slides are also available in the conference schedule So don't worry about copying it down. So as I was just saying to recap the recommendation We're highly encouraging extension authors who are getting started today to build containerized care and function specifically and if you know Go, there's an easy way to do this as part of customize there is a package called K YAML and It has a function framework package That can be used to help you easily author extensions. So we're going to look at we're going to walk through some examples of how to use that today Let's get started with an example where we're going to build a transformer that does something really simple What it's going to do is add a single annotation that has a static key and the user controlled value So in other words, we're determining what the key is the user is giving us the value That's all they get to do. We're going to insert that annotation into all the resources. They give us simple So in our customization, we want to write something like this. This is what the end users should get to write as As you can see, we have a transformer configuration It has its own kind the value annotator and we're using an API version that is unique to our company In the transformer configuration, the user is going to be supplying our extension with that very important data that they want us to put in the annotation So understand what this extension is going to need to do We're going to first take a step back and review what's going to happen to this configuration when customized finds it as part of running the customized build So going back to what we learned earlier in the presentation Customize is essentially a series of generators that are going to feed their data into a pipeline of transformers within that pipeline Everything is configured with care and amul Inside the transformer pipeline itself the output of each transformer in turn is passed to the next one in the line as the input And then finally we get our final amul result So in this case here, we're making a transformer Which means customized is going to find that configuration of writing in that transformers field But then what is it going to do? How will it know where to find the code we're writing? Turns out that transformer configuration I showed you previously was missing something super important The part that points customized the implementation we're about to build Currently, this is actually done with our annotation on the customer resource The one of the open caps that I mentioned earlier Proposes a way to remove this concern from the end user which would be really nice For now though, this is what the end user configuration is actually going to look like as you can see there's a direct reference in there to a docker container that we're going to build and When customize gets to this step in this transformers pipeline It's going to be able to read that annotation and say oh I know where to find the annotate the implementation for this value annotator type And it's going to go get your docker container and executed It's going to specifically pass the resources that is accumulated so far from that pipeline that we're looking at To the docker container in a really specific format. We're going to take a look at next So that input format is a Kubernetes resource model API of course because everything customized is and it's called a resource list This resource list is going to be passed to the extension on standard in and It looks it looks like this. It's got two fields that we care about today And these fields are determined by the way by that caring function specification that I keep mentioning So the first field that we're interested in is the items field Items simply contains that list of resources that we've accumulated so far And that's in other words the resources that we need to process in our extension The second part is called the function config field and the content there probably looks pretty familiar by now It's that same client-side custom resource that we're using to define our extensions configuration That's coming straight to us from that transformers field in the customization It turns out that resource list is actually both the input and the output format required by the care and function specification So when we receive that input on the left in the form of resource list We're going to need to do our extension to work and then we're going to need to output resource list as our result So specifically what we're going to do here in this example We're receiving a config map as the in the items list and our job is to add nametations So we're going to go look in the function config. We're going to see the important data that we need to put on as the annotation We're going to put that on to our config map and then we're going to write the Transform config map into the items list the items field of the resource list and Finally, we're going to output that resource list on standard out to get it back to customize So would you actually implement this? No, no way customize has a really powerful annotation feature built right in so you wouldn't bother doing this The purpose of this example is to just get started with showing how easy it is to reproduce simple customized file transformations using the KML SDK So let's take a look at that. This here is the code The entire implementation of the simple extension fits in just 26 lines of go Let's do that on the interesting part So the important thing to notice here is the signature of that function that were implemented in the middle It takes the list of items that we're getting through the resource list So the framework itself is going to take them out of the resource list that received as input reading from standard in and pass it to our function here items field Then we're going to do the work that we want to do and in the body of function And we have to return those items and again the framework is going to take care of Taking those items that we give it and writing them into a resource list opening on standard out as required by the specification The other thing I want to point out about this example is the pipe E and set annotation helpers Camel has a lot of stuff like this That helps you do simple YAML manipulations for common transformations that you might want to implement now Let's move on to a more advanced use case here We're imagining that we're a platform maintainer and we want to promote our company's best practices for spring boot applications We also want to avoid having to teach and our end users how to wire up ingress for their apps So we decided to do is create a client-side customer resource for using customize It's going to be a customized extension Notice that we're going to be asking the user to give us two inputs in this case the image that's containing their code Plus the domain that they want us to expose their application at Now likelihood a real implementation of this would have a bunch more options, but to keep it simple We're just going to go with these fields Let's start once again by looking at the input and the output specification required By the carry on functions less occasion So we have a resource list of course on both sides, but in this example We see the client-side custom resource that we just looked at from our customization So now it's the spring boot up and Then we're going to work with an empty items in this case because if you think about what we're doing here We're not transforming anymore this time. We're a generator so if you look at the Output that we need to create we have some items in our items list Specifically, we've got a deployment a service and ingress the network policy We're gonna have to generate those from scratch and we're gonna have to make sure they're all wired up together the way The user asked us for The first thing that we're going to do to implement something more sophisticated like this is define a type and go that corresponds to that custom resource that we're exposing as You can see this type is gonna have the standard metadata inspect fields that you would think custom Kubernetes object would have and we have a spec that actually has three fields so we see the domain and image fields that we looked at earlier in the end user Configuration example, but we also have a replicas field here. We're gonna implement that one as being optional So you're gonna see how that's easy to do as well Since this struct is going to be populated from YAML it also notice bunch of YAML tags here That's gonna be really important to remember The next thing that we're going to need to do is implement a filter method on that type that has that same signature That we're just looking at in the basic example It's gonna take in a list of items coming from the resource list We're gonna do whatever we need to do to those items Then we're gonna return the resulting items and or an error as applicable The framework is once again going to take care of reading for standard info us and it's gonna call this filter function on our type and It's gonna take the items we give it put it back into a resource list and then it's standard up Camel actually has a variety of tools to help you implement the business logic of your function as well Earlier we saw some simple helpers that help you manipulate individual nodes like by adding annotations like we're doing in the simple example But we also have some higher-level tools here such as the template processor that we're using in this example Here what we're doing is we're creating a bunch of templates for all of those many resources that we need to generate and We're putting them into an embedded directory Then we're using the abstraction type itself so that V1 alpha 1 Java spring boots we're using that directly as template data Temporate template processor type actually also has a bunch of other interesting features that we're not showing here Including the ability to select subsets of resources and apply patches to them, which is a super handy operation Another thing that we might want to do especially because I just said in this example We're using the user input template data is validate that input before we actually run our filter Luckily, the framework makes this super easy to do as well All you need to do is implement this validate method and the framework is going to automatically call it before it calls your filter then We might also want to implement defaulting and this is gonna how we're going to make that replicas field optional It's the end user in the example didn't give us any replicas field And we're going to implement this default method that detects that and sets it to three for us So this is also going to be called automatically by the framework before filter is called The framework even helps you make your custom resource API sustainable by versioning it over time This version API processor type that we're showing here essentially wraps related Implementations and acts as a dispatcher based on API version and kind The firm has even more tools than we have time to show off today Notably it supports patching including for CRDs and at the container level which can help you do a bunch of really handy operations It also has a suite of selector and matcher tools that can help you target exactly the resources that you want You can even help you kickstart containerization by generating your doc file So that we've seen an example of how to build Extensions I'm going to pass it back to Jeff to discuss some best practices that you should keep in mind when authoring for customized Thanks. Okay best practices First keep your extensions declarative the whole thing about customize is to keep everything declarative so Nothing should depend on flags nothing should depend on environment variables or any kind outside resource or to the extent that you can limit it The output should be deterministic so think in terms of unit tests if your input doesn't change your output shouldn't change either so You want to have some predictable outcomes, right? As continue just mentioned leverage the API version one of the reasons We want to use custom resources rather than inventing some other new way of doing this is to leverage all these concepts that we all know so use the API version field to keep the your existing users of your extensions happy while you move into the future with maybe possibly breaking changes and Finally testing is important Anybody trying to get you to use a framework the first question you should ask is well, how do I test things? So this should be recognizable that anybody's done go unit testing so if you're working and go and of course you can use any language you want because the The underlying format is a container, but if you're using go you can use this Where you see the word processor just put in transform or generator or whatever and you can specify the inputs into that thing expect and specify the expected outputs and You can also specify inputs that generate errors so you can check that the errors are what you want That's important for validators because that's the whole business in life is to report errors Stepping back to the bigger picture Customize is part of a CD Pipeline and it wants to work with git and we recommend using what's called a hydrated repository. So Imagine you do a deployment you run customized build and you're using all these extensions and now you're using containers in your pipeline You've just introduced some risk There's the usual risk which we've all been bearing for many years now pulling down the containers of the internet You mitigate that by using a trusted catalog, right? But the new risk is now you've got an extra piece that's running in your CD pipeline and something could go wrong So what you do is you put your own customers build you put the output into a repository Usually with a robot, right and then a robot also reads a repository on say a tag event and a place to the cluster So now if something goes wrong as the British would say your cluster goes pairs up pair up or pear shaped anything You can do a rollback easily by just telling the robot go apply the last tag You don't have to worry about running customized build or going back to the original dry material Finding the tag you checked in and then going rolling back to that tag and then running customized build again Okay, so here's the link slide And that's where customized lives. There's extension documentation there. There is The KML live framework that we were just talking about is listed there. There's some caps that are talking about future plans Especially the catalog which is a cool project. There's a contributing file at the site I would recommend anybody here who is interested in writing go or contributing to Kubernetes in general To come take a look at the contributing file look at issues that are marked help wanted turns out Customized has been a great place for many people to start working on Kubernetes Lots of people started here even just writing documentation and now they're doing releases or now they are 60 li tech leads. So It's a good place to get started and and and get involved with the 60 li That's it Awesome All right. Thank you Katrina and Jeff for amazing presentation. We are now have few minutes for your questions So actually have few questions that our users asking online First question from Daniel. How would you how would we combine customized with helm charts? Is there any ways that they can be you know used together? Yes, you want to take it or should I take that sure yeah, I can take it We have built-in helm generator function right now That you can use that it has evolved over time And the latest version of it requires you to set additional flags on the command line and vacation of customize But it works as a generator. So we talked about generators today Part of that generator pipeline is just going to read your your home chart and send the results through to the next thing in line Another thing that you could do is you if the built-in extension doesn't expose the helm options that you want You can build an extension generator extension like we were talking about today that that renders helm in fact The kept folks have created an extension generator for helm that follows the carry-on function specification So that's yet another option that you can use To to bring those two tools together Awesome, I think another question is very similar. Would it be possible to combine go templating customized? No, I just got done saying that the one of the guiding principles is we're not going to do that at least We want the raw material the material the configuration at rest to be just normal Kubernetes YAML No templates and no no designs domain specific language. I guess Yeah, the caveat on that was that you might have noticed we had template processor as part of the extension guide there so Template if you want to use templating we recommend that you hide it behind a formal API, right? So you're defining your formal type it has the fields your end users are working with pure kubernetes data like they would always with customize But then within your extension if you so choose you can use templating in there and you're not exposing it to your end users at that point So that's an implementation detail. Yes, you could write it in parole if you wanted Awesome, there's actually I will get a take last question from my virtual attendees and then I will be asking you But actually I can answer that question. Would it be possible? What is it possible The compatibility you know of customize with tools like Argo CD or flux So if you look into the documentation of flux or Argo CD, there's actually a the whole chapter around cost how to use customize So get upstooling fully supporting right now The customize so any questions here? Yeah, I want to ask So I happen to have the sad task of maintaining the capability of installing manifests across a very large Set of kubernetes versions today. I support all the way from 117 to like the head of master on KK Does is there any provisions for making it easier to like? As a for example, you know certain apis get deprecated and removed and later kubernetes versions new versions get added It would be nice to have some capability to express that like hey if it's like a 122 cluster You can use this new API, but it does not use the old one or whatever, right? So So you're talking about you've gotten multiple clusters running in different versions and you would like to have Customized manage that or you'd have maybe use is can customize the use to help you manage that I Don't want to have to like maintain distinct manifests for distinct kubernetes API versions over time, right? I want to have the same manifest but tweaked like It's the 116 cluster. So I have to use the like the beta API for CRD is not to be one because it didn't Yeah, I mean, I may be wrong. This sounds like a perfect use for variance. I mean this is kind of Customizes stock and trade you figure out what you have in common between these different environments or the different versions And that becomes your base and then you use a different customization to modify those very modify your base to head off to your your various variants after various versions if you wanted to sort of abstract over the different like API changes for For some end user say say or if you want to do yourself you could do that an extension Because extension is what one of the tools that we have allows you to select on on different resources that exist in the source So you could even theoretically Identify that the input stream has the older API versions and as part of your extension spec You could you can include that cluster version you could like inside your extensions business logic You could look up Well, what API versions are supported here and you could even do the conversion internally since you're writing a full program there You can do essentially whatever whatever you would like you just got to expose the data that you need to Deform the basis of those decisions in in the API that that that you're configuring and then And then you could take care of that within the extension And since you're you're writing a full program there You can you can use the testing framework that we that we were showing off there to make sure that it's going to behave as expected Right now. There's no built-in API version modifier So if all you were trying to do is like use like say a deployment at beta and v1 or something And you wanted to switch that to just v1 I don't think we have that but it would be relatively straightforward to build such a thing All right. Thank you Katrina Jeff We we need to conclude our presentation because we're running out of the time if you have any questions to speakers Please reach out to them. You need you can talk at the booth and thank you very much and enjoy the rest of the cube