 Thank you everyone for joining us. Welcome to today's CNCF live webinar, breaking tradition, the future of package management with Kubernetes. I'm Libby Schultz and I'll be moderating today's webinar. I'm going to read our code of conduct and then hand things over to Dimitri Kalanen, engineer at VMware. A few housekeeping items before we get started. During the webinar, you are not able to speak as an attendee. There is a Q&A box in the chat. On the right hand side of the screen. So just drop any of your questions in the chat and we'll get to as many as we can. This is an official webinar of the CNCF and as such is subject to the CNCF code of conduct. Please do not add anything to the chat or questions that would be in violation of that code of conduct. And please be respectful of all your fellow participants and presenters. Please also note that the recording and slides will be posted later today to the CNCF online programs page at community.cncf.io under online programs. They're also available via your registration link you use to sign in and the recording will be on our online programs YouTube playlist as well. With that, I will hand things over to Dimitri. So take it away. Hey, folks. Feels a little weird without all the little zoom boxes of seeing everyone else. But that's that I guess we'll do for now. Well, thank you for maybe later viewing this the stock. I'm Dimitri. I work at VMware in the map BU unit. Mostly working on Kubernetes, but I do have quite a interesting background, you know, coming from Cloud Foundry. And, you know, before that working on various consulting projects with Pivotal. So today we really want to talk about Kubernetes package management with Carvel and specifically one of the Carvel's project called Cap Controller. Carvel is an OSS project sponsored by VMware. It was born probably around two years ago, maybe. It's actually a toolkit of various Kubernetes related tools. Now, some of those tools are generic. They're not necessarily specifically written for Kubernetes, but they work well with Kubernetes. And some of those tools are obviously very much Kubernetes specific. So we have a little site there Carvel.dev that actually lists out all of the tools. And we are also talking about revamping that stuff to make it a little bit more new user friendly. So tell us what you think when you visit the website. Here's a Slack pound Carvel on the Kubernetes Slack instance. We're very active there. Actually recently reached a thousand members. Really excited for me seeing kind of a community grow over time. And we welcome anybody to drop in chat with us on Carvel related questions or maybe even just how you use Kubernetes. We're always eager to learn what you all are doing. So one thing that you might have seen from recent KubeCon is a talk by one of my colleagues, Shata Rupa. There was a keynote called Breaking Traditional, the Future of Package Management with Kubernetes. There's going to be a link that you can click in the slides and I'm sure you can find it on YouTube. The talk kind of talked about what we have been up to within Carvel community and what are some of our desires and visions for package management. So it actually serves like a good background maybe for this talk. Now this talk is maybe more of a continuation to kind of dive in a little deeper into the details and I actually love doing a lot of demos. So we'll be doing a lot of live demos. Hopefully things go well, but if they go wrong, that's actually more fun than usual. Alright, so one of the slides from that talk was this specific slide. It was calling out declarative Kubernetes APIs, immutable bundles, distributor of OCI registries and really picking the right tool for the job. And when I say the right tool for the job, what I mean is being at the abstraction level that your problem deserves. So most users don't necessarily have the same exact problem, it's similar problems or it's really kind of stacked up in different ways. So at least from my experience, I think it's important to find the tool that actually solves your problem and doesn't actually do too much extra. Because if it does do too much extra, then you're dealing with all the extra complexity or maybe vice versa, right? You found a tool that does bare minimum, but it's not quite enough for your problem. One of the actually other things that we'll touch on is using OCI registry as a storage system. Now, obviously, we'll use OCI registry for container images, but as you probably have noticed, the industry have turned into using registries for a variety of other artifacts. And we also caught up on that trend, I would say, quite a bit early and are very excited in how we use the registries and how we deliver some of the software. So let's jump in into this last bullet point, layered approach. So Carvel, as I mentioned before, is a toolkit, right? And there's various tools and these tools try to follow UNIX philosophy. What I mean by that is we want to have small, sharp tools that know how to help you solve a particular problem and not do anything beyond that, right? And we have a collection of tools that help you with configuration building. For example, the Kubernetes configuration building, we have a tool, for example, that allows you to manage Kubernetes resources in bulk as kind of seen as an application. We have tool that allows you to interact with registries and do interesting things with the assets stored in the registries. And there's a few other tools. One of the tools specifically we'll start with is Capsioli. Capsioli is that tool that allows you to manage multiple resources in bulk, right? You can think of it as QPCTL on steroids. One of the first demos is to actually look at that tool and deploy something with it. Since I've said different problems should probably be solved at different levels of abstraction. So Capsioli is really a CLI that you get to use maybe yourself or maybe in your CI system. But we also thought, well, there's definitely cases where it's much more useful to have an abstraction within Kubernetes that represents the deployment of some configuration to your cluster, right? And that's what AppCR is all about. It's really meant to describe how do you fetch, configure, and deploy your application to the cluster. The next, oops, oh no, pressing the wrong button. All right, the next layer that we'll actually talk about is two other CRs, package install CR and a package, right? So this is where we get a little bit into maybe more traditional thinking around package management, right? So a package CR represents a piece of software at a particular version that, you know, there is this concrete thing and a package install is actually an intent to install that piece of software. Now, as you can see over here, we have a package install arrow pointing to the AppCR. And that's intentional because we want to reuse the lower level of abstractions in our higher levels, right? So we're not throwing away all the useful work that we've done. We're building on top of it. And you as a user can decide which layer you really want to be at. And, oh my God, keep on pressing the upright arrow. So, all right, the next final top level, at least in our current stack, is package repository. And a package repository is all about delivering a set of package CRs to the cluster, right? So similar to a Linux distribution, right? A repository contains a collection of packages. In our case, a package repository contains a collection of package CRs that could ultimately be installed. So we'll take a look at this entire stack as we go along and see what happens. Let's dig in live. Alrighty, so I'll jump over to my terminal here. And I do have a cluster over here that's authenticated. And I'm using CAP CLI to interact with this cluster right now, right? So what I'm saying is let's list all of the applications that we have installed to the system. Now, we haven't currently installed anything, so the list is empty. So if we go ahead, and I've prepared a few handy commands to run for ourselves to kind of explore some of the stuff, if we actually go ahead and deploy the sample YAML from GitHub, we'll see that CAP is going to tell us what things we will install with this YAML, right? So this is actually from one of our Carvel simple apps running on Kubernetes. It's a little go app. But in our case over here, as you can see, we have a service, a Kubernetes service that's being added, and we also have a deployment that's being added. And the deployment is pointing to some image that's been built somewhere. And similar to tools like Terraform, maybe CloudFormation, you get this little nice diff that tells you what are the exact operations that will take. Now, one of the benefits of using CAP is that these are the only operations that will be taken, right? So you know exactly what would happen to your cluster. This gets a little bit interesting, of course, once you actually start changing the objects, not just creating them and deleting them, right? So let's go ahead and actually press confirm, and we'll have our app being deployed here. And it succeeded pretty quick. And we can actually inspect what kind of resources got created. So here I've used a simple app as the name. So we'll say simple app. And we'll see that simple app now is composed of six resources, right? Deployment and service, that's the stuff that we have put in. And then the Kubernetes cluster also added in the endpoints for the service. And then, of course, the deployment got broken down into the replica set and the pod and etc. So that's the basic of CAP, right? You're able to, for example, go ahead and say, okay, I no longer want this app on my cluster and you're able to delete it. And it will find all of these resources and get rid of them. One of the, you know, maybe basics around CAP operation is that we're not trying to do anything outside of just the basic Kubernetes feature set, right? We're not trying to hide the APIs or anything like that. For example, the way that CAP knows which resources are associated with this app is really a true label. So all it's doing is it's generating a unique label and getting that deployed. All right, so CAP CLI, you know, could be used in your CI environments, for example, if you're really pushing out the content to your cluster, or maybe you are, I don't know, maybe you're for whatever reason playing locally yourself, right? And gives you ability to converge the set of resources in some kind of a deterministic manner, right? Rather than trying to hunt down which resources were in the previous version of the thing that you were deploying and removing it and whatnot. So with CAP, of course, you know, so that was a simple little example, right? And, you know, we can, of course, CAP is used in production environments to deploy more, you know, serious software. And so, for example, here's an example of, here's an example of CertManager from 161 release from GitHub. We'll install that real quick as well, just to showcase that, you know, it's happily deploying things. It does show you a little progress log telling you what's going on, what kind of resource is getting applied, you know, some resources are getting applied in parallel, some not, we're waiting for some pods to start up and whatnot. But all in all, all it's trying to do is just deploy, converge, make sure that the resources are in a particular state. And we'll go ahead and also delete the same thing. So CAP delete dash ACM. Let's say yes. All right. So that was the lowest abstraction level that we typically start with, right? Very simple. Now, in our next example over here, we're actually throwing in a little bit of a configuration challenge, right? So, let's take a look at this config, right? So this is the config that we deployed from the GitHub. And it had a service, had a deployment, and it had some, you know, environment configuration going on there, and in a particular URL. Now, we also have a tool within Carvel called YTT. YTT is all about changing your data structures that are represented within YAML, right? We have various approaches, right? Some approaches that are allowed is templating, for example. Another approach that's very common is overlaying. And some folks, for example, mix them, right? Because sometimes the overlaying is much more useful for after the maybe environment-specific configuration versus maybe templating is useful when you're still kind of reconfiguring your configuration in the first place, right? So we allow both of the approaches to be, you know, either solely used or mixed together, found it very powerful when dealing with especially large amounts of configuration. So let's grab that and do a little deploy over here. So we'll have, well, actually, let's see. So we'll do a deploy over here. And in our case, since we're just updating the simple app that we previously installed, it's actually telling us that we're only going to change replicas to be set to 3. And this is based on that overlay, right? That's saying, hey, we will set replicas on the deployment within, you know, the config YAML that it found, right? So let's just match this particular object. And so let's go ahead and say yes. And I think to nobody's surprise, you know, if we go ahead and say, okay, get pod, right? We'll see that there's three pods now running versus what it was one pod running before. All right. Nothing too complicated, right? So just scoping the discussion for a second to this two tools, CAP and YTT, right? You get a variety of possible users, right? You could be doing all kinds of things with, for example, using other tools to generate the YAML, but then doing a CAP deploy, or maybe you're actually using YTT to generate or modify, let's say upstream configuration, right? If I wanted to tweak something in a cert manager YAML that's been published with YTT and then maybe send that to Cube CTL or some other tool, for example, mixing and matching is what kind of brings the power to some of those tools, right? We're not trying to tell you what is the entire tool stack that you should be using, right? We're really trying to say, okay, you have a problem on your hands. We have this particular tool that solves this particular problem. Okay. So easy so far. All right. Let's get a little bit more interesting with maybe a non-CLI option, right? So CAP deploy is great. You might be doing it yourself, or maybe you're doing it in a CI. That's all cool. Some of our users do appreciate a GitHub's first approach, right? And so for that, we actually have a project called CAP controller. CAP controller actually brings several tools together in a fairly opinionated way. And that's what actually provides that stacking of the app CR to the package installed to the package into the package repository. So let's actually go ahead and install CAP controller with CAP deploy onto the cluster. I think we're installing the latest version of CAP controller here from GitHub. We'll just go ahead and say yes. Of course, you could be using Qubectl to do this, but I'm just using CAP deploy here. And what we'll do is actually check out what is it doing inside this number three directory. So what does app CR look like? So actually, that's not a directory that I wanted to take a look at, which, oh yeah, maybe app CR config. Oh yeah, that is the directory. So app CR is really composed of three pieces, fetch, template and deploy. We felt like this is the right amount of balance that allows you to mix and match certain things, right? For example, you might be fetching things from a config map in this case, right? So we have a config map that actually contains a bunch of YAML and an overlay. And you're able to combine it in particular ways. So in this particular case, you can combine it with YTT. And finally, as a last step, you're doing a CAP deploy, right? And so what this app CR is meant to describe is your way of deploying this application. And you can imagine, right, if you have 100 apps, it's very beneficial to have that consistency across all of them, right? They always get deployed, let's say, by app CR driven through Git, for example. Now note, we also have a service account over here because app CR doesn't give you any kind of privileges. You do have to bring your own service account so that the app CR will use it and actually deploy it. So in our case over here, we have a config YAML that we want to deploy. So let's go ahead and do that, the CAP. So we have a CAP controller already installed on the cluster. We will actually go ahead and do a delete of simple app because that's where we're going to be installing through app CR. So that hopefully gets rid of pods real quick. Here it goes. All right. So we only have CAP controller now. And we will go ahead and apply the, we'll create a service account here. So I'll actually copy this out over here. We'll create a service account so that our app CR can deploy things. And we're using in this example, of course, default namespace, but this is not really namespace specific. And so we'll go ahead and actually deploy our app CR in a config map to our cluster. Here it goes. So CAP is actually smart enough to wait for app CR to reconcile. And we can actually see that app CR reconciled pretty quick. And there's some nice status information that's available to you. We have our spec of how we told app CR to do things, but we also have the status getting that, hey, the deploy has succeeded. And note over here, the deploy, that output may look very familiar. And the reason why it may look very familiar is because app CR actually internally uses CAP deploy to deploy the application. So we're building on top of the lower primitive within this higher level. Now, you also notice that the fetch step succeeded over here. It fetched some stuff from a config map, nothing too exciting. And finally, the templating steps succeeded over here. And we do have this nice little handy inspect output indicating what's been created. So now after that app CR has succeeded, you actually get to see via CAP that simple app exists here deployed three pods over here happily running, right? And, you know, we can go ahead and, for example, change something inside this config map. Let's say we want to scale it down to four and we'll go ahead and apply the same. We'll even make it a little fancy here and show you that that's what we're changing here. We're changing the replicas for yes, and then the app CR should pick up. It might have already picked up. So, okay, get pod. Now there's four pods running, right? All right, so that's the app CR level layer that we've been talking about. And let me flip back to the slide real quick. So we've covered the CAP CLI. And then we're also covering right now the app CR. Now, let's take a look at the app CR that's a little bit more complicated than what we have, right? So what we had was just, hey, read some stuff, read some content from a config map. But that might be useful for maybe a local tweaking, local development, maybe even you can imagine building a little CLIs that allow you to fill in the config map from your local resources. More on that later maybe beyond the talk. Come talk to us about your use cases in the Slack channel. Here's, on the other hand, a more complicated app CR that actually quite powerfully captures what most people want, right? So they have maybe a Git repository somewhere that contains some configuration and they want to be able to manage this configuration a little bit. And then finally they want to deploy it, right? So same exact steps. In this case, though, we're only fetching from Git. We're doing a little bit of YTT massaging for that configuration. We're deploying this, right? So let's go ahead and again, we'll do CAP delete. Actually, let's do a, this copy paste this. We'll just do this here. Oh, actually, this is, oh yeah, that's true. Okay, so you can actually see that it's changing from inline to Git. It's changing YTT from doing nothing to, for example, including a particular set of directories within what it fetched. And then it's actually getting rid of this config map because there's no need for it. And so that happened awfully fast, but here it is reconciled. We can actually take a look at this. And as you can actually see, because we're fetching from Git this time, there is actually a little bit of handy information that's being included over here. We're actually showing what is the, what is the commit that it fetched. And, you know, here you can actually see that it converged based on the changes, right? So previously, our config map was containing, I guess, a deployment and a service. And after it's been fetching this stuff from the actual Git repository, there's been a little bit of changes. And you can actually configure App CR to show you the exact diff for it, similar to how we're doing locally. But at least in this case, it's not configured to do it. And then we also have the little inspector here. So back to Kapp.ls here. You can see we're back to App CTRL being now managed by the Git source. Okay, so that's a Git repo with some YAML and, you know, nothing too exciting. Because App CR is really somewhat agnostic, right, to the content that it's fetching, right? And the split between fetching, templating, and deploying is really this powerful way of thinking about stuff, right? We can do all kinds of other things that, you know, a lot of the common use cases call for, right? So here we're fetching from a Bitnami chart, Git repository at a particular reference. We're actually saying let's do a little bit of helm templating, this particular NGINX values. And this chart, I guess, supports the configuration of NGINX server. And then we're also on top of that. So not just we did the helm template, but on top of that, we're also doing YTT overlay that allows us to change something that is not configurable in that chart. So we are removing external traffic policy in this case and actually setting the type cluster IP on this particular service, right? And so this is a little NGINX App CR. And let's actually go ahead and grab that and deploy that, all right? So we'll go ahead and say, so we are creating the NGINX App CR. We are deleting our simple app because I guess we don't need it anymore in this config. And we're also applying the NGINX values. So let's go ahead and say yes to that. So that may take a little bit since it's going to, you know, deploy the resources to the cluster, fetch the images, etc. You can actually see, well, not quick enough, I guess. You can actually see incremental output as things progress, right? So we don't just update the status at the end. We actually update the status as the App CR reconciliation progresses. And as you can see over here, the NGINX helm chart here was, it had the deployment and the service and the config map, I guess, NGINX server block, and ultimately deployed that. And supposedly there is some NGINX running here. Yep, there's a NGINX port running inside the default namespace for our installment. So that's, that kind of showcases, right, the power that you can really, you know, we could actually be within App CR, you know, there's like different ways of fetch things. And so you could be using a helm chart directly to fetch it from the helm repository or you could be using Git. Or you can actually be doing something that we've been doing at VMware more recently, is also putting in configuration into the OCR registry and actually fetching the configuration from what we call bundles from the OCR registry. But the greater point here is that we're able to mix and match those things within App CR and that gives tremendous flexibility. Actually, one thing that recently somebody was really appreciative of is the feature of being able to decrypt certain configuration via SOPs as part of the templating section. So we do have a SOPs configuration that you can provide over here to, for example, use the age backend to decrypt some of the configuration that might be encrypted in your Git repository. And in theory, you know, if you haven't noticed here, fetch is an array. So you could actually be fetching multiple things. So one of those things could, for example, be your encrypted, you know, credentials maybe, right? If that's how you manage your credentials in the production, right? So a lot of the flexibility here, because a lot of people have different choices made for their production environments, for their dev environments, et cetera. All right, so let's keep on going here. So the next example that we're going on is an App CR that actually uses this concept of an image package bundle. So what exactly are these image package bundles? Well, to break that down a little bit more, maybe in a more visual way, let's actually look at this slide over here. I'm not going to press the present mode because it's a little bit, don't see my screen, so it's a little bit terrifying. So we'll break this concept of a bundle into a few things, right? So to deploy something to Kubernetes, you need some configuration, right? Now, that configuration may come in different types of formats. For example, it might be a configuration that's a plain YAML manifest, right? It might be, you know, Helm template. It might be a YTT template. Maybe it's some new templating mechanisms that have recently been invented, right? Now, previously I've mentioned that similar to the rest of the industry, there's a recent trend that lets you use registry because it's such a common API now that's available everywhere. Let's use a registry to actually store content that's not directly contained in images. So we've started doing that quite a bit ago with this tool called Image Package. And one of the things that we've kind of organically grew in Image Package is this concept of a bundle. We want to be able to not just to store the configuration of something, but we also want to be able to reference other OCI artifacts so that we can build up a graph of the software. And so this image kind of tries to showcase that, right? So you may have some, let's say YAML configuration in File 2, YAML, File 3, YAML. But we also have this dot Image Package images YAML that references this other images that are being, let's say, referenced by this File 2 and File 3 YAMLs, right? Now, we do have another tool that actually easily is able to evaluate, like, here the X number of images that you've been using and then actually create the images YAML for you. But the general concept of being able to build up this graph is quite powerful. What's even more powerful about this is that when we do this reference to another image, it could, in theory, be another bundle, right? So you can kind of see how we're building up this graph. And Image Package itself doesn't actually have an opinion what's inside those images. It doesn't even know, right? It just knows how to deal with OCI Registry references. Now, these OCI Registry references don't even have to be in the same registry, right? You know, if you've seen, for example, you know, you may want to push your configuration up into your registry, but that configuration is actually pointing to the other registries. Maybe something to, you know, in GCR or Docker Hub, et cetera, et cetera. Because I don't know, depending on your level of confidence with some of the external assets, right? You may be just consuming the external assets directly versus like trying to rebuild them yourself directly, right? So how does this actually look like when somebody is building Image Package bundles, right? Well, it actually looks very close to what you typically do with Docker commands, right? So you have this Image Package push that takes a reference, and you also get to specify the content of the bundle, right? And so inside this My Bundle directory, you typically would have that Image Package bundle sub-directory that contains a little bit of metadata and the content that you want to put in there. Now, one of the things that you could be doing now, now that you have this graph of the software, is you can actually take that graph of software and move that over to a different registry. As for example, a lot of users who really appreciate this feature are the users who want to have an air-gapped environment, right? So they're able to say Image Package, copy this bundle to a tarball, and so that recursively grabs everything that you ever need and then brings that over to the other registry of your choice, wherever you import your tarball that contains, let's say, 100 images, and then you're able to plug that back in into, let's say, cap controller, or I mean, a warranty, you may have entirely different set of tooling that you're using for deployment or something like that, right? So that's where it gets interesting, right? Once you have this graph, you can do cool things around it. Now, more recently, we actually added functionality to also grab the core sign signatures. We know the entire graph, and so that means that we can actually check if for each one of the nodes in the graph is there an associated signature or multiple signature, right? We can also start bringing in things like S-bonds and attestations, right? Once you have the graph, you can do all kinds of additional things that production system, for example, may want to take advantage of. And finally, once you've carried that over maybe to your own registry or maybe you've kept the bundle in the original location, you can always pull it, unpack it into a directory. This just unpacks the contents of the bundle directly. It doesn't actually bring over all the referenced images or anything like that, but it does give you a list of referenced images that you can take advantage of and put it in to consume it via some other system and actually use those references, right? Now, because we're requiring everything to be digest base, digest referenced, right? We have that guarantee that your bundle digest represents the entire graph that it captured, right? So again, you can, for example, co-sign the bundle itself, right? And then that actually translates into the confidence that you're bringing over the entire thing, and it's all exactly the same digest. So that's a little bit of image package. And so where does that leave us, right? So in our example over here, we're actually using image package bundle. I think this is one of the images that we use in our end-to-end test suite for CapController. But you're able to download this bundle and then, you know, run some configuration munging on it. Cable is the little tool that actually inserts the new updated references, and then finally you do the deploy here, right? And so if we want to actually apply that, so we'll actually, again, I'll copy this over here. And so as you can see over here, we're adding our app CR that uses the image package bundle. We're actually deleting, in this case, the nginx-related bits, because I guess this example no longer needs it. And so first step, right? Fetching, we will take advantage of image package pool, grab the bundle and deploy it. And you can actually see it here again. Nothing too exciting again. We're back to our deployed applications pretty quickly. So that's how you would use a bundle. Now what's interesting about OCI registries is they do give you this, you know, location where you can find various things, right? And so once you get to that, you can get a little bit fancy with, for example, your tags, right? So here we're saying use the particular tag, right? We would also, for example, if you really wanted to do this for a production, we would probably do something like cable through it and actually grab and have your YAML have the digest, right, to be referenced to that exact image. However, in this example, we're saying, you know what? We don't want to know which tag is going to get picked, which image is going to get picked. It will happen automatically based on this tag selection policy, right? So in here, we have Sember as the way to decide what to fetch. And so within this OCI registry repository, there's a few tags. I think this one was using V1. This one will most likely pick V2 because I think that's what it is. And so let's go ahead and actually apply that as well. So we'll say, I think it's that. And so here we're changing to the tag selection. We're also throwing in some YTT stuff, not that it's really necessary, but... So here it goes. It goes in and image package will talk to the registry, will figure out what's going on, which is the tag you want. And so if we look at the app status, we will actually see what happened here. So you can actually see that it picked tag VO. Now by default, actually, we don't pick pre-release versions, right? Because pre-release is not necessarily maybe production, man. However, you could add a little bit more configuration over here that, for example, says include the pre-releases. Or maybe even one of my favorite features may be include certain pre-releases that have, let's say, RC in them, right? And so that kind of acts as a way for you to filter out what do you actually want your environment to be consuming, right? And in different types of environments, you might be consuming slightly different artifacts, right? All right, so that was an example of how we use more flexible fetching to determine what you actually want to install, right? So far, we've gone through the examples of Capp CLI, doing the deploy directly. You are responsible for running the Capp deploy, right? Wherever you are, right? We've looked at AppCR that allows you to explicitly say concretely how to deploy, how to fetch, template, and deploy your application, right? And Capp Controller will facilitate that continuously. By the way, maybe I haven't made that clear. Sorry, this is something that's maybe been not talked about, is that the AppCR deployment process happens continuously. Actually, for AppCRs, the default is fairly short. I think it's about 30 seconds or something like that, right? So this loop, if actually, is happening all the time. Now, there's definitely a few optimizations to react to the system in various ways more quickly, but at the least, right, there is that reconciliation really happening. And so if your artifact is changing, right, like you might be pointing to an origin develop here, right? An origin develop keeps on changing. Capp Controller will pick up those changes and redeploy them, right, at that, you know, at that time. All right, so AppCR, you get to specify concretely how to deploy something. Now, in a higher level block, right, if we go back to our slides over here, the next layer of this notion of a package and package CR, well, sometimes users don't want to actually decide how to deploy something, right? They don't know. Maybe some other team is responsible for determining that. Maybe somehow they receive that piece of software. So they want to not worry about it, right? And so that's exactly what package CR is about. Let me just kind of focus on this particular CR. So we have kind package CR, right? It has a name, ref name here, and it has a version, right? So this is what's, you can think of it similar to how CRDs have a fully qualified name, right? We expect each package to also have a fully qualified name, right? So that there's no funky, maybe, eliciting conflicts going on and whatnot. And we also want a version to be provided, right? Now, this two pieces of information is actually, this section and this section is what's required in package CR. Everything else is optional. Obviously, it's good to provide all kinds of useful metadata to whoever is consuming this package CRs. But for our example here, really the focus is the ref name, the version, and the template section. Now, the template section is actually app CR type embedded in, right? So this is a similar pattern how the deployment in Kubernetes embeds, for example, a pod spec, right? So this is exactly the same thing. And this is ultimately how this, how gets translated is that package install that operates on the package CR that reads it in, right, through your reference, will create that app CR based on this template, right? So all of the functionality that you might have been using directly with app CR, right, could be interior used here, right? Now, for our own usage, we have mostly stuck with image package bundles because they provide you that nice guarantee of immutability, especially when pointed through digest, right? But you may, for whatever reason, want to hide what a piece of software, how the piece of software is actually being installed, and maybe that piece of software is being installed from a Git repository at a particular Git shell or something like that, right? Or maybe just continuously changing, right? So that's the package CR, right? And once it gets loaded into the system, right, you can now use package install to install it based on a ref name, based on a particular version selection. Now, no, the version selection kind of sounds like that tag selection that I previously mentioned in app CR. Well, it's actually backed by the same little struct. And what that allows you to do ultimately is allows you to specify the policy of which version do you want to select, right? And so because the package CRs are actually require you to have versions to be sember format, you get to potentially put in some interesting constraints here that have your system automatically evaluate and kind of maybe keep things updated continuously, right? So let's actually go ahead and install this package CR to our cluster over here. So we'll grab that and we'll do, oops, unauthorized. Well, that's unfortunate. I think my session expired for the cluster. Let me stop sharing for a second. Let me real quickly reload my token here. Apologies for the little pause. Maybe a coffee break for anybody watching. And let me grab the appropriate configuration here. It's going to be one second. Just keeping it real with the live demos. That's right. I was kind of, you know, I did the refresh my tokens and whatnot right before, but I wasn't actually sure how long and when they expire. So that was a calculated bet that hopefully maybe they last an hour, but I guess they last a little less. All right. So it looks like it looks like I'm back. So let me screen share again. All right. Hopefully we're back. So as I was saying, we're going to run the cab deploy. And here it goes again. All right. So here what we're doing is we're adding a package CR to the cluster. We have an app over here. And we have a package install. And in fact, what's actually going to get a little funky here is that both package install and app, they're called simple app here. And so let's go ahead and actually still go for it. But because package install installs the app, it might get a little confused, if you will. So let's let's see what's going on over here. So oops. So we have, I guess it finished. All right. And so it still was pretty quick. We have the app, that simple app. And we also now have a package install. Oops. Is it not package install like that? I actually don't recall what it's supposed to be. It must be package install. Yeah. The shortcut is the short name is package. And I've been using it for long enough to remember to forget the long name. All right. So we have a version two that's been installed. And we have the app that ultimately the package install created as a result of this package CR. Now we also have package record here. Right. That's the thing that we imported version two. Here is it in its YAML shape and form. And it actually tells you all the information that we saw in the YAML. Actually, one thing I forgot to mention was the little section over here in the package YAML that I hidden away is the little value schema. So you can actually know what potentially your package is exposing as configuration. We'll get to that I think in a moment. So package install successfully installed version two. Sounds great. Nothing fancy again. Let's move on maybe to the next example then. So in the next example, what we have here is the following. So the package install can be configured with certain values. And unfortunately, I think this example doesn't actually showcase the values that it's how it's configuring. Let's say maybe this one. Yeah. So here's an example of, for example, package install allows you to specify certain values. And these values are referenced through a secret. And inside the secret, you could be dropping in that configuration. So this is very exactly the same actually as the app CR. In fact, this secret reference will make it down into the app CR that gets created. Now, on top of that, some things are not configurable through values. Well, somebody authored some software, and they just don't have a certain configuration up that you really need in your environment. That's ultimately how we actually YTT, for example, includes templating and overlaying. Because we do see that 80-20 match between 80% of templating, let's say, and 20% of overlaying, especially in production environments where there's different kinds of ways of configuring things. So in this example, we're saying package install, please use this secret as an additional overlay on top of this package so that by the time it gets installed, things get modified a little bit. So in this case, we're actually setting replica five in our overlay. And right now there's only three pods. So if we go ahead and then again, I have real handy, got paste. So I'm cheating a little bit over here. I'm using this dash P flag. And what that means is actually don't try to delete anything, only do the create or update actions, which is handy if you're just kind of iterating on stuff and maybe you don't want to provide the entirety of the config every single time. And so as you can see over here, the diff is add the little overlay on the package install and we actually have, well, our secret is hidden away from us because who knows, maybe it's sensitive information, but the overlay YAML includes, you know, setting to five replicas. So if we go ahead and cab deploy this, hopefully a few seconds later, we can see that now there is five pods instead of three pods, right? So our overlay is actually modifying, our overlay is modifying the underlying package in a particular way, right? So this is a really great way to adopt something from let's say upstream, right? But you don't necessarily have, you know, ability to modify it and you don't want to fork it. You don't want to, you know, do something special just for yourself, right? So you get to use the overlay to override and add in your opinions. All right, so here's another example, package install of nginx. It's not that exciting, I would say, since we already see an app CR taking advantage of that nginx. So if you actually look at the package YAML, you can actually see that this is straight copy of that app CR, put it in here, wrapped in a package CR. We give it a name nginxtestcarvel.dev, version one, right? And in package install, you're saying, I'd like to install version one of nginxtestcarvel.dev, right? And you're providing those values. So given the time where I think we're running a little low here, I'm just going to skip over this example and maybe jump straight through this example of package repository, our highest layer. So in our picture over here, our highest layer was package repository. And maybe you can't really argue that it's a layer on top. It's more of a wrapper. Since a lot of, let's say, users might be consuming a lot of versions of the software, right? How do you distribute your package CRs, right? Well, you could definitely throw it in into, let's say, an image package bundle or a git repository and have another app CR that keeps on updating your cluster, totally valid way of using it. But we figured, let's actually add a high-level concept package repository that allows us to specify where is this repository located through the fetch section, right? And ultimately, being able to install things from that package repository, right? So an example of this would be... So let's go ahead and delete our app over here. So this is deleting the package install, the package, the overlay. So we'll just go to kind of a clean, nice world here. So your ghost finished. So if we go ahead and say cap ls, nothing here, right? We only have cap controller installed. And so we'll go ahead and apply the step nine over here. Step nine, here it goes. Actually, that's a step eight. Let me get to step nine. So we have a package repository and a package install. We'll go ahead and say yes, right? And what would happen? What would happen is that if we look inside our cluster, the package repository brought this three-package CRs of different versions, right? And our package install actually decided, okay, I'd like to actually install version one. Now, there's actually a few variations I put in here for just the sake of entertainment here. So this is how you specify an explicit version, but you may actually specify, just give me the latest thing, right? We're not specifying any constraints, right? So in this case, the latest thing would have been 2.0, right? Versus 1.0 that I explicitly selected. And remember, I said that we don't actually do RC or we don't do pre-release builds, right? So minus RC is the pre-release build. So you could actually enable it and say, okay, well, actually get me the latest any build, RC or not, right? So that's that. One thing maybe worth mentioning real quick, really running out of time is, okay, well, package repositories, right? Who's creating this package repositories? Well, recently, one of the teams was in VMware, Tanzu actually made an open source project called Tanzu Community Edition, right? And so an example of a real package repository comes from those folks over there. A really great job at making that all happen. So if we go ahead and say TCE-F, and I think it's 9.1 TCE YAML, I'll just go ahead and add this repository here real quick. So we'll go ahead and say yes, and give it a few seconds here. And so previously, we only had this set of packages right now. If we go ahead and say, show me the packages inside this namespace, we got lots of different packages, right? Now, as you can see here, we actually have multiple versions of CertManager that the TCE community has included in. And you can actually see package metadata, which is kind of a more combined view. It doesn't have any versions or anything, just the package names. And so you can see we have CertManager, Contour, et cetera, et cetera, right? But this is just an example. And in addition to our other repo that brings in some other packages from our earlier examples. I'm now going to go and do maybe an install of CertManager. I think you all got the point already that it just, all it is, is a standard now API using Kubernetes package install, something, right? And it's done. So we'll maybe get back to the window here. So the last slide I had was, you know, please join us in the Carvel Slack. We love the discussions. We want to hear your feedback. We want to hear your ideas. Carvel Dev is also a good starting point to find out more about all these Carvel tools. And do tell us the suggestions of how we can improve any of this stuff, what are your use cases and how we can be helping you to actually solve those use cases. And I'm going to, I guess, flip over to the chat over here and see any questions here that might be worth calling out. And by the way, if anybody wants to discuss their questions further, jump in, jump into the Carvel Slack. We can discuss there too. So what do we have here? So hello from Slovakia. Hello from Brazil. Hello all. Hey, all to you too. Hello from... Is it Prem or... Or is it Prem? Prem from France. Just to level set package management with Cade that will be discussed here as fully upstream compatible, not VMware Tanzu specific. Yes, absolutely. It's all OSS. It just happens to live within a VMware Tanzu GitHub organization, but it is just like all the other projects, Contour, Valero, they're all open source. Anything you can take to to use yourself. And that's been already replied. Any insight on difference between Argo CD and CAP? Well, CAP CLI is a CLI, right? That allows you to do that deployment. So it's a little different in the sense that Argo CD is a system to... It's a high level system, right? So arguably one would want to compare Argo CD and CAP controller. That goes a little bit into... We feel like the feature set that we have and the way that the APIs are designed is more Kubernetes centric rather than Argo CD. And we're also like delaying that we've got going on. On top of the CAP CLI, the APCRs, the packages, etc. So that's our take on it. And the impossibility with Helm and customize huge topic. We do have some notes in the YTT documentation, how we feel like YTT is comparing to some of them. Are these atomic operations? Can we have custom template engines tools? So some folks try to extend it a little bit. We do want to work on out-of-the-box extensibility a little bit more. But we haven't had the chance to actually collect more feedback from a community. Do come over and tell us in the Slack channel what kind of tools you would want to extend. What kind of integrations would you look forward to? Or maybe how would you want to extend those things? And by the way, I think I realized your name was Prem now. That wasn't the location. So hello. All right. We had a need for something. Hey, Scott, the package is amazing. Thank you. Thank you for that. VMware TCA supports Carvel Pack. I have no idea about VMware TCA, but I'm sure we can find out what's going on in the future for you. We have a need for the combination of customizing subs. Great to see you're considering it. Yes, I think, you know, subs we got the integration for already customized. We don't have it yet, but we'd definitely like to hear the feedback of how you all would like to use it. What kind of features? Customize does have a lot of features, and there's definitely some considerations in how it's best to expose those features. But otherwise, we're open to ideas. All right. Thank you so much for joining us. Thank you so much, Dmitri, and thank you everyone for joining. We had a great turnout today. Everything will be online shortly, as well as the slide deck, and catch us again next week for another round of live webinars and live streams. And thanks again. Thank you.