 All right, good morning, and we're on time. Hello, GitOpsians. OK, fine. Yeah. There we go. That was perfect for post-travel morning. I like it. All right, welcome to not another episode of GitOps Guide to the Galaxy, but welcome to ArgoCon EU 2024. Today, we have a panel with my good friends Andrew Block, Dan Garfield, and Christian Hernandez. Thank you, yeah. There will not be jokes, but I do expect applause anyway. So today, expanding ArgoCity content management using OCI artifacts, that is a mouthful. What does it mean? Let's get started. OK, I'm going to go ahead, and I'm going to kick this off. It's kind of my baby. I love it a lot about it, and I kind of want to talk to you about who we are, maybe. So let me introduce myself. My name is Andy Block. I'm a distinguished architect at Red Hat. I've been working in the GitOps and Argo space for goodness gracious, got to be three, four years. So not a lot. Been working with a lot of folks here on the stage, in the community, and kind of want to talk to you today about what OCI artifacts are and what they can mean to Argo. Dan, do you want to give an introduction to yourself? Sure. So my name is Dan Garfield. I co-founder and chief open service officer of CodeFresh, but now acquired by Octopus Deploy and running open source in the Argo office there. Yeah, so Chris Hernandez, head of community at Acuity, self-proclaimed GitOps kingpin, and very excited about what OCI means to GitOps in general. And you're your MC here. Oh, right, yes, of course. So I'm your MC today. My name is Hilary Lipzig. I am a principal reliability engineer at Red Hat and host of the Red Hat livestream GitOps guide to the galaxy. Hence the shirt. Most of us know there are two primary sources for applications in Argo. It's either going to be Git or Helm. Certainly everyone has access to either a Git repository or a Helm repository, right? Yes. But maybe not. Maybe you're in certain environments where those aren't available. Maybe there are some networking constraints, firewall rules, regulations. I have many of my customers where they're not allowed to have a Git repository in their deployment, an operational environment because it's separation of concerns, development, operations, or certain policies from an organization standpoint don't allow that. Or they just can't deploy a Git server in that environment or talking edge environments. However, what does every Kubernetes environment have access to? Anybody? A container registry. Because we're typically using container images for our applications. So why can't we go ahead and take advantage of the infrastructure that we already have in place and reuse that for Argo CD? And that's what we are going to talk about today in leveraging OCI artifacts. Now, what the heck is OCI artifacts? I hear there were artifacts. And an artifact can be a lot of things. It can be a jar. It can be a Maven module, a Python module. You name it. Well, let's talk about what OCI is. OCI is the Open Container Initiative. It means another thing. But in this context, OCI is the Open Container Initiative. It is an open governance structure that really focuses on container formats and runtimes. There were some container wars, conflicts of interests, stuff about goodness gracious, seven, eight years ago. God, we're getting old here in the Kubernetes world. But really, the OCI group came together to create these standards to define how container images should be packaged, how they should be run. And recently, there's a lot of thought and interest around this concept of artifacts. Being able to leverage container images and a lot of the OCI structures in many of the same ways that we leveraged container images. So OCI artifacts allows us to use container repositories for storage of other things aside from container images. We're talking things that we are using today. Helm charts can be stored in container images. Wasm modules, software-able materials artifacts, they are already able to be stored as OCI artifacts. Now, why can't we go ahead and take that same concept and extend that to Argo CD? So as I mentioned, OCI artifacts, probably OCI Helm charts has been supported in Argo since version 1.8. Now it's back in 2020. I know other things happened in 2020, so we need not want to remember that. However, this we want to do want to remember as a way that we can enable OCI already in the Argo CD ecosystem. And it's really easy to get started with Helm and OCI. If you haven't used that as a source type in the past, create a secret that has the property OCI enable, probably enable OCI to true. And you can start leveraging Helm charts that are stored in your OCI registry. So why are we here? We're going to talk today about taking advantage of that and reusing it to provide more native capabilities around OCI and Argo. So there's an enhancement proposal that was submitted about a year ago by some guy. Can't imagine who. Looks familiar. And some folks also contributing, also familiar. So everyone who put together this enhancement proposal is sitting here on the stage. There's one other person, Michael, who you saw earlier. He was on the talk previously. We're all very interested in the opportunities for OCI artifacts and Argo CD. We're going to talk to some of the ways that it solves challenges, some of the areas that we're thinking about, and really how we can get the community to help build this content together. Really, this isn't about the four of us, five of us, et cetera, building it ourselves. That's not why we're here. We're all just in common here. People talk about Argo. It is we want to work with Argo. We want to contribute to Argo. We want to make Argo a better place to work and thrive. So some of the areas that we're thinking about, and we're going to talk through of them, everything from client development, OCI application sources, CLI integration, user interfaces. These are all things that we're thinking about. But we want to, A, talk to you all, see what you think, because we're just a couple of guys and gals. We want to hear what your thoughts are if this is an interest to you, or if you think we're just crazy, which is possible, but at least in this context. Both things can be true at the same time. That is very true. So I'm going to sit down. I'm going to stand up. Hilary, go ahead and start standing and ask us questions and get some questions from the audience. Wonderful. Thank you, Andy. I was going to bring note cards and be like really wasteful and throw them around, and then I forgot my note card. So we're doing this from the phone, and I'm sorry about that. More technical anyways. It's more technical anyways, but it would have been fun. Anyway, like I said, there will be no jokes. So just to all of you, can you kind of individually talk a little bit about your interaction with like OCI artifacts up to now? And we're not going to make you go last because you were just talking, and we're going to start on the other end with Christian. Yeah, so with the OCI artifacts, so with the integration that is possible with, especially in Argo CD, but just in general with GetOps that we have the ability to scale at some point. So interacting, being able to put in payloads into an OCI registry opens up a lot in terms of application deployment, in terms of scalability, and then things like that. So from early on, I saw the ability for scalability and the ability to store payloads of things that we're trying to do at scale. We worked previously on this standard called CNAB, which is Cloud Native Application Bundles. And the idea of that was very similar. Basically, you kind of package it as an OCI, and you have this bundle that you can throw at any infrastructure, and it knows how to self-start and install and come up to the desired specs. So you could basically throw this at a Kubernetes cluster sitting on a container ship in the middle of the Indian Ocean, and it would just bootstrap and work. And so OCI can be abused to do lots of things. I don't know if it's an abuse, but it's an extension of the format. So obviously, we're all familiar working with Docker images. That's where everybody comes into OCI artifacts. But just the controls and the way that we think about the structure of those repositories kind of informs this. Because most people, when they want to ship a change, they don't think about shipping a manifest. They think about shipping a binary. So they're like, I built it. OCI ran it. It tested, and it pushed to the repo. After that, it should get deployed. And they don't necessarily think about the get-off side of the equation of, OK, well, actually, we need to version our manifest because we want to do rollbacks later if we need to. And they probably shouldn't have to think about that. Because ultimately, they just really want to get it deployed. In terms of OCI artifacts, I became a contributor to the Helm project to bring OCI to GA. It had been in tech experimental phase for about two years. So I joined the community to help really drive that to GA, which we did about two years ago. Since that time, I also became a maintainer on the ORAS project. So ORAS is the OCI Registries and Storage project. It is the underlying library underneath Helm and many other CNCF projects. So I know, and actually, ORAS is being used currently within the Argo CD project as well. So that makes it for more of a natural fit for that type of library to be the bread and butter underneath this integration that we're looking to develop and deliver. So going back, actually, one slide to our proposal here. Andy, you wrote this proposal. And I think we've got a little bit of it from you. But can you expand on your motivation on this just a little bit more, especially kind of with the proposal in front of people and the opportunity for them to really take it in? Yeah, so the proposal really is meant to kind of get thoughts from those who may be in similar situations that I've seen personally. I have worked with my customers and have seen as well. There will be another slide at the end that talks more about more of the concrete implementation of this proposal. Is that a push forward moment here? Can I bring up one of those scenarios that you were talking about, Andy? Yeah, go ahead. So we just had a talk from Michael Crenshaw and Zach, Aller from Intuit, talking about the problem of basically like how you manage your repositories. And in Intuit, the way that they do it is they have their Helm charts, their stuff in one repository, they render all of those, and then they commit those to a separate repository that is the source of truth, right? So with this proposal, you essentially would take that second repository and make that an OCI registry. Right, what I'm kind of envisioning and thinking about here is your CI process or some tool would go ahead and leverage some capabilities. We're going to be putting into the Argo CD CLI that will allow you to create that artifact bundle and allow you to push it to that container registry. We're going to bring this natively. So you don't have to think. You can just say, hey, I have a source of a directory source that could be a Helm chart potentially. It could be just a regular Kubernetes manifest. It could be customized templates. Those get packaged up and pushed to a container registry. And then the repo server would pull it down. The controller in Argo would then go ahead and render them the same way that they would be rendered from Git. And you get all the benefits of Argo CD in the same way. The idea is that we're just treating it as another application source. So if you can go ahead and pull from a Git repository, you can pull from a OCI artifact and then render them the same way. You can reuse the same plugins, the same everything. That's the goal. So my pre-prepared questions are going to make you repeat yourselves. I'm going to want to avoid that a little bit just for the better for the audience. Go ahead. I'm going to give you a mic. Please. I'm just wondering if then OCI registries are treated as the source, would it be possible then in Argo CD to integrate additional policies? For example, as part of OCI 1.1, you can add as part of, for example, a container image, a signature, and then it's all within. It's like a dependency tree. So would, for example, policies such as only pull this artifact if it has a signature, would that be within the scope? Or would that be implemented in external policy management with Argo CD? I think it could be involved. The security opportunities here to strengthen the entire providence of our GitOps manifest. We now open up an entire ecosystem that already exists in Kubernetes. And now we can bolt onto them. Now within Argo, we don't have to reinvent everything ourselves. We can go ahead and reuse a lot of what's already out there. You hit the nail on the head. Security is definitely an area that I'm thinking about as well. Anything else, Gus? Well, on top of that, there's definitely the security aspect of it, especially if we're using things like OCI and people are tagging things where you can have things like signatures and things like that. I also like to look at it from the perspective of building off of what Dan was talking about is that who here are using rendered manifests? Anyone using rendered manifest pattern? Yeah, so that's one of my favorite things to do. And OCI then becomes, like Dan was saying, kind of like the delivery mechanism, the delivery. Now we have a way to bundle our delivery into, hey, this is my deployable artifact. And I actually think that I actually did a talk at GitOpsCon about how I think we're kind of abusing Git in a way by using it as a database, using it as a read and write heavy database to where it wasn't really meant to be used the way we're using it now. And it just happens to be able to now withstand that. But I think with OCI and being able to render those manifests, if you're using rendered manifests into a bundle, now we're taking a lot of stress also off of Git as well. And I can also see a policy that we can throw into our configuration that says, if you have a wildcard that says, dev, don't deploy it. And certain other type of rejects is that we can apply to applications that says what type of content we can actually bring in and bring not. I also realized there also is a mic back there. So I think for questions, isn't that the one that I think? Yes, there's two mics. They can line up over there and over here for questions. I mean, I don't mind getting my exercise, but at least that might be more formal. Yeah, it might be more formal, easier to keep track of. I also need to keep us on time. OK. So you know, that is my job. There are two mics. If you can just head over there and start there. OK, so before we take this question, because my pre-prepared question links into some of the things we just had, and I want to get that out. So what limitations in Argo CD do we see running a foul of with the current vision for OCI? And it's kind of a path. What are the current limitations that Argo CD has that we're going to hit? One of them is going to be around immutability, because tags aren't necessarily, they are mutable. You can modify them if you have a 1.0. 1.0 doesn't always mean 1.0. One of the other benefits is that you can use digests, and those are immutable. So we'll see some interesting new patterns. And I'm hoping, personally, that folks get a chance to see the benefits of using digest shaws instead of tags, because of those concerns of immutability. That's exactly the same thing I was thinking. And the challenge with that, of course, is how many people are using image updater today? Argo CD image updater? A handful of people. Like that semver pattern of like, oh, I want to deploy 1.x. That makes a lot of sense in this proposal, because you could basically just say the same thing, and you've got the source of truth, which is awesome. But there's a tension there, because then you aren't using the digest. And so you do lose immutability. So you have to have some, I don't know, I think we need a better way to solve that at some point. You also bring up a thought of maybe using the similar concept with image updater for application sources to have it be floating. He was already thinking of new opportunities, I know. I know, right? He was fancy. All right, well, let's transition over to Mike there. Hi. About rendering manifest. Doesn't it collide with the concept of not using the Git but a storage and some kind of binary instead of code like Git? Because when you mentioned rendering manifest, it's one of the problems. It's not to have the transparency to see what was changed exactly. So that's really where it comes down to rendering the manifest at either the source or at Argo, because you can do it either in one of two ways. You can render them fully within your CI pipeline and push the OCI artifact. Or you can push the raw manifest, and then have those be stored in OCI artifacts and then have Argo render them the same way they render today. There are a couple of different patterns, and we'll see the patterns continue to evolve as we move this proposal forward. Yeah, and I think rendering manifests further earlier in the process. You're not then, if you're doing it there, you're not doing it at the Argo state, which means that you're actually, if you're doing it at the Argo state, which is happening now, you're actually mutating your source of truth, and a lot of things can happen. It should be safe, quote unquote, but you are mutating them when Argo renders them, versus rendering them earlier on and creating a payload, put that in a mutable system, like OCI, then have Argo read that as well. You lose some of things like being able to diff on the fly, but you can next see the diff either in the registry or on the Argo CD UI itself. There are certainly some valid use cases for why you wanna render them on the fly. You wanna inject secrets or other things, right? Don't inject secrets. Injecting secrets is bull crap. Stop doing it. Use references to secrets. I mean, use references to secrets. You're mad, you're so right. That's like super common, is that people actually do that with Argo. There are some plugins that enable it, and I don't wanna call them out because I don't wanna make anybody feel bad, but it's like kind of crazy that people will take vault secrets and just shove them into manifest and stick them into Kubernetes, because they're in plain text, people, hello? Like, are we? Okay, alright, alright, I'm gonna, as much as I so agree with you, I'm gonna be crazy pills here. As much as I so agree with you, we have two minutes and at least one more audience question. So you know what, let's go do that, that TED Talk over drinks. Go ahead. Yeah, not so much of a question, but more of, I totally agree. I created the request for customized port for OCI in Argo CD that is also linked in this issue that you have. So the sooner the better and this is exactly what already exists in Flux, so yes, please, it's the short answer. Yes, it does only use this out there too, so we're trying to bring a lot of that same parity to the entire ecosystem. So if you are interested. If you are interested in helping out. Three locations. Right. One of them is joining the brand new Slack channel that is now on CNCF Slack, Argo CD OCI integration. We now have an implementation enhancement proposal, so it's basically taking that proposal and putting the actual tasks on them and then providing your own input. You can now go ahead and engage in that and then just engage in the community. Yep, of course. Anywhere that you feel like getting involved in the community is a good place to get involved. So let me see, how did we do? Oh, we have like 30 seconds left. I'm going to call that on time. Thank you very much. Thank you, everyone. Thank you.