 I got the thumbs up to go ahead and get started. So this is Ingress Engine X in 2024 plans, but some of these plans have been going on for a few years, so we're excited to talk about what we are planning to attempt to do this year. So we'll go ahead and start with a little bit of an intro. Most of you probably know me if you've been to this talk before. I'm a Solutions Architect now at A Surveillance, maintainer of Kubernetes. I write those lovely release notes that you all don't read, I will continue to say that until I get more people reading them. I've also done a bunch of stuff around the networking. I'm the author of Networking Kubernetes, and I have an AWS course on a Cloud Guru. And yes, with that sweet, sweet beard, I do love to dress up as Gimli. So yeah, my name is Marco. You can find me as Gakko on GitHub. I'm SRE at Giant Swarm, already about 10 years into open source and working with Kubernetes since 2016. And I became a maintainer of Ingress Engine X project just last November. So besides that, I'm interested in Chrome and Canvas plans at home. Be careful, don't beat them by them. And we thank you for helping us fix the Helm charts. So if you have any Helm chart issues, it's his fault now. So we're gonna do a little bit of an overview of the complexity around Ingress Engine X as I like to remind folks why sometimes it takes a very long time for us to get releases out and all of the things that we maintain that aren't just the controller. We're gonna talk about the fun trip with Ingress 125. A little bit about Lua. Y'all are probably tired of hearing me talk about Lua. How many Lua developers do we have in the room? I ask that question all the time too when I get the same response. Two, three, Carlos, can you go get that guy? Thank you, a tiny bit? Okay, we're gonna talk about this new fun thing. I don't know how new it is, but it's new to me. Ingress JavaScript, everyone keeps asking about gateway API. Obviously, we're going to talk about that. We've got a lot. I know there was a lot of fun new announcements that are coming up, a lot of things that are going to V1 now as well. So we're gonna have a conversation around that. And of course, I'm gonna talk about the Helm chart. Mark is gonna talk about all of the awesome things he's doing to help make sure that we don't break things in the Helm chart releases anymore. And we have a surprise for you all this year. So I look forward to having a conversation about that. Go ahead, next slide. I don't remember. Oh. I forget the slide. So we have a lot that goes into the application that we have to manage. So obviously, the controller is written in Golang. It's on an Alpine image. We also have the Nginx versions. And there's a lot of code around the dynamic stuff with Lua. So there's a lot to maintain. We've got also, how many of you guys know that we have a QCTL plugin that doesn't work on Mac AMD64? Go on. We also help support two monitoring frameworks. We've got some Grafana dashboards that we put out there. Go ahead. Some third party plugins. So there is also, we also do support any Lua plugin that you would like to run. So for our build time, we also compile Nginx. There's a lot of options. There's lots of things. So we have a custom Nginx build. So we have to do that across. Oh, I don't have that as the next one. I have seven static configs. So 12 supporting container images, 30 Nginx modules. And that's the one that I like to talk about a lot. So what are three dependencies compiled across four architectures for three Kubernetes versions for Helm and all of our static deployments. So we have a static deployment for AWS. We have one specifically. I think there might be an Oracle one in there now. So if you can't use Helm, we have static deployments for that. 69 command line flags. There's lots that goes in here, 100 plus. So for each one of these annotations and command line flags, we have end to end tests for all of these. So there's a lot that goes in to maintaining this. And I have two core contributors right now. And we're gonna talk a little bit more about, we've been working with SICK Contributor X about defining the roles and the responsibilities so that we can have folks come in and maybe help support updating those container images, managing those, making those a little bit more efficient, maybe looking at, I mean, even just supporting the end to end tests, we have a container image that we do all of the testing, all of the builds happen inside of that container image and someone has to maintain that. So there's a lot which leads us into sometimes why it takes us a long time to get the things like this. So with all of those configuration options, those clouds, the clouds, this Helms, the versions, there's a lot of testing that goes into just this and getting this released out. So yeah, it's a lot of work and I just wanted to, again, be aware of that and please read the release notes. There were some breaking changes in that. How many people broke their clusters when they tried to deploy 110? Nobody? Good. When we talk about this, we look at some of the issues that we've had. So it took us, when IngenX got released for 125, it took us about nine months to get there. We'll talk a little bit more about that. Why? When we go talk about the lower functionality, I think the other one was 11 months and the last time we tried to do an upgrade, we actually broke things and we had a bunch of seg folds in our end to end tests. So we actually had to downgrade back to 119. So when folks are inputting things like, why did this happen or why did you do this? There are specific reasons that we do talk about a lot in Slack and in our community meeting to discuss why we're doing the things that we're doing. So we try to make it as visible as possible what we're doing. From those conversations, all of our community meeting notes are, we have those in GitHub and Google Drive and all those fun things. So when we get questions like this, we are aware that context is missed because it's really hard to follow those conversations in Slack. It's also hard to follow why we're doing things that we're doing. We are working again on getting better with why we do things in the release notes. But one of the big reasons that it takes us so long is that we are supporting a lot with dynamic configurations via Lua. And we depend on OpenResty for all of those libraries. And so we have to wait for them to support it and then we have to do our testing. So it does take some time for them to make that change and then for us to integrate that change. Here's all of the things that we support via Lua that I'm aware of that we've looked at. There's probably more, but most of the dynamic functionality is based on Lua. So I have a new number to add to that previous slide. Go ahead and go to the next one. So with that issue, one of the thoughts that we've had, I think I've had the issue open for about a year. We had two issues open, one looking into supporting Rust as a replacement or engine X JavaScript. Neither one of those got a lot of traction. So from this conversation, you go to the next slide. The widest possible community using a popular programming language. How many JavaScript developers do we have in the room? That's a lot more than the Lua ones. Not as many as I thought there would be. Carlos, write them down please. Yeah. But the idea with using engine X JavaScript or the rationale is that it is supported by engine X. So the thought would be that as they develop new functionality, it will be available with new versions and we'll be able to do upgrades faster. Now, that is also a large overhaul. If you go back a couple slides to the, there's a lot of functionality here. I think the next slide was the right one, sorry. I've seen examples of those two things. I have an X on here on dynamic plugins. We'll still allow dynamic plugins with the idea is that we're going to try to migrate or attempt to migrate the functionality with NJS. That's the one that we've decided on. I have to give Ricardo lanes to work on, otherwise he's going to go and try and do things like with things with wasm and, you know, he's going to still try to learn rust, but this is what the project is going along with the Lua functionality. So the thought with this is that in one 11, which is the next minor release, we'll have the NJS libraries available so that folks can start experimenting with this on their own. And I'm going to start opening up issues and start working at how we can start migrating some of this functionality. If that functionality is even possible in the current NJS libraries and have this conversation with Nginx. So we're working on this. So this is one of the big things that I'm working on and looking for help from the community. So if this is interesting, migrating from Lua to JavaScript or trying to do some of these fun things in JavaScript, please let us know. Again, we meet every Thursday, 11 Eastern for our community meetings and we are in Ingress Nginx Dev in Slack. So that's one of the first major things that we're going to try to accomplish. How many people have put in a GitHub issue asking us for Gateway API? Okay, that person isn't here. We've had a lot of ask around Gateway API support and we're going to start that implementation. So the idea with this is that as the functionality is available, we'll release alpha versions like we've done before. I think we released the 125 Nginx alpha version. We did it with CH route. We did it with the V1 upgrade from the network API. So once we're ready to do that, we will do a major version upgrade to V2 and we'll release this with Gateway class, Gateway and HTTP route support. So that support will be there. And again, asking for the community if there is functionality that you do want, it will be available for folks to implement and help us with that functionality. As folks ask for more, I'll open up issues for each one of these pieces of functionality and we'll ask for feedback on what we should be implementing from either from a maintenance perspective or the folks that are going to want to help with Gateway API. So we've got a lot of asks from the community around this and we're trying to get this pushed through. So my goal is to have a beta or at least be ready to try to do the V2 by Salt Lake City. Go to the next. Yeah, so that being said, so far, thank you very much. All of those changes James just told you will also affect the way we are modeling our Helm chart right now. And before we can actually implement them, we are kind of agreed that we need to improve the situation around testing our Helm chart and also releasing them. Some of you might remember this accidental release of 483. Well, that was one of those mistakes we currently have in our CI, but we want to work on that regarding testing. So we recently implemented the Helm unit test plugin that basically takes a manifest and you can write your test cases for that. We already have a bit of a coverage for some of our manifests, but first step would be implementing at least minimum tests for all of the manifests in the Helm chart. And that way, each time someone is opening a PR to our Helm chart, for example, they can just go and extend the tests we already have. It makes things a lot easier then. Second, we have integration tests. They are being executed by the chart testing, a CLI of the Helm project. Right now, there are some CI values around, but they kind of have the wrong naming pattern and therefore they are not getting picked up at all. And so in the end, I think we'll also need to reorganize the integration tests we have right now and make the match with real world use cases. So that's what I think currently takes around 45 minutes every time someone's pushing to a pull request. And maybe we can even improve that and get your feedback if your change is breaking something quite earlier. Overall, I think those changes to the testing thingies will improve the way reviewers actually can review our pull requests. In the end, you don't need to know each and every detail of how Helms being implemented or whatsoever. But as long as there are tests, I don't need to explain that to you, I guess. As long as there are tests, you can be quite confident that the change is at least not breaking something. And that is what we already have in the controller with all the, I think, more than 100 E2E tests. And I'd like to reach that in Helm chart two. Yeah, back to the release workflow. After changes got implemented, we want to push them out. And right now, there is kind of an optimism which just releases a new Helm release, a chart release, once someone's pushing the version or bumping the version in a chart, Yammer. That led to the before mentioned 483, which wasn't a patch release at all, but included minor, if not even major changes. Because we tuned the security context a lot there. And we just, in the last weeks, got a bit of feedback that this is somehow working, but in some user environments is also not working. And yeah, so in the end, what I want to say, accidental patch release, we weren't really good on that. And we want to do that better in the future. So how do we do that? First step would be actually having all the chart related changes in change logs. Right now, it's just a regular expression doing something like if the commit message contains chart, then put it in a change log. That's not how it should work. And after that, we also want to- I was just happy that the reg X worked. Yeah, at least it's a reg X. I mean, you can also pick them by hand. Yeah, in the end, we also want those change logs to be included in the GitHub release. Right now, you only see something like ingress.nx, Helm chart, version number, that's it. There's nothing about the changes at all. And I think here again, if for example, James is doing a release in the end, then it should be as easy as just doing some comments and then you got all the changes in the change log. After that, when we then have secure and reliable basis for testing and releasing stuff, we can also focus on actual feature changes or in that case, finally dropping pod security policies because they've been deprecated in version 1221 of Kubernetes, removed in 125 and those versions are already quite old, but we still have them. We have pod security policies in the chart. You can still enable them. I think for someone running on old Kubernetes version, that's really required. Okay, but in the near future, we should really drop them. What's the successor for them? There's a pod security admission and the whole pod security admission controller knows for now several so-called pod security standards. We are already compliant to the most restrictive one. It's the restricted one actually since version 49. That's the change we accidentally already introduced in 483, yeah. But this is not applying to change root mode. Since the change root mode requires some more privileges and therefore breaks the pod security standard restricted. In the end, so we are in a kind of a soft mode right now because we are supporting it. We are compliant to it, but there is no enforcement. So even if you have pod security admission controller enabled whatsoever, the namespace is not configured for it and that's something we like to change in Helm or at least the way you can then deploy the chart to your cluster. So the newly created namespace also and already includes all the labels required for the pod security admission controller. And since we just recently had some issues with provider specific stuff, we will also try to improve the situation around that. Right now, our chart, the real Helm chart is mostly supporting like each and every provider. Since most of the time you just need to add some annotations, for example, to your load balancer, but the static deployment manifests we provide right now. They are not so good at that yet. For example, on AKS or on Azure in general, you probably need to add an annotation. So the health check is finally working. That's not a curing when you're an external traffic policy local, but anyways, that's something we could at least document or even improve in the static manifests and also put some information about that inside the chart. I'm not sure we have a static manifest for AKS. We don't have them yet, but we will probably. Who's running Azure AKS with Ingress Engine X and would be willing to help write a static manifest for that? And the hands go down. Yeah, I don't think that's too complicated. At least I know someone who needs to support customers doing that, sadly. Yeah, anyway, so there is a lot of stuff around load balancer setup, security stuff when it comes to providers. AKS, for example, is also recommending you to turn off the automatic mounting of the service token. Side note, Ingress Engine X needs API access, so please don't do that. It's just breaking everything. So you can easily ignore that warning or yeah, we should add documentation about that because doing it manually, just to get the automation disabled, that's a bit difficult and I guess we won't support that because then users are like, yeah, I set it up manually. Yeah, you did it, but you did it wrong. Oh, okay, that's not our fault in that case. Anyway, so far for providers, I think this will be more important in the upcoming months and yeah, go ahead, next slide. That's something not really dedicated to the chart because it will affect the whole controller in the end, so there needs to be something done at the go level also. Right now, we have the container, we have only one container for the controller and the Engine X, they are all together and imagine you have a huge cluster, 200 nodes, 2000, 3000 increases, I know environments, where it's the case and yeah, in the end, your Engine X, your ingress controller will consume about three and a half gigabyte just for that static chunk without even handling any traffic and now imagine you're actually getting traffic like a lot of load and HPA is scaling up to 100 pots. Fine, then you have 100 times the three and a half gigabyte of static chunk. That's for example, one reason why we in the end want to split up the control plane, so the ingress Engine X controller and the data plane, which is the Engine X. Another reason for that are some security pitfalls, I mentioned change route before. For example, where we want to prevent someone who is actually getting into the Engine X can access the service account token because with that service account token of the ingress controller, they can basically access all secrets in the whole cluster since we need to access certificates and yeah, so splitting that up will also probably fix some security stuff, but in the end, it requires a lot of work and that's probably also something for version two and then. Yeah, I think that also affects, I think some folks have asked this for like rootless containers and they've also asked us for like read-only containers and it's a little difficult for that. I know it is possible, so those things doing this will help us with that perspective. So we're working on that, no timing because I know Ricardo has been working a lot on the 125 upgrade and I think this will be his next priority. I'll have to tell him and I'll keep showing him this recording that he has to get this done this year as I'm looking at the camera, Ricardo. Yeah, and one last thing for the wise, I guess we have separate dependencies. We already tried to, for example, remove call from the image many, many times. It's not working. And yeah, if we have different containers, I think that will not be a big issue anymore. So for example, for remote code execution whatsoever, if we only have the dependencies, the binaries and the image we really, really need and we split them up, then it's also going to be more secure for that reason. Okay, next one. And I think we also brought a surprise today, but that's what James is talking about now. So if we're going to implement Gateway API, it's not really Ingress Engine X anymore. So we haven't asked from the community that I think you all will actually help out with this one. We're actually going to rename the project. And we're asking for your help. And please don't make it Ingress McBody phase. So when we will do the V2 launch with the Gateway API support, is when we'll do the project change. And like I said previously, that is slated for Salt Lake City. That is also recorded. And we'll see how well that goes with the implementation. I do want to give a shout out to Nick Young and Rob Scott for helping us out. They've given us a lot of direction so far on the Gateway implementation. And of course, if you have any questions, comments or thoughts on that, we again are in the Ingress Engine X Slack Dev. Good. Oh, I have other thoughts. But anyway, one of the things that I did previously discuss is that getting involved is not just code. We talked a lot about our code contributions just now because that's what we are working on as maintainers. But we do need, like I said, there's not an AKS static manifest. There's lots of nuances to each one of the cloud providers. So it'd be helpful if we had cloud specific examples or if you're doing something interesting and weird with Ingress Engine X, please document it and put it out there for other folks to read and learn about. We'll accept those PRs. Documentation always needs refreshed, updated with new examples, new versions, things like that. I've already talked about the JavaScript developers, so we're looking at that and also working on with Ingress getting support with the NJS library. And we talked about CI, end-to-end testing and image support. We do a lot in GitHub. We don't do a lot in the actual cloud environments. So working on things like that, replicating issues. I think we probably need to start adding some more labels for cloud specific things and just getting folks to help out to test those for us. Because sometimes it can take a half a day to set up an environment to replicate an issue and you multiply that out by the last count, I saw 485 issues. It takes a long time sometimes to try out some of these issues. So we are definitely looking at for folks to help support and try out issues. And we're all working on, again, with Sig Contributex to finally grain define these roles and responsibilities to help out with the contributor ladder so we can get some folks added in as maintainers and approvers on specific things. So lots of ways to get involved. That's not just the scary lure, the JavaScript or the massive gateway API implementation. So if you have any thoughts or questions around that, please let us know. Go ahead. Like I said, Ingress Engine Xdev. Users is a support channel. We've got the new contributor doc, which I know needs revamped because I think it's just a straight copy from the Kubernetes contributor doc. We have our meeting notes and then we have the playlist where you can listen to us argue about how to do all of these fun things. And we'll start probably having design sessions to and reviews on the gateway API implementation. And that's the feedback survey. I think we've got, yeah, I knew it and a little early because Ricardo was in here to talk. He loves to talk about Ingress Engine X and what he does on Sundays with it. Any questions, comments, thoughts? I have stunned you all into silence. Awesome. Thank you. Thank you.