 Hello, how's everyone doing today? Sure. Getting your steps in? Awesome. Well, I've got half a stick networking here and one of my founders, so I'm going to fail miserably. I'm looking forward to that. My name is James Strong. I'm a staff solutions architect with ChainGuard, and we're going to talk about what's happening with Ingress Engine X. You present myself. I'm Ricardo. I work at VMware, but I don't do Ingress Engine X at VMware. I do on my weekends. Sunday. So if you need a PR, ping him on Sundays. It's true. Works for me. I do a lot of stuff with networking and Kubernetes and things like that. Also, security and container images. Come check out the ChainGuard booth. We'll talk about CVE reduction and we have a raffle. I was required to say that. Yeah. I have already presented myself. I am also the creator of KewPook. I don't know if someone uses KewPook like the plugin for the application. Great. Antonio uses it. So you already got the sticker. No one is going to get the sticker. Just Antonio. Yeah. So our agenda for today. We're going to talk about the CVE review that we've had. So we've had some fun news lately. If you all are using Ingress Engine X, you probably know about this, but we're going to talk a little bit more about those particular ones. We're going to talk about our year-end review. So some of the stuff that we've done this year. Some of the things that were awesome. Some of the things that probably upset some of you. And we're going to talk about our 2024 plans and a community discussion. But real quick, one of the things that I want to talk to you all about is one of the things first and foremost is that we're all volunteers and it's very difficult sometimes to see issues, to keep track of all of the issues. You know, right now I think I have, last time I checked, I have 417 issues. And yeah, it's very difficult to keep track of all of this between the three maintainers that we have. So one of the best things to get our attention is to join the community meetings. Who's joined the community meeting for Ingress Engine X? Thank you. One person. But really, honestly, it's very difficult for us to keep track of all of them. And if there is an issue or if there is a bug, there's something that's wrong that you want to bring to our attention, please bring it to the community meeting. And we've got all of that linked later in the presentation. Slack is also a good place to reach us. I'm not sure if everyone knows, we have like Kubernetes Slack. There is a channel for Ingress users where you can make questions and have support. And if you actually are willing to get some new features or discuss some bug fix or something that really engages the maintainers, we have a development channel as well. So usually it's kind of faster. Yeah. Slack's a good entry point. And then the community meeting as well. So who has seen these CVEs? Yeah. Did you read all of the final articles? Don't read articles about your software, especially when you release CVEs. So we've had a lot of fun working with security on fixing these for the last year. So a lot of things that we've rolled out this past year have been to help fix these CVEs that have been coming. So a couple of the things did break folks, but we want you all to be aware that behind the scenes there are things that we are working on. So when we release things that look like they're breaking changes, they're also to help security issues. Ingress and Gen X and Ingress in general is a highly privileged controller. It needs access to SSL certificates and it's the entry point to the cluster. So we have to do as much as we can to secure Ingress. Lots of things are insecure by default. There's lots of things that you have to tweak with Kubernetes and with Ingress to make it a little bit more secure. And that's what we've been doing this past year. So a lot of the things that we've done, and I'll let Ricardo talk a little bit more about some of the security updates that we did to fix some of these CVEs. Yeah, so who is here just for the CVEs, actually? You can, I would be, like, just with the maintainers. So we've got two. That's nice, thanks. So if you ever got access to the Ingress and Gen X controller, like did a CubeCity OSX and got into the controller, you're going to see that a lot of things that you do on Ingress Object and the other things, we went up on the Gen X configuration file. So we have a bunch of Lua code and a bunch of templating, but some of those things, they're going to be on the Gen X configuration file, which is actually a demon running that can execute code on that container. Because we use Lua, you can also write scripts that will be executed. So first of all, when you trust your users to create an Ingress object on your Kubernetes cluster, you are expecting them actually to try things that they shouldn't be trying, right? So trying to inject code, trying to extract some information, or at least, like, you trust them and know that if they do that, they'll do probably for some good reason, right? When you allow users to run untrusted code, unchecked, you're going to have security issues. Yeah. So we've been trying actually, so the way that today Ingress controller is structured, and we promised that last time on Detroit, and I'm going to renew that promise. Sorry, folks. It's that because of that configuration file, what happens is you run everything together, right? So you have the same container running that reconciliation, going to the Kubernetes API, getting the certificates, and so on. That's the same one that's exposed to your Internet. And this is kind of dangerous. This is kind of bad, right? Because someone can just create something, some Ingress object that will end up being a Ginex configuration file that can end up being like someone from externally getting all of the secrets of your cluster, right? So the thing is that we've been trying to isolate that problem, those problems. Eliminating them is too hard today because we support a lot of annotations, as you may know. We've been trying to deprecate some, as you may know as well, and some things that are old, some things that are not used, and we've been trying to close and make sure that when the admin allows the user to really add Ginex configuration files using snippets, they know what they are doing, right? So you are allowing your user to create configuration files to write in Ginex configuration files. Do you want to do that? You are allowing your user to add random rejects on the annotations because you want to allow them to have random URLs. Do you want to do that? But because it always works this way, we cannot just go and say, okay, this doesn't work this way anymore because you folks are going to yell at us and say, hey, you broke my production cluster with everything that was working, and now I am not trusting them anymore because I'm worried. So that's what we've been trying to do in a way that, hey, we are just trying to make it a bit more safe, and if you want to run the risk of allowing your users to put configuration files, go for it, but that's up to you. I think what Rob's going to tell us is just to use a gateway API. Yeah. 2026. That's the promise. One of the other big questions that I get a lot is, why are you all running an old version of Nginx? I think I've closed about six issues that I've asked this. I get a lot of questions in Slack about this. So one of the things that we do, you see the list here. Ricardo said Lua. I'm going to say Lua. We use a lot of Lua plugins and a lot of Lua code, and because of that, we're using Open Rusty. We use Nginx. We can support that version of Nginx, because the last time we did this, it generated a lot of core dumps and it caused a lot of issues. We are going to work on an experimental build of some of the things we're going to talk about in our 2024 plan, but because of that, I wanted to be on record. This is why we don't support the latest version of Nginx. And just for the record, we use all of those Lua stuff because we need to allow you to hot reload the bag cans. That's why we do have all of these Lua stuff there. And now that we have some plans to remove that and maybe reduce the size of the image and then say, okay, folks, we can evolve with the right version. So 2023 review, about 501 PRs. That's a lot of PRs. I didn't get a chance to remove the Pinupot from that, but that's a lot of PRs to manage and improve and work through between 30 people. Now for a couple of friends from Sig Release helping us. Thank you, Carlos and Adolfo. We've had about 10 releases. We're on a pretty good cadence about one a month because of Curl. Thank you, Curl. We have to update Curl. We have to update all of the underlying OS dependencies. We are working on V10. We are going to be some changes in V10. How many of you read the release notes that I so carefully handcrafted? Thank you. Awesome. How many of you know that we also have a Google Dev mailing list where I also put those release notes out there and release information about? No? We have a Google Dev mailing list. Ingress Engine X Dev at Kubernetes.io Please join that. We put out release notes there as well. We're trying to do as much as we can to let folks know what is happening, what's going on. We have a Slack handle. We have the Dev mailing list. And we have the two Slack channels. Talk about some of those major changes that we talked about a little bit. I will let Ricardo talk about those a little bit more. Annotation validations. What are annotation validations? Basically, we never validated all of the characters that you added into your annotations. We could add a random code into that, trying to do escapes and it would pass. Now, if you have a Boolean annotation it will enforce that it's a Boolean annotation. It's a server name. It's going to try to enforce that that's like a valid FQDN and not like james.whatever.com quote drop database, whatever. That's it. Path validations. How many of you all use implementation specific? Yeah. It's very specific in its name. Implementation specific. It is up to the controller to implement that specifically. So we have to do some validation around that. Sorry. That was a little iteration for the day. Go ahead. Who uses exact or prefix almost as always as a path type here in the implementation specific on Ingress, maybe? Depends. So there was a problem when the Ingress API, another Ingress controller was being created. I'm looking at you Rob, because you're doing that. We as Kubernetes maintainers forgot to add some validations on the API and then we couldn't add that anymore because the API turned it into GAE. So if we do that, it would break our clusters. And then we realized that we allowed things like regexes on the path and they shouldn't be allowed. Then we allowed some other random characters and they shouldn't be allowed. So instead of adding that on Kubernetes API because it would break all of the other implementations as well, we started adding those to Ingress controller and it was part of this latest release and it actually was one of the folks that didn't read the release notes. And GRIP were moving the V2, so I think V1 has been removed or did I put that in the wrong spot? You're looking at me funny. I'm just trying to remember that one. Let's talk about some of the fun breaking changes that we did. The big one was disabling server snippets who got caught by that one who didn't release notes. Again, we tried to make it very obvious about a breaking change. We're going to talk a little bit more about the sem-verm, how we're going to interpret it from that perspective and what you would expect from us when we make a major, minor and patch release. So you can disagree, you can have philosophical debates, but at the end of the day we have to maintain some semblance of cadence with releases. And then of course all the CVEs. So anytime there's a Go CVE, there's an Alpine CVE, there's a Louis CVE, there's Curl, that's Love Curl. It's a great tool. Anyway, so we also have to put out fixes for all of those things. Release cadence, like I said, currently monthly we are looking to implement nightly builds. So being able to put a new version out every night or at some time something, some cadence really around just updating things. Making sure that when we do the APK update we don't have to worry about that. And then making the releases a little bit easier because we have those nightly builds and folks can test against those. And then just always more testing. I think I had a slide for the past three talks where we talk about just all of the things that we have to test over 400 end to end. There's just a lot of testing. And there's also been a lot of performance and there's also been a lot of cloud-specific questions. So working on trying to deploy Ingress Engine X in those configurations and test those. Because the cloud also adds a lot of other options and tweaks. So this is the thing where most people are either going to disagree or agree. When we talk about major, minor and patch releases, so I'll start at the bottom because I don't think anyone's going to disagree with me on this one. Patch releases are going to be probably 195. CVEs, we do those monthly. Dependipot, Golang, all of those fun things I just talked about. We go up the stack where people are going to probably start disagreeing which is fine. You can come be a maintainer with us and you can help change our minds. Alpine versions, anytime there's a new Alpine version, we'll bump, minor, we'll track it. Config changes. So this is the one that's going to break, that broke folks, that this is going to be the big sticking point for most people is that if there is a security change the cadence now is that we'll put it out there where it won't break and then another release coming out, we will flip it to the more secure option. We're trying again to make Ingress secure by default. That's also my tagline for my job. New features, as we add new features, changes in the helm chart, things like that and then anything upstream from the Kubernetes or KK. VBata 1 was the big one for us but that goes into the next category for major changes. So architectural changes, we're going to talk a lot about control plane, data plane split, Ricardo talked about it a little bit beforehand and then breaking backwards compatibility. I don't think anyone's going to disagree if we're going to break something from that perspective how annotations work removing annotations, things like that. We're going to break backwards compatibility from the API, we'll do major changes. VBata 1 was the big one and then the CPDP split where we're going to talk a little bit more about. Gateway API is probably going to be one. Yeah, Gateway API is probably, control plane, data plane split is going to enable us to do Gateway API so we'll have a V2 where control plane, data plane split and then most likely 2024, 2025. We'll get on V2 of Gateway API. As soon as my wife allows me to work on V2. So some of the fun things we're going to do in 2024 I'm going to hand some of these off to Ricardo. You want to talk about the TLS pass-through problem? Yeah, who uses TLS pass-through? Cool. Did you know that it's not handled by InginX but by the Go controller and you have really bad performances on that? Yeah. So why have we been doing this split control plane, data plane? We have realized that and was like, okay, we need to fix that. And there is a history on that. When this was implemented, it was part of the commercial license of InginX but now it's free. So we've been fixing that. There is a PR already ongoing. It's working-ish. There are still some things to be fixed. But the idea it's actually not only to fix but to solve some really old issues that have seen people asking for proxy protocol or people asking for some better performance or even I guess allow lists and deny lists. The big one that I think may cause some companies controversy that rely on some of the lure pieces is that yesterday in the contributor summit asked if anyone was a lure developer. Do we have any lure developers in the room? Do you have one? You're welcome. I'm going to talk to you. Someone make sure he doesn't leave. We're looking to migrate the lure functionality to NJS. InginX JavaScript library. How many of you are JavaScript developers? A lot more than the lure developers. So that's kind of the idea is that we want to be able to move to a supported language so we can get more contributors to help out with that. And it is also going to be supported by the InginX F5 folks. So we're looking to do that next year as we move through a lot of some of these. I could talk about the experimental InginX image. We have someone from the community who has tested it. They've done the build and we're working on getting that integrated. I don't think this is a PR. I think they just did it locally and tested it. But again, it's going to cause core dump issues and things like that. So that's why we're going to say it's experimental and we're going to put it out with the nightly builds until it is supported by Open Rusty. Want to talk about mod security? Yeah. Someone using mod security? Okay. We have some folks. I'm not sure if you know, but mod security was backed by a company, Spider Labs. They stated that they're not going to keep that anymore in 2020. They released it to the community to be supported. As a company, they won't be supporting it. Yeah, that's true. But for us, if you have ever seen the compilation of the whole InginX takes a lot of time, a bunch of C bindings and other things, it works on some architectures, some others. It takes some more effort to work. And there is this new project called Coraza from OASP, which they have implemented almost the majority of mod security parser in Go. And the speed, I think they have actually stated that the speed of this parser is a bit slower than the C one on mod security, but it works. And we've been seeing a bunch of other projects using Coraza as a replacement for mod security. So you have like an envoy binding, HAProxy ingress is using that as well. And we have spoken with them to say like, hey, maybe we can try to do that on InginX. And they said, yeah, cool. So we have different approaches to do that. But the idea is we think mod security for their efforts, I think it was like this is admin in 2004 or 2005, has always been like a great project. But Coraza seems like the evolution of the architecture for InginX as well. And then we keep talking about control plane, data plane, splits. So like we talked about, some of the security issues is because InginX and the controller are running in the same pod. We shouldn't be doing that. And we're going to work on splitting that all out. There's a lot of legacy code. There's a lot of legacy paths. I don't know if you again if you've seen our build we're doing a lot of stem linking, a lot of installs. There's just a lot to work through to break those two pieces apart. And trying to understand how we're going to do the communication sync because the ingress control is going to generate the InginX config and then data plane is going to just use it. So hopefully we'll be able to again break out the responsibilities and increase the security model. But again, this is also going to help us with the gateway API implementation and with that I'm going to go ahead and let you talk about that one again. Or we keep talking about it. It is definitely on our minds. We're always thinking about it. We always again the basic questions around CVEs, gateway API and the InginX versions. We don't hear them. We're always thinking about them. And the last one is reducing the container builds and third-party images. So we don't just maintain the container image for Ingress InginX. There's lots of other pieces. The end-to-end testing I think we've got about 11 or 12 container images that also have to be updated all the time. I think I have a PR open for which one. The certificate book has a CVE in it right now. I know that. So again, reducing those container builds, reducing all of the dependencies that we need. One of the things while we were investigating, can we get rid of curl in our image? If you know anybody who works for what's the dependency, the max database? It's a GUIP. Oh, GRIP. Please don't set up a CRON using curl. Anyway, so just trying to reduce all of the third-party dependencies that we're using again to help reduce that attack surface. That's a lot that we have planned. Again, InginX, control plane, data plane I think are the two major priorities because they will set us up from a secure perspective and it will help us with some of these other pieces. Community aspect, as I talked about we have the dev support channel, we have the user support channel, contributor docs. We also maintain all of the documentation for the annotations and all of that. We do need folks to help out with examples. So if you're using the new version of the AWS Cloud load balancer and you want to test it out and write up a documentation on it, external auth, any type of weird corner edge case that you think would help other people, please go ahead and document that and work with us on getting that out there. Like I said, community meeting and notes are in there, they'll be posted and we meet every two weeks to talk about CVs and a lot of these issues and with that I think we're good. I don't have a last slide so I guess we can open it up for questions. We went through that a lot faster and I thought we were going to. We got 10 minutes to have questions. I mean if you're here I asked. Questions, complaints, you can just come here and beat James, I don't care. No questions. We are going to implement Gate to API if you are thinking on that question. So we're doing a great job. Cool, thanks. There is one there. One question. Don't know if we have a mic. Oh, we have the mics. Ricardo is going to come to you. On the last slide you had talked about control plane and data plane split. Can you please speak to a little bit more about it? Once there will be two different containers for controller and the proxy will that have its own set of challenges or right now it's a little bit messy so can you just elaborate on those points? Everybody hear the question, I can repeat it. You just wanted to understand a little bit more about the control plane data, plane split and again I will let Ricardo talk more about that. Yeah, so let me come here because I'm short. So I'm not sure like as an example if you have ever seen like a Qproxy or like other other architectures where everyone connects to the API and they do the programming, right? So Qproxy as an example has some experimental project that we've been trying doing that split to reduce the burden on the API, right? The idea is the same. So you would have some like two or three controllers they keep connected to the API. They do the whole of business logic like get all of the ingresses, get all of the secrets, get everything and then those control planes they go to the data plane and say hey look you have this new configuration. The data plane actually can be connected back to this control plane, right? But in fact this control plane is not going to be Kubernetes API anymore, it's just going to be like a QE or like a GRPC service or something like that saying okay there is this new block of configuration so data plane just go and configure that, right? So they are not connecting to Kubernetes API anymore which is kind of what we are trying to really achieve and because also watching Kubernetes API it's kind of expensive, it becomes expensive when you scale a lot we will have just like one thing or like three controllers watching those the API and all of the objects ingress, secrets endpoints, endpoints slices name spaces and generating the right payload for the data plane. One of the biggest things that it will help out with is that ingress itself needs access to the TLS secrets and Nginx doesn't really, it just needs to know where they are at on disk so being able to configure those and grab those isn't the ingress controller piece so again it's about making sure who is responsible for what and making sure the access controls are there because right now the ingress controller, the image has Nginx installed in it we want to break those out and make sure they are two completely separate containers. And the scaling of these two will be separate they could be separate because they will be separate pods will be separate deployment so it helps out with that perspective so you will only need three controllers and then maybe you will need 30 Nginx so you can scale the data plane separately, another benefit. Right now you would need to scale both of them. Just to be clear, we are going to start simple because trying to break that way that I told you was pretty hard. We did that it passed almost all of the end-to-end tests but when you get some problems like keep alive losing connections and other things doing that GRPC stuff it was kind of annoying getting the connection established again and so on. So we are going to start doing simple in the sense like you have one pod and two containers but the container that is running in Nginx should never be able to reach the API server. Maybe mounting the same volume or something like that and based on that we are going to start now breaking who does the reconciliation who does just the configuration and the reload and so on because the huge break was going to be there is a PR that if you want to take a look it is like 6,000 lines or something like that and a lot of changes and almost everything passing almost if you want to test it. Yeah, okay. Anyone else? Join the community meeting and we can talk more about it. Rob, you are out loud. That's a payback. I promise no heckling. Actually though you mentioned that you have opportunities to contribute and there are a decent number of people here. How would they get involved? What are the ways that they can really like are there specific concrete things they can start with or that I know you have got a lot to handle. Yeah, we probably didn't do a very good job of explaining walking through that. I know there are some old examples on the documentation that don't work that probably need updated. I know we removed off of HTT bin and I think some of the examples are still using HTTP bin. Little things like that. Again, corner edge cases for uses. So if you are doing something off the wall please provide that example. We get questions all the time for the static deployments because right now I think we support digital ocean, AWS, things like that. So if you are using a specific cloud provider and there is a static manifest for that that could also be helpful. There is a good example like on HelmChart as an example. Our HelmChart is very complicated. Yeah, but we have some folks from Giant Swarm helping us to fix and it's kind of amazing because they are using production and they say like hey we have all those bugs so can we just keep fixing? Yeah, sure. So we figured out we've been trying, we've been figuring that there are some problems that need to be fixed and we know that they are like a consulting that does a lot of things and they actually deploy in production and the truth is that I don't deploy in Giant X in production because I don't work anymore with platform engineering. So I don't know what's going on and I think that's something that actually sometimes we miss Kubernetes developers like we don't use in our day to day things that we've been using so we need those feedbacks including Ingress. So that's great. Great question. I have a question for Jamie. So as a Wolfie maintainer, have you had any luck in using Wolfie base OS for Engine X Ingress? Come to my booth and we can talk about it more. But the answer is yes. We do have an Ingress Engine X controller on Wolfie. Anyone else? Thanks for coming. Thanks, folks.