 Welcome, everyone. It's about time we start talking. So here we go. I guess that's the best option to start this. So, Janet and myself, who will be talking to you about the past decade of SIGAPs, Janet has been contributing to SIGAPs, primarily in the workload space since 2015, and she's been co-chair for SIGAPs for almost the same time. My name is Maciej, aside from being an active contributor in the batch primarily, but also helping with the workloads API. I'm also helping with the kubectl, and as of last fall, I'm also one of the Kubernetes steering committee members, and my first contribution dates all the way back to December of 2014. There are links to those contributions of ours, if you're curious about what they were. The last person that is in charge of SIGAPs is Ken, who was unable to be with us. Ken is actually contributing to workloads primarily, similarly to Janet. His first contribution was a significant refactoring of the stateful set controllers back in 2017. So the SIGAPs, if you look at the space that we are covering, we are actually covering three major API groups within the Kubernetes ecosystem. Those three API groups are batch, so if you think about jobs, cron jobs, the biggest API group that we own is around the app, so all sorts of stateful and stateless controllers, replica controllers, or daemon sets, which span all the nodes in your running cluster. And lastly, the smallest group that is actually one of the most important, which ensures that whatever disruptions that are happening in your clusters are not affecting your application's HA functionality. So what the early days look like, because like the title said, we want to go through the past decade of the Kubernetes project with a special focus on what the special interest groups apps was doing over the past 10 years, even before SIGs were initiated. So the first officially available tagged version of Kubernetes was in September of 2014. That was version 0.2, and it was released with a V1 beta API. There was no API groups at the time, only single group. Everyone we're using just that. It was beta one. And there's several few resources, and that's literally all the resources that the Kubernetes cluster had at the time. There's literally just pods, replication controller. If you don't know what a replication controller is, you can think that it's almost identical to Replicaset. We renamed it later on, but both replication controller and Replicaset are still available because they were G8 at the time back in the history, services, endpoints, events. That's probably something that a lot of you are familiar. And there's one very funny name, which is a minion. Does anyone know what a minion is? Cool. What is it? Yes, it's a node. At a later stage in the project, we renamed the minion. I still don't know why that change happened. I really like the minions. I checked earlier today. We weren't first. I really was hoping that the Kubernetes project came up with the name minions first, but unfortunately, that was already despicable. Me was released in 2010 from what I checked earlier today. And we were in 2014, although based on what the founders of the Kubernetes project were talking about, they were the primitives behind the Kubernetes projects were even long before that past decade or the past 10 years. It was even before that. So the day before, after that September 0.2 release and all the way until 1.0, which was released later in June of 2015, there has been a lot of change, a lot of turmoil. If you would look at the code base, it was shifting back and forth in so many places. Everything was being rewritten. The API went through a significant rewrite. So you can see in the on the slide, we quickly went through a V1, beta 1, beta 2, beta 3, the entire machinery around API conversions and the ability for users to pick whichever version that was suitable for them. All that was slowly happening in a lot of pain and a lot of hard time. Additional changes, like I mentioned, Kube CTL wasn't a thing in the initial version. It was actually Kube CFG. Out of curiosity, I was looking earlier today and it seems that there is a project which is called Kube CFG, but it doesn't have anything to do with the one that I have on the slide. If you look through the source code, I think that was around 0.6 or 0.7 when we renamed Kube CFG and rewrote it from scratch as Kube CTL and in the shape and form that you know and work with on a daily basis. And like I said, we did rename the minions to nodes at the same time. And lastly, from various different changes, there was a basic authentication was added because initially Kube did not have any security primitives. So like I mentioned, Kube 1.0 was released in July of 2015. At the time, we were able to scratch some of the API surfaces. So we released a fully supported V1 version with still exposed V1 beta 3 version, although a lot of people and a lot of examples were changed such that a lot of people should be using the V1. Out of the interesting changes that also happened at the same time, we already approached and exposed pluggable schedulers. So the slow we started building the primitives to allow expanding the building Kubernetes capabilities, although it was very in very limited way. And at the time, the pluggable scheduler still required you manually rebuilding the entire Kubernetes from scratch to have that functionality. And there were initial primitives around the basic admission plugins. So with that, I'm going to talk about the rest of the refinement. After 1.0, we have huge investment in the workloads controllers. So for example, on the check just talked about replication controller. So we start trying to add rolling update functionality to it. We started with using the Kube control rolling update command to experiment with the right behavior. And then when we are ready, we add that to deployment API. So deployment start controlling replication controller for the rolling update feature in 1.1. And then we add more workload APIs to support different types of workloads. For example, for the for the demon workloads running on each node, we have the demon set. And for the batch workloads, we have the job API. And then in 1.2, we add the replica set API because we decided that the replication controller name doesn't really resonate because it has a controller in its name and it's a controller. So we think that replication replica set is a better name. And also the replica set has a better support for the label selector. In addition to the equality based selector, it also support the set based selector. So besides the equality selector, you can also say that the operator is in a set or not in a set. And then we are adding, we added the pod disruption budget in 1.3 that allows you to specify the disruptions that you allow for your workload. And then in 1.3, we also added the patch set. So patch set is for the stateful workloads. So at that time, we start feeling that the stateless work, stateless workloads is not enough for Kubernetes users. The Kubernetes users need to run their databases, for example. So we introduce the patch set. And then that's for that stateful workload purpose. In 1.4, we also added the scheduled job. The scheduled job is for scheduling, running a job at periodically, at certain time. And then in 1.5, we renamed the patch set to Steffo set because the patch set name comes from the pad versus cattle metaphor, but it doesn't really resonate with everyone in the world. So we think that stateful set will be a better name. And then in 1.5, we also renamed the scheduled job to crown job because scheduled job is pretty long and tedious to type. And then also users can confuse the scheduled job with the scheduler. So we decide to rename that to crown job so that it's easier for people to differentiate. And then we also made a bunch of changes to the API. So like Majek mentioned, we introduced the API group concept. We started with adding the batch and extensions API group in 1.2. So batch group is for the batch workloads. And the reason that we're adding API groups is that you can have different APIs evolve at a different time. For example, you can have the batch workloads evolving from V1 beta 1 to V1 beta 2 to V1 beta 3. And then the other APIs can stay as their original V1 beta 1. And then the extensions API group is really for experiments. So a lot of APIs started with being in the extensions API group, including deployments, stateful sets. But then in 1.3, we added more API groups. So we added apps, API groups. And then we added we added the deployments, demon sets, stateful set into the apps API group. And then the pod disruption budget comes into the policy API group. And then eventually the extensions API group becomes deprecated. And in 1.1, we had the third party resource API. That's for allowing you to extend the Kubernetes API. But then that has some known issues. And we wanted to make some breaking changes to it. So in 1.7, we introduced customer resource definition. And that allows everyone to extend their Kubernetes API. So we also added a bunch of new features to the controllers. For example, in 1.6, in addition to the rolling update in deployments, we also added the rolling update functionality in demon set. So that functionality is directly built into the demon set API. So demon set directly interact with pods, which is slightly different from deployment because deployments interact with replicas set. And then in 1.7, we introduced a demon set history object for keeping the demon set history so that you can also use that history to roll back your deployment, your demon set. And then at the same time in 1.7, we are also adding the rolling update feature in stateful set. And stateful set also directly touch the pods. And it uses the same API, which is called controller revision for the stateful set history. And then in 1.7, we also added the max available feature so that you can specify how tolerant you want to be for the disruptions. For example, if you have a stateful set with one replica, then you probably don't want to tolerate any disruption, so you will set it to zero. But if you have deployment with more than one replicas, then you can set it to a higher number. And in 1.8, we added the backup limits to the job API so that you can set how many times you want to retry the job before you set it to fail. And then in 1.12, we added the job TTL controller so that you can clean up the job after the finish because that's a pin point for the users when the completed job just lingering in their cluster and take up the space of at CD. So at 1.9, we added the application, the apps API group. We moved all the workloads API to GA. And then to do that, we need to make sure that they are all consistent. The API fields will be consistent. The rolling update features are consistent. And the status conditions also represent consistent behavior. And then besides that, we also added a bunch of other features. So for example, in 1.4, we introduced the cap, which is Kubernetes enhancement proposal. This is the process for you to propose change to Kubernetes. And you just write something and explain what you want to do instead of just filing a PR directly for the change you want. And in 1.6, we also added a lot of improvements to the performance and scalability so that the cluster can have 5,000 nodes. 1.8, we introduced special interest groups, which is SIG. And SIG apps was one of the first 12 SIGs. And now we have 24 SIGs. And with that, I'm going to hand it over back to Jack to talk about the beta API that's there forever. Yeah. So as you saw from what Janna was going through the initial 20ish version, there was a lot of changes, a lot of controllers were being written. And then suddenly around 120, the SIG architecture, so one of the initial SIGs, the one that is responsible for the overall architecture of the Kubernetes, figure out that there are several groups in the API surface of Kubernetes project that are in perma beta phase. So they wrote together an enhancement, which described a clear requirement that any given API group cannot be in a beta phase longer than two releases, after which it has to either introduce another beta phase or be promoted to GA. We within the SIG apps were affected by this change, primarily by the Cron job API, which unfortunately, as you see on the slide, was in a beta since one point, if I remember correctly, 1.8. So that wasn't that bad, but there were worse offenders like pod disruption budgets, which literally since the moment that they were incepted, they were in touch. There were some minor changes in the API surface, but nobody found the time to and push this forward to make sure that this is fully supportable. And in parallel to that, a lot of users between whether that was 1.8 or 1.3 and 120 started heavily relying on on both of those resources. So we needed to do something. So I was pitching a lot of changes around the Cron job controller. And I found eventually after multiple conversation, a person and actually ally is over here. Ally helped me rewrote the entire controller. There is a presentation that I that I gave several years ago about writing controllers back in in Copenhagen from 2018. If your curious what it is to write a good controller, I totally recommend that. Just a quick recap of what was said over there. Previously, controllers were written such that we were always looking at the API and pulling what's going on. Over time, Kubernetes introduced a pattern that is called shared informers, where you just register your hooks and being informed about the changes to particular resources. So with that, we needed to actually rewrite and a lot of the controllers that we previously had, literally as the shared informers were created, they were battle tested and rewritten as we were talking. So the changes that John was talking around the API surface is one point of change that was happening. But in parallel to that, a lot of refactoring was happening in the code base. Unfortunately, the current job controller wasn't one of them. One of the reasons was that it required additional primitives that did not exist at the time in the Kubernetes client go library, which allowed us to to queue the the cron jobs to be processed at a later point in time. Several several mechanisms were added over time that allowed us to eventually rewrite this mechanism. So around 121 and 122, we were able to promote to first of all rewrite the controller, and it wasn't like a nice rewrite. We literally had to start start from scratch. There is a presentation that I did with ally during the pandemic. If you're curious about the details with regards to to the rewrite one of the the benefit of the rewrite was a significant performance improvement in the cron job controller compared to the previous one. But the other changes were also required. So I also mentioned the pod disruption budgets more than from Google volunteered and helped push the changes that were required. Thankfully for the PDB, we did not require a significant rewrite. There was only a handful of issues that had to be resolved because we've already gathered some feedback from the users over the so many releases, and we want to make sure that all of them are addressed. And by the time we GA it in 121, we were able to address those. So as you see the Janet a lot of the stuff that Janet was covering was the first phase of Kubernetes growth was around the workload. So primarily the stateful and the stateless workload controllers, whereas the post 120 ish. So the past I would say two or three years. A lot of the changes is actually happening in the batch in the batch surface. So a lot of changes primarily to the job controller, the job controller. It was GA very early on and Janet was talking about it. But it had a couple of flaws. One of that flaws that that I was aware when I wrote the the controller originally in 2015 2016 was the fact that the job controller needed the pots that were finished to still be around to be able to calculate it finishing status, which was a problem if you run a job with I don't like 1000 to or 10,000 or even 100,000 pots, because you don't want to have 100,000 pots lying around until the job finishes. So although over significant period of time, improve the functionality of the job controller, allowing to actually scale the the functionality of the jobs to even greater length, additional changes that the index job, that was actually one of the requirements when I was working initially with Eric from Google. Eric, I think once we put the job controller, one of the first issues that he opened up was like, we want to have index job. But it took us only six, seven next years, until we were able to pick it up and introduce functionally like index jobs. Or the fact that we started running, and we moved off of the stateless and statefuls because they were running just fine. And we started focusing on the batch. So all kinds of HPC workloads, or AI ML these days, a lot of those workloads are being run on Kubernetes. And suddenly we we've identified a lot of hurdles and problems that people were facing when they were trying to run these types of workloads. So we started working on stuff like elastic indexes, retriable pod, failures, back off limits per index and so forth. What interesting thing that I that I did mention in the previous slide, but I'm also going to mention here, was a cron time zone, snuff food that we had with cron jobs. So that also was one of the oldest issues that we had in the Kubernetes project, where people initially requested us to add the time zone capability setting in our initial response was that you should be using some kind of like UI to translate to whatever the time zone is on the queue controller manager. But we eventually agreed and we added after we were able to GA the cron job controller, we added an expanded the functionality. Thankfully, the library that we were using behind it was supporting the time zone functionality. So we build on top of that. Unfortunately, we all made mistakes. And the mistake that we did was we upgraded the library, which was all in fun. The unexpected bit that came out of it was that people will suddenly able to set the time zone on a cron job through an unsupported variable. So if you would use cron underscore TZ or TZ in the name of the time zone, in the spec schedule of a cron job at one point in time in Kubernetes in the past, I don't know, three or four releases back, it actually was able to pick up the time zone correctly, which was a mishap on our end because we didn't notice that the library that we were using for parsing the schedule actually exposed this functionality. So it took us additional couple more years to be able to prevent people from setting by first warning them that they are using unsupported functionality. And then in parallel to that, providing them with a supported way to setting the time zone. Because if for whatever reason we decided to change the underlying time zone parsing library, suddenly a lot of people would be broken. So it took us quite some time. I think in 130 or 131, I was able to finally drop this entire functionality. And currently, if you try to set this variable, it will fail validation on a schedule. But for first quite some time, yes, there was there were a lot of problems. But going back to the main topic that I was discussing about. So around 2022, May of 2022, we actually formed a working group that was devoted entirely to helping people run all sorts of batch workloads. Although we initially were targeting people from the research community. Over time, it turns out that a lot of folks that are interested in AI, although primarily the training side of AI, we're interested in what we were trying to achieve within the SIG apps and a lot of the investments and a lot of the growth within the batch workloads is currently being driven by the by by recommendation coming from this group. The group was formed not as part of the SIG apps, but as a entirely separate working group, because they are not only focusing on the API surface and the controllers, but they are also looking at stuff like the scheduler changes, or Kubla changes to support drivers like the DRA topic that Patrick was talking about earlier today in the keynote. So what is the future? And I think that's probably a question to all of us because we are here to make sure that you are able to run all sorts of workloads. We covered the apps, the initial phase that Janet was talking about that we allow people to run the initial replica set deployments. We added the jobs to be able to run tasks that require a completion phase, unlike the the state the stateless workloads initially, we added the support for stateful workloads. Eventually, over time, we expanded the surfaces for both the workload controllers and batch controllers. We are still working with the batch working group on expanding the capabilities. We will be definitely working with the the broad AI community to make sure that both the training workloads, which is partially covered by what the batch working group is talking about are working perfectly fine. But also there is the topic of running the interference, so serving the models, which is still an area that requires a little bit of more discussion, a little bit of more thought because based on the initial discussion that we've had, multiple people are approaching the same topic with different API resources that Kubernetes provides. So if you are curious, and if you have some interesting thoughts, particular use cases that you want to talk about, if you have problems, issues that you want to know how to resolve or share with the broader community, we are meeting every Monday at 6pm European time. There are other times on the slide if you're curious. There's also a dedicated Slack channel for the apps working special interest group. We have our own mailing list. So whichever medium of contact you prefer, we welcome all of you to join us to share your stories, whether that's a success story, fail stories, like we tried to do that today with Janet sharing with you all our successes and failures because we're not, I don't know, whatever. We're simple engineers that do make mistakes the same way everyone else does. And there's only a fine amount of time that we have to accomplish so many tasks that are that exist. And the number of contributors is a problem within the Kubernetes. So if you are interested in, and I'm fully aware that contributing to the apps work special interest group is not an easy feat. And it requires a lot of patience and a lot of personal investment upfront to be able to provide any even sometimes a small fix to any of the controllers. I still encourage you as much as I can and as much as I have my own personal time. I'm happy to guide a person through the initial contributions, like I have through so many years. You can ask ally. I don't think heba is around, but I was helping her to go through the entire CAP process, how to submit it, even though her change was rather small. I was helping her to make sure that she is comfortable going through a lot of initial paperwork that is sometimes intimidating. But at the end of the day, it is a required point because it serves as a documentation for us in the long run. So thank you very much. If you have any questions, we will be around here for a couple more minutes. And thank you. Thank you.