 Okay welcome everyone this beautiful afternoon. I'm happy to see so many of you join me for for SIGUPS updates. So who we are? SIGUPS updates is being glad currently by three people. That's myself, Ken and Janet. I know that Janet is looking is joining us virtually. Kenneth unfortunately wasn't able to join us, but those are the three people that he would talk to or that you would encounter if you join our bi-weekly calls. One of us will be usually writing a call and if you're trying to submit proposal those will be also the people that you want to talk with about getting the approvals for the proposals. So like I mentioned we do meet every other Monday at various times. I put them on the slide since we are on the east coast that will be in at noon. The next call will be happening on November 14th. So if you're interested in presenting any particular topics for SIGUPS feel free to pop up and present whatever you're interested in or whatever you're working on. We're more than happy to listen to your use cases, your problems, your issues or even if you're having a PR that hasn't been touched or you're awaiting review, you're more than welcome to join our calls and bringing up to our attention. There's a lot of us, there's a lot of the work, but there are only a couple of us that are pushing the boat forward. I encourage you to also join our Slack channel. You can ping us there on the previous slide and I can probably return or maybe even two slides back. There were our GitHub and Slack handles so you can probably figure out who is who. Alternatively if not Slack or meetings there's the last option is email group. If you want to contact us this way, this is your preferred way of contacting with us that's totally fine as well. So what do we do, what the SIGUPS does? I've been getting and answering that question since the beginning of this week is SIGUPS actually still doing something? Or is it stale? And the answer is yeah we are active, we're actually very active even though the majority of the work as you'll see in a minute is driven by the output from the working group batch. There's still a lot of stuff that we're trying to to push forward with regards to the controllers and the applications that are running on OpenShift. So that's that. Let me quickly before moving forward, let me quickly go through what we've been working on for the past two, three releases. What are the main issues and the main features that we delivered? So the G8 features that landed in the past two releases that are worthy of mention is index job. So if you've ever worked with job by default it will just create this many pods as you specify but there is no ordering. You can't have any particular information about which pod is picking which index. So if you're thinking of something like a combination of a job and a stateful set that's an index job. Basically every pod has a specific index assigned to it and if it fails it will be restarted with that number. That number is also injected into the pod so if you want to use that number for processing your data that's perfectly fine use case for that. Another feature again on the job API that we've been pushing forward is job suspension. We are seeing a lot of batch workloads and I'll be talking about the batch group which I already mentioned as well before. A lot of those inputs around the job improvements is coming from the batch workloads and they are trying to produce primitives within the Kubernetes API especially the workloads and the batch groups that will allow to create a higher level controllers which will be capable of driving the batch execution in a much smarter more sophisticated way. So basically the job suspension is nothing more than being able to stop the execution of a job. Whatever was running will be stopped and then it will be restarted as soon as you unsuspend the job. The other two features which we were basically aligning various controllers some of the controllers deployment was probably the most featureful when it comes to various rollout strategies controller and the other controllers like daemon sets or stateful set were missing some features. So for daemon sets we work on something that is called Mac search which basically allows you to tell oh that during a rollout I'm actually allowing to have more pots than the nodes being rollout during the rollout which might be useful of course a lot will depend on what is your particular use case because with daemon sets obviously you need to be very careful because they are running one per each node so if a pod is using specific resources on the node which are limited then obviously you won't be able to fully reuse that mechanism. So have a look at it. It can improve some use cases definitely. The other leveling up feature for controllers that we did was for stateful sets we introduced something that is called min ready seconds which basically allows you to tell oh my pod my application has to be ready and running for this many seconds before I'll add it to the service and can serve traffic to it. So this is more for use cases where either your bootstrap of the application is taking some time or you want to warm up your caches for your application and give it a little bit more time than normally you would. So you can delay when the application is joining the stateful set. The better features that we promoted over the past two releases the first one is related with deciding which pods should be removed first whenever a scale down is happening in a deployment. We kind of left this one hanging for a little bit for a little bit while we're looking for more input on this but basically the current built-in mechanism works such that it will pick the youngest pod in your deployment to be scaled down which is not always desirable for some use cases. So we would like to have a user provided input and an information point how to scale down your deployment. The other one is again jobs like I mentioned work group batch has been pushing us quite a lot with regards to improvements. It's basically exposing information about ready pods. So when you're running batch workloads and going in thousands or more than a hundred and thousands pods per job it is useful if you can have an information about which of the pods are in which states. Beforehand we only provided information about active which is basically anything that is running or is pending. There was no distinction between actually executing pods in a job so that feature exposes that information and a status. And finally the time zone in cron job which is probably the most requested features within Kubernetes literally dating back to the first implementation when cron jobs were called scheduler jobs. We've been for very long not maybe not necessarily not interested in that but and not a good word. We were blocked technically from providing this functionality because we would have to rely on user provided or cluster owner provided database with time zones and we did not want to and to provide some that kind of enforcement for cluster administrators. As soon as Golang started providing their own built in database with time zones we figure out that yes that's a point where we will always have the information about time zones present on the cluster even if it will be built on top of a scratch images it will always have that kind of information provided and we can rely and in the worst case we will rely on the Golang built in database. So that functionality is present since 125 it was promoted to beta with 127 hopefully I haven't seen any complaints yet but if you try it out and you don't like something let me know I'm very happy to hear about that. We don't have anything in alpha which is a little bit surprising when I was presenting when I was preparing the presentation I thought that we had something for alpha and we do but it's something that we are starting to push to alpha in 126 but I'll get to that in in a minute. So the stuff that we're currently working on if this is something that interests you I'm more than happy to hear either after the presentation or feel free to find all the links to the appropriate features and the Kubernetes enhancements repository which has then the further links to the implementation details feel free to to follow them and leave your information about that. So one of the features or a downside of the job implementation was that it required the pods of a job to be around for the controller to be able to figure out oh I completed the job because I was looking at the at the pods that completed. We didn't have a mechanism smart enough to count the pods that were completed and then yeah. So the problem was basically that if you had a job which went into thousands or more than thousands of pods and a pod garbage collector which is responsible for removing completed pods would kick in we would lose the information because the pod garbage collector is responsible for removing pods which are not needed which basically if it completed it is not needed any longer. But since the controller relied on calculating so it would again and again create the same pods over and over again. We knew about this limitation from day one when we started implementing the jobs and this was the issue was created to fix this problem as soon as possible and as soon as possible was since one 1.2 or 1.3 I can remember when we introduced the job it took us almost 20 releases to to push the feature although did a great job introducing a finalizer which is responsible for counting the pods which completed and we don't have to keep them around which is a great improvement especially again for the batch work clothes which are running in multiple and thousands or more pods. There are two beta features that we're working on for 1.26. One is about giving the users the ability to decide what happens with their PVCs. Currently by default we are not touching your PVCs in the stateful sets because we're trying to treat all the data owned by PVC owned by stateful sets as a something that has to be maintained but there were requests there was a couple of users that requested these features. If I remember correctly Mark is working on this feature and basically there is the ability to say and stateful set spec that oh yes I'm fully aware that if I will be scaling up or scaling down specifically it's okay for the controller to remove my PVCs but it's a explicitly opt and you that's not a behavior that we will be changing by default in a stateful set. Another feature again coming from the batch work group is retrieval and non-retrieval failures for jobs so currently a job controller is rather dumb in case when it comes to failures it will retry every single failure of a pod. If a pod will end in a non-zero with a non-zero exit code it will always retry such a pod even if the failure will be permanent I don't know if there will be some execution errors or something like that it will always retry so that can at some rare cases it can cause cluster exhaustion if we will be retrying the same pods over and over again. So the feature is about giving the users the ability to specify oh these are actually the exit codes or information that we embed in conditions so for example if your pod gets evicted by a scheduler because of not sufficient resources or by the kubelet for similar reasons you can say that okay I'm expecting that this might happen in some cases and you don't have to retry those pods similar for exit codes if you write your application in such a way that you have nicely codified exit codes and you know that everything up until let's say 100 is a retryable error but everything above 100 exit codes is it should not be retried because it's a fatal error that it won't help and we will be just retrying forever and wasting resources basically. So that API actually adds that, adds those capabilities to the job API. It's pretty it's pretty big I'm fully aware because I've been reviewing that a couple of times since the very early alpha version we in the beta the API transformed a little bit we're hoping that it will be a little bit more expressive but yes definitely give it a try if you if you've been planning any work related with retryable and not retryable fairs for jobs you've been running some batch workloads that code that could benefit from this functionality I'm more than happy to hear from you especially be that this is currently at as it will be as a beta in 126 so it will be on by default but what is most important I would like to get at gather as much as possible feedback before this feature goes GA because at a beta stage we still have quite a lot of flexibility when it comes to changing the API and I know that there are some edge cases which might be problematic to express in the API that we currently put together and there's a bunch of alpha features that we were trying to put together there's a pot healthy policy for PDBs which is basically we discovered recently that the PDBs are always counting every pot doesn't matter whether it's a terminated pot it's a pending pot or it's a running pot it will always count all of those pots in your PDBs which might be a problem for some cases if you know that a pot is terminated you might not necessarily be interested in taking that pot into your account during PDB calculation so there is a new spec field being added to the PDB which allows us to to have a little bit more wiggle room with regards to which pots should be calculated for PDBs the next one is about stateful set numbering so if you've ever worked with stateful sets you probably know that stateful set will create this many replicas as you specify and it will start counting them from number one until whatever the replicas number you have but there are some cases if you're thinking about migrating as specifically migrating your stateful set between clusters you would like to be able to move pots one by one or in groups I don't know group of two or whatever your application is capable of doing for those cases we will currently allow you to set the initial number of your stateful set so you can say that oh my stateful set actually should be from 3 to 5 and we will be creating pots that will be numbered from 3 to 5 and not from 1 to 5 as in a default case lastly which is something very dear to my heart and we've been trying to push this feature for for a little while now and probably it will take a couple good releases still in the future is consolidating the controller's statuses and conditions so controllers all of the controllers were written by different people I'm trying to think I don't think there was a single person that wrote at least more than one controller usually there were various people at various stages of Kubernetes maturity and the downside of that approach was that we did not have unified approach how to express the information about where my workload is in its life cycle which if you're working with just for example deployments you probably wouldn't have to care but if you're a person that's trying to build something on top of the current controllers then if you have to codify separately for deployment separately for stateful set separately for for daemon sets to differently read what the status means what the condition means it is getting a problem it is becoming a problem because you need to know what controller you're working with to be able to to properly operate it at a higher level ideally we would like to have the ability to say oh if an app is ready it's ready it doesn't matter whether it's a stateful set it's a deployment daemon set we should have a way to say in a consistent way that yes deployment daemon set stateful set is ready and I can fully rely on that that's not a thing currently and we've been putting together proposal we've been discussing on various spaces we've we've raised this topic a couple of times during the sig apps calls we've raised this topic with API machinery and API reviewers because we are trying to add to the existing API which is probably the biggest issue at this point in time so if you have your opinions I'm more than happy to hear about them there is a cap open about this issue feel free to have a look at it leave your comments there every kind of input is more than more than welcome before I close the call we still have a couple about a couple minutes and I want to leave a little bit of time for questions there are two topics that are very important for the overall stability of the project and controllers as well I point I'm pointing and you can get the links if you open the slides which are already uploaded to the sketch one is about the increased reliability reliability bar so that basically means every single PR don't be surprised if you will be pushed back with asked to add more tests for your PR that is fixing some area even if even if you're touching some area and not necessarily the test that you are being asked to provide are touching the stuff that you're modifying the reason for that is there are some areas controllers are roughly okay ish in some areas that I remember but there are some edge cases that we are not we don't have them covered as as much as we would wish in the in the unit test especially because that's the simplest feedback that we can get when you are changing any kind of controller and the more confidence we have the better obviously for us if we can deliver features because if if reviewing a particular controller requires me to set up a cluster with your change to ensure that you haven't broken anything that obviously takes a lot of time to be able to to review every single PR so that of both of these topics are basically talking about that lastly I was talking about the reviewers cohort that we started a couple months back and I see Haba here Haba is one of the folks that we are trying we're trying to mentor and teach how to review the controller PRs it's not an easy thing I would say that's one of the toughest areas still in the Kubernetes code especially that eventual outcomes or the the effect of any change might might triple to so many users as we've heard earlier today during the keynote so with that I think that's roughly oh yes the batch work group I did mention that a couple of times throughout the talk earlier today so the batch group work group is basically a combination of several SIGs that we joined efforts to improve the overall ability to run batch workloads so that's that's a cross-cutting work from SIG apps SIG scheduling and SIG node because any kind of batch related workloads affect those three areas and we're trying to improve the output of all those three SIGs we've been having folks from various different organizations from various products presenting about their use cases what they are missing on cube why they are building something else on top of Kubernetes APIs or what kind of things they are currently missing in the Kubernetes API that we can pick up and and improve in the long term and add to the to the core Kubernetes APIs so if you're interested or if you have any particular topics for the batch work group feel free to ping me or any of the other batch batch leads or just pop up into the mailing list or the Slack channel we are meeting every other Thursday and it's a 10 p.m. Eastern I think that's wrong that should be 10 a.m. Eastern sorry I'll fix the slides later on yes it's a 10 a.m. Eastern time so with that I think if you have any questions I'm more than happy to answer them if you're shy or anything like that I'll be around for a little while and I'll also be at the Red Hat booth during the booth crawl later today so feel free to say hi or anything okay thank you very much