 Thank you everybody for coming. I'm glad to see good turnout. You're all probably interested in sidecars. Yeah, more people will probably keep coming, so I hope it wouldn't interrupt you too much. So, I'm Sergei Kanjalev, I work for Google and I'm chair of SIGNOTE. I'm Todd Neal and I work for AWS and I'm a SIGNOTE reviewer. So, sidecars. You know this construction wouldn't probably steer well. When I was a kid, smallish, I was spending time every summer with my grandparents and my grandfather has a motorcycle with a sidecar. Imagine sitting in a sidecar, there is no belt, nothing, wind in your face and so exciting. And I was small and imaginative, so I was like, oh, I'm steering a motorcycle. Oh, let's go left, let's go right. And unfortunately, when I need to watch my grandfather really carefully where he is going right, then I would pretend to turn my wheel and go right as well. But I was imagining that I'm steering the motorcycle. So, Kubernetes was feeling similarly to me. Sidecars were steering the pods. It's like, yeah, it's not right. We need to fix it. So, we decided that we need to fix it and now it's fixed. We will be, so now, yeah, no imaginative steering wheel that actually steers the motorcycle. So, let's go to the next slide. So, what are sidecars? Sidecars is a special type of container. It's a pattern that you will use when you implement your workload. So, let's say you have an application, you want to continue this application. It may be existing code, it may be something that you write specifically for Kubernetes, but nevertheless, it's something that implements your business logic, you deploy it and you want it to work. And sometimes you need to work and you need to communicate with something else. Like, sometimes you need to collect telemetry out of this container. You have different ways to do that. One of the ways to get a specialized process that will go run and like collect logs when needed. And you don't want this process to run inside your container because you may have different dependencies. It's much easier to just run a separate container inside your port and you're golden. Same concept with networking access. You may have different networking requirements and you want to deploy it as some sort of containers that already exist on the market or you build it yourself. Other things like security. Somebody will download certificate and keep it up to date all the time so you don't need to worry about it and data access is also an interesting pattern. We see some drivers being installed as a sidecar. And the reason people also install the sidecar, not only because they can control it more precisely, they can configure it specifically for your port, but also you can account resources for your port. So imagine this driver is not installed as a sidecar, is installed as a demon set. In this case, whenever your workload goes to the demon set, demon set becomes a shareable resource that may experience noisy neighbor problem. So as a container using up all the resources, you out of resources so you don't have your data access there. It doesn't work really well. So sidecar eliminates this problem. Now, if you run sidecar inside your port, you are responsible for that and you charged for these resources. So you can limit it, you can unlimited, it's your choice, but you are responsible for these resources. And it's a very powerful pattern in this regard. In fact, there was a talk on a day zero talking about some data access pattern for AIML that using sidecar as we developed specifically for this purpose because they wanted to attribute all this cost of encryption and decryption to the port rather than making system administrator pay for that. Anyway, this is sidecar pattern. And out of all the existing concepts that we have today in Kubernetes, we have innate containers that runs to completion, runs in order, and we need regular containers that start in almost a sequential order. And then whenever they all terminate, port will terminate. So in this model, it's quite easy to say, let's take one of these containers and make it sidecar. And it works. So it was working for many years. This article was written, I mean on previous slide, articles written on like 2000 single digit year, I think. Yeah, so pattern existed and it worked very well for web apps. Web apps are unique. They never end. They keep going. And whenever container crashes, it will be restarted. So it works perfectly fine for web apps. More and more, I mean, some people complain about sidecars in web apps because termination order is not declared. So if you want to gracefully terminate your port, then your main container may finish after your sidecar finish. So imagine you have a network providing sidecar and your container still needs a sidecar, but sidecar was already finished because there is no ordering termination. So there were problems like that, but this was edge case. Like nobody cares about edge case that much. I mean, it will crash a couple of times. Yeah, they know how it goes. But hopefully you are not that person. You want real sidecars. So we solved this problem. And another problem is that it was much bigger and it was in the face. Like it wasn't like this edge case, like when graceful termination happens sometimes, ordering is not right. This is a real problem. You put monitoring container in your job and the job never finishes. Monitoring still wants to run. It's running, it's running, it's running and sending zero metrics unless you will manually terminate this job. So another problem for long running jobs we experienced is if job is 10 hours and after first 15 minutes your monitoring container suddenly decided to allocate too much memory just out of blue and it was umkilled, it will never be restarted. So 10 hours you run blind without any monitoring, which also not ideal. So we wanted to restart sidecar. So many problems is jobs and we heard all these problems as a community. It was even before I joined Kubernetes. So I heard that they heard. I saw very old issues describing these problems and documents how to solve this problem and why it wasn't solved before. So let's get to the next slide. So to understand why it wasn't solved before we need to understand what's happening now in Kubernetes. So as you may heard on Keynote Kubernetes is everywhere. It's ubiquitous. It's a new Linux. So people running everything on Kubernetes. Many, many different types of workloads. And for all these workloads Kubernetes is pretty much works. It's okay. It's not bad. It can manage your workload reasonably well and you can hug things together so it will not even look ugly. So it's great. And the problem was that even for those workloads that we had, we didn't have it very reliable in Signode. So Signode experienced a problem when we implemented many, many features and suddenly we felt that those features are not all polished. Like we have a bunch of Permabaitas. Permabaita is something permanently in beta. We had a bunch of code that is not well tested and not implemented as conformance. So we needed to do something about it. And we started working on eliminating all those Permabaitas, cleaning up code, improving test cases. And finally we get to the very stable state. So now if you look at Signode and Kubernetes, we are at 40 feature gates and 19 of them is experimentation. So now we get into this like maybe one year ago. We get into the mode when we're not only deprecating and cleaning up stuff, but we're also experimenting and we're experimenting a lot. And main directions when we experiment is new workloads, especially EIML that are very popular now, but also like just before EIML they were HPC, which are as important as EIML. And also we want to understand hardware better. So hardware is not relative here, but new workloads are definitely something that we want to support better. So we've been on this good point when we are reliable enough so we can start experimenting and introducing new features. And we also really, really want side cards for new workload types. And when I say new workload types, I keep repeating about jobs and stuff. But also we have things like open telemetry that get so popular and Fluent Bead became such a ubiquitous part of Log's collection. So all these agents want to run inside your port and they want to run more and more inside the jobs that you're running. So to accommodate those scenarios, we really needed side cards. So again, remember this key, trying to steer the side card, thinking that it's steering the motorcycle. I want you to understand that side card feature is not a great new addition. It's not a major, it's not something that didn't work before. So it's worked before, especially for web apps. What we're fixing right now is fixing side cards for new workload. And specifically for jobs, when we will start, keep restarting these jobs. And now we want to start talking about implementation of side cards. And Todd here will walk us through the implementation. All right. So Sergey reminded us about why we want side cards, the problems that we had. And I'm sure everyone's probably tried to work around side cards before. I started writing bash scripts and containers so that you can try to control the order of their startup with like files or amount between them. And then do the same thing on the reverse side. And now you've got like this half baked system D implementation and bash in your pod that's really running. I see some shaking heads and everyone's like, yeah, that's fun to write. So how are things better now? So a sidecar is an init container with a restart policy of always. And we can all go get lunch now because I think that's the real one you know. The first thing you'll notice is like Sergey said, there's not a new type of container. Sidecars are just init containers that continues to run. And there's some neat things that come about from that. If you want to sequence your regular init containers with your sidecars, that just sort of falls out of the implementation. In this case, we have a secret fetch standard init container. It runs first because it's first in the init container list, then our network proxy can know that the secret fetch container has already finished and it can start up. Then the log sender can start up after and then finally all of our main containers can start up. And how does it really look though if you look at like the pod lifetime? So here, time going right towards, or towards increasing towards the right. Our secret fetch container starts, runs, finishes, and once it's finished, we're still in the init container section. We have a network proxy. It starts up as a startup probe, a pre-start hook. Once it's running, we're now ready to start our log sender. We don't wait for the network proxy to finish. We have a log sender now, has a startup probe, maybe a pre-start hook, or post-start hook. We run that. Everything's great. And now we can simultaneously start our main containers. Now we're ready for the pod to finish. We can stop our main container simultaneously. And now we have our sidecars. The log sender was the one we started last, so we terminate it first because there's sort of this implicit dependency that the log sender needs a network proxy, so we need to terminate the log sender first. And then we'll stop our network proxy and finally our pod is completely done and terminated. One way to think about this is that our sidecars, lifetimes sort of bracket the main containers and they sort of bracket each other and kind of get this tower of annoy looking figure of the pod lifetimes. And that lets you sort of make some assumptions about container lifetimes that really weren't possible before without doing a lot of hacky stuff on your own like we skipped one. So initialization is pretty standard but termination ordering is a bit more complicated. So we'll talk about it here. You have pre-stop hooks and then you have the stop container call that the CRI or is made through the CRI to actually stop containers. So with sidecars the pre-stop hook effectively tells you that, hey, the pod is terminating. Termination process has started. So pre-stop gets called simultaneously for main containers and sidecars. For the main containers, as soon as your pre-stop is completed we call stop container which ends up in a sig term and eventually a sig kill if you don't exit. But for sidecars that's not what happens. For sidecars we actually don't call stop container and you don't get a sig term until it's your turn to stop. So in this case log sender is pre-stop hook finished but we don't call a stop container and we don't get a sig term until all of the main containers are done. So your main containers can effectively rely on the fact that your sidecars containers will not be stopped until all of the main containers are completed. Similarly the network proxy, log container may depend on network proxy. So we don't call stop container, we don't send the sig term for the network proxy until the log sender has actually finished. So that's sort of how the termination works, a bit more complicated than initialization but it gives you some nice guarantees about the lifetime and the relationships between the lifetime of the various containers. And with any new feature there's always ways to use and misuse it. So we'll go through a few of those. Everyone's familiar with requests and limits on containers and should always set request and limits. Sidecars, still important. One thing and I'll get back to this in a few minutes but if you're trying to figure out like what are these sort of sum of pod requests for this pod like the how much or how many CPUs does this pod require and you go look at the containers like oh it has one CPU and there's a knit container but actually that knit container is a restart policy. If we look at the lifetimes those are both running at the same time now. So if you ever sort of made assumptions that hey I'll just sum up the main containers and compare them with the max of the knit containers that's how I get pod requests. That calculation no longer works so you really need to consider the lifetime of the pods and the resource requests and sort of take them into consideration with your your allocation planning. The great thing about sidecars is you get this serialized startup ordering but if your startup is really slow it can turn into one of the bad things about sidecars. So in this case our network proxy is taking a really long time to start up and because of that the log sender we don't even begin to start it until the startup probe and pre-start hook have or post-start hook have completed. In this case log sender also has a really long startup probe and post-start hook and what you end up seeing is that just sort of lengthens the time before we ever even begin to consider starting your main containers. So sort of a best practice minimize the time before your containers go ready so that we can quickly start all the other containers. Termination ordering or the termination sort of the same problem. There's a single graceful pod stop time period and it's shared between the main containers and all the sidecars. So in this case our main containers everything gets the start hook immediately we send sick term the main containers exits and then our log sender gets a sick term and it just ignores it. I don't want to stop and so what ends up happening is eventually we run out of time and once we run out of time we just quickly try to shut down the pod as rapidly as possible. So best practice exit your pre-start hook as soon as possible and then shut down fully if you're a sidecar once you get that sick term because you're a sidecar once you get a sick term you know that no one that depends on you is alive anymore so you're good to shut down to and that will let us quickly tear down the pod without exceeding the grace period. And then lastly just a few things. I said that the resource calculation sort of summing up requests for a pod has changed so I'm certain that there are some metrics or cost accounting tools that someone has written to like allocate cost to teams. Those are all going to be wrong in one two nine unless you go update them to accounts for sidecars. So sidecars are in the internet container section so just be aware that you need to go check all that out and make sure it works correctly. And then there's at least been one report of a mutating admission webbook that dropped the restart policy field on internet containers so if you have any of those be sure to test those out with sidecars and ensure that it doesn't break anything and other than that just go out and use some sidecars. Go back to Sergey to finish out. Yeah you may have heard that I mean we spend a lot of time drawing all those diagrams with dots like six terms here what if it's here would be better and we've been spending a lot of time debating and understanding what will work the best and we not did it alone we did it with many many people and those people not only signaled maintainers and signaled reviewers approvers code reviewers it also people who actually implement in this so we can confidently say that we thought about almost everything that you may have to think of when you implement sidecars so if you haven't thought of something or you think we haven't thought of something maybe we just it's just impossible to implement so some things are just not achievable in this physical world but I also want to talk about something what's next talking about sidecars you you enter the room and you say in sidecars and immediately I see it and people saying oh you know what I wanted from sidecar I want something else so what we decided to do we decided to take this approach with smaller footprint we said that we will solve specific problems with the specific field and we did so we did alpha in 128 we did beta 129 beta now has termination ordering that Todd was talking about in at length so behavior of 128 and 120 and difference a little bit so if you want to implement something that will be surviving till GA at 129 is your version and then adoption wise we already have for Istio blog post explaining how exactly you need to you can try out sidecar containers with Istio and GCS fuses one of the data drivers I was talking about they also had a sidecar and already have a commit saying like let's remove all this old code and we'll all get the sidecars and it will be much better for everybody so we have an option we have testing already so welcome to try it out as well and upcoming so we have one small feature that we also want to deliver to GA this feature will change behavior a little little bit most of people wouldn't notice it it's a behavior for restarts of sidecars while pod is terminating so imagine you're taking 30 minutes to terminate your pod for some reason maybe you need to uninstall drivers or like clean up major file which is huge in this case if sidecar still running we want it to be keep being restarted if it crashed so it's a very hk scenario but it's important for some use cases and we really want to cover all of them so the last feature we'll take for GA as a part of this cap but a part of this cap we also had more goodness that people asked about it's one point or another and this goodness may or may not be implemented there is no guarantee but it's something we hear a lot and we may want to think about it more as part of a follow-up on sidecars so things like security boundaries like right now there is no difference between sidecar and non-sidecar and people really concerned about it because sidecars may be envoy proxies have been injected and it needs a little bit of privilege to modify a network table and people may not want to have even like allow SSH access to this sidecar that this has a little bit more privilege and they maybe want to allow SSH to something that is containers that is less privileged and just using this network tables what else other things is resource reuse so there are two types of sidecars one sidecar is logging you want your HPC running on dedicated CPUs and all the logs and everything else running as far as possible to not interfere and other scenario is networking when you want to run your sidecar as close as possible to your workload so like networking calls wouldn't take extra hoop so you want to run on same numonote or something and then other features like crash back loop off and better um handling we all we hear all these features they are not coming as part of this cab but we may start working on it later and we also wanted to highlight something that definitely will not be part of Kubernetes anytime soon so one of the interesting requests we hear is when sidecar needs to be declared in some other asset that will be automatically plugged like attached to the port so port became like not a single yaml file but bunch of yaml files that will be constructed together on the fly so I don't think that will be that will happen anytime soon it's a neat idea though but it's not something that we actively pursuing right now and also we've been debating a lot because we want to replace like emulate system D inside the port so do we want to make port an orchestrator of containers so you can say one container is dependent on another container and dependent on third container so random specifically in this order and if container number two restarts you restart like start from very beginning you need to start the first one again we decide that we don't want to go this direction anytime soon so port will stay a single deployment unit you deploy main container it runs to the completion or runs forever but it doesn't have any orchestration inside the orchestrator so yeah that's something we wouldn't implement and there are a couple more features I listed here that is like percentage of usage it's another interesting thing that probably not happening anytime soon but it's a neat idea so if you have any questions we left specifically more time for questions because we believe that it's a hard topic you can go to the microphone if you want to leave feedback here you go if you want to run grab lunch we are not offended at all lunch is very important for your health thank you so great presentation thank you very much I have a question since you mentioned that you thought of everything could you possibly share your thoughts behind why is it filled inside an unit container and not a zone set of containers why did you make that decision to define sidecar containers basically piggyback on meaning containers so the main idea was that we wanted to make sure that something running sidecar is running while in need containers are running and we also knew that not every in need container wants to run with a sidecar so if we will put it in some other collection we will need to have coordination with in need containers after that and we didn't want this actively have it already right so the in need containers without restart always run as in need containers and only then the containers we or can you actually mix and match yes we can make a mix and match an order will be preserved so first container will run second container with restart policy will start up and only after it's fully start up second third container will run so ordering is preserved container could be just a simple in need container not a sidecar yes and it will run to completion before the next sidecar container will run okay yes thank you in 1.29 how are you planning to implement the order of termination is it depend on the order of in need containers that we configure yes the reverse order that we started them up in so the basically look at the list of in need containers of your sidecars and going in reverse is how they're terminated got it thank you piggybacking off of the first question right now the field set to like either never restart or always restart is there a more option is there a idea for adding more options to that restart field or is it just going to always be like never restart or always restart there are ideas not for in need containers but for in need containers basically support all of them like we have three never on failure and always right so we kind of support all of them I mean you can only specify always so rely on default and default will be either all but theoretically we can introduce three of them and wouldn't be a problem the ideas are for main containers for main containers beyond never on failure and restart some ideas of how can we say that container is so important that it will restart alongside entire port so it will basically terminate the entire port and restart as a whole port together so that's kind of new values it may be the same field maybe different field but that is new life cycle ideas that we have for ports all right thank you hey thanks for the talk I just wanted to ask I know you about service meshes I meant I saw that you mentioned that Istio now offers native support for sidecar containers do other service meshes not offer the support how does a service mesh handle requests and things from sidecar containers in general can you if you can speak to that bit so they're really they're just they're just regular containers in the pod we just start them earlier and then also define a termination warning other than that they just act like regular containers within the pod so if you have a sidecar that you're running as a main container now for the most part you can probably just turn it into a sidecar by moving into the net container set the restart policy always and you've gotten serialized and known initialization termination order so it should work out of the box okay I guess I was a little confused why you know what Istio means by native support maybe it's a better question for Istio but yeah so right now how Istio works it has a this injection proxy and for every port it will inject the sidecar sidecar is basically envoy proxy preconfigured it will inject it as a regular container so what they implemented is if they detect it's 129 and it has sidecar support they will inject it as a new type of like in it container with the restart policy always and it helps a lot because now they cover in it containers that they have better termination so I mean it's the same functionality but better okay got it thanks I appreciate it yeah I think like what a survey I had mentioned with Fuse it may allow some of them to remove some container or some logic that they're doing now where they're trying to sort of enforce this initialization order termination order themselves and just rely on the fact that that's how that's how sidecars work now so I have a question so once the port runs is it is it the default setting of the sidecar that it's going to run automatically once it starts so we will from a Kublet perspective we will start you you be inside car and then we will start sidecar and we will monitor its execution if it terminated or finished we will restart it and then restart it and then restart it so we will keep it alive for the duration of port being alive okay gotcha so if we're doing any type of a code change is sidecar can be added can be of any value to the main container can you repeat the question in order to do any code change will the sidecar will be of any value to the main container I mean doing any code change and testing it I mean if you have already sidecar implemented as a regular container yes I mean minor changes will be needed mostly changes will be needed in how you terminate so typically sidecars today will receive a sick term and we'll wait for a little bit to provide services for main containers now you shouldn't wait you just like terminate and you know that it will be terminated in the right time so probably you just remove a little bit of code but other than that it will work the same okay gotcha thank you so this kind of blurs my understanding a bit of like init containers and main containers so init containers now are just kind of ordered and main containers are running in parallel is there any like interest in making that maybe more flexible where you could define like layers of ordering and parallelization or yeah I think there was a on one of the slides on the things that are not going to be done soon anyway or like sort of implementing a system d like the container a depends on container b depends on container c and just build a dependency tree and figure out how to start up like this is sort of like a minimal api that gives us the ability to solve a lot of the problems that we've had with using sort of extend extended sidecar containers before without introducing a through a whole new set of features and maybe one more if you only have one main container is there a reason not to just use these init containers exclusively now or yes the sidecar containers do not extend the lifetime of the pod so okay that's the other sort of key thing so if that was your main container out so if you don't have any main containers and like it will terminate it wouldn't do it start okay thank you very much