 Hello everyone. How is everybody doing? It's great to have a lot of moderated people here. This is maybe a bit of a mouthful. Let's start first with a quick introduction to two of us. My name is Alois. I'm working for Dynatrace, and at Dynatrace being responsible amongst other things for everything we do open source-wise, also a number of CNCF projects like Open Feature, Open Telemetry, and Captain. I'm also co-chair of Tag App Delivery in the CNCF, and a lot of this motivates the target I'm having, or that we are having here today. My name is Alex. I put my full name there to distinguish myself from certain news headlines. I'm an engineering director at Canonical, so I work on micro-kates, charm-kates. We build upstream Kubernetes, but I'm also on the governing board now for Open Feature and a contributor there, and also work with Alois on Tag App Delivery. So, before we get started here, there's a slider link. You can actually use this slider link. We will have a couple of polls as part of the slider, and you can then also interact with us, as well as slide on top of the chat as well. So, yeah, just take your phone. I'll leave it here for a second. Yeah, give it 30 seconds, and I'll switch over. Okay, everybody should be fine right now. Let's use some magic. Okay. Yeah, first question. We're really trying to make this a bit interactive. How many of you are familiar with skit-ups? And by familiar, we think not just knowing the concept, but at least having worked with it, or maybe using it in production. That's how it goes. It's a good distribution. Yeah, that's a good distribution. That's what distribution should look like, yeah. I feel like in statistics class right now, it's almost a perfect distribution. Okay. All right. Have you ever built an operator? And again, this is not just building an operator for fun, but building an operator that you actually are also using as part of your Kubernetes deployments. Interesting. Interesting. Give it another 10 seconds. There's people who coming in later have built operators. Yeah, it's all the operator people. Okay. They're finished to deploy their operators, maybe. Okay. And one more. So, this is like an open-ended question now. What do you think about day one and day two operations just put in some of the challenges that you had so far? So, day one, we really talk about deployment. We'll go into a bit more details. And day two is like really running them in production. Just throwing. And this is a word cloud, right? Yeah. So, you can put, it sucks. It's tough. It's challenging or, you know, backups, knowledge. Yeah, exactly. Observability. That's interesting. Actually, this tells us how interesting our talk's going to be. How much we are on point. Upgrades, yeah. Yes. Observability, logging, knowledge, YAML, build-outs, can-feeding, documentation, disaster recovery, and education. It's good. Awesome. Awesome. This is very useful. Thank you. Everyone for participating. Yeah, thank you. Obviously, you can keep adding to this. So, but now let's get started. Let's start with some things you probably already know or at least familiar with. That's also why we were asking those questions. Obviously, if we talk about the two operations, we have a couple of concepts that you should be more or less familiar with. We'll dive into details there as well. One is that a declarative and extensively API that Kubernetes is offering. Like you can use the primitives in Kubernetes. You can extend it with your own CRDs. You can use composability. I think that's what pretty much everybody here is familiar with. We were asking about GitOps. GitOps as a concept of more or less syncing your environment here and matching the system state with the desired state of your environment using it to update and manage your environment. We had quite an interesting distribution there. I think we're going to be pretty much on point because we won't be talking. We put this in deliberately. This is the basic drawing of GitOps. Maybe what you would put on the box of Cornflakes. Alex today will go much deeper into what we're going to do there. And the third one is really about operators and CRDs for custom logics. You should at least have some understanding of operators and CRDs and how they help to build out certain things, especially when we move more towards the day two piece. You will see how operators and controllers play a role in here. I think everybody is quite familiar enough, but we will dive into some of those topics more in detail. When we talk about the one and two operations, there was the word backup on there. Backup is actually one of the things we're not going to cover today. What we're really focusing on is really looking at an application like full stack all the way down to infrastructure from your intent to deploy until you retire it from production. And one of the distinctions I really like about day one and day two is using an analogy from another area. In this case it's base travel. Not that it's rocket science, maybe sometimes it is a bit of rocket science, but I like to compare day one operations with launch control when you're launching a rocket. So in launch control what you're actually doing, you're obviously trying to get this rocket off the ground into space, right? Think about launch control is that everything on launch control is focused on whether getting that rocket up there is something that you want to do now. And usually your process, if something is not right, is abortion or rollback oriented. So a lot of the day one, if it doesn't work, you rollback to previous state that was working to a lot of good configuration. And the interesting fact is that this is the same actually what you do like it's base travel. However, there is this point when the rocket goes up and it reaches like that rust that it has multiple cheese going there so you can't eject that capsule on top anymore which you could usually explode off but suddenly you can't anymore. So you have to continue doing what you're doing so a rollback is no longer possible. And that's where you move to mission control because it's exactly the same when you look at space operations. This is a totally different team and their goal is not to abort but to get the system that is available by modifications back to a stable state. So the main differentiation is really here. This is okay, we abort if it doesn't work. Day two operations will really keep the system running and modify so it keeps running again. Like think again about space travel you can't tell the astronauts who are in space right now we don't know how to bring you back to Earth but we're going to launch another crew tomorrow and see how it's going. Welcome being part of our cannery space travel team. That's what's not going to work and it's pretty much the same here when we think about those two options. And this actually takes us over why Kubernetes is good for both of them and with this and handing like physically over from launch control to mission control right now to Alex here. So I think it's a really interesting question and we all kind of arrive at that conclusion without really considering why. Kubernetes is becoming sort of the posix of the modern age. It's a substrate that you can effectively deploy out many different types of interval components and it's very extensible by its nature. So it really makes it a perfect environment to use as your mission control because you can deploy it on top of any other infrastructure substrate and effectively the API is extensible enough that you can start to define custom types of behavior and custom types of primitives that mean something to you. The last thing is that it really takes us in the direction of being a declarative process in that we write something, we deploy that, we reconcile it and that gives us the day one. Day one is where you start your journey in this. So starting out, what does it actually look like? Well, I think most folks will be familiar. We had sort of a median distribution there. Most folks are familiar with GitOps, but here's a little crash course reminder that effectively you can achieve GitOps in a very simple way. It's a repository that's sinking back and you have this set of activities that allow you to use the fundamental components of Git in a way that can provide you a lot of value directly. So authentication, also having checks and tests as part of that gating system. In this example here, this could be a repository that has an application workload. It might be an infrastructure workload. We'll describe what that actually means later on, but it's some type of declarative configuration. Now, in reality, it's typically a lot more complex than that. This illustration shows what actually looks like when you start to put this in practice. So as you see here, we have this idea of managed services. So these could be your Kubernetes cluster in production. But how do we go through this? How do we get to this point of having all these fancy services on the right? Typically speaking, the thinking right now is that we want to have GitOps to reconcile not only the infrastructure declarative config, but also the application workloads. But of course there's a challenge getting there because our infrastructure declarative configuration might have several different topologies that it's deployed on, whether it's OpenStack, whether it's directly onto some bare metal provider, whether it's in a cloud provider. So we need a way of describing that that's reproducible in a way that we can template to go forward. So you'll see that one of the ways of thinking is to actually have some sort of management control cluster where you are saying, here's the desired intent. I need you to think about the components that I need to get there. We'll come on to talk about some CNCF project later, but you can probably theorize that if you're familiar with these, you'll see where I'm going with this, in that the infrastructure primitives, that might be give me a load balancer, give me a managed object storage, are things that you can describe as declarative configuration, that are then provided by a cloud infrastructure provider. And so where does it all lead to? Well, when you start to build these managed Kubernetes clusters, the idea is that you can provide the application repositories, so folks who want to just deploy a workload on top of them, you can give them a batteries included model. So let's walk through how this actually works. In this sort of scenario, you have a declarative configuration for say some infrastructure. I need an API gateway, a piece of object storage, and some other bits and bobs. And they're all the primary substrate for this application repository. I might have a WordPress application that needs all those things, but to get there, I need to have a certain amount of things pre-provisioned. And this is the real heart of the day one activities. It's provisioning and deployment. You'll see that what happens in this example is that we have a management Kubernetes cluster that's created. That management cluster contains all sorts of primitives that describe these types of managed services, right? I want an object storage of a certain size in a certain place with a certain set of controls on it. Equally, I also want to create multiple Kubernetes clusters. We're trending towards having ephemeral clusters that are short-lived, rather than large node clusters, generally speaking. And so there is an interesting onus on this ability to create Cates clusters themselves that are like cattle. And so once you've gotten to this point where you have this idea of these definitions and they're being pulled by the GitOps workload, whatever that might be, you need to reconcile them. So the infrastructure primitives and the providers say, hey, I understand what an object storage is. I understand what an API gateway is. And it will be contextual because it will understand the annotations of that particular cloud service that you are correlating to. And once you get there, that's not really where the journey ends because if you deploy out like that, it's pretty much like firing a gun. It's a black box. So really, you need to start thinking about how do you aggregate your observability between application level and infrastructure level? How do you differentiate? This is where we start to see day one operations has a lot of toil because what we're seeing companies do is just use the fire hose of putting all logs, all messages, so from the entire node all the way up to the application on top into their observability systems and sort of just saying, that's great, we've got observability, but not really knowing how to then use that as a control loop for SLOs and other governance. And so in a nutshell, one of the key abilities that you get from following this kind of pattern is declarative reproducible ways to build that infrastructure, but also when you start to implement the feedback loop of the observability being aggregated, you can start to modify the declarative config and go further into the remediation side of things. Yeah, maybe before we jump in here out of curiosity, we don't have a question in the slider, we're asking you here, who's running a scenario where they have a management cluster and an application cluster that are more cattle-like? Okay, interesting, so for the video, I think it's probably four? Yeah, probably. Yeah, that's why I wanted to bring it up, like we have that cluster more or less manages the other clusters. And now let's move into the data piece here. First of all, we have to extend a bit of our minds that Alex talked about from day one to day two, what we want. We talked about infrastructure that we are defining and we talked about application workloads. And we talked about, like, this synchronization that the operator predim provides between a desired state and the actual state of the system. So what we have to do, we have to extend our desired state so that the system for day two operations actually knows what to do. And there's a couple of ways to do that. So if you look at the idea of alerting a modifying state, you have three options. The first one is anomaly detection, which more or less means the system is behaving not as it is supposed to be. This can mean that you have response times that you're usually 50 milliseconds that are not suddenly like 120, but your SLOs or your SLAs might still be at a couple of seconds. So it's technically fine, but the system is not behaving as it is usually behaving. Then you move to alerts where you say once you go beyond a certain threshold, the system is not right. Or obviously you move to SLOs where you're giving your system a certain error budget and you're reacting on this error budget. SLOs are actually the most declarative way that you can interact with the system because you want the system to stay in this error budget and then ensure that the controls that are running in the Kubernetes clusters take care of this. Everything else is like pretty direct imperative. Once you reach that value, do something in the system. The second thing is day two wants to actually modify the desired state. Think of any... You can do it in multiple ways. Like scaling would be an example or disabling, for example, a feature flag. Think about having an immense workload on your entire system. Yes, obviously you can scale up, but you can also use feature flags. I already mentioned open feature, which now is the CNCF project, which we'll come to a bit later on again as well. To disable a feature, for example, not query life data, but to move back to a cache level. So changing the runtime configuration of the environment so that they can better react to those situations. And day two operations actually wants to do that so actively reconfigure the cluster. And obviously you have, at some point, leaving the imperative mode because you're moving to an imperative mode, you have to tell the system what it should do when it is in a certain situation. You technically have this. Even it's usually hidden very well. If you look at classical operations, that's what you would have in your remediation workflows and the scripts that you're usually running. But you also need to provide disability to a cloud-native system as well. What you need forever, especially in a cloud-native environment where we run a lot of small, composed services, is a slightly different approach than with the traditional application. And there's always this term coming up about AIOps and AIOps is a lot of things to a lot of people. And it is more that's on this slide, but I still want to share the basic idea, the basic problem that you need to solve if you want to make this actually work in a massively distributed system. So what you see on top here is you have this service that they're very right actually failing, but obviously it's impacting other services. The result out of this is obviously that you will get alerts for all of those three services without context. So you would not know how to react on this situation, like which service should you remediate. In the worst case, you would try to change all the three services while there's actually one that actually needs to be changed. This, by the way, also has impact on how you model operational workflows. You don't model them on the entire system, but you need to model them based on specific services. So these are a lot more or less your context less alerting. What you're now doing, you're enhancing these context less alerts with tracing. Think about it that way. If you have a trace based on those alerts, you can reconstruct the causal chain that you have on top. If then on top you're adding like temporal information, like what component failed first, plus additional information like how things can be related to each other, like how a failure mode can be related to a slowdown or how certain failures at the application level can be related to the infrastructure. You're actually arriving at something like this, which can suddenly differentiate the impact of a failure mode to the actual cause or root cause. I know not everybody likes the term root cause, but I think you're kind of getting to it. So what we actually did here, we used traditional monitoring combined with tracing and some additional knowledge we have about our system to come to a way better way of actually alerting and defining these kind of things. So that's very high level what you really want to get. And so this is way more important than being hung up whether this is AI, whether this is machine learning or whatever. It is a system that can reconstruct based on context. So what we now need to do in order to move to a day two operation scenario is actually a couple of things. Alex talked about the whole GitOps configuration that we have for day one and we're now starting to extend it for day two. The first thing we do, as we start everything declarative first, we are adding the declarative monitoring in there. So your monitoring configuration should also be part of your system configuration. This today, and I think that's what you will see a lot of with these topics on day two, it's possible, it's not like possibly in a standardized means where they can't configure monitoring in a way that all monitoring backend systems will eventually understand it. It's very much focused on the project or solution that you're going to use, but most of them provide you with a way to declaratively configure what you want your monitoring to look like. And obviously you need to also specify your goals like your SLOs and these kind of things should be specified declaratively as well and pushed to your system. And you add your remediation logic on a pair of service bases or how you want to have it. You want to have this version controlled and linked to the version that you're actually deploying because as you're rolling forward with new deployment, you want your remediation logic to roll forward with this as well. And this logic can be basically everything. It can be just a link to a container image. It can be a script that you're providing there, whatever it's kind of needed for the environment that you want to work with. And these logic blocks are usually very simple so they're not super complex workflows as you might be used to from traditional applications where it's like workflow with 15, 20 or more steps. As they obviously are always about one single service, they're usually more or less little if than else statements like if this happens, then modify the service this way. And usually they come from developers because developers know how you can reconfigure a system properly. And then they're composed at runtime depending what we learned before in the AI upside, how things actually work. So what we're now doing as we are doing our GitHub sync, we're not just thinking our declarative configuration, how the application should be deployed and built. We're also thinking over our declarative monitoring configuration and our remediation logic. Obviously doing this per se won't do us a lot of good because Kubernetes wouldn't know what to do with these new CRDs that we just built. So in order to make this work, we need what we call in here like the monitoring controller. The monitoring controller can control, call it controller operator, whatever you feel more comfortable with in this case. What it is going to do is taking that monitoring configuration and pushing it towards your observability system. So the idea should be your observability system is not configured out of band and it's like split brain type of situation. But you actually let Kubernetes really be the control plane here to also take control over the observability system. So it uses this configuration and then deploys it to your Prometheus or whatever solution you're kind of using there. Including creating alerts, creating metrics that you want to calculate. For example, if you have an SLO definition, there's a project called Open SLOs and other projects like Capcom have an SLO definition as well. They create the proper metrics to do the metrics calculation for you so that you don't have to think. And then what you need is actually somebody watching over this. What does the observability platform tell us about what's happening here? And this in our case is our Day 2 application controller. This is either push or pull the way how it's interacting with the monitoring system and it's more or less depends on security boundaries or what you feel more comfortable. In many cases, it is actually more a pull because depending on whether you have an on-prem monitoring system where it's entirely a separate component. So whether the monitoring system is allowed to send notifications into your cluster or you're actually fetching them. So once your Day 2 app controller actually detects that there is a problem happening in the system, what they do is they look up this remediation logic which is technically the CRD that they are working with. Again, it's a similar example where we have if the load on a certain system is too high and response time is going down then scale to cached content delivery. That's the if statement it would find. It's most likely if metrics are rough or if a certain type of alert arises then modify a feature flag or whatever you want to do in this case. So what this then does it spins of a remediation job and job in this case is really a job in Kubernetes so you're most likely modeled this as a job because once it has finished running you can't get rid of it. So there's no need like having all of this remediation actions being constantly available as containers. And they can do two things. One is they can talk to external services changing some system configuration, modifying it in some ways maybe even talking to some of the cloud APIs but even more importantly and that honestly right now is a bit of the controversial thing the remediation jobs might actually actively change the declarative configuration so they're intercepting with your GitOps workflow. And in a GitOps scenario that's currently the only way you can actually change this because if you would change it right in the cluster the GitOps controller would revert your change because suddenly the desired state and the actual state is out of sync and it doesn't know the desired state that it would have in there. So this is kind of a bit of an unsolved problem in here but I should give you an idea on how I like to build the two operations pieces in there. So it's really about anomaly detection using really as it goes to inform you about what's going on through this correlation and then really doing this context-aware management so adding additional context management to the workloads that you're deploying and I think this brings us to how we built this on top of the entire project. So really to just build upon the previous slides there are several projects within the CNCF ecosystem which are picked at random because they fulfill the requirements here. But what they all do is they enable you to build that workflow that we just described. So being able to use Flux or Argo or any number of GitOps based tools is extremely convenient when you need to be able to create a management cluster that represents the infrastructure and the workflows that you want to apply on that infrastructure. One of the things that's really interesting to me and this came through TagApp Delivery was Crossplane when we were doing the due diligence for that. Because Crossplane is a very interesting project in that it represents all the resources in cloud providers which isn't something traditionally something like cluster API would do. And in fact you can create load bounces and block storage and all those other artifacts that are convenient. And even more what's useful is I think as I described in the previous slide you have the app repositories and then you have the infrastructure repositories. The two together you can have a singular repository that describes the dependencies of application in the same repository in a YAML definition. And that brings us a lot closer in the day one to being able to get to a point of a very ubiquitous cycle throughout this infrastructure life cycle. And what's interesting especially I think is that I mentioned cluster API not having the same capabilities of Crossplane but of course that is now a subset called by Crossplane and you can use cluster API directly to cut out that particular process. What's nice I think is that we have a lot of LEGO bricks inside of the CNCF ecosystem that are convenient for you to choose your journey. And I also mentioned using tooling such as tracing with Jager. One of our ambitions of the Open Feature project is to add baggage so that you can see the feature flag that was enabled at the particular time. As that gets scraped back to the monitoring controller you're building a picture up to look at what the date was at the time and what features were running. And as we mentioned earlier on if you have an evaluation context you can say well you know in real world if an application falls over and I can see that it's because of some sort of back pressure or load related I can have a feature evaluation context that says okay well make this commit back into the repository that dials it back 25% or so. So we're starting to see the building blocks there and I think really we haven't presented any more so than in the Kepton project which is also a new a newly thought out project in terms of building some components that are really innovating in this space. You know looking at things like SLOs and using those in a way that you can actually make changes to enable that day to journey. If you haven't checked out Kepton I think that I would do it in Misjustice now to try and explain it but Kepton very much follows that path of being able to give you a greater picture on the context of application right so if you have a particular workload that's running in Kubernetes you can also see those relationships and dependencies and manage the SLOs in a really convenient way and I think this sets a tone for tooling of the future and that day 2 is a problem that we're starting to get closer to solving but as was mentioned in the previous slide there is definitely a challenge with remediating jobs I mean putting credentials into that job and then making a commit back into the main on your Git branch doesn't feel great to me but we're trying to close that gap I mean we've got projects running at the moment with what's called a GitOps reverse sync where you would actually sync backwards through another GitOps repository into the declarative config so I think what we'll start to see is that GitOps becomes more of an ecosystem to self balance this footprint of your infrastructure as we go forward but the undertone here is that really we're starting to put a lot more trust in automation and we're starting to think that we can narratively define all of the failure modes and have ways to over a gradient scale those in an automated fashion then we're not going to be able to accomplish that so in summary we've tried to distill I think the key tenets of what we would like you to hopefully take away here is that day 2 operations is very much about trying to remove the humans out of the equation here and not rely on automation unless you have the right observability signals being correctly remediated and acted upon GitOps, Kubernetes and operators are just enablers for this pattern really they're convenient enablers this is a pattern of operation this is how you manage the life cycle of your digital estate and SLOs and error budgets we're starting to see now more than ever are becoming the inputs for those corrective actions for operators those are becoming the driving force for how an operator makes that decision how the remediation job actually starts to build that picture of what it's going to transmit back to the declarative configuration I think what we could actually show here you could really use the core concepts of Kubernetes as a powerful plane to build all of them and really stick to this declarative and operational model it's also why we went into some more details how it could build day 2 and also for day 1 bits and pieces it is actually possible for non-cloud native tooling I think also you see how projects in the CNCF fall together nicely once you stick to these core concepts that we have defined for Kubernetes as a powerful plane and how well they are connected playing with each other also you find our Twitter handles here for any questions or how they want to engage with us that QR code is a different one that for the working group on platforms is part of tech app delivery tech app delivery has a working group now platform working group it used to be called corporate if delivery which kind of was doing the same thing but people feel more comfortable talking about platform and platform engineering so the goal of this working group is go deeper into defining these blueprints addressing these problems how they can be solved with Kubernetes and projects we are even right now discussing building out a reference implementation not one that covers everything but that just helps you to get started on these topics it is part of our work we definitely invite you to join working group platforms here if this is something that is interesting to you if you have an opinion on it if you have questions on it because that is what we will be focusing on a lot within tech app delivery to help you build those platforms bring projects together with this experience I hope the input was useful you saw a bit of the way we are working in the working group that was more or less looking at sketches that we have been working on and implementations and where we are taking this maybe time for one or two questions left now so how do you make this audit friendly in a highly regulated environment that is a really good question I used to work at two different banks I completely sympathize you just get an AWS service catalog and you have five things you can use in there and it is really tough I think that it is a wider question in terms of the governance audit cuts several ways whether that is looking at the security protocols in place whether you are looking at FIPS, CIS et cetera or whether you are looking at SOC2 and more of the controls that you have got on your PII et cetera is that because you are able to effectively represent your entire digital estate in a series of repositories that get pulled into reality it makes it very easy to find one place where you run the audit scanners over so as an example and this might be only a subset to the answer of your question but as an example we would at bank scan the repository to check where simple things like images were coming from was it coming from the wild was it coming through a squib proxy did the did the permissions use host network simple misconfiguration issues kind of what I had in that check-in test you could extend that to have a suite of audit tests in there so that when it gets to your production the security team then don't remind you you have made a problem I think also to that end with the GitOps process there is nothing to say you can't deploy out security scanners like trivia et cetera environment and use the observability from trivia trivia gives you Prometheus metrics it's the same thing right it's just another signal that feeds back into the remediated action what you could do in that case and I'll demo this on Friday in the open feature talk is that you can use feature flagging to actually actively in the cluster get the API service turned capabilities off so when you detect say from your security scanner there's a Prometheus metric that's returning on one meaning there's a CVE in it you can turn off the admission controller and transfer it to a shutdown mode so there are all these interesting capabilities that were signed to build on and as I said shameless plug but open feature plus these kind of practices go hand in hand for that very good maybe we'll jump in here actually this is the most audit friendly thing that you would ever find because everything is run by Git then you could even run in Git blame like most operational processes that you're running today rely on people knowing what they're doing and documenting it actively this is actually all documented you have every single change you have every single action in there you have access control on all different levels enabled it looks like a bit scary because the system is doing something by itself but it's never doing anything you don't want it to do you can always add manual guard rails and you know exactly every single state transition of that system that is more auditing that most systems today actually have if you really stick to the model it only breaks and that's the important thing if you're doing something out of band you have to really design around those principle principle like Alex mentioned the open feature if a feature management system uses to be a separate system that handles feature flags separately you're suddenly breaking that entire audit chain because you have one system deployed you have to audit log for this one and then suddenly an out of band system changes something on those systems so you have to really avoid those out of band changes and then you're actually getting a perfectly auditable and control level system as an anecdote I've seen things like service now being used to create temporary ephemeral accounts to then make a manual imperative change within a cluster that then is completely unregulated and unordited whereas this way the principle would be that you have a GIT group of administrators within your VCS that can make a change in a PR merge that change in and it's far more traceable this is what I found out so on prem there's really little distinction so I typically work on bare metal or with open stack or some other virtualization there's no real differentiating I guess what you're driving at is the whole infrastructure provisioning piece so with crossplane and with cappy you have two components a bootstrap provider which gives you the flavor of Kubernetes there's an infrastructure provider and there's about 15 different cloud providers what's interesting about this is it's a democratization so if I have a startup and I have a rack and I want to show it to the world I might write my own infrastructure provider and then when somebody writes a piece of cluster YAML and they put Alex as company the provider will start the pod and it knows how to reconcile that and it connects to my API in London so what's really interesting about the CNCF project and I mentioned POSIX is it's a great leveler it's a great ubiquitous platform that you can start building towards so for bare metal as an example we have a MAZ so metal as a service and open stack providers and simply all that does is it creates a pod for open stack and it knows how to talk to that open stack endpoint or that MAZ endpoint there's even a VM web provider etc I think the key point again is they're not doing it out of ban that's what they keep hearing us repeating over and over once you move say here we have our Terraform scripts not nothing against Terraform at all but if we're not integrated in this process and they're handling a lot of these you will suddenly use this connection or connective tissue that you want to have in there but nothing is holding your back because you could put any logic as part of the controller operator but that's what you should eventually move to also for those on primary sources okay great thank you join me in thanking Alex and others