 Hello everybody and welcome to another OpenShift Commons briefing and as we do on Wednesdays we like to talk about operators and I'm hoping you're joining us from the previous live stream on OLM so it's a good segue into another aspect of our operator framework community. We've got Jesus Rodriguez and Christof Laprun from the Java operator group and they're going to give us an update on Java operators and all things around that so I'm going to let them introduce themselves. You can ask questions in the chat. We'll have a bit of a live Q&A at the end and I hear there's a really skookum demo coming so I'm looking forward to that as well so please do ask your questions in the chat and we'll rock and roll now so Jesus take it away. Okay I'm Jesus Rodriguez, a principal software engineer here at Red Hat and currently working with the operator framework operator SDK team and also working with the Java plugin. Christof? Hi folks I'm Christof Laprun. I'm a principal software engineer working in middleware or application services as it's called now and I focus on right now focus on Java operator SDK which we discussed today. Awesome. Okay so the as you saw this last week with Rashmi and Varsha the operator SDK from operator framework works closely with QBuilder. The two projects were effectively written to simplify writing operators. The SDK team has worked closely with the QBuilder community to kind of collaborate on a simple and similar project structure so the projects that the SDK will scaffold look almost identical to the QBuilder ones. SDK also uses controller runtime and controller tools upstream to kind of base a lot of its operators on and the important part too is the extensible CLI so we have the plugins which Rashmi covered during the last presentation and in these plugins we are scaffolding out a variety of different projects and operators so today we offer Ansible, Helm and Go and as you'll see we're starting to look at and scaffolding out Java. So why Java? Why look into offering operators in Java? So as you may know Java is one of the most popular enterprise languages and there are tons of developers out there already using the language and as workloads start to move to the cloud it's natural that they want to also move the operational knowledge to the cloud as well and in these operators if you've already got a lot of your expertise in writing your applications in Java you tend to want to write the operators in Java. Today typically when you go cloud native the first thing you have to think of is like oh we probably have to write something and go which is a fine language but it's kind of out of most people's comfort zone. This is precisely the reason why the SDK went after using Ansible and Helm because we had folks that were had expertise in Ansible and written their installation in Ansible so this allowed them to reuse a lot of that and create an operator in Ansible so it's natural that we can also try to enable folks with Java. Also the DevOps philosophy you know it's typically the folks that are writing the applications which in this case would be Java would also have to maintain and do the operational part portion of this so if they can use their same language skills it would just make it way easier and allow them to reach success. Frameworks. Java has a breadth of frameworks available and tools that make it easier to write both applications and as you'll see later even operators. Performance. Java has historically been a performance language and with the operator Quarkus has enabled a native mode which Christophe will talk mentioned briefly in Quarkus and that has increased performance a ton. There's actually a few sites out there that compare the native Java with you know the Quarkus native with regular Java and go and it's it's amazing how fast Quarkus has gotten. Christophe? Yeah yeah I just wanted to add to what Azus was saying. So far the big questions because it's been possible to write operators in pretty much any language right as long as you can have a loop that you can watch events from the Kubernetes server you can write an operator in any language I mean and even though Java so far I've been used in lots of applications and it's typically quite performant and the performance characteristics so far were more targeted at server applications and that came with several optimizations that were not really suited for cloud workloads and that's why Red Hat looked at improving that specific aspect with Quarkus. The idea with Quarkus is to make Java cloud native basically and improve the performance aspects of Java specifically to enable fast startup because typically Java applications used to have pretty slow startup because the virtual machine is optimized for long running processes so in a cloud environment where your applications are running in containers that can be shut down pretty much at any time that's an handicap that you don't want to have so you need fast startup and you need also to be able to share the space and the memory with other applications on your cluster right so another aspect was also to work on the memory side of things so that you can have a Java application that doesn't take that much memory either and the result of that work is Quarkus which is a which comes with a pretty neat tagline which is a personix about Java it's meant to be developer friendly as we will see in the demo and it's supposed to be optimized for cloud workloads even though it can work in what we call JVM mode which is basically traditional Java it still has a lot of optimizations for that mode where a lot of the work is deported is done at build time as opposed to runtime for many traditional Java frameworks and the big I mean one of the big innovations in Quarkus is making it easy to compile your application natively because Oracle came up with a project called Grail, GravVM in a couple of years back but it's been quite difficult for people to use it effectively for non-experts and Quarkus makes that a lot easier to natively compile your applications and it comes with a best of breeds of libraries and standards and one aspect also of Quarkus is that people are able to write extensions for it so that you can plug your extension into the framework and make it support native compilation and do whatever your extension is trying to bring to the Quarkus world make that work do that work sorry at build time as opposed to to runtime next slide please yeah so if we were to maybe I don't know if you can see very well because the image is quite small at least on my screen but it's a summary of what Quarkus brings to the table in green you can see it's basically showing the space that's taken in memory for a Quarkus application as opposed to a traditional Java application and that's a typical REST application so the rest end point and the second one is the REST plus CRUD CRUD being create read update and delete so it's basically accessing a database I mean doing basic database applications so those are typical application workloads that you could have on the cloud and basically in the gray the gray color shows what it would be for a typical Java application so you can see it takes a lot of memory and it's pretty slow to start and in the blue color shows the Quarkus version but running in Java virtual machine mode so the what what I called the JVM mode earlier and in green is the same application but combined natively so as you can see for the memory side of things it takes significantly less memory like about a tenth of the memory from a traditional Java app and it starts really fast like from 4.5 seconds something to not even a second to start up when running in native mode so you can already guess the advantages of running Quarkus in native mode in containers and when it comes to operators obviously that that neatly ties I mean catches up with go operators in terms of efficiency even though you can probably still write go applications that are slightly smaller than Java than Quarkus applications Quarkus is competitive and it has the advantages of obviously being known to Java enterprise developers so they don't have to as Jesus was saying they don't have to learn new languages new language next slide please so that's where the Java operator SDK comes into play because even though you have Quarkus and several projects open source projects to I mean clients for Kubernetes APIs that you can use in your Java applications to access the Kubernetes server and interact with them so far one point that was preventing or at least making things harder for people to write operators in Java was the lack of the equivalent of controller runtime that go provides for for developers and the aim of the Java operator SDK is to provide that equivalent of controller runtime but for Java developers so obviously the goal is not to have a one-to-one feature parity but rather to have something that is familiar I mean that uses Java ideals so that it would be familiar to Java developers and the end goal is to simplify writing operators and controllers in Java obviously it relies on the fabricate Kubernetes client that Red Hat developed a while ago already and provides integrations for different Java frameworks in particular Spring Boot it provides a Spring Boot starter and what we discussed today in greater details Quarkus extension and that makes things even easier and just recently the operator framework team has developed a plugin for the operator SDK CLI that the laws as Jesus was saying to scaffold Java operator projects using the operator SDK plugin before you go on to the next one one comment I wanted to add is you know like you mentioned earlier that you can write today operators in Java just using straight fabricate client but the aim of like Java operator SDK just like controller runtime is to kind of hide or at least abstract some of the concepts away so that you don't have to know all of the raw Kubernetes and try to make it easier to write and do your controllers so that it makes it easier to enter the space early on when operators first came out you know the first thing everyone thought is like if you're writing an operator you're going to be a Kubernetes expert we found that not all operator developers are Kubernetes experts often the reason they're writing operators is because sometimes the business says hey we need to go cloud native you need to learn to write an operator and a lot of the users of that we found were in that base of I've got to write an operator I don't know go I I know Java or I know Ansible how do I get to where I need to solve this problem and I think that's where these projects are helping folks move along yeah definitely the goal is really so that people can focus on their on their business logic as opposed to the low-level machinery that is required to basically wire everything and write your operator at the low level and we see that in in the demo hopefully next slide please which I think yeah so let's talk a little bit and Gritter details about the the quacus extensions for the the Java operator SDK so even though the Java operator SDK does improve the situation quite a bit when it comes to developing operators in Java the quacus extension goes even further it removes mobile even more boiler plates I don't think I'm not sure we'll see that in the demo but we can talk about that maybe it makes it as I mentioned it makes it easier to natively compile your application another aspect that comes with operator development is the generation or at least writing a CRD for your custom resources and typically that's not really straightforward I mean if you if you've ever written a CRD manually without any help it's it's quite tricky to get it right and the quacus extension automatically generates the CRD for you based on your on your code and we support we have a dev mod support so dev mod is something that that quacus brings along which allows a fast feedback loop basically you leave your application running as you write your code quacus restarts the application when needed and do things on your VR so that basically you don't have to stop your application do compile your code restart the application and and basically it makes everything faster and more enjoyable as a developer I think we can maybe move to the demo now there was one more this is right before the demo so to tie things up together as Christof mentioned earlier there was the Java operators plugin for the operator SDK so in the operator framework side like I mentioned earlier we had go ansible and helm and we've had requests for hey I'd like to write operators in Java we traditionally with the other three were basing things on controller runtime and we started to look at possibly creating a Java version of controller runtime and that's when we found the Java operator SDK and that made it easier to get to this point to write a plugin to allow the scaffolding of the Java plug operator so one of the reasons we did this it's also helps to integrate with the operator framework SDK and Q builder so you've got a set of users that are familiar with that technology already that will want to also possibly have users that use Java together we are looking and we offer OLM and scorecard integration so OLM is the operator lifecycle manager which comes with OpenShift and it's also open source that you can install in a bare metal coming in a bare Kubernetes cluster that offers the lifecycle of an operator so we help to build bundles and integrate with that scorecard is a testing facility that we offer with the SDK that you can write custom tests and there's some basic tests that allow you to go through pipelines and just do validations of your operator and like I mentioned the CLI with this plugin it is it uses the same CLI as the other as the operator SDK so if you've generated and scaffolded out a go operator it uses the same API to create a Java one and what we're writing out instead of starting from scratch and going through and figuring out all of the things that we needed for Java we know what a Java operator SDK and the corkis extension require to create a basic project and that's what we're scaffolding out and going from there okay next slide I think is the your demo Chris I was not sharing the screen can everyone see my screen not yet give it a second try again well we're doing this if anyone has any questions please just ask them in the chat wherever you are in YouTube or in Twitch and we'll lay that in here but um try the share screen again button because you may if you have two terminals set up it may be giving yeah I think that was the it was showing before we started so there you go and we're back to the small font so if you can yeah I'm gonna switch back to presentation mode and hopefully that doesn't mess things up okay and everyone see working up yeah okay I'll just stop my video for now okay so uh let's try to write an operator live let's see how it goes big undertaking right but not really I mean we see so I'm gonna use the operator SDK CLI to scaffold my code so maybe maybe a little bit let's talk a little bit about the use case I'm trying to solve so I mean obviously we won't get into all the nitty gritty details of creating a production ready operators here the idea is that as a developer I want to write an operator that will take my the I mean an image reference I have a Docker image of of an application I want to put on my Kubernetes cluster and I want that application to be accessible outside of the cluster so typically if I want to do that with Kubernetes I have to create a deployment create a service and on plain Kubernetes I have to write an ingress or a root if I'm targeting OpenShift so in this case we will do a simple use case where I have my app already I'm not going to worry too much too much about which port is exposed it will be outcoded to be ADAD in this case and because that's quite a I mean that could get tricky quickly but um yeah so my operator basically wants to I just want to write a simple custom resource where as specifying in my spec here's an image ref do your work create the deployment for me create the service create an ingress and and expose my app so that I can access it so I'll I'll try to do that now and the um I will scaffold my uh my operator using the operator SDK init command which starts the scaffolding process and I tell it to use the quarkus plugin and specify a domain name and a project name so the domain name will be used for uh things like package name in Java but also for the the group name for the the CRD and things like that the custom resource definition and things like that so let's run that and let's look at what got generated and maybe I should open the project first okay let's do that again I should have created a project first directory because now I messed up my directory let's do that again okay and the SDK is nice enough to tell us that right now we scaffolded a basic project but I don't do much as we would see so if we look at what got generated we have a typical Maven project with a pump file that adds all the the the needed information to get started we also have a make file a git ignore a resource file for application properties for quarkus and no no classes as of now so if I start my dev mode my quarkus dev mode uh it should uh I should say beforehand that my laptop is quite old and not very gifted in terms of RAM so running blue jeans plus docker plus my ad and everything is not that fast so it doesn't do quarkus a lot of justies unfortunately in this case but yeah the quarkus is starting now and you can see that it listens for on port uh 5005 so that I can attach a debugger if I need to and it should get going soon while that's while that's loading one thing I wanted to make a comment about um I know you mentioned make file and most folks that are familiar with java noticed the palm.xml and wondered probably wondered why there's a make file um the reason there's a make file is the main cli for the operator stk um for let go and ansible we have a make file that we have so that you can do make build for make bundle um you can do your scorecard runs and things like that so to keep the the flow the same between the different languages that's why that make file was created but that was users back to you Christophe so it started but it failed and basically it's telling us the same thing that the operators did get told us there's no api right now in the java operator parlance what you do is that um actually you would create an operator instance and then create a resource controllers that implement a given interface and you implement those methods and then the sdk that's it's magic and and the operator is wired up but in our case we don't have any resource controller now because we haven't created any so that's what we'll do now using still the operator sdk cli and this time uh if you notice i keep the the corpus deadmod still running and the command i'm using for the operators sdk cli is create api this time and i need to specify two additional parameters i need to give a kind for my custom resource because basically creating an api in operator sdk parlance is is the kind of creating a custom resource because that's basically what they are as far as kubernetes is concerned so i give it a kind which will map to a class name and in java and and a version uh that's used for versioning my api so if i run that uh command um we'll see that it should create hopefully usually it's faster than that i guess that's the demo effect i'm sure what's going on now why isn't it picking up the code because it doesn't come out that well no because i'm stupid and i didn't switch to the directory i created again so let's do that again and now it should work um yeah see it created an exposed app uh class and as you can see as i'm looking at my class you can see that quarkus reloaded the application automatically uh while i was looking at my class so let's look at the class a little bit um this exposed app class uh is in the package that we defined uh using the sdk init and it extends the custom resource class that's found in the fabricate client and if you know java it's parameterized by two additional classes which map to the spec and the status class for the for the custom resource basically the new java operator sdk is opinionated in that way and try to make sure that you follow best practice the best kubernetes practices where it's it's a good practice to for your custom resource to separate the desired state which is specified by the spec property of the custom resource and the status which is quite which is supposed to be completely under the control of the of the cluster which i mean there are cases you probably don't want that but let's not get into that but basically i have that dichotomy between the desired state which is represented by the spec side of the custom resource and the current state which is the status um so if we look at those classes right now they are empty um because even though the operator is dkc li there's a lot of things for you it can't treat your mind yet so maybe there'll be a feature for to the door i don't know we're getting there yeah uh so those classes are empty so let's add um well maybe let's look at what quarkus did while we were talking some more so you can see that first that's actually the yeah quarkus detected that there are there have been changes to some classes so it restarted the app and then our extension which is uh owned by the operator is dk processor is doing some work is registering i'm not sure why there's double there but let's ignore that for now but it's registering registering classes for reflection and that's used later for native compilation basically any can any classes that are access reflectively in java you need to register them properly so that brawl vm knows that you intend to do stuff with them at runtime and the quarkus extension knows that the spec the custom resource and status class uh have some reflection aspects i mean i've said i've accessed their reflection in the code so it automatically register them for you so that when you natively compile your operator you don't run into issues at runtime where the native binary cannot do the operations you want that's one nice things that quarkus does for you and then it processes your the controller the exposed app controller which is not doing anything at the moment and we're gonna look into it later and it associates your controller with your custom resource which is automatically the exposed apps that i can i o and registered with the group i can i o which are the information to pass to the operator kcli and also it's generated the crd for you using the v1 version of the spec you can tell it to generate the v1 beta one as well version of the crd spec but it doesn't do that by default and it um let's see and then it failed because let's see what why is it failing oh yeah because the crd was not found on the cluster so when it tried when quarkus tried to start the operator on the controller it basically saw that i mean when it's trying to register the watcher to watch the events for your custom resource the operator the java operator SDK has a mode that you can deactivate actually but that lets you check that the crd is present that's useful in development during development because as you can see here i forgot to deploy my crd so my operator cannot really work if the API is not known by the cluster so let's change that actually and maybe we can tell quarkus there's a there's a property that we can change in application property to tell the java operator sdk extension to automatically apply crds to the cluster during development obviously don't want to do that in production but anyway that mode should apply only during development when you're running in dev mode so let's change that property to true and quarkus should figure out that the application has changed after a while and yeah I'm sure enough it restarted and it regenerated the crd for you and applied it to the cluster right there you can see it got it got applied and now my operator properly started it's running but nothing happens because obviously the controller doesn't do anything so let's change that let's look at the controller now actually a little bit more um so your controller class has two things that well I mean several things but first thing is that it needs to to be annotated with the controller annotation because that's where we can configure several aspects we can configure aspects using either the annotation or the the application that properties file and it also needs to implement the resource controller interface which is parameterized with your custom resource class and what that gives you is that you need then to implement two methods well really one method which is the creator of that resource this other method in it is called in some cases when you want to register even sources we will get into that here but that's something for more advanced use cases I don't think we'll have time to get into that here and the creator of that resource method is basically gets called automatically whenever something happens when not something but when one of your resources is created or updated on the cluster so we can see there's no nowhere here do you need to create a watcher and under the low-level stuff it's it's just automatically gets called here so let's add a field here and maybe we'll add some some logic here as well we're going to just print all that we received a resource a resource we we get directly the resource object so now since I added the field getting it ref on my spec I can typically get it um I think the app got restarted but I discovered a bug earlier so I suspect that maybe we might have an issue yeah the CRD didn't get generated properly so unfortunately that's a bug I need to address let's restart and and also create this cluster resource I look at my my cluster right now there's nothing in it in my name space um let's see so let's apply our custom resource and hopefully in my controller I added just an output so that whenever the creator of that resource method is called we'll get the image we retrieve the image ref a field from the the customer source and output so let's apply our custom resource this will work and yeah you can see that uh the operator uh responded they are complete but it's getting ready uh maybe I should stop the demon now and open the field for more questions and I could show a lot more but uh about for a lot of time so maybe let's open the field I don't know maybe because we have more targeted questions that we can answer one comment I would make is that as you can see it is um pretty simple to create the you know the basic boilerplate stuff of the operator and then now like Christoph said earlier you can focus on the business logic of your operator and not have to worry so much for oh you know how do I get my reconcile function to get called what do I need to do here now we've you know the framework will call your create an update resource that's where you update and handle your reconciliation logic and again focus on the business logic yeah maybe I can do add the logic a little bit while we keep talking okay so I don't know if we have questions or anything I got one more slide after this yeah maybe we should go to this the slides here I mean yeah just to show quickly the well yeah let's go I just took it sorry no no problem so as you saw it was Christoph's demo and there are quite a few samples in the Java operator SDK project and available to look at as well as the corkis extension and the operator framework SDK has non-java ones that you could look at as well quickly because we're running out of time the road map for these projects and the plugins is effectively you know adding better testing support one of the common questions we get on the operator framework side is now that they've learned to scaffold and create their projects how do they test them they there's both unit tests and integration tests that they want to focus on and that's clearly going to be one of the big questions that they're going to ask when they create their Java operators as well today cube builder and SDK have an ENV test MF test that allows the test to run a real API server so that you're interacting with a real API server but not necessarily a full-on cluster we are looking into those types of testing for the Java stuff as well find see what would be the best thing better dependent resource support there's usually your primary but then there are things that you have to monitor and and effectively watch while your operator is running again integration with OLM so OLM is one of the key features in OpenShift for managing operators so we want to be able to allow your operator to be run and managed by OLM as easy as possible so that you don't have to worry about all the details about the Java operator plugins that we've showed the scaffolding out the initialize and create API as Rashmi and Varsha mentioned last week the phase two plugins which allows out of tree executable plugins so today the Java operators plugins is compiled into operator SDK phase two will actually allow you to create your own binary and put it into the path and have operator SDK discover it and run it this will give the project the ability to have its own releases and we could ultimately migrate it to a full-on Java application today it's written in go it was just trying to get it out with the constraints that we had at the time metrics and webhook support so one of the key things that operators typically want is how well is my operator running which how much memory am I using the how often are the reconciles happening and then there's other things about your application that you want to expose so we're not only are there some default metrics we also want to allow folks to add their own metrics that they want to monitor for their applications that they're deploying and then lastly your requirements the things that you want to see in these projects we would welcome your input there's a variety of projects that you can contribute to and offer support and or you know as simple as creating an issue open an issue to at least show your interest of what feature that's missing or you would like to see so that we can prioritize it and lastly I think Diane put this in the chat there are a few projects Java operator SDK porcus extension the plugin and then we've got qbuilder and operator SDK themselves we meet every Thursday 4 p.m utc that's 11 a.m. I think eastern and there's the zoom link there's also a discord chat I forgot there's also a kubernetes uh you builder users and or kubernetes operators as well as a operator SDK in slack so you can pretty much find us everywhere I'll grab those and add them in in the video edited this so that's great too so yeah I didn't know you guys had a discord server so that's great actually and there's all there's a couple of questions in the chat too as well and there's there's always the conversation we have about quarkus versus spring boot and you know java stuff and we did talk at the very beginning about the value proposition for using quarkus and the quarkus extension instead of just a standalone java library but I'm wondering if you could just address that again in the in this q and a there's a question here in the chat my understanding from the java operator SDK project is they also look at supporting spring boot and plain old java christoph can probably um dig into more I mean explain that a little better I know from the plug inside from the scaffolding we've also considered scaffolding you know spring boot or some of the other frameworks that an operator would look like um trying not to just pick one um because I know like with go when we scaffold it out it's pretty the only thing we worry about is controller runtime and then you're pretty much a free-for-all for what you want to use there there's always that balance of being opinionated but also giving you the flexibility so to address the question it's the same I mean the same question could apply to any java project right why use any framework to do anything basically you use a framework because it makes things easier like I said mentioned at the beginning you can write an operator using shell if you want I mean um a bash create as long as you can deal with the reconciliation loop uh listen to events I mean kubernetes events and and react to them pretty much any language you can use the question then becomes a familiar are you with that language um and are you being productive with that language for your the use case you're trying to solve in in this situation why use quarkus well quarkus says I mean maybe I didn't do a good job of demoing everything but I'm actually I didn't demo all of that I wanted to but um um basically with quarkus you have that uh fast feedback loop when you do something you run the demo then you write your code and then quarkus tells you oh your code doesn't compile or in that case for the extension it tells you or you didn't deploy your crd so I can't do anything or you don't have a resource controller implemented so your operator doesn't do anything obviously if you're familiar with the framework you don't need that help but for developers who might not be um that's still useful another aspect also that I quickly mentioned uh the quarkus extension uh I mean quarkus if you use java quarkus is the I mean obviously I'm biased but in my opinion it's the best framework uh java framework to use if you aim to compile your app natively at some point because it makes the native compilation quite a breeze compared to similar java projects that are out there but not I mean at least in my opinion don't bring the same level of productivity that quarkus brings so the quarkus extension in that case automatically registers for reflection the classes that need to be if you don't use the extension you would need to know that and know how to register those classes yourself which is not trivial things to do so though that's one thing the second thing is that it looks at your code it automatically updates your crd for you when needed applies it if you want to do the cluster so you don't need to do that and we are looking at other features um I think I think yeah you can you can write you really you really answered that um I think and john's uh who's the person who's asked this is um is he is also a fan of quarkus uh so it's not a it's not a quarkus versus spring boot conversation it's just more about um what the benefits of the web framework were and I think you articulated that I mean yeah quarkus isn't I mean obviously it's it's maybe optimized for the web but it's not uh necessarily simply a web framework it goes way beyond that hopefully we show that here yeah and being opinionated is not a bad thing it doesn't make you just and you may have a bias but um working this closely and working on all these plugins and everything and making them um uh really um speed up our our java application development processes and making these services available is really a huge contribution so thank you christoph and asus for all of the work that you've been doing and the community has been doing and I really highly encourage people to try the projects just like it says there on the slide in front of you take a look at them give us your feedback um come to one of the community meetings um let us know um what's missing and help us figure out what's next to do really looking forward to um some of the phase two um work that's being done for java so we'll have you back when um we get to that point and do some more demoing but um if you're using um this java operator in the sdk in production or if there and you have a story or some lessons learn uh reach out let us know um we always want to hear from you and get your feedback that's what really drives the innovation and the roadmap um we're not making this up as we go along we're really doing this based on community feedback so um real important that um you reach out and um log an issue that's one of my favorite things um even better um go in a uh pull request and make something happen um that's even more fun but i'm sure you can find um your peers in these community meetings um in these different uh servers and especially i'll add in the slack kubernetes um channels as well for this um in in the chat afterwards but this is really your opportunity to um get involved give feedback and help drive the project so um kustoff great demo um and i will share the slide deck and get you guys back here in not too distant future for the next upgrade so all these slides and the previous operator sdk talk will be available shortly as a blog post on openshift.com or cloud red hat dot com slash blog or one of the many places where we put our blog these days um and it will everything all the URLs resolve to the same place so i i'll keep track of that as well so thank you again for everything you've um you've done and you shared with us today and um give us your feedback that's the key takeaway come back come back and join us again so thanks again kristoff thank you thanks everyone take care