 Dobro. Zdaj smo si počučati, ne? Zdaj. Dobro. Zdaj smo si počučati o... To je bilo misljenje titul. Zdaj smo si počučati o resociju, pletformi, in sovne, sovort. Kako se počuče? Zdaj. Zdaj se počuče. Zdaj. Zdaj. Zdaj. Zdaj. Kraterina? To je neko, da je njih, da vse to izgleda. Skupaj. Kaj, ki vse dobro na to? Mamo. Kaj, ki kaj? Kaj, da vse dobro na to? Daj, da je kaj? Kaj? Kaj? Kaj? Kaj? Z nekaj, da je tukaj. Koša je? Zelo, da ne vidimo. Zelo, kukaj, zelo, tako je več več nekaj, kako smo zelo v inustriji, nekaj? If you want to prepare a meal, you need to find all the ingredients you need. And for some of those ingredients, you're going to go to a supermarket next door. For some others, you're going to be adventurous and go to woods to find mushrooms or to the market and so on and so forth, right? When you need to prepare a meal, you need to gather all the ingredients first and then assemble them all together, right? That's how cooking works and that's pretty much what we are doing in our industry as well, right? You need to figure out to find the cables for networking servers, these drives, and so on and so forth, right? You need to find all the pieces and each of those pieces do not mean much individually, but when you put them all together, you get something meaningful. You get the cluster that can run your applications. You can get the databases. You get this or that, right? It's very similar to cooking. Now, since the long time now, right? We don't anymore go in to the shop to buy servers, cables, and so on and so forth. Some of you might be doing that. Some of you might be running your own data center. That's great. But we got a new thing, new thing, like 15 years ago, 20 or nobody knows when, and that is pretty much giving you similar effect like going to very, very big supermarket to shop for ingredients. You get the menu, you choose what you need. Do you need networking in AWS? You get networking. Do you need storage? You get storage. Do you need a server? You get the server. Now, the problem with that approach, not the problem, but all those are designed in a way that they give you most of the time, not always, individual ingredients, individual components, so that you can assemble something that is exactly as what you want, right? Because otherwise, Google Cloud and Azure and AWS could not predict what is it that you need in advance. They give you ingredients. And then you, ops person, or dev ops person, or platform engineer, or whatever we call ourselves these days, you would actually assemble all that together, probably after receiving a jail ticket, saying I need the database for yesterday, or something like that, right? You're assembling all those ingredients into something meaningful, into a meal, or in case of what we are doing, let's say a database, or whatever you're supposed to assemble. Now, that is very different. That menu, this what providers give you right now, ingredients is very different from the experience when you go to a restaurant, right? And that's the experience that developers are most of the time expecting today, right? They don't know what is exact, what are all the ingredients, what are all the things they need to assemble to get a burger, or a soup, or whatever they want, they want a ready to go meal. I want the database. Don't ask me about VPC and subnets and what's not, right? So that's usually what we consider today being platform engineering, right? You understand ingredients, you're the cook, but you're not offering those ingredients to everybody else in a company, you're assembling them into certain number of choices, right? Burger against soup, or whatever the meals are. The database, a cluster, a backend application. Now, what we are going to do first at the very beginning, and you will see it will change drastically later, we're going to use crossplane to see one quick example, because what I want to talk about is not really what I'm going to do now, but I just need to get there somewhere. So we're going to use crossplane. Crossplane is a way how you can create a control plane, how you can create that menu that allows people to choose what they want. And behind that, you know what the recipe is. The something equivalent of a recipe would be what we call compositions in crossplane that will assemble all those different ingredients to create something meaningful, right? And you will see that in very, very soon. Now, what we are going to try to do today is I'm going to follow the example of being in a restaurant ordering a dish instead of ingredients and try to see how we can make developers feel, get some very, very similar experience with our services. So we're going to skip the setup. Wrong button. There we go, because we did that in advance. And we're going to do two things, three things first, right? We're going to create a cluster, because cluster is required to run something. We're going to create a database. We're going to create an application. And then something bad will happen, but we'll get there. Yeah, she has dual personality today. She will be acting as a developer first and then something else. We'll get to that something else, right? So here we have an example of cluster, right? This is a cluster that, think of it like ordering something from the menu, bit some special requirements. A person, imaginary person here, she would say, hey, I want something in AWS. I want it to be, yes, I want medium-sized nodes because I have no idea what's happening in AWS. I want it to be three nodes. I want some namespaces. I want traffic to be installed in the cluster. And I want that cluster to be populated with credentials so that I can access AWS from that cluster, right? Now, I don't have time to show you all the magic, all the ingredients behind all that. Find me after the talk, I can show you all the thousands of lines of YAML and some other code in it. But the point is that as a developer, you just say, what do you want from the menu? One of the items in a menu is a cluster and create a cluster. And Katarina is going to create a cluster now, probably. We didn't recur. So I'm not 100% sure what will happen, but we'll see. There we go. Cluster AWS Store, huh? In the namespace something, A team, right? And that creates a cluster. And if you want to see what are all the ingredients in this specific thing that you ordered, we can do crossplane trace and you will see that all the components that are in this cluster are listed there, right? This is what is happening in a background as a result of you invoking, creating an instance of something because that cost composite resource. So a bunch of things happening there, that does not matter. What does matter is that takes approximately 20, 25 minutes to create all the components required in AWS. It's a bit slow-ish. So what we are going to do is, we already created the same thing in advance. We are faking it a bit today, right? So the same manifest that she executed earlier was executed before this talk, and you can see, okay, now we have the cluster. It's all up and running. I don't care about any of those things because what they really ordered is a cluster, so let's just move on. I'm not sure what's next. So I need to discover. There we go, yeah. So the next thing is a database, right? That developer, Katarina, needs a database. That's another cross-plane composition that creates custom resource definition in Kubernetes that allows us to create resources based on it, and then behind the scenes, things will happen. And in this case, we are creating SQL claim based on AWS. If we will postgres, version 14, medium-size, there will be database in that server in myDB. Since that server will be created in the management cluster, we are going to push the credentials of that database to a secret store in AWS, and then we are going to pull it in a different cluster, the cluster where application will be running, the one that was created a few minutes ago, and there is some SQL schema that will be generated in that database, right? So that's fully operational postgres database with all the banks and lease vessels, with users, with everything, with authentication move to a different cluster and with schema. So we're gonna fake it again. Katarina is going to apply that manifest. It takes, you can see if you do cross-plane trace, we can see what are all the components involved in running that database. There we go, all that stuff. Since it takes time, 15 minutes, we did a copy of that in advance, so we're gonna fake it that we fast forwarded for those 15 minutes. And we should have the, there we go. Database fully operational, all the components, all the ingredients are there ready. Now, there's one more thing missing before we get to the real part of this talk, which is, you'll see what it is. Anyways, I will need an application. There is a definition of an application. This now was not done in advance. We are doing this now because it doesn't take much time to deploy application. I'm saying again, I don't know what are deployments in service meshes and virtual services and this or that, right? I'm obstructing application into something called up claim. A couple of parameters run it in that namespace, use that image, that's the port, that's the host and do it in that cluster that we created earlier and connect to the database using the secret of that database so that you can speak with it, right? So now if you do just kubectl, apply, et cetera, et cetera, et cetera, or some other things, there we go. Apply and if we do the trace, you should see that the application is running, right? And it will be connected to the database and it will automatically happen. I think that I'm using dapper behind the scenes and so on and so forth, right? Those are the resources that were created. You can see by the name what it is, the deployment namespace and so on and so forth. Well, that's all great, right? That's what most many teams, not in this, not using this tech, not using those tools, but that's what many teams I speak with do when they think about platform engineering, right? How can I enable people to create stuff, to update stuff, and to delete stuff in an easy way without them spending 75 years learning Kubernetes, AWS, and everything else in between? Now, if you go, yeah, there we go. Look at it, we already did that. The problem that we are having, if we consider platform engineering being just enabling people to operate, is that they end up being blind, right? That they can maybe see some basic things. Maybe the status of what they created is true or false or something like that, but most of the time they're blind. They would like imagine the restaurant where people are having, being blindfolded and then you don't even know whether your mirror was served, whether the plate is empty, whether you got something, what that something is and so on and so forth. What we're really needing is data operations. People need to be able not only to create resources that they need in an easy way, but also observe resources. And that observing resources need to be tailored for specific use cases. Not some generic dashboard that works for everyone and go figure it out, but dashboards and data created specifically for the services that they are instantiating. And I think that this is where I passed them. There we go. And she's not the developer anymore. Now she knows about observability. Yeah, now I know. And I actually asked my question, what is so bad about those blindfolds? A few years back I've been in a dark restaurant. Everything was completely dark. We conserved a surprise menu. Blind waiters were actually bringing it to us and it was a lot of fun. So why is this so bad in our situation? Well, the big difference is, a few years back I was having fun with my friends. In my free time I'm up for adventures and raising the unexpected, but definitely not something I want to have at work. And definitely not something I want the developers I'm working with having to face. Because it would be the equivalent to this situation. Let's just think about you're building an app, you're running a different production, everything is running fine, you're moving on to the next features, but suddenly an angry customer is calling you. And they say that this and that feature isn't working, you have to get your stuff together, you have to fix it right away and you have to leave everything behind. This is already where the first problem happened. It was actually a customer informing you that something is not working. While it should be the other way around, that you are noticing that something is going on and that you can proactively do something about this, inform customers and by that already soften the situation. But too late for that. So you continue troubleshooting, you have some logs, but they don't really tell the whole story. You maybe get alerts but way too late and it's super hard to pinpoint the root cause and by that also to fix the issue. So what can we as platform engineers do about that situation? How can we make this better for developers? In my opinion, one of the most important points to get this right is to allow developers or to enable them to trust your platform. Because by building up trust between the developer and our platform, they can actually deploy confidence onto the platform. And how can we do that? One very important point, transparency. Developers need to know what is going on in our platform. Is the platform up and running? Is the problem they are facing maybe related to our platform? So we need to be transparent about that. We need to give developers insights to the platform and then later on do the same for their applications so that they can be confident and trust their applications in the end and avoid this situation. This problem is also very much reflected in the Dynatrace data for observability report of this year. In their 88% of tech leaders have claimed that the complexity of their tech stack has been growing in the last 12 months. And about half of them is thinking that it also will grow within the next year. About 86% have claimed that their tech stack is causing an explosion of data that is beyond humans ability to manage. And 81% said that configuring monitoring tools and analyzing data is distracting teams and is actually slowing down innovation. And what makes all of that even worse is that only 9% of applications are fully instrumented for end-to-end observability. The gist of all that, we have a huge problem. The complexity of our systems is growing. We don't have proper monitoring or observability configuration because it's hindering innovation. It's slowing us down from getting features out. And that results in very few applications being actually instrumented. So, I think or at least hope that we can all agree we have to do something about it. And what we can do about it is making observability a key goal of platform engineering. And according to the state of DevOps report from Papad in 2023, already nearly 40% of applications have defined observability as a key goal of platform engineering. So, we're definitely not alone with this opinion, but still it's less than half of companies that really focus on observability in their platforms. So, what we did today is we built a demo to show you how you can integrate observability to your platform, to avoid your developers sitting blindly in a restaurant hoping that they will get a meal one day, maybe also not. And I will show you that soon. Just one very quick slide before that. I wanna quickly talk about what observability actually is. Basically, observability goes beyond traditional monitoring because back in the days with monitoring configuration, we were always expecting some errors and were then creating dashboards and alerts for those errors. So, basically, we could handle known unknowns very well. But we had a problem if the error happened that we couldn't expect. So, a so-called unknown unknown happened. And in this case is where observability really shines because observability combines data from various sources, mainly metrics, logs, and traces, but it can be very, very more platform sources to get data from, to analyze it, and then to prepare it in a nice way to the user, help drop shooting, inform them about problems, and so on. Today, we're going to use dino trace for the demo. It's obviously my tool of choice, but all the concepts we are talking about are applying to any other observability tool as well. So, yeah. Okay, let's get started with the demo. So, you have already seen those crossplane battle trace commands that we executed. It's a great way while developing crossplane compositions and trying them out to see if everything is up and running as expected. But still, you don't want to have this as a troubleshooting tool in production, right? I mean, I love using it. I don't get me wrong, but you always have to connect to the cluster. You need to have credentials and also you need to know your way around Kubernetes to actually then troubleshoot the resources. So, let's see what we have in dino trace, actually. In here, we first of all have a Kubernetes application. And here you can see, we should probably make it a bit bigger. You can see that we have two clusters here. Why do we have two? Well, actually, we first deployed the crossplane observability demo cluster, which is just a local kind cluster. And this one is our control plane. This is where we deployed all the resources earlier in the demo, too. So, the application, the cluster, the database has been deployed to that one. But then we also have the 18 cluster, which is the one that we just created during the demo, but faked a bit, so created a bit earlier because it just takes some time. Yeah, just those things are just because it wasn't connected to internet earlier, so don't get distracted by them, please. But let's open the 18 cluster. And in here you can see already we got a very quick status of the cluster. We have several problems, everything is up. We have two vulnerabilities that we should definitely care about, especially because yesterday it was just one. We have CPU and memory stats in here. We can click around, we have a few more charts. We can see if there are any problems like on the first glance and so on. But still, if I'm a developer, if I have no idea what's going on, this might be a bit hard for me. So, what Victor and I did is we went ahead and created a dashboard that is basically an entry point for the developer. And this is a very simple dashboard now. It just shows, hey, do we have vulnerabilities? All our nodes and pods healthy, so really nothing special. But what it also shows is that this cluster overview dashboard can be customized to your clusters and to your needs. But the question you're all probably asking, where is this coming from? The nice thing, nothing I'm showing you is created manually. So, everything has been created with crossplane. And if we now go back to the terminal and I show you the cluster that or the configuration for the cluster, oh, sorry. And I show you for the configuration for the cluster that we actually deployed. You will see that suddenly there is a dinotrace configuration in here. And this is because Victor very nicely added dinotrace configuration to the cluster composition. So, it's like one more ingredient that we are adding to our big recipe. And you can see it's enabled. We need to pass the ID of the tenant or the URL of the tenant that we wanna create the dashboards and the Kubernetes clusters on. And we need to pass some credentials. And that's it. Those four lines of Yammer is all a developer needs to do to get this dashboard and the dinotrace configuration. So, quite doable, I think. And what we also can do with that is for different clusters we have different people using them. So, the A team cluster is a developer cluster. It's used by developers to deploy their workloads. But, for example, the crossplane observability demo cluster. So, our control plane is used by platform engineers. So, what we did is we created actually this overview dashboard that you already know. But we also created a crossplane metrics dashboard that gives us more details about what's going on with crossplane. Especially, for example, like the average time to readiness that I like to use to check if everything is working as expected, and so on. So, by that, you can provide different levels of information for different users because not everybody can handle the same amount of information and not everybody needs the same amount of information. Okay, we did not only deploy a cluster, right? We also deployed a database. So, pretty much the same story. We can navigate to the Clouds app and select databases. And in here, we will see that we have, yes, this one it is, a Terraform database or a database that is called Terraform. And again, we can click around, we see some metrics, and so on. So, where is that coming from? Same thing, but in this case, not the cluster, but the database composition. So, if we open the db.aws, you can see, again, down here, we have a dinotrace dashboard configuration, not much more configuration, we are enabling it again, we are passing credentials, and we're giving it some information about the application. So, which cluster, Kubernetes cluster is it running on, how is the app deployment called, which namespace do we have, and a select channel, which we will need later on. And again, that's all. I, as a developer, who doesn't know anything about the internals of our platform, have to do. And this result in the cloud that I just showed you, but also, again, in an application dashboard that we created to give developers a first insight, oh, sorry. In an application dashboard that we created to give developers a first insight in the application. Again, it's just a sample dashboard, but we have in here vulnerabilities, we have application stats, and also stats about the database, because they belong together. You also see the latest logs here, but still, everything is a bit boring, right? So, let's go ahead and generate some load. We have prepared something on DDoSify, and what we do in this test plan is two steps. The first one, we are creating a video with proposed request, and the second one, we are getting all the videos. So, let's hit next. Let's be very, very brave and do 10,000 requests in 20 seconds, because, sorry? Okay, 100,000. 100,000, okay. It's a live demo, right, so. Okay, let's click next. We are sending them from all over the world, because we can. Let's save it. And let's start it. And this is now running and sending all the requests that I just showed you. And while it is starting, I hope that we can already go back to dinotrace and actually observe what is going on in our application. This is maybe a bit of too much of a time frame. Let's change it to 30 minutes. And yes, we can actually. So, here you can see that we already have many traces for our application. It's, in that case, many get requests. I hope I can also find a post in here. Let's see. Not yet. Then we will just use one of the get requests. And now we have a very, very, very simple trace, because we have a very simple application, but still it's good enough to show the idea. So, we can click on it and we can get information about what's happening. We can see, hey, somebody did a get request to our application. It resulted in 200, okay. It's running on port 8080 and a bit more like request headers, timing, logs, and so on. And by that, we already have much, much better troubleshooting possibilities than the developer we talked about in the first scenario. And we actually also need it, because if you saw the notification up there, in the meantime, we got a message from dinotrace that we have multiple application problems because maybe we have been too brave during the presentation. You go, you go brave and you die. Yeah, so we can click on that. We can utilize everything we just showed you to troubleshoot that problem. And basically, what we wanted to do is we wanted to show how you can really make observability easy for the developer by integrating it into your platform. And with that, oh, sorry. And with that, ultimately, remove those blindfolds, give everything the meals that they want to have without being scared of the meals suddenly disappearing or having something inside that you don't want because you know exactly what is going on, the waiters know exactly what is going on. And I think that's the situation we should aim for in our companies as well. And with that, at least from my side, thank you very much. I don't know if you wanna add something. Questions? Anyone? About anything? Life, universe, no? Ah, there's one. Yes. To beg, beg in the application. Oh, yes, so I used actually go templates to write part of my compositions. Yes, cause now Crossplane supports functions which allow you to define it in any way you like as long as you provide the output back to Crossplane. If it's a network, you should see it, you mean if it's a network that prevents you from seeing the dashboard itself, if the network is problem, you should be, if you did the observability in terms of dashboards, you should see it there and you should be notified, assuming that, you wanna answer? Yeah, assuming that you can access the dashboard themselves, right? Now if your network is messed up that you cannot access that either and assuming that also you did observability in a way that if you cannot reach something that's a bad thing, right? Ultimately you can do something like, what's the service? Uptime that verifies whether your dashboards is accessible as well. Anybody else? I think back there is one question. I don't know, I don't see, yes, yes. Yell, scream. Through the CLI. So I tried to summarize and you correct me if I'm wrong. When I executed Crossplane trace, we saw crossplane resources but can we add dinotrace there as well? Yeah, something similar to that. And also when it comes to other crossplane resources, so for example, when a function is wrong, let's say, if you could really see what exactly happens, what exactly the crossplane controller, let's say, has executed and, yeah, what trigger for those? Not necessary, you cannot see all those things from the crossplane trace command because there is a limited space there. But generally speaking, if you describe the resource, any resource, including resources that are, for example, creating dashboard, you get events of what's happening in statuses. And you could import those into dashboards as well, which I think that is what Katarina did as well. Exactly, that's just what I wanted to say. So crossplane is great in emitting all information it has in events. And you can read all those events in Kubernetes and then with the dinotrace query language, you can actually process them. You can also create workflows, like when you notice, hey, there's a new claim, you can get the things underneath them. It's a bit hard to do. I'm currently trying it and I'm getting there so it is possible. It just requires some trial and error and failing and hopefully succeeding in the end. But it's definitely possible. Thanks. The setup? Yeah, the part that you scared. The composition. The setup, yes, yes, yes. So... Should I show the script or the command? Not the setup section of the slides, I think. Was that the question? Yes. Yeah. So whenever I create slides, I always put setup even though I'm executing in advance or if you want to follow the same slides you can actually get to the same point. And there is a setup. I like niks, so there is devbooks that spins up all the tools you need and then probably creates a cluster, setup.sh actually creates everything. Thank you. If you do that, I have to apologize a bit. There are a lot of dinotrace steps that are not automated, so you need to click around a bit in the UI, but you can get a demo tenant to also do that. And maybe I will have time in the future to also automate those so that it's less clicking. Two more? Okay, I heard two more questions, is what we are allowed. There's one, one hand, two hands, I don't know. Pass the mic, throw it. Maybe I could go first. Maybe a silly question, I'm not sure, but would we be able to do the same setup you did or something similar, but not using dinotrace? Maybe some other solvability tool, I don't know, Grafana? Definitely. I mean, I haven't used Grafana too much, to be honest, so I'm not quite sure about the internals, and I'm sure the Kubernetes app or the Clouds app are different there and behave differently, but building dashboards, adding them to your composition, as long as the observability tool you're using has an API that you can actually add in your composition and that you can call, you can do this with any tool you want, also with non-observability tools if you wanna do something, you can have a lot of fun with it. One of the takeaways from this talk, among others, not the only one, is that you can assemble resources whichever you want in compositions, and they will be orchestrated and created ultimately one way or another. Last question probably, there we go. Sorry. Maybe not a question, but observations, so essentially we did the same by building our own compositions if you injecting custom instrumentations also security scans and things like that, so cross-plane is pretty powerful in terms of what you can do in terms of building your own compositions. So, good stuff. Thank you. Thank you. Is that it? Are we finished? Gonna kick us out? Absolutely. Okay, then we go.