 How do we manage infrastructure and services and applications? Now, the answer to the question might seem simple and you might say, hey, I have a CLI and I use that CLI to interact with some API. That CLI may be talking to AWS API or Azure API or GCP or Alibaba or maybe Kubernetes. But APIs are only a middleman between us and the real deal. And that real deal is a control plane. And you might be thinking, hey, control plane is something that only Kubernetes uses, but that's not true. When we execute CLI like AWS or GCloud or Terraform or whatever we're using, it sends the information about the desired state to the API. And control plane on the other side picks that request from an API and does something. Control plane is the one doing the actual work. And it's everywhere. It's in AWS, in GCP, in Azure, every cloud provider has a control plane. And then for applications, we have Kubernetes. And in the past, we had Mezos and Docker swarm and we have Nomad. They're also control planes. Control planes are essential because the platforms that we communicate with through an API cannot just do things exactly as we'll tell them. We don't do that anymore. They're responsible for picking up information about all the desired states that we have for all the resources and making sure that they're created, that they're created in specific quarters, that they're maintained, that they're always up and running, and so on and so forth. Control planes are at the heart of everything and they are everywhere. They know what to do, when to do it, and how to do it, whatever that it is. And that it, that something is happening constantly because the role of control plane is not only to create stuff that we tell them, actually, we never even tell control planes to create something. We tell them that there is a desired state of something and then control planes are making sure that that something is created and that that something is maintained forever and ever, or at least until we change the desired state to be something else. And that something can be even, hey, this resource should not exist anymore. Another problem with control planes is that there are too many of them and most of them are proprietary. They're hidden and we have no control over them. We cannot modify them to suit our specific needs. We cannot see how they do, what they do, and so on and so forth. Most of the control planes, with the exception of very few, the main one being Kubernetes, are proprietary and hidden and we need to change that. We need a control plane that is open source. We need a control plane that is available to everybody and we need a control plane that will unify all those control planes because when you think about it, hey, if you need to deal with Archer and AWS because everybody's a multi-cloud now and I need to deal with Kubernetes and potentially with many other control planes, the job becomes a nightmare. It becomes complicated and simply not really effective. So we need a control plane of control planes, between the universal control plane that must and I stress must be open source. And such control planes should not only control all the control planes, but also suit our specific needs. What one company needs or one team needs is not exactly the same as what everybody else needs. So we need to be able not only to have control plane of control planes, a universal control plane, but also tweak it and make it do the things that we need and not what somebody else thinks that we need. Not some vendor that is trying to figure out what are the needs that can fit into everybody's use cases. That's frankly speaking impossible. Crossplane is trying to do just that. It is open source, it is community-driven and it is hoping to become a universal control plane that can be used to control all control planes. It is a control plane of cloud, of whole cloud, not of a specific vendor. And I just realized that I forgot to introduce myself. My name is Victor. I work for Upbound and we are the major contributor to the crossplane project. Today, I will not be able to show you everything that universal control can and should do. Instead, I will focus on one specific use case, one specific scenario, and that scenario will be around databases. How can we manage infrastructure and services and whatever else we might need related to database servers and databases and schemas and everything else? And I have a couple of requirements I want to fulfill. To begin with, I want to enable everyone to manage everything while making it safe and by removing the complexity. Now that might be confusing, you might not understand what I mean by that, so let me explain. I believe that we are all experts in certain areas and some of us are experts in, let's say containers, Kubernetes, others in infrastructure. There is a third person or third group and expert in databases. Somebody is very good at writing Java code. Somebody else writes back and single and so on and so forth. But we all want to be also self-sufficient. If I need a database, I should be able to create a database. However, the problem is that if I'm not an expert in databases, I might not have all the knowledge I need to create a database server, to manage the server, to create databases inside of those servers and to create users and to create permissions and to manage database schema and all the rest of the things that we need when managing, in this case, databases, but the same story applies to almost everything else. So I need help from somebody. I need the help from a database guru or infrastructure guru or a person who really knows how those things work. But I do not want that help to be based on, hey, let me open a Jira ticket and then you sit on it for a couple of days and then if you have nothing else to do, then you would create the database for me and when something goes wrong, then I will need to open another Jira issue and so on and so forth. I need to be able to create and manage servers and databases and users and everything by myself without involving the guru. But that guru is the only one who has the necessary knowledge. And that means that the database expert, the infrastructure person, the guru, should create a service that I can consume, should create a service on top of that universal control plane that I was mentioning before and dumb it down for me. That service, that interface, should be limited to what I care about, like as a person who needs a database, but he's not an expert and hide the complexity that is unavoidable, but still shouldn't necessarily be visible to me. I as a developer might want to choose, hey, I want my SQL or Postgres database. I want it to be small or big and a few other things, but I probably shouldn't care about hundreds of other tiny little important tweaks and details that the expert cares about, but I don't. I as a developer want to be self-sufficient and to manage my own applications and my own infrastructure and my own services, but limited to the level that matters to me. And that's where crossplane comes in, that's where the universal control plane comes in, and that's where composites come in as well. Composites are probably the most important feature of crossplane. Composites allow us to compose different pieces of infrastructure and services and applications, compose different resources into a bundle that is well-defined, easily digestible, and yet hides all the things that potential users should not care about. So I'm going to do the following. I'm going to pretend that some moments that I'm an operator and in some other moments that I'm a developer. But the ultimate goal for me as a developer is to create database servers, databases inside of those servers, users and permissions, and whatever else I might need. And finally, I need schema for that database and there must be something else. Yes, I need to be able also to safely connect to that database. So basically, I need a whole deal and there is a lot involved even in that simple example. So here's what I'm going to use. I'm going to use Kubernetes API because that's a sort of universal API for absolutely everything. It's not only about applications anymore. I guess everybody knows that. I'm going to use crossplane and especially crossplane composites to create all the infrastructure and all the services and basically everything that I mentioned and probably the things that I didn't mention except schema. I'm going to manage my schema through schema hero. But that will be transparent for me as a user and you will see soon how it's somebody else's job to figure out how to connect all those things, how to convert all that complexity, all that madness into something simple that is easy to digest for a person that doesn't know all the details about those things. To summarize, I'm going to use Kubernetes API because it's universal API. I'm going to use crossplane as a universal control plane that will manage absolutely everything and I'm going to use crossplane composites to bundle database servers and databases and users and everything else. And I'm going to combine that with schema hero that will manage the last mile, the schema of that database. So let me show you a definition that I will use as a developer, not a database expert to create and manage my own Postgres database. And that definition is in this case, in this file, Google Postgres YAML. Now there are a couple of interesting things here. To begin with, it's extremely small, it is limited only to the things that I care about, not all the details that are required for that database to run successfully. And those details start with SQL claim. That's the type of the interface that I'm using, the type of the resource definition. It's a custom resource definition. And the interesting thing about it is that this custom resource definition does not come from a vendor. This was not created by somebody else, from somebody, by somebody outside my company. This was created by a person or a team in my company that understands what are the needs that we have for SQL databases. There is a name, name can be anything. There is a namespace, that's a namespace inside of the control plane cluster. That's the cluster where Crossplane is running. That's the universal control plane. And I have a namespace called a team over there. That's my namespace. I have elevated permissions in it and I can create my resources over there. ID, you just unique identifier and then we're coming to composition selector. This one is important. And through the labels, I can fine tune what type of the database I want. In this specific case, I'm saying, hey, I want the database in Google and I want it to be Postgres. I could have specified Azure or AWS or on prem or anything else, whatever is supported by my company. And that can be one provider or 10 providers. And that could be one type of database or 20 types of databases. In this case, I want Google and I want database type to be Postgres. I could have skipped specifying those labels. And in that case, the provider would be whatever is the default in my company. And the database type would be whatever is default in my company. But I wanted to be explicit. I wanted to make sure that that database will be running in Google and it will be Postgres. And then there are a couple of parameters like, hey, I want Postgres version 13 and I want it to be small. I do not expect heavy load on that database. I do not expect to have much data. So I'm choosing small. It could be medium. It could be large. It could be any other value that is agreed by me and whoever else is managing this composition. And that somebody should understand the details. We're going to go through details later. For now, think of this as developer facing, non-expert facing interface that I can use and I will use. So let me create this resource and see what will happen. So Q-Cuttle, apply, dash-file name, Google, what did we say, Postgres, YAML, right? So I created the resource just as I would create any other resource in a Kubernetes cluster except that this Kubernetes cluster is acting as a universal control plane and it is doing much more than deploying applications. In this case, it is, as you see, creating database servers and databases and what's not. This will take a few moments. I will watch what's going on by executing Q-Cuttle get managed and that should show me which managed infrastructure or cross-plane resources are being created, which ones are finished and so on and so forth. Now this will take a couple of moments. So I will fast forward to the end of the process. I think it takes like five minutes give or take and I do not want to bore you for all that time. Okay, I'm back and it's finished. And we can see by the status is that it's ready. It's synced and it is runnable. So my Postgres is up and running and I can prove it to you if I go to my Google console. So let's do that. So where is my SQL? Here it is. And there's the A team Postgres. The database is there and it's up and running and it contains, no, it doesn't contain everything we need. We are going to get there in a second. For now, think of this as being fully operational Postgres server it still doesn't have the database. It doesn't have the schema. It doesn't have the users. It doesn't have all the rest of the things that we need. Like if I go to databases, there are only wait for it, wait for it. There is only Postgres, which is a system level database. I would need my own database there. But we are going to iterate now and see a more complex example. Even though it's not going to be more complex for a user but maybe more complex for a person creating those compositions and deciding how all that works together. I'm still a developer and I will say, hey, you know what? I need also my SQL database. The Postgres is not enough. I need two database servers. I need my SQL. So how would I do that? Well, the explanation is very simple. I can have manifest like this, which is almost the same as the previous manifest except that this time I'm saying, hey, database should be my SQL and the version is 8 because that's the latest version at the time of this recording of my SQL. Postgres was 13 and I still want small database, small database server. Now the new thing here, apart from saying my SQL instead of Postgres, is saying, hey, I would like the connection to the database to be written in a secret and that secret should be this one. And there are a couple of other things that will be special this time. And you will see how this definition is still equally simple for a developer but does much more from the operational perspective. So let me create that one as well. So that's kubectl, apply, dash dash find name, google and this time my SQL. And then I can watch the progress just as before by saying kubectl get managed. And we should see now two database resources being active, one of them created. Here we are. There's the Postgres database, that one is finished, it's runnable and the system is creating my SQL right now. But it is doing a bit more. You can see here by the resource, this time we are creating not only database server but also a database for me inside of that server and a few other things. I will soon show you how all this magic happened. For now I'm still focused on the developer experience later on we will see the operational side of all that. But now I'm going to fast forward to the end of the process because again it will take five minutes give or take until Google creates the database and I don't want you to look at me looking the screen for such a long time. So fast forwarding to the end of the process. Here we are, the database was created and we can see there is my SQL together with Postgres and there is something special about it not only that we got the server itself but the database was created as well. You can see there we go, it's called .dot and the database is there and we have connection string as well and let me actually show you the connection string that might be interesting so that you can see how a user in this case me a developer can now use the database or find out how to connect to the database. Here we are kubectl dot dot namespace so I'm going to my own personal namespace or the namespace of the team and say get secrets and we can see that a team my SQL is here so I can find out the connection string and the information how to connect to the database from here and if you don't believe me we can do this kubectl dot dot namespace a team get secret a team SQL or my SQL actually and then dash dash output yaml right and here we are there's the password there's the public IP and there's the username you probably know if you're using Kubernetes this is all base64 encoded I would need to decode it but that's simple command just pipe the output of a field to base64 dash d okay so I have my SQL server and that server fulfills all the requirements of my company yet it was very easy for me to create it and manage it that server has the database that I need I got the connection string and all that is happening in an extremely simple way while still providing all the levers around the security hardening and everything that my company needs even though I didn't show you that part yet but that will bear with me for a few more moments actually let's do it right away right let me show you how all that happened now I'm an operator or database expert or whatever I am the person who understands that stuff and I have two important definitions first is this one and this one is the definition of where is it let's go to the top composite resource definition this is the custom resource definition actually this is a composite resource definition which cross-plane in turn converted into custom resource definition that was the one that I as a developer when I was faking to be a developer used and this one defines a couple of important things first of all which information will be available through the secret after the cluster is created after the server and the database were created and a few other things like hey this will be called this custom resource definition will be called composite SQL or SQL claim if you want to claim it and that's what we did because claims are limited to a namespace which is a better option almost always but what matters here is open API v3 schema that one defines all the fields that I as a developer could use before there is the ID if you remember and there are some parameters and inside of those parameters there is a version in the size so basically this defines the interface that everybody else in my company can use to manage databases and can manage any type of the database as long as that database is implemented so let me show you the second one one implementation of that so I could have implementations for Azure for AWS for Alibaba for on-prem but in this case I'm going to show you the one that is for Google because that's the one I used and here it is right let me go back to the top and show you all the details I will not go into details though but all the things in a very brief way so where is the top here we are we have one composition that says hey whenever somebody wants to use Google Postgres then this should be the definition of that something and that something contains cross-plane database gcp cross-plane IO version and it uses cloud SQL instance and it has certain values hardcoded because like imagine that everybody in our company is running in US East 1 if that would be a choice I would expose it as a parameter but it's not in this case instance type the things that nobody cares about except me some settings like availability type regional tier which tier to use at least by default and so on and so forth so this is a sample of the things that I might care about but majority of people in my company don't that's simply baked into everything my SQL or actually Postgres instance and are things specific to Postgres and then we have patches these are the things that will be overwritten whenever somebody specifies something as a parameter so the version parameter will be passed as a database version because that's what Postgres in Google Cloud expects similarly we have the parameter size that you already saw before but you saw it as small or potentially medium or large and this is where it gets transformed this is where I'm saying hey if somebody says small for Postgres in Google small means this and medium means that people do not know what that means people do not care they really care whether it's small, medium, large or whatever we agree with similarly we have a second implementation of that same composite resource definition which in this case is going to be applied whenever specifies that somebody wants Google but this time my SQL instead of Postgres and then we have similar dense like before okay I'm going to use Cloud SQL instance from GCP as I said before this could be AWS it could be Azure there could be as many implementations as somebody wants and then there are some specific parameters for my SQL instances and what really matters here that we are overwriting the parameters like version just as before and the size just as before but this time specific to my SQL databases in Google and then we're having where is it the second resource here which says hey I want to have a provider config that will take the connection information from my SQL and reconfigure itself and then I'm going to use the same config to create the database that newly created the database server and that database will be called dot now this dot is the same as this one that was created here here it is right now this is simply an example and normally compositions would be much more extensive we would have users we would have permissions we would have networking we would have VPCs and all the things that you might or might not need I mean if you don't need it then you wouldn't specify it so right now we have a fully operational my SQL database in Google why in Google because I as a developer chose Google and why my SQL again because that's what I chose but for me the interface is exactly the same I'm just making few choices and the implementation differs and the operator or the SRE or whatever the role is these days is deciding what are different implementations depending on different scenarios now the only thing missing here is to create a way to manage database schemas and for that I'm going to switch from cross-plane to schema hero and the reason why I like combining those two is because they're both Kubernetes type of tools so they can easily speak to each other and easily be combined and live in the same ecosystem and that's really one of the huge advantages of cross-plane it can be combined with the vast ecosystem cloud native Kubernetes ecosystem while other tools might have difficulty doing that so I'm going to show you the definition which is stored here and it says hey there are two things first of all the database where the schema should be applied is called dot that's the database that was created by cross-plane and is currently managed by cross-plane forever and ever no matter what happens and this is the connection right I basically have the address which consists of the password and the user and the host which right now is set to some default so I'm going to change that soon and we have a definition of one table normally you would have more tables and more stuff here in schema but one should be enough to demonstrate which says hey create and manage table called products and this is the primary key these are the columns and so on and so forth right so this is a Kubernetes way of defining database schema and in this case using schema hero the only thing slightly problematic here is that I couldn't combine it 100% with cross-plane because cross-plane creates Kubernetes secrets with the information how to connect to the database in one way and schema hero expects it in a different way so I'm going to quickly retrieve the information from the newly generated secret and modify this file I'm going to do that by doing a bit of copy and paste there we go I'm retrieving the password I retrieve the IP and now I modify the definition you can see here it says routes this is the string of the password and this is the IP of the database server managed by cross-plane so all I have to do is kubectl apply dash-dash-file name named and then schema hero db yaml right and there we go it was created those kubernetes resources were created and now schema hero is managing the scheme of the database based on the desired state defining Kubernetes manifest so here we are I have here table plus I'm going to connect a new database here we are my sql what will be the name the name will be dot and the host I think that this is the host here we are and the password is this one oh that's the wrong field here is the password route is the username and the database is dot dot dot dot test it it's working here we are one two here we are and you can see here I know it's small resolution but I don't think that I can change the font size in this application and we need to trust me even though it's hard to see that in the table section there is products and products has the fields ID name and price so there we are that's all there is to it I showed you two personas I as a developer I can do everything I need to do I can create my infrastructure I can create my services I can deploy my applications and I can focus on things that matter to me and the reason why I can focus on things that matter to me is because SREs or CIS admins or DevOps people or database people or whatever the different expertise are created compositions that are easy to digest and I can focus on my stuff they can focus on providing services to me instead of responding to requests I could have went to the other partner and said hey dear John give me a database create a database for me create a database server actually and then do it here and then create a database for me and I need the schema and when I change the schema I will need to contact you again and so on and so forth so instead of me depending on others I depend on services provided from others which is a huge difference there is a huge huge huge difference between providing services to others and responding to requests from others if we do that if we provide services then SREs and operators and DevOps and what so not can focus on writing and defining and managing services that are consumed by others and those services can be specialized and fulfill certain specific needs that my organization needs that some teams specific teams need instead of being generic and all that was done through a control plane which I didn't have time to go into details but you need to trust me that cross plane is a control plane from now on is managing that database it is watching for drip detection it is doing reconciliation it is constantly constantly monitoring the database and making sure that it is in the correct state which is a huge difference compared to how we traditionally do things execute this CLI command and then something will happen and if the state changes the actual state changes in the meantime then well what can you do so please check out abound that's a company I work at check out especially check out cross plane if you haven't used it yet you will like it I promise you that because it's absolutely amazing as long as you're comfortable with Kubernetes and if you're already using it then come to our selection and ping me and let me help you and answer all your questions and so on and so forth thank you so much for watching see you next time cheers