 So, I want to talk today about how to install Kubernetes on Demi. Because everyone wants to go into the cloud and wants to trust those cloud providers. Many of the companies still want to have their Kubernetes and plus in their own data center. And many people think it's very difficult to install Kubernetes on Chrome. Like you have to install the operating system and all this kind of stuff. But actually, if you're using the Y2, it's only around 30 lines of code you really need. And I want just quickly show what kind of lines you need, what kind of tools are used in the background. And then I want to demonstrate it. So, this one is not working. This one's done. Also not working. So, that's the idea we want to do today. I just want to go to the slides about 20 minutes. I think we don't do any questions during the slides because I really want to go quickly over them. Give you an idea of what I want to do. And later on I will use all those tools in the demonstration. So, if you have questions, I'm happy to answer them during the demonstration. Because then I can show you the tool, I can show you the configuration. Because in the slides sometimes I shorten the configuration just for the purpose of showing you. So, there's not all the information inside. So, but I wanted to give you a quick idea of what I'm doing, what I want to do. So, the demo is 30 minutes and after that we could have just a quick Q&A. I personally work as a data center network engineer for the last 15 years at Strato, which is a Berlin-based hosting company. So, we own our own data center. We have one data center in Berlin, one data center in Karlsruhe. And therefore I have personally quite a bit of experience to work in a data center and what kind of technology, what kind of hardware you're using there. The magic tool I want to show today is Typhoon. Not anyone know Typhoon? Okay, no one. So, it's great. If you know and call as they have the commercial variants, that's Tectronic. But you have to pay for Tectronic, only 10 notes are for free. Typhoon is actually free. It's their open source counterpart. It doesn't have all of the feature, but it's more or less very capable of installing Kubernetes cluster. So, you're installing a small Kubernetes cluster. It's almost all the time very up to date. And you have a very short declarative configuration for Typhoon. So, as I told you, it's only a few lines of code. It's actually free. So, it's free in the code and it's free for cost. So, you just can use it and install your data center. So, it's very practical for just lab environments or for data center. I personally didn't try the data center. So, my scale is normally around a dozen of notes. I'm not sure if I will release the scale to a few hundred notes because as you later see, you have some sort of SSH loop to put config on the machines. But maybe there are some sort of limits how far it will scale. And if you want to look at it, it's on GitHub and it's free. Right now, they're delivering Kubernetes in the version 110.2. So, it's absolutely recent. That's the latest release you can use. And so, you don't have to use fork 4.0 or this extra month propagation flag because it's in beta in this version. You can install it as a single or multi-master. It depends on your needs. In my demonstration, I just want to use the single master. It's easier to get it up and running. You can choose between two different network backends between Calico and Flannel. I personally like Calico much more. So, I chose it but there's really no difference. I mean, from the point of fork 4.0, you can use both of them. But I personally like about it having Calico much more. It's TLS enabled by default. And as well, all the airbag boots are there. So, you can also have it just to see how the TLS is working, just what the airbag is working, just play around with you. So, it's really an awesome tool. You can also direct not only against your private cloud in your data center, you can use it against AWS. Your private cloud is going against digital ocean and also Google Cloud. So, you just have to learn this tool once and then use it against many different cloud providers. Right now, there is support for two Linux distributions. Surely, it's in container Linux. The distribution from CoreOS because it's part of the CoreOS brand. So, certainly, they have support in container Linux. And CoreOS got acquired in January this year from Red Hat. So, a few days ago, they added also Fedora Atomic because they are now part of Red Hat and therefore they have to support the Red Hat distribution or the container Red Hat distribution as well. But that's, the Fedora Atomic is still in beta. I personally didn't try it. So, maybe it's working, maybe it's not as wide. I can't tell anything about that. So, now I want to come to the configuration. The lines you actually have to write there will be three slides containing the configuration. I had to put them apart for the visibility reason. The first one only describes actually your provisioning endpoint. You have your custom name, you have your matchbox endpoint where provisioning is actually done. So, you configure matchbox via the stool. I will explain matchbox later on. You describe or you choose which kind of container Linux channel you want to use and which version. Test install means I downloaded the CoreOS before in the local directory just to speed things up. And autolog in TTYSO is very, very nice for debugging because then you won't have to type the password because on the serial console you are not going automatically. So, I would use that in production for deployment as it turns it's very handy. So, that is just describing the basic idea of how the cluster looks like and what kind of endpoints you want to use. The next one will describe more points about your cluster. So, you have the domain name, what's the cluster, what kind of endpoints you want to use. So, I just use an example.com domain. You know, SSHP, it's certainly one of the shortened. You have the SADM in this directory. There will also actually one file, so I'll just split it up. The last one is you define your controller. In this one I have only one controller because I do single master. If you want to do multi-master, you will just extend the list with more information. I have the second one too. And I only have in this slide one node but in real deployment you will put in more nodes. You need the MAC address for the node because you want to do the MAC deployment. That means the server, our PX booting, and you need the MAC to identify the machine or kind of package. You're actually wondering how is it possible to just use maybe 50 lines of code to configure a whole Kubernetes cluster. The idea or the solution behind that is that Typhoon doesn't stand by itself. It uses a whole tool chain of tools to do the jobs. But all the tools are abstracted away from you. You can look into it, you can tweak them, but normally if you just want to use Typhoon you don't have to use that. But I recommend everyone to look into them because you get more knowledge. And the first one you're using is Terraform. Matias mentioned already, so you have your infrastructure as a code just described, or you just described Terraform manifests to describe infrastructure. Typhoon is using heavily, or is heavily based on Terraform. You're using U2Q to generate all the Kubernetes configuration, the TLS certificates and everything Kubernetes needs. You also generate U2Q as self-hosted Kubernetes cluster. So we are going today to deploy a self-hosted Kubernetes cluster. That means the control plan of Kubernetes is actually part inside of Kubernetes. So you can do all the rolling upgrades which are really nicely done in Kubernetes with your own control plan. Therefore you have to boot a feeable control plan and then switch into the self-hosted control plan. All the stuff is done by Typhoon. You don't have to do it by yourself. As a basic operating system you're using container linux. Container linux, for the configuration of it we're using container linux config and the one which is actually configuring the container linux is the tool called Ignition and the last one which you're using to get all the pixie boot and all the stuff out to the machine is Matchbox, which we will see later. So Terraform, I don't think I have to explain it much more. I mean it's tool to design your infrastructure in code and it has many popular service providers like you can go to Packethost, AWS, Google Cloud, whatever. So there is a whole list of providers describing your application in code and the actual idea is that Typhoon is actually a Terraform module. So it uses all the resources Terraform gives kind of Typhoon use, it's just a Terraform module. Bootcube is used for self-hosted Kubernetes cluster. So it's just generating the temporary control plan and all the necessary assets of that. So like the empty assets and actually manifests for the static or for the boot control plan also the cube config. So it's just a tool generating config and the second one is configuring the ethereal control plan with the actual control plan you want to use. Container linux is actually very often called CoreOS but that's not right. CoreOS is the brand, the company is CoreOS and the distribution is called container linux but very often it's called CoreOS but actually that's not the name of it. The interesting thing about CoreOS it doesn't have any packet manager so you can't get installed or young installed whatever. All applications are containers or system D units. So you install CoreOS at the boot. It's immutable if you want to change something at the configuration you reinstall it. And therefore it's very easy with Typhoon just to do the process over and over and actually Typhoon will do it for you. It will reinstall the machine for you. And as I told you before they got acquired by Red Hat in the beginning of the year so if you don't want to support Red Hat you have an alternative. You can use flat cardinus which was forked a few weeks ago from the Kindfold folks here in Berlin. Their fork is what's called flat cardinus. That's the car in the train with the containers or in the train with the containers coming out on top of it. And they approve of CoreOS so they ask them could we fork because we want to do commercial support because Red Hat doesn't give you commercial support for CoreOS but for container linux. Here you're put by commercial support. So the idea is to configure container linux using the container linux configuration which is a very easy, very human readable YAML file which you can write down with a declarative idea what the machine should do but actually the machine can't consume it. So you have to use another tool that validates your container linux config and it will also transform it and this step is mandatory. You have to do it. So that shows it on purpose because you don't want to debug and not booting a virtual machine in the cloud you don't get any SSH located into it so you can't look at what kind of errors in the conditional. So there's always the two between the container linux and the ignition config to validate and transform the config. With the config you can configure user, storage, network, system D, units, etc. and docker so it's a very easy format and I think it's very easy to see and to use. For example, a user is defined like that so I think I don't have to say anything about it it's very easy to read, very easy to understand if you've ever written something young for whatever config management system you would easily understand that. That's an example for storage so you're mounting a disk format with Sparta FS say let's do a root label and that's the idea behind then you can put a file on it so I'm using the root file system and I'm writing a file into it so that's a very short example normally you put it in a Kubernetes configuration which is much, much longer and you could also put in system D, unit, files, whatever you would like so that's the idea of how you configure container linux and the whole configuration or the whole configuration specifications online you can look it up if you want you can do the tool which you need to get from the container linux configuration to the ignition with the config transpiler so that's the tool which will translate it and also validate it it's translated and looks more like this so you have the container linux config you're using config transpiler you're getting the ignition config the ignition is running on the Kubernetes on the core or container linux host is putting the ignition config and it's writing the system D files or whatever you configure to the system the system is booting up and then you have a config up on the Kubernetes node on the left hand side you see the container linux config on the right hand side you see the ignition config and I think everyone would agree that we don't want to write that by hand so I mean you could do it that's not the point that you can't do it but I personally wouldn't like to have this syntax in them or whatever to edit it so that's always go to container linux go to the config transpiler go to the ignition file the ignition is actually the tool, the demon the system D unit which only runs once at Buddha it runs very very early when the unit run fs is started before the root file system is mounted before the P would change from the unit run fs to the actually operating system is done so you can manipulate every disk you can do everything, you can format them, you can repartition them you can put L, V, L on top of that so that was the biggest problem with cloud in it because cloud in was running when the whole root process was done and then you couldn't repartition the disk with ignition you can do that you can build a rate analysis whatever you want it's very powerful so before the user space and what ignition is doing it's pulling it's configuration from a sort of truth, in our case it's pulling that from the matchbox server so it brings up to the matchbox server the matchbox server is an HTTP and GRPC servers GRPC they're using to configure it so Typhoon with the help of Terraform is speaking GRPC to matchbox to get all the configuration to the daemon and the ignition client is just doing a curl to the HTTP endpoint and it's getting all the ignition files adjacent based configuration to get the machine to the state where you want it matchbox is easy to deploy it's just a binary which starts the go binary or it's also delivered as a container so just go and do docker whatever or docker pull and start docker compose so for matchbox has the idea about groups which matches a machine to profiles based on labels the labels are most likely mega addresses because that's the most of the idea which uniquely identifies a certain data center so you could also use the UUID for stage level or the serial number of the main board or whatever you want but I never saw that before it's possible but all the installation of matchbox I saw there using the mega address because it's unique and it only changes if you change the network card but normally you're booting from the onboard card so this mega address is not changing at all so only if you replace the machine the other one is the profile which you have in matchbox the profile says what kind of type this machine should become so it senses that it has the idea that the template is pointing to the right ignition file for the purpose of the machine so we have a nice picture of that so we have different mega addresses we have the groups here the groups are matched on the mega addresses and the groups shows them then which kind of profiles the machine should become so that it should become a quantities master or work or entity or whatever so you have the idea of between the groups and profiles and a group file looks very easy so the important part is here you have the mega identifier which identifies the node and you have the profile which is attached to that special node with that mega address and for that node the profile looks like that this you're saying what kind of ignition file you should get like the XED YAML and what kind of operating system which version and you're also saying the core as config URL is the URL for the ignition file here we're downloading the ignition file you can put your variables inside the UILG and the mega address which is most of the time used instead well not the combination of most of the time only the mega address so just a quick recap of the many tools how they interact with each other so the ideas we have the typhoon module on the left side bottom which is a module of Terraform so you're just executing you're just starting on your local machine Terraform then speaking the first step to bootcube a binary on your local system it's generating all the necessary assets TLL, TLS certificates and cubecon for whatever you need and it's putting it in the secret in the asset directory but then Terraform is doing a grpc call to matchbox in this grpc call it's still container-node-config matchbox has an integrated container-linux transpiler which translates the container-in-config to the ignition config and then you can start your server the gsmars, dhc, dhcp and pxl, why am I left totally out of this talk because it's just mega address and gsmarm they should rematch the mega address and the p address and the name you have in the config but that's easy to do that, the machine boots up and goes to the matchbox because it's an ignition config it becomes some sort of type like a master, worker, etc if the machine is booting up you're copying why scp a bootcube generator config to it and after a few minutes you have a cold running Kubernetes cluster so on this I want to demonstrate now so hopefully it will work excuse me, I have a question I don't really know the tool named ignition and from what you told it seems like the only purpose that it serves is to take the yaml file and then to put it into json actually the config transpire which is just translating the body to it it's just a part of ignition ignition is actually the tool which is running on the container Linux system during the first and this one is putting the json file and the json file is called ignition config and the tool is called ignition and the translator or the compiler or this is called container Linux transpire so the names are really horrible and you get a few minutes to get all of this right because I don't like the names I don't like how they call it but I mean I can't choose it yeah I don't like I personally think it's very complicated so and therefore I want to tell you guys that you don't have to go through the same amount of effort to get everything straight and out how the different tools interact with each other and also how long does it normally take to totally reboot the core OS so not the core OS but the container Linux and it really depends on the hardware because the most time a modern bmw server spends is in the BIOS process so initializing on the network card, write cards, whatever so the boot process is really fast you will see and the boot process takes on this example only a few seconds because I'm using virtual machines instead of real hardware I do pixie boot them so I do the same you would do with real hardware but I don't use real hardware because of the point that every time you reboot it will spend minutes in BIOS or post the initializing for reference like how big is the binary like 300 megabytes or 250 whatever so it's really small thank you will you post your presentation anywhere yes I'm not sure where but maybe just link in the media and speaker deck or whatever but we will do that somewhere probably we will find I'm not sure we will post it in the upstop or just link to somewhere but yes so actually the whole demonstration is running on a virtualization machine actually standing at my home so I hope the DSL connection doesn't die so what I'm doing is I'm having here all the configuration you need we look into the provider first the provider is just to get the certificates that's the most important part because the DRPC connection to matchbox I use TLS to authenticate my clients so that's just for the certificates and for a real cloud provider you would put in your password for AWS or whatever they use and that's the whole cluster config I have I mean most of the parts we saw now the edge keys we learn a few more of nodes I have 5 nodes in the clusters a little bit bigger but not really a huge one so what we are doing is here we are starting Terraform we will have the point of apply and you get what you view what you want to do and just type yes all the configuration we swear they are generated and you are ready to go I also start now the virtual machines so I use KVM for that and those KVM machines they really do pixie boot so they are empty they don't have any operating system on them I use two this panel one for the operating system and one this I use for 4x later on now you see the machines already are putting their stuff from the matchbox so right now they are just putting their image like the NANDROMFS and the first machines already booted the NANDROMFS and they are putting their initial files so without BIOS post stuff they are booting really really quickly up and we are now looking why the serial console on the SNOTE and what we are doing right now is actually installing the most important part we are installing 4S on STA so I am not just pixie booting and let them into machines in the NANDROMFS and then using machines so I will boot them up with a RAM disk and from the RAM disk I install we are installing 4S on the actually machine that will take a few minutes not a few minutes a few moments and then the machine automatically will boot and we will see hopefully that short because I am connected on the serial console so we will see if it will boot any questions so far on the other terminal you see that typhoon is still trying to copy all the assets to the machines because the machines are not initialized yet the pre-boot process the real boot is starting pre-installing environment we have an SSH team running but it is running on 2222 so as an administrator you can log into it but Tyrofone doesn't know from this high SSH port which is waiting for the real boot and there the SSH port is on the normal port number and then you will see that the configuration is pushed out questions when one of the machines restarts on which services does it depend it probably has to call the ignition again right no if the machine reboots as I told ignition is only running once at the first point so if the machine just reboots it will just come up with the same configuration so you are not depending on your matchbox server, PC environment maybe for the PC or for the DHCP and for the configuration if you don't put the IP address into the ignition file then you may be dependent on the DHCP server in every movie boot or failure scenario but you can easily put your IP address into the continental loose configuration and configure the server statically so you are not depending on anything so just for the install process and as you can see all the machines they got or it's running away so the SSH connection was successful for example here for node 2 and you push the configuration to the server the next one is we want to start bootcubes so that means now we are starting the if you will control plan for Kubernetes and then want to switch to the persistent control plan of Kubernetes so I tried that before so that's on the machine running so I am not sure how far it is right now no containers started you actually can't see it very well because the size is not so great actually that's one line the one line means and bootcubes wants to connect to the to the IP server which is not yet there so bootcubes we are downloading the bootcube container and then we are starting it and what bootcube actually did is to generate those files that's the bootstrap registry and you have on the controller on any node you have the cubelet so the cubelet normally gets its information what to do while the IP server but we don't have an IP server yet and do is just put static manifest into this directory the cubelet will look into this directory and pick them up and just start the bootstrap control plan so then you have a control plan you have an IP server and then you can speak to the IP server to instance the persistent IP server so then the real IP server you want to keep is just an object inside of Kubernetes and then you are deleting those files to Kubernetes cluster and can do rolling updates of all your parts inside of Kubernetes so if you look now we have already the port container so they are still downloading and on the other on the typhoon script we are still waiting for the bootcube process to finish and it will finish when we instantiate it to the persistent control plan so now we have a few more the schedule is already running the IP server is already running so what we can see is that the it's really hard to see in this size the interesting part is like the point where the IP server we see here that the IP server wasn't there then it got in error and then the bootstrapping control plan was ready and the bootcube injected all the Kubernetes object for the persistent control that are all those informations which they are not nicely in here and now you can see that as watching the Kubernetes RP about the HUB schedule you can imagine that it was over there so now we have to start the ports inside of Kubernetes that will take a short time then actually the Kubernetes process is finished so the other end on the typhoon module we are still waiting for the bootcube to finish at all that's still big enough so now the RP servers running and you're waiting for all those ports to come up actually over something it won't work it's one of the two things you still have to do manually so I think it's difficult to start and go right away so but actually it's done bootcube is ready and the typhoon is stopped so you can now export that's one of the parts which bootcube was written and the secret directory is your auth you get that and now you can go on to the Kubernetes RP and now you have almost running a few things are finishing up but right now you have a running Kubernetes cluster it was really easy to do I mean we just started the script and waited for maybe I don't know what was it 10-15 minutes and now you have a running cluster and most of the time you're waiting for pointed ports are downloading from the internet so if you have a local registry for all the ports it will go much much much quicker so now all the ports are running so your Kubernetes cluster would be on all of them a few still but that's the point to get Kubernetes started actually I wanted to show them before right now but the point is I forgot a very crucial part before because it's still too you still have to do it automatically and actually I am forgot it because if you want to install Qobit in the cluster Qobit is just ports instead of Kubernetes so all the ports have to resolve the API server of Qobit and that's not done in the normal in the normal and configuration of Terraform because you would have to change the the secret and there you have the manifest which are Qobit Y SSH you would have to change the I think it's the control manager and you would have to change this information to cluster first because that means that the Q controller is not asking the host resolver to resolve Rp or registry.cobit or rp.cobit it's asking the internal and the sreserver from Kubernetes and you need that to resolve the servers which are using in Qobit so I can start a Qobit and all the ports wouldn't reach the Rp server so I'm not sure what to do you want just to see the web interface but we can't really get the ports up and running I'm really sorry for that or we take and here I will re-deploy it in 15 minutes or maybe 10 minutes the cluster is ready with the right configuration and then I can quickly show the Qobit web interface with all this function all it's proud and it will just take 15 minutes to re-spin it with the right configuration sorry for wasting your time on the first try yes one question I just want to know the Qobit software that you wanted to show us you would want to install it in order to run the Qobit disk cluster or what are the advantages of just getting the Qobit? the idea is to use Qobit because Qobit is a software to find storage systems for the spare disk I have to give them to Qobit and Qobit is that we export in those disks like an NFS file system to online Kubernetes so I have shared file system from a Kubernetes port and they can move around in your data center they can run from one server to another server and they take their volume with it so one machine dies one physical server dies but just re-spin or the Qobit is just re-spin for you on other machines and they are getting their volume the data for your high-speed server or whatever automatically beneath it so that's the idea behind that that you don't have to worry about any incident in your data center and on the other hand it's also a Qobit data store node just dies for a hardware reason whatever on the machine just goes crazy then Qobit will re-deploy or reschedule your data to other Qobit machines or other ports where Qobit is running so your data integrity and replication level should be always met and that's the idea because in our own data center you can't or you don't have some sort of Google Assistant volume or whatever you can buy a big huge NetApp but at one point in the first step you get the same idea that you want to wire NFS but what is the point or what will you do if one NetApp doesn't scale for you so your big queue you need the next one but then you have partitioning you have a few data on this NetApp and a few other data on the other one what would you do if you need the data on both of them or if one NetApp doesn't or it's not able to scale the ION performance you really need so you can't really get the second one because you need one partition for your IONs so in Qobit as we saw before it just scales linearly so you just put more work on it so if you need to double the ION just double your nodes and you will get double of the IONs if you want double of the space just put bigger disk inside of it so you control your own data centers so therefore I think I'm suffering to find network as much flexible and much more agile and as big storage box because it's easier to get a big storage box to put it in there but in the long run you don't know what kind of storage requirement you have in one year, two years or four, five years so the NetApp you have probably an investment period for 36 months or longer and maybe your requirements are changing and you're much more flexible software so that's the idea behind that and it's really a very nice solution to use it to secure your data center now okay Any other questions? Well we also for you should the data decoupled or detached from the hardware is there what kind of a backup system will be constipated or is there like a physical backup you know where is the data kind of stored and not a lot of data centers use LTO tapes to backup the data that's a quite interesting story yeah first you can actually choose first so the complete reliability comes from the fact that we actually store data multiple times for example replication factor 3 so we make sure that they're stored on different racks for example so we can really power off one rack and we will rebuild somewhere else those integration codes in there you can have for a backup story it works like every other storage system if you want to have a backup just put it on some other system or replication and onto another co-write system so this is also working so this particular thing is really it could be also in different data centers in different countries then yeah sure so it depends if you would like to have multiple copies and multiple data centers always in sync there's the speed of light yeah so then everything else is a sort of replication that's possible so I think what we would have seen here is 5 or 6 more commands on command line then we would have complete up and running co-write faster yes I don't know if we want to wait to just go over 4 years and the commands are very very easy so what we would have done is actually creating the co-write name space because all the pods are in a separate name space just for the reason that it's more tidy, more easy to over them you would define a co-write as well as a good where it's free to start but basically you have to execute a few commands like that and those and then for the client you have more two down on the side so it's really really everything is there so you have only to really follow this instruction if you're running a command line you have to have a few pre-requisites like you have a second HD type of machine and a few run and CPU and the resources are valid on the machine but installation is mostly very simple and got even easier with the operator my peers and his colleagues developed in the last I don't know months and then you have this storage defined storage with the software you find storage in your own data center and you can use the same ideas like the big cloud provider so yeah this solution costs money but in my experience it's worth to spend this money because the utilization of the hardware is with 4 byte much higher than with other open source projects like CEP so you never get good performance out of CEP with 4 byte you have quite a bit of more performance so they say officially you're saying 10 times that personally measures 10 times but it's much much much faster so we already know the step we are copying the secrets over and then it will just take a few more minutes it's easy that pods have some sort of cross effects to each other maybe with a device or it will become better I didn't look into it too much I just saw the announcement and read the blog but I personally would think you could do in a corporate environment where you are the platform operation team and you have internal departments using the Kubernetes cluster and different namespaces and you have our rules on top of that I probably think that's okay to use but in a public environment if you're a really untrusted user on that I wouldn't do that but it's not really dependent on forward I wouldn't do that with any software in Kubernetes because Kubernetes is right now not multi-tenant capable so you have a very loose isolation of namespaces but it's not what you have in OpenStack with different tenants this idea is not there when it is right now what's the state? I don't know so maybe with an iso-rider everything would be fancy and you couldn't see files being accessed I hope you will do a thing I was thinking about why don't you do some sort of videos videos about that a little screencast which I should just play if something goes wrong but I think where's the fun of doing it live if you have some sort of videos in your backhand just to show it because your head is working I do it in the lab and everything is fine and later on you start in your own data center and things are not working what was this guy telling me? I have a shift and you've never tried this approach for clusters larger than maybe 10-15 years I want to know what you do in real life for large clusters you told that you work in a large data center you have multiple clients how do they manage large clusters and do they have any workflows that are closer to scientific research when you have multiple nodes that communicate not only to shape data but also to shape computations are there any way out of the box ready solutions for that so actually yes I'm working with ISP but we're not using Kubernetes in production today so we are using Kubernetes in our data center for internal projects so their projects are in the idea of a few thousand of nodes so we have internal backtracking to whatever in those environments but we don't do it as a customer project so you can't work in my company we have a cluster right now so what we are doing is OpenStack right now and OpenStack we are installing wire, puppet and pixie boot so we just put a very small image on the machines and then the puppet action counter is configuring everything so it's a whole different approach of configuration management of data center and of structure management and I also have a not really a follow up question but a separate one there are many tools nowadays for those kinds of automations but they all seem separate and they have all different data eyes I'm wondering if there is at least one tool on the market that allows the app that is actually running on the device to somehow we have shared library or something like that pull information from the environment and find out how many resources are available and do some kind of not runtime type information but runtime resources I don't know polling or metadata information is there anything like that or not? I don't know if I really got your question right you are looking for a tool which is managing the data center like the resources in the data center so you have a few thousand observers and you want to visualize them to a certain level you want that the machine is booting up and then do some sort of service discovery and is getting its purpose like why is it the server here you just have a plan what I mean is that nowadays communication is kind of one direction you have the operator and you have the application that is running on the device and the operator says how much resources he wants to allocate for different tasks different programs, different executables is there a way for executable or for the application for the binary to request from the environment that runs in how many nodes and resources are available I think I got it I think what we want to do is if the binary can request more or less resources if needed so the binary can use more communication that's in a very broad way like the auto-scaling functionality which you have in the public codes and you have an auto-scaling functionality so you mean if the if the processor cores are overheating and there is too much load it automatically expands the thread pool and allows more if the CPU is overheating it's more like how many percentage of CPU utilization you have on the node if your port is running above maybe 60% that your threshold and a second port is spawn then that boosts maybe 30% if they are rising to 60% as well then a third one is spawn if they are going beneath maybe 15 then one port is reduced or deleted so that kind of schedule you have already I think we have had an open stack and in the Kubernetes the auto-scaling functionality but it's not like on a more complex idea like you have too many queries per second on the database and then you are spawning a next database instance like that it's more like a very rough metric and most of the time I think they are using just too pure utilization okay and another question is let's say you have a SuperLive data center with probably 10,000 nodes and it's not homogeneous so you have different devices with different internal specs let's say you have a rack with maybe PCIe and many devices and then you have a rack with SSDs like the normal setup right and you have multiple different binaries with all very different requirements from the hardware and one way is to manually assign the test that are more memory intense storage intense to those faster racks and some other devices to other tests is there a way just internally inside Kubernetes or open stack to make some kind of a formula that automatically maps the task to the device depending on benchmarks or maybe internal specs I don't think there is a tool that starts that automatically what you can do as an application developer in your company is that you can mark your pods with labels or known affinities so you could say this podge should use a storage class which is called FAST and the storage class FAST just has SSDs inside then some sort of a hybrid process would use the storage class back up and then you have a large, slow and hard disk inside of this podge so you could influence the idea what kind of resources are used and also you could label Kubernetes nodes with different labels for CPU or GPU if they are there or different network speed but there an application developer has to mark their pods with the right label to get a schedule on those nodes so it's no magic there so it's really you have to do it but you can influence it but it's not what you want or what you're looking for like a magic tool you're just putting the load inside and there's some sort of tool which just observes the binary and then says oh this is the limit or this is the limit and it just will increase that so that not I'm aware of but we do need so I need my environment so we're creating the name space for it and then you install in the name space the forward config and the operator so you're waiting for a moment to get this pop up and running right now just looking at the what name space so we don't want to see all the other name spaces so the operator is running the operator has custom resource definition which we can use now to actually deploy what we want so we have to use the service config I have to replace at least 64 and the service so what we're doing right now is we are starting the 4x software as a pot and we're starting the web console which is giving us the nice 3D data on we're starting the registry I would say the brain of all of it which coordinates that meta data servers doing some data stuff and the interesting part of the part where we're actually storing the data are the two data pots we have them on the two nodes and what we're going to let on to the client as well so we have a hyperconverge node where the disk and the client are in the same box so you could do the hyperconverge so now it depends how quickly my connection is to get the parts downloaded but it shouldn't take too long well it's going on can you also answer the following question there is the back console running does it itself generate the HTML pages and push it to your device or there is some kind of a special data format that uses to push the data of the statistics and how it runs internally I think what you should answer is a question so it's just the UI for a particular point yeah it just runs on the whole device that runs all the applications it's not just a single problem this cloud is actually the application that's what it's all about which is talking to the registry over in API so there are many many different microservices and they're all talking to each other okay and we're just having to get the back console to configure everything I don't know yet I'm still downloading but you can see that's from the machine to the internet so that's really really many times so one more question and it will probably be finished I'm not sure if I have any words I have one question so this deep point with Typhoon can you use it for autoscaling let's say if you have a pool of free nodes can you autoscale the cluster particularly for the 4 byte if you have spare hardware with MHDs inside with MMA you can just scale them as far as I know right now not the operator could do that but I think that's certainly a point where the operator could be improved in the future to just get devices by itself so if you want to add an additional node to this cluster so you don't have to run the Typhoon to get this yes you have to to get this one and then you have to edit you have to edit the server's configuration and has to put your new node inside of the time which the server should become so it should be a registry node just in this time once and then you will apply this to Kubernetes and then this node gets the hot schedule and that's no automatically like just you're putting just a new node inside of it and magically Kubernetes knows what purpose this node should fulfill so you can put the node inside configure the Typhoon and install Kubernetes as a work node on top of then you apply it back to the cluster and then the 4-byte operator is picking it up and you can use it what I'm actually doing is just forwarding to the machine to get the good so that's the screen you can see if you log into 4-byte for the first time it's a very easy setup step I mean just create an account and a password license key you don't need for 40 days or whatever it was other 40 days 45 yeah so you shouldn't do that in production but for a lab I mean it's really easy to do so you don't have to get in touch with those guys just go on and just have a look see if you like it if you don't like it so and if you like just apply it so that's the web pool I don't want to go to all the stuff it's very well information but the important part right now is that we have the 3 nodes the data data server you have to format those disks to the right and that's data so as soon as you did that and the main task now I'm started so the 4-byte ports are now formatting your spare hard disk in the server and as soon as the formatting process finish they are available to be consumed by 4-byte in the 4-byte application what we do in parallel is to install the clients and the client is the client config which just says like the service config which should be the client there is a hot description for that so in the meantime you see the nodes or the data nodes and metadata nodes are already there in a very short time the node should be there and ready so now the 2 clients already and who gets the right node 2 clients and just install the ports on them so right now we have full functional full functional and full installation so now you can start to consuming the resources so what we need is to install the secrets that are just the base 64 encode credentials I end up into the that console then we have to install the second mistake I already almost did that's also one part which I don't like right now but it's already changed option isn't it you told me so you have to enter the co-wet tenant ID into the storage class because the storage class reference it and right now you still have to go into the co-wet GUI get this ID and put it in there actually there should only be the name my tenant but that's not working right now but it's fixed in the upstream release so now we create the storage class and what we now create is the persistent volume claim so we're doing this volume claim and as Matias told before crowbar is a projects compatible file system so we are doing the persistent volume claim as a read-write menu so it could have different reader and write up to the filesystem so different parts can read and write on the same volume if you do that with block your file system gets destroyed in a second but here you can do it so right now the persistent volume claim is pending and the persistent volume is also pending so that's the persistent volume is there and the claim is also there at spawn so we created a claim in crowbar if you're going back to the volumes refreshing it now we can see actually the volume crowbar does know about the volume right now so that's the volume we just created, we didn't create it in via the crowbar army created by the crowbar needus army and crowbar needus was speaking to the crowbar army to generating this kind of volume and what we now want to do is to start a little application just to see how we could mount it and down here we mount the volumes the application is just a busy box with a sleep command for one hour so just to get a shell to enter it and we start two of those so the two servers are started and now I want you to lock into one of those and I'll take here the first one the 86 or we'll start with the 86 it's 24 it's 24 I was on the wrong part thank you and I start so I go to the directory full byte so we have mounted nothing is inside of it right now so what I want to do is to split the window to increase that and go into the other machine but not so we have the two for the first one it would be the PLN stuff here so we go inside of that we're doing the same directory and it's still empty because we didn't write anything what we want to do is to do a watch one and look at the directory so yeah it's still empty so now I do a touch and hopefully everything is working the files are appearing on the other one so I really mount this device or this file system on both machines and if you look into that you can see that the servers are running on different nodes so I truly have a shared file system behind the machine so if you do that with block storage your file system will be corrupted in seconds so you can mount it on many machines so it's like NFS but much more scalable and that will close my presentation