 Okay. Hello everybody. My name is Walid Shari. I am basically a community lead of two groups the AWS for the eastern province in Saudi Arabia and the blocker meet-ups. I'm I like community, and I like sharing. and this presentation is about openness. Open Shift . Oculus جوسف ، هل تريد أن تقوم بشأنك؟ نعم ، أنا جوسف مايا أنا مانكلاوت آي دكت وايفانجاليست at Rodeon Schwartz in Germany ونحن نستخدم كاتي 3 و 2 سنوات ونحن نستخدم كاتي 4 قليلاً ونحن نحاول نحاول أن نحاول أن نحاول لك قليلاً ونحاول أن نحاول لك حيث أننا نبدأ بالنسبة لك سنبدأ بالنسبة لك لأننا نقوم بشكل مخطئ ونحن نقوم بشكل مخطئ ونحن نأخذ كثيراً ما نتوقع بعد المنسبة نتوقع أن نرى بسعر هذا جميل لننتقل الاشلاء وإذا كنت تساعد في المنصة ونأخذ التفكير أُسعد الخراب وقد نحتاج قابل الخريب على نظام هرار وكنت تبقى حتى أن مفيلة لا تنضر فتاة وغبط وكتاب مخطئ نحن نتوقع على نظام This installation was done on one, two, three master nodes and one, two, three worker nodes. And you can see the specs of these nodes. This is the OKD console view. If you want to look at the vSphere console view, you can see that basically we have these VMs and we have a bootstrap node. It's not needed actually now. We can just power it off. The whole by power of the bootstrap node, the vCenter. OK, good. We used vFsense but it actually delayed us. It caused so many issues so we dropped it. So that's what you should see from the vSphere. You should see the VMs and you should see the IV addresses because they have the VMware tools built in. And one important component of the installation is the load balancer. That's how the load balancer should look like and that's the end points at the end of the day. If it all looks green, that's good. But if you can see that the load balancer already has the bootstrap also but it doesn't matter. And the ingress just needs two nodes. So there is one node which is not participating in the ingress which is OK, fine. So that's what we're going to see at the end. Now, what's the plan? The plan is that we start the installation and then we can talk about it. So we need to go to the console. So everything, we try to do everything me and Joseph from the documentation. So you have to start with OKDIO basically and you go to the documentation. The documentation will point you to a GitHub repo for the OKD, yes. And on the OKD there's guides and under the guides there is user provision infrastructure vSphere Terraform. So this is one of the installation that is when you provision the infrastructure using Terraform. The other installation which is, go to the slide. In any OpenShift installation, OKD installation or OpenShift installation basically the installation ends to be OpenShift install create cluster, that's it, yes. So that's as simple as it is but different environment will have certain different prerequisites so that's what we're going to take care of. So let's go now and see. So I have here if I want to do OpenShift create cluster OpenShift install pressing check especially if you're running OpenShift and OKD check if you have the right OpenShift install you could be running a different version of OpenShift install so the first thing that hit us before is which version that we're using. And I just say create cluster I prefer to give it a directory where it saves its artifacts and in this directory I already have an OpenShift install. This OpenShift install has my environment. So let me not run it and show you my environment how it looks like. Sorry, let's install config.yaml So this is the ML file but you don't need to worry about it because this gets generated by the installer itself. OK, we can generate it. So it asks you things like for example if I am installing on vSphere you need to know what is the vCenter what cluster you have inside vSphere what's the data center what's the data store and if you want automated install what is called infrastructure provision installation you just give it two things the IngressVIP and the ABIVIP and these are floating addresses that will be maintained by Kiba LiveB. Of course you need the administrator you need certain privileges you don't need full administrator privileges and and you need the SSH key so that you can do some troubleshooting I actually run extra step that I am using so just in case the internet or query goes down or there is an issue or there is contention so I am doing also an offline registry. This is this section OK, now This is optional, you don't have to do that but at least in lots of company environments I think you will mirror the images just to have them in place and are not dependent of external registries yes so if I run the installer OK because the file is not there actually I copied the wrong file there it will ask me these questions but it will ask me which public key I need so I don't need to write any ML I just select which key I want to use what platform so basically it's the same for any other platform just vcenter so it's vcsa.lab .local this is a homelab one machine then R620 I use a name and I have this sphere .local and the password if I get this right it will connect to this sphere I didn't get them right so it's the password but it could also use the install config I think and it won't ask you yes but it's good to see the questions if this phase will use the OK it's connecting let me find the password maybe I'll forget the password yeah it's correct OK last time the live demo so this is expected OK so it was the password so it's the same for Azure if you enter the credentials it will connect to the system and provide you with information at this step you can be sure that the connection stands yes yes so in my data center I have three data stores I have one cubecon which is basically empty now it's an SSD disk around 890GB OK so I select this data store and then it will ask me about the network again as Joseph said it gets this information from the vcenter if I look in the vcenter I have these networks so basically I will select this network OKD OK I need the floating IB address and this IB address should be none used floating so this is one of them and I need another IB address for the ingress so this is the other one now this domain is my home domain it's local there is a name cubecon let me OK so I get it from the DNS it's cubecon 2020 EU I will tell you why I need to get it I need what was it again sorry you get it grab ABI from the DNS let's double check 20 EU cubecon 20 EU that's my cluster name pool secret it's also a fake secret in the chat you can take it from there I don't see the chat OK good I can what is this pool secret I will paste it OK I have it here so I have this document always the pool secret just take unless you want red height operators you can put pool secret from try.obanshift.com now this will start the infrastructure provision we have been trying it we have a problem with the virtual IB the ingress but several people on the Slack have success with it so you see now what it's doing it's pulling the image so you don't need to do anything you just sit and relax you just basically fill the information you didn't even touch the ML5 and you sit back and relax and it will pull the image and it will create the infrastructure so it will pull the image and cache it so if you do another one most likely it will not be doing it and that's what happened here it's reusing my old image now it will create infrastructure resources don't get confused it's a image cost in fact it's a message and if you see here it created the folder on the center it created the folder and in this folder there is a template it's powered off because it's a template it's not a VM and from this template it will start cloning what machines are required so because I didn't customize it so by default it will create three masters and then it will create three worker nodes and of course it will start with the bootstrap okay if we go back okay it's still creating the resources so let's go to the slides you could show maybe in vSphere yes how it creates the resources in the vSphere okay it started so it created the bootstrap node which is the most important node to initiate the cluster installation it will be required during the cluster installation it will run the etcd which is the key value data store that keeps information about your cluster and then it will start the three masters okay later on it will start basically the worker nodes because the worker nodes depend on the masters to start and for this the only requirement for this installation the only requirement is just to entries on your DNS okay and this is all in documentation in one hour it cannot really tell you much but in the resources I have basically asking you to join definition or join OpenShift TV they have already created several videos and several blog posts regarding the installation steps and the different options that you have but it should be deleted because we know in our setup it does not work should we show an installation with Terraform because maybe you're in Iraq and get it running yes but I just wanted to show how easy it is this one so it will wait until the bootstrap completes but let me delete it as Joseph said so the deleting is very easy actually you don't need to enter any information you just say destroy the same command and this is good because basically it cleans up after itself so you can see now I don't have the VMs I just have the template and most likely to remove this also okay so this is the easiest way to install OpenShift on vSphere okay the other way as I showed you in the documentation in github okay where is my here using Terraform so Terraform is a very popular infrastructure as code tool and instead of using instead of provisioning the machines yourself doing the template and cloning them and whatever you just enter the information that you require and it will install it for you what is the information? it's exactly the same as we saw before the cluster ID the cluster domain what kind of base domain you have what is the vCenter server here it's called vSphere server but it's actually the vCenter server the vCenter user which has certain permissions and these permissions are defined in the documentation in OPDIO so normal stuff okay now what is not normal it needs the bootstrap ignition control plane ignition so this is basically the OpenShift install program that will give it to you okay let's go and we have a cluster by the way they're running so we need to destroy it also so if I go to the Probe Zero so this is my Terraform directory and I have a cluster running so what I need to do I need to delete it and I delete it this time using Terraform Terraform Destroy Auto-Abrood so it's a you should take care about sometimes to delete also the Terraform TF state file the Terraform source state we had a few moments of trouble with that if you don't delete it and Terraform will have something in the history and sometimes complain about it it's good to know that the OpenShift installer is also based on Terraform so and if you want to customize something you could in theory change the Terraform files they are in the OpenShift installer repository build it and you have an installer exactly matching your infrastructure that's one nice thing about the installer yes so as Joseph said my first initial try was with Terraform and I had several issues because basically the Terraform assumes that you have a cluster and they just have one host so to solve this I added the host to the cluster and it assumes you need resource pools and to have a resource pool you need more than one host in a cluster so what I did I enabled DRS to basically get over this requirement but because I was playing on and off there is a file Terraform.tf state was created and I had other issues and I couldn't solve them so basically Joseph showed me how to do it if you have a Terraform delete the state file and start fresh Terraform is a declarative tool but we have found out that there are some corner cases and you don't need really to learn a lot about Terraform all you need to do now if you go back to vCenter we'll find that basically our cluster is done and this is the other cluster we're using the IBI so now we can start a new cluster okay what we need to remove we have certain files that are hidden I need to remove this it would be good to know what's the purpose of these files because if you don't delete them and you call the installer twice you also will have trouble that you install config change won't be recognized until you delete these files and they are hidden so it's very easy to stumble about above that just delete them in the start and you're fine yes so the log file all the log file from the previous state Terraform tf state is the most important one to delete and the open shift install state because also there are certain things really certificate if I believe that especially if you cross the 24 hours that might cause you installation issues now I have deleted the files and what I need to do if I'm starting from scratch I need to add the context of my environment in terraform.tfr what's this context this context is basically again the cluster name what domain you are going to use what's the cluster domain the vcenter credentials and the vcenter name what data center you're going to use are here the data store is seeing ssd 01 which is okay I have another one and what template you're going to use and this template you need to upload it actually earlier terraform doesn't upload it for you and you need to upload it to a content library so if you go to the fees and if you say content maybe it's easier to go through the menu and if you say where is content libraries yes content libraries and you find basically I have uploaded the template here now one section is that it requires an ignition file so this ignition file you need to create them using the open shift install command we are fixing mac addresses because we need to use the htp so the req was it for user provisioned infrastructure you need to have a certain this is around it which are dns for the host names and dhcp so that basically you can you can you can then you can exactly yes so what we are missing now in this file is the ignition files so how are we going to do this again using the open shift install program so if I say open shift install on okay I was trying to get my history but open shift install create a situation let me I can sorry oh okay basically it will ask me the same question but if I have a main if I have an open shift I can get the other one okay let's do it open shift install create install config sorry I think yes I think so I have one there's a backup one nice one nice side effect is that the installer will delete your install config for some reason so you should take a backup of that before you call the installer let's reduce the compute to 2 just in case and yeah it looks fine and the okay it looks fine the SSH key is okay yes so we can SSH and 2 CDMs okay yes okay so if I have the install.config I just run the installer which is again open shift install create this time I need the ignition files so that my terraform can consume them create ignition configs instead of files make me honest I don't need to specify directly because basically need to read so there was the ignition files here I need to delete them let's delete them let's terraform no yes was worrying me about them yes so let's delete them and start from scratch okay now what's the result the result is the ignition files that were created if you can see so I have the bootstrap ignition file master ignition file and I should have worker ignition file yes now what is this ignition file think about it as Fedora Core OS how to configure itself okay now I need to it's applied to the VM in the first start Fedora Core OS will wait for ignition file and we provide it to Fedora Core through terraform there is a specialty on vSphere because the bootstrap IGN file is rather big I think it has 200 kilobytes and it's not possible to provide a vCenter with such a big ignition file that's why maybe you can show us a short ignition file yes I can there is one trick I do when I copy the file I copy extra characters to make sure that I copy the right file because sometimes you think that you copied the file but you didn't so okay you have to provide a chain of ignition files for the bootstrap can you show that again Valid do you want me to show the ignition file yes and no the terraform TFOS okay let me get the worker file so that they can copy it there yeah okay so as you see as you see the worker and the master ignition files are very short because the bootstrap IGN file and the reason is that in the ignition file there is a source defined maybe you'll see it in the first line and this HTTPS address is at least for the bootstrap ignition maybe Valid can show it in a second yes okay again the same trick I copy extra characters for the right file and here is the bootstrap ignition it's really long okay and it's in base actually it's not in base 64 it's JSON so if you want to look at it use jq. and you can look at it and it has all the information that you need so it has basically what kind of ignition file now Fedora core OS uses version 3 I don't know why it adds another key I guess this is a cluster key but that's my key and this key I'm not sure what it is so it has all the information it needs to configure the Fedora core OS there's a base 64 or there's certificates there are lots of files basically there are lots of files and their content they are written to the of course directories as a storage and they are also defined some services that are started automatically on the first boot but maybe you can show because I tried to explain that since a few minutes maybe you can show the Fedora from TF again yes can you show the ignition there is a bootstrap ignition yes so the bootstrap ignition because it's big yes because the ignition file for the bootstrap node is so big you have to provide it in a user web server for it that web server can serve this ignition file and the bootstrap node will only get this small you see it here on line starting at line 43 this small part tells ignition set the master ignition for the bootstrap will be served from this location and you have to provide either a GitLab a repository or a web server that serves this file and with this way you can provide service for VMs with very small ignition files but the big ignition will be pulled afterwards in the second phase from the web server you don't need a full web server some people just do a Ruby or byte to serve this file but whatever but that's good because for example if Joseph didn't tell me about this I would have forgotten to copy the bootstrap the technician file to the web server and that's one step you need to do and an extra step you need to do is that a carelet so you have and don't carelet from your from your memory just find it and use the same command that is here M-E-R-L to make sure that basically because this is the most important one this is what triggers the whole installation if you cannot get to this file the whole installation will not work basically where's the file it's not copied it time okay we're expecting this okay is it frozen? yeah my machine frozen okay okay let me open another terminal yeah the beauty of Tmuk so I'm using a helper node and this helper node was created by Christine Hernandez let me you know the idea of this because I'm using Tmuk I can attach hopefully okay I can attach I need to reset and find out how to reset okay so okay so the ignition file works so we can start somebody can find out how to reset Tmuk to this size it's the same size but I'm not sure okay so now what can I do I have the ignition files and I have the TF bars really I can start the installation how do I start the installation maybe you should key the screen so we don't get a character mess this one it's I need the reset Tmuk reset I don't remember it it doesn't happen often Tmuk reset window size so I use Google to find out how to do this is there any way to the rate max yes there is Tmuk minus V okay control Tmuk attach minus no control B plus Q which makes flash number sorry there was a command detach if you go up yeah here it is yeah okay but the problem is that I cannot be attached from the other one because it's stuck maybe is it okay close terminal okay I just need to close the other one now can you can you clear the screen valid because the yeah okay so I need so I need terraform apply auto approve yeah you can do plan if you want but that's the quickest way now what terraform is going to do going actually to create the infrastructure for you it's communicating with the V center to get the state it's doing the virtual machine template checking if it's there or not checking the data stores doing everything that OpenShift install does because OpenShift install at the end of the day is using terraform okay so you can see it's creating the machines creating a folder V center and see if this is done V center I go to home I go to VMs and templates okay it created the folder okay D4 and I can see it started creating the machines okay just a matter of time it will start them up normally you have nothing to do with the insights but we will show you a few important steps just in case if you have trouble getting the system running so if the VMs are up I think Wally we should start with the bootstrap VM to age into it but at first we will show in the sphere if something is wrong with the boot procedure because I spent some time on the slides I would like to show some the slides are just for supporting us while we are talking because there is lots of things that you have to wait so the takeaway if you want to ask me what's the takeaway from this whole hour the takeaway is that if I want to install OpenShift or OKD OKD4 it's very simple it's just OpenShift install create cluster what's difficult the difficult part is the if it's a user provision infrastructure the user has to provision this infrastructure how can he provision this infrastructure there is basically the ocb helper so if you go to github and you say ocb helper the ocb helper node this will give you all the services that you require so DNS server, load balancer, what server, dhcb, pixie and whatever method you need and actually if you go at the end it have the UBI automation for vSphere if you click on this one quick start this other repository which is under Red Hat official getRebo you might have do you want to install from over or do you want to install with pixie or do you want to install static IPs are you a restricted network you don't want the cloud provider so all the options you have are there OK let's go back and see OK the nodes has started so now what we can do we can check what's happening with the bootstrap OK the bootstrap came up it has taken the ignition file and you got an IP address the hostname is bootstrap and we should SSH into it to show this to face installation oops error oh no the tags does not have the permissions for the tags OK change but it should it should work without that I think normally the tags is for the for the storage I believe but I think it will work because the beauty of OKD4's installation is that it will start CVMs it gets the ignition files and you are done the installer mostly the OpenShift installer will provide you with some status messages but normally you can you don't need the installer if the resources are created now we will there's a question yes yes this fake yeah I think they are working on at least the sample operator because it's degrading after some time is it 100 8 it is 100 yes we tried it before that's what I know maybe we are there isn't one host yep OK so that's what you usually do because it takes time so basically you try to figure out is everything OK or not the first thing that the bootstrap needs to do is to boot images it's dependent on containers to start so yes it will do an image that's good and it will keep booting how do you can check that you do PSAUX bootman yes I have bootman it's still booting images it's booting into faces before we are thrown out I think it will pull two images in this images there is a release payload it's like cryo VM tools and then it will install them and it will reboot and that's sorry and if it reboots the first phase of the installation of Fedora Core is finished and every node also the masters and the workers do have this two phase installation and I think in a few seconds we will be thrown out here yes and another thing when it pulled the image from Fedora Core or as download the CI server from the download site it pulled the latest image but each OKD release has a specific image to it so if it needs to upgrade or downgrade you will see that basically it will take care of that you don't need to worry about it because sometimes in other different installation you will use the official OKD repository some other instructions will ask you to download like for example this is Greg he's also from the OKD working group he has instruction that you have to download the image and you download just the latest image don't worry about it okay let's check the masters the master will not do anything until the until the install is you could show the masters what they are doing so there is lots of errors but that's fine because the way the control cycle in Kubernetes and on all these systems to get to their desired state so if you look at the logs it's full of error get error because basically the bootstrap is not ready yet so it's trying to go to this IB address and this board which is basically the ABI internal which is the machine config server this is where it will get the master will get the ignition files from yeah maybe you remember we showed we showed before the Terraform vast file and we had a little steps also for the masters and workers and they are all they are pulling their huge ignition file from the load balancer and the load balancer in the beginning only contains the bootstrap node if the bootstrap node has finished with this installation it will serve the ignition file to the masters and at this time you won't see this message again because the master will load the ignition files configure itself and start with the two phase configuration maybe we can show we should have thrown out from the bootstrap node let's see not yet okay just in time and it's very fast to connect it I love it because basically the two things I love about OCD and OpenShift the images they use for host the Fedora Core OS or Red Hat Core OS they are very minimal and they are very easy to maintain and you are not supposed to configure them except using the ignition file and except using basically a desired state you are not supposed to SSH into them now if you look if you look here this is a different version than the one here this is this is July July 26 this is June 29 so basically it did downgrade itself so that it can fit to the requirements of this release version of OCD again okay so we are getting more images I think we will get over 1216 images correct yeah I think so yeah and it's good to know that in OCD and OpenShift 4 the operating system is an implementation detail you don't have to patch host for security packages but OpenShift and OCD will take care about that if you upgrade your cluster you will get a new operating system it's automatically installed during the update and a rolling up upgrade on each node of your cluster so you have you must not take care about OS upgrades on your own so if you look at the journal you can see that I uploaded the release image and now it's starting the bootstrapping usually the bootstrapping takes 10 to 15 sometimes 20 minutes hopefully not much because basically the images I have them close by and if you see these characters this is basically Arabic characters in Linux but they are really messed up so don't use Arabic in Linux so what do you think about showing OCD get POTS because I think we could see something already yeah I tried so I have the three images so what did it do what did it do with these images so if you check Portman or not yet Portman oh no try control sorry yes try CTL there is a T too much yes it's a live demo expect me to do machine not yet you can do BES but I don't think there is something yet change it takes a little bit it's rather slow for other images you see the log okay starting at CD that's the most important thing because what the bootstrap will do it will be its on key value data store for the whole cluster and it will try to provision another cluster so the moment at CD is up and running then it will try to scale out to the other nodes the master nodes and the moment it scales out to the master nodes they will start the machine config server and the bootstrap node job has finished you can kill it then still in the same images so now because it's starting the CD certificate most likely we will find we should see we should see I think it will pull okay so how to start how to start as I said you go to WCDIO and you get basically you get the instructions from there or from this side okay now one of the things it will tell you if you go to the release page one of the things it will tell you it will ask you to run this command OSEADM so you don't have OSEADM you go to the GitHub of OCD and get it from the releases pages so you see this is the release and if you go down this is all the images that are contained on the release you'll find basically which this is the client which is open shift client so it can be actually you can submit a bug report that should be OCD client doesn't matter yeah the moment you have this client from the release page as you saw you run this command it will pull the open shift install and OSEADM and QCTRL and then you are in your way yes now this is the menu means you haven't seen it in the beginning so what kind of you don't need to create any admin it will create it for you now if I want to do the caching I just set up certain variables and they use the OSEADM release mirror the instruction is for this connected environment but you don't need really need for this connected environment even if you need to make sure that you are not dependent on the upstream just in case you can mirror them into your okay so the installation we said there are two types there is the full stack automation which is what we call IPI with infrastructure provision install and the UBI which is not fully automated and you can see here the fully automated it will build your network it will create the load balancers it will configure DNS it will provision the VMs for you it will do the operating system install it will generate the ignition config it will generate the ignition config as you saw that we had to do it for Terraform but even in Terraform the installer can take care of that open shift open system support for the OCD it's usually Fedora Core OS I think there are talks about centers correct? yes now the other thing about why do you want the automation is that you can do the old scaling out of the box if it's usable vision you need to make sure that your cloud provider in this case the vSphere has the capability to do this and I believe in 4.5 it has the capability to do this that's true so this will be fully automated yes let's go back because I just want to take the chance to look inside if we have progress okay now maybe you can show OC get pots outside of the bootstrap node because now we are in the bootstrap node and we could do a watch maybe so one of the things the artifacts that it creates it creates this odd directory and it has the cube config which is basically the configuration you need to connect to your cluster so if I say export cube config auth and the cube config I can see OC getBots minus A if I don't have yet because it's just a CD it's still starting I'd like to do a watch on that too yes so another thing I can do using the old shift install I say wait for I want to check the status for the bootstrapping is it finished or not I can say bootstrap complete yeah wait for bootstrap boot I type correctly so it doesn't tell me much so I increase the verbosity by increasing the log11 again it doesn't tell me much so what I can do I can actually trail the open shift log file there's a log file it will tell me what stage it is on so it can tell me that the waiting for commit ABI to come up so unless the ABI server is up I cannot use OC because OC is basically a client for this Kubernetes ABI everything is restful everything talks to the ABI and if the ABI is not up on the bootstrap I will not be able to talk to it okay so as we saw the installer is very simple it's text user interface it takes the cloud credentials the vSphere for example the vCenter the administrator what data store do you want to use what network you want to use and bootstrap creates a cluster control plane creates the bootstrap the bootstrap basically creates the master nodes so that's the diagram if you want to see so from one machine the installer it will create the bootstrap node and then the masters will depend on the bootstrap node to get their ignition files into basically provision themselves into the desired state of the cluster and then the worker nodes will talk to the master nodes for the applications ingress basically usually gets installed on the worker nodes we'll share the slides, these slides are from DevNation to give credit to DevNation and Barcetar and Eric these are actually the slides from the master class for the open shift operators so it tells you basically how you start and stuff like this okay I don't think we will have time to do the auto scaler so let's see it's strange because it's a pulling it's rather slow so maybe you could you could do a watch on OC get pots or namespaces so maybe you see a little bit more now normally you should yeah you would see a list of pending pots this is already good because it's talking to the API server already you know the API server is not up to us because it's an apatite response does it let's check let's go to yes this message comes short before you see a lot of pending pots I am again on the bootstrap node to double check what's happening okay I have more images but they are not really enough I want to check if these images are actually resulting in container runs I don't try control bs okay that's better so what I have I have the hcd I have the hcd metrics I have the hcd member and I have the machine config server that's really good because the machine config server is the one that is responsible how do I check again I can also check from the load balancer from the load balancer if it looks again but that's not true the bootstrap is starting to serve the equation file to the masters this means that the masters the masters should be able to pull them and start over yes so maybe you remember that before the masters were were pulling for an ignition file and now the bootstrap node can serve this ignition file this machine server is responsible for that and the master pulled it configured it and now we should but now we should see something with osiget it's a third try here it says it's still waiting for the Kubernetes ABI I believe the Kubernetes ABI should be ready but it's not ready because you're posting it what might I say is not actually given I am I am switching over normally everything is fixing itself it's magic I can reboot the nodes during the installation and don't worry about them they will get to the same state also so they are in a loop to get to the desired state normally you can't you can't break anything because the system will fix itself so we had we were often in a situation where we thought we have broken something but it's rather easy to fix it because okd is our open shift 4 is based on operators operators are first class citizens these operators will take care about their their pods they are responsible for and fix itself yes if you have a problem you can also go to Slack and open shift dev in the Kubernetes Slack is basically where you can ask your questions when you have problems or open shift users these are two nice channels if you have open shift or okd I believe okd is mostly open shift dev correct Josie yes very awesome most time yes I think give me a first give me a first try I think now we should see the pending pods I have a strong fee ok yeah because pooling is slow normally it should not take so long you will see a list of pending pods and then the cluster the containers will creating the masters will configure itself themselves ok I mean performance is ok for the bootstrap the bootstrap specs is really low for cpu 8 gigabyte but it's doing good the master let's see actually the registry that's the one maybe that is I guess my registry is the one that is blocking this using mirroring yes this one maybe is the one that is causing us the headache not really it's the not really it may just take a little bit of time yeah give it a little bit of time I want to see the pending pods everybody there as well I want to see the okd dashboard is my proof of life that's why that's why I showed it in the beginning just in case normally an installation takes 30 minutes if you pull from an internal registry it takes about 30 minutes so we were talking a little bit about the insights because I think it's more interesting to see the insights as to have a dashboard because you should see what normally happens if you install it most normally so this is my home lab it's an old Dell server like 5-6 years old it has 40 magical cores and basically memory is good enough around 320 I killed everything just to basically make sure that we don't have any contentions or anything I didn't shut down I killed actually everything but I mean at work if I do this at work Joseph said maybe 15-30 minutes max if you have good SSD if you have basically a powerful server this is really an old server and you can see the hertz it's really 1.7 GHz it's very very it's good for a home lab I love it so let's go back to the boards but we can let it run because we have the next platform we will install it on Azure and we can let do the installation what it wants to do and look at it in the end but could you do have you seen some pending parts or last try? not yet so basically if you look at the let's see the logs I hope the tagging I don't know why it's taking some time for a couple of the other demos so don't panic we're not panicking we are live you're live if it can't go wrong it will yes and if it takes too much time it must be live actually I was worried that I'm not to be fresh I wasn't really fresh this morning I wanted to do the automated install because I know it's doable I know people have done it but there is something in my environment that is blocking me from doing it so I couldn't figure it out yes so the automated install for vSphere very straightforward you just need two entries one for the API and one for the Ingress and they are floating and they are and this IB addresses will travel with the nodes so basically you don't need to provision a load balancer and you have to have DHCP because it will talk to the vCenter to create the VMs so it's really very nice I will get it done okay so what are the issues that we had as Joseph said the Terraform TF state so I like this one I like to state there is so many one of the problems with OpenShift and OKDE is that so much information so much videos are there outside but if you follow the official Red Hat and the OpenCommons OpenShift Commons I think you should be safe and the Slack of course so and what is the difference between OpenShift and OKDE when it comes to installation first of all the release format if it's OpenShift it would be 4.50 or 4.51 but OKDE you will see this date if I go back to the index and you see the different releases you see basically they are really a little bit different but for documentation it's OKDE.io if you go to OpenShift there might be some discrepancies on how things are done so it's better to stay OKDE.io so this is the table I like also that you can as he said changeLock if you click on the version you will see what changed in each version and also the commitment I like this that leads to this entry and actually you can see what can you upgrade from and what we can upgrade to you cannot assume that you can upgrade from one release to the other release straight away you have to check basically there is another GitHub from a Red Hat Engineer who created the like chart that basically draws all these upgrade ads to you you can see what Kubernetes release version is using for OS OpenShift 4 is viewer Kubernetes with the extensions using operators so if you ask me why should I use OKDE first of all it's an open source platform I can use it in hybrid cloud as far as I know it's the only hybrid cloud open source platform that is out there which is holistic it will give me logging identity providers it comes really maybe this is an advantage and disadvantages if you need something quick maybe you can use K3D or K3S but then you have to do lots of stuff to get it up and running in a good shape when you see this this is not really a tag actually when you see this one if I go to this one for example and I go to Quay I found out that if I need to do really OKDE and OpenShift you don't use these tags you use the digest so you have to do this trick to get to the digest you have to use the image and then try fetch tag when you fetch the tag you fetch the digest you fetch the digest because everything in OpenShift and OKDE is digest it takes our life a little bit harder but it's much more immutable much more secure much more good practice for configuration management let's go back to Joseph Woods yes can you do an OC get one that's what yes on the first try yes almost yeah oh we have already one running container one running POT or yeah here it is running the network operators always the first network is installing on the masters and yeah from now on things will go their way so OpenShift is an operator responsible for several networking functionality and OVN is the actually okay I need to show you this so another default configuration between OpenShift and OKDE OpenShift will use the SDN is OpenShift SDN OKDE uses the Kubernetes OVN which makes it easier if you are going to run Windows container later on so one requirement for Windows container is to have the SDN as Kubernetes OVN which comes by default network configuration option for OKDE I think it's okay Diane if we explain that because on Azure the Azure installation will have the same POT if I go here and if I do BS okay so things are running I have the Kube ABI okay so the ABI I think is done yeah the ABI is done so I think the bootstrap completed but I could not find the requested resource it's okay let it do its job it will work so maybe in 10 minutes the master as the workers will join the cluster and we have to approve the CSR but maybe we can maybe we can in the meanwhile start the Azure installation so maybe we could be on a UI afterwards and then you could explain the operators again afterwards so so Azure installations okay so if you install OKDE on vSphere on GCP on Google Cloud the Fedora Core Image is available there or you can download it for vSphere or the installer will download it in Azure it's a special case because the moment Azure was having Core OS as an official image it doesn't have Fedora Core OS image yet so the installer has in the marketplace so the installer has a hack and if I go back to the releases so Joseph wrote a workaround and this workaround was available until beta5 but after beta5 this workaround by mistake got out and we have a bull request for it to bring it in so there is yes there is a bull request to bring it in it was merged today it was merged so we can use the do you want to try the new install I don't know if there is a nightly but I think yes there was a little problem the procedure because the Fedora Core image is not available the marketplace is the following the open shift installer downloads the Fedora Core image from the Fedora Core site extracts it decompresses it and uploads it again to Azure the problem with that is if you have a slow internet connection like me and your upload is slow then teraform will time out so our procedure the only workaround at this moment is to use the open shift installer not locally on your PC in an Azure Cloud Shell because if you are already on Azure things are much faster yes so the open shift installer is open source for both open shift and OKD the OKD is basically you have to go to the branch of course and that's where basically you find the open shift installer and if you want to participate or help on the open shift installer for the different platforms so Azure as I said the main thing is the open shift installer so I already downloaded this beforehand and the one I downloaded as I said the first thing to do is always check the installer so the version to make sure that you are not using something that you don't intend to use because this happens the moment you go into different version or testing different releases you might get messed up unless you have a system in place so it tells me that this is from commit because as I said Joseph basically made the bull request I made the bull request after him because I thought my bull request was better but it was the same so I closed mine so basically this was from a bull request it was not committed yet but as Joseph said it's committed today to save us time I will use this because I already know that this one works and you can see this you can see this yes now what I have already cluster up as usual basically here so I have one cluster up maybe I need to log in maybe this is a PCI I think no no it is local yes this is PCI local yes here so this is a developer screen in OkD and this is the default open shift monitoring so you can see the bots and they are not related they are not in they can see them and you can scale them and whatever if I go to the administrator I will see the status that's not good but this is Vsphere again again no oh yes okay that's why it's not good you see the confusion when you have more than one cluster okay so how do we get how do I know which cluster is up I can as usual I have the kubeconfig file so I export kubeconfig install directory hot and the kubeconfig and I ask for the routes what routes are there and actually I'm looking for a specific route which is the console I know what it is but let's check I will not get it this way because I need to ask in all names places in all projects but this is a pre-installed cluster isn't it but this is a pre-installed cluster just in case that we had issues yes so I can go here for example I can delete this so this is a pre-installed Azure OkD cluster on Azure that you leave us that it's working in print in Siri I don't have a storage configure but I have managed premium this is the differences now so in VMware you will have the vSphere built in driver by default which will provision a storage class called thin if you need the CSI you need to do it as an after step in Azure it will use basically Azure storage so this is using Azure storage and you can create different tiers you can create different storage classes for this so let's kill this cluster I have to actually and this time I will use actually the Azure Shell as Joseph explained because of the uploading and downloading of the image this could be faster doing it from an Azure Shell Azure Shell times out every 20 minutes so if you want to get rid of this time out use Tmuk and you do watch whatever so basically open Tmuk and just run a water or you press enter from time to time but this is not fun or you ask kids to come around okay okay so I'm in this directory why do I want to kill because basically the quarter reasons actually let me kill the other one the life because of the quotas so we have enough I have enough now but just increase okay let's try cluster let's directly no but this is how you destroy a cluster so if there is anything you learned today it's how to destroy a cluster we'll do a this at work Oolert Oolert takes a little bit usually fast and if I go to the console okay I am in the console actually if I go into the console and I check VMs for example I will see some VMs starting to disappear or starting to go into a known state so I have two clusters one is running in North Europe Switzerland and one is running in East U.S the one I'm killing I think the one I'm killing is the East in the North Europe one I was expecting to be contention last red had someone that was contention in AWS in U.S but today there was contention to be honest with you it's either Azure is really good or I was lucky today to provision these clusters so the quota system works because they didn't give me my requirement is very small 80 calls maximum but it's like took a week to get me this quota okay let's go back to the command line so while this is destroying we can actually create another cluster to be careful when it comes to Azure when it comes to your cloud because it's not your machine so you have you are limited on the size okay so slash home which is non-resistant disk is quite good but it's non-resistant anything you put there it will get lost if you disconnect the terminal and slash home slash your user so my image is is around 8G if I look I have I think I know I don't have okay actually this is so the trick is and it's available in the Get Rebel of the OCD is to make directory slash home slash cache you create the cache with the non-resistant one okay and you link to it from your persistent one this won't be necessary still will be finally will be available in the marketplace Azure and Red Hat are working on that so we hope that in a few yeah in a few reasonable time it will be available and you can call this command from your local PC I love to create directory so that I can limit my the evolution of my artifacts for the installations so I created directory and I love to have debug level high as I said the key takeaway how to install OpenShift is OpenShift install create in no matter is it vSphere is it GCD is it Azure okay press enter I am missing something create a cluster so what do I want to create I can create a cluster I can create a manifest in this time I am using the automated install so I want to create a cluster asking me which platform in this time it's Azure which region you have to be careful with this because there is a quota and the default quota is 20 CPUs this is not enough you need at least 28 unless you customize unless you customize your installation which one do they have I need north Europe this is what I have quota already agreed by Microsoft and now this is a domain this is another requirement domain in Azure in your own account okay it's cluster name what do you want to name the cluster let's name it yourself okay the book secret okay the trick I do is I have this in the dashboard so basically I cannot write Jason I am not very good in Jason it's fake part that's all in Jason but I have to put it in the dashboard okay so it goes it fetches master machines it basically checks with Azure do I have the credentials the credentials was done earlier and I'm not good in video editing like the YAM so I didn't record it so basically you need a certain service account with certain privileges it's all documented it's very straightforward it needs lots of permissions I don't know why but I don't I think it's very good to have a service principle with so much permissions but it's required yes so now that's the step that Joseph told you about it's trying to obtain the Fidora Core OS forget the message obtaining Red Hat Core OS it should say Core OS and that's it maybe I should do the request and then it copies it into the cache file that we created and it it starts decompressing it if I want to check now we saw how we check the infrastructure one the on premise if I want to check I just do refresh here and I should see North Europe resource group okay nothing yet it's very slow the console slow understand the Azure console I don't see North Europe yet okay now we see resource group resource group is a concept in Azure where basically it's a container that contains all your artifacts your virtual machine, your storage your network, your security groups so we see network security group we see disk be careful that this is the one that are costly if you are running open shift select the right disk if you want to customize these are the ones that really cost and you can add GB if you want but this is another time so it started creating the virtual machines it started creating artifacts I have two disks and I have two machines I have the masters where is the storage where is the storage account okay and where is the where is the storage account good question but it should be yeah it's a virtual machine it's already running there should be an image and a storage account let's see again let's refresh sometimes it's just a concept okay or we can straight go to storage accounts okay and we should see which are we don't see the North Europe yet strange because it's still the compressing let's see what's happening on the shell still the compressing the image it's yeah so basically that's good by the way because it's creating the machines yeah while just creating the VM it's not really installing them it's creating all the artifacts that it needs before actually the image is ready which is good optimization but I think I think I think the machines should be created after the image was uploaded are you in the right or was it an old one maybe it was an old one what I selected North Europe correct yes yeah so it should be and you cannot see it here because it's not there yet let's do again because I think in the installer code the Terraform starts after the decompression so Terraform shouldn't run it will take a little bit until the decompression has finished maybe it was the old one so this is North Europe oh yes so this is the one that is deleting yeah I think it's not I wish there is a way to say how this resource because that's the reason you're right yes you're right can we have a look if it's still decompressing yes yes it's still decompressing the download was faster than the decompression and the deleting is this low okay so I jinxed the Azure I said Azure was good today can we peek over to vSphere yes I hope we don't because there was lots of confusion for that but maybe it helps I'll fix it in the video editing how's that okay it's still downloading take a while it's okay yeah but that's too long for a boot scan it's downloading the timeout is 40 minutes so I guess it didn't timeout yet but even if it times out you can actually it's still going on okay so we have bootstrap not finished on vSphere and the decompressing of an image and this is my fault I should have done it earlier if it was on the cache and that's the difference now you saw on vSphere it was on the cache so it started straight away but here it's not on the cache so basically it's taking it's time to get the image and then decompress it and decompressing and uploading so we should see a new resource group if we do this yes so if you go to the resource groups we should see a new north Europe resource group it takes a little bit because the web UI is always even if the resources are created on Azure the web UI takes a little bit okay let's see on the console on the console it's telling me that basically Terraform has started it downloaded the Azure provider and it's initializing the modules that are in the Terraform called manifest so there is a bad bootstrap, there is a DNS there is a master, there is a vNet these are all resources the bootstrap is an OpenShift resource that Terraform will use to provision to provision basically the OPD and that's why that's what happened here when I said the Joseph code got missing here because the Terraform provider was changed here and it requires certain changes on the code and by mistake I guess it got dropped so there was an upgrade to the Azure provider that's why a lot of a lot of people had reported problems with error messages from the installer telling them that some page blob block blob problem occurred yes and we fixed that it was only a small change that was necessary and it was merged today and I think in the next release it will be it will be in next release will come out the week after next week so you will have Azure support in an official way as you see so one type of storage containers in Azure is base or block block are aligned to 512 and that was the error message the other issue that was dropped was the source so the source is a cache image so this was the other change so it's very easy to participate on the fixing things especially if you have Joseph backing you up as a mentor it's really amazing so that's the issue I have a strong feeling but let's see the compression has ended and that we should see a resource group yes so we are seeing resources we see Dnets we see Azure private DNS so it's recreating the subdomain let's go to the portal and see where I am in the portal okay so let's refresh okay we see in you Joseph Joseph here so if I refresh here also because it's also these are interface of Azure you need to refresh several times I think it will create like 22 records at the end now what do we have we have a load balancer which is expensive and we have basically the public IP address in the virtual network we don't have the VMs yet the first thing that is crucial is maybe you can show the storage account it should already be created it's still it's creating it let's see it so I love basically the cloud shell on Google and Azure it's really nice very very convenient if I go to the storage accounts yes okay this way also works yes okay and I refresh I should see north Europe yes here is Joseph north Europe and if I look the first entry it's the first entry you can take it is it just the first entry yes I did the storage account so why did it okay you press okay so this is the storage account for the resource group that created recently okay and I have different types of files we are in containers and basically there's one for the bootstrap Ignition file okay it's up it's uploaded and the other one is the VHD so this is the initial phase of the Azure installer if you have these two you are in good shape and you saw it's 8GB that's why you have to be careful when you are installing several releases because you could run out of space okay let's go to the resource group again it was creating an image so the VMs should also be created soon maybe we have one if you scroll down yes by type next few seconds let's refresh this is the take away for Azure refresh it's a nice thing about the Azure IPI installer is that you have to take care about nothing you have to prepare the service principle sometimes it's documented very good it can have a service principle in a few minutes and then you only call the installer create cluster Azure enter your credentials and here you are it takes around it's a bit confusing because now if I want to check do I have the right service principle it's not really service principle if you search for service principle most likely you will not find it it's in the you go to the application registration and then you say view all applications this directory and that's the service principle I created and actually it created others it created it created something for Ingress and I think that's the reason why the initial service principle has so much so high permissions that it can create other service principles on its own so I guess this information is safe because there is a password related to the service principles and there is permissions and this is what you need to check you need to check that you have the directory graph and I'm not sure if this is because it worked for me before but the last installation somehow it's not granted for me but it will tell you here there will be a warning and then you just follow what the warning tells you in my company only a special group of people is allowed to grant so I personally cannot grant service principles on my own and that's why this is a true face procedure so in VMware there is something called Vmug advantage have one year evaluation license for almost all products of VMware in Microsoft there is the developer basically subscriptions which will give you like 150 US dollars per month access to Azure and it will give you in the beginning if you don't have an Azure account it will give you 200 free account and later on you get like a quote 150 US dollars per month and you cannot use this in production this is basically development test work but it's good enough it's really excellent so this is what we call developer subscription or something like that it's still creating the blob images for the machines now if we look somewhere we can actually let's see let's see if it created the virtual machines or not yes so we have really this is OKD this is not the current installation sorry we don't have first upload it and creating a VM it's always confusing if you have lots of resource groups in Azure to mix things up yes now if you ask me do you want automated install or do you want to use a provision to install it depends if you are in company that is heavily regulated and you cannot change the host names the current automated install for a reason it will create a hash so you cannot know the names in advance unless you change your process so do you want to be on control or do you want the installer to be in control so if you can answer this question then you can decide if you want it automated or user provision so user provision has advantages and automated has advantages it's not like I should always go with automated everyone has an advantage OK if you want more slides the other thing that I like about OpenShift the concept of operators OKD or OpenShift basically are operator so if it's OpenShift you can call it operator container platform if it's OKD you can call it operator Kubernetes I don't know so what are operators think of them as that you have Joseph and you have Kristian Hernandez and Chris Short and Diane and everybody from Red Hat in one room and anything you can ask about OpenShift or OKD they will answer it for you so basically you are putting all of the operations or the developers in one package that's what it is it's basically application lifecycle management capitalizing on all the knowledge that human operators have so they are just regular bots and there is a controller so it's a controller and a custom resource and everything you configure using the custom resource definition so basically even how to manage your Kubernetes environment becomes much easier yes so this is how an operator look like this is custom resource definition this is just a hypothetical example if I have something called cars this is how I will define it and it becomes the first class citizen in Kubernetes so what kind of first class citizens do we have go ahead Joseph in OKD3 we had also API extensions but I don't think that they did it with operators I think they were extending the API server itself but with custom resources you actually can extend the API of Kubernetes with own resources and this is a mechanism that's not propriety for you know what I mean for OpenShift but it's plain Kubernetes and if you write the operators there is an operator SDK we also use it in my company it's very easy to write operators on your own if you have written operator with that and this is source code you will immediately see how they work because there is always yeah maybe you can show one actually Chris Short Chris Short did like two hours workshop on how to write operators using an idealistic person an idealistic person on I mean he will say oh no this is the wrong use of Ansible is not a programming language but for somebody like me from operations it's the easiest way to get operators running up and running on on OpenShift.org you can use go there's also a great operator on talks by Michael Revnek as well that you can find and there's an amazing group of people in the Ansible world doing work with operators so yeah tons of stuff I just want to give you a heads up at the top of the hour we are going to cut off the live stream so if you want to and it will continue with the OpenStack stuff and if you can take more time if we need to but just let you know yeah are flexible maybe we should have a look on the VMs yes they are creating I have the masters creating okay so the bootstrap okay the image is uploaded the VMs start nice and as Diane said there's lots of resources for operators one of them is actually learn.openshift.com it's very easy you don't need to provision your cluster you can use one cluster for one hour and there's lots of small tasks that you can do within one hour one main task is the operator framework you can understand the ABI you can understand what's the ATCD operator you can use it for the application it's not for the infrastructure and you can use the operator SDK and the Ansible Kubernetes which basically Chris Shaw did a long workshop covering it and there's an Ansible Refresher even if you want to do the Ansible Kubernetes module I think with the operator SDK you get a CLI where you provide a few very few parameters and it will create an operator for you you can immediately compile it and do a few things and you have only to fill I think two files it's a controller in the the API where you can specify fields for your new Kubernetes resources and it's lots of fun to write operators especially if you for example if you want to create a database like a MongoDB operators that take care about that or if you want to create VMs you can use operators that create VMs on Azure we wrote an operator my colleague wrote one that creates an VMs on Azure with the help of operators in Kubernetes yes so this cluster is running in Azure this was created early this morning and if you look at the selection you have you have AI machine learning you have the open data hub operator yeah it's a community operator so basically you don't need to subscribe and it's very easy to install you just click install hopefully it will work this is a live demo okay so basically status unknown it will try to get images and so that's how easy you install the machine learning deep learning environment it basically clicks succeeded and now I have a complete machine learning environment using operators so what does it have which is the main interface for data scientists, data engineers to write machine learning models Apache Spark for data management Prometheus for monitoring and Grafana Airflow for workflows so if you are not using and there is seldom also for workflows Argo for basically GitOps and Kafka so basically I installed all of these without any knowledge in any one of these in less than few seconds that's how easy it is now you say okay you installed it but you still need to configure it yeah no problem you go here to open data hub and there is if you go back to the open shift comments videos there are several videos related to how I installed open data hub and from definition also there is some stuff related to this because I saw because I saw I was using a GP provides a GPU support I installed NVIDIA GPU operator a few weeks ago also on OKD it works absolutely smooth and yeah it's a lot of fun and you can play with your creativity to invent new things because you don't have to care much about how things are installed you can use operators and it makes life more easier for you yes so it's creating so today because maybe the other guys from the KubeCon are also creating things yeah can we send them a message in Slack please talk for 5 minutes talk for 5 minutes but I guess they are botting virtually now I mean everybody is in his own way but if you are if you see that it magically goes away and in the end you have your UI you can look in and have fun but what's also nice with Azure is that you can immediately use the auto-scaler I tried it on my own it's very easy to do that on Azure and I think it's a very important feature to save money that you enable the auto-scaler it's on I think thank you for reminding me because I have a problem because the one problem I have is basically I didn't configure the storage and the open shift samples which are commercial subscription to Red Hat I don't think they are commercial but they are basically you need to use the boom secret from Red Hat but if I go to the administration cluster setting I can see there is an update ready for me so this is the second cluster that I installed now how to update and just press this button update to another version and then you can select yes I want to go so now while we are installing one cluster we are updating another cluster and it's and it's your workload is still running so because the upgrade it's a rolling upgrade it will install all the operators you can see this that in this menu here you will see that after a few seconds the versions of the operators change and in the end always the last steps is always that all the nodes get restarted and this is a rolling upgrade is always one master and one worker that gets reported and your workload if it's cloud native will run and will be not interrupted by the upgrade procedure that's very nice okay good earlier there was in maintenance status updated now it's gone I haven't seen it before so I guess this is it means that when the node was upgraded or something when the node was reported anyway the upgrade will not continue because it thinks that it's not safe so the cluster version sometimes we'll have this sometimes it will say it will not be enabled but then and I guess because I have one operator upgraded if I get rid of this one most likely this will be very smooth but even with this one degraded most likely it will work if you if you put that operator into a removed state that will fix it okay that's okay but this is a cluster operator how can I do it how can I do it how can I okay okay I see 1051 Joseph where is it no I don't see it it's coming you can see my screen so you can tell me yeah I'm gonna drop it in the chat okay are you looking for a full secret no no it's basically how to disable the sampler operator I think it's handy to disable it we should disable it by default unless somebody wants it من now but can you is it possible to update if you remove it okay thanks yeah really remove state that that should do it for you okay Nice okay let's try i go to the console i go to the cloud shell i open a new shell create unpackOK now they are getting in completed state فتحوشًا، حسناً، أعمل على العالم، حقًا؟ لذا أحتاج إلى أعمل على هذا الآن لكن يجب أن تستطيع إستخدام مختلفة مع كوب كونفج يجب أن تكون لدينا أن نظر على أجل موافق في سفر لأن فقط أخبركم أني بحاجة إلى المستخدم، حسناً، أنا بحاجة إلى المستخدم أستطيع أن أستطيع التعليم فهي فرق الشمس فرق الأموراتر. هذا ما أحبه عن الأموراتر. أنت تستطيع التعليق الأموراتر. وكذلك. لا نفكر هذه. لا نفعل أي دموز. لا نفعل أي مقارات. كنت مستعدة على تقريبها. سنرى now in the console. أرى. ثم سنذهب إلى هذه العقل. حسنا. نعم. أفضل جيدًا. شكراً لك. إضافة. دعنا نرى إضافة. لذلك سننتظر قليلاً. ونرى إضافة سنذهب. حسناً، دعنا نذهب إلى إضافة. إضافة. هناك شيء أفضل مع إضافة. أعتقد أنها تأخذت كثيرًا. هل يمكنك أن ترى إضافة إضافة؟ إذا لديك إضافة إضافة؟ نعم. إضافة إضافة. يجب أن تقوم بإضافة إضافة. إنها كل إضافة، إذا كانت مستقبلية. لذا لا ننتظر. إضافة. إضافة. لقد تأخذت كثيرًا. ربما كانت تأخذت كثيرًا. هل تأخذت كثيرًا من الإضافة؟ أعتقد أنها كانت لديك إضافة. لأنه يتوقف لن توقف. لأننا كانت لديك إضافة. لكن الآن يبدأ تأخذ إضافة إضافة إضافة إضافة. يعني أننا يجب أن نكون... إذا كانت إضافة إضافة. نعم. والآن تأخذت إضافة إضافة إضافة. نعم. أعتقد أننا يمكننا أن نضافة إلى... أريد أن نضافة. yes, I want to say that it's really nice that on every platform the installation process is almost the same. it's even possible to use platforms that are not officially supported like they I installed OKD-4 on FOXMOCS by selecting a platform önق. وصفت أكثر من فائز جميلة للتشاهدات. بحث أن المجمع تأتي على مجمع الطريقة. أعطونا أن نفعل هذا بشكل أسئل. فأجل أن نساعد الذي يريد أن نتكبر بشكل أسئل. أعطونا الاسلام بشكل أسئل من فائز جميع. وقعوا البلدية عن تلك الثورة. وصفتها بشكل أسئل. animation.ru where you have less control about what happens but yeah, you have to not take care about it easier Yes, the bootstrapping in Azure is done. So that's good. So we can kill the bootstrap but we don't need to. Why? Because basically the committed installed. I think they kill it. It's an installer with the controller normally. Yes, so now it's basically the master's are in control. ما ؟ نحن لدينا المقلسة لذا لا يجب أن ينظر على هذا ماذا ؟ نعم ، هذا شيء مجددا لقد رأيته لذا أحتاج إلى أن أرى هل أنت حقاً حقاً نعم الأسئلة الأحيانة مع المقلسة بسيطة هذا هو أبعي لا لا ، هذا هو أبعي لا ، بببضي هذا هو الأخير ، أسرع بذلك بسيطة هل يجب أن يؤدي المقلسة المقلسة ، ندينا أسئلة مع المقلسة بذلك أعتقد أننا علينا أن نقوم برسولها فهذا يأتي بالنسبة لأننا نرى لماذا لا يأتي لأننا لدينا بعض الأشياء بالنسبة للمشاركة لقد قمت بإضافة أوكا ديون ويسفير في جانوري ويجب أن تعمل فهذا يأتي بالنسبة للمشاركة ويجب أن تعمل في مجموعة نعم, ولكن يأتي بالنسبة للمشاركة لقد قمت بإضافة الأشياء بالنسبة ويجب أن نبدأ بالنسبة للمشاركة فقط في البداية ونجدها على الأساس ولكن يمكنك أن تؤدي لنا يعمل على إضافة ويسفير في أساس ونتمكن من العمل لقد قمت بإضافة لقد قمت بإضافة إضافة ونجدها على إضافة أعتقد أن أشسم تأتي بالنسبة للمشاركة وكذا أعتقد إضافة لأنها لأنها سوف تفعل أحبال أرهاب ولقد ولكن الأحب السامر العالم is covered in the install boot يجب أن تأخذ التأكد من التأكد من الوصول. ولكن إذا لم تفعل هذا, يجب أن يعمل. هناك another way to customize your Cloud Shell and you get a larger storage if you really need to do several ones. You can also create a VM and run the installer there. It's also faster than doing that in the Azure Cloud Shell because the Azure Cloud Shell is rather limited in the resources. Yes, so as long as you are in Azure, that would be okay. And hopefully, okay, actually you reminded me. If you want to help out and make Microsoft adopt Fedora Core OS faster, there is a voting here. Really? Yes. So if you can vote to this issue. Only seven votes. Everybody vote now. Put the link in. We should provide this link somewhere in Slack. Oh yeah. I'll take it. I don't understand what happens. Walid, I think the hour is over. The only thing you can go on a little bit longer if you think you'll get the Azure one up. It's just the live stream will stop. The blue genes will continue on forever. Okay. I don't think that it will be ready in a couple of minutes. I think my expectation is it will take another maybe 15 minutes from now on. Walid, are you out there? Walid, would you like a last chance to proof of life on Azure? Hi. Yes, I'm here. I shared the link on the chat. Basically, this is an application from the Gram Shipley. I'm an excellent hacker. Okay. So basically. So I start sharing my screen. Yep. Go for it. Okay. I remember Gram Shipley. Favorite ways to waste time. Yes. So basically, I'm killing bots. So here it is basically I killed a bot and it's been recreating it very fast. And the developer experience, you can see the monitoring is really good. So this is for this project, the Wild West. So the dashboard is up. I like this. This is like this. This is last week release. It's a stable release. And I like it. It's perfect. I didn't need to do the batch. Everything looks fine. So all the operators are working fine. You can see the memory. It's really not utilized still. You can see the network transfer. You can see the number of node 6 nodes and I can click on them. And see that it's the Joseph cluster we created earlier. What else do you want to see? And from operations point of view, it tells me that I can update. There's one release after this. So I guess if I click again, let's see. There's only one release stable after that one. Okay. And how long do you think that would take? It depends. It depends really. How long is it in Saudi Arabia right now? That's the real question. Sorry? How late is it after midnight now? It's 5 minutes after midnight. Yeah. We're keeping you up. I think that's a very good proof of life even though the update is failing. I think you might let it go. Did the vSphere 1 complete? I deleted it. So we thought that that's it. So we deleted it. I'm sure I can bring it up again. I'm trying the automated one for vSphere really. I want to get this thing up here. Okay. So the update failed. Okay. This is still very early release. I mean, not tested very well yet. This is last week and the update is like 2-3 days ago. Holy. Yeah. So nobody is playing this game. Yeah. Oh well. I think it's at this point. Somebody is playing. I can see. I can see. Yeah. But yeah. Here I'll hit you up a little bit. A few more times here. Oops. But I think it's, I think we've hit the end of our energy for demos and for the day. And I'm really glad you could do that. And this is a great slide too for all of the. If you go to full view it in full production presentation. A great way to end the day here. Kill and pods. Because there's no better way to pass the time. When you're at Kubecon over the next few days. Kill and pot. This is probably what a lot of people are going to be doing. Yes. This has been a wonderful day. And I really appreciate the open stackers for getting up early out there in Australia. You for staying up late. Charo for, you know, sticking in there from the beginning of the day. It's been pretty, pretty wonderful. If you want to hear more, I highly encourage you to check out okd.io. And join the okd working group because you'll get the announcements of new events. Tomorrow there is actually an okd working group meeting on the Fedora calendar which I think I'm going to share my screen just for a half a second here. I'm going to take over. Did I leave? Oh, see, there I am. I'm shooting you down. Okd working group is meeting. And I'll move this over. And then the Fedora calendar. Grab that link. If you're interested, this is a great way to subscribe to this calendar. I'll put the link in here. And you will get all the info. Tomorrow we'll be online again just for an hour while we run through. I'm totally amazed at how many and all of the demos completed. I think the only one was the IPA over. Had, I think we just ran out of time debugging it, but we did manage to put in a live issue into okd. So kudos to everybody. Thank you very, very much for your time. And I'm looking forward to seeing a lot more of you all in the coming months. I know I heard from a few folks here who are looking forward to deploying it in production. So as you do that okd in production, we want to get your feedback. We want to hear your war stories. Hopefully they'll not be war stories. But and as always updating the docs and sharing the slides to all of this stuff is key. So OpenStackers, if you can share your slide deck with me, that would be awesome. And I will endeavor to get this the all the videos up by the end of day tomorrow. But I look forward to seeing you all online tomorrow as well for KUKON. So if you're looking for us I should be in the Red Hat Slack channel somewhere. And there are tons of good talks coming live at KUKON tomorrow in the next day. So stay tuned. And we will hook up at the working group meeting tomorrow and let everybody go back to their days. And that's then probably one of the most fun days in easiest working group events I've ever done because I made everybody else do the work. So I'll love it and we'll do it again. So take care all. Take care. Thank you. Thank you very much. Thanks.