 Yeah, why don't you intro it and then hit it off? Okay, awesome. So so welcome everybody. This is this is hyper ledger I mean it used to be like hyper ledger Budapest, but I mean it's getting basically pretty much international So I'm not quite sure if you know, I mean the location makes sense Today we're gonna have a very interesting topic. It's gonna be high for hyper ledger fabric on on Kubernetes Which is basically getting to be a very very exciting topic and basically The expert for today is is David. So David basically the the floor is yours Okay, well, hello everyone Thank you for assisting this meetup. I hope that you learn something from here some some tips so Today today's topic as Daniel said, it's about how to deploy a hyper ledger fabric network on Kubernetes So I'm going to introduce myself. My name is David Bejo I'm a software architect and software engineer at two companies for software, which is my company and then Altia which Works for your IPO and then I part of building various network One is for your IPO, which is a company that manage three more than designs in Europe And I have other contests in the United States for some for a startup Where I use these tools that I'm going to show you during the following two hours My experience in blockchain is primary on hyper ledger fabric. I started like two years ago and We're going in 2020 and this is the primary technology that I have been working on and I'm also developer and maintaining the Kubernetes operators for fabric. There are more Kubernetes operators, but Because when I started in 2020 the first The first things that I that I saw in the product was that it was very powerful but in order to deploy a network in An hour or two hours, it was Very operational heavy operational. So I thought, okay, how can we Improve this process and then I saw when the operators were going to See more in depth in the following screens Well, so having said that, let's jump into the agenda We are going to do a quick introduction of what we're going to do What network we're going to deploy, how many organizations we will see this and then the basic Logic of how a Kubernetes operator works. We have to keep in mind that a Kubernetes operator is not only for hyper ledger fabric There are Kubernetes operators for other products. For example There is a Kubernetes operator for Postgres and what does this operator do? This operator uses the deployment of new Postgres cluster and has logic built in not for Kubernetes, but for this product specifically Then after we finish the introduction we will do a demo One hour more or less, we will deploy an orderer organization We will create a channel, we will deploy a chain code and add another organization And then deploy a chain code again for this new organization to pick up the new changes and add a new orderer organization This I don't think we will make it, but let's try to do all of these things And then the remote chain code development, this I wasn't able to do it and this was a lot of A lot of time, so if you stay more in the in the meetup, I will be able to tell you more about this feature And there of course, if there is any question During the meetup, I think Daniel will be able to pick it up and Ask the question and we can answer it And then 30 minutes after the demo in order to answer all these questions Okay So let's jump into it First, I want to do a recap of the components that are involved in a blockchain network In this case, in the Hyperledic network We have the orderer, which is responsible for the consensus of the network and to produce the new blocks And an orderer has, as we see here in the picture An orderer has sign certificates and TLS certificates, which is the same Let's make it the pointer, okay It has TLS certificates and sign certificates, which is the same that the PR has The PR is responsible for managing and executing The chain code, functions in the chain code, and it also has the sign certificates and TLS certificates So as we can see, one of the tasks that we will automate is the generation of the certificates And this is something that the Kubernetes operator can do for you So we have the PR, we have the orderer, and we have the SDK client, which can be an API that you have And these of course have sign certificates, which are used to sign the Messages that are passed into the PR in order to verify that the source is a legitimate one Having said that, then let's see what is the goal of this method So we will have three organizations These two organizations, one and organization two Will be PR organizations, we'll have two PRs Here we can see, two PRs for the R1 and two PRs for the R2 And then we will have the orderer 1 MSP, which can have in this case two orders We will start with one and then we will have more orders at the end of the method And then we will create a channel demo With some channels, in a channel you can have as many channels as we want If you can follow the scripts, maybe after the meeting, then you will be able to create As many channels as you want and as many channels as you want So this is the goal, two organizations for PR and one orderer organization with multiple orders And of course a channel and a channel So how are we going to do this? In order to create the channel, we will use HyperLayer Fabric 2.3 Which introduced a feature which is the channel participation API This channel participation API allows us So in the past with 2.2 and another versions The ordering service needed a system channel And this system channel added extra operational tasks in the project And with the channel participation API, this is removed, there is no system channel So you can easily generate the block for the channel And then ask the orderer node in order to join this channel This also eliminates a lot of privacy problems because in the past, in the system channel All of the orderer organization needed to be in the system channel And therefore they had access to what was going on in the network So this was something that was improved in HyperLayer 2.3 and this is something that we will use The second thing is, the second item is In order to perform all of these operations Then there is a QCTL plugin which is built into the operator And this is the commands, the tool that we will use in order to use this installation External build that we will execute in Kubernetes This is when we go to the installation and execution of the same code We will deep type into this part The code can be found in this link, it's already in the chat in this zoom call if you want to check it And of course we are not using Pyptogen in order to generate any certificate And one of the questions that you may be asking yourself is why a Kubernetes operator? I know that there are a lot of good tools in the ecosystems There is Celo, there is the blockchain automation framework But these tools focus on a lot of blockchain of HyperLayer products, in this case Fabri, Gindy So the idea of this Kubernetes operator and why we need this Is that these will help organizations that are already using Kubernetes In order to deploy a HyperLayer Fabri network using what they already know And what they already know is Jamel files Which they already use in order to deploy other applications Such as deployments, services, ingress, etc It extracts ourselves from the bootstrapping of the node As I discussed earlier the generating of the certificates It is based on Kubernetes, so this can be a premise on any cloud, Azure, AWS And the Kubernetes operator is also adding extra features For example, the certificate renewal and more use cases like monitoring, etc So this is the goal of this Kubernetes operator to ease the operations needed on a HyperLayer Fabri network And how this Kubernetes operator works So this is the user, this will be us in this meetup And the user issues modifications to a custom resource There are three custom resources that we will use These are the peer, the order and the certificate authority And there will be a program running which is the operator And this will keep track of all the custom resources in this case And when a new peer resource is created Then it adjusts themselves to the current state And the current state is deploying, duplicating the needed resources in the Kubernetes cluster That is if you're familiar with Kubernetes This is create a deployment, create a service, create config maps, secrets for the certificates, etc This is the main logic that the operator can do Of course this can be extended for any use case And you can do your own operator for any use case that you have in mind If we go more deep dive into the operator So this HLF operator is the program that runs It's subscribed for CRDs which are the custom resource definition And this box, this big box is the Kubernetes cluster And to this Kubernetes cluster, external applications connect In this case if we see at the right there is the QCTL HLF plugin Which this is what we are going to be using But you can also with Golan create your own API And then interact with Kubernetes creating CRDs So you could basically with this operator build a SaaS application that creates These PRs, create disorders, create these certificate authorities Just by creating this custom resource definition So in fact I did some proof of concepts and this was this work in the parcel You can try to do the same, create some APIs that extract the end users from this deployment And we can jump to the demo Maybe I went too fast maybe Are there any questions, Daniel in the chat Anything that may be interesting to check before the demo So basically I don't see questions at the chat But I mean it's a good idea so we might as well stop for a second Anybody from the audience having some questions If not perhaps I would have one question So generally what's the what's the benefit of using a Kubernetes operator So why is it better than just you know I mean using ports and services and standard stuff The the primary benefit is the automates that is built in So that is the primary benefit We will check the parameters but At the end there are the custom blizzards that we talk about This custom blizzards definition has a lot of parameters So with a Jamel that is one file You could create all of the rest of the files So the deployment, the service, the ingress etc And also there is logic built in If we maybe we can take some time and go to the chat We go to the chat operator and then the charts These are the charts that are used Let me put this in the chat So for example the peer This is the chart, these are the templates These are all the resources that are created This is what is going on behind the scenes So when we have a CRT of the peer at the end The operator generates this certificate and invokes this chart And in this chart there is the TL certificate already generated You don't need to go for example to the fabric CA and generate the certificate And then paste it into a Jamel For example and then executing and creating again all of these resources So this is already built in Of course not every, I mean the rest of the food generated There is custom food generated that cannot be automated So in this case there is a QCTL plugin That we try to automate as much as possible But this discusses its own limitations This is something that it needs to be improved Maybe with the help of people that are using it Super thanks, there's actually two more questions in the chat I would just leave them perhaps, I mean you might as well answer them So one question is do we still need Kubernetes charts? I guess the question is ham charts Or can we resolve everything using the API? And the second question is pretty similar How it is different from her chart deployment Okay this isn't on YouTube, no? How is it different than chart deployment? This is a very good question It's not, I mean at the end, these are Jamels You could create I'm not sure how deep to go because we will see this in some minutes Where we start to deploy the parameters that are there And how it differs from a chart deployment But how I see based on the previous project Then you have a chart, then you have some parameters And then you have some deployment that automates the creation Of the secrets inside the cluster So at the end these mimics the functionality of A Kubernetes operator running in the cluster And then generating this for yourself It's more, I mean it's not different Than deploying it with a help chart It's just faster So in one hour we will deploy from scratch To organizations with API and we can add more peers And then an order organization And then create a channel and then install it So this is what is good for this project That in an hour you can create a network We have more questions, no? I don't see actually more questions at the moment Okay, in some days Is this solution scalable? The operator is not a single point of failure What the operator does is just We will go back to this So imagine if the operator breaks And then it doesn't work The peers and order and certificate authorities That work created by the operator will keep working Because there is no dependency between the peer Between these components of the operator It's just that there is a new CRD creator in this case Then it adjusts the state And then adjusting the state is just deploying the application It happened to me that maybe the operator restarted Because we're in spot instances in ASU For whatever reason it restarted But then the peer orders and certificate authorities Are not affected by these people They have already been deployed So it's like if you are running them too And then you have the help dealer in the server And then the help dealer classes The applications that are already Have already been deployed Then will be already there So there is no It's a single point of failure But it's not like something that is so critical The only impact will be that you will not be able to create Or modify peer orders and CRs But the components already there will be working I think we have more questions I don't know if I answered to you, Adrien How to deploy the chain code This is how to make it There will be QCTL plugins how to make it You can install the chain code approved and committed Passing a configuration, a network configuration WorkCausDV will be hosted to manage the state as I don't see The CausDV is in the We can see here in the charts The CLF period Emplates So as we can see here in the deployment There is the volume for CausDV But there is an if here There is a conditional that if it starts the B Then we create all of the pods So this is basically the CausDV is a site card It's designed to be as a site card to the peer And it also has the logic to explore the metrics Which is really useful to find bottlenecks On what is going on in CausDV When you have complex queries So yeah, it's in a site card I will it auto check failure Angel itself This is based on the Replying to CalTune He's asking if this will check failure Angel itself This depends on the Kubernetes The JS Kubernetes automatic Heal of all of the pods that are not healthy Does it cover adding new words? Alexander asks Yeah, with the QCTL plugin yes Although for production you may want to build your own workflow And does it cover the renewal? This one yes And this one you can use in production It covers renewal for orders and peers I don't know We have more questions, Daniel Oh yeah, there's one more question in YouTube I'm a hypervisor beginner Having problems with exploring with Fabric So basically we had one session It was like, I don't know, two months ago It's on the Hyperlegia YouTube channel It's called Hyperlegia Fabric and Monitoring If you take a look, you can find some hints How you can set up Explorer with Fabric And some sample quotes as well That's Digby's answer He's talking about the Hyperlegia Fabric Explorer To see the blocks and all of this At the end, this will need a network configuration Which can be in JSON But this is for every Hyperlegia Fabric network So there is not really logic built-in in this operator In order to deploy this Explorer But you should be able to get with the command That we will show the data To start this Explorer One thing that I want to comment is that For example, for the monitoring of this There is an option to use Service Monitor from CoreOS And this will use the If you have Prometheus operator installed in your cluster You can add this option and then It will start to get the metrics from the PRs Or from the CowsDB that you have configured And this is for all the components PR, Orderer and CAs So there is one more question And I would propose So let me just cover this question And then you might as well have a longer Question and answer around at the end So the last question is that I was wondering if deploying Hyperlegia Fabric On Docker SWAR is a good practice Or is it better to use Kubernetes? Good question, but the Docker SWAR At least for enterprise is already deprecated There is some activity going on in the community But I mean They have stopped development from Docker So it's not recommended to run it on Docker SWAR Basically because the underlying Docker SWAR is dead Basically So at the end He says, I'm supposed to switch to Kubernetes? I think so Kubernetes at the end have surprised Docker SWAR Okay, so let's move on to the demo Let's go to the demo I'm going to ask, can you see the screen? The letters is everything okay From your view Yeah, I can see the code actually Yeah, it's really good You can make perhaps if just one little bit bigger But not very much, yeah Perfect So for anyone that wants to follow this guy He will need a Kubernetes cluster At least 1.15 We will use 1.22 We will need a kubectl plugin And we will need help Okay, I'm seeing some questions I will We will answer it after the demo Or when we go to the point of the question Okay, just for the people that are asking You can keep asking at some Google store Okay, so these are the prerequisites for this cluster To see the plugin and help Then nice to have for exploring what is in Kubernetes I usually use Lens I don't know if some of you know about this tool But basically you can have like This is Lens And you can see the code, the deployments The secrets Everything And then you don't have to go with the kubectl Common doing these things This is something that if you are going to deploy I would recommend If something fails Or if you want to get the logs Or etc Then Let's jump into the kubectl plugin These are the commands that we will use In order to add this plugin to the path at the end We will first I will not deep dive into this But you basically download it and see And then I'm removing what can be there in this folder Because I have run it multiple times Then unzip it Then I make this un-executable And then I move it to the USR local bin kubectl-hlf This is very important because kubectl takes all of the binaries That start with kubectl-dash And then you are able to execute them As this kubectl space-hlf And then you can execute it Then let's execute this This will move the binary to the To this path And then we can see if this has been successful We can execute a simple command Then go And we will be able to see all the available commands So with this we are set up But of course we need a Kubernetes cluster for this So in order to provision a Kubernetes cluster I have provisioned it before this talk Because I Didn't want to spend time here But I'm going to deploy We're going to deploy this blockchain network In my local And for this we are using kind Which is basically Kubernetes on Docker I'm sure that some of you have Know about this But I just want to show the web page So this is Kubernetes in Docker And then you can spin up Like a Kubernetes cluster with 10 nodes With five nodes Different configuration And this is very useful for testing Applications in Docker In Kubernetes without having to go through Minikube Because Minikube has You may know it only supports one node So if you want to do some serious testing Then these two can help you And this is maintained by a Kubernetes team And we can create The Kubernetes cluster with this command This is the default configuration This is one node only But this will create us A Kubernetes cluster with this version 122.2 I think by the time of this presentation There is 123 So this may be possible But we will stick with this Of course this cluster has been created And then we can access this This pure config Let's put it in this path This is bureaucratical We're not getting any value from hyperliferry But this is something that we need to do So this is the pure config And this is useful to Copy and paste it into Lens In this case We can add it into Lens In order to see the status of the cluster We go here in the left Add cluster Then we can paste it as text And there we go We have connected to the core of this cluster And we can see the cluster, the nodes We have only one node The workloads Which we don't have pods in the default namespace But we can see all the pods in all of the namespace And then we see that the The core DNS All everything is on So we're good to go to the next step Which in this case if You are doing this via command line We can basically get the nodes and get the pods In order to verify that there is cluster connectivity And these are the prerequisites Then the next step is to deploy the hyperliferry operator In this case This is the only chart that we will need So the goal of this step is to deploy the Hyperliferry operator and the custom resource definitions If we go to this repository HLF help charts We can go to the chart And then we can go to the HLF operator And then in the templates This is what Co-ordinates will create It will create a deployment with the operator Some service metrics And the CRDs And this is what This is the Important part Because this will create If we go with one This will create a custom resource definition That afterwards will allow us To create With a jammer CA in this case With these parameters So this is the This is part of the installation of the client's operator So in order to install this We need to add the repository Because this is hosting You have pages I have added already the repository So this will not do anything And then after installing the repository Then we can install And then at this transition time I'm specifying here with the Daseff flag While specifying the jammer file That we can see In this case And it has the parameters In this case where it's in a beta version Which is something that we will release In the following week And there are more resources But this is what we played most of the file So let's execute it Help me install And then we can go back to Rens It will take some time Let's see Let's put a bit That much Okay Now it's deployed And if we go to the lens Then we see here the path And this path is the one that has the logic To react to this CRD that we saw before So we need to wait until this path starts And then we can keep going Then if you have some parameters to change Then you can run the help upgrade And this will update The resources, the deployment and so on Then in order to check We can check the CRDs As you can see here there are four Because at the start of the operator We try to automate also the genesis The genesis for the system channel But we stop doing that So this is why we have four So the ones that we will use are the Fabric CA The orderer nodes And the Fabric peers And these are the custom resource definitions We can check the codes We can see that it's running So we are good to go Then of course we can check the logs of the operator controller And as we can see There are only info logs So we are good to go also Okay, so I'm going to remove the resources And then let's start building the blockchain network We will create these three directories Which will be the ones that we will use In order to store the YAML files And then we are going to start deploying the peer organization But before doing that I want to just recap Of what we're going to do So right now this is the goal So right now we are with the org1 msp And we are going to deploy the peer organization Which obviously involves creating first the CA And then the peer And then we will continue to the order Into deploying this And then at the end we will deploy the organization So let's create the Resources for the CL This will create a YAML Then I'm going to apply And then in order to save time I'm going to apply And then we're going to review How this is What are the parameters that we have here I remove this So we have created a file with this name In Kubernetes In this name space And if we collapse all the parameters We have a lot of parameters But this is because This is passed then to the file configuration So at the top level The most important parameters are the CA Which is how the sign certificate authority Will behave And the TLCA Then we have the storage here For the WCA Then we have the In this case the service That will specify that this is a raw balancer If we are deploying this on AWS On Amazon And we have the metrics In order to specify how this How the Fabric EA metrics should be Should be configured Then the course This is something that is passed to the Fabric EA The database Which you could link this Fabric EA to a Post Recycle database In this case In order to ease the installation We are using SQLite 3 And then the host And the host on the Istio Is related to a question that you Asked before How do you suppose network nodes If I create You create a separate load balancer for each So in this case You can opt in to use Istio And to specify the inverse headway That you want and the port That you want this to listen So based on the Istio and based on the host Then the CRDs needed for Istio will be created And then if you have configured the public IP And the Istio in your cluster Then you will be able to access this Fabric EA Or Fabric PR or Fabric Orderer node From the outside or from external source And then let's jump into the Into the CA parameters Actually the CA and TLSCA are very similar But basically there are a lot of parameters For example the host that they use for the security Some configuration If we allow identity to be removed The CRL The CRL which is a file That basically keeps track of all the identities That are no longer valid Or that you don't want them to be valid Then the CCR This is for newly generated certificates You can configure also the intermediate CA And this is the name of the CA This seems like another parameter But it will be the one That will actually configure the CA In this case this will be the CA that will be used For signing certificates So signing certificates for example The SDK client that we sold at the start Will use the CA The PR will have also a certificate from the CA And there is another at the end Which is the TLS certificate authority And this TLS certificate authority Is used for the server communication Then we can configure here the identities This is very useful In order to have a default set of identities So that you don't have to let's say Create more identities Do more operations after the VLCA is deployed In this case we are creating a role A client Which has as a user a password And has basically full control In order to create new users And create users with basically any role And then of course the max enrollments of the register Which basically can be in this case is minus one But it can be as you want The maximum enrollment that you want This user to have And then you can configure your CA The CA that will be created in the operator And these are the parameters Which is what the basic parameters that we allow for now To be specified And when we expect the CA that is created We will be able to see these values in the CA And if we take a look into the TLS CA Then it's the same So it's the configuration of a certificate authority By default in order to not make it more complicated For the CA We created like two CAs One for signing and one for the TLS And then with this you can You have almost any use case covered And this is actually the recommended way For hyperlabel, not to issue all the certificates from one CA But from two CAs Based on the usage that they're going to have Okay So let's see if the CA has been deployed We can see the lens So we can see here Let's filter for the default And or one CA So we have the CA already Deployed We can see the logs This is the readiness proof That Kubernetes Inbox the certificate authority In order to see if the CA is still working or not So this is working 200 error code And we can continue for the next step Which is use this fabric CA In order to create a appear And what would we need in order to create a appear Then the first thing There are some checks here That we can some checks that we can do Okay, we have executed this command And the state that the operator is Exposing is that is running And then we can get the pods And see that two out of two running One out of one run Then the next step Will be to create the user That will be used by the operator To generate the certificate So we are using this command Which is QCTL-HLF This is the plugin that we installed At the start of this meetup And then this is the name of the CA that we have created You do not see that Or one CA Or one CA is this name Or one CA Then the user will be the user name Of the user that we will be creating The certificate will be the password For that user The type will be the type of user That we will create This can be PR This can be client As we saw here In the identities This can be also client And this can be also admin In this case As these are the certificates That will be generated for the PR Then this needs to be PR And then the role idea And then role secrets These are the values That we will configure in the Fabricity CRD And then the Meshpady Which is needed in order to Generate this user Then we can register This has registered the user If we try to register it again We will create a problem Okay Registration of PR fail Is already registered So this is fine And then we can create In this case we are creating the manifest For the PR So let's do the same Let's create the manifest And then let's execute it And then we can deep type Into the parameters that we will have Of course this is the quick start Let's say the database will start Into deploying a network With this coordinate operator When you are deploying production networks Then you will want to tweak Let's say this configuration of course Because we are going to say a lot of nulls Because a lot of entity values Because we are running this in our local So let's start with the line 8 There is a possibility to configure The COSDB supporters With the image, the school policy And that in this case will have it This is why it's null We can configure the COSDB password Using a password In case that we are using As a word state database COSDB Configure some parameters Such as discovery period Docker socket path Which I don't recommend setting this Because Docker is dedicated For Kubernetes But in 2020 when I started This wasn't dedicated You can still use it If you are running on a Kubernetes That has enabled the access to the Docker socket Then the standard chain code builder This will trigger the default behavior Of having an out-of-the-box external builder That builds the chain code inside Kubernetes And also runs the chain code inside Kubernetes Or you can have your own set of external builders This is an array And you can configure the name This can be My external builder And the path that you have in your In your image So for this you But for this you will need a custom image That has this external builder setup I have used this in the past For setting up remote chain codes But by default the name can be like this It can be an empty array This will be it Then this is Very important All these endpoints If you don't specify these endpoints Then one endpoint based On the Kubernetes cluster IP Will be Will be let's say Mounted in the In the Kubernetes operator So for this if you are running on a Production network You will want to say Or 1 pr0 kfs.es 443 And then for the bootstrap Other pr Endpoints Will be this one External input Also this one So you can configure all of this As you please But for demo purposes We can leave this empty And then the operator will Guess these parameters And we will be able to see what are these parameters later Then for the gossip configuration These are two very important parameters Which is the org leader And just the direct election I believe that the recommended Now in the latest Version of the recommended is to set up org leader To use the direct election force But you can do it as you please actually Then host services This is very useful to To run in private networks When you have access to changing to dns Host This is To define the external host that will be accessed And this is only if Istio Is is enabled in this case Then logging Parameters The SPID The replicas Resources for the For each of the Containers The chain code The cost db The cost db spotter The PR So this is just Boil plate in order to configure What resources Do you want For your blockchain network And then this is the one of the Most important parts Which is Has to do with The generation Of the certificate So As we can see here in the secret There are We have enrollment And then we have Two Self-selections Which are component and tls What does What is this component useful So this component Will be Will use this CA This CA name This is what we saw in the fabric CA CRD This name CA This CA port So this will connect This will tell the operator Okay connect To this CA host Which is inside the cluster Actually And to this port And then in order to verify That the CA is the one that we want Then we can specify the CA the tls certificate Of the CA Which is this one in base 64 And of course In order to enroll We need the username password That we have registered In the previous In the previous commands And then The same will be For the tls But with the with the exception That For the tls we have also The Some parameters for the csr Which this is needed in order to Generate a certificate With the host names or IPs That we want to access From this case we have local host In case that we want to Jump into the To the container and do some operations And then we have the external IP Of the cluster Of the core of this cluster You can also put here a C name But For this This will work also And then more boilerplate For the service Not port Service monitor This is for the monitoring By default this You can enable You can disable it To have some parameters here This will need a deep dive Because Is not only to configure the service monitor Is Is also to Configure also the dashboard Simplafana And etc Stdb For the operations we will be using Level db You can use Cards db Also As we want And the storage part Which for the chain code Cards db and the peer We have Configure this The chain code I want to Make a point On the chain code The chain code is not The chain code that is running It is the chain code Is the file server That hosts The chain code And we will see this more in detail Because this has to do With the external builder With the current external builder So We will see how this is managed And we have the type of the Of the image As and this parameter The update certificate time This When this is changed This is actually a date When this is changed Then the certificates are renewed And then the PR is restarted So this is how The certificates renewal is Implemented in this case Just changing this The operator Knows this And then it Initiate the process of Renewing the certificates Okay I'm going to do it Some water One moment So Right now we have deployed The PR The PR zero We can see some logs In order to see that There is no problem here Just a PR There is no channel The PR is not part of any channel Neither so And this blue Tells us that Everything is okay So Having the PR If we go back to this picture We have deployed DCA and this PR And now we're going for this chain code This Google is still the chain code But until we create the channel The created for the the ordering service We will not be able to approve and commit the chain So let's keep the mardowns We can do the checks here In order to check that this is running Stayed running And then the pods One out of one In the PR two out of two Because this is running The PR at the the chain code file server And then we are ready to interact with the PR There's some question from Adrienne Do you think we can add an organization The fabric network which is in order Or in this cluster With With the QCTL plan The QCTL plan is meant for for local network So I don't Recommend this way but you can try You can try At the end how I have work in previous project Is that I develop an API And then this API connects to multiple coordinates cluster This API has this logic of connecting to All of the components that they have So this API can keep track of three clusters From cluster and then act accordingly Okay, I don't know if I have replied this question But let's keep going So now the goal is to generate the network configuration Which is a general file that we will use To connect to the PR Because we need to have a certificate And this certificate needs to be issued by the CA That we just created So first let's create A user admin We will create a user admin Pw of type admin With this control and control At this password Then We register this user Then we enroll it And this will give us this file PR or one jammer If we see this file This is just some jammel with the certificate And the private key And this has been generated by the public CA So these are like some tools in order to To bootstrap the network and to start interacting with the PRs As possible Then we will inspect the The organization one And then this will generate the or one jammer Which we can see here And in this we can see the organizations Which have only one PR And then the PR with the TLS CA certificate authority But in this file we need to add the users So and this is the command that we will Execute in the next step Which is to add the user to this configuration This is like a Like a boilerplate code But you can have like If you have three PRs This will work the same And then in the or one jammer We have the org one msp And this the user is added So with this we are able to pass this configuration To the heaps detail plan In order to install the chain code Which is one process that we will leave in the background Because this takes like Four five minutes In order to install this chain code in Kubernetes So we will just run it in another window Just keep going with the With this repository With these tasks Then okay the process is now Started And then if we see in the Kubernetes cluster There is a new path Right And this path what is doing is building The is building the chain code And if we see there are four containers So the three first in the containers Have to set up the chain code volume To download the chain code source And guess what this container is doing This container is going To the PR Chain code server that we Will talk about in order to fetch the source Chain code And this chain code can be go Can be java It's basically your Your chain code that you have The chain code that you have developed And then it will build it And then at the end it will upload This is the final container This will upload the result Of the compilation of the build It will upload it to the PR chain code server So that when this is started Then the results can be fetched We can actually see this With a screen with a pixel with a diagram And I guess that this will be better So this is external builder At the top we have this install the chain code Which is a process that we are following right now The SDK client start installing the chain code Which is the command that we have just issued And it's issued to the PR Then the PR in the container invokes The external builder binary This is a binary that is in the PR image And then how it's configured by default The image that we are using interacts With the code that is API In order to create this path And the path this path is what we already saw In the lens this path is this one And it's the one that is building the image And then when this path is launched Then it fetch the chain code source From the FS server Which these are the two containers that we have This extra box is the path And PR is a container and FS server Which means file system server Is the other container And then this is used to fetch the chain code source And then at the end it uploads the build output And the same happens when the PR wants to run Wants to run this chain code So the SDK client execute a transaction On a chain code in a specific channel And it executes it in a PR Then the PR sees all external builders And then selects one If there is any of course You can have non-external builders So it invokes the external builder binary Then this external builder as it's configured Interests with the point of the API To create a path And this path well there is a error missing Which is fetching the chain code The build output from the FS server And then when this is fetched Then the path starts and connects directly to the PR And this will be the process that will be happening When we get to the point of running transactions If there are any questions Just put it in the chat and we will try to answer it As soon as possible about this process Because this is like the new way of Deploying channels in the coordinate Because it also gives you the ability to track Of metrics, of usage, resource usage Since if you do it in Docker You don't have access to these metrics So this also gives you extra visibility And observability Okay, so this is still going We don't worry right now about the example Let's create the order organization So let's create this folder Orderer1 Let's see Orderer1 is empty, this folder, perfect And then the same as we did for the PR Let's create the CA And apply it Then we can see here in the orderer1 CA This is basically the same that we saw This cost starts generated by default By the plugin But it's the same CA So the same configuration for 1 CA can work For you, for any CA Just changing this name And changing this cost Also this cost Which basically are for the virtual service And gateway for the HD So if we go to the lens This is being created I think right now this is created So let's deploy Then we have to do the same as we did with the PR We need to create a user in the CA So let's just create this Create this user This is created If we execute it again It will give us an error Okay And then we can create The orderer node manifest So we're going to create the manifest Apply the manifest There is a difference between these two commands Because one is for creating the Jamel The base For the Jamel And the other is for actually Applying it in our code in this cluster Let's apply it And then let's check What parameters we have in the orderer So Let's zoom a bit I will go from Most important to less important So these Parameters that are in line 9 And line 10 And also line 11 Are The most important parameters The bootstrap method This is a requirement In order to use The channel participation and API Which Requires this parameter To be set to true And we can actually Go to the Hyperledger In order to see How this is transmitted So this is the Create a channel for the latest Hyperledger version And then If we look For PartTC So Remove the channel This is actually a tutorial that I encourage you to follow In order to understand how the channel participation API work But in this case If you have worked with the orderer Configuration What this Is doing is just enabling Enabling this Key in the orderer And also the general bootstrap method The bigger We have to set this value to null Why? Because if we don't set it to null It will try to find the genesis And this is what we don't want We want to use the channel participation API And that We don't need any file For the orderer to start We just need The ordered app And An endpoint in order to access The admin API Which is a parameter that we have here If you want to access The admin API Via Istio You can configure it here The host For this endpoint For this port The store gateway And the port So you can configure this For the orderer One Admin For can be The one that you have So this is how you Will configure The Istio For the admin Endpoint And why do we need An admin endpoint Because the joining And The list in all of the channels In order to support This feature Hyperladyfabric created Another server Inside the orderer And this can have A different CAs In order to authenticate And etc So this is why there is an admin Istio Apart from the normal Istio Property Which is the 70-50 port I hope this is clear About why there are Two Istio Configuration And when is the Admin Endpoint used But this is basically for the channel participation API And then if you are using In minding that you are using A 2.2 or you are creating a 2.2 method Then you can set this to false Then this 2.5 method And then you could Put here the gms64 And this will work This will configure the orderer So that it Fetched the channel From this content Okay So Let's keep going We have the hostalysis Useful for private networks Ad, full policy Replicas This is useful in order to test scenarios Where I know there is Why you want to test resilience And then you can set this to zero You can set this 72.2 So it's only zero or one In this case we want one The resources For the Orderer container The same limit sequence And then This secret enrollment component This is what we already saw in the PR So this is a standard We just specify the CA host Which in this case we are going by the IP And not port The CA name The DNS certificate In order to verify that the CA That we are using Is the one that we want In order to avoid the hacks And then all idea The user and password that we have created In the CA In this step, if you remember In the right The one that I'm selecting To CTM and CLF CA register with the name With the user, orderer and secret orderer PWM Which is the same And then for the TLS the same Plus the host Then more Properties Not port Service monitor Which can be also for monitoring Of course This is a more configuration Then the storage Which In this case we are selecting two gigabytes You can select just the class that you want And The tag we are using 2.3.3.0 And last We Update the certificate In the case of updating the orderer certificate This is more Complicated Because The operator can update And renew the certificate But the certificate Of the orderer will need to be updated Also in all of the channels That they belong to So this is why it should be To have like so If I present the picture To have Here To have some API That you can Or some script that you can manage All of this to renew the certificate And then take this certificate This TLS certificate that was generated And then replace it In all of the All of the channels that the orderer belongs I hope This is clear Of course Until you Go through the process You will have those And understand But this is the basic configuration If you take this out of the box Then this will work Okay, so now We can go to the To the lens And we can see here on node one This is the path of the orderer node So this has been created We can check the logs In order to see What are the logs And we see blue Which means that it's okay Then we can continue Generate Order manifest Create ordering service We have created this We can get the pods Here Everything is running And then let's create the channel So in order to create the channel We have to generate A network configuration For the In this case for the orderer MSP And node one MSP So we create this And then points that we'll be taking Will be the ones From the From the word in this cluster So let's execute this And For service We can see here The node configuration In this case there are two organizations The order MSP Which has one orderer And the one MSP Which has the peer And of course with the certificates For the orderer And the certificates for the peer And There is this extra property Which is the channels Which is the default configuration The default orders and default peers Which will be used In order to interact with the network This is This is Interpreted by the golan Because at the end This binary It goes to the To a Uses the golan So having said this Then let's Register the user Let's enter the user For the Orderer For the orderer organization Let's add the user And then we have the same In the order services We have the same but We have just Added to this The orderer MSP Then we can generate The block As you see these are all utility tools You have not seen really Any command line From the hyperlabel fabric But this could Be extended for any use case That we have And this could be cool running On CICD For example Jenkins So This Is really powerful Can help you in order to Boost from new networks So in this channel Generate There is an output parameter Which is the file that will be generated The name of the channel In this case The orderer organizations Which are the order one And the orderer organizations Which is the order MSP And in the next Part We can see Using config Config the accelerator We can decode the demo block And then Open the The block json And if we vote in the block json A very big json file But As you can see here in the old one MSP This is a site application And we have two Components application And the orderer And then there are some policies for the channel For example Who can There is There are the policies for the orderer Which are these ones Which is for example Who can update this part Of the channel And there is for the application Also in this case The order one MSP has Some policies for that means This is useful basically to see that The number of organizations that We are adding to the channel Are the ones that we want So if we see here order one MSP And in the application And in the order we see The order MSP Were fine So Based on this we can Enroll The user that we have generated before In order to Join The orderer mode To the Let's say The CEO Of the orderer So we have enrolled it Then A new channel has been generated Which is admin DLS Or the service Which is a file that has A certificate And a key And then based on this certificate We can join This network This This orderer Note to the channel This identity This is the name of the orderer node Which if we go to the If we go to the Coordination API Then we can see that this is the name But this matches the orderer node That's one matches with this one And we of course Pass the demo block Which is the one that we have just generated So we can join it Then the status code that retires The admin API of the orderer node Is 201 And this means that the Orderer node is in If we try to execute it again Then we get that 405 Which means that This orderer node is already On the channel Okay These are info messages from the 2CTL plugin And then On course We can join the PR To the channel I just want to pause here I don't know how we are about time Right now How do you see it, Daniel? Sorry Can you just start with the question I was We are at the point that We have added We have joined the orderer node To the channel We are at the point of adding the PR To the channel So I don't know if there are any questions In the YouTube I just want to pause In order to try to solve Any doubt So we can stop For a brief Like question answers Round if you like Or if that's the question Okay We can continue And at the end We will continue What is left is just Join PR to the channel So we will Need to enroll The same This is very boring But at the end this is the These are the steps that are needed So we need a user for the PowerXCA In order to be able to later Enroll it And get a certificate So these are the commands So this is what We ran before CnRoll With the name of the PowerXCA that is running in the cluster Then the user Which is admin Ccredadminpw And this is the output And then we can inspect the cluster In order to get the network config And then We can add The user that we have just Generated To the network config In order for the Golang SDK To fetch this To be able to use The certificate And then Based on the org1.jammel The network configuration We can join this PR Which is org1.pr0 Dot the name In this case Using the admin that we have added To the network configuration So if we execute this Then channel join it And if we go to the lens One moment We can go here To the lens on one PR0 And we will see here that Creating Major demo Getting block information from block storage So this means that The PR is Already in the network Then The first thing that we need to do After joining the PR is to Add the anchor PR in order for Other organizations to Find all of the PRs Of this organization And then we already Have the web Adding the anchor PR This is the transaction ID And There is a utility In order to see What is the height Of this channel Of all the PRs So we can just execute this With the config org1.jammel And the channel that we want And this will give us Just The URL And the PR And the ledger And we will see this Being updated As we add more blocks So now we are At the part of installing the channel So When we try to install the channel This is already installed Then we can query The channels that are already installed And this is the Name and the hash And then we can Approve it For Our organization And then we can Commit it For the person who is not Approve or who is not Don't know the parameter So in order to approve This organization We need the network configuration Which we have recreated Before And this Is for the chain code definition So this needs the package ID That this Organization will use Because this is approval The version 1.0 Whatever version that you want It can be in this stream The sequence which is one The name of the chain code The endorsement policy And the channel We are approving this This chain code definition So as we can see here At the right this chain code Was approved and this is the Transact ID And then we can commit And then we will see also another Transaction And then as we have Committed and as we have Already installed the chain code Then we have some utility functions In order to invoke this Chain code In this case we are executing Chain code invoke Passing the configuration And the user, the PR Where we want this transaction to go And the chain code Channel and function That we are invoking And also some parameters that we are invoking In this chain code So we can invoke This chain code for the first time Because It hasn't taken Much longer But if we see here in Lens There is some other chain code There is another path that has been That has been created ccfabcar And this is what We were talking before In this picture In the standard builder So the standard builder binary Has interacted with the API And it has created a path And then this path is connected directly To the PR And this is why this is working Because we are from the SDK client We are executing it in the PR And then this is connected The PR is connected to this path And this path Is the one that we have here Actually we can see the logs We can see also In this case I think This depends on the Of course in the chain code In Java for example You have a lot of logs In order to debug and you can Like output custom logs And they need containers in order to Download the chain code output For the file system server Which is the one that we see in this screen That has The output The build output Of the chain code Of course we have Invoke a translator They need ledger This is the fab card This is the Chain code that we Can deploy And then we can invoke the chain code Just query This will be a lot faster Because this does not involve A new block to the blockchain And we can just query And get all the cards And I think We will not have time For doing more but The idea Was to update the channel Also there is here Create a second organization Which We're not going to set it but let's go through What will be the process We have already saw First we have to generate the CA manifest Then apply the CA In order to create CA Then enroll Register The user in this new CA In this case the CA For the organization too Then we generate the PR manifest Using the enroll ID And enroll secret that we have Used That we have registered Before Then create the PR And then this comes The second organization to the channel So for this we will need to create A second Network config In this case for the two MSP And this will output The files In a file, in a folder that is The Qtoconfig And this is a custom folder In order to be able to Add this organization To the channel If we see this folder I think we can see it Here Qtoconfig In the organization Or to MSP MSP Here basically there are all these CA search And CLS CA search I think I can do this With org1 And then this will generate A Configuration file here In the root Sorry for not explaining it so well But we are shorting time But basically we will Just Reference this config tx Jamel which has The identity We will try to open it It has The List of organization MSP Points to this MSP That we just saw And the same for the org2 In this case we have an inspector But it will be the same And based on this config tx Then the Qtoconfig plugin Will pick it up And then it can Add the organization With To a specific channel With the network config With the user in the network config And this is the organization that it will add To the To the channel And then after Having updated the channel Adding the new Organization We could respect The channel And this is something that you can do Between the steps And this demoization If we take a look As we have saw At the start with this block json So this is a json representation Which Is hard at the beginning To get used to But is one of the best ways because You have You can see all of the configuration that is In the channel Related to the policies and related to everything But here the main groups Are the application And of course the order So Apart from this The rest is standalone So Very basic is what we did With the first organization So In order to add it We need to generate the network config Which will be org2 Of course we need to enroll And then we can join The PR2 to the channel The PR of the organization to the channel But of course We have added The PR to the channel Now we need to approve And commit again for this new organization To have To be able to execute transactions For the PR2 to endorse The transactions So we will need to install The chenco In the PR Note here that the network config is org2 Then You can query All the chencos installed In the PR of the organization to And then Both organizations Will need to approve the chenco again Note that this sequence Is updated So sequence Before it was 2 now is 1 And the policy now Involves organization to Also Both have approved Then one of them can Commit And of course we can execute These two Comments in order to Do some tests in the chenco So I don't know, I mean I think we can go with the discussions Of course I can keep going because There is a section also to add more orders Just to Create the yaml for the order 2 Same as we have created order 1 Then we create order 2 Then we create order 3 Then we create the manifest for order 3 Then we apply The manifest for order 3 And then we join Both to the network To the chenco in this case And once we have Added these Orders to the chenco we can add them Also as a consent Which this involves also The network configuration Registering a user Enrolling it Adding the user to the network configuration And then we can Generate the transaction The channel update In order to That this channel update Will be used later To update the channel in this case So this generates the protobuf The channel update protobuf And what this does is Fetch the channel Configuration And then adds to the Consented Safety It adds this extra order Then the channel is updated And then the same is Perfor For the order 3 In this case There is this channel update Protobuf being generated And then the channel Finally is updated And then of course we can expect The channel at any time So this is the There is a mix here of utility funding Utility to functions Typed integration With hyperladyfabri But at the end this allows you to Create Genomic APIs also In order to create these Diamonds that we saw In order to customize it with your Domains, with your resources With your parameters And Hopefully it can help you In the future Daniel, I don't know if We can go to The discussions part Yes I would propose First of all Thank you very much for this Super presentation and Demonstration I think it was Very much the Diving to hyperladyfabri So I would propose to have some Let's have the Q&A session And then we might as well Have some discussion as well If you guys have any questions Both in the chat Or I'm taking a look on YouTube Feel free to ask You can speak I think Also if someone has a Question I mean I don't see some questions Perhaps let me start With question So my first question is It's almost a product So you plan to Bring it to a product level Or Have it under hyperledger Or an open source Or what's the general Road map This product as you said I am using it With The companies that I'm working With And this is why I'm maintaining This product Of course Anyone can contribute This will be ideal Because right now I think there are two contributors Only But the idea is to keep Bridging the gap Between The start of a blockchain network And how to manage it And this will be Admin console that was released Like three or four months ago By IBM Or adding more Utilities, for example The certificates in new one Utilities are very good one And an obvious one That is already in the operator But As you saw here In the jammels There is not much Much else that we can do About this Because these are the parameters that Fabry has Already And What we can do is add a layer Or a sugar layer Let's say That makes it easier For the For organizations and companies using this This operator And I mean the idea is just to keep Contributing, to keep updating it And to keep improving This Developer experience I have not thought about Turning it into a product So this will be open source All the life Okay, super That sounds great So I'm not quite sure If there's question from the audience My second question would be You mentioned a couple of times I'm not quite sure So let's assume Some general So we install hyper leisure Fabry That's cool But let's just assume that after running Fabry for why we got Some administrator tasks as well So I guess like Upgrading chain code is possible Pretty simple The next one is revoking and renewing certificates I'm not quite sure how difficult Is that for instance This is If we say this We can do it live So in the lens We have The light We have the custom Resources In the left definitions HLF confus over And then we can go here to the peers Then we can do the peers And then we can edit This is the Jamel This is the Jamel that Is a story And Then there is this property You update certificate time Now it's now It has no value at all So If we change this To a date When the certificate is updated Then it will update This date in the status Of this resource And then it From now on it will keep track Of when it has renewed the certificate And then you will be able to Just update this date to a Future one So that You want this to be renewed On the 6th Of November The operator will pick this up And then the certificates Will be renewed Invoking the Fabric CA And then updating the certificate In the course of this class So let's just save it And then we can see the logs Of the operator This will be here Fabric PR This is I think from now You can see here That this now Created Not trying to update that So This is just Renewing the certificates And Now the PR is being terminated As you can see here If We see the secrets PR0 The TLS for example Take the certificate And then let's go to SSL Some random website Where we can Based the certificate And then see the Valid from and Valid And this has been the certificate that has been renewed For now It's the same, we don't have the previous certificate But if we had the previous certificate We will be able to see That the serial number has changed And We will be able to have proof because The valid from a valid 2 Of course we have created the PR One hour ago The valid from a valid 2 will be the same But if we expect it with OpenSSL we will be able to see this That It's from Now, not from one hour ago So This is how it's implemented right now So you just change A property in the PR And this is the same for the orderer And then it's automatically renewed So it looks great actually I mean It's much more difficult to do it Actually manually Fat I mean A lot of blockchain networks Have this problem right now Yeah, is there Any question from the audience Side Perhaps in the meantime I will have one more Question I'm rather a developer Or user In the direction of Kubernetes So my question is So How is the usage Possibilities of such Kubernetes Custom Operator Or Custom Resources I mean is it like vanilla Compatible Is it something that can be used like With Google Kubernetes Engine With Azure Kubernetes OpenShift In my experience It can work With Almost any Kubernetes cluster I have deployed on premise I have deployed this on Azure I have deployed this on AWS So as long as it's a Kubernetes It doesn't matter if there is a Layer on top of Kubernetes At the end we are using here Basic CRDs from Kubernetes Deployments So there is no Special Features Just From any of the cloud providers You can use any Kubernetes cluster So basically as soon as Custom Kubernetes Operator Is supported Then it should work Yeah, it's Related to the version In this case So this is the minimum Version Plus So as long as you have this version It will work I don't know if that was the question Yeah, I think yes Yeah Perhaps one last question From my side So generally How do we Can you just summarize? I mean there were some hints But can you just summarize a little bit once again What's with networking So I mean we got Kubernetes We want to do some At node port but somehow publish Service With ingress Which might be actually Different in different Kubernetes Versions so I Like I don't know it's nginx Or nginx or Contour or So there's a bunch of different Network settings Possible in Kubernetes So how is it possible To deal with that situation Very good question for Right now the demo that We just did It uses node port In order to access the PR Access the orders But this is useful for development For Small proof of concepts But for production In this case The optimal solution Will be to use Istio Why? Use Istio and not use nginx and ingress for example Because Istio supports Layer 4 with SNI And it's way easier to Have this kind of routine So if we go From the administration machine Which is us, we are the administrators We will access The HLF components through Istio Remember that in development We just go through node port Node port is a Port that is exposing The coordinates nodes And then it redirects to the HLF components But this Forces you to Expose a lot of ports And for production what you can do is Just have in the Istio part A public IP linked to This ingress And then this Istio can route To the HLF components Of course you will need DNS Services The DNS to point to these IPs And This DNS needs to be configured Also When generating When creating the PR Resources Because if we see here In the PR1 Jamel We have here host So if we are creating For one KFS.Test Or one PR PR0 KFS.Test This DNS Must point To the Istio At the end So this is the tricky part And This It's highly recommended To have it automated And to have it in place So the recommendation will be Use Istio Have a public IP that points to this Istio And then Configure These CRDs The February PR Also the February CA with the host That you have pointed to Istio And the virtual service and gateway Are already generated By in this case Hyperlate fabric Operate in this case So I don't know How do you use In this case NGINX ingress Or Istio So I'm not very much familiar With Istio unfortunately It's an exciting technology but I still have some gaps In my knowledge I did see it like NGINX And with contour a little bit Yeah Perhaps Istio is just more advanced Yeah For layer 4 of course And it is As you see here the SNI Which is an extension That is included in the clean hello This is at the start of the TLS Communication And this is something that is supported Out of the box by Istio And it's the The go to NGINX ingress In order to To configure it to For this Hyperlate fabric Network So in all of the deployments that I have Done I have used Istio In the AWS, in Azure And also on premise And this is what is Integrated already here So if we say Host To integrate this If we say here Or 1.0 KFSS And in the ingress We put it like this Then you will try to create The resource definitions For Istio And what are the resource definitions for Istio These are What we see here The gateway and the virtual service And this basically tells Istio When there is this Specific host In the request, then root To this specific peer Or to this specific order Or to this specific fabrics here And this is the configuration that Managed that is managed in Istio So I recommend to check It out, I will put it In the chat of Zoom But is the go to solution Is the go to solution For routing to Okay, super, thanks So basically We got like Technically 5 more minutes So again Is there any question From the audience side Or perhaps any Topic that you would like To briefly discuss Regarding of Hyperlegia Fabric And custom resources And operators If there is not any question Then if you are on YouTube In the comments, then you can post Any question and I will Try to answer it offline I'm checking the YouTube channel as well But I don't see any questions At the moment, but yeah Perhaps later on Later in the comments Yeah, exactly Okay, so Okay then, I mean Basically if there's no more questions From the audience I don't have more questions at the moment So it means that basically I would like to very much thank you For your Presentation, I mean it was A super challenging deep dive Into Fabric and Kubernetes So thank you again It was just super from my side And then I don't know if there's no more comment Or question or Or anything from Anybody, then we would slowly Basically close this meetup Yeah, well I just want to say if you want to Contact me, I'm on LinkedIn David Viejo And also on github You can contact me I think it's David Viejo Pomata But yeah, I mean just Great Google, I think This name in Google And then you will be able to Reach me out If you have any questions or any doubts Or if I can help you Yeah, I mean and after the meetup So if you send Like the slides Or even the github repo Then I can post it basically I mean both under the YouTube channel And in meetup as well So I mean all the links will be accessible Yeah, I have posted it In Zoom at the start Of the meetup but I will pass it to you Okay, so if there's nothing more Then again, thank you very much For this super, super presentation I would slowly say Happy I mean Have a new weekend But it's only Thursday so I mean have a happy Friday basically And thank you very much Again and that was For today and thank you And bye Okay, thank you everyone For staying up until now See you soon Bye