 Great, you're live. So hey, good morning, good afternoon, and good evening, everybody, depending on the time zone. This is Daniel Segu, and in this Hyperledger Meetup, I'm just gonna give a presentation practically and raise some issues on Hyperledger fabricants and more generally on consortium blockchain challenges. Perhaps with you from my side, I'm basically a software architect dealing with different blockchain challenges. I would say not just consortium blockchain challenges, but generally blockchain challenges in like more than five years. And then I would say this consortium topic is very interesting. So one big area at a time dealing with is Hyperledger fabric, of course. The other word is a little bit like the classical blockchain world. It's like more like Ethereum, Polygon, and then like the open blockchain world, I would say. So the idea is for today, basically. I got like 14 slides that I'm gonna cover. It looks that way, it's more like kind of a problem statement, basically. So I will start basically with pretty high level issues and then I try to go a little bit more into the details of like Hyperledger fabric problems, but it's gonna be more like a problem statement. So I'm not gonna be like necessarily answers. I will try to give some answers, but I won't go very much deep into the technical details and technical answers. Again, it works like a problem statement and hopefully there's gonna be like a couple of subsequent meetups and presentations on different answers for the problems that I raise actually are here in this presentation. My proposition would be the following. I mean, I'm sure as I just raise issues or as I just raise problems, you might as well have answers or solutions for that. I would say I would finish actually my presentation. You are very likely to share your experience. It just looks that way. I just wouldn't like to stop my presentation in the middle for like 10 minutes or 15 minutes discussing one thing. So I would propose the following. I can likely stop my presentation and answer questions if you don't understand anything. Anyway, my network is not too stable at the moment. So it might happen that I will be dropped out for like 20 seconds during the call. So we might as well have some unintended breaks anyway. But I would say if you want to react basically for such topics, for such issues, you might as well know some solution or you want to share some best practices or ideas that you know or some experiences, you are very welcome to do it. I would say firstly during the presentation, you can welcome to share it in the chat. And then at the presentation, we're gonna have a lot of time. So actually we can discuss topics. You can very likely share your experience on possible points and on possible solutions as well. So that would be actually my presentation is like, I would say it's like 40 minutes. I'm not always very good estimating the time, but I think we're gonna finish in 40 minutes. And after that, we have like 30 minutes more to discuss different topics in details or if you have like experience, hands-on or maybe best practice solutions for different points, then you are very much encouraged to share it. So this is the agenda for today. I'm just gonna start very much, yeah, from a high level perspective, starting with blockchain and enterprise systems, application development, cryptography. So these are basically issues that you might face if you deal with consortium blockchain solutions. Then I get one slide, or what's my point of view of agile development in the blockchain scheme, especially in the consortium blockchain scheme. And then we're gonna start going a little bit more deeper into technical parts and infrastructural parts. So I'm just having one slide on fabric infrastructure and orchestration, some consideration on that. On privacy, that's already getting to be more fabric-specific. Then on designing for high-levelivity and performance, then one big part is like... Yeah, he'll be back in a second. As he was saying, his network was a little unstable. He'll be back in a second. While we're giving him a second, if there's any questions, comments, feel free to come off mute or type and chat. So again, that was the unintended break. So actually after backup recovery and cryptographic material, these are like a little bit more technical, but again, I'm just gonna raise questions practically, not necessarily give answers. And then at last, but not least, we can speak a little bit about my proposition of further community contribution. And of course, we can very likely discuss anything at the end. So that would be the plan for my presentation. And I will start with the first slide. It's very much high-level. So basically what is actually a consortium blockchain at all? And then I would just... I mean, I have more experience with the two edges. So I have like 15 years experience with enterprise systems. I mean, enterprise system is a one-organizational system. Even if you install like a distributed database or something similar, you get one organization to deal with. So to develop, to deliver it, to install, to maintain, to operate, to make governance for your system, that's practically one company that you have to deal with. One way of communicating, one infrastructure team, one customer, one set of requirements and so on and so on. So that's like classical enterprise system development. And I have some experience on the other hand as well. It's like the real public decentralized blockchain. I would say that's in a way the other side. So basically if you want to set up your mode and do something with real power polygon, then the idea is basically that code is low. So basically you can set up your environment and attach to the network, but you have to accept the network. So you can't do anything. This is the network. You should actually behave the same way that's the code basically is coded on the public blockchain network. Certainly there might be some exceptions. So if you are very much a hacker, you might as well consider like doing like double-spending attacks or something similar with like bribing or having a lot of miners or bribing basically validators and so on. But that's not the usual setup. So the usual setup is that the blockchain is pretty big. It's like having, I don't know, eight thousand nodes at the Ethereum network. You're just one node and then basically you have to be conformed with the rules that already existed by the rest of the network. So what is a consortium blockchain? Is it something which is more similar to an enterprise system? Or is it something which is more similar to a decentralized, to a fully decentralized blockchain? And while this is hard, this is hard to answer because I mean it depends. So if you have like a consortium blockchain with like 50 companies, I would say that's more like, you should behave more like as in a decentralized blockchain. So if one company practically attaches to the network, then the company should basically accept the rules of the rest, I would say. So that's more like something which is a decentralized blockchain. But if you have a consortium ledger between two or three companies, that's more similar to a classical enterprise world, which means that, yeah, I mean, you should not deal with one customer, but like with two or three, but basically the requirements and the expectations are more similar to an enterprise system. So what about hyperledger fabric? Well, hyperledger fabric is tricky in this sense because it's a complex framework of realizing many different scenarios. It's an enterprise blockchain, I would say. So you can have like channels with 30 companies which behaves more like a little bit like a decentralized blockchain. And you can have like channels actually with like two companies as well, which behaves more similar to a classical enterprise world. But the situation is even more complicated because you can have like mixed solutions. So just imagine that you have something which is a channel with 30 companies. So you might as well say that works a little bit similar to a decentralized blockchain, but in this channel, you get a private data collection with three companies. So it means that actually in your blockchain, there are systems or there are parts which are more decentralized, having 30 companies on board, but basically there are parts as well which are more similar to an enterprise system. So it's challenging, I would say. Consortium blockchain is challenging and hyperledger fabric can be challenging as well, basically. And you should basically prepare for these challenges if you deliver or run hyperledger fabric. A total different aspect of developing something with like hyperledger fabric or consortium blockchain or blockchain at all. And then, so from a development perspective, the question is usually if you do like classical application development or if you design a cryptographic protocol. And I would say the two things are totally different from each other. So if you do like a classical application development, then you need application developers. You need to set up the business rules and implement these business rules and that's like classical application development. And of course, I mean, you can find a lot of good application developer on the market doing that stuff. But if you design like cryptographic protocols, you need to have cryptographer and basically it's not just the competence, but the project having like a project needed to have like, like designing basically a cryptographic protocol is totally different from an application development project. It's because designing a cryptographic protocol is usually risky. It's more like research or research on the end development. And otherwise again, you need like cryptographer which is a very beautiful, but a very specific profession. And so the point is that like like 10 years ago or the original idea from blockchain was to design kind of a multi-party cryptographic protocol. So it wasn't necessarily the idea at the early stages to do very much classical application development. It was more like, I mean, a play for cryptographer or network professionals with cryptographic background and so on. But lately it goes a little bit, I mean, very much to the application development side. I mean, even with solidity, the language for solidities is pretty similar to like, like Java or JavaScript. And if we speak about like hyperledger fabric, of course it looks that way and you can develop your chain code, like for instance in Java as well. So you might as well imagine that, hey, I mean, we can just have some, a couple of like Java or JavaScript developer on board and then we are done. They can do the development. But the point is this is not necessary too. So if you design blockchain, it's always critical to evaluate where you land. So is it really a purely application development task or do you need to have to deal with like different cryptographic stuff? And usually you need to deal with cryptographic stuff because there are like a lot of certificates in hyperledger fabric. There might be some requirements that need to have like cryptographic background basically. So you might as well say, hey, I got a different database. I don't want to store all of my information in blockchain, but I would say some of my data basically is in a standard database. So then it might come sexually requirement. Let me extend actually the cryptographic guarantee for transactions and for data validity for that database as well. So for instance, if I assign a transaction with my key, I mean, I get the data in my blockchain and I would have to have the same guarantee for my blockchain storage as well. So then it gets actually to go a little bit to the cryptographic world. And that's, yeah, I mean, that's a critical question. So if you start actually with application development in mind, but you need to have like a lot of cryptographic design at the end, then sure it's not gonna be a good idea. So basically if you know at the beginning you need to have cryptography and cryptographers on board, it's gonna be more like a research development project and not necessarily something which should go live in a certain timeline and in a certain budget. So let me just continue with the next slide. And it's again, it's still general, but after that I'm getting to be like more and more specific or technology specific. And this question is what about like with agile development and blockchain? And yeah, I would say agile is very fancy at the moment, but basically the original idea of blockchain is totally the opposite of agile development. It looks that way that, I mean, the original idea for instance, a solidity code was that you deploy once, you can't change. I mean, there are already some workarounds how you can do it, but that's somehow the original idea of thinking. You get one code, it shouldn't be developed in an agile way. It shouldn't be developed practically as a mission critical code. Basically as a piece of code that you put into a satellite which leaves for instance the, I don't know the Mars as an example. So basically that code should work on the long run as well without any change. That's like mission critical or Sandra Santopano says it's like a defensive programming. That's the original idea for blockchain development. But of course, if you have like a small blockchain, of course in an enterprise world, you might as well say, hey, if you have like just three companies, yeah, why not? I mean, three companies can agree basically in something. So if they can agree on a feature request on a daily basis, so why shouldn't we actually develop and deliver or changes on a daily basis? So the idea is basically our proposition for this part is that you can do basically both in a public blockchain world and in a consortium blockchain world as well, agile development basically up to a proof of concept or up to basically a prototype version or up to basically something which is the first version of your code. It looks that way. I mean, in Ethereum, you can practically deploy your code differently. I mean, if you deploy 100 times your code in a daily basis that's gonna be like 100 different smart contract. And it's not a problem until until you don't want to go live with it. So if it's like a demo application or if it's a prototype or a minimum value product that you can do it 100 times, if basically you went live with it and you have some errors in that, that's a problem basically. And this one is pretty similar in a consortium blockchain as well. So the idea is not to have like free enterprises that you have to deal with but let me have like 10 enterprises. So if you want to, you know, I mean make a daily kind of a daily development or daily feature request basically and you have like 10 enterprises, you don't have a chance of affecting these 10 enterprises like on a daily basis. So if you have like consortium blockchain, the idea is to get a similar, have like one prototype blockchain. It's, I mean, it's run by the provider, for instance. And you can deploy on that on a daily basis. You can do as a JLSC one, but basically have some kind of a break. This break should contain basically the final specification. So you can do the final specification based on these interactions on a prototype environment. But basically in this break, have a final specification, have some quality assurance. And if you are in the enterprise world, then have basically a yes that each of these enterprises agree basically on the code and on the functionality and what they see. And then as you start to go live in like Ethereum, it means deploying to a live network. In like Hyperledger Fabric, it means that you don't install for a very limited Hyperledger infrastructure, but you deliver your infrastructure to the enterprises as well. So as you start to deliver actually in production, it won't change very frequently. Or if you want to change very frequently, it's gonna be difficult in enterprise world, you need a lot of coordination usually for that. So that's our proposition in terms of it. Yeah, it looks like we lost Daniel again for just a second. He'll be back as he said earlier and he has some network connectivity issues. While we're waiting again, if you have any questions, comments, feel free to come off mute or type in the chat. So fortunately, that was a break actually as I changed slide. But let me see the next point. That's like Fabric infrastructure and orchestration. Most modern software are designed practically for kind of a micro-surface or as a kind of a containerized application, which basically needs kind of an orchestration below. And that's actually the modern way of delivering software and delivering like a program development. So I got more experience like with Docker swarm, Docker, different Kubernetes versions, OpenShifter and then stuff like that. So there might be some very good orchestration designed for blockchain applications. But my experience is basically that these orchestration and these containerization technologies wasn't or weren't designed for blockchain applications. Wasn't or weren't exactly designed basically for blockchains. And the problem is if you have like a Dockerized environment and you get no orchestration, what you do basically, you get like, I mean, let me just take the example of Kubernetes. What you do, you just have like bots and you can scale up and scale down bots in a very easy way. If you have like OpenShift to get some, you even get some user experience for that and some. But the original idea of like Kubernetes that basically you, if you scale bots, you scale for performance or perhaps for higher availability. But the idea for blockchain is absolutely contradicting with that. So if you scale generally a blockchain node, so if you have like more nodes for instance in a blockchain network, you scale for something which is a security or for decentralization. So practically, if you have a normal application in Kubernetes and you scale up, for instance, like a web server port with like two more instances, it's gonna be faster usually. You can put more nodes on that. If you have blockchain and if you have like, instead of like five nodes, you have like a thousand nodes, then basically your system won't be faster, it will be slower because actually more nodes should validate your transaction. So as I mentioned, blockchain scares for security and decentralization. And it is not something which is, I mean, if you speak somebody who is like a Kubernetes professional, it's not something that they get it right away, for instance. So perhaps for this reason, like increasing number of the bots in like Kubernetes is, it is easy, but if you need to increase your blockchain nodes, it's usually not so easy. So if you like increase the blockchain nodes, you know, I mean, in fabric, you have just to join basically one more peer or one more organization to the network, it's a struggling. But basically it's not just a challenge from a technical point of view. So it's not coincident that it's not exactly the same or the easiest as ETS with the Kubernetes bot. It is actually something different than from scaling a bot. So scaling a bot and joining like another peer or another organization is something two different things. So they shouldn't be basically mixed up or mixed with each other. So that's one thing. The second one is something which is agile development. So installing or upgrading a smart contract. Again, if you have like two organizations, it can be an easy task, but if you have like 30 organizations and they want to agree or they should agree before installing anything, or I mean, depending on your endorsement policy, they shouldn't necessarily just agree, but basically they should actually sign the transaction as well. It's not something which is a daily activity. For this reason, actually integrating stuff with CICD pipelines, they are not easy from a technical point of view. And hence I would say it just doesn't necessarily have make sense basically doing such a thing. So again, the major message, I mean, even most state of the art orchestration technologies were not designed for consortium blockchain infrastructure. Just an example. So for instance, the idea is that you get like a lot of pots and all of these pots basically have persistent storage in a way that the storage basically is dependent with each other. It's basically you don't have like independent pots storing something, but you have a lot of pots storing basically more or less the same data, which is even connected with like a lot of cryptography as well. It's not something for that basically Kubernetes was designed for. And then so if you know actually a much better orchestration technology, which is designed for blockchain, for consortium blockchain, then yeah, please share it. But I'm skeptical if you've heard this word is already there. So let me just continue basically with more fabric elements and that's basically fabric and privacy. So in terms of fabric and privacy, so you got a lot of things that you can use basically and these things are sometimes, yeah, I would say with like contradicting with each other or not easy to use all of them. So you get like channels. Channel is a good idea, but basically you can use private data collections if you want to make data more private. But you have to pay attention basically because even if you use like private data collection, you should consider using transient field or veto transient field because with that you get like a little bit like different piece of privacy then you can have like organizational private data collection, for instance, and you can even have a lot of like design patterns using organizational private data collection in the fabric documentation and stuff like that. Then there's the identity mixer, which is more of the zero-modage proof and verifiable credential based approach. Unfortunately, it's not really full-scale. As far as I know, you can hide like the signing identity of a people or of person in one organization, but you can't really scale this up like for hiding the organizational identity and even if that's somehow not really productive ready. So for instance, the revoking of certificates is not implemented as far as I know, for instance. Then you can use like trusted execution environments. Of course, I mean need some hardware plugins and of course you can design like privacy construct, self-developed privacy construct as well. And of course it's a question is which should be used, which shouldn't be used. I mean, if you use everything, then things gonna be more complicated, that's for sure. And not just with installing stuff or at delivering your network, but basically at maintaining and operating your network as well. So it's just one proposition. You perhaps have something which is better, but one high-level idea is that to try to match all the privacy requirements with the existing privacy structure. So I get the basis, this table. We got data, for instance, data one, data two or information one, information two. Again, it's very much high-level. And then we can have different strengths of privacy as well. So for instance, probably the strongest level of privacy is physical privacy. The second level is like strong privacy as an example of encryption. We might as well have here like another column, it's like strong, strong post quantum privacy, which still perhaps doesn't really exist or at least not in practical application, but it's coming. So we might as well strong privacy and even strong post quantum or in the future strong post quantum privacy as well. Then we can have like weak privacy, like pseudonyms and for instance, no cryptographic privacy, like we do something with different policies, for instance. And so what I usually propose is to match somehow the requirements with the possibilities. So we got here the R, R is the requirement. So what should be required for data one, for data two, for information one, for information two, and so on and so on. And on the possibility of Cypher Azure Fabric, I got like more categories. So you can do something which is F, it's called fabric support out of the box. So if you need like physical privacy for data one, you can have a different channels, you can do it for instance with fabric out of the box. You can have something which is with fabric, but fabric doesn't support out of the box, but it needs some custom development. This is like I, and then I got R and D at the end, which means fabric doesn't support out of the box and it might, it doesn't even need like development, it might need some research and development as well. Sorry, this is a different category because if you have like R and D in the project again, you can't, usually you can't do it in timeline and in budget. And what's the proposition is just somehow to match like even with customer the requirements with the possibility of what fabric can provide. So saying like, hey, I mean, if you need like for this data too, you need a strong privacy, there's no strong privacy guarantee in a cryptographic sense out of the box in Hyperledger Fabric. So it means we can do some development or some research on development, but it's gonna be like more free from a budget and from a timing perspective. So that's one way of thinking it. I mean, usually we started sometimes a project in a way that, hey, let me just use all of the privacy elements from Hyperledger Fabric, but it's not the best idea because I mean, even if these elements are there, it again, if it increases the complexity of your application, then you will have problems if the customer is not prepared for the complexity for instance. So it's usually just a better way, just somehow to match the privacy requirements with the possibilities of what Fabric can provide you. Basically, something similar is the same. If I don't say privacy, but like cryptographic guarantee. So for instance, there's like a cryptographic guarantee that like a data can be changed with the help of a transaction. And then basically that transaction can be signed by somebody. So basically in some sense, there's a cryptographic guarantee that this data cannot be changed by an unauthorized person. That's like the basic idea of Fabric. But if you just bring in the requirements, perhaps you can't give the same cryptographic guarantee for like data consistency for all of your data or for all of your requirements. So it's again, it's just the best ways to like start with kind of a similar table and then have an idea which is the data that really requires like cryptographic guarantee and which is the data which doesn't. And even if it's that cryptographic guarantee, again, there are many ways of cryptographic guarantee. So you can say even if that's a digital signature, you can at least distinguish like classical digital signature or perhaps quantum resistant digital signatures as well. So the next big topic is designing for performance and high availability. And then, yeah, so I saw like here many issues here. So for the first one, so really designing for performance, I found it's tricky. So basically your blockchain performance depends a lot of things. I mean, most critical is your chain code logic and your read write sets and the level of consistency, that's very critical. If you want to read out data from the blockchain then coach DB performance hacking there will be very much dependent on the orchestration or perhaps multi-orchestration. So if you consider deploying your network to multiply organizations, then you might not have just one orchestration technology, but perhaps several ones. And of course, your performance very much dependent on your network, which is again, if you have like multiply company, it's not just one network, it's several network connections. And the storage as well, which is again might be differing in different companies. So usually what I see basically a practical way of designing for performance is more or less you build your system, at least in a prototype space, but basically with your final chain code logic and with your final infrastructure setup and you make some measurements, that's the easiest way. I'm not quite sure if there's any more practical idea. You can have like, there are tools for this like Hyperledger Caliper, but it can do some simulation, but it can't really simulate, you know, like multi-organizational, multi-network, multi-storage, multi-orchestration setup. It's not something which Caliper does. It's here to make some idea how the system performance a priori can be designed or can be developed. So I mean, of course, one way of doing is like developing your system, rolling it out and then measuring performance, but usually that's not the best idea, that's not necessarily best practice. It's much better if you can make some performance assumptions in the design phase or in the developing phase and not just after rolling out the system. There might be some questions with high availability. I mean, the question is, what is high availability at all? Here, basically, you can somehow consider like high availability in one organization, for instance. Then here, it's again a question. So there's the fabric high availability. So you can say, hey, high availability in fabric sense, yeah, we get like two peers or three peers in one organization and that's highly available. But unfortunately, usually that's not the case because you should consider like high availability in the underlying layer. So how does your, I don't know, docker swarm infrastructure look like or how is your infrastructure below your docker swarm? So if you have like multiply Kubernetes clusters, for instance, or if you have one storage below the whole system or one network, for instance, and so on and so on. So usually just designing organization for fabric high availability is not enough. Practically, you should actually design the whole infrastructure below to have like real high availability. And there are some tricks, for instance, like, I mean, even if you have like several peers, like my experience at Discovery, if it's like this Node.js library as Discovery for the first run, it might drops actually transactions if like one peers goes down. So there are some fabric specialties as well. Another thing to consider is like multi-organizational high availability. It's again, it's a little bit different like from one organizational. It's the question, what happens if one organization goes down? Is there like critical organization in the system from an availability perspective or not? And it's something which can be like designed with the help of endorsement and order errors. So endorsement policies and order errors, so it's like more fabric oriented. And of course, you might as well consider like crash fault or Byzantine faults, but I'm not quite sure if that makes sense in an endorsement policy, but like if you design your ordering service, of course, the easiest is rough, but it's just makes to crash fault tolerance and it doesn't give you like Byzantine fault tolerance. And then some factors that can be considered is like data availability, like replication factor. So if you have like a piece of data, how many times is it replicated in your organization and how many times is it replicated like among the organizations? So these are some considerations for high availability. And I would say sometimes again, just to design system for that is tricky and it doesn't only involve like designing your high-paragile fabric. So the next big topic is something which I say like multi-party enterprise operations. And then, so it looks that way. I mean, let me assume that, hey, we just installed our consortium blockchain network. It's running, it looks cool. Everything works fine, but usually this network should be a little longer. So the question is how much coordination, how much agreement do you need for operating this network? I mean, let's just imagine you get like, I don't know, 20 different companies. Even organizing a call between these 20 companies is tricky. So it's a lot of coordination, basically a lot of coordination effort and a lot of agreement is needed if there should be like a decision carried out. So basically the question is if there's anything or if there's any best practice or ideas for something which is a multi-party governance or multi-party enterprise governance or multi-party enterprise operation. And I usually mean like the IT staffs. So let me just start from the scratch that we just have the idea. We have to change something. You know, I mean, there's like, I don't know, containers on Dockers one and then they're running, they're running our whole network and then something should be configured on the network. So then, for instance, the question is what should be configured and the level of coordination and then remember these, for instance, for that configurable item. So I start to use like a little bit like IT words, but like if you just say, we got one configurable item and then this table for instance on the right, I just try to have like an idea that we can have different configurable items and for each such configurable item, it's an interesting question. If changing that item needs like a coordination of the parties or an agreement of the parties, or is it something that can be basically changed by one company? And there are many things. So for instance, if I have like a container property, an environment variable, which describes my log level, for instance, on my local fabric node, then it's something that I can basically change on my own. So I don't need like coordination for that. But like if I need to have like, I don't know, if I need to have like an administration, basically like I would like to change the batch time of my application channel, then basically it's like a channel property change. It requires basically an organization agreement, at least a majority. I mean, it depends of course on your enrollment, but basically you need endorsement, sorry, but you need probably an organizational agreement on all of the organizations that is sitting in this channel. So it's certainly kind of change, which is more complicated and you should prepare for this additional complexity. Then if you need to like change, if you have like Fabric 2.2 and you get like system channel, and basically you want to like change a parameter in your system channel, it requires probably that all organization I must agree basically. So that's one thing. The other thing is how can such a change be executed in a multi-party or in a possibly in an automated way? And then, I mean, of course, on a low level, Fabric provides some possibilities, but if you just want to somehow automate this whole thing, it's not necessarily gonna be easy. One other thing which is perhaps like classic thing, it's not so fancy, it's CMDB, that's change management database. So if you like change one piece of anything on your full network, it's a good idea to administrate what you changed. And then it might be surprising because like in classical Kubernetes, for instance, you get like the pod log. So pod log describes basically pretty much what change on that log. But if you consider like, you know, I mean a consortium setup with Hyperledger Fabric, then perhaps some of the administration transactions are basically transactions that are executed somehow. So it's gonna be such a change which is very, very relevant. So which is actually a very high impact change and it will be executed in a way that it won't be reflected in your Kubernetes pod log, for instance. So it's a question is, where's your CMDB? How, where do you actually just register log basic your changes? And then how do you do it in a multi-party, in a multi-organizational way? So sometimes, you know, I mean, I changed something. I mean, the whole system or part of the system got screwed. Let me just try to figure out what happened. Perhaps it wasn't only because of my change. So it might be a good idea to, to consider CMDB a change management database in a multi-organizational way. So similar things are like multi-party logging. From a technical point of view, you get very, very good tools for logging, like in Kubernetes or in Docker. You can have the elastic search and keep on the stuff, for instance. But these texts were designed for individual companies, I would say. And usually, like if you want to monitor your log, basically your whole network, then in most of the cases, like for debugging one error, you need to look from several companies. And of course, you might say, hey, let's just centralize it, let me just have one server in the middle. And then everybody just sends the logs there. But it's usually not, it's usually not accepted. It's because it's usually not accepted. So it's like, if one company sees the logo of another company in the network, then even if that log basically doesn't contain very private information, even if the fact that like one company knows that another company does something at the moment, it might leak actually a private information, which is not accepted. So again, multi-party logging, similar for multi-party monitoring or ticketing. So there's very good solutions for monitoring your fabric networks, like you can do it like Prometheus, Grafono and stuff like that. But these tools were designed practically on for individual companies. So what happens if you need to react for different things or monitor different things basically on a multi-company perspective, if you need like just handling ticket from a multi-company perspective, you might as well say, hey, let me just centralize the monitoring and ticketing, but it's again, it's just usually a logo basically. So one last point, how can fabric support such operation? I mean, there are some use cases like deep technical for upgrading or installing chain code. You can find some lapses where like fabric operator or operations, hyper-ledger labs or operation smart contract. So there are some initiatives for that, but I would say, so this is a big topic. And I think most of the problems here, even if there's some initiatives still partly unresolved. So that we have my, yeah, actually that's one of my favorite topic, which is backup recovery. So, I mean, and then I'm just gonna cover very much, you know, roughly, I mean, what is back recovery at all in a consortium blockchain network on a very high level. So again, we can start with classical systems, classical systems in classical enterprise system. We got like persistent data storages. These are backed up and restored. That's like backup recovery on a high level. So what is backup recovery on a public blockchain? So in a public blockchain, you usually don't have backup recovery. There's a big network. We assume that the whole network won't go down. So we assume that all the 8,000 Ethereum nodes will not go down. If your node goes down, what you can do basically, you can bring up your node and you can re-synchronize with the network. You can even do something which is a snapshot, but usually snapshot means that you don't synchronize your node from scratch, but from basically from something which is half re-synchronized. So that's like public blockchain. It's not really something which is the same as in a classical enterprise backup recovery. So what's the situation with consortium blockchain? Well, I mean, it's a good... Okay Daniel, we lost you. Yeah, sorry. Shall I have just a brief? You still hear me? Yeah, we can hear you. From consortium blockchain, we lost you. Okay, okay. Ah, sorry. It's again, it's my network. My network is not highly available. So the question is, what is backup recovery in a consortium blockchain scenario? And then you might as well say, hey, it is something similar to a public blockchain. We get one channel, 20 companies, and then basically we assume not all companies gonna go down. The data will be somewhere. End of story. We assume it's similar to a public blockchain. But we can say, hey, we got just two companies in a channel. So basically it might happen that both companies will be somehow corrupted. So we should backup our system somehow. And then things are more complicated with hyperledger fabric because you can have like things that you get one channel, like with five companies, but you get the private data collection and the data in private collection is shared just by two companies. And then the question is, which is backup recovery, what should be considered? I mean, one way of starting this stuff is like a little bit like considering, again, I will not have like answers. But one idea might be is just to access basically each relevant piece of data kind of a replication factor. And then replication factor might be, I mean, you can even replicate data in one organization. So if you have like three peers or four peers in one organization, that means basically that your data is replicated four times in your organization. And then you can consider something as data replication across organizations. So if you get like one channel with three companies, then practically your data is three times replicated in terms of organizational replication. And then you might as well consider if you need like kind of a classical backup of your whole system, or you might assume that, yeah, so the data is like, I don't know, two times replicated in an organization and three times replicating at cross organization, then we assume that not every single will go down basically. So it's a question a little bit like data replication factor. And then again, the problem is you get like fabric data structures. So if you have like just channels, the situation should be easier, but you get something as well as private data collection. And then even if you are in a channel, then you get much less replication factor with the private data collection in terms of organizational replication. And you can have something as well, which is organizational private data collection. Actually it is used anyway, when you install like chain code, for instance, and if you have like organizational private data collection, it is replicated only in your company. So it's a question if you have to somehow backup, make a backup and restore stuffs. And things are more complicated because these things are not independent from each other, but they are connected with like practically cryptography. If you're unlucky and if you have like probably 2.2 you have even a system channel, which makes things more complicated. And again, so what's important, these are not necessarily like independent things, but these are usually kind of dependent with different like hatches. And then again, it's cryptography that's basically a whole distributed authenticated data structure. Authenticated data structure means that individual data or pieces of information of your data structure are connected with cryptography practically. So again, that's backup recovery. I got one more slide. Basically this is like, I just covered very fast. It's like maintaining your cryptographic material. And sorry, if I say cryptographic material, I don't just mean the crypto config, but you should consider basically that there are a lot of cryptographic elements in the chain for the first time like keys and certificates, but perhaps on the long run, like zero-knowledge proof stock might be coming actually in the future. I mean, you already have like zero-knowledge proof and verifiable credentialing fabric if you use like identity mixer, but as this whole field is being developed further, you might as well have like much more such structures, such cryptographic structures in your blockchain network. And the problem is that, I mean, one thing to install these stuffs, I mean, even installing these stuffs, is not necessarily simple, but it's more complicated and more challenging and this complexity is usually underestimated, very much underestimated, maintaining this cryptographic. So depending on, I mean, every piece of cryptographic material that you somehow install must be maintained actually, like in perhaps not daily or weekly, but at least in yearly operation. So for instance, if you get like just the simplest that like X509 certificates, which exists at the moment with fabric, there's a life cycle for that, just create and roll, suspend is not supported at the moment and revoke. And this life cycle should be supported actually for the different X509 certificates, which are, I mean, which are differently created usually. I mean, the basic idea is similar, but usually you create them in a different way, so you have two little different way, and then you get like signing certs, admin certs, user certs, these are usually E-certs. If things are getting more complicated, you get your own CAs, you get fabric CAs in the system, so you can have root CAs, you can have more CAs in the system, so you can have like sub-CAs. If things are even more complicated, you don't have even like fabric CAs, but something similar. So even practically with like the easiest stuff, which is the more most classical, I mean, cryptographic material, that's like X509 certificate, things are different. So I'm not quite sure if there's like tool for, I mean, really for monitoring your certificates in fabric and to like rolling your certificate just by clicking something. If anybody knows such a tool, I'm very happy to, please share it because I think I mean, I mean, the whole community gonna love it. So it's again, we are somewhere here, if we do like application development, like with maintaining, operating, application and the infrastructure, that's the key device of development, and we can have something which is, which is more like designing a cryptographic protocol. It's basically maintaining this cryptographic protocol, which is just as operation, as maintaining task, it can be very, it can be challenging and it is usually very much underestimated. So that was basically the points that I listed for fabric challenges or consortium blockchain challenges. And then the question is, I mean, perhaps just my point of view, I'm not quite sure, but if you share my opinion, community contribution or further community idea, again, so if you say, hey, what I listed here, you know an answer and these topics are not important, please share with me, I'm gonna be very happy. So I mean, seriously. But if you find that these topics might be challenging for you as well, or if you saw like applications where these points are challenging, then I would propose to start kind of a community contribution on that. And my proposition would be to design something similar as ITIL, that's information technology, infrastructure library. And I mean the original idea of ITIL. So original idea was just to summarize problems, solutions, best practices and tools for operating basically an enterprise software in your company. That was the original idea, like with service operation, service design and service transition. I mean, recently it has became like a philosophy, so it's more philosophical, but I mean, I would just have the original idea of ITIL just from the original practical side. So I would start actually summarizing something similar, systematically collecting and analyzing. And basically only for consortium blockchains, we can start hyper ledger fabric as well, possible solutions, possible problems with consortium blockchain and hyper ledger fabric, possible solutions for that, best practices, something somewhere has some experience with that and it got some things which are anti pattern, so rather don't do it, staffs. And of course tools, we got a lot of tools basically, but these tools usually focus on one or two pain points of this whole consortium operation. So it would perhaps be a good idea just to summarize tools as well, in terms of a tool, I don't know, source basically the certificate renewal process in hyper ledger fabric, but won't do anything with like multi-party coordination of different companies and so on and so on. And again, so the idea would be just to design, just this one for consortium blockchains and then reaching somehow the major lifecycle items, so at like designing, planning, rolling, installing, maintaining and operating consortium blockchain networks and basically hyper ledger fabric. You can find basically a document here as well, this is like an extended version of my presentation, a little bit like summarized in text as well, so I can likely share it. And then basically that was my presentation. I just a little bit underestimated the time that was needed, but then I would say, I would stop my monologue and then again, we can start like discussing and then if you have like any experience sharing on any of these topics, then just please feel free to share it. Hi, this is Alex, I just wanted to maybe bring two topics that maybe could fit at the beginning on your list that you had for problem statements. One would be related to the balancing of data between what's on the blockchain in fabric versus off chain, because from our experience, very few customers actually build UIs or applications on top of, for example, CouchDB or even talk about level DB, I think for that. So most people actually have to sync the data from the ledger to some off chain, right? The relation database typically, or for that matter, no SQL or whatever it might be. But then the question comes, how do we ensure that we don't have gaps in data between what's in the ledger versus what's off chain? And from our experience, we've seen very little about that in terms of best practices and how to do that type of data verification. Yeah, one aspect. That's a very good point, that's a very good point. So one pain point that I saw as well, if you store off chain data and off chain data, not in terms of like a sync private data collection, but like in a totally different database, then the question is, so basically the point is, if you store data totally off chain, you expect indirectly the same guarantee that blockchain does. So you would expect that basically this data is consistent with the blockchain data, which is not necessary true, because if you like writing to like two storages, I mean blockchain, some transaction data warranty in the blockchain, but it doesn't give you guarantee with some other data source. So the question is with an existing storage solution. And if so, yeah, that's a problem. That's not an easy problem anyway. So that's one aspect that I saw. The second one, in blockchain it is, I mean in hyperledger fabric, it is guaranteed that the access of your data is basically driven by digital signatures. So if you don't have the keys, you can't sign. I mean, there might be some exceptions, but generally if you don't have the keys and you can't sign transactions, you can't change the data. And as soon as you bring data outside of hyperledger fabric, you will lose this guarantee. So it is a question if that's a problem from a customer perspective or not, but certainly it's something which is, I mean, it is overseen. So we just say, hey, we can store data in the blockchain that we just bring outside of the blockchain. Sure, but then we don't have the same guarantees. We don't have the same cryptographic guarantees, but blockchain does in terms of accessing data in terms of digital signature and perhaps not even like consistency. Yeah, absolutely. Yeah, but then the second one when we've seen questions coming up often is, okay, you have these blocks on the chain, but what actually goes in a block, right? I mean, we see actually transactions happening, going into, let's say, possibly multiple transactions go in a block, but what's actually the business records that make up those blocks? So we really don't see much in terms of tooling and either like with the old Explorer, it would be really nice if over time there is a facility through which, okay, I look at an Explorer, I see the block numbers, but when I click on a block number, for example, I can actually see those, I don't know, those JSON messages or whatever got stored right into the world state and I can see what made up that block, because I think to me that visibility is really not there yet and it's just a little bit of a black box. It just in terms of, and I think ties a little bit back to with the balancing question, but in any case, that was the second point. Yeah, I mean, I totally agree. I faced actually the situation in like, so like Explorer is more like, I would say something for developers sometimes or professionals. So like having hyperlegion, I mean, I mean, there are two problems with Explorers. One problem is that it's end of line. End of line, sorry. It's not developed further, which raises some serious problems if you need like the latest, you know, I mean, Docker image and so on, but otherwise it's not something which is for end users, I would say. So perhaps even end users or users with not so much, you know, I mean, high professional background would like to see something from the network and then perhaps hyperlegion Explorer is not the best tool for this. Yeah, I mean, that's my experience with Explorer. Daniel, I mean, I have actually some news I can share about Explorer. So it was end of life as a project, but it has recently been restarted as a lab. I just dropped the link to it on the Zoom chat. There have been at least five or six, if not more organizations that showed up after it was end of life and said, hey, I was using it and I'd still be interested in developing it. So that includes Walmart and some other organizations. So if there's interest, they're getting together actually tomorrow and I dropped the link to that as well to talk about what the roadmap is. And so if there's interest from people and restarting, exploring, kind of taking it maybe in a new direction, just be aware that that's something that there are a number of community members currently looking into. Thanks, that's interesting. And that's an open call tomorrow. The details are on that link. Anybody's welcome to join. Yeah, thanks for that update. I mean, it's very helpful for me, absolutely. So again, I mean, if you have like any ideas or any experience in terms of blockchain, consortium blockchain challenges, just feel free to share. If you have like, I mean, if you have like experience or tools for solving any of my problems that I listed, I'm very happy to hear that. So no problem, just feel free to unmute yourself and share. Otherwise, I just try to read the chat as well. So you can share in the chat as well if you feel like. Hi, my name is Zachary Jones. I'll share something briefly here. It was mentioned about referencing things that were off-chain data, whatnot. It's kind of an explorer concept. We had built a solution that involved a lot of different types of off-chain data. We were synchronizing things related to private equity and we had many different types of companies that were a part of the process and not all of them were necessarily represented on-chain as entities, but we still had to have record of the data that occurred in those systems. We just developed a signing method for the data that was in those external systems. It was a mixture of the URL and a kind of unique reference, including the daytime that something had changed in the external system. Almost like a Git commit hash. And we had developed that for some of these systems and then would commit that to the chain. And along with the URL that was actually linking to those external systems, it made it easier to be certain whether or not a particular transaction on the chain related to a particular state in the external system, because it kind of warrantied the exact, daytime stamp of the data that was there. So that's just one strategy that people are heading down the kind of direction if they have a lot of data off-chain. The rest of what we did was utilizing private data and we actually used some of the on-chain with the hyperledger native versions of private data. But then as well, we had some data that we had to store in a different type of database that was coordinated between parties. And we developed our own method that was kind of a duplication of the process for storing private data. But we again signed data that was stored in that other database and then committed that signature to a transaction as part of a block so that there was confirmation of whether or not that data was changing outside of the scope of the hyperledger network. That sounds good. Just let me have just one tricky question. Were there any problems with like, I mean, did you consider like backup and restore between like two, between fabric data and off-chain data? Or it wasn't a problem because you just practically used for reading all data from the blockchain. We had a little bit of a pass on that complexity because there was one primary node or entity in the network that was us that was coordinating most of the other activity for the other entities. So entities were only pushing data back into the chain through us, maybe not centrally since I mean, they were, entities had to have a sign off whenever they pushed certain kinds of data that would fall under the scope of what we would have needed to a backup. And so since we were kind of the, maybe not the moderator, but just under the facilitator of all of these parties interacting, it made it a bit easier for us to be a global walk, so to speak, against a backup process that would force synchronization to other entities as well. Thanks, awesome, that sounds great. Yeah, so again, the few topics like relevant, that would be, I mean, few of these topics like relevant, that would be somehow my idea is that let me just summarize, I don't know, perhaps by hyperledger, there's some either in a documentation or basically perhaps as a hyperledger lab, although that's not called, it's just, you know, I mean ideas, summarized ideas. Again, kind of different problems possible solutions, best practices to do, not to do. And of course, tools, I mean, it would be very interesting. I mean, as far as I know, we already have like a tool list, it's just or a list of tools for hyperledger fabric. It just sometimes it is not clear which tool solves which problem. But anyway, again, so if you have like ideas, suggestions, or perhaps basically experience to share, just feel free to unmute yourself and then just go ahead. So like one point that was raised in the chat, like, sorry, just, yeah, just go ahead if you started to say something, sorry if I just interrupted you. Yeah, if nobody shares it, it's okay, like one actually question is like, which is the orchestration technology that you propose like for running hyperledger fabric. It's again, I mean, it is Kubernetes or different Kubernetes version, which is mostly used or basically which that's the technology that enterprises start to use, I would say. But again, my experience that it's not really for consortium, not really optimized for consortium blockchain. I mean, even if you got like, even if your fabric infrastructure is based on container, the logic behind Kubernetes is not exactly the same as the logic behind the blockchain at work. So if you happen to know or if you happen to, if you can start just like other orchestration technology which is like more optimal for hyperledger fabric, just feel free to share. Hi there. Hi. Can you just print, paste this last link on the slide in the chat, consortium blockchain networks that run? This one. Yes, which is there? Yes, absolutely. Anyway, I mean, this whole section is recorded and then after the call, perhaps tomorrow or not today, but we're gonna share the slides as well. So basically you will get everything. Thank you. Thank you. It's again, this is just a document. It's like a little bit like more detailed summary basically of my presentation. So it's still not really the structure that I proposed here. But yeah, I mean, perhaps we can get something similar as well. Sure, thank you. So there's one question, is if there's a preferred cloud platform to host hyperledger, I don't know, perhaps somebody? Probably the platform matters more than the tooling that you have around it, right? I don't know if the question is about SaaS solutions for hyperledger or just cloud providers from our experience, it doesn't matter if it's Azure or if it's GCP or AWS. Yeah, there's one interesting aspect. Looks like we may have lost Daniel again for a second, but he'll be back. Sorry, I was just reconnected. So yeah, so the idea is somehow that that if you multiply enterprise, they will use like different orchestration technology or perhaps even different container technologies and then perhaps even different orchestration technologies. And that's again a pretty big complexity that might come actually in the project. So I don't know, is there like any more points, experience that you want to share basically in this call? Because I would say if there's no more points, then I would just close this call. Again, so the idea is somehow that we're gonna have like subsequent such meetups. I think there are like two pieces in the pipeline, scheduled somehow for next year, where not so much on a high level positioning, but I mean really on different tools and different solutions or different solutions for different problems and different tool supports. So perhaps they can give like more answers and more details. So it's like one thing that will be truly presented, one thing that will be truly presented that's like February operator and February operator will solve some of the issues, some of the operational issues, I think not all, but perhaps some will be solved by February operator. And there's gonna be like a meetup and presentation on that practically next year. One last question here that, so Hyperlegia and DeFi. So basically Hyperlegia, I mean it's the question, Hyperlegia is the umbrella from Linux Foundation, but if it's a question of Hyperlegia Fabric and DeFi, there's not much DeFi application on Hyperlegia Fabric, it's not much crypto, it's more like a consortium ledger, but for instance in the Hyperlegia project, you find like Hyperlegia Bezu, Bezu is an Ethereum node and then it can participate in public networks as well. And so that practically you can run DeFi application on that if you want, it's again, it's the question, if that's like Hyperlegia Fabric or Bezu, if you take Bezu, then it's solidly debased, can participate even in public networks so there's some correlation or combination of Hyperlegia Bezu and DeFi application. If that's the question, Fabric is still not much there, I'm not quite sure if like running DeFi application in a consortium network makes sense at all, but like tokenization makes sense for sure and then there are some tokenization initiatives from Hyperlegia Fabric, you can find in Fabric samples like the three major types of tokens, like fungible, non-fungible and hybrid tokens, kind of ERC 20, 721 and 1155 style tokens. So I would say our time is slowly running up. So that was basically my presentation and thank you very much for the participation and thank you very much for the discussion at the end. And then basically, yeah, I mean, just let we see some more points in details next year. That's great, thank you, Daniel. And if you want to sit, I see that some people asked about slides. If you want to send me the slides, I can send it out to everybody who registered. Sure, I will send it after the call. Great, thanks. And thanks, everyone. Thank you, everyone. Bye. Thank you.