 Hey, we'll be starting in a while and I know this is the last session of I mean of this region and it's been like There were so many interesting sessions so far I know you are like oh one more session and then no boring session We we have put together an interesting site of slide deck in this presentation In fact, this started as like conversation with with anina, right? So we were discussing on some of the use cases like I was asking questions on hey How does it solve from academic word and then there were questions back on how can we implement this and we both side together and put our Tarts on on the slide deck and that's what we're going to present in today's session Oh to you and you know Yeah, so can you share can you see my screen? Yes? Okay, so yeah So Okay Yeah, so in today's session as I said we are going to discuss about blockchain design patterns where we will be putting our thoughts together from both Academy of Word and you know can you download the slide they can can I share the screen? No, I will download it. No worries Okay, so meanwhile We'll be presenting the topic of blockchain design patterns where we'll be discussing Advantages and disadvantages of some of the commonly known patterns and then from academic point of view what really works and what doesn't work and where is that? research gap that like we still need to Proceed in it, right? So these are some of the hard questions that we would try to address in today's session and For that with you today We have like anina Who's assistant professor at Department of Computer Science and Engineering from St. Gates College of Engineering and I'm our own working as top stopper engineer at Walmart global tech and a hyper leisure TSE member So a little about the context of our talk as I don't mention I don't and I have brainstormed and put together our thoughts on common design patterns used in blockchain application development So it's actually an attempt on stitching together our thoughts from both the perspectives so we actually really value our time and hope you can walk away with a lot of good information and That being said we have tremendous amount of information to look at today And we will be talking about common design patterns and challenges we will be taking a deep dive into on-chain and off-chain world data and interactions and The other area that we are going to touch is data versus network earnings And yeah, we have a lot to go through and hand this off to Arun and he will be discussing the common patterns used in application development Right. I'm sure most of us have come from application development background who We have been developing multiple applications, right? So whenever we think about web application, we will have a database Stored in our enterprise systems and then we will build back in engine It could be microservice architecture, which is talking to these database databases and when we talk about database again It could be like a single instance of database which is monolithic Storage or it could be again distributed database storing maybe shards of data or maybe distributed across multiple regions for different optimizations applied for multiple geolocation's and then we will have kind of typical user interface which is talking to these back-end services and Which is finally accessed by the user, but does this really work in blockchain case, right? So that's the question and even in in in general web applications We have something called as like we do implement authentication and authorization mechanism at the back-end API slayers and We relay those requests or network gateways But when we apply that to blockchain technology, right is and many people if if you're like unfamiliar with it Blockchain does provide storage and if you consider blockchain itself as a storage engine Then you try to directly replace What we have in our traditional web application with blockchain that may really not work because When we send the right request, right? And let's take a typical database application when I send the right request those right request would be And then can you go back to previous slide? Let's let's take a typical application right when we send the right request to database Those right requests will not be committed until we say commit this particular data to a database application So similarly if you consider blockchain operations, you can consider like transaction execution or that particular phase where blockchain decides whether to proceed process the transaction or not to be first phase and then the commit phase where in Term if you have to talk in terms of hyper ledger fabric That's when ordering service comes together and it gets involved and creates the block So if the block gets accepted, that's the commit phase for us Hey, so this is how we can coordinate DB right versus a blockchain right operation and The smart contract is is the one which is going to perform the crowd operations onto database But we are really not sure how long it takes for that particular block to get created And how long that that particular operation to be completed because transaction execution happens at its own phase We cannot really guarantee and then having this kind of application design is it's not generally preferred We cannot give a deterministic response to user with it And of course circuit breaker logics doesn't really work with these applications Yeah, um, so now let's look at and alternate to that So we said we will not be able to design our blockchain application as if it is a database system Now what what can we improve over it? Let's say we we bring in um Like for every request we bring in immediate response saying that Okay, now am I this is my back end up back end engine, which is also acting as a blockchain client node for my actual blockchain applications over there As soon as I receive a request I respond back saying that okay, I received the request So tell it to user and tell the user that I really don't know the status So you keep polling for the status So with this approach the problem is we may end up having multiple poll requests to be signed a blockchain node It not only affects the blockchain node in terms of long wait Which we have to do But it also has its own effect that Smart contract may never get executed or may Continue in the in the loop of failures, right? So until until that point we will not be able to Keep users waiting if this was my payment application based on blockchain I cannot when user says pay this thing to somebody else. I cannot make them wait forever and also Polling mechanism on blockchain actually like it overloads the re-request on blockchain So probably we need something else like there's there are better alternatives to design applications Now let's look into another approach So in this approach what we have done is Put a cache database in between blockchain and our client node Which is also our backend application And in addition to that we have something called as event listener. So what this event listener does is Any blockchain you take fabric or sawtooth or base or anything There is a concept of transaction receipt or like event It at multiple layers. It could be at block level. It could be at commit level or it could be at Channel level in fabric. So what we can do is we can ask blockchain system itself. Hey I send you a transaction you tell me when it is completed or you keep telling me information Whatever is happening within you and based on that I'll make my decisions. I'll I'll do something at the application layer So that's what this system indicates and what does cache db used for so cache db in here is like When I receive a transaction, I need to first store it somewhere so that I don't lose my data So I let's say I have some temporary storage. It could be uh It could also be like a pops up topic like like a kafka engine which which can store data for three days or Five days or ten days, right? So Let's say I cash all the requests and I stream them from this blockchain client into blockchain At its own phase and whenever I receive data from the blockchain. I tell it to user Hey, here is the current status of whatever you did earlier But probably this is also not Not sufficient. This is good, but this is not sufficient model for us What if user wants to query what happened with their data in the past? Like there is no way that because we are just streaming requests Streaming solutions from blockchain and giving it back to user. There is no way we can know what happened in the past So that's where we have our next architecture Pattern which is like in addition to having Event listener we what if we store the event listener data into another database Now there are multiple advantages here And of course like back in the engine can be used to perform retry operations on the failing data It can know what what failed and what not failed based on the event listeners data and in addition to that This I mean I'm calling it as analytics data over there, but the reason for calling that is When we receive events from blockchain, it's not just the data which we are reading from our own transactions And as we know like blockchain is decentralized application where transactions could be flowing in from multiple parties So this database is actually data collected from all over like every other different parties who are involved over there So with this approach, we have Local cache of information as and when transactions arrive. We are caching them We are sending them to blockchain We are waiting for blockchain commit operations to complete And then we are also having analytics database and then a query mechanism or Repurposing that analytics database to perform any other operations that we require right now One more thing which I would like to call out here is that in in terms of that cache database in in in the diagram with the If you consider that cache database If you replace that with some kind of permanent storage, we can also use this design pattern as Off-chain storage of our data. I don't need to put everything onto my blockchain I can keep them into my storage like replace cache db with a permanent database Put your contents into that database and store or move only the data which you want provenance for onto blockchain Now this particular half-chain data trust which which you see on the screen Is because let's say we have all this complicated structure or so-called complicated structure We are waiting for blockchain commit to complete and then we are storing it in a in an external database before we process it for analytics applications So how can I trust because this data which I store outside the blockchain is still entrusted data, right? I mean even though it is coming in from blockchain I need a way to guarantee and for some of the Applications if this databases are in protected environment because it is for their own usage We really don't need anything extra. That's all we need But on few cases we may need to audit the data which is stored over there So that's where we can bring in additional component Let's say as scheduled data verifier, which I'm calling it or here But in reality it is it is an audit module So what exactly are we auditing here? Let's say you randomly pulled some data from your analytics data or whatever you read from blockchain And then you try to query your blockchain in your leisure time when your payload or when your throughput requirements are low and whenever your blockchain node is capable enough to answer all those sample data queries And audit them through this module. So with this approach I mean as soon as you identify there is some screen There is some change in data against whatever you stored off chain You will of course do that analysis separately, right? So what happened like who changed it what actually happened on your onto your external database But this process of auditing will allow you to not only Identify but also rebuild or replay whatever happened earlier So most of the blockchain solutions, they do have option for you where you can ask it Hey, I don't I want to know what exactly happened from let's say block number 10 I want to know what happened from block number 12 So the event mechanism and in most of the blockchain solutions, they provide you that option You can replay it from particular block to particular block and maybe Let's say from block number 10 to latest block And rebuild your database so you can process your Off-chain data as if you can you can now again start trusting that database So there are such multiple models that you can follow Now Having spoke about all these what do you think I mean about these design patterns and what do you think are What what do you think works best with these things these design patterns? So, uh, thanks Arun. So, um, let's dive into the benefits and challenges of this architecture first So, uh, this microservices Usually kind of satisfy the increasing need for a more fluid flow of information. So coming to our benefits first So even based architectures are usually asynchronous without blocking So this actually enables the resources to jump freely to the next task Once their piece of work is actually complete without worrying about what happened before or what will be happening next And uh, services are usually loosely coupled That means they generally do not need to have a knowledge or they do not have any dependencies on other services Including their implementation details or any other transport protocols And it's generally easy to scale using microservices Maybe you can increase the resources to the most needed microservices rather than scaling an entire application And this actually enables scaling to be a bit faster. And of course it will be cost efficient as well So when we are using it with a queue it can actually kind of recover the lost work by something called replaying And you know, this can be really valuable to prevent data loss when a consumer needs to recover data So, of course with never ending a list of benefits even driven architectures have some challenges as well So the first challenge that I personally think is they are very easy to over engineer because you know The services can be really complex because an application can include descents or even hundreds of different services And they all need to communicate securely And coming to debugging it becomes more challenging with microservices because an application consisting of multiple microservices And with each microservice having its own set of locks Tracing the source of problem will be a bit difficult And coming to the testing perspective unit testing may be easier with microservices While integration testing will not be that easy because the components are really distributed and Maybe these things might be simpler if it was closely coupled again So perhaps the most Significant challenge is of course the data and transaction management because you know because of this asynchronous nature They must be carefully handling the inconsistent data between the services and they should be you know able to handle incompatible version They should watch for duplicate events and They don't actually support asset transactions and it can be very difficult to you know Kind of track or debug and the choice of adoption is completely dependent on the adoption that we are actually building So coming to some of the process challenges. So let's think for a moment What if if you are taking the common design patterns that we have discussed now to our real life use cases And of course, there can be many questions in our mind. There can be questions on scalability There can be questions on security. There can be questions on privacy. And of course, there can be questions on cost So maybe to circle around It actually I would like to bring your attention to something called blockchain triumph So usually it is explained in context of public blockchains. So the public blockchain dilemmas or it's also called impossible triangles So it's the idea that a blockchain can have Decentralization it can have security and it can have scalability to varying degrees But it cannot have all the three to that sufficient extent at the same time So what I mean by scalability here transaction performing Transaction processing performance that is tps. That is a number of transaction process per second So maybe I can take an example say for example If you're considering the example of a bitcoin it is widely considered to succeed in security having never been had and You know decentralization both in governance and development But coming to scalability due to its limited block size and comparatively long block time It is actually compromising that thing So in short every application is expected to have all these three things But it cannot have all the three at the sufficient extent at the same time Okay, so maybe we can move to a real life use case and we are actually quoting this from the official documentation of hyper ledger fabric so Agriculture is actually providing livelihood to you know more than 50 percentage of the people in developing countries especially like India But you know climate change is at the same time passing grave risk to their activities as well maybe you know the slightest increase in the frequency of a drought or If if there is an increase in rain it will actually present major challenges for food security and water availability to them and to the question on how this payout is actually made usually The agreement actually exists between the insurer and the inshore under specific threshold weather condition when I say weather conditions It can be you know consecutive days with or without rain or something like that So if you're thinking for a moment, who can add those information? I mean who can add those weather data. Is that the farmers? No, they cannot add that Is that the insurance companies a big no Is that a third party? So academically or theoretically speaking at this possible if we can introduce a trusted third party like the meteorological department because they can provide the needed legit information And what if they are not willing to join the network? Then how can we make payouts? Then what is the solution? So this is where an oracle comes to picture So all of you might be aware of what a smart contract is So smart contracts are just pieces of code or you know business logic that actually live on blockchain And they execute on certain conditions or logic that smart contract developers has built and the main focus is here on data Without good data to make decisions on what happens or what transaction should execute This smart contract cannot work on the way it is intended to work They cannot make a decision on its own and the outcome actually depends on the data that actually get passed to it So if wrong information is getting passed to the contract the claim won't be accepted And there is no way to back up transaction because it is on the blockchain So maybe we can come back to our example And see how oracles can solve this problem So we can simply put oracles as trusted third party that actually supplies data to blockchain So maybe for a smart decision if the information is added by a trusted third party like oracles in the form of data concerning the conditions Like concerning the weather conditions The payouts can be automatically triggered and the insurance company or insured party cannot intervene to stop the payments or change the conditions So basically this goes like this Oracle is required to perform some sort of action as a trusted data source And they can have the ability to reconcile multiple data sources and feed them as one stream of data So when I say reconciliation, it can be you know different methods like weighted averages or it can be based on reputation scores Or it can be based on complex cryptography. So oracles are basically it can be considered as gatekeepers So maybe we can think we can put in another way as a blockchain Subjective is to reach consensus You know extrinsic information We cannot you know provide along with the transaction data because other date other nodes would actually dictate information coming from an Untrusted source, right? So therefore the information coming from the real world should come from a party Which is a univocal source whose reliability is undisputed for all the nodes and the oracles to be more precise They do not insert information into the blockchain directly But they actually gather or garner data from different sources and they actually you know kind of Supply to the blockchain So when when What when smart contracts concerning this extrinsic data is kind of executed The court then calls for the right information from a trusted oracles and yeah, it gets executed Maybe the examples of oracles can be you know iot systems such as props or sensors or they can be even humans that operate directly on the blockchain So maybe coming and and then we can come to the different types of oracles Oracles are usually classified into two main categories The first one can be a centralized oracle and the second one can be a decentralized oracle So centralized as a name suggests it means that one particular data source or api is having the control over your smart contract And if if if it is happening like that there is Your smart contract is now no better than a regular contract, right? So if you are using centralized oracles that are not distributed They can actually introduce single point of failure and they can actually you know since they are operating on non deterministic data Their reliability needs to be trusted where availability accessibility and level of certainty about the validity of the data actually depends only on one node When we are actually coming to decentralized oracles It can be considered as a group of independent blockchain oracles that provide data to the blockchain So every independent node or oracle in the decentralized network actually, you know independently retrieves data from an off-chain source and bring it to the Chain and the data is then aggregated And you know, so the system can come to the deterministic value of truth for that data points Maybe there are different as I mentioned earlier. There are different reconciliation techniques. Maybe we can use any one kind any any of them and we can come to a Deterministic value of truth for that particular data point And if you are actually migrating to decentralized oracles It can resolve the single point of failure problem in the centralized trust model But these models can actually bring higher latency For the data processing with less efficiency when compared with the centralized data model. Okay So when if we are actually fetching data from an external source the trial or impossible triangle can be redefined as While fetching a data from an external source a system can have Decentralization accuracy and it can satisfy cost but only to varying degrees But it cannot have all the three to that sufficient extent at the same time. Okay So basically our discussions around oracles showed that Achieving a fully trustworthy oracle platform for blockchain is still at an early stage and it requires more research Attention and you know efforts need to be Excerpted in this direction. So specifically we need to integrate, you know Noble security and privacy mechanisms to interest to the existing trust models Maybe to detect prevent and mitigate some civil and pollution among oracles to solve this design and implementation challenges and maybe we can We can actually try for establishing new standards for unified data format formats and decentralized governance so having this in mind as blockchain grows in popularity and adoption becomes more mainstream the question of governance is still a concern and You know the governance can give them the voice and control that they are actually comfortable with So maybe uh, I can quickly call arun and he's a better person to portray the governance designs patterns. So over to arun Sure, I mean now so I I guess this this is a great discussion so far Like we started with some come of some of the common design patterns. We looked into a scenario where Hey, this is the only way in which Data from external source is the only way in which this we can proceed from there to make a smart contract decision And so I mean I would put this entire process if you can go to next slide like I would put this entire process itself as a data governance related Concern right so whatever we we are following following those best practices in terms of design Following those best guidelines in terms of choosing right model for putting the data right right data into the system so I would put that into data governance aspect right and Why do we need that because of not just for Securing our own data, but also because there would be regulatory requirements It will require us to maybe protect the data in certain ways or Restrict access to that data or maybe restrict from processing that particular data the way they want Maybe even make modification or retention related policies So I mean I would put these things aside for a while and if you if you see apart from data governance There is also one more important aspect that we will need to consider with oracle patterns right so with oracle patterns Some of the good practices that we have oracle service provider smart contract Running in our blockchain system and which has the only access to external services Anything that our smart contract requires We'll talk to this particular smart contract and then let's say if this was the fabric network We have a channel where all the consortium members are part of And then there is another channel where we are allowing Oracle service provider to join in and provide the external data source to us But the important aspect with this is how well we manage this network itself So network governance is is very critical over there along with data governance I mean the diagram is is more intuitive than what I would what I could express We need to restrict access from from these services In inbound traffic from these services there should be it should be more of Outbound traffic I mean request which are which can be made from consortium members Than inbound request coming in from Oracle service providers. So With with these thoughts, I mean this this this puts a picture around How can we In the I mean when we started how we started with the design patterns I mean what was the missing gap or how do we manage the off-chain data versus on-chain data And we also touched upon the governance right some of the key areas that we need to touch upon or at least keep in mind when we design our applications I mean, I I hope this was a useful discussion from both of us And thank you all so we are open for questions and we are happy to answer answer your questions Thank you all I see a question which says What are recommendations for query from blockchain based systems? Um, so this is a great question. I know many people have this question So it depends on the trust boundary. I mean you could of course directly read from the Let's in this case. I'm assuming you have external database like externally installed couch database or your peer In hyper ledger fabric, you could directly read that but it depends on uh, trust boundary If if your trust boundary is such that you are the only party who can access that database And of course you are free to do that However, it's always better that we have uh Like an external copy of data the way we want, right? So because the let's say even when block pruning kind of concept comes in or um, when where we want to Stop the blockchain or maybe like archive a blockchain for a certain period of time The old data still plays significant roles for analytic purpose. So maybe that's where you may consider uh, like data dumping ground and use that for your applications than directly using the couch dv instance I see another question. Do you think a chain code could be used as decentralized record? If not enough and or society under the value transaction of failed printing condition Right. So, uh, if you if you remember the diagram which we were showing and in that we Showcase one of the way in which oracles can be designed where We have a smart contract specific smart contract or specialized smart contract Which is going to help read external data. So you could of course convert. I mean this smart contract could be implemented in any Um protocol or in any language including the chain code way of reading external data And chain code in fact, um, I mean helps you in governing it much better, right? You can not you not only are agreeing upon majority But you could also put a measure such that hey if this is the person who is signing the data or who is approving the transaction Then the weightage for that person is is like more like one of the technique Which Anina discussed is around putting weightage around those wereacle service providers so that we could trust At varying levels based on their reputation in the past I hope that was useful answer And do we have more questions? I think I see one more question It's such an environment. I did not really understand if if someone What should be recommended architecture look like and what sort of oracle would best fit such an environment? I probably did not understand and happy to take your question offline And understand this question in more detail. Okay. Um, I think we are To the end of this session. We are three minutes over time Thank you all and and I hope you all had a great hyper ledger global forum this year See you in the next session in the americas region Thank you. Bye. So hope all of you had a wonderful session and see you next time. Bye