 Okay, welcome everyone to our weekly firefly community call today we are going to be hearing from Peter brothers about message sequencing in firefly and kind of how the internals of the system work. It's kind of an important part of how the system works. And we'll have some open discussion after that. And this is a public meeting it is being recorded and the recording will be public as well. So just want to throw that out there. Again, welcome and thank you for coming and I'll turn it over to you. Thanks Nico so we're ready to go. We're going to we're going to do, I guess, probably part one of message sequencing. The slides here are going to go down into the weeds, but I'm probably going to not go all the way down today because that's just a lot of information. And I really want to make sure that we've talked about why fireflies trying to solve the problems that fireflies trying to solve and try and make make it clear the value that message sequencing is one very important thing where there's actually heavy lifting code inside of firefly is probably the most complicated piece of actual coding firefly. Because it's almost by definition something that none of the individual components that underneath firefly can do for themselves. So what I want to really address head on sort of why is that all the sophistication around ordering is collider just about ordering things. Sorry. So just about ordering things always firefly. You know, is it treating the things underneath it is just dumb things that need to be ordered. No. However, ordering is something that always ends up getting built on top of these underlying technologies and fireflies there to stop you needing to build things like firefly on top of the underlying the picture here I think I think in America it's fun ball is it. I have no idea actually what that thing's called. I've seen these things in in in in in playgrounds rounds around these these next to the woods, and they are absolutely fantastic analogy for how business networks need to deal with things in order. And we'll see we'll see we'll come back to that and it'll be a little bit clearer in a minute I'm not just inferring the whole business networks are like a bunch of bunch of kids playing in a playing in a yard. Although I'm sure we can all recognize a few similarities at times. So, so the sequencing is one of the most important constructs that enable multi party business processes to function. This is not new to blockchain in any way shape or form. And definitely not not new to money party systems. The whole suite of technology but used to be called a list called business process management technology but used to solve business processes inside of an organization, and it's incredibly valuable inside of a money party system inside of each organization itself has considered this one of the top problems for the whole of my career. And it, if you've got a business process inside of just one organization, how does it work. Well, you create an instance of a long running process instance. You create an ID, and then you create a flowchart and the flowchart might be expressed in BPM or some sophisticated technology like that you create a flowchart that says this happens. Then this happens, then the decision happens, and it branches left or right, and then this happens. If at any given point in time, it's like quantum imputing it might have gone this way or it might have gone this way or it might have gone both ways at the same time. You can't build a business process, it's impossible. What you end up building is that every single step. You have to build compensation logic that says well, we'll assume we're going in both directions and then when we realize that we've actually gone in one direction and not the other will go back and will sort of compensate for the one that we did previously that we didn't mean to do. And it's horrible like that. So the way a sensible business process works is just straight through process from beginning to end. Now if you've got one central system inside of one organization, that's not easy to achieve business business management software in itself is super complicated, but it's a process that's solved by creating a highly simple central system with a database inside of it inside of one organization, and they'll be a database and they'll be messaging systems to coordinate the actions like that's what people have been doing for business process management. You know, a decade and a half now people have been thinking about this problem of doing it inside of the business. When you do the multi party version of that things become interesting because you've got multiple people with that own autonomous independent systems. They've got their own sources of truth. And this is where blockchain is magic, genuinely magic for these multi party systems, multi party system people have been building them for many, many years. And the blockchain based ones have a construct that no one's had before, which is, we can have an agreed sequence of events. And it does mean just a little bit of extra complexity for the poor developer of the business process. This is what we're using and firefly cares deeply about just developers building code. So we think about it as West API is fundamentally, and it's just the same business process that would be created for one organization, you have to model it such that each is formed by a different organization. And when a organization decides to move to the next step. They can look at all of the previous history. So if I'm, if I'm this step here at party three. I can look back and I can see the sequence of events that happen in an agreed order, and we'll talk about how that's possible. I can look back and I can decide to take the next step. The one tweak the one little bit of programming awkwardness that we can't get away from is that I have to wait until everybody's agreed that I've stated what the next step is in the sequence before I can trust that it's true. I can't just say, only decide that this is the next step and go, I need to say, I initiate what I believe the next step is and then I wait until it's been confirmed on the blockchain that my record, you know signed proof from me that I want to initiate the next step in the process has been confirmed, then it's confirmed to everybody and no one can now insert a step before that. But if we just try to do it at the same time, then I assume that because I said it. The point that I said it nobody else had said anything else. I assume that it's the next step will get into trouble. So I'm going to just go back to the analogy that we talked about at the beginning of that fun ball game where you've got all of those people throwing balls into a into a funnel. Basically, fun balls got multiple exits on the phone which makes it more complicated but we're going to talk about fun it's only got one hole at the bottom. And the, you can, as one of the parties in the network and this scales to as many parties you should like so just simplest to explain to you. If you've got Bob on the left or Sally on the right here, each of them, they know the steps they are performing, and they know the order in which they perform. Well they want to perform. So what they're doing is just saying, I'm going to do the next one I'm going to do the next one and do the next one. And they're not waiting for them to be confirmed. Think of it, like throwing those balls up into the up into the, the fun ball funnel, right, your, you know the order in which you sent them, like if Bob's got blue balls and Sally's got orange balls here. If he throws, he said he threw ball one, then he threw ball two, and he threw ball three, but he's no idea whether his ball one is going to come out the bottom of the funnel before or after Sally's ball one. If they go in the air at the same time, or if they land in the funnel and they're rattling around the funnel at the same time. If they go away, for either of them to know at the point they throw them, which one's going to come out of the funnel first. But it's not hard, you just need to look at the bottom of the funnel rather than the top. You've got to go throw the ball and then you're going to wait and you've got to see the order that comes out. And that's, it's not a hard concept, and it seems pretty trivial. It's the thing that has been impossible in these multi-party networks before. If you just have West APIs to each other you know you didn't have a blockchain in the middle you didn't have a multi-police system with you know the current generation of technology suite. You were just using West APIs and messaging. You could absolutely have an agreement over, you know, I talked to you and I say something. But if you've got five people in the system, you can't without a lot of sophistication, basically building something like a blockchain, you can't agree a single order between everybody. You need a consensus algorithm, something which is going to make it final, what order the events occur. So that stuck, it's maybe sometimes an under, you know, in the world of tokens and the like, it's an underrepresented value in its own right. The fact that blockchains at their core are a sequence of events. And it, it's worth just mentioning, it's also the thing that makes tokens possible. This same construct is what allows tokens to be built on top. So just like you can build a very well proven business process, like total conservation, the value of a fungible token or uniqueness of an ID for a token and propagation of ownership of, of a unique, a unique ID in the case of non fungible tokens, you can build that on ordering concept constructs and Andrew last week was talking about how far fly beats is worth actually bring those in as tokens and like into into fireflies API. You can also build arbitrary business processes on this fundamental, like air and water of the blockchain ecosystem piece of piece of tech. So what's coming, which is, wouldn't it be great if all of the information that we wanted could just go on to this sequencing information, like tokens work because all of the information that backs the things that tokens provide you ownership information, as goes on to the blockchain. And my business processes just can't work that way, just almost universally, the data itself can't go on to the sequencing tech. And that's for two reasons. One is the data just isn't the right regulatory shape, right, maybe it needs to be deleted at some point in the future. Maybe it's too sensitive to be shared on the blockchain maybe it's just competitive information you're giving away by putting all that data on the blockchain. One problem. The other is, it's really expensive to do this multiply agreement on data. It's really expensive these consensus algorithms are hard. So if you slam huge amounts of data you say look here's my, here's my 100 megabyte PDF file, stick it into the ordering construct, like into the actual contract itself, you jam up the pipes. The ordering construct it has to be sequenced one after the other you've only got one, one pipe here. So it's not appropriate to try and push all of your data for those two reasons through the pipe itself. So that's why file fly has all of this sophistication built into it for coordination. And using the technology and most of the, the, the architecture you'll see this, this term aggregation being used. And I was talking that I go with Niko have a little bit this morning and I don't know if the terms white or wrong but I wanted just to explain what we mean, like there's multiple things happening, but there's only one source of truth of the order. There's a true source of truth which is the blockchain for the order in which things happen, but the blockchain does not contain the actual business data. And maybe there's not just one piece of business data, maybe the thing that I'm trying to order is a step in the business process which has a little bit of Jason metadata, but with some signatures inside of it that say look I attest that this is the state of the business maybe there's 10 attachments that are each megabyte in size and all and then there's the core blockchain thing and like all those five or 10 whatever it is, like bits of data need to be distributed to everybody. But they can only be processed by everybody in the order that things happened on the blockchain. So you need to sort of treat the blockchains as a source of truth, and then wait for the other things to arrive before you can act on it. And that's really where that heavy lifting is inside of the inside of the firefly. So, I'm definitely going to talk through this one in some more detail. We're going to flip we're going to sort of go on the high level of some of the more advanced constructs that build on top of this like how do you mask do advanced privacy on top of some of this stuff. So we're going to talk through this diagram and to end, and we're going to talk about how firefly is built and how as a developer against it you need to understand this piece. It's built to give a global order, a global order sequence of events to real private and shared business data of any size and shape in a business network. And it comes with a building contract that allows you to do that, but the only thing that goes on the blockchain is just just the sequence of events. And it is compatible the architecture with a next step that we expect to be taking in the coming months to allow a blockchain transaction to be as sophisticated as you like, and still to be coordinated. So it starts with transaction submission. So just like on the previous chart now we've got we've got, you know, we've got Bob and Bob and Sally here, and Bob's in number one and, and, and Sally's over here in member number two. And they've got an application. Okay that application is connected to a firefly core, which is the API server, the firefly. And it's connected to it using West API's and, and WebSockets could be web hooks as well but just taking the example of it's West API on the WebSocket. So you just got you've got a West API connection from this app to this call. Now a really important point is that this app in a real world production scenario might be being scaled up being scaled down. It could probably be the back end of a web app so you're probably running five instances of it. Firefly is designed to still be able to provide these assurances that we're talking about here, even if you've got a horizontally scaled application. And the architecture. No awesome points of detail here but aren't fully fleshed out. It's also supported where firefly could be horizontally scaled. So that my firefly can have a few instances, and your firefly can have a few instances. And they're still, you know, this is still just my one firefly and that's possible, because underneath it we use a database and we make a lot of use of the database and firefly. In order to be able to provide a very simple contracts between firefly and the application to be able to take away genuinely many thousands of lines of code that might be needed to be in the application to be pushed down to firefly. And the app just doesn't have to worry about it because firefly relies on upon a consistency that's a highly available database. So we rely a lot on that. So the app you just say to the your, your, your firefly, you just say, and the message, right, send the red message and the green message. And firefly will do all of the batching and other, other, other things that are necessary to efficiently use the, the technologies to efficiently use a messaging technology to efficiently use a public storage technology like IPFS, it's very inefficient to take like a kilobyte of data, and every time you have a kilobyte of data, create a new shared file on IPFS, hugely inefficient. So firefly does all the batching to say, I'm going to put hundreds of these inside of one, one file that gets stored in an IPFS or messages that gets transferred across. And each firefly node is responsible to making sure that the order of your events is preserved. So if you send a green message before red message, the green message is going to come out the other side for everybody in the network before the red message. There's no assurance at the point that you send that rest API and you say acknowledged by this, this, this, this message has, is being sent. So there's no assurance at that point that it's the next message in the actual sequence, because Sally over here in her organization, her app is sent a blue message at the same time that we were sending the red and the green messages. So there's this arbitrator the blockchain in the middle and the blockchain locks in the order. So the firefly cores both send a request through the blockchain interface which is pluggable. So we talked about the majority of the, the Ethereum, the standard development on the fabric which is probably going to be a topic for another one of these coming up soon. And then what we've done in the past to prove this out on quarter that you have a an interface to the blockchain, which is is about reliably getting it onto the blockchain and then detecting an event when it's made it on the blockchain. So these on this side, the green message goes on first, then the web message on this side, the blue message goes in and the blockchain orders it green, blue, red, and those events come out to the both sides to both sides, and the events go back to the applications. So really important piece in your application, if you want to be a business process that's genuinely multi party safe. You can't treat your step as processed. You can't act on it, you can't update your core systems of records inside of your organization. You can't treat it as confirmed until you process it in the order that it comes out of the blockchain, not the order it goes in. So this application sent red, then green. Okay, if it didn't wait for where to be confirmed as the next thing before it sent green. But all it's doing is it's put is queuing things up for processing. It needs to process, say green and red, apologies. It needs to process not green, then red, it needs to process green, then blue, then red. And if it does that, then it's assured to have performed its processing in the same order as Sally has. So Bob and Sally have both processed red, sorry, green, blue, red, green, blue, red on both sides. Now that is like I'm laboring the point because it is a change in the way that you program. You, you have to say to your user experience, for example, you can still have beautiful user experiences, but the user experience will immediately say, thank you, pending. And then we'll update a few seconds later to say confirmed, or it might say rejected. Think about an example like spending. I have a business process where somebody in the business network says, I've got a credit balance. And the other members of the network say, I'd like to bid on that, that credit with bananas. And everyone's waiting to do it. So this might be happening in very short period of time that credit bananas can only be given to one of those parties. So, so all of those parties that are saying, I want to purchase this credit bananas, because right now, at the point in my application I look at the states. The credit bananas is eligible to be set to be sold. I'd like to buy it. That's great. You said I'd like to buy it. The outcome wants his order with everybody else who says they'd like to buy it might be. Well done, you bought us, or it might be. I'm sorry, someone else got there first. It's a really simple example where you can see the real world implications of if you just assume that because you learned your local database and your local database said that bananas were available. I'm going to buy them. Thank you very much and you didn't wait for it to be confirmed. And then you could end up assuming that you've got something that you haven't. And with an inconsistent state between you and the other members of the business now. The way this has always worked in the past before blockchain was well that was all great. Everybody did it and then reconcile afterwards you had a conversation logic legal disputes that could take weeks or months, or you just waited for confirmation to happen on the phone and humans or with ups stuff flying around the world. You know sending paperwork that's been signed and you need copies of paperwork that this construct means that you can do this digitally and it's and it's really powerful. Peter, do you want to question you want to take it now. Yeah, let's take a question. What if the contract in BC layer reject a transaction question is because it will help to understand if we need to model our logic to adopt to the behavior of Firefly. So if a, if the on chain transaction. And we're talking about a mode that you can't do immediately today, the moment which is custom on chain logic. If you have on chain logic that is making decisions based on what's on chain as the source of truth. Then it's important that you admit events from the blockchain that describe the different outcomes. You should write your blockchain logic to admit a yes or a no. And that means that that what goes in is a red request, what comes out is a red yes or red no, depending on what the blockchain data set. And this is about the aggregation that we're not going to go through in detail. There's charts like this that talk about how it works that we're going to go through once we had enough of these that everyone feels like we want to be talking in a sort of level of detail how it works in this core. But fundamentally, the application like the risk the point it receives the red. It's got the on chain information of the result of the transaction pass fail in between some data generated from the blockchain. It's got that conclusively for sure from the blockchain. It's also got the off chain data that went with it but it can correlate it with. So you get the value of the event sequencing, because you're not just rejecting it from the blockchain, just to be really clear, you're admitting a rejection event from the blockchain, which the application can process off chain. So the core system said I'd like to buy the credit bananas, the core and locally that that's a request center to purchase the credit bananas, what comes back out is on chain. There was a bit of information that said no, an off chain is it's this credit bananas with this particular RFID tag in the box. Well, that data might have been off chain and the yes or no mother been on chain. When the application gets it for flies going to make sure all of that's been aggregated back together again. So you get an event that comes with all of that data, as well as that that very usually very small piece of information that's on the blockchain itself. I think it's, it's also worth repeating, I guess the assumption here, which is the default smart contract gives you the ordering doesn't really have any logic in it. But when you look at when you look at in the source code has no logic so you always get the sequence emitted and there's no, there's almost zero chance for that to be failing. So it's more likely that it's going to be fading because there's networking issues, there's there's other things and far flies built around those make sure that doesn't get populated into the application layer. So, using far fly with the, with the basic functionalities, you don't have to be overly concerned with the failing on on with the on chain logic. Yeah, because we're 33 minutes past I know we started a few minutes late but we've always said with these, these sessions we don't want it to be dominated by just having to listen to one person talk to a bunch of slides. So we're going to, we're going to try and cut in a couple of minutes and just get to open questions which could be on this or they can be on anything else. There are a couple of bits that we're not going through today. The first is how on earth is it possible to do all of the things that we just talked about on the on the last chart. This is like an example of what how if you chose not to use firefly, and you really wanted to solve this problem correctly, the level of sophistication, you would have to build into application layer to not have to build yourself. You know, it's not use firefly so so this will go through at a point where it makes sense to sort of go level below the previous diagram. Now, the next one I think is something but probably a bunch of people on the call excited about doing blockchain which is how the heck do you can you do that without leaking information on the blockchain in the case that this is a private sequence of events. And in the case, and this is true almost universally including when you're using channels, including when you're using channels, some of the data needs to be hidden from the parties maintaining the channel. Maybe I've got five parties on a on a fabric channel because I've got the supplier, and I've got the auditor, and I've got the purchaser. And, and I've got the shipping company all on the same channel because they need to see the sequence of events that doesn't mean that all of the data fields and all of the, all of the pieces of the transaction need to be seen by everybody. So if the blockchain is an open one it's not like a fabric channel that's it's like the everybody channel that you set up on your fabric environment or it's an ethereum chain so everybody's involved in it to enterprise ethereum chain like or hackalogy of a suit, then it's going to be maintained by lots of people they need to not be able to see what's going on, and you need to be able to hide. Who's performing the transaction, you need to be able to hide the data that's part of the transaction. That's a hard bit and here's what we'll go through in some number of weeks time when we're ready for it. You want to hide the fact that this sequence of events are a sequence of events, because that's metadata leakage, leakage. The fact that five events happened, and as five events that happen together, you can form that you can deduce, it's probably one of these transactions that was happening. It's considered leakage to have the same identifier on chain in all of your transactions if you're trying to keep things really, really, really obscured from people. It's not something to solve, but it is something that's solved in in in Firefly we believe. So we look forward to a conversation in a little while where we talk about exactly how Firefly does that, that masking in truly private transactions, and not being able to preserve order and avoiding the big problem in ordering is actually not how do you order. It's how what do you stop if something goes wrong, how much of the world stops because you don't receive something. Now how do you deal with that situation that's actually the hard thing and it's always been the hard thing in message message sequencing way way way before before watching. There's another me waffling waffling on. So we wanted to sort of go to the open discussion section.