 Welcome everybody. This is our very first Firefly community call. We're really glad to have you and excited to kick this off for these first few community calls. We're going to have a subject matter expert walk through a technical topic on Firefly kind of starting with architecture and flows through the system to help people get familiar with how Firefly works, how it's put together, and all that good stuff. So Peter Broadhurst is going to be leading the session this morning. So we'll basically take the first 30 minutes of the call. Peter will walk us through that. And then we'll leave the last 30 minutes up for discussion Q&A and whatever topics people want to talk about. I'll try to keep us on time. And with no further ado, I will hand it over to Peter. So, and you're going to keep me to time, we want to make sure that we try and keep the second half of this just for open topics and those topics don't have to have anything to do with the first half of the meeting either. So we were thinking about what's the very first target to kick it off. And I think we'll be trying to influence the schedule for what other topics we do and who comes along and talk based on feedback from the community. We've had enough feedback to know that the next one is probably going to be tokens. We're going to start and what feels like the beginning, because it's thinking about as a developer and Firefly is for developers blockchain or not blockchain is very important thing. So the first thing I want to do inside of Firefly, particularly thinking about the ping pong example we use during the kickoff where there's a sort of mixture of coordination of some stuff happening on chain some stuff happening off chain. I want to do some of that. I want to do some things which I'm sharing with everybody in the network some things that I'm just doing with individual members in the network privacy first approach which is so common in enterprise scenarios. How does that work was the sort of fundamental model here and what do I need to sort of understand about the model, but let's start from where you, you start as a developer which is what, what's the first interaction with the API. So we're going to start here with sending a message, a message is probably that if you sort of put blockchain transactions aside and we'll talk a little bit more about those later in the session to lots about them in future, future sessions. So to put the simplifying talking to really sophisticated blockchain transactions aside, and you, what's the most important construct inside of sort of Firefly and it's probably this construct of a message. The message is a, an envelope really of one transmission of data with a signature if you choose to choose to have that from one member of the network to one or more members of the network. I think the reason to think of it like the envelope, rather than the whole that rather than the letter is because inside of that that message you include data that that data. When you actually send the message you can just put it there and Jason, right, you can just say look, I only want one piece of data in this message and its value is a string of tests. I could put in a sophisticated Jason objects inside of this message, I could put five pieces of data inside of this message. But you could also have data that you're putting in the envelope and you're sending to somebody by doing over send to an individual private message to somebody or broadcast to all of the network. You can do that, based on data that you already have because you received it from somebody else, or data that you have because you uploaded it because it was 100 megabytes file. So when you're sending a message you can either reference existing data that already exists in your private view of the world on your Firefly, or you can add data in line in that in that payload. So probably the very first hello world you're going to do is going to look at something like this thing on the light. I'm going to say look here's my data it's just in line. It's there in the in the message itself. And you can do this using for example that the, the sample application in Firefly samples that Andrew showed at the kickoff. And we'll make sure there's good things to that. And you just say look here's some data I want to send but you, but in the future you might find you refer to this people with the upload it. And we'll talk a little bit about how blogs work later. Go to the header of the message. And the reason we put it inside of the header that it's like the address on the front of the envelope. It's, there is like some some work done in Firefly to make sure that we hash all of this stuff in a way that the combination of the unique idea that gets allocated to the message, all of the data that's been attached to the message, and these extra things that you specify explicitly in the header, everything that goes into the header comes back in the head of all that gets hashed together so that this message has a unique idea that has a really unique signature in the world it's just one transmission and that that you know you can look at and everybody who's involved and can see. So in the in the header I say things like do I want this to be pinned to the blockchain what transaction type of it. So you might want it pinned over on many cases where in this party party network you wanted to be audited on your side in Firefly with the database you want it to be all still on the other side. In their Firefly, but actually you don't want any of the overhead of putting it on the blockchain at all. There's no, there's no reason you have to do that just because you're using Firefly as part of the party network. So, so you choose if you want to transaction or not on here and you tag it to save a type of message to services like my particular request message or, you know, a category of the message. So the topics we're going to talk about in future sessions it's like an ordering context. Everything on the same topic has to be ordered with everything else on that same on that same topic within the scope of the group. So, so that's an important construct we'll talk about the future sessions when we talk about message ordering. And then if I'm not broadcasting it to the whole network I need to specify who I want to send it to. And you do that using a construct for the group. We'll talk about why having an explicit construct that is a group is necessary, and what that the value that that provides later. But just know for now when you're sending your first message if you want it to be private, you have to specify who the recipients are. And if you specify the same recipients over and over again, it will all be resolved to the same group. So if you want to really detail with this at the moment you might want lots of different groups of the same distance and you can customize it with a name. So one last thing is there is the construct which more is happening, as Jim and others work on fabric channels and the like the construct of a ledger, but I want to attach this to. Many networks only just have one one ledger that everybody can see, but there are other networks where you actually want to have different ledgers that have a subset of people who can see the tests of that ledger. I guess the key thing to know is there's a big difference in firefly. Just because you're all using the same ledger doesn't mean you all need to see the same data, because in real business scenarios, it's much more common for different parties to be able to see different parts of the transaction. So you know all the testing to that transaction, but that is, but every party can see every piece of paper. So anyway, that's, that's a message. It looks pretty simple. It's very simple to send just to post to message to send the message, and that goes into the system. What happens then is it sort of gets split down reconciled with everything else that's already there so maybe you reference the existing data. Maybe it's the first time a group's been used that got resolved down or maybe that group's happened to be used a million times before. All of that gets resolved and a bunch of things get stored locally inside a firefly. So these things you'll see there's lots of rest interfaces to get on firefly and the UI shows and it's continuing to evolve to show all of these bits of bits of data. And you can see sort of like one piece goes in and it gets split apart, and there's good reasons why it's being split apart. But in this session, we're going to talk about what some of that splitting is and why it happens. So the three logical things that you might have put in the same message. They get split apart in the data model that's part of firefly and the message with the header and the hash and all of the like, that becomes one thing in the system. The group is another thing that you can go and look up and that gets referenced by hash rather than ID and that particular case. And the data, all of those individual pieces of data whether new or the ones that you've already had, they all exist as unique resources and with rest interfaces in the model. Just a couple of other things that we're not going to talk about much more about in this session, but I wanted to make sure you're aware of data is split between the stuff that is of value. And it's like in line in the data that gets stored in the database. Database is obviously not really suited to storing like 100 megabytes long. That's not really the right, the right balance in fact probably get above megabytes and probably you should be thinking of different storage approach. The data exchange plug in that's a pluggable capability of a firefly that comes with the built in one that uses a file system. You, that's where these those will be uploaded so you can upload them on the firefly API and you can download them and you're on the nice multi part upload that you'd expect, but where they get stored. So that's a, that's a pluggable thing so blogs are interesting. And then the other thing just to mention here is that groups, we can sell down to identities. And the inside of a group each identity is a combination of an actual identity, the moment the only entities that exist organizational identities, and those need to have been registered ahead of time. But there's the architecture will definitely support anonymous scenarios down the road. And, and also a node identities. There is a plugin for identity, which is really a placeholder at the moment it just maps one to one where you might have an identity which is something very, very flexible like a DID, and then a resolution plugin that will allow you to resolve that down to finding identities and other identities in the system. But just just know for the moment that there's a network map. And when you run your CLI, it's automatically registering all the nodes in your in your CLI created environment during the CLI startup you'll see a bit where it says registering or registering node. It's making sure that everybody in the network knows about the existence of that identity. And that happens under the covers, using a broadcast, which is one of the flows that we'll talk through now. So, my message, little input message better data reference to a group, whatever went in, got split apart into a message of group and data. Happening in, you know, get reduction scenario this is happening in volume, lots of these are happening. And they are being accepted into the system, but there's lots of chain of events that needs to happen from them going from their initial state which is pending to actually being confirmed in the system and we'll talk about the existence of ordering in future sessions. But fundamentally what firefighters about is saying, we can have a shared source of truth. We can have a shared source of truth. And it's very important in many business noise that we know who said what in what order. And just saying it is one thing. That's the send. What matters is everybody's heard. So things go through a life cycle where they, you get an acknowledged best API call you get an accepted 202, when you send the message. It doesn't become a confirmed message, and the event that we'll see at the end, but results in a confirmed message, until all of the things that need to happen as part of that have happened. So first, I'll probably the most interesting one, which is a pinned transaction, a very simple transaction, which is just establishing the or the originator, and the sequence of the message, very simple transaction, is tied to a batch of messages. And we very deliberately and have made it so that pinning this is from from all the previous generations and the learning there that you actually do need to batch things together to get through the group that you need. So batching is an automatic building construct for pinning inside the firefly that will work across the edges, etc. So, so your messages get assembled with any other messages that arrive in a particular time window up to a certain cap so 300 messages arrive within a couple of seconds you assemble those together they go into a batch. And, and then that batch gets a set gets a transaction assigned to it and the transaction exists in firefly. So you can actually see that transaction sort of set has a set existing your local database of firefly as well as propagating to the to the blockchain. As a placeholder here with it's really important to understand that firefly is trying to simplify. All of the things you might do in a multi party system we're talking a lot about a case where the transactions very simple, because we're focused on the on chain off chain coordination. The components that are pluggable like if connect inside of the system, interacting with super sophisticated smart contracts is really important to this ecosystem. We're not talking about it, not because it's not important. But we will we talk about because really a lot of heavy lifting that firefight does is to enable all of the things that exist off chain to coordinate with those on chain things. But it's not it's not important. It's just that the focus of that is all of the existing communities. And we using all of the investment and fantastic innovation that's happening inside the individual blockchain communities across the blockchain communities. That's what far flies there to do it's there to harness that it's not there to be the next fantastic implementation of a hash time lock on chain transaction right that's not the firefly communities the firefly communities world is to take that fantastic implementation that's just been done and make it possible for me and mortal engineers that by false that web applications to build beautiful experiences on top of that. So watch this space for us to talk lots more about transactions in this session, the example transaction is super simple, and it comes out of the box in firefly because it's so simple super simple. So, um, the journey we've been on so far is that we've got messages, we put a message in, it got cymbal into a batch of messages and we said well the transaction gets associated with that and stuff needs to happen. There's another, another thing you'll find pops up as a side effect of sending a message, and multiple ones of them pop up for each individual batch and the UI and the API's help you join the dots between the message and the operations. So operation is an actual thing that actually happened in a plugin. So you've got to plug in for the blockchain, you've got to plug in for the data exchange that's doing things like transport transporting messages privately between two participants with the right level of transport security. So it's an operation for something like storing the data in public storage like IPFS. Those operations are probably like that they're infrastructure specific, and they have a result, right, depending on the way the plug in works that result might come immediately. It may come in a few seconds time. That result may be a success. It may even be a failure. So we have an operation, a little resource and then very, very small resource them inside of inside of Firefly for each one of these operations each, each sort of bit of the plumbing that's happening on underneath this this bigger piece. So you can understand what happened, and you can see what how to correlate the orchestrator with the low level pieces so if you want to go and look inside of IPFS, you need to know the IPFS ID where do you look well, because it's IPFS specific it's going to be in the operation. The operation is the mapping from a general. I need to pin this data, I need to store the state in public storage to to I did this in IPFS and this is the idea so operations are super useful. And in this case, because we're in the pin transaction, there's an operation that's going to be sending the, and this, this case let's think of it as private data could be IPFS but think of it as private data. So this is sending to Gabriel a piece of a piece of data, completely off chain that happens in one operation off chain, and then the transaction being submitted to the blockchain is itself a an operation. And what you'll see is a lot of work has been put into making sure that these transactions leak as little as they possibly can. So things like the blockchain plugins are able to do things like anonymous submission of the transactions but that's critical specific so that's in the plugin bucket. But the transaction data itself in involves an ID uniquely generated for that one batch. So it doesn't leak. It's a hash of the batch which allows you to match the thing that you receive off chain with the thing that happens on chain, an idea of the transaction goes there as well as saying characteristics have been uniquely created. And then these pins and we're going to talk a little bit about those more in future sessions there's not time today to go through them in detail. And there's a pin for each message in the batch. And the pin is represents the, the, this message is sort of place in the overall sequence. It doesn't get sequenced in the actual order to the blockchain puts the transaction in an order, but it's the thing that identifies in a way that is obduscated for privacy. So you don't have this like you do five transactions on the same topic within a private context, not a broadcast context, you don't want them to be the same, same value with those five transactions. That's the cleverness under the covers that will go through in the future, the future. One of these sessions, these pins are there so what you'll find when you look at the blockchain transaction is there's this sort of hash pin for each message inside of the batch. I mean it's a very efficient way to use the blockchain so you can get a lot of a lot of performance out of the blockchain because you're biting per batch which might contain hundreds of transactions. So in one transaction it's just got this list of hash is inside of it. Okay, so, so we've, and just refer back to blobs here the blobs have a separate, they travel separate to the messages themselves. So in the case where we've got private pins message in this case. Sorry, a public let's say sorry apologies in this particular case this is a broadcast. So we take a blog and the blog is going to be put into public storage. So first into your own local storage, and then when you send a transaction that needs that to be broadcast to everybody will check if it's already been put into published storage if it hasn't will put it there. And then the, the data that flows with your message just has a reference to where we've stored in the public storage. It's not as extensible to any public storage mechanism it could be a URL to some central location, it could be an IPFS ID, which is the current plug in that's that's there to out the box. I'm going to just talk a minute and we might come back to this in the future session. It is possible to do completely unrelated to blockchain trans messaging as well. It's available, and you can give it a try so if you need some things that involve the blockchains and some things for latency reasons or other reasons don't don't need to go on the blockchain at all you can do that. And they just get mapped up and basically all the data that needs to go gets grouped together into an individual message, and that message gets super simple the message details of the group because you need to know who else is in it. There's still about that with each sessions. And then the data gets bundled together, but not with hundreds of other messages like just just those get bundled together, and they get, and they get sent. So that's how unpinched transfer works. And then if we look again at pinned and we look at how blocks work in the case of just sending those privately. It's very simple. There's an operation for them, and they get sent separately to the batch. And because they're large, it makes sense for them to have a separate transmission cycle, and also a good data exchange implementation is going to do things like chunking and streaming and checkpoint restart all the things you'd expect from a managed transfer sort of protocol. Peter just a quick time check probably about five more minutes for the segment. Thank you very much. Okay. So we've got as far as all of the stuff's gone right and we broke it down. Our input message groups data went into a batch few operations happens in the transaction happen. And we're going to talk a little bit more about then I'm going to talk now about the other direction, right. I sent something in what about when I actually want to receive information from firefly. And we're going to just talk briefly about this today and then we're going to have whole sessions we talk about sequencing and complexity that's involved here. And the important thing to know is in the case of pin transactions. So this is the most interesting case. The order on the blockchain is the truth. The order on the blockchain is the truth. It's the blockchain interfaces. Whatever that plug in is to do a viable reliable event streaming from whatever the blockchain is. So if connect for a theory and does reliable a collection of information from the blocks and transactions that happening inside of inside of your theory and network and all of those topics and logs. The order, quarter connect does that with the, with the Java interfaces that allow transaction streaming based on timestamps and quarter transactions and fabric under development and plug in does that with it native eventing system of fabric. So it's responsible for delivering them to firefly in the same order reliably as they happen on the chain firefly just says, thank you very much. I put that in order. And it puts it into an ordered sequence of these pens. And it does that because, while the order on the blockchain is the truth. The message is ready or not is not just did the blockchain better wife, the blockchain bit my wife first, then the private data, then the blog, the blog my wife first then the private data, then the blockchain transaction by any combination can happen. So very importantly, it's in a particular point in this in this list. And it might be the other things before or after it that are in the same sequence, have or haven't arrived. So probably one of the heaviest lifting pieces of engineering inside of the core of firefly is the aggregator, and the aggregators responsibility is for working out when a pin becomes ready. The other stuff is assembled the off chain starts being aggregated together, the pins ready, I can move on. And what does moving on mean, moving on means turning the fact that that pin was something that I'm interested in. I had a right to see, I've got the data, it's all available that fact becomes an event. The message is go into the system from one member, and then they come back out again as events to that same member, and everybody else. And that means that everybody can process them in the same order, whether that's the person who's center, or everybody else. So inside of firefly there's these two things which look a lot like cues. There's the list of pins, which is just a list since the beginning of time and everything that's been pinned. Whether I know whether I know I'm interested in it or not at the time of hitting it. It's an interesting thing you don't know at the time that the pin comes in whether you're interested or not, because the other things need to come in before you can tell. And then the ones that I am interested in and they complete go into a list of events. She means we store those events only once. And that doesn't mean just one application can listen to those events just like anything that's in the messaging space that deals with these kinds of scenarios, we have a construct of a subscription, and the subscription is a very simple thing it's just a pointer to a particular point in this list of events, and a filter to say what events you're interested in. And then a connection to a particular transport of which we have to web sockets on my hooks, very pluggable there easy to plug a lot of other things down the road. And where you can say basically every time an event happens of this type, deliver it to my applications using this transport, and then that pops up inside of your application as an event. And then a whistle stop tour, I know that's a very dense 30 minutes, we sort of built up to this overall picture, or what we've got is we've got, I send a message, and I get events out. We're going to talk, and maybe in a future session we're going to talk about how we can turn that into an API, so that there's a request message and there's a response message and they need to be aggregated to wait for a response to a particular request, we're going to talk about that in future sessions. But the fundamental model of firefly is event driven, either messages, or blockchain transactions, or blockchain transactions caused by messages, right, all of those combinations, trigger events, and then trigger processing. And this is the really important thing that firefly recognizes is that that processing is about apps, not just the on-chain processing is important. Applications written by Node.js full stack developers who love react applications, mobile developers, back-end core Java developers, developers using ESBs and other integration technologies on premise. This is very important in the ecosystem. Firefly can be that connection point where you can say, I have a core system, it wants to invite some data and broadcast it to people, something happened on the blockchain, it turns into an event. That's the model. And then inside of the box, the sort of paddling feet of the swan under the surface here, you then have access to all of these things that need to happen for an efficient system. And by from the learning of lots of projects, and sort of several actually going to end up implementing all of these, we can make them a pattern, we can make them plug it all, and we can give you insight with user experience and West API, so that you can see what that's going on, and you can see under the covers of what's happening inside of the environment. And that's what we wanted to take you through today. In this section was just like building up to this picture so you can see what we're talking about messages, groups, batches, operations, data transactions, subscriptions, events and pins, what the heck we're meaning, but actually what you care about is messages and events. So, hopefully lots of contributors on the on the on the line as well and obviously you're on extending and enhancing the internals of these as well and plugging in as well, which is another key point. Okay, how did I do Nico is that just about on time. Perfect. It was an excellent landing. All right. Thank you very much for that Peter.