 Welcome everybody to our weekly firefly community call. Just a couple of quick announcements before we jump in. Last week, one of the questions afterward was about a contributor guide or a process for becoming a maintainer and just wanted to let folks know that that is in progress. Yes, there is a draft PR open for the firefly repo that has a proposed process for becoming a maintainer. We'd like to get that merged soon and plan on doing so. Just want to give the existing maintainers a chance to look at it and comment on it though, one of them is out on vacation, so that will likely be merged on Monday. I see Dano just joined, he brought that up last week. So Dano, that was the process for becoming a maintainer. The documentation on that is going to be merged here very soon. Cool. I saw that Arun mentioned it too on another one of your calls. Or that was the hyperledger meetup. Yep. Yeah. Yeah, and then there's been questions about how do I get started? How do I become a contributor? There's documentation on that in the same PR with some helpful links on where to find good first issues and that sort of thing, so working on building out the community documentation there. The other thing that I wanted to note is, so we had a great meetup last week and there is another one tomorrow. Tomorrow's is aimed at the time slot that's friendly for hyperledger India and APAC region. And John, do you want to, I see you're on. I don't know if you want to comment on that at all, but I just wanted to let folks know about that as well. Thanks, Nico. Yeah, I think that we had an excellent turnout for the hyperledger meetup. I think one of the things that I have as a takeaway is we did it as more of a studio presentation. And I just want to make sure that we work out the kinks with that integrating with the YouTube live stream. If we do another presentation, which I'm open to, then we may want to just consider doing a direct zoom presentation versus the studio model, just to make sure that the video comes through clear and perfect. Yeah, that's that's a good call out I watched the YouTube recording afterward and was also disappointed in the absolutely crushing compression that zoom placed on the video. Yeah, well, I have some thoughts on how to make that better for for tomorrow and for next time around so. Okay, good. So this week, if you want to, you know, redo that in a, you know, future time and do it as a direct zoom presentation because those always seem to turn out perfectly. Okay, thanks. Alrighty, those are the only announcements that I had. So today's topic is firefly plugin architecture and configuration. So I'm going to go ahead and share my screen now. So, especially as folks are curious about, you know, how do I, how do I get involved in doing the firefly and where, you know, what does this look like to commit code to it. I think this will be a especially important topic. First of all, can can folks see my screen I think I am sharing it. I've got multiple monitors here though. It's the white black letters on white. Excellent. I hope you like my Google slides theme I spent a long time working on it. It's, it's very readable and high contrast sarcasm there I need I need to get my designer friends to make me a cool firefly Google slides template because we don't have one yet but that would be nice. So, yeah, let's jump right in. So you've probably seen this slide, if you've been around for firefly meetups, or we've probably shown it in other community calls as well. So, we're going to kind of dive into this a little bit more today, talking about, you know, how, how do all of these things actually work in the code will actually go look at some of the code and just a little bit as well. So we're going to kind of break this down so a firefly node is, but when we talk about a firefly node we really mean a machine or even like a group of machines, possibly you could have some of these things running on multiple machines that is sort of one unit in the firefly network that belongs to a member a member can have multiple nodes if they want for redundancy. Sorry, they can, they have a node they can have multiple cores if they want for redundancy and then for each of the infrastructure pieces, sort of the redundancy model there is left up to the particular example of that piece of infrastructure so for instance if you're using Postgres, it might look one way for you know how you scale Postgres how you make it redundant and solve for HA and those sorts of things. So, I think this you know this this diagram is sort of helpful. I've put together another one though to specifically look at Kate what is like if I'm thinking about this from the code, and I want to visualize it that way. What does that look like and so I've kind of built a stack here that I want to talk through so at the bottom of the stack you have the firefly core this is the like the firefly core is written in go lang. So this is like the main go process that starts up, read some config and then does a whole bunch of other things. So on top of the core, you have a series of interfaces, and this is sort of the base layer of like, these are all of the types of behaviors that fire that firefly can do and that you can do with firefly. So you can persist things to a data store. You can obviously do things with blockchains. So there is network identity, you can exchange data with other members of the network, you could store files publicly, and then there is event transport for notifying members of the network of events so these are, these are implemented as as interfaces and then there's kind of the next layer that is like specific implementations of those interfaces. So if, if you want to write a plugin for firefly. You can swap out, you know anything at this layer by implementing the, you know the methods that are expected to be on something that implements these interfaces, hopefully that makes sense. I will kind of keep building this up so for the persistence layer there, there's no, no sequel in there right now but we've thought about, okay at some point somebody who's probably going to want to know SQL database. So that would kind of slot in probably right next to this one right now there is a SQL abstraction layer though that lets you use different types of SQL databases, and you know abstracts the specifics of each one. And then I'll, as we as we go up. I'll show the specific ones that are currently there and implemented. It's worth noting that this diagram is representative of like the things that are there and the things that are in active development right now. So there are other things like tokens. I have not tokens are actually being worked on. I have not put them in this diagram because I don't know exactly where those building blocks will fit yet but I imagine there would be probably another whole vertical column over here that implements tokens. I need to, I need to chat with Andrew someday about how that fits with the blockchain interface though and what the overlap is there. So if you want to build a plug in. You, you're basically going to write a thin layer of go code that implements, you know, one of these specific interfaces down here that knows how to talk to the particular micro service API database or whatever it is that you want Firefly to work with. So hopefully that makes sense. Another layer. Now we're going to jump out of probably in most cases, the, so all of these things are in the Firefly core go process so that's, that's kind of what we talk about when we mean the Firefly core so that's, that's usually all going to be a single process. We go up a layer, we're going to jump out of that process, most of the time now to a connector layer. Now the connector layers are optional, but for the blockchain nodes, each, each blockchain definitely uses one right now so we have each connect and work on quarter connect that connects is an active development as well. So, basically, each of these things is going to talk to the specific service that it's been designed to adapt to this interface. So, we'll just go up one more layer here so you can see the full picture. So then finally at the top you have the like the specific implementation or sort of we often describe Firefly as a micro cross. Sorry that should say micro services not micro processes. It's sort of a micro service architecture where you have a lot of different processes that have some sort of well defined API that's exposed, and they can communicate with other micro services in the system. So in this case our SQL extraction layer can talk to either PostgreSQL or SQLite. There's some thin client code here that talks to each connect, which then is designed to work with Ethereum nodes. Same with FabConnect. For identity, right now we have one identity implementation and it's right now is somewhat coupled to Ethereum. It uses on-chain identity. That obviously needs to change as we move to Fabric at Quarta to a more general purpose identity system, but the on-chain identity is what's there in the code right now. Right now the, so Firefly Data Exchange HTTPS is the name of the repo that this is the current implementation of data exchange within Firefly, but again all of these things could be swapped out. So if you had another application that could exchange data privately and securely, you could write a plugin. You could write some go code to put slot in right next to here that implements the data exchange interface and talks to a different system right here instead of the Firefly Data Exchange HTTPS. IPFS is used for public storage and that's naturally a really good fit there, but same case here. If you wanted something other than IPFS, you could build something that would just slot in right there. Also worth commenting on the event transport. The event transport interface is you could swap it out for something different if you wanted to as well. You know, in an HTTP world, web sockets and web hooks for HTTP requests, callbacks, makes sense as a way to communicate with other applications. So this is how if you're building a Firefly app, this is how your app would get notified from Firefly when something happens. Somebody sent you a message, you sent a message that got confirmed. There's all sorts of things. So right now, we've talked about in the past how you can create subscriptions to receive events from Firefly. This is the implementation of how those things are sent to your application. So if you wanted something other than web sockets, you could build some other thing and you could put in some go code here that sends events on some other transport other than web sockets if you wanted to. That's totally something that you could do. So hopefully that makes sense. That's kind of it in terms of the slides that I had to share. At this point, the next thing I'd like to do is go hop into the code base and actually show, you know, how do we configure these things. That's actually my next slide. And where are these, you know, if I want to contribute to this if I want to write a plug in. Where is all this at in the code base so that that's what we're going to look at next before we jump into that I just wanted to pause. Hopefully this this diagram makes sense. Yeah, I just want to pause to see if people have questions on this before we go actually look at code because if you're lost now you're probably going to be lost more later so I'll pause here for questions. I mean, from the connector layer right so is that I mean, are all of these already complete. Are they working connectors or it's yet to be built. Great question so so each connect is here it's done. If you if you run the CLI to create an environment this is this is the vertical slice that you get for the blockchain layer. Jim Zhang is working a lot on fab connect right now it is in progress. And I think it's, it's partially working but it's not totally there yet. Cordo connect is earlier in progress so these, these two slices right here are being worked on right now. Those are those are great places. If you have experience with fabric or Cordo, and you want to jump in and contribute to the code that kind of these these four blocks right here would be a fantastic place if folks want to jump in and get involved. And that would be go right. I believe, I'm not sure admit is on I believe Cordo connect is Java. Yeah, connect is being written in go though. Yes, that's correct. Both Ethereum and fabric connectors are written in go because the underlying libraries are mostly available in go. Cordo is all Java so our connectors will corner implementation is Java based. Thanks, yeah, we can. Yeah, yeah, great question. Thanks. Looks like rich has his hand up. Yeah, if you're just transmitting Jason objects, and you're using the HTTPS connector. Do you use IPFS or is that only when you transmit blobs of information. Good question so I to be a little more specific so you are doing a what which API call are you making when we're using the web hooks. So if we're using web hooks with HTTPS, and we're transmitting a JSON object or string. Does it go through IPFS, or does it just go straight through the connector on the right. So if the, if it was a broadcast or if it was a public message. It would use the payload would be stored on IPFS that the data would get hashed the hash would be stored on the blockchain. The payload would be stored on IPFS. If it was a private message. Then it will use the data exchange. If you are definitely presenting a blob I believe if. Yeah if it's even if it's just Jason it will still traverse the data exchange instead of IPFS. Thank you. Yep. Good question. Any other questions before we go look at some code and config and stuff. Okay. I just switch my desktop over here. Hopefully you should be able to see the s code now. Yeah, we can see it. Okay, cool. So this is an example of a firefly can config file. So I want to start actually showing you the config. Some of you have probably seen one already but I just wanted to, it's very relevant to this concept of plugins because there's obviously for each of these things, they're going to need configuration. And each one of them is going to be configured probably very differently than the other ones. So I just wanted to talk through a little bit of how that happens so there's, there's some stuff that's built into the firefly core that you can configure here like obviously the logging level. The HTTP, the API, the rest interface is sort of that's built into that base layer that kind of that gray box at the bottom firefly core. There's there's also an admin interface that you can configure. And this is used for. You can actually firefly has a concept of like dynamic configuration where if you're standing a note up it, you can. It's default. So this is like a default config that the CLI has generated so it stands a firefly note up in this pre init state. So it's basically just sitting there, going okay I don't have all the config I need yet. So I'm waiting for some config. And there's actually an API endpoint on this admin interface that you can post additional configuration to. I'm not going to get into all the mechanics of how you do that. If you're curious that the CLI code base does that you can have a look at what it's doing there but you through this interface you can actually post configuration to it. And there's another endpoint you can call to like get it to reload the config and reload all the plugins. There's some other stuff in here like where it will serve a front end from if you want it to serve a front end, you could take this whole key out and it will disable that. And then like the name of the node and its organization so that those are all kind of like core things and then the rest of this is our plugins. So, you'll notice how each of these things sort of corresponds to at this root layer, one of the, the interface layer blocks on my little stack diagram. So, starting with blockchain. What type of blockchain do I want to work with. And in this case it's going to be Ethereum, and then you define another key called the name of the type that you put there. And then, so you could have an Ethereum blockchain that uses something other than if connect if you wanted to if you wrote a, if we go, actually, you know, I'll just here I can slide back to here. I wrote some other piece of code here that was something other than if connect. We could put something here that's, you know, something other than if connect, and maybe provides a different way to talk to if connect. I'm not sure what the advantage of that would be but it's very modular. So it's there if you need it. I'm going to show you some specific ETH connect things and I'm actually going to show you where in the code will find these things a little bit so you've got a URL for each connect this is it's it's HP API. Just a side note these URLs may look funny because this, this particular config file like I said was generated by the CLI so it's designed to work in a Docker compose network, which is why everything has not IP addresses and things like a DNS name of each connect underscore that's a Docker compose thing. We have a an instance and a topic as well. So we can configure database stuff here again we define the type and then have a key that has the specific things that that database implementation needs. And so on and so forth. So down here public storage or using IPFS IPFS has actually like two different APIs that it needs. And then, and there's data exchange down here. So, I'm just taking a quick check on time. This is probably good enough on the config file for now, want to actually hop over to the code and show you like, if I was going to build one of these things, where, where in the code base would I go. So, just to kind of highlight how cool the, the plugins are let's have a look at postgres.go so this is under it. If you clone the firefly repo where this is a firefly core code base we're looking at here. So this is going to be under internal database postgres postgres.go. It's very short. We're basically just initializing a postgres driver here and defining a couple of different strings that might be slightly variable between different types of databases. And that's, that is about it. Likewise, if we look at SQLite. There's a SQLite.go. It's also very short basically does the same thing. It's basically just setting up the driver. If we look at config, not a lot of config here. So all of the, all the grunt work of all the database stuff is being done through a SQL common package here. And this is where we've defined all the stuff like, okay, I have this type. How do I get this type into the database? How do I get it out? How do I query it? How do I sort it? That's all implemented in all of these files. And then like using a different type of SQL database is, is pretty much as easy as just, you know, configuring it. Initializing the driver and then wiring it up to this, you know, sort of this common SQL layer that we have here. So as an example, we, we started with a different, we started with Postgres and a different database called QL. It was, you know, it's all written in Go. And it was kind of cool, but it had some quirks and we decided, you know what, we're going to, this has problems. It was really nice for unit testing, but not really a production quality thing. So, you know, SQLite is a lot more commonly used. So we're going to swap out QL with SQLite. One of the developers was able to do that in like, I don't know, I think it only took him a couple of an hour or two at most and he probably got something else done at that time as well. But it was really easy just to swap that out. So that was pretty cool. Look at one more thing here before we wrap up. So if you're going to write a plugin, we talked about how certain plugins are going to need specific configuration. There, there is, if we look at, let's see, internal config. If we look at config, this is where like all of the core configuration is defined and all of the, so these are like all the specific keys that you'll find in the core configuration, as well as all of their defaults. So if you're ever curious, hopefully, at some point, we have amazing documentation that will, you know, just show all of these and mark down somewhere. We're not there yet. It would be cool to even auto generate that. But if you're curious about what a default setting is in Firefly, this is the place to find it in config in the, in the config package. So this is, this is the core config. But what about for a plugin? So let's look at ETH Connect. We'll go into internal and go into blockchain, Ethereum. Look at ethereum.go. Sorry, this is, yeah, I want to look at config. Okay. So, so this is the, this is, these are the config keys for. ETH Connect. So basically, we're going to define all the known keys here. And you can see the same. So like instance topic, there's some other things that weren't set in config like batch size and batch timeout. These are all the things in the config file that are going to be down here at this layer. So we're defining all of the, all the keys. And then we also define defaults for them as well. And then here's where the defaults are actually set. All of the config is managed with the, the Viper library in Golang is kind of what is orchestrating all this config and sort of like rolling it up into all the different places that config can come from. So it can come from obviously your config file. When you run Firefly, it will say you could pass in a config file with the dash F flag, or you can override stuff with the environment variables as well. So you could prefix something with everything's prefixed with uppercase Firefly underscore. So I could say, you know, anything that could be defined in this config file, you're going to use basically underscore to separate the different levels here. So that's how config gets rolled up into here. And then in the actual implementation of the, the plugin. Basically all of these known keys can be referenced and that's how your plugin can actually read things so let's see if I can find an example. I didn't. Yeah, anyway, I won't spend time scrolling through code to bore you to death but basically this this is where those, those keys are going to be read. I'm sure there's some in here somewhere. Anyway, that is, we're at the bottom of the hour, and that was about all I had to present on this topic so hopefully this is a, it's a huge code base, and this was just like a quick fly by of like, in this, you know, tens of thousands of lines of code code base but hopefully it's a little bit of enough of a taste that if you are curious about like, going to see how things are put together. You at least have a picture, you know this diagram in your head of kind of how all the building blocks stack on top of each other, and just a quick little pointer into the code to know where to get started.