 All right, welcome to the September 26 2023 Indie contributors call few things to talk about on the agenda. We'll see where the conversation goes. We are recording the session and we'll be posting the results after after the call. A reminder that this is a Linux foundation and hyper ledger foundation meeting so the Linux foundation antitrust policy is in effect listed on your screen the hyper ledger code of conduct linked on your screen is available. I posted the agenda link on the screen so people can sign in. And add their name anyone new want to introduce themselves on the call. Lots of people we know. All right. Any announcements to be added I know I I W is coming up the hyper ledger summit is coming up in San Francisco, both of those should probably be under the announcements. I I W is in two weeks, and I believe it's the hyper ledger summit in four weeks I will get those added to this. Hopefully people are going to those events we can get together in person and have these discussions. All right, any other announcements for anyone else any agenda items to be added to the meeting. All right. I did want to point out this, I should have looked up the item in any note and plan on there is a PR. Sorry, there's an issue that requests a PR be added to, I believe it's plenty. Not certain. I think it's fine. But basically to replace the earth's implementation of indy BLS signatures with the Indy version of BLS signatures so with the archiving of the earth's project in the BLA signatures crate rust crate was moved out of Ursa and put into indy as a repo and adjusted to put it there. We could really use that in the indy plan I believe is as I'm sure I'm not quite sure if it's not a plan if I think it's plan. It should be updated to use that and not use Ursa any further. It should be a pretty straightforward dependency change and then run all the tests and verify all the tests. So, if anyone is looking for something. Hopefully pretty easy to contribute to the project that would be a good one. I would look up. I'll try to find the PR number or sorry the issue number to replace it to identify this. If anyone wants to look up now and add it that would be great. That's where we are. I did want to ask about those that operate networks to give an update if you have any on progress towards indy upgrades. Lynn and or raid primarily. Yeah, I'll start. So the indy co test that upgrade to 20.04 is complete. Yeah, and then the we started on our demo net. And around some serious issues. I don't know if we want an explanation of what happened there here but it would be helpful. Well, have you and Wade shared it. It would be actually be helpful if you could. Yeah, I'd be happy to give some. I mean, but the, the demo net is. 47th of the way upgraded right now so we're, we're on our way we worked around the issues and are, and are continuing with our demo net upgrade and then we'll have scheduled to the main net upgrade as soon as the demo nets done so we're, we're making good progress and we're just a little bit behind schedule. We were hoping to do like a network every three weeks or so and we're, you know, we're working just a little bit behind that schedule but it pretty close right within a week of being able to keep up with the a three week upgrade per network so far. So, can you give a quick summary of the issues of the encountered on demo net. At this time, I did talk a little bit about the test net upgrade and the issues I had there, and then somebody said, Hey, is there a document that describes that and how to troubleshoot the issues that you had there. At a very high level, it seemed like if we tried to connect one of the issues was if we tried to connect to add a new node to the network, but that node was not able to connect to the primary node that that things went south it was caused a bad situation and so I did document that and I checked it in and I'm in some troubles getting the merge there it was approved but the merges hasn't happened yet, but I created a document so just so everyone knows that document with the troubleshooting stuff that that I promised last time is complete so then the demo net issue is a is a unknown as to what caused it what happened was I added a node to the demo net after doing the upgrade is normal part of the upgrade the did not remove the node from consensus the keys and all are still the same add the node back in with the same keys to as if the node had gone down and started back up but now it's 2004 and I don't think this is a 2004 issue but what happened when I added the node back in was the network almost immediately went out of consensus and the other nodes on the network began what I call a split brain thing that has happened several times to me I don't know that anyone else has ever experienced it and so I don't it's just kind of weird because I've seen it so many times but the split brain means that three or four of the nodes keep the same primary and then the other nodes decide that there's a different primary and they start looping around with few changes to fix their issues that they're having where they don't agree with the primary that the other ones have and so it's a it's like almost like two networks trying to communicate together on the same thing it's a that's something I believe I created an issue on and I've documented but we haven't ever been able to get in and fix it so I did try and spend some time this last week debugging and going through to try and figure out what what caused it and how it happened I came up with some ideas and tried to prove or disprove the ideas and one of the ideas that I came up with a shared with Wade and then sorry Wade but it turns out that it was just kind of circumstantial it seemed like the DDoS protection thing might have had something to do with it because I started seeing these kinds of behaviors it seemed like right after we had added DDoS protection and stuff but I wasn't able to prove whether that had any any correlation so you know take a couple weeks of effort probably to set up something and replicate the issue in house it did to get to the bottom of everything and figure out what's there because I think that it didn't happen at all on the test net but it did happen on the NDCO demo net and part of the reason that and I think it might have happened is because of the extremely large size of the audit ledger on the demo net it's a 1.3 million or something so let me tell you the work around in case anyone ever sees this essentially the work around is to get a copy of the ledger from 1 of the nodes is working when you get it back up and working when you get it back in consensus and everything's working well get a copy of the ledger from 1 of the working nodes unfortunately that means you have to shut down the node to get that copy and so you may have issues with your ledger staying up if you don't have enough nodes on it but that that's what I had to do I just shut down 1 of the nodes just stop in the node and then get a copy of the ledger and when I copy that you know almost full copy of the ledger over to the node and then start it up again it comes right up doesn't have any problems so there's an issue with with the catch up somehow it seems and so and it was several hours last week that I poured through the logs and thought about trying to go through it and I wasn't able to determine the real cause there so so is that then good best practice is to always start with a copy of the ledger when you're starting the node starting a new node for the demo net that's what I have to do and that's a that's not a bad practice if you if you're concerned about whether your ledger is going to come up or not is that's probably what I'll do on my main net just to be cautious is get the other ledger put it out there and then and then it comes right up you don't have to wait two or three hours for consensus and sorry for the catch up to happen and and it works and it seems to work pretty well so just to clarify Bruno from I did up here just to clarify the difference with this problem from the one we had when upgrading the test net is that actually you starting a node completely empty and it never managed to catch up that's it that's correct and what happened was I even though it went out of consensus it still looked like it was trying to catch up and I waited like an hour and 20 minutes but then it turned out somebody was trying to do a demo at that time and that messed them up and so I had to stop waiting to see if it actually would fix itself and so I I got in a lot of trouble but besides that the yeah I tried waiting and I and I went back as I was looking through the logs it it actually made it to 800,000 or so in the hour and 20 minutes it was it was moving and still moving and it might have finally caught up and returned everything back to consensus etc but it being out of consensus and everything went slow the response times were slow and so it caused a major demo to be not working and that was that was big trouble for me. Interesting. So since that is doable and it reduces the time for a node to begin it seems like it's a it's the right thing to do starting with a copy of the ledger from a valid node. Right it's a good workaround and still be great if we could find the bug but it right now that's that's what we have as a workaround is to Yeah, you get a copy of the data directory and to make sure when you copy the data directory over you change the name of the sub directory to be the name of the node and then you're good to go. It's just added added overhead having to just to do that. You got to get the copy to the to the Stuart and they need to load it on their system so there's you know some additional work there that needs to be done depending on how much access they have to their to their system. Yeah, it just seems like you would get that at the beginning of the outlet upgrade process get one copy always use that one even though things would have moved on by the time you don't need to get it right before the as long as you have the majority that's what it sounds like the majority of the ledger, you don't have to have it perfect right because it's always going to be behind. Yep, it's just the logistics of delivering the copies of the files and getting those loaded on the machines. Okay, awesome. Okay, so now you're back on track and you're just doing the fifth, sixth and seventh nodes on demo net. Yep, that's right. The last two notes that I upgrade the first two were where didn't work very well and then the last two have worked great so the work around is working great. I mean, I'm sorry I'm trying to clarify. So some nodes you didn't have to copy the data directory but some nodes you had to. I mean, when you're great you have something that's practically the same and that shouldn't be. There shouldn't be so much catch up I'm sorry I'm confused. So, as I understand it, when you start up a new node, it reaches out to all of the other nodes and builds up its data directory build up ledger it from the data directory and I think what Lynn is saying is if you have start from an existing one. It can process it and then continue to extend it to whatever is missing. Okay. That's right. Any comments from you wait on progress. You're you're mostly and still in the documentation phase. Yeah. From my standpoint the documentation for the sovereign side is done. We just need to start notifying a select few stewards and get them to get on board with updating their migrating their nodes. So I can sort of to start with at least one at a time, go through the process with them and see how that all goes. I fired a link in the chat to some documentation that Lynn did on the troubleshooting. So, the other thing that Lynn and I have also talked about are in the progressive of doing. Mostly mostly Lynn is getting the documentation that in DCO and sovereign uses for setting up nodes and everything documented and included in the indie node repository so that it's probably available for everybody and this is more of a single single copy where we can all contribute to it. The directory out there and I've got the AWS one mostly done and then I'll be working on the other documents to get those out there and then I was going to add the validator preparation guide as well I think. But it reformatting that to be marked down is turning out to be a little bit heavier than I thought but it's a it's good to get it out there so I'm going to continue through and work through that and hopefully get a lot of that done today but Is that a Google Doc. Google Docs right now but what we're going to do is make them as Markdown and put them in the indie in the repository. There's a, I use a Google extension called Docs to Markdown. Yeah, that's what I'm using. I use that to start with but the somehow I haven't formatted it right and then all the sub numbers get to be 53 54 instead of being, you know, one dot three or whatever it is and so that the numbering is the start of it and then, and it's some of its outdated so it's taking some time to get an updated as well so it's, it's, it's all right, it's just taking a little longer than I had hoped. I also found chat GPT helps with converting the things like that. It's the only place I'm using chat GPT these days is to format Jason into CSVs and markdown tables and stuff like that it's surprisingly good at it. So I did have one other follow on a quick question I see Renata you're on the line here. I know you have some expertise in looking at logs and debugging what happened on networks and why things happened. It, it, are you okay if I put together something and send gather some logs and and send you the stuff to see if you can help to figure out what went wrong with the split brain scenario I'm seeing. Hello. So, I'm not sure that my knowledge is actual right now. But of course, could you please share your logs and I'll check my back and help somehow. And small joke I tried to generate daddy method throw to be the chat. It's so stupid. It's absolutely awful for the specifications. That's funny. Okay thanks I'll put together as best I can what the scenario is and what I think happened and send you the logs and thank you very much for having. If you can have a look at that. All right, any more on network upgrades that was great that was lots of lots of good stuff thank you man and thanks Wade. Awesome work land on on getting the two networks, almost there. And that is. I assume that includes cat herding with the stewards and things like that right. Yes. We send out a notification say please update within the next two weeks and yeah. And then it turns out we have to be on the line with them to add it back in and we thought early on that maybe they could just add their own back on and to start back up but it turns out that it's that doesn't work you have a meeting with them because it is regularly something that they haven't that they've missed because it's essentially is adding a new note to the network with an upgrade involved so. Doesn't surprise me. Good stuff. Okay. Excellent. Okay. So, this is a close system summit follow up. I threw this on to the agenda and thought we could have a discussion about it so this is an open discussion I don't have. I'll start with a couple of questions and I see we're not as here which is awesome might might be able to help out so if we think about Indy on Bezu. I mean, at least a configuration of how you and guidance at the absolute minimum of how to run indy on Bezu I assume there would be code involved in how to do it for a, a, a, of an agent and areas agent talking to the network so that would be needed. I guess I'll start with a question and just see where we go and just open up everyone encourage everyone have conversations so if I have a Bezu network I have a bunch of clients that are operating. Presumably, I'm, it's at least a permission network. So we want public reads from it, private permission rights. So, the biggest thing I wonder about is that an enterprise has a bunch of objects that have to be stored files effectively JSON structures that have to be stored. Where would that data go. What's the, it has thought been given as to where that data would be put. Presumably it would not go on chain, or maybe it would. Would it go on chain would it go somewhere else what are what are the thoughts on that. We can use examples of other DAD methods used smart contracts deleted contracts. Yeah, this is that it's possible to keep objects inside smart contracts like a storage. So, of course, it should be in, in a chain. So, I believe, right now, to share our status. We are on the design stage and started some POC. So moments that we tricky moments, challenging moments that we see is a signature validation, because there is other ethereum signature that is not a D to five five 19. But we have to implement it. So it's not a blocker. In fact, there are some solutions. Anyway, we can implement it by ourselves just we don't want this idea because it's just cryptography. But anyway, it's possible. So right now we are looking at the different options and try to understand what we're going to do. Another challenging moments, of course, is a question of a format of the documents. So of course we can't absolutely duplicate the current API in video, but we think about making it as close as possible. And here we have a question about the storage, for example, because we want to keep to store the full DID document. Not just partly like meme, like a trip transaction and have a full DID document. But of course, every time, all the time when we try to change something to make it better, we need to change a normal API. So again, it's not something critical because we can do these changes to make it similar with the current API on the video level is the K level. And it will be okay for areas to update in video version, for example, without big changes. So our plan here is finalize some small demo, maybe, and just check in some specific moments. And after this, maybe create an open source hyperledger project to think together about this because right now we are close and dear so and brainstorm these topics and will be really, it will be really great to discuss it with the whole community. I sent a note at Hart Montgomery suggestion had heard back to Matt Nelson at Bezu, but I haven't heard back I don't know. I don't know him. So it was an out of the blue request but we'll see. I can Steven I can follow up with hard on that I'm going to talk to him later today. Okay. Daniel's queuing up some questions go ahead Daniel. This is mostly stemming from a general unfamiliarity with Bezu. So I had a couple of questions and I'll have to give credit to Sam for thinking up some of these questions as well. But so with Bezu and the proof of authority model. So like the more permission model for consensus. Does, is there anything like mining that is still required in that consensus model like is there any of that, you know, hash puzzle wasteful thing going on or is it completely different. It's completely different. We decided to use could be ft protocol. And yes, this one. So it's possible to do proof of authority for this consensus protocol with result gas price. Good to know. The other question I had was how closely related is Bezu to the Ethereum project is Bezu itself, like built off of Ethereum, as in like the chain code for it or is it just an Ethereum compatible blockchain and then if we, if there are close to close ties to the Ethereum project is there anything like if the Ethereum community decided to go and do something that wasn't really in line with our goals would we be stuck going along with that or what are the opportunities for divergence. If that's the case, I guess. I got it. Honestly, speaking with the community is our next goal. So speaking about Bezu projects, it's like an addition for Ethereum nodes, like private transactions, maybe you know about this feature for Ethereum. We have some additional teeny node, teeny addition for Ethereum node. And it's a Bezu. Yes. So for our project, we suggest to implement smart protocols, smart contracts, sorry, for deploying needs to Ethereum network. So it might be deployed with a Bezu network and for a clean Ethereum network, because we need Bezu for consensus, mostly for consensus part and very data specific. So basically you can use Bezu on the main net you could use a smart contract on Ethernet main but the reason we would use this is for the privacy and the permissioning. So we have a separate network spin up instances of Indie Bezu that are private and permissioned. Yep, right now we divided. We have a design a small PC. We divided authentication part and role model and SSI part. And finally, we can use smart contracts for a role model for trusty stores adding you know to throw trust, throw stored accounts, only for Bezu part. And if we can imagine a business case when we need to use Ethereum without Bezu, we can use just smart contracts for SSI and so for the AD and for an increase. And, and so your thoughts are that actually storing the full and on credit objects on the ledger is not actually like there is a way to do that, particularly if you're using your running your own network. Yes, it should be okay. It is a question about revocation. Tail files, as usual, but while we speak just about the documents it's absolutely okay storage like a full object. Yeah, okay. Revocation, yeah, you're right revocation becomes the issue but right now we're still the, the revocation entries, we have a relatively small still constrained by the size of the tails file. The tails file wouldn't have to go on the ledger itself. So, interesting. Other questions. So honestly, while we are working on some and design some moments, we are open for any discussions and we will be happy for any feedback. So it's not a project that we keep in secret just to be in a process of publishing our work. And during this, we will be happy for any discussions. Excellent. All right. Other questions. I mean, I, it looks to me like this would be a good way to go. But, you know, this is, this is the right thing. So design and POC spikes. Have you actually were not have you gotten, you know, I mentioned it would be sort of a layer I'm envisioning at least a configuration layer probably some code and probably an SDK for agents. So those would be the repos with the heavy lifting, it wouldn't be similar to that, that Bezu would be like Plenum, there would need to be a node wrapper around it in similar to any node today and then in the SDK. So is that the sort of structure of it you're seeing I'm just not exactly sure what the, what the structure would be have you gotten far enough in your design to be thinking about that. Mostly the same we don't plan some additional layers here. We just plan to have a network without consensus and business parts. It's just a smart contracts for Ethereum nodes. And speaking about SDK parts. We are going to integrate like a first MVP POC we firstly want to integrate it in current video and Indy SDK to make it possible to try in the current applications anyway. Without new specific tools as we like our community. Yeah, yeah. Okay. Awesome. Any other questions from anyone. Small addition to maybe activate our community. That's, there are, we understand that it is really ambitious project, and maybe not the first try. The main, why we are working on design and pure small pieces concepts right now because we want to be sure in this option. And the second item here is critical item. It is your feedback. So, right now I'm feeling like a blogger. But I really like your feedback to understand do really need it. So any comments. You're welcome. Just to, to, I guess, comment generally. I like the idea of being able to move to a, you know, more up to date a bigger community have a little bit more support in that sense. But it seems like there's some pretty significant challenges, especially with the migration from existing networks to to the new networks. So I think that would be a pretty significant challenge to overcome there. But in general, I'm in favor of getting us to a point where we have a better maintained ledger base to work from. So, yeah. It's a tough problem. Yep. Certainly the activity on Bezu the scalability looks impressive. And boy there's a big. There's a big community. We're not up. Here's a random question. I'm on the hyper ledger technical oversight committee. And last week, we had a presentation to enable the transition of firefly from a incubation project to a grad graduate project. And so I got exposed to a bunch of things about firefly. And I just wondered, is firefly a potential tool that could be used on the agent side as a way to talk to the ledger. So firefly enables is sort of a is known as a toolkit for connecting and allowing you to build your applications with a variety of ledgers. It occurred to me in the middle of that as I'm sitting here thinking, oh boy, we would definitely need a an agent side interface. Is there any possibility that or or does firefly have a potential of playing a role there. I'm seeing you looking up firefly so you probably haven't thought of this one before. I'm guessing that's a, I don't know, but anyway, I throw that out there as a possibility that's what fireflies intended to be. Whether it would help us in this particular situation, and particularly ideas of connecting to, and whether it could be used for two different networks let alone. Let's simplify the agent work for for connecting to an indie on Bezu, first of all, and second, would it actually give us flexibility in in multiple networks. It sounds like an interesting idea honestly for answering I need read more about hyperlens with firefly. So it looks like it's possible. If we speak about a new ledger, of course, it should be okay if we speak about agency then looks like we need to update agency for following this idea but honestly, I don't know I need to double check what is firefly. Let's throw that out there it is. Again, it looks like a very mature project it's got a number of you know it's very active. As I mentioned transition from incubation to graduate status last Thursday. It's got some maturity and whether that is another Lego piece that could be put into play or not I don't know I just. You know it's obviously kind of fits what they're trying to do whether it is actual a good one for this particular use case that's a different question. And Stephen and indie folks, hard on myself or other members of staff are happy to make any intrusion need to the firefly team if and when you want. So that's that's not a problem. They'd be excited to work with you I know they. They use dids in some capacity within firefly already some. Okay. All right, anyone with final comments or should we just wrap it up there. I have a quick quick question. Yeah. I stumbled upon something I'm going to put a chin the link in the chat, and I'm speaking backwards today. The other day, it seems like a, we're talking about basic as a replacement. Yes, or for indeed I saw that the other day and I was wondering if anybody has thoughts or has investigated because these sounds like is a non ethereum drop in replacement. It's written in the read me I haven't played with it at all and I'm wondering if it is any. Yeah, I should have. I should raise that and it's a good point. Dave McKay is someone we've worked with, and Vera did is his organization so he's worked with the Ontario government and done on that number of things with northern block and various other places in the community. And he directed me a while ago about the same idea which is to use a theory and bezu as, as a way to do a alternate to to Indy. And I assume this is what this link you've given a million was that work. I think so. I think so but he definitely has raised that idea, and, and, and begun a conversation about it and done a bunch of work on it so run out of this is something you might want to look into with with him. Dave McKay. Don't know about problem here I don't know why that name is there but Dave McKay is is the flatten. Yeah, he's definitely in the Canadian community and very good as a company in Canada he's, we've seen him a number of times. Yeah, so can in Bezu. It's not using a theorem. Yeah, but he is using Bezu. So, anyway, what is interesting but looks like exact exactly a copy of for the current India approach. So like skeleton of transactions without any validation. Yeah, it's a good idea we right now we think about having for the documents with verification methods and to add functionality of the methods. I think I would be a necessary component I agree with you. Yeah, just really interesting thank you. Yeah. Okay. I'll add that. Hey, any other comments. Thank you that was good discussion much appreciated. We'll go from there. And, as noted, feedback interested for DSR on the designs and POC spikes they're doing related to this. Good stuff. Have a good one all take care. Thanks. Thanks Steven thanks everybody. Yeah. Thank you. Bye bye.