 Hello, good morning, good afternoon, or good evening everybody, depending where you're zooming in from, and welcome to the Hyperledger Foundation in-depth webinar with Splunk on observability and blockchain, a deep dive into Fabric and Basel. We are delighted to share with you this next Hyperledger in-depth webinar, and we're very excited about today's presentation. My name is Tomasz Sedej and I'm an ecosystem manager at the Hyperledger Foundation. Today I will have the chance to introduce our panelist Antoine and also take you through some housekeeping rules. So if you ever joined our webinars before, you know that we have some housekeeping. First, all are welcome in our global community and we are committed to creating a safe and welcoming environment for all. So please follow the Hyperledger Code of Conduct when interacting with each other or talking with others in the community as well. You can find our Code of Conduct in our website and also on our wiki. All Hyperledger Foundation member webinars are held under the Linux Foundation Entertrust Policy, which you can find on our wiki and also on our event page on our website. We are recording this webinar, so you're also welcome to come back and check out our webinar library where the recording will be available along with the slides. This webinar is also being live streamed on YouTube, so welcome everybody joining us there as well. Now we encourage these sessions to be very interactive and engaging, so please stay active and the more active everybody is, the better experience we will all have. So feel free to raise your hand, get muted and speak up. If you would prefer not to speak up, you are also welcome to use the Q&A box to ask any questions you may have or you can also put your questions in a chat either here on the webinar or also on YouTube and we will come to them as we go. If there are some questions that might not get answered since we ran out of time, you're also welcome to join us on our Discord channel and ask them there to get them answered and I will share this information after the webinar. So without further ado, I'm very happy to introduce Antoine Toulmy who is a senior engineering manager at Splunk and we really look forward to this talk, so Antoine, over to you. Thanks Thomas. I'm very interested. Thank you so much for your interest in me. It's a pleasure to be here today and to talk to the Hyperledger Foundation members on behalf of Splunk and I hope that if you have questions you'll stop me and ask questions throughout. Thomas is here checking the chat, don't be shy. Yes. We will talk about observability in blockchain and these are two different topics that are just coming together as a both mature and it's really a privilege for me to kind of introduce you to WT if I made today. Hopefully you've been exposed before. So a little bit about me. I'm a senior engineering manager at Splunk. I've been at Splunk for over two years. I've been working in blockchain. I'm also a contributor on the open telemetry project which I will talk about today. That's a little background. I'm based in San Jose, California. I'm active on the Hyperledger Discord. I'm a member of the Metric Committer, sorry, of Hyperledger Bezu and I've been contributing lately to Fabric. So we have a lot of work in this hour because we will be talking about different things and trying to combine them. We will talk about observability which is a mouthful and explain how to correctly pronounce it going forward so you can do it at home as well. We'll talk about the type of signals that observability contains and then we'll go into open telemetry and explain how this project is helpful for that purpose. We'll go to blockchain. We'll talk about multiple computation and the different trade-offs that you have to make when you work with blockchain and the aspects of distributed systems which are amenable to observability and the different challenges that we see there. And so in terms of content, we're going to start with a primer on observability. We're going to talk about this application for blockchain and we'll take a peek into the future. So first off, observability, referred to as well as O11Y or Oli if you don't have time. And Oli just rolls off the tongue so it's much easier for people to use. I'll try to stick to observability today to keep it a little bit entertaining for you so you can count the time that I misnamed it. But yeah, Oli is how you would go about if you're at a conference in your lifetime. Observability lets us understand the system from the outside by letting us ask questions about that system without knowing its inner workings. So this is a definition I picked up from the open symmetry website. So it's interesting because it's allowing us now to know exactly what to expect out of the system without having to actually know its code. And it's a common and thorough probable standard to monitor software, to really debug and understand and make it transparent how it works. So we can have a common way to instrument the software, a common data model for all the data emitted by the software and a set of common naming conventions and semantics. So we know exactly what to expect and what are the different names that we're going to be using. A good example is a false name would be set as false.name as an attribute inside observability. So in terms of signals, if you're working with observability, you're going to have three big types of signals today which are kind of mature and maybe more will come later, but right now these are good data points that we're able to monitor and track. The first type of signals and we'll delve into that in just a moment is metrics, which allow us to record different types of data points associated with a particular quantity or latency time or number of records. So for example, a max of an event would be encoded as a gauge that will allow you then to really see over time the discrete value of an event. You could also use a bucket style system to get a histogram where you'd be able to arrange a set of data points to make sense of that. Or you could also use an aggregation method that would give you the sum of a set of data points of a period of time and then give you a sum. We also have the ability to record the execution of a particular program using the tracing system. A tracing system allows you to understand the different execution paths inside your program. Now we're going to weigh on that quite a bit today. We're going to talk about how this plays out with blockchain because obviously there's some use cases there which would span either one actor or multiple actors of a distributed system where you could trace those things. In this particular example, we are seeing three different web services which are communicating to each other and are using different execution paths that are correlated together according to the spans that form the trace. And finally we have the good old log. The thing that everybody's been used to, you know, traditionally in the origin of the term log is that you would take a piece of wood and throw it overboard to kind of measure the speed of your ship and then you would record the data bringing the speed of your ship in the logbook. When computer programs came to be, the logbook analogy got reused and there was actually logging what was happening inside of programs using some explicit measurements using some explicit messages. And so we'll talk about that and we'll talk about what's the evolution of that signal is all the time. Now for metrics. Metrics with open telemetry now have a data model which is now considered stable. It's been a long, long effort from the open telemetry community. So here in that particular case, I'll take you to this past page here and you can see in the open telemetry.io website there's a complete expose on those type of metrics. What is interesting here is to understand the different types of models. So here we have this high level diagram that I showed you earlier for the different types of metrics. We also have the different types of mapping that we could see from different models of from the open telemetry metric data stream to how it would be mapped to a time series so you can make sense of it. So if it's monotonic and it's a sum then it can become a counter. If it's not monotonic, meaning it goes up and down, it can become a gauge, it's a gauge map to gauge Instagram to Instagram. There's a couple aggregation strategies that are very much custom to different types of metrics you might care about. So for example, you could see a sum being aggregated over a short period of time and then you might see discrete data points there. Or you could just do an aggregation over the lifecycle of your complete program. So for example, a number of times someone tried to connect with you over peer to peer that would become a total counter of everything that's been not so far. And then interesting me, if you're looking with gauges, you can also have example of value in samples here where you sample the current value of that gauge over time. And a beautiful representation of a histogram where you could see like there's disparate values and events over a period of time and you're going to aggregate them. But instead of just taking the sum of them, you're going to cut a distribution of those values so you can make sense of them, so you can do advanced analytics based off that. There's one more which is currently experimental I'm discovering at this moment around exponential histogram, which is going to be able to create more recorded measurements in complex formats. I'm not sure we're going to talk about that quite yet. In terms of stability, this is the main types of data sets. What's interesting is this is now mapped to everything that you might be familiar with. If you worked with Prometheus, for example, there's some good conventions and guidelines to how this is mapped over to that model. You can see that right here in terms of compatibility and all that. So this is all coming together and it's starting to kind of come in mixed sense. And it's interesting because it was a whole lot of work from the open symmetric community to come up with a model that was stable and that's now delivered as a $1 version of that. And similar to what you would see with, you know, if you have analysis, there's the enhancement system where people can propose enhancements to the model. But it's highly codified and requires the community to come together and adopt and ratify these changes. Now traces. We talked about traces just earlier and traces are built around the notion of a span, right? And they're made of multiple spans coming together, being assembled to create a complete trace of execution. And a span is just a lapse of time of a particular execution inside a program. A span has all sorts of metadata to help you understand what is happening. So for example, this is a capture of a span from the open territory website as well, where you can see all the attributes of a span from the type of platform that's being used by the client, user agent. There is a few metadata value which are useful to kind of filter and understand the use case of the span, such as server here for the span client. It can also be internal. It can be client. So you can this way kind of discriminate between different types of span and kind of make sense of how they're being used. And they have all sorts of disinformation that is then being compiled together to form a trace, right? This is done by keeping track of your immediate parents or having links to other spans as you go in sequence. So when you put them together, it starts to look like this. And these are two different views of traces interpretation. This is kind of a view where you would look at it from a point of view of inside a program where you're looking inside the execution of a particular call here to insert the block inside a particular chain. And this is you talking to the tri-tree to see exactly how to get the value, going deeper inside the tri-tree, then running the EVM, trying to do an upcall, trying to get the state, et cetera, et cetera. So really getting to the bottom of where the time is being spent inside the program. Very good for all sorts of work around performance. In the same way, you can also map it between different rest services. For example, this is an example that is taken from our documentation at Splunk where we have multiple services coming together and trying to work out, you know, from a front-end perspective, all the time that is being spent talking to different services. What's interesting here is that we didn't start just at the service, we went all the way to, et cetera, database, for example. So database, much time is being spent working and talking to Redis, working with those different services. And you can see here, there's a little bit of an apps. It's getting a little too long because when we talk to the payment service, it's getting a little too long. So we're able to map out performance issues and see if there are any errors being reported in Red that tell us a bit more about the states in the execution of that trace. Now, when we talk about logs, keep this comical aspect going. Logs are well known for this should never happen. This is usually what you see in logs. The problem is that they're curated by humans. So they're only going to be as good as what humans make them. When you're a person and you're trying to do the best you can at looking at your particular log, you may think this should never happen, but the code changes, the circumstance changes, it's really difficult for people to be accurate in what they log all the time. So it's half a comment, half an actual instruction. Sometimes people are highly disciplined about this and actually put an error code in their logs, but most of the time that does not happen. And sometimes logs are also used incorrectly, or they're used the best way to do what's possible at the time, such as tracking time, for example, where we're going to just have a system out, print a line here, says so far migration downtime is downtime so far. So very clumsy type of logging, very difficult then to make sense of. And then you have to use all sorts of regular expressions to then get the data out. So the good news here is that there are two ways those logs can become more useful all the time. One is we can start metricizing logs. So you can see how many times of those occurrences you're seeing this particular log, for example, you could start also extracting value out of that log. It becomes even easier if you're using, let's say, Golang, Zapplog, and JSON outputs, so all the fields are being outputted separately. And you can also now associate logs to traces. So new logging frameworks will have the ability to tell you, hey, is there's a trace that is currently being recorded as part of the execution of this code? So as you log this, I'm going to add a trace ID and a span ID to this particular log record. When I admit that log record, then you will be able to also say, hey, I'd like to see the execution path of this type of logs and I can make them come together. So in short, if we start us on Oli, this is what the web is going to look like. Finally, we made it. Awesome. So how do we do that? We adopt open telemetry. Open telemetry has this broad mandate in this part of the CNCF, the Cloud Native Computing Foundation, to kind of come together and finally standardize all the different aspects of Oli, observability, from tracing metrics, logs, et cetera, having all these stuff figured out. Now how it works is, again, going back to the specification work that the team has been doing, and it's a whole lot of the specification where we talk about codifying logs, codifying traces, codifying metrics, all those things that we may take for granted needed to be kind of rooted into common definitions. What also works is if you're going to use any of the open telemetry agents or SDKs or anything like that, the whole community agreed on environment variables and configuration sets that are going to be the same cross-board for all the SDKs all over the place. So all those values, all those limits, all those ways of exporting and importing data are actually now set stable in the same across all components. So if you were to work with a C sharp program, for example, and decide to export its traces, you would be using the same environment variables that if you were to use it with a Java program, which allows you then to set a, let's say, Kubernetes set of pods with the same environment variables. And before you know it, all your data is flowing seamlessly to your system, so you can see what is actually going on between all those services. So this is a beautiful focus. It's a gain in time in operations where you may get its value out of existing software just by enabling a few environment variables without having to do a whole lot of leverage. So how it works too is using the open telemetry collector. So the open telemetry collector is a product that is published by the open telemetry project and it's based in a group program and allows you to then collect all the information from all the different signals and perform actions on them and then export them to a destination of your choosing. So if you were to deploy inside your environment, you might not want to go straight to the cloud, let's say with your signals, you may want to kind of have a point where you can apply back pressure, check the health, have some level of high availability, some things. So you would send all your data here to those receivers, so it could be a common fuse data, it could be the native CLP, GRPC formats that you would expose with different traces or it could be Yeager, which is HTTP based, for example. Those receivers would get all that data. Another example of receivers not featured here is just reading from the file, right, as it's being written to. Then you would batch this data, you would also manipulate the attributes, you could add information, such as the host, the Kubernetes cluster, anything you want to kind of characterize the data itself and then you would maybe process it so you would be removing sensitive data, you would be able to mask critical information, stuff, things and then you can send it to a destination where choosing could be any system that accepts this type of data, right, it could also just be a database where you're choosing or anything like that, or is free or long-term storage. So this collector is part of your strategy when you start using open symmetry to kind of take care of your data sets and transfer that data to a safe place. So that gives you a whole lot of life here in Highports. Now let's talk about blockchain. And I'm not even the author, like this is not even my idea, this is coming from Intvora, which is a major Ethereum provider of APIs. And if you look at the timestamp, it's coming from 2020, right, in December. So this is from Nicola Cochio, I'm liberally quoting him here. He is saying that if you use observability, it's monitoring, but then it starts to become more about why. Why is this happening? How this monitoring is impacted? And you want to start looking for monitoring to delve into debugging, to catching failures before they occur, and generate to answer all those questions that we don't know yet, we should be asking, but that we will be urgently asking when something goes wrong. So you can see already that there are blockchain providers out there who embraced observability, kind of get an edge in business to kind of have better operations so that they can have better coverage of this type of these cases. So I gave a workshop on Beesu two weeks ago, two weeks first. I'm going to reprise just a little bit of a few concepts there. So we can talk about exactly where observability plays out. And I'll be somewhat going over some of those concepts. However, if you want to see the workshops available on YouTube and have a link at the end of this talk, so you can take a look. So Beesu itself is a different client, which is actually a misnomer, it's actually a server, it's a peer-to-peer agent. It runs a single process, it's independent. So as a blockchain client, it's supposed to grow and be able to do everything you would want from a blockchain, right? So it's got all its data, get there from all exchanges, just sending collections, receiving collections, and it can integrate the chain for you. So you can ask it for the latest block, block information, collections in that block, all this information should be available to you because you're running a client and you don't trust anyone, right? So if you want to get all the data local to you, check them by the database of your client. What that does is that it creates a complex software stack. You're now building a number of network activities, such as the peer-to-peer interface, the RPC interface, the HTTP or WebSockets, or even IPC now. You have, if you're doing mining, another network interface for Stratum, and then you may be exporting if-stats of the WebSockets to the server of your choosing to report your health and your stats. So in effect, Bezu works as different components. The first one is it's a database. It's using works to be storage. It's have multiple stores, and here's why it has multiple stores. It has to store a number of different things, has to store the blocks, has to store transactions. They also has to store the trial of those transactions, the receipts of all the transactions for a particular block, the world state. So the accounts in the view of those accounts, and those accounts themselves have storage that also need to be stored and has to be cryptographically compliant using the tri-3 as well. So all this information has to be stored inside the different types of databases, and they all have to be indexed as well, so you can have fast retrieval of that data. And it's all using RocksDB, which is a file-based storage mechanism. Bezu is also a transaction pool, so it has a set of activities which consist of receiving collections in whatever orders being sent or propagated to them. You can send them over just an RPC, but you can also propagate them using a GOSCIP protocol based on depth particularly between different clients. That's how your client would take care of it. So if you're not mining yourself, but you send a collection to your client, your client says, thank you, I cannot do anything with this, but I can probably send it to the rest of the network, and maybe a miner will eventually get that in their pool, and then they'll apply rules to eventually order that into a potential block that will be up for mining with everybody else. And of course, Bezu is a network agent, so it's completely independent. So in the course of configuration, it's going to require a set of truth and few sets to make sure that it's able to communicate with others and check their data. So you never have your client blindly trust what's coming over the network. It's going to get some Genesis block because it's on hash, so it can check that whatever blocks it's getting going forward actually have the right parent hash. It's got its own consensus engines. It's going to be able to apply some rules about how to validate blocks. It could be that she's in the hash, click, to BST, all that, and then set of boot nodes to discover other peers. So it's able to connect to other peers and network and start working. So for example, for discovery, it's using a UDP-based messages first to boot nodes, then recursively to other peers to start connecting to more and more peers and field buckets into Kedamiya table, which is going to be filled with eight buckets, if I remember correctly, was on the 256. Yeah, using a distance algorithm based on the E node of the other peer. So here's an E node, right? This is the actual public key of that other peer. It's not compressed using the SC16-256K1 curve. And then you have here, you have the host, then here you have the TCP port, and here you have the UDP port. So those in those information is what you get in the boot node, or you can also use that as static. And then you can start having what sorts of exchanges like that where you start getting more and more of those. And then you wrench them and you try to not be eclipsed at that by just having a few few of them, you want to have as many of them as possible. So you're safe and secure. But it's also an equal client, right? So with TCP, you will be embedding your node identity with the public key. I just showed you earlier, you will be able to encrypt data that you send to another node and say, Hey, this is me. Here is me. Hello. And then you can negotiate support calls, such as if other exists, such as Whisper or IDFT, so that you can start exchanging messages. The life cycle of Hyperlegio Basu is you start, you see where you are, you load up your data from your files, you get new blood from your peers, check whether you're at the top of the chain or not. If you're not, then you start syncing and you reach the head of your clinical chain eventually, and you keep getting and dating new blood from your peers until you get to the top of the chain, which will then allow you to be in the new blood because now you caught up so you can check whether the transaction applies or not. If it does apply, then you mint this new block and you give it a parent hash of the latest block you know about, and you can either propagate that block and hopefully you get rewarded for that, or you can get new blocks from your peers and you also propagate them because you want to keep going and want to keep up with the rest of the network. And you keep going like this, syncing and creating yourself a situation where you want to stay on top of things so you can keep working through the life cycle of your client. Finally, it's also part of a consensus. So you have different organizations that will work together and so you would want to understand all the aspects of the consortium itself if you're inside an enterprise consortium. This is a particular example and we're going back to that. We have a consortium that is made of people with different clouds, different providers, different set of implementations and so it's important to understand how to work with others especially when you're trying to have a operator in the middle, trying to understand exactly how things are going in terms of infrastructure, collection rates, things like that. The base also works as just an RPC server, you know you can send it a little bit of HTTP like this of a post or you can use WebSocket IPC or just even a GraphQL API. So it's a complex API server, you can do a number of things, you can batch, you can use it with wallet to have metamask directly and all those options you will want to much work. So in recap, this is what we're going to go into now. If you're looking at Hyperledger Basu or any blockchain client really out there, you're now working with four different softwares come together in a Frankenstein kind of way where you have a database, a HTTP agent, a queue and an API server all come together into one single process. So a lot of it can go wrong, a lot of it is intertwined, a lot of it needs to keep up with the latest and greatest. It's extremely expensive and it's great for monitoring and tracing. So let's talk about that. If you look at the database and the way Basu is using it, ROG-CD metrics make a ton of sense. And also what makes sense to understand how the memory is being used if you're in Java, you have the ability to get all the garbage collection metrics from your JVM metrics. You will also want to understand the writer and index of tracing. So how much time do we spend in the indexing the data that is being stored and how useful is it. And to be more specific here, we've seen and I've seen personally a lot of time being spent by the Basu committers to try to understand exactly all the complex use cases around manipulating the multiple trees to store them on disk. And then more recently to also take care of the bonsai data structures. The bonsai data structure is actually cleaning the data over time to avoid having an exponential growth of your tree. But as a result, there are sometimes some interesting edge cases where bonsai might not work. So it's been interesting to see how the debugging goes and tracing here could be helpful. Or an API server tracing can help you to kind of map it on the status map like I showed you earlier. So you can use the HTTP headers and there are a slew of different standards there that came up that are starting to kind of come together now. So you have W3C headers, you have Yeager headers, you have B3 headers. It's interesting because it can capture the identity of the request. You can just add a HTTP header for that. But it can also capture the state. So if there's an error, then you can record that as an error. They can report this as the round trip of it. And it's interesting because you can also continue this inside the server. So you can continue looking inside where the time is spent when let's say you ask for a block, for example. So that's a question that you might want to ask yourself. Like if you're running Besu, what's your average latency? Is your JSON API server really spending a lot of time answering questions and where does it spend? It's most of its time, right? What's really expensive? If you get it from a point of view of message queue, so debugging your collection pools. And one of the things that Besu has done for the committers have played with is having better metrics. So understanding like, how big is my pool right now? How busy is it? And then you also have the opportunity to trace a collection by itself through the pool to a block. So trying to also add to a trace here, the collection ID so you can then map it and see how often it's being added to a block. For example, is it added often? Is it being discarded? What's going on, right? And then when you do mining with Besu, which is possible, you can also have metrics around solutions or active miners, which is otherwise kind of a blind spot for Besu. So you want to know how much activity you're getting on the network and are you really getting a good set of solutions or do you have a lot of bad solutions coming from miners? Finally, for syncing, so whenever you get new blocks and all that, having a way to kind of measure the time it takes to insert this new data inside your database is kind of interesting. And for peer-to-peer, that's I think the most promising field. We have some metrics already exposed by Besu around peers and peer counts, number of connections, disconnections over time, anything around discovery as well, number of time discovery was ran, etc. But having the ability to also trace messages between Besu clients would be great to understand like who is talking to whom and creating a high level view of the network itself. And that would be the future of tracing across clients. So here's a quick view of some of the business metrics. And so you can see we have different things that actually don't relate to each other. We have a block number here, which is current block number over time. We have the JVM. So we're able to really get a different set of interesting metrics from that view. And also on the flip side here is a set of fabric metrics that we're able to get from the Pomifist metrics exposed by fabric in the same way. So in this particular case, we can see the collection rate. We can see the count of industrial proposal duration. So how much time does it take to endorse number of entries written. So we're matrixizing the logs here, which is cool. We're also matrixizing the errors. So we're just counting the number of errors being reported. And we can see the current block number for this fabric requirements. So on that note, and for fabric specifically, because I talked about Besu a bunch, we had the RFC opened with the Hyperledger fabric community for open telemetry, which is voted in and merged into fabric RFCs. You can take a look here on this Github repo. We also have pull request for fabric Java chain code that was merged. Let me take a look at that for you. So this one is interesting because it's happening inside the chain code itself, the chain code is the smart contract. And so instrumenting the smart contract is kind of interesting to kind of understand this. We have some number of open telemetry metrics also that we can enable there to track the states and the metrics as you go about this. And the big thing here is when you invoke, these creates a particular span that is going to execute around the execution of your actual chain code execution stub. So when you invoke your stub, we already created an environment for you to start reporting the states and the value of that. What does that does for you is that it gives you a trace every time your chain code executes. So if you're able to configure your chain code execution to report traces, then you can see light up in your environments, your tracing environments, the traces of every time of those invocations, results of those invocations. For example here, if we see an error, then we'll be able to report that. And then we'll be able to tell you on average how much of the time it works. And sometimes when it does not, right? And you'll be able to also report if it's ending, you can see the latency of your execution. So this was an easy kind of add to that execution framework. But also we started looking at adding GRPC interceptors in Fabric itself. So Fabric is using GRPC to have ability to communicate between peers and between different elements. So there's a number of interceptors which are here already. And so what we did is we just added it in the upper geometry, go SDK. We have the ability to add interceptors which add additional headers to the GRPC message. And so by adding those, we can then actually understand the propagation of message inside this distributed system. Then if we report those traces, we'll be able to very easily create a service map of your Fabric cluster. So this is in progress because it takes a little bit of time to get the kings out, get the community also mobilized around this. And this is something I'll talk about in five times at the end around the next step on that. For Bezoo, well, I'm a community on Bezoo. So I could go a little further. We have a little bit of an update on that, documentation, sorry, on that with an open telemetry page. And you can just enable open telemetry using metrics enabled and metrics protocol equals open telemetry. You can use Splunk, you can use others to visualize all the collected data. And we have an example of how to configure a connector to get all this data and send it to the Splunk for example here. How it works is we have an open telemetry system which is a type of metric system that is being exposed here that actually creates an open telemetry and an SDK provider that is going to report on a political basis to an endpoint of your choosing that is being configured as part of your environment viable and also has the ability to get all the data. This is meant to be intangible between the promifuse metrics provider or open telemetry metrics provider. So a couple more examples. In the JSON RPC service, for example, we have the ability to create a span around any execution of any JSON RPC method. In that case, we create a service span where we give it an address of the host and port that is calling you. We give it the parent if there is such a parent as you know it could be extracted. And then we start the span with that. Once the span, once the execution of Vertex is done for the Splunker method, we either record the exception and we make it an error or we just end it. So what happens there is that we also, in that particular context, add headers to the response. So you're able to say, oh, okay, I'm talking to Besu. This is what it looks like. And I'm able to, if there are any request attributes such as the trace ID on the request that is incoming to the JSON RPC port, map it to an existing trace, record that as a parent, so we can actually coordinate this trace to any other trace from the system that is talking to Besu. And a fun one is pipeline pipeline is used to sync and to perform batch actions on the back end of Besu. So for this particular one, we try to create a generic span that we're going to use to trace the activity of a pipeline. And so we can actually see when we run this, the syncing activity of Besu, and we can see where it spends its time inside this. So the task span itself is started here. If there's any issue whatsoever, mark it as an error, of course, then we'll end it in the end. What's interesting is to eventually pass it down into more specific items, but we can do that using autoswantation for XDB, for example, so we will have to do as much manual work to give it all this metadata. Okay, so the interesting part here is that I talked a little bit about the interconnectivity between all those clients and we have trust and not trust scenarios, right? So if you trust everybody in your consortium, you would be able to just have everybody send all the data to one place. And then from that place, the operator could look at all the traces, all the metrics and actually make sense of what is going on and help you proactively fix your systems or debug why one of the actors did not get metrics or traces in time. But if there's no trust, you can also choose to only share some of the data. And this is what's happening here. We see different tiers. You can see only some of the logs. You can see only some of the metrics. You can just see the organization ID and a little bit of them, right? So this allows a consortium to kind of create a health dashboard for themselves when they can really see how it's fairing and whether there's any data that needs to be corrected and brought to the operators of the consortium. So let's talk about the future and bright future. Okay, sorry about that. So the future for me, and this is my view of it, is that we're going to see new types of activities and actions which are going to be enabled by combination of observability and blockchain. I'm going to delve into three different types. The first one is I see it as trace central. I just talked about trust and no trust scenarios. I believe that in the future, you will always be on the subway around executing and performing actions, and you won't have time to operate a monitoring of the building system. So you'll start working with actors that will dedicate themselves to get all your signals, the metrics, traces, even your logs, and start proactively reporting to your issues, which means that you will have a lot less in terms of work to do to keep up with the state of your network, and you'll be able to go much faster to destination as you see this approach. And even better, then you can coordinate all your separate trends between each other so there's no crushes. And this type of collaboration and coordination will become more and more common over time, and what would be great is if we talk all the same language, I mean start with standardized on this type of technologies, and we can really go quickly to create those consortium level activities. So I see that happening already with our existing spark offerings, for example, with our existing customers. The other thing that really excites me is I want people to start being able to use systems for free, and say you have a voting website, for example, and you're a user here, you have your secrets, you're performing actions on the websites, and you're kind of left to your own devices. It would be a free website for users to use, even for you to set up a vote, for example. And if you want to get insights in terms of metrics and traces, then you would pay more money to get additional data. And the kicker here is that this website, this SaaS provider would not need to build all the dashboard and all the back end of showing you all the activity of those users, which is pretty extensive and proper of your task right now. Instead, if they were to allow you to download a feed of all those metrics and traces, then you can create your own dashboards using your own solution that allows you to actually make sense of what's going on on the website. It's less work for the SaaS provider. It's more control for you because then you can swap between different providers and you can keep using your own dashboards for your own use cases. And you keep your IP to yourself instead of the dashboard you create becomes very specific to the use case that you have of leveraging the website for your activities. And finally, task providers. So, signing traces, checking the execution of a particular trace. If you work with the provider and let's say you work with a major commercial entity and you're sending them an invoice, then you will be able to see how the invoice is progressing through the system by checking traces. If you sign the trace, then you can for sure know that they're executing that in good faith and then trying to go about your invoice the right way. If you can do that, then you can also use also of newfangled technologies such as zero knowledge proof to see without sharing data, you can verify that the system is processing your claim and going about your execution correctly. So, I like this particular person on Twitter because always ask open-ended questions like this and they get some pretty good insights from different people. So, the gist of the discussion seems to be it's not yet a thing, it's not yet happening, we're not yet using zero knowledge proof to provide trust in execution of other programs. But I can see the conversions happening between observability and zero knowledge execution. So, further reading, infra on observability, here's a web blog post that I showed you earlier. We also have a blog post on Besu on the spawn.com website on Hyperledgeo being an observability pioneer and I kind of punted a little bit on our workshop. We have a full workshop on YouTube you can watch and you can learn to extend Besu, it's pretty awesome. So, if you want to read more, that's interesting there. Okay, so I talked a lot and I can't wait to hear from you all and love to hear your questions. Yeah, thank you very much Antoine, it was a really great and dynamic presentation and I really liked your analogies, you know, with the subway map or the lock. And so, to everybody joining us, please raise your hands and ask any questions you may have or use the Q&A box. We still have a couple of minutes left, so we can still address those questions. Doesn't seem to be any open questions, Antoine? Not always. So, I'm going to give you a couple of suggestions where to find me and how to work with me if you're interested in this and you happen to be listening to this on YouTube in a couple of weeks and you're wondering, well, that's great, but really what's in it for me and how do I participate in this? So, the Hyperledger Foundation is interested in getting better at how to execute on this particular agenda. There is definitely going forward an agenda to work on this as a community with Hyperledger and I've received some interest from people. There's one PR just a few folks that, you know, reached out to me. We're going to create a task force with Hyperledger to kind of discuss how we want to go about this in a more, you know, closely dated way, especially for fabric. And I hope that we will be able to then quickly iterate on this and create also some interesting technology sets for this. So, we'll be on Discord. There'll be a channel for that and there's a task force group. And if you're interested, chime in. I've got a question. Anton, I was wondering, using open telemetry and Beesoo, I could like get traces to see how information is going on different nodes from... Well, you know, it's like I work for one of the companies inside the blockchain. I'm not going to check on every other one, it's only mine. But maybe I need to know how the others are getting my information. I'm not sure. Yeah, so, and I think this is crucial to distributed systems and this is where I think we have to kind of keep innovating on this. If you go back to this diagram, which is I think we used to kind of represent this scenario, let's say you are number three, right? You're working in your Azure environment and you're only sharing some of this information. You would be able to keep yourself your own information because you want to operate your own cluster. You don't want to talk to anyone else. Maybe you just want to know. But as is well known with blockchain, right, in distributed systems, it's the only type of system where another machine don't know anything about can break your system, right? So, let's say org number one starts to mess up with its own metrics and their system is down, their host is no longer available. You want to know about this and vice versa, right? So, you would be able to come to a an agreement where you would share voluntarily some of the data, maybe not all of it, and you would be able to then create a dashboard, which would give you a view of all the participants at this level. So, this is actually a dashboard from Splub. It's using stretches around the data that is being shared by consulting members to kind of update dynamically. Kind of gimmicky, but bear with me for the example. To kind of update, say, okay, here, actually means I'm getting data from that. That's good, right? This data did not choose to share its data around this consortium. So, it's going to be checked as no. So, you can check here. You can see at the top level, right? Data health is okay. We're getting the logs. Blockchain health, we're getting collections. We're getting blocks. Everything's working as it should. Infrastructure health, well, we have to check. This Azure environment here is starting to report issues. Sorry, it's reporting issues now, okay? So, it's reporting data but showing issues. So, hopefully, what we'll be able to do is we can create open telemetry collector scenarios where you can route the data to your own. You can also route the data and some of it only to a central open telemetry set of collectors that will give you a view of everything at once. And we have examples that showcase that that you can use to kind of make sense. I'm actually, you know, a little bit on the spot but I can show you that. So, we have under our signal effects org under Splunk Auto Collector. We have an example here that shows you hotel logs routing which I built. And it's kind of interesting because it's showing a first Auto Collector which is going to talk straight to Splunk and the second Auto Collector which is going to be routing some of the data on me from this file log listening to without and then sends to another Auto Collector which could also choose to go to another entity could be going to Splunk and it's going to another Splunk could be going to provider of your choosing. What's awesome and what, you know, as a big open source advocate, this Auto Collector is open telemetry thing is all Apache licensed. So, it does not need to change based on your provider. It could be that you use Splunk today. You want to use a different provider tomorrow. It still works the same way. It's good. It's standard. It's helpful. It's a good investment into your future as a company to be able to operate and monitor and debug what's going on. Great. Thank you. Okay. Hopefully, this was helpful. So, a comment from Todd. Thanks, Todd. I appreciate your comments. I'm saying I did a good job. Yeah. Thanks again, Antoine. I think we'll need to wrap it up. We're almost to the top of the hour. And thank you, Anna, for your question and Todd for your comment as well. So, as Antoine said, if you're watching this a couple of weeks from now on YouTube, you are really welcome to reach out to us or Antoine and the slides will be available there as well in our webinar library. So, you can always check this one out and also reach out to us. So, thanks again, Antoine. You covered such a lot of useful information. It was really a great presentation. And thank you for your time. So, as mentioned and also as Antoine mentioned, we do have our Discord channel. And just let me share my screen. There we go. So, here you are. We're very welcome to join our Discord community and explore these channels related to hyperledger projects like Fabrik and Beisoudet upon us presenting, but also special interest groups, working groups, our regional chapters and much more. And you can find the topics that are of interest to you. Also, Hyperledger Global Forum is happening in Dublin, Ireland from September 12th to 14th. And it will be a great opportunity to meet in person as well. Now, thank you, everybody, for watching. Thank you, Antoine. And thank you, everybody, for joining us and for all of the great questions. And if you have anything additional, please feel free to reach out to us and we will be able to help. Thank you, everybody. And have a nice day. Bye.