 presents. Okay, thank you for inviting me for this presentation to present our research group. And I'm going to talk about the integration of hyperledger fabric and ROS2 for distributed robotic systems and some results of our real-world experiments in our lab. And so for my presentation, I'm, I will start with a small introduction about our research team, tears, in University of Turku. Then I will give an introduction of hyperledger fabric and ROS2 and then go over some real-world examples and scenarios. And then finally talk about the results that we had. Turku stands for Turku Intelligence Embedded and Robotic Systems and our research verticals can solve edge intelligence, resilence autonomy, distributed systems like swarm intelligence and collaborative autonomy and so on. And in this presentation, we will explore how blockchain technology can be integrated with advanced robotics to enable trustable data sharing and robot control. And we will examine real-world examples and scenarios of scalable, collaborative and reliable robotic systems and discover their potential in various industries. I know you know, but I'm going to give some introduction a lot of fabric also. As you know, hyperledger is one of the biggest platforms in the permission of blockchains and it is an open source project based on Linux Foundation. It provides projects and frameworks to businesses and developers to build the blockchain networks and applications. And it aims to ease the collaboration between developers, enterprises and businesses in the field of distributed ledger technology. And as the advantages, it offers a robust set of features that makes it suitable for integrating blockchain into distributed robotic systems. Its permission network architecture ensures that only authorized participants can join the network and access shared data. And the modular architecture allows for flexibility and customization, enabling developers to tailor the blockchain network to their specific requirements. And it supports smart contracts, enabling the execution of automated businesses logic and the blockchain. But if I want to talk about the robotic side, I'm going to talk about the ROS, which is basically it's going to be the main talk today. And what is ROS? Before the introduction of ROS, robotics development was often lacked a standardized framework. Developers had to build their own custom solutions for communication, coordination and integration of robots components, leading lots of efforts and collaboration among researchers and sharing the codes and resources were challenging. The absence of a unified platform made it harder to prototype, test and deploy robot applications, increasing developments time and complexity. ROS revolutionized robotics by providing a common framework, fostering collaboration, promoting modularity and simplifying the development process, accelerating the advancement in the field. First, ROS developed, ROS is also known as ROS1. It was initially released in 2007 and has since become widely adopted in the robotics community. It provides flexible and modular framework for developing robot software. It allows developers to break down complex robot applications into smaller reusable components called packages, promoting code modularity and usability. It uses published and subscribed messages system, which we call them topic, to enable communication between different components of the robotic systems. And it was primarily supports single host systems and is mainly used for research and prototyping applications. It has a large ecosystem of packages and tools developed by the community making it easier to leverage existing functionality and collaborate with others. If you see, I put an example of how ROS1 works and how robots can communicate with each other. In the presence of ROSMaster, we can have publisher know that subscriber know where the data can be transferred through the ROS topic so that multiple robots can easily communicate with each other and transfer the message between each other. Here you can see the ROS communication where, for example, camera can be as a node which publishes its data on a topic. We can also have a node for processing the camera data and one node for, for example, recording the data. And processing node can subscribe to the camera topic and publish its processing results on another topic. And on the other hand, the recording node can subscribe to these topics and record all the data received from the camera node and processing node. But ROS1 has also limitations. It doesn't provide native support for real-time systems or safety critical applications. Package developed for one specific ROS distribution may not always be fully compatible or works seamlessly with other ROS distributions. So sometimes we need to change the whole codes and whole codes for other distributions. It was primarily written in Python 2, which has become an outdated version of the language. And also it was designed for research and development purposes, rather than for deployment in security sensitive and security sensitive environments. It lacks of native support for external networking and distributed system. It was designed for single host systems and communication between ROS nodes was primarily intended for local machine communication. With the presence of these limitations, ROS2 emerged. ROS2 is a modern framework for building robot applications. It improves open ROS1 with enhanced performance, scalability and support for real-time and safety critical systems. ROS2 promotes code usability, collaboration and interoperability, making it easier to develop advanced robot applications. With distributed architecture of ROS2, it enables communication across multiple machines and its modular structure separates components into nodes for improved flexibility. And message-based communication in ROS2 uses a published subscribe model for data exchange. With this architecture, ROS2 provides a powerful platform for building advanced robot applications. Here, if you can see, basically topics in ROS2 are transferred in this way where we can have multiple publisher and subscriber nodes. The biggest change that came with ROS2 was the selection of the DDS middleware for the communication layer, which strengthens the communication between robot components with its decentralized public subscribe architecture. And we don't need any ROS master in the ROS2. ROS includes mature open-source libraries to be used for navigation, control, motion planning, vision and simulation purposes. The 3D visualization tool, like R-Waze, named R-Waze and the simulation tool, Gazebo, are seen as useful tool for robot developers. And also apart from this, open-CV libraries are used for detection purposes in ROS2, which we have also used in our scenarios. Now let's move to some of the real-world examples and some scenarios that we had regarding integrating ROS2 and fabric. Over the past two years, we've been working on integrating fabric with ROS2 using both Go language, Golang, and more recently with Node.js. For ROS2, both Go and Node.js client libraries are community backed, as the main supported languages are Python and C++. So if I want to talk about our first scenario, a real-world example, a hyperledger fabric DLT network is interfaced with ROS2 for managing autonomous robotic fleets. In this scenario, we introduced the framework for integrating ROS2 with a hyperledger fabric for distributed robotic systems and also analyzed the impact of integrating fabric in robotic systems with a performance and a scalability study. And experimental proof of the proposed framework for an inventory management application with ground and aerial robots. Here, this diagram shows the framework architecture and key components. The fabric network is hosted in a cloud with a Go-based web interface for visualizing the ROS2 data stored in the different channels and the command interface for inputting user instructions. Fabric is hosted by a set of organizations, contain peers, certificate authorities to verify their membership in the network. Existing of the private data channel is one of the key components differentiating fabric from other blockchain solutions here. In this framework, robots are members of the fabric network and also shared ROS2 network. You can also see the important applications of smart contracts for industrial robots like ledger can be used to record sensor data, ROS2 data type, or so on. And web application can be used for visualizing data and sending comments to the robots and so on. If I want to give an algorithm example of an application for such purpose, you can see this algorithm. Another benefit of DLT in addition to demutable and auditable records, as you know, in the building security features and identity management, which can be exploited. For instance, in interfaces for users for control in their individual robots or fleets, any collaborative decision-making process can be implemented through smart contracts, which we are basically using to ensure that all robots obtain the same result. This is applicable to role allocation or resource distributed problems. For the experimental part of our scenario, you can see the implementation diagram of the different software models running in different nodes here. We have used the ground robots, EAI dashboard platform, and the UAV X500 quadrature frame embedded with PIXHawk 5X flight controller running with the PIX4 framework. For this experiment, we used six optitrack cameras for navigation of the robots in our arena. Robots were running ROSNITIC under robot 18, but the localization and object detection were all running in ROS2 foxy. We have used ROS1 bridge to forward data from NAITIC to foxy from ROS1 to ROS2. Fabric applications were running onboard the robots, but connect to the peers running on the separate computer in the network with the same Intel Core S7 processor. And for the object detections, we used your looks, and Fabric has been brought up for secure data management and robot control. We used two smart contracts, one for storing the pass tracking and one for storing the location of the detected objects in the asset. And for each robot, one application with different functions for controlling the assets has been used. And we have used a series of shelves set up in the room of 40 square meters containing different objects from the cocoa data set categories. And for this scenario, we have checked and analyzed the CPU and memory usage during the experiment to see if using an integrating Fabric in our system would affect the system or not. And we have also checked the YOLOX usage. And the CPU usage is in blue and memory is in pink. So as you can see, having Fabric for secure data sharing is negligible. And so the proposed framework can be adopted in a wide variety of robotics platforms. And we have also measured the latency of the data storing in the blockchain when the application nodes is running in the robots connected to the computer is running in the order and appears on the Wi-Fi. And here you can see the distribution of the latency of the main four settings where data from 15 seconds is accumulated and over 200 hertz of ROS2 data being pushed into the smart contracts. We were trying to try this with different batch timeout and maximum message to see the difference latencies also. And as a second scenario, you can see here the blockchain is used for high-level mission commands, predefined robot paths for triggering multi-robot cooperative actions based on chain recorded robotic states like docking and landing the drone and switching operation modes for active and deactivating the ultra-wideband tokens. Detected objects are also stored in the blockchain. With the ultra-wideband, I need to give an introduction about that also. What is it? Ultra-wideband technology merged as a robust solution for localization in robotic systems, especially for GNSS denied environments. It can provide centimeter levels of accuracy at low cost. It also has the advantage that it doesn't depend on environmental conditions, such as light or there is an dust like what would happen with the cameras and lighters. It is also more accurate than Wi-Fi or Bluetooth and less prone to interferences. Here you can see ultra-wideband is a radio frequency technology that is used for indoor localization, since these devices are more affordable than other indoor localization solutions like off-the-track cameras that we used for our previous scenario is attracting increasing attention. Anchors and tags are two types of nodes used in ultra-wideband localization to obtain the positions of the tags in most anchor-based scenarios. The locations of the anchors must be known. If the coordination of the fixed anchors nodes are assumed to be known by the system for our calculation, and if they have enough fixed nodes and also the location of that coordination of them, depending on the ranging method implemented, the tags can be estimated their distance from each anchor and calculate their own location using different algorithms like TOF kind of lights, which in our case is that. So for this scenario, we plan to we plan the path for our telodrome and dash goal to inspect a warehouse-like environments and store the information they get about the objects, store them in the blockchain. We utilize the fabric smart contract for locking historical data and collaborative decision-making for drone landing, and we also utilize ultra-wideband localization for following predefined trajectory while the docking smart contract triggers the activation of anchors in the ground robot for more accurate relative localization while docking. The experimental platform of this scenario is consist of the commercially tele-MAV and ground robot. As you can see, the drone is equipped with the ultra-wideband module for localization. We also utilize the drone camera for object detections with the camera fitting available to web sockets connected to a controller computer. The ground robot EI dash goal is equipped with the sensors and camera and also real-sense T265 real-sense camera for the ego motion estimation. For this experiment, we have four ultra-wideband nodes modules with custom firmware. They have been deployed for robot localization and also we have five extra ultra-wideband nodes which has been used on the dash goal platform for more accurate docking of the tele. For this experiment, like the previous one, we have the objects from the cocoa dataset categories and the arena is 40 square meter long. So on the software side, the dash goal robot runs trust melodic and the remote and the main driver. Localization and object detections are all running in ROS2. The fabric application running onboard the robots are connected to the peers running on a separate computer and network. To forward the data, we have used bridge like previous one and to obtain the camera image of the dash goal at a frequency of 30 hertz. Even though they are forwarded to the object detectors at five hertz, the USB cam package has been used which is available in both melodic and proxy. To implement the different parts of the system, we have used goal programming language whenever possible has been used to increase the potential for integrate change between the different parts of the system. We have five smart contracts implemented in the system, two for restoring each robot's past history and two for restoring the locations of the affected objects by each robot and one for updating the battery level on the teller and also landing decision making for both robots. One application containing different functions such as creating assets and reading and updating and changing the assets has been used for each robot also. This figure shows the trajectories of the robots during the experiments and towards the mission ends as it can be seen after more than three minutes the positions converge when the telodrome lands on the dash goal and the bottom figures show the distribution in the time of the object detection detected by the telon dash goal. And we have also analyzed the memory and CPU usage by fabric and your looks and we got the same result which a usage of fabric is negligible and we have also analyzed the latency and this figure shows the latency distribution of five smart contracts where we had over 200 hertz of raw data stored in the average. And if I want to show it so it'd be more visualized for you so the idea is that both tello and dash will start to explore the place and whenever they're exploring whenever the battery level of the tello goes beyond the threshold then the docking command is sent to dash goal and tello so they both go to the landing position and while they both arrive to that position their anchors on the dash goal activates so the tello use those ultra wide band anchors on the dash goal to land so we have more in the accurate landing. So this is another scenario with the same idea but using the fabric as smart contracts as running the role allocation to determine the corresponding roles for each node shown here. We have four ultra wide band localization of the system we use both timer flights and TDOA to minimize the disadvantageous and the hyper ledger fabric aims to secure data management and robot control. The implementation has two smart contracts one contract to store the past history of six mobile ultra wide band nodes and the second one that implements the role allocation algorithm and saves the roles in the network. So the core idea for the role allocation is to keep the to keep the passive nodes toward the centroid of the system inside the convex envelope and the active nodes towards the outside. So the idea is that I don't want to go inside this but using the smart contract for role allocation with basically for this idea. And for the hardware we utilize the six jet son nanos jet sons sorry jet sons the jet son nanomodules and we have ultra wide band decal wave developed modules with custom firmware to enable timer flight and TDOA and transmission scheduling and the backbone wireless network is also implemented and we have also used six octetrack cameras for motion capture system where used and one global controller with quite seven processor for robot coordination and to run the localization algorithm. On the software side the jet sons are running Ross Melodic in a 18 one computer to serve as a coordinator around localization also a Ross one to Ross two bridge is implemented that before and Ross one for jet son and also for the hyper ledger public application and the role allocation algorithm is implemented in three different ways. One way was in Ross one Python node one way was in Ross two with RCL go online and another way was in fabric is not contract in Golan with the aim of comparing the performance later on all three methods are implemented with the frequency of five hertz and after comparing these different methods. We analyze the impact of the integrating hyper ledger smart contracts for the role allocation also and as a distributed and cooperative decision making process a Ross to go implementation is naturally faster than the interpreted Ross one Python node however most implementations are fast enough for meeting the frequency defined for the role allocation the additional hyper ledger fabric interface to the baseline go implementation increments its latency however it still remains fast enough to meet the five hertz requirement for the system and as the last scenario that we have recently worked on a hyper ledger fabric network region robots with the remote operation application the fabric network is able to transport data with low latencies in the range of hundreds of milliseconds mealtime data stream and be confirmed so hashes in the blockchain the fabric network allows for control access management and auditability of the teleoperation so we proposed fabric as bridging the interface in Ross to system so using the fabric as a bridge with Ross to system through an event based design and parameters are able to data transport setting and also we introduced a design implementation and evaluation of proof of concept for near real time robot teleoperation so fabrics from single to multiple systems and comparing to previous experiments and scenarios these experiments is toward near real time remote teleoperation and control it's more lightweight and computationally efficient fabric roster bridge and be ported from go to Node.js so despite go which required predefined muscle mixes Node.js doesn't require this and it's in general it's a more generalizable and performance fabric to interface so this is a system architecture overview and illustration of the roster and fabric blockchain interfaces the fabric network effectively acts as a data transport layer between individual roster subsystems a fabric network is optimized for cross organizational data exchange and secure and tempo proof data storage and sharing event driven architecture is the key feature of this paper research where roster fabric applications execute callbacks from both the Ross and fabric networks they are interfacing uh roster events uh events occur whenever for example new message arrives or new messages are published to a given topic this approach significantly reduces data transport latency at high network capacity and as a hardware setup we utilize for this experiment we utilize the DJI tele drone and also the mockup system in there and around eight to nine to five meters for the localization so here you can see the software architecture this system was implemented based on the roster galactic framework on Ubuntu 20 and robot control and localization we're running in host one data from opti track mockup system camera as you see vrpn client ross node and used for accurate position and attitude of the robot and the teleoperation application and the basic motion controller we're running in host two two fabric applications are then deployed in each host to interface with the roster system and the latest latency is also calculated since automating message is generated in one host until the matching velocity control message for controlling the teleprosy to the predefined pass arise from another host including both the physical network and the fabric network latencies which you can see is more or less around 0.5 milliseconds and as a conclusion we leveraged blockchain technology for multi-robot systems in various industrial applications for purpose of identity management data sharing security monitoring and multi-robot inventory management and consensus we showed that using proper leisure fabric and robotic systems provides a significant amount of built-in security by having a minimal impact on the utilization of the computational resources and in overall the integration of hyperledger fabric and rostering powers distributed robotic systems with enhanced capabilities and trustworthiness and of course we are a team and I haven't worked alone and I couldn't do this without the help of my friends and colleagues and so thank you for your patience and listening thank you so much thanks thanks Sarma and participants can ask questions directly or they can write in the chat so we have some questions Sarma in the chat so our first question is about lidar and machine vision can be used to complement each other is there any focus with ROS2 R&D to match the two technologies basically I haven't worked on lidars and machine vision so maybe I cannot help you with that but I haven't worked with that so I cannot say exactly but I can search and I can I can reply you later I can ask because some of our colleagues are specifically working on lidars and we have other colleagues working on machine learning and visions aside so I can ask them and later answer your question thanks Sarma and is there mesh technology used for ROS2 solutions no we haven't used mesh technology we haven't used for our use for our ROS2 solutions Sarma you mentioned one thing like identity management so how you are using this with respect to blockchain so the the basic idea is that all the robots that are in the system are defined before so we know the identity of them so they are kind of members of the network so that other robots do not come and be like Byzantine the system currently we are working on the scenario now that we have a Byzantine robot so we are more focusing on the identity management part and you are using the ABA system or contract of the fabric which is going to be published later so you can see the result but basically the idea until now was that so we all the identities were defined in the system and one more question on ultrawide band so what are the use cases and other use cases like what kind of devices we can connect with this and if you can tell us the frequency as well that's really helpful basically my colleague is working on ultrawide band but the main usage that I have used for all these experiments is only for the localization and it is the thing is mostly we use mock-up systems but as ultrawide band are more affordable and easy to use we have I have used them in some of my experiments to see the results also but for the other systems they can be I think they can be portable they can be connected to the different robots also but if you have any questions about ultrawide band you can connect my colleague Paola Torico Moron so she is the specialized person for the ultrawide band part mm-hmm thanks and about the standards so do we have standards on this right now or which which is the standard body for this particularly what I didn't get or ros too yes so is there any standard body who made all these standards or is it what is it like is it open source or yeah it is open source so we have currently we are working with ros to galactic so it's easily used and open source and it can be used for different robots and it's easy to use also we have all the package and things in our github repository if you want for example the ones that the smart contracts and applications that we have used for our scenarios are I can send you later in the github so it can be used and compatible to other robots and you can you can easily use them yeah that will be great if you can share the github yeah of course I can add the github repository in this slide and then later when I send the slides you when you are sharing that you can go and check yeah thanks thanks so much David could you check any questions from YouTube I've got I've got a few comments to say so thanks Samo for for accepting invitation and for a great presentation you've done quite a good bit of work here and it's and it seems that you haven't really stopped that trying to integrate the the fabric with ros too so you've actually done some analytics work on on kind of how hyper ledger and fabric would would actually work in these environments and so the and I and I particularly like the way that you have used smart contracts it seems to kind of support the event driven or or enable the event driven kind of nature of of the work so you're you're I think triggering these smart contracts based on the events that happen on the on the ros too site right so that that's probably would save lots of resources in the smart contracts just being triggered when they are needed rather than constantly being kind of monitoring the system right and so the the part with the smart contracts which is kind of the the most interesting part for us in this group kind of trying to see how people are using fabric and chain codes and so on so one one of them particularly was interesting to me which was the decision making contract based on the battery level I think yeah you can go back to that to the slide because so some of the smart contracts are are are essentially about writing kind of data on the ledger right just but this particular one was about actually kind of making a decision so one thing that I that I'm a bit confused about is that so so this is a decision making that is supposed to be taken in a distributed manner so there should be some essentially entities that don't trust each other right and now they want to make this decision together without trusting each other so can you just elaborate on that scenario like why wouldn't they trust each other and what are these parties why wouldn't they trust each other yeah so the thing is we are okay for example here dash go on tell or not connected to each other they are not aware of each other's condition so smart contracts only checks the topics that is getting from the tell which is the battery and whenever it gets the topic that the battery is lower than a specific it just published the topic so the rust so on the other side in the dash go which we have the smart contract also when it gets that topic it's go to that task I mean goes to that position so basically they they do not both of them are from parts of the network but they are not aware of each other they just gets like publish and subscribe they just subscribe to topics and publish and with these they just communicate I don't know if I get your question right or not or yeah yeah no no so yeah maybe maybe I misunderstood the the point here so so what you're saying is that there is a server server and and kind of a node relationship here so the idea here is not achieving trust using the smart contract it's more about yeah trust side is for trusting the robots because you know for example we are our scenario was using them in the warehouse environment if another robots try to make a false in the system or try to for example get the drone for landing or do some false in the system that's that robot cannot see any topic because those topics are going through the blockchain so it's not available outside that so only these two can can communicate and can see the topics another thing which which which we are currently using also is that there is another thing like beside rust 2 s rust has been developed recently which is secure rust 2 with that case if we implemented that also each the each robot would have some keys and with those keys none of nothing out of those robots would see those topics so basically it's like that but we have we are using it currently we are trying to use the s rust 2 with the aviation smart contract this time for this identity and trust things but basically yeah okay so now so if you want to talk about the kind of the setup of fabric that you have right so do you have multiple organizations do you have like so because these so these smart contracts are essentially probably you don't need a very complicated endorsement policy for them right so do do do everyone in the network have to endorse these smart contracts or only the robots which are which are which are going to involve in this so nothing else this smart contracts only in by these robots so no no one else I mean you can if you're in a system you can see the topics but you don't need to endorse the smart contract right so I can I can see like in more sophisticated scenarios like let's say in a factory of future a scenario where you might have let's say machines from different let's say sections of the factory trying to work together so maybe in those cases you might you might need kind of smart contracts that are that are essentially kind of bringing more of that the collaborative decision making kind of side of things to the smart contract because then then you have the problem of trust right so because now you have external sources of trust and how do you how do you trust people coming in and out I know that you're planning to use smart contracts for identity management as well but what I'm what I'm talking about here is more or more from the view of more complicated decisions that have to be taken and more people have to give an input so here your input comes directly from the drone and the drone makes a decision based on its battery level yeah right and okay okay yeah yeah so there's there's quite a bit of things to be to be done here in more complicated scenarios yeah of course I mean we are just trying starting this I mean I recently I just started working on fabric from a PhD I mean it's been one year and a half so I'm just trying yeah integrate I was not specializing yeah yeah and one last question so if you go to the latency figures that you had well actually like both latency and cpu utilization yeah so what is the transaction throughput here what numbers are you looking at the truth the thing is we had another analysis also but I didn't put because the slides were too much for checking the throughputs to see that if changing having different throughputs would affect this or not we had some tests with different patch timeouts and maximum message but beside that we got the frequency of I mean with the frequency of around 50 hertz was the thing that we got the optimal one but we had some stress tests to check the different frequencies and different utilization but I didn't put it here so with that optimal frequency that we got we got the minimum resources utilization yeah but sometimes that's really out of your control right because the transactions that come through your your your your fabric will depend on the number of drones you have so I assume these are all scenarios with one drone and one dash right yeah only for our case I mean it's not yeah yeah so I mean obviously for this case this is a good analysis but I wonder if this would still apply if you have let's say 100 machines that are quite latency sensitive so you want to process their transactions very very quickly and so yeah what I'm saying is that this might not be very generalizable because like I did lots of kind of these kind of analysis on better if my fabric network is going to kind of be able to meet the the latency requirement that I have in my use cases right which is telecom use cases and they're very latency sensitive and once you reach kind of a couple of hundred transactions per second things start to kind of change quite rapidly but then with the latency and CPU usage and so on yeah also when after recently because with our new scenario that we are using the one problem is that as it is accumulating the data after if if we leave the data to be transferred as it is sometimes the latency increases even even so it needs for example after three minutes or four minutes after accumulating the message it starts to go yeah and there's also so one thing is actually like just writing things on the blocks and the other thing is that if you have like a kind of a more sophisticated smart contracts that are making a decision based on a particular algorithm then if you have to wait for all the nodes to endorse your your your smart contracts outcome then you end up sometimes waiting for quite a while to for everyone to endorse the smart contract and so on so yeah yeah like lots of space for you to to work on if if you want to continue working with type ledger so yeah of course there are lots of parameters that we have to take into account and if you want to if you want to generalize it of course we have to use more than two robots we have to take into account lots of things of course you're right perfect thank you very much thank you anyone else any questions i have a question i have a question so um very wonderful presentation thank you so much for your time um my question is are you able to bring the electromagnetic computation into fabric i for now i have to search for that because i don't know if i can i don't know i have to frequent like the frequencies are you able to bring like the rf and all of these things into fabric yeah that's why i'm saying um i mean like checking them or well anything i mean i mean you're on there you're on the you're on the cutting edge yeah so i'm just wondering if you were able to bring electromagnetic computation into fabric uh actually we haven't tried no so i can i cannot give you another we haven't tried so maybe if you try i can give you that okay all right yeah thank you you're welcome uh anyone else any questions if not uh so thank you thank you salma for accepting our invitation and uh this very good presentation and thank you everyone for joining today and thank you so much thank you thanks everyone thanks everyone bye