 Yeah. Okay. I can start right. Yeah, so that's your life. Yeah. Okay. So hello, everyone. I'm Salma and thank you for inviting me to present. Okay. Thank you for inviting me for this presentation to present our research group. And I'm going to talk about the integration of Hyperledger Fabric and ROS2 for distributed robotic systems and some results of our real work experiments in our lab. So for my presentation, I'm, I will start with a small introduction about our research team here in University of Turku. Then I will give an introduction of Hyperledger Fabric and ROS2 and then go over some real-world examples and scenarios. And then finally talk about the results that we have. Turku stands for Turku Intelligence Embedded and Robotic Systems and our research articles consist of Edge Intelligence, Resilience Autonomy, Distributed Systems like Swarm Intelligence and Collaborative Autonomy and so on. And in this presentation, we will explore how the blockchain technology can be integrated with advanced robotics to enable trustable data sharing and robot control. And we will examine real-world examples and scenarios of scalable, collaborative and reliable robotic systems and discover their potential in various industries. I know, you know, but I'm going to give some introduction about the Fabric also. As you know, Hyperledger is one of the biggest platforms in the permission of blockchains and it is an open-source project based on Linux Foundation. It provides projects and frameworks to businesses and developers to build the blockchain networks and applications. And it aims to ease the collaboration between developers, enterprises, and businesses in the field of distributed ledger technology. And as the advantages, it offers a robust set of features that makes it suitable for integrating blockchain into distributed robotic systems. Its permission network architecture ensures that only authorized participants can join the network and access shared data. And the modular architecture allows for flexibility and customization enabling developers to tailor the blockchain network to their specific requirements. And it supports smart contracts enabling the execution of automated businesses logic on the blockchain. But if I want to talk about the robotic side, I'm going to talk about the ROS which is basically it's going to be the main talk today. And what is ROS? Before the introduction of ROS, robotics development was often lacked a standardized framework. Developers had to build their own custom solutions for communication, coordination and integration of robots. Components leading lots of efforts and collaboration among researchers and sharing the codes and resources were challenging. The absence of a unified platform made it harder to prototype, test and deploy robot applications, increasing developments time and complexity. ROS revolutionized robotics by providing a common framework fostering collaboration, promoting modularity and simplifying the development process, accelerating the advancement in the field. First, ROS developed. ROS is also known as ROS 1. It was initially released in 2007 and has since become widely adopted in the robotics community. It provides flexible and modular framework for developing robot software. It allows developers to break down complex robot applications into smaller, reusable components called packages, promoting code modularity and reusability. It uses published and subscribed messages system which we call the topic to enable communication between different components of the robotic systems. It primarily supports single host systems and is mainly used for research and prototyping applications. It has a large ecosystem of packages and tools developed by the community making it easier to leverage existing functionality and collaborate with others. If you see, I put an example of how ROS 1 works and how robots can communicate with each other. In the presence of ROS master, we can have publisher node and subscriber node where the data can be transferred through the ROS topic so that multiple robots can easily communicate with each other and transfer the message between each other. Here you can see that ROS communication where, for example, camera can be as a node which publishes its data on a topic. We can also have a node for processing the camera data and one node for, for example, recording the data. Processing node can subscribe to the camera topic and publish its processing results on another topic. On the other hand, the recording node can subscribe to these topics and record all the data we see from the camera node and processing node. But ROS 1 has also limitations. It doesn't provide native support for real-time systems or safety critical applications. Package developed for one specific ROS distribution may not always be fully compatible or works seamlessly with other ROS distributions. Sometimes we need to change the whole codes for other distributions. It was primarily written in Python 2 which has become an outdated version of the language and also it was designed for research and development purposes rather than for deployment in security and security environment. It lacks of native support for external networking and distributed system. It was designed for single host systems and communication between ROS nodes was primarily intended for local machine communication. With the presence of these limitations, ROS 2 emerged. ROS 2 is a modern framework for building robot applications. It improves open ROS 1 with enhanced performance, scalability and support for real-time and safety critical systems. ROS 2 promotes code reusability, collaboration and interoperability, making it easier to develop advanced robot applications. With distributed architecture of ROS 2, it enables communication across multiple machines and its modular structure separates components into nodes for improved flexibility. Message-based communication in ROS 2 uses a published subscribe model for data exchange. With this architecture, ROS 2 provides a powerful platform for building advanced robot applications. Here, if you can see, basically, topics in ROS 2 are transferred in this way where we can have multiple publisher and subscriber nodes. The biggest change that came with ROS 2 was the selection of the DDS middleware for the communication layer, which has strengthened the communication between robot components with its decentralized public subscribe architecture. We don't need any ROS master in the ROS 2. ROS includes mature open-source libraries to be used for navigation, control, motion planning, vision, and simulation purposes. The 3D visualization tool, like R-Waze, named R-Waze and the simulation tool, Gazebo, are seen as a useful tool for robot developers. And also, apart from this, open-cv libraries are used for detection purposes in ROS 2, which we have also used in our scenarios. Now, let's move to some of the real-world examples and some scenarios that we had regarding integrating ROS 2 and Fabric. Over the past two years, we've been working on integrating Fabric with ROS 2 using both Go language, Go lang, and more recently with Node.js. For ROS 2, both Go and Node.js client libraries are community backed, as the main supported languages are Python and C++. So if I want to talk about our first scenario, a real-world example, a hyperledger Fabric DLT network is interfaced with ROS 2 for managing autonomous robotic fleets. In this scenario, we introduced the framework for integrating ROS 2 with a hyperledger Fabric for distributed robotic systems and also analyzed the impact of integrating Fabric in robotic systems with the performance and scalability study and experimental proof of the proposed framework for an inventory management application with ground and aerial robots. Here, this diagram shows the framework architecture and key components. The Fabric network is hosted in a cloud with a Go-based web interface for visualizing the ROS 2 data stored in the different channels and the command interface for inputting user instructions. Fabric is hosted by a set of organizations, contain peers, certificate authorities to verify their membership in the network. Existing of the private data channel is one of the key components differentiating Fabric from other blockchain solutions here. In this framework, robots are members of the Fabric network and also shared ROS 2 network. You can also see the important applications of smart contracts for industrial robots like ledger can be used to record sensor data, ROS 2 data type or so on. And web application can be used for visualizing data and sending comments to the robots and so on. If I want to give an example of an application for such purpose, you can see this algorithm. Another benefit of DLT in addition to demutable and auditable records, as you know, in the built-in security features and identity management, which can be exploited, for instance in interfaces for users for control of their individual robots or fleets. Any collaborative decision-making process can be implemented through smart contracts, which we are basically using to ensure that all robots obtain the same result. This is applicable to role allocation or resource distributed problems. For the experimental part of our scenario, you can see the implementation diagram of the different software models running in different nodes here. We have used the ground robots, EAI dashboard platform and the UAV X500 quadrotor frame embedded with PIXHawk 5X flight controller running with the PX4 framework. For this experiment, we used six optitrack cameras for navigation of the robots in our arena. Robots were running ROSNITIC under robot 18, but the localization and object detection were all running in ROS 2 proxy. We have used ROS1 bridge to forward data from NAITIC to Foxy from ROS1 to ROS2. Fabric applications were running onboard the robots, but connect to the peers running on the separate computer in the network with the same Intel Core i7 processor. And for the object detections, we used YOLOX, and Fabric has been brought up for secure data management and robot control. We used two smart contracts, one for storing the past tracking and one for storing the location of the detected objects in the asset. And for each robot, one application with different functions for controlling the assets has been used. And we have used a series of shelves set up in the room of 40 square meters containing different objects from the cocoa data set categories. And for this scenario, we have checked and analyzed the CPU and memory usage during the experiments to see if using and integrating Fabric in our system would affect the system or not. And we have also checked the YOLOX usage. And the CPU usage is in blue and memory is in pink. So as you can see, having Fabric for secure data sharing is negligible. And so the proposed framework can be adopted in a wide variety of robotics platforms. And we have also measured the latency of the data storing in the blockchain when the application node is running in the robots connected to the computer is running in the order and the peers node via Wi-Fi. And here you can see the distribution of the latency of the main four settings where data from 15 seconds is accumulated and over 200 hertz of Rust2 data being pushed into the smart contracts. We were trying to try this with different batch timeout and maximum message to see the difference latencies also. As a second scenario, you can see here the blockchain is used for high-level mission commands predefined robot paths for triggering multi-robot cooperative actions based on chain recorded robotic states like docking and landing the drone and switching operation modes for active and deactivating the ultra-wideband patrons. Detected objects are also stored in the blockchain. With the ultra-wideband, I need to give an introduction about that also. What is it? Ultra-wideband technology merged as a robust solution for localization in robotic systems especially for GNSS denied environments. It can provide centimeter levels of accuracy at low cost. It also has the advantage that it doesn't depend on environmental conditions such as light or there is a dust like what would happen with the cameras and lighters. It is also more accurate than Wi-Fi or Bluetooth and less prone to interference. Here you can see ultra-wideband is a radio frequency technology that is used for indoor localization since these devices are more affordable than other indoor localization solutions like optical track cameras that we used for our previous scenario is attracting increasing attention. Anchors and tags are two types of nodes used in ultra-wideband localization to obtain the positions of the tags in most anchor-based scenarios. The locations of the anchors must be known. If we the coordinations of the fixed anchors nodes are assumed to be known by the system for our calculation and if we have enough fixed nodes and also the location of that coordination of them depending on the ranging method implemented the tags can be estimated their distance from each anchor and calculate their own location using different algorithms like TOF kind of lights which in our case is that. So for this scenario we plan to we plan the path for our tele-drone and dash goal to inspect a warehouse like environments and store the information they get about the objects store them in the blockchain. We utilize the fabric smart contract for logging historical data and collaborating collaborative decision making for drone landing and we also utilize ultra-wideband localization for following predefined trajectory while the docking smart contract triggers the activation of anchors in the ground robot for more accurate relative localization while docking. The experimental platform of this scenario is consist of the commercially tele-MAV and ground robot. As you can see the drone is equipped with the ultra-wideband module for localization. We also utilize the drone camera for object detections with the camera fitting available so that sockets connected to a controller computer. The ground robot AI dash goal is equipped with sensors and camera and also real sense T265 real sense camera for the ego motion estimation. For this experiment we have four ultra-wideband nodes modules with custom firmware. They have been deployed for robot localization and also we have five extra ultra-wideband nodes which has been used on the dash goal platform for more accurate docking of the tello. For this experiment like the previous one we have the objects from the Coco data set categories and the arena is 40 square meter long. So on the software side the dash goal robot drones trust melodic and the robot and the main driver. Localization and object detections are all running in ROS2. The fabric application running onboard the robots are connected to the peers running on a separate computer and network. To forward the data we have used bridge like previous one and to obtain the camera image of the dash goal at the frequency of 30 hertz. Even though they are forwarded to the object detectors at five hertz the USB cam package has been used which is available in both melodic and proxy. To implement the different parts of the system we have used GoPro programming language whenever possible has been used to increase the potential for integration between the different parts of the system. We have five smart contracts implemented in the system two for storing each robot's past history and two for storing the locations of the detected objects by each robot and one for updating the battery level of the tello and also landing decision making for both robots. One application containing different functions such as creating assets and reading updating and changing the assets has been used for each robot also. This figure shows the trajectories of the robots during the experiments and towards emission ends as it can be seen after more than three minutes the positions converge when the tello drone lands on the dash goal and the bottom figures show the distribution in the time of the object detection detected by the tello and dash goal. And we have also analyzed the memory and CPU usage by fabric and YOLOX and we got the same result which a usage of fabric is negligible and we have also analyzed latency and this figure shows the latency distribution of five smart contracts where we had over 200 hertz of lost data stored in the average. And if I want to show it so it be more visualized for you so the idea is that both tello and dash will start to explore the place and whenever they're exploring whenever the battery level of the tello goes beyond the threshold then the docking command is sent to dash goal and tello so they both go to the landing position and while they both arrive to that position their anchors on the dash goal activates so the tello use those ultra wide band anchors on the dash goal to land so we have more inaccurate landing. So this is another scenario with the same idea but using the fabric a smart contract as running the role allocation to determine the corresponding roles for each node shown here. We have four ultra wide band localization of the system we use both time of flights and TDOA to minimize the disadvantages and the hyper ledger fabric aims to secure data management and robot control the implementation has two smart contracts one contract to store the past history of six mobile ultra wide band nodes and the second one that implements the role allocation algorithm and saves the roles in the network. So the core idea for the role allocation is to keep the passive nodes toward the centroid of the system inside the convex envelope and the active nodes towards the outside so the idea is that I don't want to go inside this but using the smart contract for role allocation with basically for this idea and for the hardware we utilize the six jet son nanos jet sons sorry jet sons the jet son nanomodules and we have ultra wide band decal wave developed modules with custom firmware to enable time of flight and TDOA and transmission scheduling and the backbone wireless network is also implemented and we have also used six of the track cameras for motion capture system where used and one global controller with quite seven processor for robot coordination and to run the localization algorithm. On the software side the jet stones are running Ross melodic in a 18 one computer to serve as a coordinator around localization also a ross one to roster bridge is implemented before and ross one for jet son and ross two for the hyper ledger fabric application and the role allocation algorithm is implemented in three different ways one way was in ross one python node one way was in ross two with rcl go bonad and another way was in fabric smart contract in golem with the aim of comparing the performance later on all three methods are implemented with the frequency of five hertz and after comparing these different methods we analyze the impact of the integrating hyper ledger smart contracts for the role allocation also and as a distributed and cooperative decision making process ross two go implementation is naturally faster than the interpreted ross one python node however all implementations are fast enough for meeting the frequency defined for the role allocation the addition of hyper ledger fabric interface to the baseline go implementation increments its latency however it still remains fast enough to meet the five hertz requirement for the system and as the last scenario that we have recently worked on a hyper ledger fabric network region robots with the remote operation application their fabric network is able to transport data with low latencies in the range of hundreds of milliseconds real time data stream and be confirmed through hashes in the blockchain the fabric network allows for control access management and auditability of the tele operation so we proposed a fabric as bridging the interfacing ross two system so using the fabric as a bridge with ross two system through an event-based design and parameters are able to build a data transport setting and also we introduced a design implementation and evaluation of a proof of concept for near real time robot tele operation through fabrics from single to multiple systems comparing to previous experiments and scenarios this experiments is toward near real turning when the tele operation and control is more lightweight and computationally efficient fabric ross two bridge and be ported from go to node j s so despite go which required the predefined ross two nexus node j s doesn't require this and it's in general it's a more generalizable and performance fabric ross two interface so this is a system architecture overview and illustration of the ross two and fabric blockchain interfaces the fabric network effectively acts as a data transport layer between individual ross two subsystems a fabric network is optimized for cross organizational data exchange and secure and tempo proof data storage and sharing event-driven architecture is the key feature of this paper research where ross two fabric applications execute callbacks from both the ross and fabric networks that are interfacing ross two events events occur whenever for example new message arrives or new messages are published to a given topic this approach significantly reduces data transport latency at high network capacity and as a hardware setup we utilize for this experiment we utilize the dji tele drone and also the mockup system in there and around eight to nine to five mentors for the localization so here you can see the software architecture this system was implemented based on the ross two galactic framework on a window 20 and robot control and localization were running in host one data from oct track mockup system camera if you see the with the vrpn client ross node and used for accurate position and attitude of the robot and the teleoperation application and the basic motion controller were running in host two two fabric applications are then deployed in each host to interface with the ross two system and the latest latency is also calculated since automatry messages generated in one host until the matching velocity control message for controlling the teleport to the predefined pass arise from another host including both the physical network and the fabric network latencies which you can see is more or less around 0.5 milliseconds and as a conclusion we leveraged blockchain technology for multiple systems in various industrial applications for purpose of identity management data sharing security monitoring and multi-robot inventory management and consensus we showed that using hyperledger fabric and robotic systems provides a significant amount of built-in security by having a minimal impact on the utilization of the computational resources and in overall the integration of hyperledger fabric and ross two powers distributed robotic systems with enhanced capabilities and trustworthiness and of course we are a team and I haven't worked alone and I couldn't do this without the help of my friends and colleagues and so thank you for your patience and listening thank you so much thanks thanks Sarma and participants can ask questions directly or they can write in the chat so we have some questions Sarma in the chat so our first question is about lidar and machine vision can be used to complement each other is there any focus with the ross two rnd to match the two technologies um basically I haven't worked on lidars and machine vision so maybe I I cannot help you with that but we I haven't worked with that so I cannot say exactly but I can search and I can I can reply you later I can ask because some of our colleagues are specifically working on lidars and we have other colleagues working on machine vision machine learning and visions aside so I can ask them and later answer your question and mesh technology used for ross two solutions no we haven't used mass mesh technology we haven't used for our use for our ross two solutions Sarma you mentioned one one thing like identity management so how you how you are using this with respect to blockchain so the the basic idea is that all the robots that are in the system are defined before so we know the identity of them so they are kind of members of the network so that other robots do not come and be like Byzantine the system currently we are working on the scenario now that we have a Byzantine robot so we are more focusing on the identity management part and we are using the ABC smart contract of the fabric which is going to be published later so you can see the result but when basically the idea until now was that so the old identities were defined in the system and one more question on ultra wideband so what are the use cases other use cases like what what kind of devices we can connect with this and if you can tell us the frequency as well that's really helpful uh basically my colleague is working on ultra wideband but the main usage that I have used for all these experiments is only for the localization and it is the thing is most mostly we use mock-up systems but as ultra wideband are uh more affordable and easy to use we have I have used them in some of my experiments to see the results also but for the other systems they can be I think they can be portable they can be connected to the different robots also but if you have any questions about ultra wideband you can connect my colleague Paola Torico Moron so she is the specialized person for the ultra wideband part thanks and about the standards so do we have standards on this right now or which which is the standard body for this particularly what I didn't get for ROS2 yes so is there any standard body who made all these standards or is it uh what is it like is it open source or yeah it is open source so we have currently we are working with ROS2 galactic so it's easily used and open source and it can be used for different robots and it's easy to use also we have all the packages and things in our github repository if you want for example the ones that the smart contracts and applications that we have used for our scenarios are I can send you later in the github so it can be used and compatible to other robots and you can you can easily use them yeah that will be great if you can share the github yeah of course I can add the github repository in this slide and then later when I send the slides you when you're sharing that you can go on check yeah thanks thanks so much um David could you check any questions for me too I've got I've got a few comments to say so thanks Samo for accepting invitation and uh for a great presentation you've done quite a good bit of work here and it's and it seems that you haven't really stopped that trying to integrate the the fabric with ROS2 so you've actually done some analytics work on on kind of how hyper ledger on the fabric would would actually work in these environments uh so the and I and I particularly like the way that you've used smart contracts it seems to kind of support the event driven or or enable the event driven kind of nature of of the work so you're I think triggering these smart contracts based on the events that happen uh on the on the ROS2 site right so that that's probably would save lots of resources in the smart contracts just being triggered when they are needed rather than constantly being uh kind of monitoring the system right and so the the part with the smart contracts which is kind of the the most interesting part for us in this group uh kind of trying to see how people are using uh fabric and chain codes and so on so one one of them particularly was interesting to me uh which was the decision making contract based on the battery level I think yeah you can go back to that to the slide because so some of the smart contracts are are are essentially about writing uh kind of data on the ledger right but this particular one was about actually kind of uh making a decision so one thing that I that I'm a bit confused about is that so so this is a decision making that's supposed to be taken in a distributed manner so there should be some essentially entities that don't trust each other right and now they want to make this decision together without trusting each other so can you just elaborate on that scenario like why wouldn't they trust each other and what are these parties uh why wouldn't they trust each other yeah so the thing is we are okay for example here dash go on tell or not connected to each other they are not aware of each other's condition so the smart contracts only checks the topics that is getting from the tello which is the battery and whenever it gets the topic that the battery is lower than a specific it just published the topic so the ROS so on the other side in the dash go which we have the smart contract also when it gets that topic it's go to that task I mean goes to that position so basically they they do not both of them are from parts of the network but they are not aware of each other they just gets like publish and subscribe they just subscribe to topics and publish and with these they just communicate I don't know if I get your question right or not already yeah yeah no no so yeah maybe maybe I misunderstood the point here so so what you're saying is that there is a server server and and kind of a node relationship here so the idea here is not get achieving trust using the smart contract it's more about trust yeah trust side is for trusting the robots because you know for example we are our scenario was using them in the warehouse environment if another robots try to make false in the system or try to for example get the drone for landing or do some false in the system that's that robot cannot see any topics because those topics are going through the blockchain so it's not available outside that so only these two can can communicate and can see the topics another thing which which which we are currently using also is that there is another thing like beside rust 2 s rust has been developed recently which is secure rust 2 with that case if we implemented that also each the each robot would have some keys and with those keys nothing out of those robots would see those topics so basically it's like that but we have we are using it currently we are trying to use that s rust 2 with the aviation smart contract this time for this identity and trust things but basically yeah yeah okay so now so if you want to talk about the kind of the the setup of fabric that you have right so do you have multiple organizations do you have like so because these so these smart contracts are essentially probably you don't need a very complicated endorsement policy for them right so do everyone in the network have to endorse these smart contracts or only the robots which are which are which are going to involve in this so nothing else this smart contracts only and by these robots so no no one else I mean you can if you're in a system you can see the topics but you don't need to endorse the smart contract right so I can I can see like in more sophisticated scenarios like let's say in a factory of future scenario where you might have let's say machines from different let's say sections of the factory trying to work together so maybe in those cases you might need kind of smart contracts that are that are essentially kind of bringing more of that you know collaborative decision making kind of side of things to the smart contract because then then you have the problem of trust right so because now you have external sources of trust and how do you how do you trust people coming out I know that you're planning to use smart contracts for identity management as well but what I'm what I'm talking about here is more or more from the view of more complicated decisions that have to be taken and more people have to give an input so here your input comes directly from the drone and the drone makes a decision based on its battery level right okay okay yeah yeah so there's there's quite a bit of things to be to be done here in in more complicated scenarios yeah of course I mean we are just trying starting this I mean I recently I just started working on fabric from a PhD I mean it's been one year and a half so I'm just trying yeah integrate I was not specializing yeah yeah and one last question so if you go to the latency figures that you had well actually like both latency and and CPU utilization yeah so what is the transaction throughput here what numbers are you looking at the the truth the thing is we had another analysis also but I didn't put because the slides were too much for checking the throughputs to see that if changing and having different throughputs would affect this or not we had some test tests with different patch timeouts and maximum message but beside that we got the frequency of I mean with the frequency of around 50 hertz was the thing that we got the optimal one but we had some stress test to check the different frequencies and different utilization but I didn't put it here so with that optimal frequency that we got we got the minimum resources utilization yeah but sometimes that's really out of your control right because the number of transactions that come through your fabric will depend on the number of drones you have so I assume these are all scenarios with one drone and one dash right only for our case I mean it's not yeah yeah so I mean obviously for this case this is a good analysis but I wonder if this would still apply if you have let's say 100 machines that are quiet latency sensitive so you want to process their transactions very very quickly and so yeah what I'm saying is that this might not be very generalizable because like I did lots of kind of these kind of analysis on better if my fabric network is going to kind of be able to meet the the latency requirement that I have in my use cases right which is telecom use cases and they're very latency sensitive and once you reach kind of a couple of hundred transactions per second things start to kind of change quite rapidly with the latency and CPU usage and so on yeah also when after recently because with our new scenario that we are using the one problem is that as it is accumulating the data after if if we leave the data to be transferred as it is sometimes the latency increases even even so it needs for example after three minutes or four minutes after accumulating the message it starts to go yeah and there's also so one thing is actually like just writing things on the blocks and the other thing is that if you have like a kind of a more sophisticated smart contracts that are making a decision based on a particular algorithm then if you have to wait for all the nodes to endorse your your your smart contracts outcome then you end up sometimes waiting for quite a while to for everyone to endorse uh smart contracts and so on so yeah yeah like lots of space for you to to work on if if you want to continue working with type ledger so yeah of course there are lots of parameters that we have to take into account and if you want to if you want to generalize it of course we have to use more than two robots we have to take into account lots of things of course you're right okay perfect thank you very much thank you anyone else any questions I have a question have a question so um very wonderful presentation thank you so much for your time um my question is are you able to bring the electromagnetic computation into fabric um I for now I have to search for that because I don't know if I can I don't know I have to frequent like the frequencies are you able to bring like the rf and all of these things into fabric yeah that's why I'm saying um I mean like checking them or well anything I mean it's I mean you're on the you're on the you're on the cutting edge yeah so I'm just wondering if you were able to bring electromagnetic computation into fabric uh actually we haven't tried no so I can I cannot give you another we haven't tried so maybe if you try I can give you the okay all right yeah thank you you're welcome anyone else any questions if not so thank you thank you Salma for accepting our invitation and uh this very good presentation and thank you everyone for joining today and thank you so much thank you thanks everyone thanks everyone bye