 So thank you very much for being here, Yadu, and let's hear about it. All right. Thank you very much, Marco, for that warm introduction. A very good morning, good afternoon, good evening to everyone here. My name is Yadunan from Open Robotics, and today I'm really excited to talk to you about this massive open source project we've been working on at Open Robotics. It's called OpenRMF, but specifically I'm keen to share a bit more about a practical deployment of a multi-fleet task allocation system that we've developed. And this is a solution to the classic problem of how do we distribute a set of tasks across a set of robots, given that each robot is from a different brand, may have different capabilities and different constraints. So that's going to be the objective of this talk, and I hope you guys are excited too. With that, let's get into it. So what this talk will cover are three broad aspects. We'll begin by exploring the need for interoperability, and interoperability is just a fancy word for saying different systems can talk to the same system and can also talk to each other. So this could be different brands of robots that need to talk to each other or robots that need to talk to an elevator in a building, etc. And secondly, we'll talk about how the robotics middleware framework addresses some of these interoperable challenges. And finally, we'll talk about how we can solve this problem of allocating tasks across different brands of robots. Okay, so what is this interoperability problem? So nowadays, we're seeing facility owners increasingly adopt robotic automation in their sites. So if you're a hospital owner, or a airport owner, or a hospital or a hotel owner, for example, you may want to introduce robotic automation into your facility. So you might get a brand of cleaning robot to do some cleaning, a delivery robot to do some deliveries, maybe a security robot for patrolling. But the downside is that all these robots typically will come from different vendors, because a robot that's very good at cleaning will be made by a different company than the one that makes delivery robots. So inherently, when we bring these different robots into the same facility, we're introducing challenges for the operators and facility owners, primarily because these robots don't communicate with each other. They're made by different vendors, so they use different communication protocols and messaging systems. And as a result, it could lead to a lot of problems in deployment and also some operational challenges. So one common challenge that arises is you have two or more robots that go down the same hallway. So if you can imagine a hotel or a hospital corridor, which is pretty narrow, if you have two robots coming in the opposite directions, they'll think of each other as an obstacle and kind of do this robot dance to try to go around the other robot. But they'll just remain stuck in the end because they can't talk to each other and say, hey, give me a way or let's take a different path. For example, another complication from this is getting these robots to talk to the same building systems such as doors and lifts. So typically what happens is if you have a lift in your facility that you want a robot to take the robot vendor will come in and do a lift integration for their specific robot, which the second robot feed cannot really use because it uses a different messaging format. So in this case, when you want to introduce a second robotic fleet, you need to do a second integration. And these integrations are not really cheap. So the cost really adds up as you scale robotic deployment. So yeah, and moreover, from a practical standpoint, if you look at operators who need to use these robots to send out to send out robots that dispatch certain tasks. It's not really an easy job because maybe the task, the requirement is very simple, like I need something to be delivered from here to there. But if you have multiple feeds in your facility, the person making that decision goes through quite a bit of process they need to first log into the different dashboards that come with each robot vendor. So for example, we could have vendor a has his own dashboard vendor B has their own dashboard and from from that the operator would need to log into these different dashboards and see which robots are available. And from that decide which one has sufficient battery to do this task, and then finally use that dashboard to send the task to that robot. So some of these robots may be ready to deploy right now some may have low batteries some can't do deliveries for example they could be cleaning robots. So the decision making process is extremely manual. Sorry, Marco, I got a notification saying the screen sharing stopped. Are you able to see it? Yeah, can you try resharing? Sure, let me try that. Are you seeing the screens now? I'm sorry, are you seeing the slide? No, we also lost your video feed. Your video feed is up, which is missing your slides. All right, let me try that again. Yeah, I think it's charging up. Please go ahead. Sorry about that. Okay, so we were talking about how the process of submitting a task can be really manual and involves a lot of decision making when you have multiple robots. And the solution I want to talk about today is open RMF or the robotics middleware framework. So this is a fully open source platform that we've developed that opened robotics. And the goal here is to really provide a common language for robots to interoperate. But apart from that, we provide a bunch of tools and libraries to do a bunch of different functionalities we'll talk about in the next slide. But some of the takeaways are this is fully open source under Apache 2.0 license. It's actually be maintained by open robotics. We have tons of community collaborators. And we provide out of the box integrations for several robotic fleets doors and other lift systems. So what does RMF do? It's very good at a few things. It does global traffic management. So this is what we're seeing on the top left corner here. So once we have different fleets integrated with RMF in the same facility. It's continuously monitoring where these fleets are trying to go. It can be planned the routes for these robots if there's a conflict and it will avoid situations where there's two robots in the same hallway, for example. Secondly, you can have RMF communicate with other automation systems you have in the building already. So you may have a robot arm that dispenses beverages. You could have a trade return system, for example, and then you can you can integrate these systems with RMF and have robots automatically interact with them to accomplish tasks. Another big capability of RMF is it allows these different fleets to share the same lifts and doors. We talked about the challenge with multiple integrations previously. Whereas now you would do a one-time integration with RMF for every door and lift in the building. And anytime you introduce a new fleet, it can readily use that same door or lift. And lastly, you can do intelligent task allocation. So this is a functionality where you can just submit a bunch of tasks which have semantic meaning. And these tasks will automatically get allocated to different robots and RMF will ensure that they get executed. And this is going to be the focus of today's talk, basically, how we accomplish this. Before we get into that, just a quick note on how RMF interacts with vendor robots. So you can see, as you can see here, you can have different brands of robotic fleets in your facility. Typically, each of these robot fleets will come with their own API. So a vendor will provide an endpoint for you to connect to their system could be the robot directly a cloud server. So once you connect to that system, you can issue commands to the robots and receive information from the robots. So through these APIs, you typically need to do a one-time integration, what we call the smart fleet adapter. And this adapter is basically a bridge between the RMF ecosystem and the vendor-specific software. So again, these fleet adapters have certain APIs that we provide in RMF. So the goal is for you to use these APIs and take the endpoints from the vendors and combine them into an adapter to bridge the communication gap. So once you do this, you can consider your fleet to be RMF integrated. And with that, we're able to get some information from the fleets and we're also able to send information to the fleets. Okay, so with that in mind, let's examine this problem statement that we have, which is multi-feet task allocation. So it can be summarized as given M number of tasks with varying start times and certain descriptions. So it could be a cleaning task, delivery tasks, etc. And then we have some number of robots across F different types of fleets. How can we distribute these M tasks across these F fleets, such that robots are only assigned the tasks that are capable of performing. So this means that the delivery tasks should not be assigned to a cleaning robot. And further, the robot should have sufficient resources to perform all these assigned tasks. So if it's a cleaning task, you have to check that the robot has sufficient battery and cleaning resources to perform the task. And lastly, and most importantly, we need to ensure there's some kind of optimality in the assignments because that's really the goal of being better than the human operator. We want some kind of optimality in these assignments. So there are certain assumptions we make. We assume that each task can only be executed by one robot. So this means that if it's a delivery task, the same robot will do the pickup and the drop off. It's not going to hand over the payload to another robot to continue on with the task. Secondly, a robot will only perform a task once it's finished completing the previous task. And lastly, we have some constraints on charging resources as well. So when we first tackle this problem, we did what everyone normally does a literature review of some of the algorithms and solutions that exist for multi robot task allocation. So while we found several solutions for multiple robots, I think we were in a unique position where we're dealing with multiple fleets. So we were able to synthesize some of the information from all of this literature and put it together in our own implementation. So some of the big takeaways from these literature review. Sorry, Mark, did the slideshare stop again? Yeah, I was going to say that. Yeah, the slideshare stop again and the video. So if you can just, oh boy. Yeah, sorry about that. Sorry, it's a backup. Yeah, everything's working. Cool. So yeah, we borrowed heavily from a lot of literature that talked about multi robot allocation. We extended this to multi fleet task allocation. And some of these approaches that we borrowed is the auctioning system. So what this means is that every fleet is basically an auctioneer so they will bid for a certain task. And let me explain how the system works. So now when users or operators submit a bunch of tasks, it goes to a node called a dispatcher. And this is kind of a central authority that maintains all the tasks that are received. And it's also responsible for talking to the different fleet adapters. And if you remember from the previous slide of fleet adapters basically that bridge between the vendor fleet and the rest of RMF. So once these requests are received by the dispatcher, the dispatcher will then send out bids to each of these fleets. So these happen through some internal communications over Ross to and the middle where we use and Ross to his DS. So with the DS, it comes a lot of security and encryption, and a lot of great functionalities. So we're using protocol as DDS for these communications, but this dispatcher node will then let each of these fleets know that hey, I have a request to perform a delivery task. Something needs to be picked up here and dropped off there. And then each of these fleet adapters, knowing the capabilities of their robots will respond with a with a proposal. We receive a bid notice from the dispatcher and then we submit back a proposal and this proposal typically includes information on whether you can perform the task or not. And if you can perform the task which robot you're going to sign it to and when will be completed. So the proposal is received. The dispatcher node, then goes through all the proposals within it for certain time window and then decides which fleet to award the task to. And this criteria to award the task can be configured. If you prefer having fastest completion time you can set that as the criteria or the metric. If you prefer overall completion time. That's also a metric you can use, or if you want to do something like first in first out. That's also possible. It's a very flexible selection system. So the great approach with this is that it's a fully distributed system. You don't have a single node that's aware of all the capabilities of all the fleets. This responsibility is offloaded to the fleet adapter. And another great part of decentralized approaches is it's easy to add on another fleet. You don't need to change anything in the system. So I would just add another fleet adapter that has maybe, I don't know, drones or something. And that's able to respond to the same tasks without changing anything else in the system. So this is the approach we're going with it's a distributed multi fleet task allocation system. And one detail that I left out is how does this fleet adapter decide what is the best robot to allocate the task to. So we said that a task comes in, it goes to each of these fleets and each fleet tries to accommodate this task in the best possible way and then submit a proposal back. But how does each fleet decide how to best accommodate the task. And in order to do this, we've developed this new planner, which is a task allocation planner. It's a simple a star solver with a lot of constraints defined into it. So we have semantic information captured about what the task is, what the state of a robot is. And with all of this information we do a simple a star search. I won't go into the details of the algorithm in this talk. But it's just an approach where we, where we prune different solutions and try to find the best optimal assignments for the set of robots. So for example, if a fleet adapter is running this a star solver, it's going to have information on all the states of the robots. So if it has three robots, it knows what the battery level and starting position of each robot is. It will try to compute the finished position at battery level for each robot and then decide which robot to allocate the task to. So what's happening here is we're optimally allocating the task within each fleet. So each fleet does that, and then the optimal assignment for each fleet is submitted to the dispatcher node and the dispatcher node then picks an overall solution. So for ensuring optimality here, and then if we can ensure optimality at this level at the dispatcher level, then we can ensure some kind of global optimality, optimality, optimality among the assignments. So yeah, this this approach is not entirely useful without accounting for battery drain. So for example, initial states of robots, maybe too low to perform the first task. So maybe you have one robot in a fleet and it has 20% battery level whereas you need more than 30 to perform this task. Robots we need to charge in between tasks. And how do we automatically instruct robots to go to the chargers in between tasks. So the way we chose to tackle this problem is build in a battery modeling framework within the task allocation planner. So, so I thought I lost the connection there for a moment. So once we can estimate what is the battery drain for each task, we can then predict whether robots will need to charge in between tasks or whether they need to charge before they start performing the task. So that's what we've done with RMS. Again, with a lot of literature review, we try to study how battery drain for robots can be modeled. We understood that modeling this train is extremely complicated because it depends highly on battery chemistry. The state of the battery, the age of the battery and a lot of other parameters. And we don't want to be too rigid in what kind of battery models you can use. So now we have a kind of library that users can plug in play if they want to use our simple battery drain model. But if they have more complicated models machine learning based or some other data driven models, they can plug those in as well. So with these with this battery model plugged into the task allocation system. Here's a very raw output of our task allocation planner. And as you can see, if you could give it a set of agents or robots with certain IDs, they're starting locations and they're charging waypoints. And if you give them a set of tasks to perform where you need to pick something off at a certain grid location, drop it off at another grid location. It will output a set of assignments for each agent and automatically include any recharging tasks if required again for details on how this algorithm works. It's out of the scope of this talk, but we could take those questions offline. But I'll just talk about some other quick capabilities we can do with the task allocation planner. We can have priorities for tasks so we could assign higher or lower priorities to certain tasks. And internally in the task allocation planner, we can make sure that high priority tasks are completed before the low priority ones. Another thing we can do is automatically add tasks in between. So we already talked about injecting recharging tasks for robots, but we can also ask each robot to go to a specific waypoint or a parking spot after it performs all of its tasks. So yeah, this is kind of what the final results look like. So this is running RMF in a simulation environment here. We have three different feeds of robots and this is our RMF dashboard. So here are basically the user loaded a JSON file that consists of a bunch of tasks to be done. And then the user can just hit submit tasks and all of these tasks get automatically assigned to the different feeds and you can see. As you can see in the video here, some of these are delivery tasks. Some of these are cleaning tasks that get assigned to different robots. All right. It's just a quick video of it running on real robots. We just don't do things only in simulation things work in real life too. So here we have a cleaning robot and not a patrolling robot that are kind of moving around performing tasks which are controlled by RMF. All right. So there are a lot of topics covered in this short talk. All of the source code has all of the source code is available for open source. It's available on GitHub. These are the relevant repositories. We have our battery for the battery modeling our task for the task allocation. And we welcome everyone to try out some of the scenarios in simulation from the arm of demos repository. If you have any questions, please feel free to open a discussion at this link. We're super active in developing and responding to discussions as well. With that, I'm happy to take any questions.