 It's really my pleasure to welcome everyone to this panel with our guest and with our guest, which he will hear more about soon. So it's my pleasure to introduce Professor Sreya Sundaram, who is going to be the panel moderator. I think all of us or many of us know Sreya, but Professor Sundaram is actually an associate professor in the School of Electrical and Computer Engineering. And he's also the co-director of the ICON Center and we're very thankful for him taking the leadership in this panel. So Sreya, I'll pass it on to you. Thanks very much Dimitri. It's a pleasure. I'm going to share my screen here and hopefully you can all see that. Yes. Perfect. Okay, excellent. So it's a great pleasure to have everyone here for the second event of Professor Claire Tomlin's engineering distinguished lecture. It's, she gave her seminar at one o'clock and it was a wonderful overview of the problem of safe learning and robotics. So just to continue on that theme here, obviously, you know, robotic systems play an increasingly large role in society in a variety of life critical applications. So things like surgical robotics, manufacturing, construction, you know, autonomous vehicles, and the, these robots that are interacting with the physical world in various ways. So we're implementing controllers for learning algorithms, the potential for things going wrong and impacting life and property in negative ways is really large. And so it's really critical that we try and understand how to design these systems to be just be safe and secure. So the robotic systems inherently because of the interaction of the physical world are different from, you know, traditional it systems right where computer system where things go wrong may cause the system to shut down unless it's controlling something that interacts with the physical And so there are a variety of interesting challenges that come into play here. And so the system designers that that put together these robots have to make sure that the learning algorithms and the controllers do what they're supposed to do, even in the face of failures or attacks, and are to make sure that that things don't go wrong. So this is a major challenge and today we have four distinguished panelists here who are experts in this general area to provide their perspectives and to have a vigorous discussion hopefully about the challenges and the capabilities that that arise. So it's my pleasure to introduce these panelists so we'll start the panel by having each of the panelists give a very short overview of some of the work that they're doing their perspectives. After that we'll go into a discussion where I will ask questions and the panelists can can answer and the audience can also chime in with questions. And to note is that the audience does not have to wait until the question period at the end to start asking questions you can go ahead and enter your questions in the chat box and that way we will be able to ask them as they come up. So it's my pleasure to start with Professor Claire Tomlin who is the Charles this war chair in the College of Engineering and a professor of electrical engineering and computer science at Berkeley. And she's been at Berkeley for some time but she started her career at Stanford, and she's made significant contributions to control theory hybrid systems which systems air traffic control and so forth she's a fellow and I truly fellow Donald Ekman award winner and one of the few people to be both a fellow and an American Academy of Arts and Sciences fellow. So it's a great pleasure to have you Professor Tomlin. So you already gave us a very nice overview of your work in your lecture, but here in the context of this panel. So now you talk a little bit about the safety issues or the issues relevant to this panel. And so I'm going to give you access to the screen here via remote control. And so now your screen is yours. Oh, this is a format I'm not used to so I. Yeah, so if you were to know I. Okay, so I'm. You have given you access so it says waiting for you to control. Every time I try to control it it mutes me. What says viewing options and you should be able to request remote control right by your name with the top view options. Or I am happy to advance the slides for you if that's preferable. I don't see the option in those three dots you mean doesn't seem I have an option to request remote control, although it does say I can control your screen. You are controlling that. Yeah, so if you hit, if you hit the right button you should see it advance or the right arrow. Let me just try this because that's really cool and I didn't know you could do this on zoom. I'm going to try to see what I'm doing wrong. It says I'm controlling your screen and then I'm trying to advance. And it's not advancing. Let me see what do I need to do. In the last. Did I do that. Yeah, did you do that. I think try it again. Strange. Okay, well I'm happy to advance for you. You can just tell me next slide. I thought it's way cool, but I didn't know you could do that. Okay. Thank you very much for the introduction and thanks for, first of all, thanks for organizing such a wonderful visit and even though it's virtual, just having the way that you've organized it with the lecture but then the discussions and then this panel, I really appreciate it and really am enjoying my day. So I'd like to maybe as a springboard from what I was talking about earlier today, present this kind of a viewpoint so it's not a, it's how I've been thinking about safety autonomous systems and safe safety of those autonomous systems. And this is joint work with several students, but also with Alexandra Faust at Google and Jatendra Malik, who works in computer vision at Berkeley. If you could advance to the next slide please stress. Okay, so this is the, this is a very simple problem or seemingly simple problem you have an autonomous system, you'd like to navigate it from where it is now to a goal. And it just doesn't know it knows where it is it knows where the goal is but it doesn't know what's what's in between. And it has a camera, and we're going to design a perception system for this robot to be able to perceive the environment and act on both what it's seeing as well as its knowledge that it wants to get to the goal. The environment is unknown it's not really unknown because the, the, the knowledge of the environment for the knowledge of what to do comes through in the training of the perception module. So we propose, and if we advance to the next slide, Shreya's a an architecture where there are three modules, a perception module, a, which takes in images and gives them a next way point for the vehicle to follow in its path from where it is to where it wants to go. That feeds into a very standard we just used a spline place a spline based planning module then that feeds into a very standard we just use a linear quadratic regulator for the control. Our kind of maybe thoughts coming into this are that the planning and control once you have an idea of what your environment looks like they are solved problems we have wonderful algorithms to do that. Let's use the models that we have of what we have and let's integrate them kind of in a sensible way with with learning where you need it and in this case that's in in in perceiving and understanding the environment so that you can act on that. So this architecture was, it's also taking advantage of you know the recent, I would say decade of work in computer vision that is has developed. And we trained that neural net fully in simulation so simulated data of indoor texture matches. We trained it using a loss function which was developed from optimal control, and that's how it has learned to give us the next way point. Okay, so let's go to the next slide. And can you play this stress. So the first person view is what the robot seeing and then the third person view is is what is what you can see and then you see a top down kind of view. Here the goal is outside that room and it's in the hallway. It was never trained on doors or I mean it was trained on data with doors but it was never told about the semantics of a door, or the need to go through a door to get to a hallway. So it turns out it's on a floor, there's, there's glare of light right right where the goal is, and it transferred from, you know, carpeted room to glare, a kind of glare. It's with the data that we provided it's learned to, you know, do those things like go through a door to get outside of the room to go to a goal. So it's, it's become and we've we've got some, you know, more robust statistics on this, it's become more robust to glare or, you know, little faults with the camera, for example, then if we use the traditional slam pipeline where we build a map and then use planning and control on that. It's become more robust than an end to end framework so this was a stepping stone where we said okay so we want if we want to preserve a kind of modular architecture where we use learning where we need it this is possible. So if we go to the next slide please. So I want to just sort of put questions for the panel. So now, if we think, and again this is a limited example but you know, maybe relevant for autonomous vehicles that are going through environments that are unknown, although what we've shown an experiment is, is a far cry from kind of our general autonomous car right, but some challenges now to think about the first. We are relying on that perception module to give us the next waypoint. That is, you know, we, we haven't done any analysis or any bounding or any, you know, that module could fail. And, and where does it fail it fails when it gets images that are things that it hasn't seen before, or that weren't in the training distribution. So one question we've been working on is can we monitor the images as they're coming in real time and ask, are they within the training distribution or they outside the training distribution that's a very hard question, generally because you don't typically have distributions for your training data for your, you don't have distributions for your training data. Related question is, okay, can we also find some way to characterize the uncertainty in the output of the learning module. So that's that waypoint that's actually something that we're working on and this is, you know, there are what people working on neural nets have quite a bit of work in dropout or training with ensembles to try to characterize, you know, this is the waypoint you're getting, but kind of, you know, any waypoint in this region would be probably pretty good. Or that's the kind of data we'd like to see but how do we do that in, you know, how do we do it practically but also in a scientifically principled way. How should this uncertainty be then propagated through the planning and decision making loop is a question that that I think is a very important question now. And then of course this is a very simple question, what about more complex models environments. You know, typically you don't have such a good model of your vehicles so there's there's things in there you might want to learn as well so learning parts of the dynamics. But what about more complex environments people moving around. So these are all questions and challenges that I think really set a research agenda for the future in this in this area are part of a research agenda. And that's the, that's the end of my short presentation. Perfect. Thank you Claire. So we'll do a introduction by everybody and then we'll come back to these discussion these questions. All right, so our next panelist is Professor Sinyan Deng who is an associate professor here at Purdue in mechanical engineering we have a Berkeley heavy panel today as a professor dang also got her PhD at the University of Berkeley and she's been at Purdue as a faculty member and she's received the NSF career award in 2006 she was selected as a BFS at Schaefer outstanding faculty scholar in 2015. She works broadly in the space of robotics particularly bio inspired robotics aerial robotics and so forth. And so today she's going to tell us a little bit about her perspectives on robotics and safety. So Sinyan let's see if we can try that control again. Let's say control. I was successful using the arrow keys, but now they are not working somehow. It did work in the practice. This is how it always goes isn't it. There you go. Yeah, not that you're doing it. Yeah. Okay so it's me that's controlling okay. So I've been thinking what kind of work in my lab which is related to security controls, and also learning so I'm going to show you those aspects in my lab that's related to this. So this is one of the project. It's our project with which I have two computer science professors as collaborators, we are looking at cyber physical securities so cyber physical attacks. So traditionally in computer science, you're attacking the hackers and hack your computer, hack your, you know you have to strengthen the memory or firmware. So if you have a mobile robot, and if the hackers can hack you in real time to manipulate your control software to try to spoofing your sensor actually her, you know operating system, what do you do. So this is a basically four year, and they extended just one more year until like this summer. So it's been a previous few years of work in this aspect. Basically, if we look at the way it's structured traditional cyber centric approach and also control centric approach. There's a gap between, and we were trying to bridge the gap, and using computer science expertise in reverse engineering some of the code, and trying to extract some of the hidden attacks probably there, and also using control or dynamics. So there's a lot of expertise here to put it in. Not exactly the foreign control but it's a control framework that you can retrofit the system using minimum amount of extra hardware and software so that you can detect some arrows and try to recover the flight. So basically this project, the key is retrofitting so the owner basically saying oh not all the, you know, not everything is brand new with the latest software and hardware but what if you have an existing system you can just retrofit it with some added property of defend against hackers. So basically did we use, you know rovers and also drones as examples to retrofit them. So basically, you have a kind of redundant controller, where you can take some sensor reading and then compare it with what you estimated as a system dynamics and try to compare it with a real vehicle in there, and then we can try to do detection and recovery. So the details are in the papers we have but I will just show you a very quick, you know, maybe just starting of the videos. So some sensor attacks, meaning that if your IMU gets spoofed, and you're reading maybe several degrees off, and without the so called blue box, which is the retrofitting unit, you will fail but with a blue box and you can have your retrofitting unit to take over tender temporarily at least, or if it's too severe you just have it safely landing and actually try attacks for example the lock one of the motor maybe beyond certain height. And these are all malicious code that can be bad in the system when it's coming out of the factory so. And also a printing system attacks so basically they can just totally wipe off the computer I mean controller for some certain time. So this is the security really work, and we can come back later to it. So, just one more minute, just in the time gap. So learning based control in our lab is basically we want to use reinforcement learning. Actually we did use reinforcement learning to try to use this hummingbird robot to mimic some aggressive maneuvers in animals. So for example, this is to mimic flip 360 degree flip. And then the next one is to mimic hummingbird escape from a threat. And what I want to point out is. So if you have a looming threat in the front hummingbird will escape to a certain pattern and try to minimize the time of escaping. And if we use reinforcement learning to train this robot. What happens is that they will give you a very similar pattern of the body kinematics and wing kinematics to it's very interesting so the trajectory is learned. So you just give it a reward function. And of course, set it up in a reasonable reinforcement learning so basically what I want to try to say is a real animal has this type of we body kinematics and what you what you train from the robot. It's very, very similar. They do a kind of a banked turn they do it like a pitching up and roll and escape that's, you know, quick way to escape. And the starting line here is experiment result. Let's go to the next. And another thing I want to show is that talking about safety of robot. If you think about this file inspired robot because the wings are soft. And it's under literally motion. They go, they don't get tangled. And you can just, you know, that's safe to the touch. And when they, when we send send it to, you know, from a to B, like a, you know, just a point point tracking, although there are some obstacles in between. It can just bump through because the wings are soft. The body have some have have some yield because of springs. And it's quite robust. And this is just one which we have a multi model thing so you can just go through some narrow space, and then you can stand up and fly. So I've always been a very, you know, advocate for by inspiration because these things are very resilient. That's a really resiliency and the safety sometimes associated with it. They make them very attractive. I think I have only last slide left. Yeah, this is just, just show you some, you know, we look at some additional locomotion principles because, for example, this is a recent work. Well, we found out that flapping flight naturally reject gas reject disturbances because dragonflies and hummingbird and migrate thousands of miles. Each, you know, each year, and the to save energy they don't need to fight with the gas all the time, the turbulence the somehow the flapping itself get attenuated for the turbulence. So that's what I want to talk about. So I just want to maybe just trigger some thoughts. That's, yeah. That's a work we do related to this type of aspect. Well, thank you. The next speaker is Professor Inso Kwong, who is a professor here in aeronautics and astronautics at Purdue, Professor Wong got his PhD at Stanford with Professor Tomlin, and has been at Purdue, and he has received the NSF career award in 2008 was selected as one of the nation's brightest young engineers by the in 2008 as well to receive the AI double a special service citation 2010 is an associate fellow of the I double a. So Professor Wong, I will give you control. Thank you. Let me see I can control. Yes. Okay. So I have only one slide like this so I wouldn't go over five minutes. This is my lab's work in very highlight the summary as Claire presented when the architecture for this robotic exploration. Our lab also has a two major component area one is the lower left right corner that the information influence for the gravitational when it's that corresponding to perception that we understand what's going on the environment as well as the robot itself, and then the system itself. And then once we understood what's going on the next step is to take an action. And so the, the problem is now boils down to that the, how can you infer the intent in a way that it can be perceived by the robots. For example, that the robot is everything is working properly, it can do certain amount of work, but it has a some failed component, then it's performance, the limited, and they can be computed using for example which was a computation that the, the envelope is very large and the system is a normal, but that can be distorted if system is abnormal that abnormal. And then the understanding that from the multiple sensors nowadays as answers are, I mean, three of the sensors use even for small robots and drones, for example, combine them together to get generate the situational awareness. And then at the same time, the one application is the robot could be autonomous, but many times human in the loop or human on the loop so some way, the robot interact with each other with the human, then the, the how human can collaborate this automation or machine in a safe way. And one good example of this system is the airplane cockpit and this small figure shows the modern day airplane commercial airline cockpit and pilot actually a controlled the vehicle, the aircraft, just like a computer game that you push a button and then turn the knobs and so on and so forth, so that the, while automation is going on and pilot is doing something collaboratively but the, there could be a misunderstanding between the two, I mean, recent the accidents in the Boeing 737 MAX incidents that autopilot wants to one thing pilot misunderstood or once the other things that instead of collaborating between them, and they find each other so pilot wants to go up airplane wire wants to go down to some region. Then the problem is that they can we identify the pilots and the automations in 10 and show it to the pilot or the other way around from the machine's perspective, machine can understand the pilot's intent he wants to go up or go down or speed up and slow down. How can you do it and this discrepancy is well known problem in the control community called more confusion problem or automation surprise problem. So the situation awareness problem, one of the situation awareness problem is to understanding that can you capture this early now so that the pilot or automation can take a recovery measure or mitigation measure and resolve this conflicting problem. And another issue is that the automation assist the human, but sometimes she I mean many times in the human operated experts then they can do better than automation. And we call them as experts, then human does not need to support him or her. And sometimes it's just bothering it. If they keep bothering the human, human tends to turn it off, rather than getting assistance out of it. So that the, how can he make the this collaboration in such a way that machine understood the skill level of a human operator. So that is assist appropriately. So the overall system safety, for example, can be maintained rather than human just ignore the any advice from the automation, which could lead to a unsafe situation. So they can be done again using the data driven approach because modeling human is tremendously difficult and why we have a lot of data coming from this experiment. So that the from this experimental data, we develop the human cognitive human model human model again representing the skill level of the system, the human in this specific case, and then designing controller. So, yeah, then did that leads to a control actions that the how can he assist the human and or the robots in such a way that collectively we achieve the better safety. And the next slide I have actually one more slide that this is the not my lab but the our department recently have established this huge indoor us has a facility and it could be the men the vehicles any vehicle not only flying We have a Purdue has a airport and the around the airport we have multiple aerospace infrastructures, you know wind tunnelers and the proportionless and so on. And one of the hangar, which is the about 20,000 square feet and 30 foot ceiling a huge structure, we do have here represented in here. Now we have a more capability for gathering a more data safely and efficiently, and also test and validate the develop the algorithms and more safely and securely and so and also it couldn't be used for the we immediately to be used for educational purpose. And this is not we develop the develop proposed to develop this facility but we want to use this facility as a hope so that the all these icon faculty members for example, come together here and do the project together collaborative project together. So that's what I have for today. Okay, then our last panelist is Professor Shashri Mo who is an assistant professor of aeronautics and astronautics here at Purdue he's also the co director of icon. Professor more God is PhD at Yale with Professor Steven Morse and then subsequently did a postdoc at MIT before coming here to Purdue. So, Shashri, I will give it to you and I'll give you a warning at about three minute mark. Okay, sure. And I will control screen. Not work in my keyboard. See. Yes. It work. Okay. So today we're more like talk about the CF and resilient autonomy. And we will talk about the robots I guess the most impressive one might be the humanoid robot from Boston Dynamics which can be looking at, and the single complex a robot with a high degree of intelligence. And on the other hand, there could be a swarm of simple robots, which can work as a cohesive whole for some complicated missions, like exploring large area of the unknown. And this swarm of robotic swarm also has its own advantage of being robust, flexible and scalable. So we are talking about the both. And we will talk when we've come to safety for robots. Tony has presented a very nice method of for the rich by using rich process for safety, but actually here we want to want to attack this safety problem from another perspective, how to leverage human expertise to help robot to solve the safety problem. For example, if a robot is already programmed, or robot is to be programmed, how should we use human expertise to help the robot to plan to generate a trajectory with abstract avoidance. And actually sparse human inputs do help a lot. So the key idea here is to integrate human input into robust objective learning and the proposed framework here we more like a look at a robot and the autonomous system under a dream by optimal control with a tunable parameter. And then when we come to safety or additional constraints, we could introduce this additional loss function to evaluate the trajectory of the robots. So we actually utilize the external you and output of the loss function and add the feedback and design another optimal control system in the feedback loop to help tune the parameter. So this is our most recent paper presented by and it needs. And of course this is not designed are not limited to only the human city city. But here we could use that example to show you how you will be learned avoid obstacles, give me one point this you will be the drone could pass these two doors. Two way point is much better. Of course, you may argue that we could use the curve feeding and to help the robot, but if the obstacle moves and curve 15 method does not work. And here we more integrate the human the way sparse we point into the object learning of the robot. Another one went come to actually a robotic swarm, and we were not only care about safety but also resilient by resiliency means a system capability to prepare and plan for and observe and recover from the adverse events. The challenge here is that, you know, we come to robotic swarm at each robot is usually with the low cost with limited sense and precision capability, and also each robot can only communicate or current and with a certain nearby neighbors. And on the other hand, the cyber attacks could be very sophisticated, could be fully control one agent could be launched massively from multiple locations, and it could be also highly mobile. We more like to have focus on how to achieve resilience into them, perhaps one of the most fundamental problem for multi-agent optimization for robotic swarms. So each agent here has a local objective, and has the state has some local constraints, and the goal here is to for to design additional algorithm for all the robots to reach a consensus to minimize this global objectives. And with subject to the local constraints. So this fundamental problem could solve actually could be applied to a coordination of multiple Jones, and also multi agent reform and learning. One minute left. Okay, so we have solved this one resilience for this reconnaissance, and this collaboration with the Shias, and we considered time very network by venting attack time varying locations, and also only local information. And currently we are how ongoing work as for multi agent reform and learning, as we have a little gear brush postal fellow aging co supervised by Shias and myself, as this is still ongoing, as actually any problem related to safety and security in robotics or robotic swarm definitely are challenging and requires a different expertise and a collaboration of faculties from with different background, and then this motivates us to launch this center for innovation control optimization networks. And this center currently we have 52 faculty members from a department for you, and we have established a bunch of research scenes in the umbrella of robotics and robotics swarms. And this is my. Thank you very much. So I think that was a wonderful overview by the the entire panel about different perspectives and different backgrounds that are coming from. So with that, I'll stop screen sharing, and we'll enter the discussion phase of our of the panel. So, as I said, before you can feel free to type in questions into the chat and we will answer them as we go. So just to get us started, I'd like to start with a question that was related to what rather Tomlin raise at the end of hers, which is sort of this idea of when we're doing learning the, you know, there's this notion of emergence right so essentially you'd like the system to be able to perhaps predict things in a way that maybe, you know, you can't predict a priori otherwise you're just designed to control or for it right. But then there's a trade off between sort of emergence which is sort of unpredictably unpredictable behavior which does amazing things that you could not have predicted a very and safety guarantees. There's an inherent tension between these two. And I was wondering whether the panelists have any thoughts on how do you balance between giving it freedom to learn new things but also providing guarantees on safety and security. Does anybody want to have any thoughts on that. I mean I do, but I'm not sure it's the right thing to do yet. I mean, what what we've been doing because we define safety in a fairly crisp way with respect to constraint. We have what we've been doing is exploring the boundaries of those constraints. So, if you're, you know, mathematically, if you're operating on the boundary of one of our level sets, and you don't stay on the boundary you kind of go into the safe region. It means you were probably conservative, because the control is such that, you know, the best possible control should just keep you on the boundary. So that means that you were more conservative so you could probably push your boundary out a little bit. But rather than kind of starting with something very small and growing it we start with a set that we know is where we think it's, we've proven it safe if we trust the models that we're using and the specification that we have. And then we explore on the boundaries of those. But that is, like it's, it of course came as is inspired mathematically by, you know, what's, but in real life, we want problems where there's whole other regions of the state space away from the system that we'd like to be able to assess and explore. And I think, like, learning about, like, doing exploration in a safe way to gain information about the system, and then having kind of limited real exploration is some big notion that I'd like to pursue where perhaps the we try to use simulations very effectively but of course you can only simulate what you could model so. So I'd love to have this discussion and hear what the other panelists think as well. So any thoughts on this idea of how do you explore effectively while still providing guarantees on safety. This is in fact, the. I think this is important problem yet is two different problems in it. So, when I saw the community, the working on this related area. And someone more focused on that the explorations are all done in a safe environment, sometimes the simulation sometimes, you know, the labs and so on. So the end results would be safe and I'm okay. Right. And the other one is that what if this car is moving around the wire operating it also learns, then the just end results is okay is not okay because it's supposed to learn on the fly to. So these two are related but different problems or different perspective at least. And then the exploration, then the on the go is, I would say, I think at least that it could be more conservative way right be our search space exploration space is much limited because we want we want. At this time, you do not want to try to far away from the my current state, which could cause a, you know, in safety. So the, and the, from my perspective and one of the my students also working on is that initially we get the safe space, and then, and this is done by batch manner. We analyze the data and then analyze the system and learn to models and control policies and so on and so forth. And, but certainly the older data we gathered may not necessarily be reaching out or complete and it shouldn't be complete right so they we also allow the system learns on the run. And the staff size to improve or they change the parameters and could be a much smaller and then check and also the computational issues so one of the Q&A questions I saw is that the how onboard sensor at the onboard computer has a computational capabilities. That's also very important I'm using mostly small drone, and it has very very limited computational capabilities. So that's another issues from the other perspective, but really. Any thoughts on the add, I can only add maybe a couple points on the application point of view. So when you apply reinforcement learning or learning algorithm to a robot, a real robot, for example, our robot which is only, which won't have to actually control in full 60 degree freedom. And of course, the reason we want to try learning, you know two reasons one is that in those fancy maneuvers like acrobatic maneuvers. The aerodynamics so complicated dynamics itself is so complicated so model based control no matter how good because sometimes will not give you a good result. So we use we started with using reinforcement learning as a supplement to our conventional controls. And then we later found that actually some cases it's even better. So sometimes we even, you know, replace it in certain cases. So another, for example, when you're doing flip right upside down, your conventional control will try to stabilize the system. So sometimes it'll be a contradict to your goal. So in that case, we have to shut it down and just use reinforcement learning. So for application for safety of the robot, of course, we didn't crash. We didn't crash the robot so many. So for the safety, I think, when you are training in a computer. So we put we go from simulation to experiment. And usually we're pretty successful in a sense, for example, we do 10 experiments. There will be like six successful case and actually two very good cases. And of course there will be failed cases say so if it doesn't do a nice tight flip. And it will fail but it will diverge but again it will stabilize itself immediately because your commercial control catch up at that moment to at least stabilize it and lend it safely. My experience is that when you train your algorithms you have to randomize your dynamics so put random inject randomness in your dynamic parameters and take account into I mean taking to account your sensor noise, your even communication delay, basically, whatever will happen in the real world, you need to try to take into account account in the simulator so that, you know, the more you train it and it'll be more robust. I mean, that's my own my my two cents in the application side. Thank you for the Japanese quick thoughts or show you a little bit. I think that this is the there must be a trade off curve between the system performance, and also the safety concerns, and from our group who more like to formulate our model the safety and a new objective function. And, you know, just giving a pre designed autonomous autonomous control system. And then when they're at the additional constraints, we add another layer or a feedback loop feedback to that tuning parameter. So this is the way how we deal with the safety constant here. Perfect. So we had a question in the in from the audience about computational capabilities which Professor Wong alluded to and I think this also pertain to something that Professor Tom and talked about her talk about computing these reachable sets online right coming up with fast approaches. So could the panel comment on the computing capabilities and requirements for these robots to achieve and implement these types of strategies in real time, often in these, you know, resource limited platforms. I mean, this is so big research direction for computing reachable sets and I see there's another question in the chat about that, like, for multiple vehicles, if you have reach avoid games for example with multiple players, the dimensions of the problem are increasing the more, the more players you have. And the, the, the technologies that we've addressed these problems with have been decomposition, if you can decompose the problem we've had some success there. You know, you could impose some rules of the road. So some protocols to make sure that you've really only limited your tight interactions to groups of two or three agents. And then this deep reach methodology I think is quite promising that that I presented where you're using a machine learning method to solve the PDE. But these are all very good. These are all still very hard computational problems and dimensionality is an issue for doing the training of the neural net so that that training we do for perception. We do all of that offline. And, and then we port it and use the neural net that we've, we've previously trained to use it in real time. So the training is all offline and typically requires we've, we've either been using the GPU based server that we have in the lab or we with our collaborations with Google, we can, we've got some Google cloud credits there but yeah those are those are offline computations right now. Shinyan did you have any thoughts on the computational requirements for example for your little hummingbirds, I presume if you were to fly those is going to be extremely computationally restricted. Any thoughts on applicability to those kinds of platforms. So basically, the computational heavy is not training training phase, of course, your GPU in the lab. After that, we put it into the microcontroller and the vehicle. It's, you know, it's, it's okay. Yeah. Yeah, I think for competition. Even more complicated sometimes, you know, some some of algorithms we used. Yeah. I think we've come to especially autonomous system, especially Robert, I think the most heavy part is the processing the measurement right, it's better for the environmental perception part. And so we have, we have a problem always have a problem that this for number $75 co-rotors. And I think recently we're more like to try to employ the idea of agile to employ additional to just for just for the processing of the for them for the images for the one where I'm going to press the processing part. I think a competition burden, more than the competition of this competition burden is definitely one of the key to to solve this capability, competition capability thing. Great. There's a question about cybersecurity so we've been talking about safety and you know essentially sort of ensuring that things work, you know, even in normal situations is a difficult thing right essentially that the robust do what they're supposed to do. If you were to throw in adversaries that are maliciously trying to inject attacks and so forth. What are the, what are your perspectives on how things have to change. What, what, what is different from about security from, say standard robustness techniques or fault tolerance. So you have to deal with the real time malicious attack. So that means you have to have a real time monitoring of the whole system. So if the system is 40, you have to have a way to compare it with the say correct or I mean intact signals. So that's why you still need to have some redundancy there. And then we have to retrofit a system and on the other hand, we did try some learning algorithm to on top of a classical like stabilizing control for example, and then we have some learning additional learning module to deal with of let's say to try to deal with so that you can add additional torques for controls on top of that conventional stabilizing controller to deal with the disturbances, that type of thing. That's what we did. And there's a, I believe we had a e-crow I was before on that too. It's just a initial results for now. And another point that we should, I feel like I'm answering questions with more questions but if you, if you're using a learning algorithm in your feedback loop like we're using for perception. And your enemy knows that you're using a learning algorithm, and they probably can guess what type of learning algorithm you're using it can becomes like very easy for them to figure out how to spoof it by giving exactly the images which look to the human eye like they're correct and can mess the system up. So that is also quite a, you know, a challenging issue when you're using learning and somebody learns the learning algorithm that you're using. Yeah, I think there's a community work on the adverse through AI. Yeah, you can follow the AI I mean even you can. I think we had some interest in this area a couple of years ago too so you can use some similarly morphology similar to a bird, a robot bird or something you can for you can easily for the AI algorithm. Yeah. That's a good point. It's like a sword and you know, the fence. Yeah. And I think in the context of multi agent system shall show a. Do you have any thoughts on security and the additional challenges that multiple robots working together introduce into the, when you consider security. Our recent work more like, I think we can see there are the worst scenario, but consider it more intelligent attacker, which can pretend to be a good one but send the wrong information. And we also, I think we more like a proposal, perhaps a very conservative technique, and which can guarantee us to achieve. At least for convincing the best algorithm, but the cost is a little bit network redundancy and information redundancy, but we don't have to identify which one is the attacker. I mean we use all the information we received, but the algorithm will automatically filter the wrong information, the impact of the wrong information. And I think we've come to multi the cyber attacks for multi agent system and it can become hard. I think especially when we consider implementation issue about here is with a limited capability of processing and also with the limited information, especially with limited information, a global information, and issue about only currently with a certain nearby neighbors. So I will say become more and more challenging. Absolutely. The, some other questions pertain to this straight off again between safety and and learning. There was one question that came back to this idea of how do we balance off between these two and then the question pertains to is there a way, is there a notion of safety and aggression, let's say, that could be introduced into these learning algorithms to allow them to selectively trade off or perhaps, like Professor Tomlin, I think you answered in response to the question about conservative assumptions about the disturbances. If you know you're in sort of a safe region, you could perhaps relax that assumption, and, you know, and perhaps explore a little bit more, whereas if you're close to the boundary maybe have to be more careful. So is there some sort of a notion that could be introduced that allows us to dynamically trade off between these two things in a in a learning algorithm as opposed to, you know, focusing on a hard notion of safe versus exploration. I think that, yes, there is in, but I, I'm not sure how to think about it in general, I, I mean, one thing to say is that if you are further away from a safety boundary you can afford to be more aggressive. However, systems are typically designed such that the safety and performance boundaries are kind of budding up against each other so if you're going to be more aggressive that's automatically taking you to the boundary of a safety controller right because it means in vehicles you're going to go to a higher velocity you're going to which, you know, whether it's your own vehicle or whether you're looking at vehicles around you that becomes a more unsafe condition typically. So, so, you know, often, often the high performance or the more aggressive maneuver is directly budding up against safety. I think there is a trade off and maybe the development of computational tools which allow you to explore this trade off and wait it is. It's a useful tool to have. And then that's what we have been using these level set methods for. And now we're working on a problem, really not related to learning just switching between the, well we could use it in learning but switching between the performance based control and the safety based control and how to do that in a way that you, you're blending both of them yet still maintaining safety. Perfect. So I think I'd like to end off here we just have a couple minutes with just circling back around to the human side of things which we haven't discussed as much here but again was a feature of Professor Talman's talk. And so of course this idea that robots are interacting with humans and that's a fundamental characteristic of robots and that's going to increase. And the sort of perception is reality right so I think coming back to this idea of performance versus safety envelopes and optimally performing robot that skirts. The safety boundary may still be perceived to be risky or you know not well performing by the humans that are trying to control those. So, you know, then that might lead the humans to not use those I think as insert mentioned in his slides. So, how do we start thinking about designing not only for safety but also for the perceptions of the humans that use those robots is there are other tools and techniques that currently exists to think about those things, or is that an area for for significant future work. And one just as a side note to that in one project we had a bunch of human subjects study the reachable like the problem of vehicles moving close to each other and assess whether they thought this was a safe initial condition or not. And then we tiled the space and compared it with a reachability computation and humans were in general more conservative than, as you say, then the optimal control is optimal control just skirts corners and loves to do things right at the edge. So, yes, I think the work for example that shall show a presented in terms of thinking about how people, like incorporating human input into the system, not only for human assisted safety, but also for the, the incorporation of perceptions of what is safe and not is very important. So, yes, regarding for the human input, I think beside the side that present we more like a integrated humans sparse input into robots planning. We also have recently developed a method based on universe optimal control to analyze the human motions. So, the basic assumption then assume human or human motions are optimal or nearly optimal. And we want to use use observations of humans gestures for the human motion analysis. And the method is by simulation and also real human motion data, and it match our algorithm matches this, this real human data I think it's very well 99% of the data points are very well, and it's more like it turns out to be I would say, it's how to utilize control theory, and especially the universe of control to analyze the motion for machine to understand what the human motion segmentation, or and also predict what is human next emotion is also a key problem for the human machine interaction. But of course, this is based on human motion optimal or nearly optimal assumption. Great. So I think we're at the five o'clock mark here so that was a lovely panel, there are a lot of food for thought. I think, you know, hopefully the audience also had many other questions answered. Clearly this is a rich area for future exploration, more questions and answers is always a good thing, especially for all the graduate students that are that are attending. And so I'd like to thank the panel again, Professor Tom and thank you so much for a wonderful lecture and panel and visit today. Thank you so much for, you know, giving us your thoughts and ideas here and thank you everybody who attended this I hope you have a wonderful weekend, and we really enjoyed your visit Professor Tom and thank you. Thank you for hosting me. It's been a pleasure to join you in these discussions. And thank you for such a thoughtful organization of a panel and also the discussions. I very much appreciate you coming here. Thank you. Bye bye everyone. Thank you very much.