 Thank you everyone for coming in. Apologies for the glitch there on the technology side. My name is Nico Larko. I'm the director of the Urbanism Next Center at the University of Oregon, and we are thrilled to have you in this session today, and it's just great to lead up these last few sessions talking about these topics. The panel this morning and the conversations just happened with Elberto and Felipe. So many really interesting issues that were brought up there, and we're happy to be able to unpack some of those things in the session that we're about to have right now. As was said, there is a lot of pressure and funding going into AV development and deployment, and one of the issues is that they're coming. So the big question is, what does it mean when they arrive? How are they going to impact communities? What should cities be doing about this? What's the role of governance? One of the questions that was asked this morning, which we'll be talking some about today. How do we ensure that we're leveraging this technology to get to positive public good outcomes? So make sure that we're getting to the things that communities want. So very aligned with the discussions that were started this morning, and we're really excited to have some time to really go through this project that we've been working on, the night AV initiative, with the cities that are working on that, with people from Kiwi Bot, and with all of you. So we're hoping this is a good conversation. So can we go to the next slide? Maybe we can. All right, that's not exactly the next slide, but I'll just go through. So the night AV initiative includes a couple of different groups. The first is Urban is Next, so we're in charge of the research portion of this project, kind of extracting and synthesizing lessons learned and disseminating these learnings. There's Citify, which is working mostly on the project management piece of this, and a whole lot of the kind of technical aspects of this as well. And then we have four cities, the city of San Jose, Miami-Dade County, the city of Detroit, and the city of Pittsburgh, which you're going to be hearing from a number of those today. So maybe we'll introduce people. I'm not sure. I hope everyone's here and visible and can speak, but just to introduce the team itself. Again, I'm Nico from the University of Oregon, Urban is Next. Becky, do you want to introduce yourself? Yeah. And actually, Nico, we don't have quite everyone here. So if you just want to tell folks who's on the team, I think that would be the most efficient. Okay, great. So it's myself, Becky Stechler, Amanda Howell are all from Urban is Next, and then from Citify, Gabe, Story Bellows, and Karen Bridgeford. And Lily and Coral is the person from Night in Charge of the entire initiative. And I'm going to leave the panelists to have introductions from the cities later on as we go to that. So we're going to do a quick poll. Is that going to be possible to do, Becky? No, that's not possible, Nico. I think we should skip that. Okay. So the one last thing I want to say before I turn over to Story to talk a little bit about how it is that this has all evolved is that while we're talking about AVs, it's really important for us to think about this in a broader context of how we work with new technologies. That's a perennial issue. And the learnings that we're going to be talking about are widely applicable. What are the opportunities in new technology? What are the threats? What do communities want? How are they going to engage with new tech? Can it help us reach community goals? What are on the ground needs for providers, for cities, for communities? And what are the tools and levers that cities dispose of? How do we structure the engagement between public and private sector? So while we're going to be talking about AVs particularly, and hopefully we're going to get into a whole lot of the nuances and really interesting lessons learned that are coming out of this, we really need to think about how these same learnings and issues are applicable to just about any innovation arriving in cities. And the last few years are any indication or any gauge. If we think about Uber, Lyft, scooters, moped, car share, bike share, all these things that arrive in the cities, they're going to be more and more of them. So a lot of this is, will hopefully, you'll see the connections between the specifics of AVs, but then how we can really apply that to the broader context of how technology, new technologies arrive in cities, especially around transportation. So the structure of today, what are we going to do? Story's going to do a presentation talking about the project background and what are we doing specifically right now. We're going to have a panel with a few of the cohort cities and David from Kiwi Bot, who's the delivery bot provider operator on surprise assumptions. And then we're going to have a discussion with all of you on two main topics, the challenges of engaging a community around an emerging technology, where we don't know exactly how it's going to be deployed, what the needs are, how communities are really going to engage with it. And the second topic is how to shape deployment through public-private partnerships. And we're really looking forward to both the sharing the lessons learned that we have on this, but also hearing from all of you on this. So with that, I'll turn it over to Story to talk about the overview of the project. Great. Thanks, Niko. So when we started this project in 2018, we were thinking about the potential impacts that automation is going to have in cities, both positive and negative. And AVs in particular could increase safety and help reduce congestion and pollution, but they could also, due to their costs and their potential impacts on transit, make some of our existing transportation systems even more inequitable than they already are. And so I think we see the timeline for mass adoption of AVs feels a bit more uncertain now than it seemed in 2018. Our primary takeaway remains the same, engaging with AVs. Are they the worst? And so in 2018, the Knight Foundation funded a five-year, five-million-dollar initiative bringing residents to the center of self-driving vehicle pilot projects. And you can all read this, I'll actually read it out loud, that the pilot projects will be designed to engage local residents around self-driving car deployments to ensure that they reflect community impact and meet local needs. Leaders from the pilot cities will share what they learn and meet regularly to generate insights and lessons for other communities, trying to keep up with the increasing pace of testing and deployment of self-driving vehicles. Next slide, please, Becky. And so as Nico mentioned, this project is being co-run by CityFy and the Urbanism Neck Center at the University of Oregon. Nico spoke a little bit about each of us. And we have four cohort communities, which are San Jose, Miami-Dade County, the city of Detroit, and the city of Pittsburgh. And so our work was started focusing on supporting the engagement efforts of existing AV pilots that were planned for or being rolled out in each of these communities. And so I'm going to start with, Becky, if you can jump forward to the assumptions, some of the assumptions that we had going in, which started with, and I don't know if the presentation looks a little crazy on your screen. So I'm just going to keep going and assume that maybe the presentation will catch up. Maybe it won't, not super critical. But some of our assumptions were that the pace of testing and deployment of AVs was increasing and the pressure was on in cities across the country. Also that AVs may add to or detract from the quality of life of residents, particularly those from disadvantaged communities, that by engaging residents and bringing the cities to the table during the testing of AVs, there is the potential to maximize community benefits. So this is 2018. A lot has changed since then. And a lot was changing even pre-COVID. But part of it is that the pace of AV technology was advancing a lot more slowly than we had, and I think most people in the mobility space had anticipated, leading to some pretty significant market shift. The AV landscape overall had shifted, has shifted. There's a bit less, and I think you heard about this earlier, a bit less of a focus on robotaxies, particularly in-dense cities, and a greater focus on things like long-haul trucking and goods delivery. And we were seeing these shifts happen even pre-COVID before people had stopped traveling as much, and we were really focused on the travel of goods. And I think really critically as well is there's a lack of city influence over the standard AV testing models, and a lot of preemption happening at the state level. Carlos mentioned this earlier as well. And so given all of these changes and some of our assumptions that were correct, some of them that have shifted, we changed our approach in this project. And so we pivoted, and instead of the four cities working on disparate pilot projects and disparate engagement initiatives and efforts, we decided to come together and look at a single approach to, or a single type of piloting. We didn't initially think about all piloting with KiwiBot. We were exploring a number of different personal delivery device providers, but we decided to shift our cohort to all focusing on personal delivery device pilots. And some of the examples that we're using in the four cities are, as you heard from Felipe, restaurants, vegetable boxes, also talking to libraries, talking to pharmacies about how we distribute some of those goods and resources as well. And at a glance, and we've begun to roll out in Pittsburgh, Miami-Dade, and San Jose, we've had 48 customer interactions and 59 miles of streets mapped in Pittsburgh, 92 interactions and 47 miles mapped in Miami-Dade, and San Jose 103 interactions and 106 miles mapped already. And these pilots are just underway, providing really interested, the start of some really interesting learnings for us both that are relevant for the city, from technical learning, thinking about some of the opportunities and real challenges in the public right-of-way, looking at the different form factors, some of the use cases and the different models, as well as looking at some programmatic learnings, thinking about what are the opportunities and challenges around the different use cases, the intersections with public space, public policy, and some of the equity issues. And so our key objectives really are to provide the cities with the opportunity to learn both these technical and programmatic issues, and then to go back to, you know, the Knight Foundation's original focus on this project to make sure that we are engaging the community through thoughtful piloting in order to promote education and learning about these new technologies, and to better understand how the application of these technologies aligns or doesn't align with community needs. We're looking to identify some of the levers that cities have to proactively rather than reactively shape how autonomous technologies are deployed in the public right-of-way in pursuit of the public good and these desired outcomes. And all of these are things that we can think about in theory, but we really need to collaborate and to test these on the ground in order to identify the opportunities and these levers in practice. And I think incredibly importantly in engaging the private sector has always been a part of this initiative, but as we have pivoted the approach to this program, thinking about how we work collaboratively with early stage testers in the private sector is a huge opportunity for us to identify some of the places where we may be actually able to shape their business models, to think about how do we align the incentives to deliver the positive societal outcomes that we're hoping to see. And then I think it's important to understand what we're not trying to do. This initiative was never intended to solve systemic inequity and transportation and land use in food deserts through technology. We saw these technologies coming and so we want to make sure that cities have the opportunity to shape their deployment. We are not looking to make a determination as to whether these technologies are good or bad, but we do think that communities need to better understand the potential opportunities and challenges of these opportunities in order to maximize the public good and minimize the externalities. We also think that engagement with the private sector is really, really important to influence those business models, those use cases, and the regulatory understanding so that we can continue, again, to maximize the public good and minimize those externalities. And so with that, I am going to kick it over to Gabe to begin the next discussion. All right. Hey, everybody. Gabe Klein here. We're having a few technical difficulties. We had a deck, but we seem to be having trouble sharing it, but we're just going to go ahead without a deck because you don't need a deck necessarily to talk. Oh, there we go. Wonderful. So as story outline, we had some surprises. I think everybody had some surprises over the last couple of years. Some of the surprises were not pandemic related, but the pandemic absolutely changed the character of the work that we were doing. And so some of the basic premises, can the public and private sector work together? Can we describe autonomy to people and based on what they're seeing on their streets, get a sense of what that means versus showing it to people? Would working more collaboratively between public and private, would it give cities and communities a greater set over the outcomes and business cases and use cases in their communities? And so we definitely pivoted. We made some changes. And I think the program is a lot stronger for it. As Felipe talked about without Alberto, the pandemic has been terrible, but it did express itself in our world in such a way that we realized that delivery was not only a trend, but it was an absolute necessity. And so while we started out talking a bit about ride hail and various other people moving use cases, suddenly people were stuck at home. We had people that were disabled. We had people that were elderly, people that were more at risk. And so this felt pretty essential and the cities are the key drivers in this program. And the cities were like, hey, this is a use case that we really need right now. It's not something theoretical in the future. It's something that we're hearing from community members, from hospitals, from pharmacies, from other government agencies that we need this now. So we're going to have a little bit of a discussion. We have a great panel of folks that are going to be joining us. And I'm going to be co-moderating here with Nico Larco. So if we can go to slide 22. We've got Sarah Abroff, our compatriot in the city of San Jose, who joined this project about a year ago, Sarah. Eight months ago. And has been a wonderful addition to this collective. Erin Clark from the city of Pittsburgh. Hey, Erin. I'm Jerry Rodriguez, Miami-Dade County. By the way, Dr. Jerry's, we learned. Do we have Irene on or no? No, she's not. All right. And we've got, I believe, David Rodriguez. No relation, right, Jerry? No. From KiwiBuck, who we've worked very closely with throughout this process and has been really, really wonderful. And then of course, as I said, Nico, Nico and his team and our team are matrixed and we work together very closely on this program and the individual projects. Nico, anything you want to say before we dive into the group discussion? No, that was a great introduction and looking forward to hearing the conversation. All right. So if we can go forward a slide. Great. So these are some initial prompts, you know, some of the challenges that we've seen. And well, actually, these are more like the results of some of the challenges. So we're going to, you know, dive into some deeper questions. We'll take questions from the audience as well. But for the cities, you know, well, first of all, before we say, are we surprised at how you've been able to influence it? Have you been able to influence KiwiBuck's thinking, testing and operations? And what were your assumptions based on the partnerships you've had in the past versus this one, which is really, you know, curated so that you have more control? Who wants to start? Should I call on Aaron Clark? Sure. Yeah, I think we have been able through this partnership to kind of influence KiwiBuck's thinking. And that's, you know, in large part because of this grant. And the cities are in a really unique position through this grant that we do get to kind of write the rules. And KiwiBuck's very willing to work closely with us and explore things maybe they haven't explored in the past. Assumptions based on our past private sector partnerships, I think cities are rarely in the, you know, driver's seat during these types of deployments. We have some ability to issue, you know, permits to operate with some terms and conditions associated, but we're often preempted from really having a lot of control on how those pilots roll out. And, you know, the private sector is often reluctant to be fully transparent, especially when they're kind of a younger company. So yeah, it's been very refreshing to have this type of like close collaboration with KiwiBuck during this pilot. Jerice or Sarah, have you had a different experience or anything to add? Yes. Something that was very important for us is the willingness that KiwiBuck has since the beginning. So when we started asking about what we want to respect it, we would like to do like for example, the collation of the sidewalk infrastructure data, that thing they were willing to try. They were willing to really go ahead and try to meet our needs. Even that when things are not possible, or maybe they say to us, we're going to try. And that's just how we have seen that just sharing our needs, what we'd like to have over challenges now, it's a way to influence the way that has been changing or influenced KiwiBuck, but also is there the willingness to participate, to collaborate, to work, to learn, to see how can help the community and how can help us as a city, as a county, as a government. Yeah. And I was just going to add that from San Jose's perspective, you know, we actually started our partnership with KiwiBuck just over a year ago, before we were even involved with using KiwiBuck in this initiative. And we have always felt that they were a very good partner. It's one of the reasons that we involved them in this initiative from the get go. I think, you know, when I joined the city, a lot of what I heard was we don't want this roll out to be the way like scooters just landed and kind of exploded. Like we want to have more of an influence and that I think we've been really careful from the beginning with bringing robots to the city. And KiwiBuck has been a really good partner with us in terms of the push and pull, the learning on either side and taking things slowly. And we've really appreciated it the whole way. So can I ask, because I mean, there's going to be other cities and other organizations that are listening to this, and they're wondering, like, do we just need to get David? Like, is that the way we have good relate? Like, what makes the relationship work? What's making the collaboration be productive? And, you know, maybe compared to some of the other work you've done with other technologies that thrive in your city, what's different about this? And what can we learn? I mean, I feel cliche sort of saying it, but I think there's a level of transparency and trust that we have. Like, I think we are trying to be really honest from the city perspective of our concerns right from the get go, but not just our concerns, like our hopes and our ambitions and then like what, you know, instead of just taking our concerns and saying no to this technology, like how can we take them and be like, address concerns with the technology and just be, we're very open, I think, with them. And, you know, we have regular meetings with them. And from my perspective, at least, I always feel that I'm getting, like, a straight answer from David and the rest of the QEBOT team that I've talked to. So for me, I think it has a lot to do with this trust and transparency. And if we can bring those kinds of relationships to other companies, I think that's what would work really well. So, head nods from Erin and Joris, as you said, trust and transparency. Erin and Joris, do you have anything you want to add to that? Yes. From the very beginning, I think one of the main factors has been communication. We do have weekly meetings. We talk with David and it feels like you're talking with a friend or someone like that. It's not like when we say something and we propose an idea or maybe that is not possible, that it's not always, that it's like, we can try. Yes. They never say no from the very beginning. They always are willing to try, to go ahead, to help. And we don't always know what to ask or how to ask or what we need. But we have that feedback from QEBOT. We have that willingness to help, to learn, to try, at least to try. It's not like, and it's easy to talk and express what we are feeling, what we are afraid of, what we don't know, what to expect, that with other partnerships that we have. And I think it's just the good communication, frequent communication. This is going both ways and with all the teams in the project. And I think that has been the main factor and also having clear what is the role of each of the member in the team. So why don't cities and companies co-create more for better societal outcomes, for better, like to matrix, like the best public-private partnerships I've ever like around bike share. And you know, it's like we had matrix teams, public and private. Why don't we do more of that? Why is this unusual? I mean, I think to some degree, and this is like a credit to QEBOT, they're willing to do the uncomfortable work that other private sector companies might not be as willing to do. That means like in Pittsburgh, QEBOT, David individually attended some virtual community meetings that had a lot of, you know, skeptics, I would say, of this technology. And, you know, they're willing to get in front of the community, they're willing to have these kind of harder conversations with cities and to work through those challenges. I think there's like a hesitation, and understandably so to some degree, but from new startup companies to appear as they've made any mistakes whatsoever. And from the city's perspective, we expect mistakes to be made, I think. And that's why we want to work closely, especially like during, you know, pre-deployment and during early deployments. And so, yeah, I think like that's what's been different. It's like there's less of a fear, you know, to step up to the plate and kind of have those uncomfortable conversations and work through the hard things with the city. And anyone else? I mean, I got that, you know, there's like, like if you're going to do experiments, you have to be willing for things to not go right, so that you can learn. And if a company just wants good press, and if a government doesn't want to stick their neck out and only wants good press, it's going to be hard to learn together, isn't it? So David, why are you willing to do this? A lot of companies don't want to do it. We talk to other companies and they're like, yeah, I don't know, that sounds a little risky. You guys were up for it. What was the benefit? You're on mute. There you go. Hello. And thanks for the opportunity to be part of this forum. I think that some companies don't think enough about the long term. Every time we go to a new community, we have a long term approach. We want to serve the community and create deep relationships, meaningful relationships, having the understanding of where you're at, what's the dynamic can definitely make you different when it comes to providing long term value, which is ultimately what we're chasing. We are harnessing on the power of public-private partnerships to understand a new country, a new culture, a new way of doing business and this has led us to groundbreaking discoveries that are actually shifting the market. So I do believe that being respectful and opening doors in a way that makes sense with the community is just what we were taught at home and maybe something that Silicon Valley could learn a lot from. So that's our position. And let me ask both David and the cities as well. It seems like one of the things that's working well is this co-creation and in peace, in part this was created, Knight created the table. Knight said, we are going to bring everyone to this. So I'm thinking of all the cities that don't have Knight creating a table from right now. How do you do that? What's the key piece that you need to be doing in other cities or as you're for the cities from your side or David, as you're approaching other cities? Yeah, I'll summarize it by QEBOT being the robot delivery company that gets college campus students meals of their meal plans. That's what we did before COVID. COVID forced us out of our comfort zone, which are college campuses and we realized there is a lot more a robot can do, literally saving lives, learning from cities concepts like vision zero or high incident corridors. And we learned that our technology can inform cities to make better planning decisions that ultimately have an impact on human lives. It's not only something rewarding for us as human beings and a company, but also creates a new layer of the service where anyone will see a QEBOT and not only think that it makes deliveries, but rather than create social good. So I think it's part of growing up and definitely a great thing that spun out of our this opportunity after COVID. So can we talk real quick about what you're forcing Kiwi to do now? I was getting like what's actually happening like use cases in the communities. Like for, okay you Sarah, San Jose, what has been surprising or have you learned about designing a good public use case? Like you did some amazing stuff, community fridge, veggie pollution with Kiwi. And like what are the benefits and risks of trying to serve a public good through a pilot? Yeah, so you know we definitely took this really seriously. We really had a really just a huge equity component in San Jose. This was our priority. And this is kind of like two sides. Like one side is we don't want to harm any communities while we're doing this. And the other side is like beyond just not harming communities, can we actually use the robots to deliver something good to communities or to residents of the city that we wouldn't normally be able to reach this technology? And how do we get their feedback? So you know I started early when we had this just talking to various community groups of all different kinds. And I think one thing that was really surprising is how responsive people were to this. I mean no one was rejecting of a conversation with me about this. You know I think I got one response that said like please don't bring them to our neighborhood right now. But everybody else was so thrilled that I was reaching out and asking for feedback on these delivery robots even in areas where we couldn't find a use case. People were really happy I was reaching out and wanted me to follow up later on. And through this outreach we actually found a few use cases that we are really excited about and we think might really work. One of them for example is this urban farm called Vegilution. They have these vegetable boxes. We've already done a demonstration at the farm with Kiwi Bot and we're planning to actually try some vegetable box deliveries using Kiwi Bot and just kind of see how it goes. I think that we don't know if it'll work or not or help or not but we might as well try it. I think that's like the big key in doing all of this. Like I don't want to just not do it because we think it may not work. Like we have to try it and we have to adjust as we go along. Should we take some questions Nico? Yeah I like Josh's question a lot. Maybe I can put that out to everyone. So Josh is asking is the voluntary cooperation model sustainable and scalable unless cities create a strong business or regulatory incentive for other companies to come to the table? So is it possible to do this? I mean this looks good. It's working well and we'll get to some of these some of the outcomes later in the session and in the next piece that we're doing. But do you all feel this is wonderful that this is happening with all this kind of energy being put in by night helping being put in by night and all this? Is it scalable? Why not? Why would you be part of this would be one of the questions for David in other places and for the cities? Could you see this in other in other areas? Yeah sustainability is definitely like financial sustainability is definitely a big topic. I think that the necessity exists. Citizens that are paying tax dollars have limited accessibility to goods and they need to have access. It's not a question. It's human rights and it's something that has a space. How do we make it happen is part of the answer that we're solving as participating despite. When when wind vision models like this we're looking at a future where options QE but can provide a public service for free to the community if we leverage on an advertising deal or other ways like cities paying for site book infrastructure data that we collect as we do deliveries for free and other sustainability mode or even that we make arrangements so that QE we bought operation will have a cost for members of the community that can pay for it and leverage for other members of the community that do not. The most important thing for us right now is how do we operate? How do we properly engage with supply and demand parts of the community and how do we create a baseline so that the adoption of technology is not going to cause problems or steer too much for other members of the communities. So but let me interrupt you for a second. I mean the basic question then is is the cost right the cost and risk of this engagement bigger right or is it bigger or less than the benefits that come out of it right? I mean that that is the basic so you know definitely energy time risk in engaging with cities and trying to co-create here. Tell us about do you see outcomes of this which outweigh that cost and and and you know if not what you know what would need to change to make that happen I think I think that that the way to upset that risk I don't think that happens and I definitely think that just the connections and the relationships you create when you work in a community like this has long-term value that upsets the opportunity cost of not doing it. Learning how to launch properly is another value that we're getting. Even connections with other big companies that arise from this is always a good thing to have. Investors are coming to us because they know we're doing things differently and I would suggest not even for us but other micro-mobility companies to take this approach because the externalities that sprout out of the operation are always greater than the opportunity cost of presumably being here. What do the cities think? I think you know Gabe made a good point in the chat that we hope that there will be some positive outcome that outweighs some of the the costs and time consumption of these like close relationships in terms of like learnings and just gaining that public trust. I think that's something we hear a lot from the AV industry. A big barrier is public trust public trust and through these types of close partnerships I think we're getting to that public trust a little faster. Not claiming that it's going to be pretty. It's you know community engagement and getting that public trust takes hard work but again I think like he was doing that work through this and I think they will have you know more positive relationships with the communities they serve because of that work that they do. So yeah I know David spoke a little more to the sustainability but I hope that this works better than cities having to try to create you know regulatory hurdles or you know regulate the industry in order to get the outcomes we're all searching for. But yeah I'm not sure if the other cities have anything to add. Sarah, Jerice? I was going to add that I still feel like we're so new in this in this new technology and we're just really trying right now. We're just trying things and that's sort of the point of the pilot and I can't speak as much to the sustainability but I kind of want to at least start from the side because if we find some really great use case that really helps people maybe we figure it out along the way. That's an unknown answer but it's kind of like hopeful of where we might go with it. Great. I agree with Sarah. I think that one of the main goals of this initiative is to make the overall residents for people the center of the pilot of the project and if we work together in we in casual work community we're going to have the probably and we are hoping to have that positive result that is acceptance and adaptation and in this something that like Sarah says completely new we don't know we don't know what to expect we don't know and we are learning and that's one of the the reason why there's a pilot we are learning through the process and we know that we are going to have been there's going to be a lot of changes as the pilot evolve but we are open to that because we are learning that the reason why we're here we need to to learn in order to get ready for this kind of deployment and everything that comes after the delivery of the robot after the bot. And I know we're probably going to hop to the next question but I just wanted to add one thing like obviously on the public side a lot of us don't have as much experience with like business models and the sustainability of the business but I think we've been having helpful conversations with Kiwi Bot that's allowed them to be more creative. So we've like talked about scenarios in which you know how can we provide a public benefit while also ensuring that this is a sustainable business and in a lot of cities for instance you know if the way to make money is centered mostly around food delivery and restaurants well a lot of restaurants are closed on Monday Tuesday sometimes Wednesdays and so what can the companies do on those kind of like slower days to help the communities that they're serving and maybe shifting resources on those days to serve those community goods that the residents and community members have identified while also allowing them to kind of provide this this commercial service as well. Great and we're going to ask a couple last questions we're going to transition it over to Amanda and Cameron for their session but you know Brett asked a few questions like where's Big Tech and is there competition among private companies seeking partnerships? I mean one of the great things about this you know Knight has been obviously an amazing partner and funder misses their program and you know they've been very open to pivoting experimentation and you know we interviewed most of the major delivery companies and bot companies some were going through mergers and changes some were just raising funds for very particular things that they were doing. At the end of the day Kiwi was really the one company that was excited and flexible and wanted to to do this and I would say just being completely blunt I think if they had raised a hundred million dollars they may not have been. They're sort of younger, hungrier. You know we talked to some companies that have raised a billion dollars in the space and they were wonderful and great people but they're very focused on proving their particular use case. I think companies like KiwiBot and there's more and more of them now there's more of these you know young sort of upstart companies they understand that their key differentiator differentiator is going to be potentially their relationship with the communities that they are operating and not just with the government with the local businesses with the communities and with the people and so this idea of getting feedback from people is sort of in their DNA to shape how they roll out their business. So anyway hopefully Brett that answers some of your questions and I think Niko you're going to ask the last question right? Sure from that last question from Reyes. Yeah so the question was really that Reyes asked is what's the feedback loop with communities right so we're talking about you know fantastic collaboration here with who we have represented as the city's city government and KiwiBot. How does this work with the what kind of feedback are you getting from communities how are you getting that feedback? Sarah, Erin, Jerice who wants to start with that we'll start with Sarah. Sure so getting the feedback from the communities I see as like the key piece of what the city is doing in this project. So we are doing this in a number of different ways we're right at the beginning of this whole process because we're at the beginning of the pilot but we're doing it in a variety of ways in San Jose and San Jose we're doing various demonstration events so that we actually have the robots out on sidewalks or in some public place and we are asking people questions about it letting people come up ask their own questions about it. For instance I mentioned that we brought them to the urban farm and they have these Saturday weekly Saturday events and we brought them there and we got a lot of feedback from people. Another thing we're doing is we have created a number of different surveys so we've created surveys that are available on like the city website where people who see the KiwiBot can can take that survey we're going to be planning to do in-person interviews intercept surveys and observations. We're sort of casting a really wide net of different ways of gaining community feedback because we're not trying to just do the typical you know hold a community meeting and then only the loudest people get their voices heard so we're doing this in all different ways and I think I think we're really you know we are doing surveys but we're also really we're leaning on the qualitative feedback that we're doing and for some of the deliveries that we're going to do from the urban farm we're going to be interviewing the people afterwards about how they feel about it they've been you know they've opted in to to get this kind of delivery so that's what we're doing in San Jose and I know that also KiwiBot themselves have been doing a lot of in-person engagement with people passing by on sidewalks answering questions but I think this is a challenge to get as many voices as we can from as many different communities but but so far we've been doing a good job with it so I will see I don't know if the other cities have other ways that they're sharing. Yeah I think we're doing things like similarly but a little differently in Pittsburgh so we've hosted a couple virtual community meetings with the community where we're deploying so that was kind of our starting point though even before those community meetings I was engaging really regularly with the registered community organization representing that community um and so after the like questions and feedback and I know we got a question kind of asking about the main concerns of the communities um during those meetings we kind of captured those concerns and took steps to address those so one of the things we're working on right now with the community organization is um creating a community commitment that's laying out all the kind of promises that we've made during this pilot to say that we're going to be transparent we're going to maintain this information in this location um we're going to collect feedback in these ways we're going to issue kind of evaluation reports um and so I think that's been really helpful like taking those steps to increase transparency and show the community that we're taking their you know concerns and questions seriously and working to address them during the pilot period um and then we're doing things similar to San Jose and that we have a website and we've worked kind of collaboratively across the cities to create those surveys um we're working with um one of our major research institutions Carnegie Mellon here to really help us with some of that um surveying and um and kind of evaluation of of uh the the feedback that we hear from the community during this so um yeah a lot going on I think but it's it's exciting and something we don't always have the resources to do as much as we're doing right now. Teresa is there anything you wanted to add? Um in Miami we're doing in Miami they were doing the things kind of different uh part of this initiative we partnered with four city one and USP that is your band health um partnership is a local non-profit organization and the idea is that we're going to we want to work everything different than the traditional way we have been doing for example four is using uh data science to it helps us to uh identified 11 groups of groups or community groups so that the better represent Miami Bay County and the idea is that having this um identified groups we can bring the community to the pilot place when because we know that nobody's going to have this great access to the pilot and to get the insight. We haven't started the community engagement but as part of this we um USP have a community liaison framework that means that we're going to be hiding uh about 20 big persons from the community community from the uh identified communities um that we'd be working directly as speaking talking uh approaching community the idea is to have someone from the community that is trusted that the community already know that knows very well the community to go to the people to talk it's everything going to depend and how we approach them and that's going to depend on the needs and how the community behave and the community you know background cultural background language etc because Miami Bay County is very very diverse and the idea is to have someone from the community that it identifies with the community and can convey um the information about the project and get the insight to back to us they get the feedback and make them and try to help them to get this experience with that AB technology and learn about before their experience and after what they've seen and then um what why they're what why they like it why they don't like it what changed their mind what they need we want to try to better understand the needs of residents and better understand the mobility experience of a new residence so we want to make the people the center of the project and we think that having people represent we've seen it from every community or group that represents Miami Bay County we're going to get the best participation and engagement and feedback we haven't started the the community engagement yet because we are in the process of hiding the community I think four teams already identify the community also doing a personal research which is going to help us enhance or learn more about the experience of our residents so that way we hope we can learn not only for this project for future projects at Miami Bay County we can learn how to have a better understanding of our needs mobility needs and also better understanding and strengthening that we can use for other projects as well when we are talking about implementing new technologies like AV AV and we have the hope that this is going to be different but helpful for for our pilot. Perfect and that is a great transition to the next session of which one of the one of the topics that we want to talk about was community outreach and not just the how but the kind of the thorny issues around how do you do community outreach with the emerging tech Gabe is there anything you want to say to wrap this up before we pass it on? No I think this is great I'm excited for the continuing conversation. Yeah and thanks to everyone for all the great questions being put in the chat hope to continue to see these and hopefully we can continue these questions after the session as well so with that I will pass it off to Amanda and Cameron and say thank you to the panel. Great thank you so much and many of these spaces if not all of them are going to stay up here because we're going to continue to talk with them I'm Amanda Hall I'm a researcher with the University of Oregon and I've been working on developing some of the research tools that we've been using to try and understand what it is and some of the important learnings that we're getting from this from the work that we're doing and I think this conversation that we've been already having thus far really kind of tees up into it and I think you know what we're seeing in the questions that you all are putting in which we are definitely going to get to is you know we there are kind of two big themes that have been coming up and that is one about doing community engagement and how we're talking to people about the technology and then the other piece which we've also touched on already is the ways that we are working and doing a public-private partnership and I think you know this kind of in many ways like first of its kind and type of public partnership with you know these very early early partnerships with doing city-run delivery stuff so with that we do have a poll we're going to try and do a poll and if you see under the session on the session tab on the hopefully the right hand side of your screen you should see chat polls and people and if you go to polls you'll see that we have asked two questions so we're curious to know what your priority is in terms of what you'd like to talk about first hopefully we'll be able to get to both but the first is the first question if you could all vote is the biggest challenges of engaging on emerging technology if we'd like to start with that or how we shape deployment through public-private partnerships so I'll just give you all a quick sec to hopefully do a quick vote and then we will transition over into a conversation that is really specifically about that topic and hopefully back you'll you'll be able to see the results I see oh great I see in real time I think it's looking like from from the vote so far that if there's a slight oh no we're split down the middle here I think sounds like we could start with either I see one more vote for a public-private partnership so let's go ahead and start there I'm going to pass it over to Cameron to tee off that conversation okay great hi good afternoon everyone I'm Cameron Bridgeford I'm a principal at city-fi and I work with Gabe who you heard from earlier and story bellows you also heard from earlier to do program management for the night AV initiative so I think as we talk about sort of topics around shaping deployment through public-private partnerships I think the brief setup for this conversation is the AV initiative is focused on a lot of different threads but one has really been exploring that idea of co-creation of the pilot process and through developing really collaborative public-private partnerships which I think you've heard a lot about that from the previous panel discussion and I think this really comes down to the idea of like how do we how do we meet in the middle for both public and private sector so how do we align desired outcomes both to serve a public good which is obviously of the greatest importance to the public sector but also how do we build a more sustainable business model but as we've done this and tested this over you know the past two years of this initiative a lot of great learnings have come out so far I think so you know there's obviously benefits to this collaborative model but there's also risks to it that I think have to be worked through and mitigated as we sort of move through this project and so I would just start a guess to the to the folks still on the screen David and into our cities just generally opening anything you haven't shared so far about what you've learned about a pilot process that prioritizes co-creation you know thinking about David at working for a startup company and what that might entail amount of time energy resources and also obviously the processes and bureaucracy that might come with the public sector so those are a couple nuggets to get you thinking but would be happy to turn it over to anyone on the panel and then after we sort of ask this first question audience members please keep putting your questions into the chat Amanda's going to be monitoring this because we really want this to be an open forum but it's not just a panel but really we're going to prioritize your questions so I'll turn it to the panel for that question. Hello so I think that the lesson is that there that are always going to be hard conversations and you can have the hard conversations before or after and the tough conversations are tougher when they happen later so I think that as a company with with private goals having these conversations and engaging in this co-creation is super important to establish ground rules to align incentives and it leads to the type of good outcomes like finding middle grounds where we can have structural operations where the community gets benefited in a way that is aligned to the city's objectives so I do believe that a set of short conversations before launching a pilot can definitely take us a long way when it comes to being aligned long term and I won't stop coming back to this building and structuring long term relationships. Yeah I can add to I agree with what David said I was going to say that when we're doing these kinds of partnerships I think it's important to keep in mind the amount of work that both sides have to do so you know before it's not just that Kiwi Bhatt is coming to San Jose and they're going to be partnering with restaurants you know it's taken a significant amount of work from the city's side to sort of define what we're interested in reach out groups from our end and then work with Kiwi Bhatt to come up you know to agree on the different use cases that we're going to pursue so they come to us with ideas we come to them with ideas and then we really have to work together to mold that and it's hard when there's limited resources but I think that is what's really special about this project it's kind of a unique case where we have the time and money to to do that and put that forward. Drew Sir Aaron any other thoughts about how sort of the creation process with Kiwi Bhatt has shifted based on previous private sector partnerships that you've had in the past? Yeah don't want to be too repetitive but I do think this like another kind of benefit that we should probably highlight of this partnership is that you know we think we're making progress at least here into an improved like regulatory structure by partnering so closely with Kiwi Bhatt we're working through some state regulations and then trying to craft what our local regulations might look like and because we have this like very candid trusting relationship I think we can have conversations about what works and what doesn't work and we trust Kiwi Bhatt to know that they're not just trying to refrain from sharing any information or data with us but rather want to be efficient and answer the questions the city actually wants to answer so if it's you know a safety issue they might be able to give us some insight of like well this isn't really the information that you should ask us for that we should give you to answer this question and so I think that's been like a happy surprise too to see that you know we can actually work with the private sector in a meaningful way to improve kind of like how that regulation looks like in the future or what we should strive for at least. That's great thanks so much for sharing all those thoughts I think because it seems like people are equally interested in hearing about both the engagement and the public-private partnership element of this I'm just going to hop between and in order to make sure we're addressing as many of the questions that have already been put in the chat as possible so I did want to go back to a question that Kenan had asked which is about what the main concerns communities have had with the deployment of babies do you find that these different these concerns differ between individuals and connected interest groups and I think you know I'd love to hear from Sarah, Erin, Dries what you're finding so far with your engagement Erin I'm guessing you might have some things to speak to in particular about your experience in Fedsford so far. Yeah no I'd be happy to so during our community meetings we heard a lot of fear and skepticism I would say about this technology and it's not just about delivery devices as they're called here in Pennsylvania but automation in general I would say so we heard a lot of fear about you know jobs and labor and I know that was kind of talked about in a previous panel at length we also you know for these delivery devices in particular and the smaller devices that are primarily going to operate in pedestrian areas I think you know we talked a lot with or heard a lot from the community about this reality that our sidewalks and pedestrian spaces are for like people and animals like pets and window shopping and talking and socializing and we're introducing robots to this already limited space that we have and that's that's scary that's something we have to figure out how you know to manage and where that isn't is inappropriate in our cities and you know rightly so we heard a lot of concerns from our residents with disabilities and other stakeholders because they already have accessibility issues in our pedestrian spaces and to then add just another it's not even necessarily about it being a robot but it is the deprioritization of humans again in in their kind of view and so how do we ensure that we're still prioritizing humans and ensuring accessibility and safety as we kind of introduce and explore these technologies and explore their their value like societal value um so I think those are like two of the really high level things and and my department is responsible for managing the right of way and so that is a high you know priority concern of ours as well it's like how do we manage this effectively in light of some preemption that we're seeing at the state level and kind of other regulatory realities that we have and so this you know pilot in addition to a lot of other things is an opportunity for us to prepare for what that will look like in the future if more companies were to deploy I'm gonna ask a quick follow-up question from you Erin if that's okay which is just you know you had talked about you know some of the community meetings that had happened before the pilot before the robots on the ground there were a lot of concerns that were raised how has it been since the the robots have been physically there and I know you've spent a little bit of time out on the road with them um how are people actually receiving them now that they're actually they're getting to see it in real life is that allaying any of the concerns or is that raising new concerns it seems I don't want to act as if it's like resolved all of the problems but I think we saw a very drastic difference between the first community meeting we held where we had a lot of questions we couldn't like clearly answer for the community because there were a lot of unknowns with the state regulation that we now have in Pennsylvania and some other questions about the like reality of these devices operating that we needed experience like more experience and learnings before we could answer um but the second community meeting we came and we you know provided a presentation that at minimum acknowledged that we heard those concerns and we're taking steps to address them and also you know articulated the steps we had already taken um to answer some of the questions and concerns that they had raised and the amount of questions and feedback uh like negative or skeptical feedback I guess that we received during that second community meeting was fairly night and day um and so you could see that there was at least uh responsiveness from the community that like we hope they were acknowledging like okay the city is listening to us Cuba is listening to us and they're trying to take steps to like answer those questions through this pilot and so I think it's helped kind of provide some level of trust and so when the robots were actually out doing some some mapping and Julian who is with the cuba team who's here in Pittsburgh without the devices I think we haven't heard as strong of like negative or kind of major reactions and instead it's been like pretty meaningful conversations and questions um from the community while we're out with them and so I think we are seeing progress toward at least acceptance of what this pilot's all about um and what we're all trying to learn together and that both the city and and you know the company are they do have like the community's interests at heart and we're trying to work through those hard things with them awesome thanks for that Erin um Amanda and I are sort of going back to the chat and we see a pretty good question from Yasha from the Open Mobility Foundation um I'm just going to read it pretty verbatim and see what you all have to say he he reads out have there been cases where city and company objectives were fundamentally misaligned um if so how do you navigate those and he gives the example of you know essentially creating a lot of low-value convenience um deliveries on a sidewalk you know through ordering that would create sidewalk congestion for instance um that might have minimal economic or community benefits um the cities are doing good thoughts on that I I could I could begin by kind of generalizing some of what we've seen and actually it's a great question and we actually had a grid answering to this question because it is something that we can really locate so even if it's more complex we can just bring it down to the idea that KiwiBot is an API for companies to send deliveries and we deliver in the case of cities we had the situation where we have partners which are large companies that could serve a social situation so generally this is the importance of having these type of conversations that we have talked in the past where we can be horizontal about what is the outcome of this participation and how can we align these incentives we've come to situations where certain companies simply do not match the city objectives that we've said to ourselves but we've had situations where where other companies actually either tweak the pilot to serve a social business and local community in a degree or other cases where we expand this scope of what we're doing with these companies to include and involve small businesses in a way that we create for a better ecosystem so it is important that sometimes the misalignments between public and cities like might happen but I'm more interested in the times where we actually get to find these middle grounds where the companies we work with ourselves the local community and these small businesses can altogether be benefited from creating this critical massive volume that creates more and more a more sophisticated market that serves the taxpayer who is the ultimate recipient of a better service. Erin, Jerry, Sarah, anything dad? Oh, looks like Erin here. Sorry, it's Sarah, Jerry. Go ahead first. I was just going to say I think like this that question kind of points to another benefit of this like collaborative relationship like there are misalignments there's going to be the public sector has different interests than the private sector and so when you know maybe the public sector wants to move really slowly and and fulfill these community needs that aren't necessarily profitable the private sector then has the pool and they say well we have to be sustainable we need to you know make more deliveries or find this solution here and so it is that push and pull that allow us to like find that middle ground of like these business models that actually benefit both parties and so I think although yes there's going to be like misalignments or disagreements like those are the hard conversations that we kind of keep referencing that bring us to that middle ground that often don't happen between the public and private sector. Yeah, I was going to add to that I feel like with this pilot we are really being specific about the use cases that we're pursuing and that we're allowing to meet our shared goals and part of doing this to add on to Erin's point is that we're trying to learn as much as we can for this pilot to prepare for what's going to happen later on when we don't always necessarily have these really great collaborations in place like I want to know as much as I can about how the robots are delivering things how they're operating on the sidewalks and everything while I can work openly with David in preparation for knowing what's next and I think having too many robots on a sidewalk and causing congestion is actually a really big concern of ours and we're trying to learn what we can to prepare for the future to avoid that sometime in the future. Teresa anything to add to that before? Yes, we've been the same we have they have I'll say it already the cities but it's we're preparing so we need to figure out and learn what's going to happen if we're going to this robot at home so what's going to happen when we have a high volume what we need to have in place while we need to request the companies and that way both work together not only the agency or you know the government is need to adjust and prepare but have to prepare this is going to allow us to learn a little bit more about if we need to dedicate right away for this kind of devices if we need to also regulate the amount of devices that can be on the sidewalk at a given time of day if we need to regulate the time that they're operating if we need to put in place other you know different measures to to have a safe environment and because it's going to be at a moment that's probably going to have a lot of interactions between pedestrians devices and all users of the sidewalk so we need to this is going to help us to learn a little bit more of what we need to do for what is coming and what we need to ask the the companies and what we can do we cannot do and what they can do to just work together and get ready so we don't need to wait until they are here to then react we want to learn to prepare and now we're talking about delivery robots tomorrow we're going to be talking about something else different but it's going to be in the same category so we need to prepare and this is helping helping us to just establish the basis of what we might be doing or what we might be requiring the companies in and just this is just the beginning of this we will need to have more research more study more pilots but this is just the beginning of learning that's great Doris yeah and I think that emphasis on what I think we've talked a lot about about cities taking more of this proactive role rather than you know the sort of more reactive role that maybe cities have sometimes had in the past I know we're getting short on time but I think Andrea just put a great question here in the chat that I think would be a good note to sort of end on as we think about you know and move into what's next for this pilot and this work in general and so the question is about how much under the radar learning is happening among the city is how have the city staff heard from colleagues in other places wondering how to engage in similar partnerships lessons learned how the staff effort like efforts like this and maybe are you also hearing from other folks within the city yourself I know you know we talk with you all directly every week but I have no idea who else you might be talking to at the city who is who is interested so I think we could just do a round robin on that one and then we'll pass it back over for a wrap up I think guess the under you know under the radar learning is happening individually I talk with a lot of other cities about you know delivery devices but also about ab testing and deployments and Sarah and I spoke with a couple other cities in the past few months as well who have similar pilots launching and it's a conversation like we have as part of this like cohort of cities under this grant is that you know the more kind of the more we share our learnings and get on some more of the like a united front is really powerful for cities if we're all asking maybe the private sector for the same type of information or asking them the same questions when they come into our city and share the right questions to ask or share our learnings that only makes us gives us kind of firmer firmer footing as we move forward in our relationships with these um with the companies I think there's often like a lack of um you know education maybe on the public side because we're being a little bit reactive historically and so um it seems like a power imbalance and if we can balance out that that power between public and private um I think it will you know lead to to better outcomes yeah I actually don't have much to add I feel like there is so much learning that's going on sort of under the radar but we're also so in the midst of it that it's going to be something that I think I'm going to have to like summarize when when we're at the end but but I agree with all the points that Erin just made about that yeah even anything for Q about like working so we're also working with the city of LA and we are working with Berkeley and I do believe that every time a new concern especially concerns especially difficulties our eyes it creates changes in all the cities we operate on so I would say that the transfer of knowledge is absolute that now we come to a new city and cities are impressed by how much do we know just because we have these quick iteration on experiences that come from different cities on the night pilot when San Jose makes comments about the dashboard on clarity those are things uh Miami praises on the next meeting because they've seen changes so I do believe that the accumulation of experiences is definitely important one of the things that that's about so just wanted to chime in with Teresa any last thoughts on that I think I would be repeating the same but it's what we have been doing is we talk weekly and we have the core meetings also very frequent so it has been helping us to to kind of brainstorming and restoring and have been great just to hear about what they're doing because we're applying to our project and pilot and vice versa so we have been learning from each other and also whatever one of the cities know or got to know from another place in the country they show it off so and it's like expanding or resources wonderful all right well we're about five to the hour and so I think we're the last part of our agenda and we're going to pass it back over to Gabe crime to give you some thoughts on looking forward for the night AV initiative and our next steps great thanks Cameron thanks everybody on the panel thanks to all the people that are joining us um and asking questions or just hanging out and listening so you know I I wanted to try to summarize like why this work is so important at this junction in time um and I think that as we mentioned earlier we've gone through ups and downs as a society with this program you know through covid and um it's important to stay the course you know to stay consistent I think um through a downturn that's when you can have some of the greatest learnings um and it gives you an opportunity to learn in a in a different way so I think staying consistent you know giving cities and communities the ability to to have input and learn during this time as well as Kiwi bot and other companies and then preparing for the uptimes um you know I think one of the most important things we did and listen to this conversation just reinforces is we're really engaging through discussions with communities around company driven pilots um and then we move to engaging through city designed and driven pilots and I think that's really important and I also want to reinforce that we're really here to learn about pain points and opportunities we're not here to prove that AVs work like that's you know was said earlier but I just want to reinforce that that's not like our goal is to promote a particular technology or um even particular use cases um but we do know that artificial intelligence machine learning they're really in their infancy and so we need to create these spaces to study the impacts in the real world versus just theoretically on paper then these discussions that we have like this are much more more meaningful and then to sort of take it up to maybe 40 000 feet um you know we have a number of crises in our country and globally right Cameron something that we talk about uh we have inequities um you know that have really really come to light um you know in a way that they can't be ignored around race gender as well as socio economic status education we know that this leads to economic inequality which leads to a housing crisis which leads to people in certain geographies and zip codes having better access and outcomes and education and so forth than others um we also have an environmental crisis and you know it's now manifesting in a way that people are now experiencing on a daily basis like you can't ignore this anymore so in my mind as we as we flesh out the outcomes that society you know needs to see from any new potentially game changing tools or technology I would call it um those issues need to be at the forefront of the work that we do whether it's KiwiBot or you know I know we have folks on from Nuro and Uber and other great companies um or for government like we need to come together at this point in time um and figure out we have this game changing technology what's the game changing societal outcome that we can get from this otherwise is it really worth doing it to Yasha's point like if you're I looked at that tweet if you're spending a dollar to deliver a 90 cent white claw or whatever it was you know is that really worthwhile um and so with automated and I would say automated and connected vehicle technology which we have not focused on as much but we do have the potential to dramatically increase safety to lower injuries and fatalities particularly in neighborhoods of color that have been you know bisected by highways to increase job access to increase linkages to public transit and thereby by the way when you can increase the link to public transit you can extend the value of transit miles and miles into communities that traditionally have been transit deserts but all of that choice that I just laid out is all in human hands and you know so these are choices right and it doesn't always turn out that way and so that's why learning sharing collaborating amongst public private philanthropy non-profit and end users citizens and community activists for that matter are so important and I think it was interesting for those of you that heard Alberto and Felipe's conversation earlier he really appropriately pressed Alberto on Felipe on the plan for community benefit and what came out of that for me is that communities need this type of forum this experience is cross-pollination of ideas and understanding or as Cameron or Manna said middle ground to see how they can provide positive societal outcomes but positive societal outcomes that may also give them a competitive a competitive advantage or define a market segment or fundamentally change their business model and I've sat with David and watched his business model change in real time as we've had these conversations with states so in closing you know someone in the chat during the equity conversation said this is a once in a lifetime opportunity and sometimes I think there's so much noise and so many differing opinions that people decide you know what let's not seize this opportunity whether they're on the public or the private side and for good reason sometimes and let sometimes they just let the answers be defined by one sector and if I can be honest I think just letting government define or just letting the private sector define it is not the best outcome typically and so you know thanks to the Knight Foundation and their comfort creating a place for engagement in an undefined space to have uncomfortable conversations as Niko said to work with communities to let them define how technology is rolled out we are exploring all this and thanks so much to the cities it's not easy and to Kiwi Bot and everybody at night and in closing you know as many great people have said nothing easy is worth doing and this is not easy but it's really interesting work and I think we will look back on it and be thankful and hopefully we get to continue this work into 2022 so on behalf of CityFi and University of Oregon thanks everybody for coming today