 Thank you. Thank you very much everyone. It's pretty cool to be here. I've only ever done this talk online because of the last two years so it's actually good to see faces for the first time. So yeah, I'm here today to talk to you about the Mayflower Autonomous Ship. It's a really cool project that's been going on for about two years now, oh actually it's been going on for way longer but two years just for me, and it's a collaboration between IBM and an organisation called Promare. So just get out of the way, I'm just here because I think it's really cool, I think people might want to know about it, so all of my opinions are my own etc etc. So just quickly about me, so maker, IoT enthusiast, I worked a lot on the Eclipse projects for the MQTT PAHO clients. I work for IBM Research so there's this amazing house down near Winchester in Southampton called IBM Hursley. It's one of the biggest software labs in the UK for IBM and it's basically where our IBM research team works to solve problems for people where there aren't solutions yet, so typically people will come to us going, I've got this problem, everyone says it can't be done, can't find an off the shelf solution for it, and we go, ah we'll do it, and we usually figure out a way somehow, and that's usually by reaching into some of the really cool work that our colleagues in research have done and bringing it into the real world. As part of that I work at Wimbledon, IBM have got a 30 year plus partnership with the tournament there so we tend to do all of the statistics and stats capture, getting it out to the website, your TV graphics and all that kind of cool stuff, and then finally the Mayflower Autonomous Ship, which has pretty much taken up most of my time for the last two years. So what is this thing? It's a fully autonomous, cruel-less, full-sized ship capable of independently traversing oceans, which is a really, really important thing. The background from where this came about was about five, six years ago, Plymouth Town Council were looking for a way to commemorate the 400th anniversary of the original Mayflower making its way from Plymouth in the UK to Plymouth in the United States with the pilgrims, and they were thinking oh you know maybe we'll just build a replica or something, and there's this guy in the room called Brett who owns a company that makes submarines, which it's just such a cool thing, and he went no no no no no, we don't do that. We're going to make a ship that's looking forward to the next 400 years. We're going to look at the future of marine technology, its autonomy, and you know really try and do some good in the world, some you know ocean science and things like that as well. So it's kind of broken into these two parts. You've got the AI captain, which has been done by a company called Marine AI in IBM Automation, and that's sort of making a ship that can you know completely with a human on the loop, but not in the loop, get from one side of the ocean to the other, and then the other part of that is the science payload. So we've got to have a reason for crossing oceans, and in this case it's to do really important ocean research. So we've got loads of experiments and sensors packed on board, and being able to do all of that science with that and humans in the loop is a really important thing too. So you can see here is the, this is the Mayflower, this was when it was off on its first voyage. It's a 15 meter long trim around about six meters wide. It can travel about up to six or seven knots at full speed, and it's, I think it looks great. The designer is a bit of a fan of Star Trek, so if you get the spaceship feel that's where that's from. It's aluminium, so really lightweight, and it's got a really cool hybrid electric drive system, so it can use its solar panels and its batteries during the day, but then it's got a biodiesel generator that it uses when it's in rough waters and at night time and things like that. And for navigation it's got AIS, radar, GNSS, and it's got six cameras mounted on its mast to give it a really good view of the world around it so that it can pick out obstacles in its way, identify them and then move around them. This is all done with the air captain, which I'll get into in a little bit, but I'll just give you an idea of what we've actually done so far. So this isn't actually our first time across Atlantic. It's out there now in the sea. You can see on the map it's sort of on its way up to Halifax. This is sort of version 2.5 of the journey. Originally we were going to be going from Plymouth in the UK to Plymouth in the US. Slightly changed to Washington because we had some events that we wanted to get it down there for, and then unfortunately we had a few technical glitches along the way where we had to take a little diversion to the Azores. I think the team just wanted a holiday for a couple of days, but we'll never know. So they had to go down and change something in the charging circuit and then it's on its way up and then we've diverted to Halifax because just for a number of reasons it's been quite a long way. We want to get it to land quite soon. There's an experiment that we'll really appreciate being in that particular part of the ocean, which I'll get to later. And there's some really rough seas coming up and we've got rough winds here, but it's getting pretty rough out in the Atlantic for the Mayflower as well. So that's where I was this morning when I had these slides prepared and then I get this WhatsApp from a colleague. Why does something have to keep happening? It turns out there was a ship on its way to meet it. This is the Endeavour, which is an American research vessel. It was going out to meet it and then suddenly we're just watching on the webcams and suddenly they come right up to it, they hook on and they start escorting it. As it turns out, this was completely intentional. Basically it's gotten so far into the journey where the sea state is just too rough for it and they were there anyway so they thought, well, why not? Let's just escort it all the way back to port. So that's where it is at the moment. If you've been following, for everyone else who's not been following, head to mas400.com and you'll get all of the details there. So I speak about this AI captain. I'll go over that really briefly. Having a way of autonomously navigating the oceans is really important. At the moment you've got to have a really highly trained crew who can do that and if you watched the talk last night about the oil rig, you'll know that having a really good sort of autonomous system is really important, especially when it's turned on or off. For this, we really wanted to prove out all of the different steps technology to make something that was really, really safe and reliable. So the AI captain is built up of a number of different steps. So we've got our sort of data fusion layer which takes the camera footage. So we're using a machine learning model that's been trained on millions of images of pretty much anything you could find at sea and stuff you wouldn't expect as well. So pictures of tankers, oil rigs, not that we should find any out there. Flotsam and Jetsam. It's quite a lot of rubbish out in the ocean and we don't want to bump into that as well as paddle borders because around Plymouth where we've been training it, people just like to sort of float out into the Plymouth Sound and get in the way of us. So we've got all of that image data which we can analyse. We've got AIS so we know what ships are reporting to be around us which is really important. And then we've got radar as well and we need radar and camera footage as well because not every ship follows the rules and broadcasts their location, especially fishermen. They like to hide where they are when they're getting their fish so we've got to rely on radar and cameras as well to know exactly what's out there. So once we've fused all of that data together we also take a feed from the weather company so we know what the sea state's likely to be like in the future. And we then put that into sort of the main part of the air captain which is basically a really, really sophisticated rules engine. It's a piece of IBM technology called ODM and it usually is being used to decide whether someone can afford a mortgage or something and they flipped it on there on its head and they taught it what's called the Colregs which is basically the rules of the road for the sea. So when you learn to drive a car you know exactly how you should be driving. There's the same set of rules for if you are at sea saying you know how do you pass other vessels at sea? What happens if you're heading towards each other? You know do you go to starboard or to port? Whose responsibility is that? It depends on the type of vessel as well. So we taught it all of those rules really about how to behave when it sees other vessels. It's then got a second system which has been trained on something called SOLAS which is Safety of Life at Sea and that's another set of rules which sort of it's kind of like what a really wise captain would do and it allows you to sort of have a bit more autonomy around the gray areas of you know what happens when you're expecting to do something and the other vessel should go to starboard but doesn't. How do you respond to that? How do you do it in a safe way? It's a safety of life at sea. That's the main priority. We're an autonomous vessel so our priority is to basically stay away from any other ships as much as we can. Once it's been through that whole system it's been optimised. We've worked out exactly where we are and where we're going. The outcome of that is a heading and a speed and that's basically how the ship operates. It's just continually going like that. So that's the AI captain but what are we going to do with the ship? So I mentioned we're doing lots of science. If you look at the cutaway of this ship there's actually a really small bit in the middle you can just about see. It's about three metre long by one metre by one metre void and that's our science payload bay. Now if you were to look at a real big research vessel like the RSS David Attenborough or the James Cook they're massive. They're hugely expensive you know in the hundreds of millions they're crewed by a hugely capable crew. You've got all those scientists on board as well doing huge amounts of work and that's really important at sea but when you've got smaller experiments that have to go out into sort of difficult spots of sea sorry or spend maybe really really long times out there you don't want to force a crew or a load of scientists to go out at sea for a long period of time. Sea is really dangerous unless you really need to be out in the Atlantic you really don't want to go there and so reducing the risk to human life and also improving the comfort to those humans is a really really important you know sort of benefit as well as it comes down to cost you know these big research vessels that we have are massive they cost a huge amount of money to run and you really want to prioritise them for the science they they're best suited for rather than sending them out to sort of very remote areas and wasting large amounts of money fueling them up and sending them out for a long period of time. So if you've got a small ship that's autonomous and you can automate the science on board that's perfect you just send out to where it needs to go it goes along captures the data comes back you can have a small fleet of these they could go out to all kinds of places they could all link together and track data that way so that's kind of where we're going with this that's the whole ethos of why we're trying to condense all the science down and automate it. So I'll give you a quick summary now of all of the different experiments that we've got on board and these are being done usually with IBM and then a number of partner universities around the UK on board so for instance here we go we've got a whale population experiment so tracking whales is notoriously quite difficult they're never where you expect them to be and you know tracking them down can be quite difficult so we wanted in a way sort of almost passively monitoring them you know if you could put a system on a ship and that ship just happened to be passing you know in a particular area and you were able to hear those whales you'd be able to say I've heard some right whales and they're over here but how do you know what kind of you know whale they are actually or what kind of marine mammal are they or even what kind of marine creature making noise are they so this is where a lot of the work we've been doing is it's taking a hydrophone mounted on the underside of the ship and we've trained it with hundreds and hundreds of hours of already recorded whale song and dolphin noises and all kinds of other things and we're building up a massive machine learning model that can autonomously basically go ah yep that's the right whale I heard that and we can clip out that audio and send it off to the research team in Plymouth and a number of other places like the Jupiter Foundation so this has been a really interesting one because putting a hydrophone on the underside of a ship doesn't work particularly well especially when the ship is blasting along at seven knots because you tend to hear basically the sound of you getting dragged underwater so we've been doing a huge amount of work on filtering out that ambient noise of the the water being dragged along the underside of the ship and we think we're making pretty good progress there we're hoping we should have about a terabyte's worth of data by the end of our current voyage from the UK to to the Americas of the moment which should be available in the next couple of weeks hopefully and actually this is from our voyage last year so hopefully you can just see some dolphins just sort of on the port side of the ship just coming through the water there I don't know how clear that is for you but this was this was really a nice thing to see because they sort of attracted to ships generally and so we were just out there or it was on that out there on its own and they're just sort of they came come up aside and follow us along for a while and we saw this happened again on our voyage this time round too which is really nice so hopefully we'll get some some good footage of those as well and then also the audio of them if we can get the audio of them making their vocalisations underwater as well as seeing them above water I think that'll be be quite special so another experiment this is ocean chemistry and this is a really cool one because it's a a really nice reuse of technology so IBM Research in Zero came up with this thing called HyperTaste it's a digital tongue so you can train it to detect the chemical sort of fingerprint of liquids so typically being used to identify you know is this a real Scotch whiskey or you know is maybe a faked perfume or something and with the Helper Plymouth University they flipped it on its head a little bit and now have built this really awesome pumping mechanism that's on the Mayflower Township itself and that is basically tasting the seawater every 15 minutes and it's looking for the concentrations of dissolved carbonates in the seawater now they help us understand the rate of ocean acidification in different parts of the ocean so we're hoping that as we've passed along and we've actually done sort of a nice sort of up and down and then back up motion across the Atlantic we might get some interesting differences in the data to track how and where and what rate the ocean is sort of getting acidified which is all related to to climate change this was really difficult one because typically you can see the the actual sensor itself is you know in that picture there it's sitting on a nice glass of water so building this into a system where you can automatically you know suck water out of the ocean put all of the different testing fluids through and the cleaning fluids through require quite a lot of work it was basically one student's phd to sit there and build this entire pumping mechanism and build all of the complex steps so that if it got itself into a confusing state it could sort itself back out and just carry on on its own all of that data is then passed to another machine learning model which will hopefully give us all the data we've got our open ocean wave energy experiment so this is really cool so this is looking at if you have video footage of waves coming at you so in this case we're you know waves coming at the ship and you've got a really good inertial measurement unit in the ship it's like an accelerometer on your phone so it knows how the ship's behaving in the water can you match up the wave crashing into the bow of the ship with then the ship lifting up and going over it because if you can do that you know the weight size and dynamics of the ship itself you know how much energy was in that wave that just crashed into you which is really important to say looking at you know predicting the amount of damage that coastal infrastructure might take if you're looking at you know how fast is this cliff going to degrade or maybe this oil rig how much damage that's going to take or even maybe if you're looking to put a tidal energy harvesting system somewhere you know working out where's the best spot along a coastline to do that so this is all data collection at the moment we're just collecting that data and then team in Liverpool are going to be looking through this data and hopefully starting to build that machine learning model for future voyages and then we've got our open ocean tide and sea level and this is something that I never considered until I heard it so obviously it's really easy to measure the tides around the UK or any coastline you have a big stick you put it in the water and you see the water go up and down you can measure how high the tide is and we've got really good models on the coast to know exactly you know at x o'clock it's going to be three meter high tide in the open ocean it's a little bit different the the moon the gravity from the moon will suck the water up on the ocean so it is much higher in places and much lower in other places but there isn't a very easy way to measure that and a lot of the gnss navigation systems because you're rocking around on the ocean quite a lot don't give you a very accurate reading of meters above theoretical sea level with the the model that the earth is that we have so this is using uh actually it's some off the shelf gnss receivers that you can get if you want to do a real-time kinematics these things are great plugged into our ars reply and they're sitting there and they're measuring the really accurate readings between each other and then they will be compared so they're actually mounted on the uh sort of near the uh front of the the mayflower and then sort of off to the side as well so we've got a nice triangle pattern and they'll be used to track how the ship is in the water and then also the height of it as well um and this is just you know really useful data to be collecting about our earth you know i'm sure most of us have all heard the phrase of you know we know more about space around the earth than we do the oceans and this is you know it's one of those things we're trying to understand so those are just some of the experiments we've got on the on the ship at the moment we've also got a host of other sort of what we call wet sensors so things like a a CTD fluorometer uh oxygen sensors temperature sensors and they're all mounted on the underside of the ship measuring the water uh we've got a depth sensor as well um we didn't quite read the documentation on that so that worked that's been working absolutely perfectly but we panicked because the data feed we get was going great great great we're getting all these numbers we're going oh brilliant we know how low the surface of the uh the sea floor is underneath the mayflower and then it just went to zero and we went hmm that's a bit odd and after a while we realised it's because the mayflower had gone over the continental shelf and uh we'd reached our maximum rating on the depth sensor the depth sensor was working absolutely fine uh but we just we just didn't realise that zero meant in or too far not too low uh so yeah that's a bit of fun um so how are we doing all of this uh some of these you know as you can see it's a raspberry pi in a box um or it's a hydrophone uh but a lot of it's using you know we're trying to use machine learning for some of these experiments we're trying to store all of the data we're trying to monitor it we're trying to optimise when it runs as well so we had to look around building uh what we call sort of our science at the edge box or our science pod um which we started basically in June 2020 uh in the middle of a pandemic so uh we didn't have much to go on we had to sort of scrounge around what we could uh ordering stuff offline off the internet trying to get things together um and we basically ended up with this which is a suitcase full of raspberry pies um which i'm actually quite proud of even though it does look very messy on the inside so we've got four raspberry pi four bs uh we've got a little small mini ups uh a little switch uh about 12 terabytes worth of ssd storage uh and then we've got our our hydrophone receiver and an rimu in there and stuff and that's basically all mounted inside a peli case uh and it sits inside the main flower actually you know it's really tiny takes up about that much space which we had to make sure of because some of the other experiments like hypothesis are much much bigger um the the brief we were given was basically it's going to get knocked around we're probably going to stand on it it might get wet so try and make something that's going to survive all that and that's kind of where we ended up with with the sort of you know bunch of wires and stuff in a peli case um you can see i don't know whether that's too too clear on there but um you can see a sort of rough system diagram of how we how we built everything uh the the rough idea of this was that uh we were told you know if there was something would go wrong which you know in cases it did they would cut power to us the ship had to make sure that all times the ai captain uh was the thing in charge and if they were going oh we've got to you know start shutting systems off we were probably one of the first to go and that's fair enough you've got to you've got to maintain the safety of the ship so we had to make sure that when we got that hey you're being turned off signal we could safely shut down all of the experiments close all of the files make sure all that was saved away nicely and then sit there and safely shut the whole system down but sometimes we just wouldn't get that at all so that's why we have this timely for UPS there it's not to keep us running for hours and hours it gives us about 20 minutes to shut the whole pod down and put ourselves into a nice safe state we learned a lot about raspberry pies during this including how to build watchdogs to watch the raspberry pies which are watching each other to make sure that if you get yourself into a bit of a weird power state it all still carries on and you know you can you can start the whole thing up again but the great thing about this was that you know we get these data feeds we've got most of the compute work is happening in that box we're doing all of the machine learning running all of our models saving all of the data and we get a very very small bit of satellite bandwidth to send back the status of how all of this stuff is going so what you're looking at here is just a an update on you know the status of the experiments it's the temperature on inside the pod it's a little bit warm it's fine the CPUs how they're going and you can see there in the top left some of our experiments go from on to off quite a lot and it was something we realised halfway through we're recording loads of camera footage and then we realised well when it's dark you can't see very much at all you can see a perfect picture of the ship because it shows up lovely in the infrared but you can't see any of the waves you can't see anything about it so you know looking at turning off some experiments when you don't need them really important save data looking at where you are as well we're travelling across the Atlantic which means we're slowly passing through all those time zones so we're using the location of the ship to work out you know where are we when does the sun rise when does it set let's use that as a basis for all of this this kind of stuff um it's all built pretty much on open source um there's some you know the experiments themselves will be running on code written by the universities and IBM but everything supporting this box is basically node red got python grafana influx db running on raspberry pies um and we've you know a lot of the stuff we've done we're sort of looking at open sourcing and giving back you know in some of the node red nodes that we've had to build for this um and you know it's one of those great things where you can whilst you're sitting on a very cold dock in Plymouth on a February day and they go hey we're going to change where this sensor is you can just drag a node across the screen and boom it's gone you've not tried node red it's fantastic give it a go um so I'll sort of finish off I've got about eight minutes left to talk about data feed so as I mentioned we've got all of these open data feeds the ship is um out there in the Atlantic it's got statcom we're feeding that back to an mqtt server which is open for anybody to access it's at mqtt.mas400.com and on that you can get all kinds of stuff you can see the status of the ship itself so where it is how fast it's going what direction it's heading in how some of the solar panels are performing the batteries and all that kind of stuff you can look at the science experiments see what's running when it's running how it's running uh see the health of health of all the sort of the experiments and the the pod itself as well um and it's as I said it's open for anybody to access to so there's a documentation page that you can go to uh where we've listed exactly how all of this stuff works um and we've you know quite a few people have started building stuff over the past couple of months so we've got some you know really great models so if you look on the top it's a small mayflower on a uh turntable that will always point in the direction that the actual ship's going and it lights up in in that respect we've got a Twitter bot which is reporting uh where where it is and it's progressed from the UK to the US people have built maps for us and all kinds of things um and you can you know it you could probably get from nothing to something in about 20 minutes we should have probably done a workshop on this but we didn't didn't quite have time um so overall I'd say this has been about two years worth of work where are we going to go next the ship's going to get to the US in the next couple of days probably uh it's going to get to Canada because Halifax um whilst we're going through we're going to be listening to all of the whales uh hopefully because lots of right whales up there and then we'll be getting to Halifax and we'll be offloading all of the uh the rich data feeds as well so we've got all those lights of light data feeds uh the rich data feeds will be coming soon after so that'll be all of the video all of the sensor data from the um uh wet sensors on the underside of the ship that will all be open we're going to make sure that everybody can get access to that so if you want to look at you know plotting the temperature of the seawater across the Atlantic Ocean you can do that if you want to look at the oxygen concentration you can do that and then once all of the uh research teams have found their results and things like that they will all be put open as well um you can also uh go to mas 400.com which I think I mentioned and watch all of the live webcam so you can just literally see what's happening with the ship see how it's getting on uh and what it's doing and uh currently there's a sort of another ship hanging in front of it ominously as they slowly get towards Halifax so looking a bit further into the future it's going to spend some time uh probably going up and down the US coast uh we'll still be capturing data as we're there um and we'll be popping into quite a few ports along the way to let people go and see it so if you happen to be on the east coast of the US in the next 12 months um you might even be able to you know go and see the ship itself which will be quite cool and then looking even for a further future um we're looking at you know there's going to be different versions of these boats potentially in the ocean doing different things so maybe there will be a specific boat that will be doing a particular job maybe doing replacing a role that we you know we currently have ships that will go out to certain uh science boys every single week or month to corroborate the data that those science boys are collecting we can automate that and do in a ship um what I'd love to see is having a fleet of ships uh I think having a whole fleet of these uh ships going out and coordinating amongst each other um to capture data especially if you're trying to do triangulation say if you're trying to look for whales and things like that you've got one audio source that's not quite enough you know maybe you can do it with uh with more um so that's that's probably sort of uh the where the the the the near future of this is going to be um but generally we're trying to be as open as possible about it I think the the last word that I got which was literally on whatsapp about half an hour ago uh was uh because of the weather out there it's it's on its route to Halifax and yeah they've it's been fantastic to give you an idea of how this has been running uh we have a team who are working down in Plymouth who have been watching all of the same data feeds that you or I do so uh if you go you know look at the webcams you can see all of that stuff and they're sitting there watching it maybe with a bit more data and they can see the ship on its way up there but they've had someone sat there for 24 hours a day for the last month or so uh since about 27th of April just keeping an eye on it and making sure that everything's okay um which has been you know quite a tough job for them and they've they've been doing you know even over weekends and bank holidays and stuff like that um so I think they're all looking forward to having it a good rest which is definitely well deserved um so I think I'm actually a little bit early but I think is that okay that's fine um thank you everyone for coming I've this is a lot of people so uh I'm really happy that everyone was was interested in hearing about this I'm around for the rest of the weekend so if you want to ask questions you want to do something uh come and ask me um if you want to know exactly where the where the ship is in its progress and you've got a decked phone you can find out by calling 555 4014 um uh so you know why not um but yeah otherwise thank you very much