 Hi, my name is Suzanne Romain. I'm here to present some of the work done by the FMA Innovation Project. We are working under a grant from NOAA. We work with National Marine Fisheries Service with Fisheries Management Collection data, and we work very closely with the University of Washington Department of Electrical and Computer Engineering. Our research cycle involves first acquiring the imagery data. Our team has built and deployed custom camera systems to collect this data, and we also use opportunistic videos from surveillance cameras. We then get that data and we annotate it based on the needs of our desired algorithms. The University of Washington develops the algorithms and together we analyze the results and determine whether or not they are useful for fisheries managers. So the Pacific Observer Program is a great program for us to get the chance to work with. They cover the Bering Sea, Aleutian Island, and Gulf of Alaska fisheries. They train over 450 field scientists and then manage that data collection. There are a number of users that use it. Of course, first and foremost, fisheries managers for quota management, but fishermen watch it pretty closely, too, because they're interested in staying in quota and avoiding bycatch species that they don't want to see. There are over 54 stocks managed in the region, and those stock assessors also use the Observer Program data as part of their stock assessment, as well as for ecosystem research. The environment is insanely challenging. There are so many different target fisheries and gear types, and every vessel is different. They range in size from 30 feet to 350 feet, and for the machine vision perspective, this is pretty interesting because we have a lot of different types of images and very specific data points that we can pull from types of images. These are some examples of the types of imagery that we work with the most. On the left are the fixed gear or long line imagery. The fisher coming out of the water one at a time, silhouetted against the back of the boat. They're very complex. There's a lot of changes with the water conditions and the lighting conditions. It was an algorithm that took us quite a bit of time to work up. On the right-hand side, we have shoot images, and you can see there's some benefits for us. There's a solid non-changing background and solid lighting, and it gave us the opportunity to get to our species ID algorithms a little faster, although both camera approaches came with some issues with the environment. Obviously, the images on the left were much more complex, and then the images on the right, the camera was so much closer to the subject that it was difficult to keep it clean. This side is a summary of the majority of algorithms that we've been working on. I'm going to focus just on the shoot type of images, fish species identification algorithm that we have been working on. Our shoot track has been developed to discern species, count and size. The size algorithms we have in a real-time device that can size halibut as they're slid off the boat. A counting option we are now using for multiple fish being dumped into a shoot in a basket of fish. We can detect the individual fish, separate them, and then identify them to species and count from those dumps. They do require calibrated images, and the results of those calibrated images are object detection algorithms, tracking and segmentation and classification and length algorithms that get put together for one product device. We did build custom camera units, as mentioned. Along the bottom, we've got them deployed out on various research and fishing vessels, and we used video recording, security cameras for some of them, capitalizing on the motion detect. They turned out to be very good in terms of flexibility around lighting as well. We also used machine vision cameras with a trigger, and that enabled us to test different frequency of light response. FYI turns out red-green-blue does the job. No need for fancy cameras and different light frequency response. The calibration procedures involve using a board like seen at the top right of this slide. In that image, it's a calibration board. What this does is rectify the effects of each individual camera setup. So the intrinsic calibration accounts for the effects of whatever lens choice you have in relation to the camera sensor, and what happens is the pixels can become distorted as you move from center based on your lens choice. They have a wide angle lens. The edges of those images will look different than the center of those images. The intrinsic calibration is the relationship to the camera and its environment. So this allows one to place the camera in the scene, and if you look to the right, you'll see these two original images, the calibration board and the fish. And then the bottom one is a rectified image that is what we feed to the algorithm for training. In these shoots, the camera is set up at one side of the shoot, so the image is pretty distorted when it comes right out of our cameras. After we get our calibration images, we label them, and depending on the algorithmic needs, sometimes the annotators need to actually put a box around the fish with the shoots. One of the benefits of the enclosed control environment is that we can generally detect the fish as an object because it's so different from the background. And with the shoot identification projects, we've also done some genetic sampling for closely appearing species that we want to make sure the identification is right before we send it into labeling. Both of the software programs we use for this are open source. Calibration routines can be found at opencv.org, and the annotation is called application we use is called label image. And that GitHub link is posted up here. We also, our lab developed a custom version of label image that allows for tracking. So if you have images of fish in motion, please contact us. We're happy to share those annotation tools. The classifier is a two level hierarchy, and this allows us to expand our actual species list. We rare species, it takes a while to get high confidence return in terms of identifying them accurately. And this allows us to move up a level and possibly respond to areas where we know they're sensitive species, but we know we can't rely on the algorithm to find them or get them to species every time. This is an overview of our classification process. And it's great. Moving into this work, we realized that we were one of the few groups able to start early on this, and we wanted to ensure that the algorithms we built could be moved from region to region and applied to images that they could be applied to. To that end, we incorporated an active learning strategy. So after training our CNN algorithm that takes a long time to train, we run a second algorithm that allows for auditing on the back end. Basically, we run our images through the classifier and the algorithm itself, the active learning strategy, selects images that the algorithm has low confidence on and sends them back out to us and we can then label them. This is our initial results from the support approach and it can be found in this document here cited at the bottom of the slide in detail. And the number of species that we have very high accuracy at recall has increased through the years as we've had more training data. And as we pursue algorithms that address our long tail distribution, pretty much any of us working in this industry are going to have an uneven training distribution to start with, because it takes a while to collect these images of these rare species. And the ACE complementary is our first really successful attempt to get at these difficult to identify rare species. Our final push on our identification algorithm was to see how well does it do for species that humans have a hard time separating. And along the top of this slide, we have rockfish species that field staff have had a very difficult time separating accurately in the field. We collected genetics information on these species during a research cruise and then ran them through our algorithm. And sadly, the algorithm will actually very excitedly the algorithm outperforms the field staff. So we're pushing it even further and starting to introduce non controlled imagery to the algorithm to see how well it can respond to observers in the field using any camera they have to take an image of a species like this that is difficult to identify. And we shall see that is an ongoing experiment. And quickly, two different types of image algorithms I just wanted to show you a quick picture of and some sources. The human presence detector, we developed this to use as a trigger on our fixed gear vessels because we found that the data collection is a lot of data. In any way that we could focus on just the areas of time that we're interested in collecting imagery is a good thing. And this is a region based algorithm. So if you have a human activity on deck that can be translated into a data point of Hall starting Hall ending. These algorithms can be off the shelf and deployed quickly to any kind of image that uncalibrated image set that you might have for this. The second one is our detecting of a species on conveyor belts. We're using it with salmon in this instance. If you're working with processing plants, sorry, I can get these started. If you're working with processing plants and you have an individual species that you're really interested in keeping tight data collections on this could be an option for supporting the staff at least in Alaska. That is our idea is to give observers another backup set of eyes because they have a number of other duties to perform and the plants are huge. So they have samples to collect and places to be and they can't be at every belt at once. And lastly, I have just a page of references to follow up on this. We've put a lot out as a group and our directives early on was to develop algorithms that would be open source and released when we could get them worthy of other groups. And I can hope that we built them in such a way that they are expandable to other regions. The one thing that has benefited the most in this particular project is all the collaborators that we have been able to work with. It would not have been possible without these collaborators, the commercial vessels and the processing plants that were willing to let us collect images on their vessels and then utilize those images to develop these algorithms. As well as the halibut commission and the NOAA research cruises, as well as the environmental defense fund who has kept their eyes on these kinds of projects and kept the documentation flowing and the connections flowing between researchers and stakeholders in this industry. Please do reach out to our lab if you see anything that I've presented that you think might apply to images in your region. As mentioned, we spent quite a bit of time developing these algorithms to ensure that they would be expandable to include new classes. Thank you very much.