 Next up here, we're going to be hearing from Adrian Meyer, so making sure we're all ready to go on that. Excellent. So a little bit about Adrian Meyer here. He is a data scientist for remote sensing and machine learning at the Institute Geomatics. I can't even pronounce that part. I'm not going to lie. I don't speak that language. Currently working on several big projects, mainly focusing on automated image analysis, telemetry, big data processing, deep learning, digital reconstruction. Holy moly, this guy does a lot. And he's interested in the interest of the animal biologist by training matured during analysis of geotag wildlife imagery in Cape Town, South Africa. Adrian, you've got some serious nerd shops there. So thanks for joining us. Where are you joining us from? Thanks for the awesome introduction. I'm joining today from Basel, actually from the little suburb of Basel called Mutens, which is where our more or less unpronounceable school is situated. How do you pronounce that? The Fachhochschule Nord-West-Schweiz. Bless you. Now, see, you can do that. That's among your various skills you can pronounce, but that is awesome. I must learn your ways. So I will turn this over to you. And it looks like you're going to be talking about something that is completely blocked on my screen here. Oh, come on, Zoom, get out of the way. Just analyzing solar panels in Switzerland using aerial imagery. So awesome. So I'm going to turn this over to you and go for it. OK, perfect. I'm going to try to do the screen share thing here. That's OK. Well, thank you, everyone, for being part here, for being interested, for wanting to listen to this talk. It's going to be about a specific application case of detecting and analyzing solar panels. This is a type of big data problem that we have been working on since about two years now at the Institute of Geometrics of the University of Applied Sciences in Northwestern Switzerland, which would be the appropriate translation for the German term. I'm not working alone on this. We actually have a couple of student projects going on here. And my two professors, Dunia Jordan and Martin Christen, who many of you probably know, they also join in this project. And we perform it together with the government of Switzerland. Most precisely, the Federal Office for Energy. They are interested in the amount of solar energy installed in Switzerland and the energy department of the Kanton-Argau, Martin Herthach and Peter Barmit, are our two project partners. The Federal Office for Energy in Switzerland has released already a map of all the roofs of Switzerland. So they do know where the buildings are and they do know how steep the roofs are and everything like that. And they made this awesome map here to the right, which is basically a type of cadastral system where you can find out how much potential for solar energy usage your roof or the roof of your neighbor has. And they incorporated relatively complex roof geometries, I would say, like level LOD2. And you can see, you can click on a single shape here and it's going to give you the suitability estimate, the roof area and also how much Swiss francs you can earn by installing panels there. So this is the side of like installing new panels. What they do not know yet is how many panels there are. So we have this data set that's based on the Swiss Building 3D data set from the official office Swiss Topu, topographical office of Switzerland. There the roof shapes are incorporated. And what we also have of the same office is the aerial imagery. So we have 10 centimeter ground resolution, which is the news data set and 25 centimeter ground resolution, which results in about uncompressed 200 megabytes per square kilometer. And we also got PNG tiles that you can take out of this, which we use for our machine learning approach by one megapixel and they result to two to three megabytes as PNG. So we have this, we have the aerial imagery, we have the vector data, we have the roof size and the solar potential. We know that there are around two million buildings in Switzerland, which could incorporate solar power. But what we don't have is the location of each system. We don't have the size and we don't have the type of solar panels. They know some, the Federal Office for Energy knows some because of land subventions like money plans paid out for the owners of these panels. But it's far from complete. So this is what this project is about, getting a complete idea. You're not the first ones doing something similar. This was from Stanford, a deep solar project in 2018. They already used a type of segmentation AI system based on Google maps imagery, which has a much more course resolution than the 10 centimeters or at least back then had a much more course resolution than the 10 centimeters we are working on. So we are really trying to figure out now how far we can get this. You get these common types of object detection, which most of you will be familiar with. You can classify the whole image. You can localize an object within the image. You can detect the objects and instantiate them. So ideally if knowing how many cats there are in a picture, or you can do the most difficult approach like finding the actual shapes. My work here is based on a project I did in 2017, where I detected wildlife on aerial images done by a thermal infrared camera. And I used TensorFlow back then really successfully to arrive at a detector that can in real time identify certain animal species on thermal sensors. So we thought, okay, let's use this. Let's use the faster us in an approach as an initial base to find out where these panels are on our tiles from all of Switzerland. You guys are very welcome to try this type of analysis out yourself. I made a little tutorial notebook about this on Google Colab. I can show you quickly on a browser how it's going to look like. The link is here. So this is Google Colab. This is all the code is documented. Make sure when you execute it that you first downgrade NumPy out of compatibility reason because this code is already a bit older than a year. And then you have to restart the notebook and it should run completely through it down with the data set. And then you can actually run a little solar detector for yourself. Okay, so it's going to throw you this error, but there's no problem with it. You just click restart runtime, say yes, and then you can redo it. So this would be that. And it's going to result not with quite, because it's preset with only 3,000 steps learning and to not take up too much of time, well, it's going to result with some similar images. But what we got out of taking this code then was with pretty high accuracy already finding out where panels are, not the precise shape but where they are. So okay, this is not exactly yet what the federal office wanted, but it's already going in the right direction. What we could use it for then because the average precision for photovoltaic systems was actually really high with about 92%. And for thermal, it was like about 62% because a lot of them were detected as photovoltaics. They tend to be a bit smaller, they are a bit harder to detect, they have a higher heterogeneity. So we thought, okay, we go for a multi-layered workflow. We split our data set into these tiles, we use faster RCNN to identify the tiles. And then we wanted to start a workshop where a few professional experts they actually label the concrete geometries of this. So subsequently we can try and mask RCNN to find the actual geometries. I'm going to run you through this, you don't have to look at it, it's not too much detail because we have actually nice images for this. The mask RCNN, a lot of you might be familiar with this, I think Metaport started it a couple of years ago. It's actually quite a recent technology to find out precise segments of objects. It's an extension of the faster RCNN, so it's similar deep learning, but it's based on masks, like the name suggests. So we had to generate these masks somehow. And not only that, we can't just do, okay, this is all photovoltaics, the red one here, and the green one is not, for example, those would-be-roof windows, I know they look pretty close. We also need to instantiate them. Instantiation means like in the image, we need to give each of them a single ID, each system that is separate from another. So we actually know how many there are, and we can form concrete geometries. Last year's EuroPython, we had a short code sprint where there was a lot of cool inputs, and then we programmed a small cloud contribution client also from the input that came out from the code sprint where people can actually label the data, and so it's very specifically for a project. So it's a bit hard to take this over to another project. Then we did a workshop. We invited a couple of trained people and let them label a bit more than 30,000 polygons on the 8,000, more or less 8,000 image tiles. They are not very evenly distributed throughout Switzerland, like because we had a much better databases from our project partners in Agau, we had more there. Agau is like one of the cantons of Switzerland, but we also took some population density cities outside of that area to get labels. I'm not going to run through all of these 8,000 images with you. This is just two examples. I was looking like solar panels are a relatively good thing to label because they generally don't have extremely complex geometries. So there's a lot of 90 degree angles that's easy to do it with pointing other objects you might have to do with a brush type of tool. Here you can just digitalize them or digitize them. Then we generated the masks, all the elements are instantiated as colors on PNG images that have a fairly small size that fit exactly over the aerial imagery tiles. You can see what the labels look like. So we have a lot of systems, solar systems that are actually smaller than 10 square meters. That doesn't necessarily mean that on one roof there's only 10 square meters of solar panels. There might be multiple rows of 10 square meter solar panels, but then generally most connected panels are rather in the smaller section. They would fit on one of our tiles. And the question is, okay, we get implementations for both for PyTorch and for TensorFlow. Sure there are a couple of others, but there we have great support from the community from these two. And we know that there are examples using mask RCNN. And we thought we were going for PyTorch as a lot of you might be familiar with the library. It's got quite a Pythonic interface. It's got a GPU support and a nice NumPy Torch tensor bridge. You get a lot of pre-trained models, which are available Torch vision. You can try out multiple optimizers with a fairly straightforward approach. You can see it how the learning rate scheduler here is optimized with just a few variables and the weight decay of the optimizer can be set manually or can be programmed easily in Python. So this is a great approach. And we had one of our students, Sam Wilstam, try this out in 2019, also using Google Collab on Avalanche early detection. It was working quite well. We were actually surprised ourselves. So this is like webcam imagery from the Alps. And the red ones are detections which could result later in an Avalanche. And the green ones are faults that do not necessarily become an Avalanche. So this was one of these projects that led to us thinking, okay, this is actually quite a cool technology to use here. We set up our own high-performance computing system, which is a 48-core CPU system with a lot of RAM and a lot of memory, but most importantly, the four high-speed graphic cards, the GPU units with a lot of CUDA cores that you need to have efficient training running here. Our front-end is a Jupyter Hub system, which is phenomenal. It's really great. If you know what you're doing, it offers a terminal, Unix terminal right out of the browser, and it can run while you leave the computer. So this is great for machine learning. You don't have to have your own laptop clogged up or running, revving really high at home. So a server-side solution is really a way to go for also these amounts of data sets that are dealing with him. We have quite a, we've used the TorchDutils data data sets option to try out some data parallelism. This is not exactly easy to implement, but it's possible. So we don't have really provided examples for the parallel multi-GPU system, but we figured out through a try-and-error to use multi-GPU support here. You can see how we instantiate the PNG images, the object masks, the label masks. Those are all predefined data sets that we can just use out of provided examples. And this would be the lost graph for one of our runs, where we did the whole data set of the 30,000 polygons. And you can see like at around the sixth epoch, it's not really getting any better anymore, the models. We're using a ResNet 50 here. And it was about expected that it needs about this time. We achieved this after about two hours of training or something like that, plus, minus. And here's some preliminary results where you can see where we're going for. We realized when you see the differences in the precision value here in the segments between using all the categories like photovoltaic thermals and other panels together in one group, the precision increases dramatically compared to when you try to have the mask RCNN set up that it would differentiate them by itself already. So what we see here is that we have really more than 80% precision running in a single class paradigm. So this was the way to go for us. You can see also that is running relatively smoothly because the dependence between the precision and recall is rather linear. It's actually OK to use as an approach. Some examples qualitatively how it looks like. The masks turn out really well. There's not too much spill over or under. There's obviously still some problems. There's roof windows that look like solar panels. Or there's these gray structures that have absolutely the same shape as newer panels that are more black and white and reflect the sunlight and the gray matter. So there's the challenges. Also, the small ones didn't do too well. The ones below three square meters. But when we cleared them out of the data set, the precision or the recall wouldn't drastically increase. So we thought we'd rather keep them in. We want to go over our labels again. Like there's a bit of problems with reflections. Sometimes labels were created. This is just a playground slide here. This is like a bitumen roof, tar roof or something like that. They look very, very similar to solar panels. So we have to clean these up still. This is still quite some manual work. There's also some complete labeling mistakes in between. But I think with scales like that, it's relatively normal. So we have a computational load for a single run inferencing all these images over the complete of Switzerland, which is 4 million images. And each image would take on the CPU course 2.1 seconds of inferencing and using the parallelized GPUs one second, which is still resulting with 46 days of inferencing. A bit too much, to be honest. We are running into really big data problems here. And we figured out that the one big reason why this is still taking so long as the IO, the input output operations between the hard drives. And the GPU. This is actually more something about job scheduling, what we're dealing with here. So we figured out a job queue approach through MongoDB, NoSQL database, where all the detections get run on more or less unlimited processes that just get thrown onto the GPUs one after another and then written into the database. This worked really well as an approach. And we are currently inferencing the whole country on plus minus 10 days. Still potential by optimizing the model. And we could reduce the inferencing times, maybe by using a different model. But that would require probably using TensorFlow, since we don't want to really develop a new mask R, CNN approach for ResNet 101 or something for PyTorch. And shifting the whole architecture is a bit of a problem right now. So we're probably not doing this, but rather optimize the model that we have. And it's not too bad. The 10 days, that's like a more realistic thing. And we are running this currently. That's taking about another two days. And then the first complete Switzerland-wide data set will be available. So another option would be to use hybrid CPU, GPU system. So to load the CPUs as well with processes for inferencing. And there is actually still, we can see the wattage of the GPUs here of the four Tesla GPUs and the memory used. This is not actually running at its full capacity at all. So we are still bottlenecking at the database level and getting the data sets in. The actual inferencing takes probably really quick. And we could still scale this up and optimize for a higher GPU load. I already mentioned this slightly, that we want to try out maybe ResNet 101. Or there's also this new ResNet plus inception V2, mask RC and N approach. But since we don't want to start this whole up ourselves on PyTorch, we either have to wait a little bit here for the community or just try it out in TensorFlow anyway. I mean, it's not wizardry. It's cool, but it's not wizardry. Obviously, some more manual labeling would be always great. Like more data is always better in our times, isn't it? But we also have some post-classification strategies which might render the other things like superfluous. We have this post-classification workflow now set up, where we run multiple models for inferencing, like with multiple optimizers, with multiple presets, or different data set handling, and different tiles cut, like a shift in the tiles. And then we want to do an heuristic analysis on which pixel is actually with which probability by multiple models used. Also, including the near infrared data is an option. There you can see a map of a 10-centimeter coverage that is currently available. So this would fit well with our multi-model and the heuristic analysis. And in the end, we could drop this all into a random forest classifier, together with cadester data, GIS attributes, like there are a lot of big statistics data sets, area statistic data sets in Switzerland available that we could use for this type of approach. And then we would possibly additionally run an exception model, like a classification approach on the tiles, on the actual masks that we have identified already as having solar panels, in order to figure out which type it is, is it thermal, or is it photovoltaic. OK, that would be the end. I'll give you this little goodie picture for the end, just underlining that also humans sometimes have difficulties telling things apart that look very alike. OK, thank you so much. Thank you. So I actually do have a couple of questions in here for you. So yeah, I love the dog and muffin scenario. That's crazy. OK, so yeah, I've got a couple of good questions here. So Bahi-Hussef, I hope I'm not butchering that name, said, can you not get some of this information, location of PV panels type, et cetera, from local distribution network operators? Yeah, this is true. And this is what we based some of our data sets actually on. But we only have a really consistent data set for the canton of Argo, which covers a small part of Switzerland. And the whole idea of the project was to have a consistent data set over the whole of Switzerland. There are data sets where there's rough point location more or less for larger photovoltaic systems. But for the small private ones at home or smaller, they still might cover more than 10 square meters. But the data is not there yet, especially not when it comes to the actual shape. All right, and then I think we have time for one, hopefully one more, maybe both. OK, so anonymous person asks, there are a lot of different solutions to detect PV based on satellite images and more are coming. What is the state of art and why are there still various companies working on new solutions? Yeah, I think actually it depends largely on the amount of or the type of input data that you have. There are quite various approaches to do this. But I think it crystallizes out in a moment that machine learning at least is the way to go. There have been approaches in the past of using more like pixel based analyzers or OBIA, these types of approaches. But it turns out that at the moment, probably using segmentation based deep learning is the state of the art. Maybe using these multi-parted approaches, like having actually multiple parallel deep learning networks can increase the reliability of these types of analyzers. So we are hoping that we are at the state of the art. By doing that. Excellent. And actually, I think I can get this last one in. So Simon asks, it was mentioned that a MASC RCNN was trained in about two hours with a ResNet 50 backbone on the mentioned hardware. Was the training done with transfer learning or training from scratch? Yeah, we do use transfer learning because we still think, although we have 8,000 images, our data set is not large enough to start from scratch. Transfer learning is much easier to implement. And you have a lot of support by the community. You have a lot of experience that is already around. And quite frankly, the pre-trained models, they are just better on a lot of things than doing it from scratch, just from our experience. So yeah, we did use transfer learning. And we started with Coco pre-trained. Excellent. Thank you so much, Adrian. So if anyone has any other questions for Adrian.