 Detection of mosquito populations using low-power IoT and this was part of our university work. It was done by myself and my teammate Umar, he's sitting there, say hi. And our professor who's from University of Waterloo, so the three of us did this project. So yeah, so what's the claim? So the claim that we make is we use embedded machine learning and we use IoT and we are able to detect mosquitoes with 80 to 90% accuracy and the system we've designed will consume on an average 5 millijoules of energy. So that is like two months on a AA battery cell and the idea is to use FFT, KNN and a base classification stack to classify what is the species of mosquito based on their wing beat frequency. So the parameter we use here is wing beat and it's a very widely used parameter for other projects that are based on for classifying mosquitoes. I'll tell you about them later. So what's the problem? The problem is vector-borne diseases that is, you know, diseases caused by flying agents, especially mosquitoes, they account for 17% of all infectious diseases and they cause more than 1 million deaths and if you guys can Google just Google world's most dangerous insect, you will get the mosquito. Yeah, just if you can just Google it, do it. So it kills roughly 725,000 people every year. That's more than the death toll that's created by terrorism. So you can imagine how dangerous the mosquito is. Okay. So this is a small screenshot that I picked off HealthMap. HealthMap is another open source project where you can see disease incidents all over the world. So the map here is for a period of three months and it's highlighting dengue and zika incidents worldwide. So you can see a major portion of the globe is kind of covered with the red dots. Yeah, so it's a big problem. So vector-borne diseases are a big problem and this map shows it. So had I included more diseases like chicken gunia and, you know, zika, sorry, zika is there. Malaria, I think a lot more dots would have come up. Okay. So what are the existing, I'll just talk a little bit about these. So the existing control measures, we have fumigation, we have mosquito gels, which we all buy from Guardian Pharmacy. We have traps and we also have breeding prevention traps. So I think all of us know about fumigation and the sprays. I'll talk a little bit about the traps. I'm sure everyone would have seen this. Someone who has not seen this poster in their home or in their HDB, raise your hands. Okay. Wow. Okay. During the recent zika outbreak, NEA had put these tiny, gravy traps. You can see them here and they were placed in the staircase in the elevator and they put up these posters in front of our home saying that don't touch it. It's to attract mosquitoes and it's going to trap them. So I wrote a mail to NEA asking, you know, how does this work? They said they can't reveal it. But I'll show you another trap by which you can guess how this works. So this was a trap made by a Canadian guy. It's called the olivant trap. So what they do is they have this tire and they put water and a small tissue in there. So the water is going to attract mosquitoes. The mosquito will come and lay their eggs and it will also put a lot of pheromones into the water saying that this is a great breeding ground. All you other mosquitoes come here. At the end of the day, what the people do is they take out the tissue, destroy the eggs, put a fresh tissue and leave the water. The water still has the pheromone. So it's going to keep calling other mosquitoes come here, lay your eggs. So slowly if you keep destroying the eggs, you destroy the breeding ground altogether. I'm assuming the traps put by any user similar mechanism, but yeah, we don't know for sure. But these posters are still there. Look out for them once you go to your respective homes. Okay. So in all these traps, what's missing? So the main thing that's missing is most of the methods wait for the disease to manifest. Like even if you issue traps, you wait for cases to come. Like even in Singapore, they waited for I think I don't know four or five people to get infected with Zika and then they put out the traps. Traps require a lot of manual installation, monitoring. You need to collect the traps, see what mosquitoes were trapped there, see what other insects were trapped. And apart from this, you also need some amount of geographic data like which are the areas where you have more mosquitoes, where you need to install more traps. So these are the challenges that are not very clearly addressed by the existing solutions. Okay. These are some of the electronic, you know, things that use machine learning, but they don't quite get there, some solutions which I'll be showing you. So one of them is developed by a scientist called Batista. He's an Italian scientist and he developed an algorithm which is basically the three steps which I showed you. We're using the same algorithm, but in a different way. So he uses a sound, the wing beat frequency of the mosquito, does an FFT on that to extract the fundamental frequency. With the fundamental frequency, he runs a base classifier and predicts what is the species. So the machine learning model is trained with, I think, 1000 mosquito sounds of each species that he used. It's difficult to obtain mosquito sounds, so it's just 1000. Yeah, but the problem with this is he used an Arduino and he used a bunch of shields and a lot of stuff and he's giving the audio sound from an MP3 player. So this consumes a lot of power. So yeah, it's a pretty expensive setup, consumes a lot of power. It's not modular. You can't install it with the help of a battery. That's the main disadvantage we found here. And moreover, Arduino is slow. So it takes a lot of time. It takes to the tune of milliseconds or even more to process one output. And what if you have three or four mosquitoes flying into your trap? So then this has a tiny drawback over there. The other solution is by Microsoft. Microsoft is doing this project called Premonition. So it's a big, huge, large scale project that's being run off the Richmond office and their target base is the island of Granada. It's a tiny island with a lot of rainforests and I'm sure there are a lot of mosquitoes as well. So they're running this project there. What they do is they take drones, nothing fancy, just the DJI phantoms. They attach a trap to that and leave the trap in the forest. The trap is going to capture a lot of mosquitoes as they fly by using the same wing beat. Okay, so they use the wing beat frequency to trap the mosquito and then they bring the trap back. They're going to take the, you know, DNA out of the mosquito, analyze it for what all diseases it could carry. And based on that, make some predictions. So it involves genomic sequencing, which is like a huge process. So the disadvantage of this is it requires a lot of cloud processing. You need expensive traps. You have expensive drones. It's a big project. It aims for a greater good, but it's expensive. It's not something. No, no, no. The purpose of this is to get the data and predict, you know, what are the possible diseases that mosquitoes in a particular area would carry. So that's what they want to do. They want to send out traps everywhere, grab the mosquitoes, bring them back in. Yeah, yeah. Yeah, that's like that. No, they want to get the traps everywhere. They want to see whether this method is actually viable to predict. Yeah, you wouldn't keep sequencing the DNA. Yeah, yeah, yeah, exactly. Yeah, you wouldn't keep sequencing the DNA, but you still need to keep going and placing the traps everywhere. You need this. Yeah. That's why so it's a big project that's going on and they're doing it on a tiny island. They still haven't gone beyond that and it's expensive. It's like really expensive. So another project. This is really interesting. I can give you guys a link to a TED talk. So there's this project called photonic fence. They identify the mosquitoes based on wing beat frequency and just shoot them down with lasers. So it's really nice. You can see the photo of the mosquito getting burnt. But the problem with this is the machine looks something like this. So huge. They used, you know, high resolution cameras, you know, multiple laser arrays, it's liquid cooled all the way. And yeah, you can imagine how much power this takes. So it's a nice fancy thing to show off in a TED talk, but you really can't keep, you know, and yeah, the best part is the guy who made it says that he wants to use it for farms. He wants to install this as a fence around large farms. And if any pest comes into the farm, shoot it down. That's the idea of the project. But there's no practical application yet. You can still check out their website there. They have it. You can contact them for a quote or something. But which one? I have no idea. You can check it out. I'll give you the link of the laser. I think the laser won't take more power than other apparatus. Okay. So the improvement we did here was we first developed an embedded system. We put the whole algorithm into a much smaller place. We implemented the classifier on more capable hardware. Like we didn't go far, we know. We use things like Raspberry Pi's and Intel Edison's to get our algorithm run faster. And the third most important part is we created a grid of these. Like the grid is a mesh. It uses a mesh network. So it's arranged as a mesh over a geographical area. And we use it to target detection events. So the second detection event comes, the mesh is going to push the data through a packet switching mechanism and tell you, you know, which particular device has caught which mosquito. So the broad aim of this is let's say in an area like Singapore, we install multiple nodes, and each node is, you know, running the detection. The second or particular area finds a mosquito that's possibly dangerous. You can intimate any that you know we caught. We caught an AD's mosquito or we caught a bun. You shouldn't, you know, really ping for one AD's mosquito. But let's say, let's say over, I don't know, Marina where you find like 100,000 AD's mosquitoes. Yeah, that's like a big event. You should, you should inform any that something like that is happening. So that's the big idea of the project. We want to create small nodes, nodes that are more powerful in terms of hardware, and they all talk to each other using a mesh. Why mesh? Let's say there's no Wi-Fi connectivity or there's no GSM connectivity. Just, and let's say you want to install this in a country that does not have great networking capabilities like Singapore. So that's the reason why we use the mesh. The solution is very closely based on, sorry, the solution is very closely based on Batista's algorithm. We didn't modify the algorithm but we've done a lot of improvements of our own which I'll tell you soon. So the outline is we just capture the wing beat frequency using the sensors also built by Batista so there's no innovation there. We capture the wing beat frequency, we classify the data using the algorithm and we predict the species of mosquito and once the species is predicted we just send it over the network through the mesh. So the algorithm is simple. We have audio samples which run through an FFT and we extract the fundamental frequency, compute the distances and run the base classifier which tells us which class the mosquito belongs to. So currently we've trained only three species as shown in this graph. So I think we have 80s, Tarsalis and which we'll use later on. I think Qlex, Stigma. Okay, yeah, correct. 80s, Tarsalis and Stigma. Those are the three we've trained for. The data is open source available. I can give you a link to that. Yeah, one second. The distances for each class. Umar, can you explain this? Yeah, I didn't do the algorithm. Okay, sure. Okay, yeah the algorithm was done by my teammate. You can ask her the questions on that. Okay, so this is the software processing steps, not much. We have an FFT with a maximum frequency extraction. We have a KNN and we have a base classifier. It's just pseudo code. The real code is can be made available later. So the first two devices we studied was we wanted to see what's lacking in the Arduino, why the Arduino is back. So we used an Arduino and use this guy called Photon. So the key takeaway from this is the FFT is too heavy to run on both. The FFT we're using has 8k samples and 16k samples and it's really heavy. If you put the FFT inside we can't do the basic classification and the other parts of the algorithm. So we did some experiments on this device but they don't solve the eventual purpose. These are the two devices which we used to actually deploy them. They are as verified 3 and the Intel Edison. The reason why we chose the 3 and the Edison are because of the inbuilt Wi-Fi antenna. So you can directly configure them to behave as a mesh and they are very really capable. So we know about their power. A little bit about the specs. I think everyone knows about these. I'll just skip. Yeah, this is a key part of our paper because we talked a lot about sailing energy compared to what Batista had. So these are four currents which we measured in the Raspberry Pi, the Intel Edison and the Photon. So over here i1 is sleep current. So sleep current is when you know nothing is activated. When the board is just on. So the i2 is sleep current, the steady state and no radio. That is the Wi-Fi switched off and it's just on. And then i3 is steady state with radio and i4 is active current. Active current is you know when the device is actually classifying and also the radio is on. So if you see the active currents are really high for the Raspberry Pi 3 and the Intel Edison. But the particle photon is really less and we all know that the particle photon also has an inbuilt Wi-Fi. So we have two mesh setups which I'll show you later. One is where the Raspberry Pi 3 acts as a mesh and the other is where we have particle photon that creates the mesh. But each particle photon will talk to an Intel Edison. I'll show you the setup later. The reason why we chose this is because the active current of particle photon is almost less than half of the active current of the Intel Edison. Okay, so our novelty, what we did to this algorithm. So if you remember in Hackware 1.8 I talked about a little bit about the first part because that's what was going on then. Then after that we did the second and third. So I'll just tell you a little bit about the first part. So both the Intel Edison and the Raspberry Pi have some hardware features that can be made use of to, you know, accelerate. So in Raspberry Pi we use the GPU. It's a very less, it's a lesser-known feature of the Raspberry Pi. All the Raspberry Pies have a GPU but the problem of the GPU is you need to code it in assembly. So that's one reason why it's not so popular but we got lucky because someone had made an open-source FFT library. We just tweaked it to our benefit. So yeah that's how we got it running on the Raspberry Pi. But you can check it out. There are more, there's an LLVM flow that's there for the Raspberry Pi GPU. It's still rudimentary but you can try it out. It's great. And in the, for multi-threading we used OpenMP. Since both the processors have more than one code, we used multi-threading with OpenMP which is very simple. This is also something I explained in Acquire 1.8. How you can use a simple pragma and get your code to run in multiple threads. And we also used inbuilt accelerators like the Neon in the Raspberry Pi which has the ARM Cortex A53 and SSC on the Intel which has the X86 Atom architecture. So we used both the accelerators to give you vectorization. So if you have a for loop, it, for loop without any dependencies, it's going to be vectorized by these two accelerators on the respective boards. This is the first optimization we did. The second thing we did was reduction and precision. So let's say you're using 32 bits of, your FFT is done on a 32 bit integer. You might not always need 32 bits. But if you use a 32 bit integer, it's going to take up your processor bandwidth. So we reduced all our integers to 16 bit while maintaining the data because we saw that, you know, a lot of them are zeros. So we just truncated all our input to 16 bit fixed point in order to make our implementation faster. So we avoided floating point, we converted everything to fixed point, we bought it to 16 bits and we saw that it's running much faster in both Raspberry Pi and the Intel edition. But the Intel edition is great in fixed point. So this is something which we noticed through experiments. Intel edition is great with fixed points and Raspberry Pi is great with floating point. But the 16 bit fixed point any day does better than the 32 bit because you have lesser bits to calculate. And the final last bit of performance, this was achieved by means of compiler optimizations. So we used MS, some flags we've used. You can refer to our full paper. So I can give you the link of the full paper where we've told a little more about the optimizations. And yeah, so with these three we achieve a great, you know, benefit in terms of processing time. So the second you achieve a boost in processing time, you have lesser energy. Energy consumed per process. Okay. So this way we save energy compared to what Batista has done. Because we keep our board in sleep for 90% of the time. So that's what we do with sleep. The second and even comes wake up process, send the data sleep. So that's the key idea here. Other than having a low power processor that takes longer time to do your processing, we have a high power processor that takes short time to do your processing and then goes to sleep. So to just give you an idea that the Intel Edison has three modes. It has a sleep mode. Sorry, it has an active mode, a sleep mode and a deep sleep mode. If you put it into deep sleep mode, it takes roughly, I think 0.02 amperes of current. So it's pretty less. So that's the idea. Put it in deep sleep mode for 90% of the time and wake it up whenever, you know, you have a detection event. So that's how we claim to get our reduction in energy. And yeah, the methodology is we train the system using the data on this site. I can, I'll put the slide up on the hacker page. You can check it out. It's really interesting. And they have some MATLAB code that's given. So you can check out the MATLAB code also. Parts of our algorithm are taken from the MATLAB code given on this site as well. And the key point is for energy efficiency. So as I said, we keep the system in sleep at most of the time. The system wakes up when a mosquito flies by triggering a detection event and it performs the classification and we transmit this data. So how does the system wake up? So we have the trap that is built by Batista. You can check that out here as well. So the second mosquito flies by a trap, I will use a hardware interrupt to wake up the processor, be it Raspberry Pi or IntelliVision. There's a trap that's built by Batista. It's there on this website. We didn't build the trap. So the trap can give you a high signal whenever a mosquito flies by. You can use that as a hardware interrupt to wake up your processor. So a little bit about the experiments I can just go quickly. So these are things we did for the paper. It's not really very interesting. Are you guys interested in seeing or should I skip it? Okay, I think I'll just skip it. So yeah, this is one bottle like I'll show you. The original algorithm run without any optimizations and both Raspberry Pi 3 and IntelliVision show that the FFT takes most amount of time. So that's what we've optimized the most using the methods shown here. So the FFT is what we targeted because the others are really insignificant. And yeah, so this is the reduction in time we got based on optimizations. So we unoptimized this all the way up to like 13, sorry, 11 milliseconds. And once we optimized in both devices, it falls down in the Raspberry Pi 3. It falls down almost 4 and the IntelliVision falls down almost 6. But since the IntelliVision consumes lesser power, even in the active state, the overall energy consumed by IntelliVision is lesser for classification. So this is the accuracy, I don't think I'll go much into these. Yeah, so this is how we deployed the system. This is the case of where I told the photon maintains the mesh and the IntelliVision is going to take the sensor. So we have multiple sensors attached to the Edison which connects to the photon which has your mesh and in the Raspberry Pi 3 you just use the mesh as it is. So this is a little bit about throughput. Basically we have a lot of packet loss once we increase the distance. So there's only so much distance we can cover. So these are graphs regarding that. So there's a lot of future work. I'll put up the slide, you can lead it since I'm running out of time. Questions?