 Hello. My name is David. I was working on a project called Project Lighthouse, which I initially planned to introduce to you today. Things were a bit complicated. There were a few interesting plot twists. Between the time I submitted this talk, then the time I finished the slides, and the time when I'm presenting. Let's start with the trivial plot twist that the prototype has broken at some point. I'm not going to make a demo of this. What's Project Lighthouse? Project Lighthouse was a project of Mozilla's connected devices branch. It dedicated to helping blind people live in their daily life. This talk, at some point, I considered renaming it how not to die when you're preparing a hardware prototype, which is pretty ironical given that connected devices died two days ago. So I'm still going to present. So, what about us? We were Mozilla connected devices. Connected devices was a set of internal non-profit startups inside Mozilla. The idea, when it worked, was to have great ideas, make great prototypes, try to convince people to give us money to turn the prototypes into actual products, and hopefully to make products that would help users. Connected devices has produced a total of zero products, which might be so much related to the fact that we don't exist anymore. Well, that's life. But we had a bunch of good ideas at some point. And since we were the rebels, we were still using the dinosaur and not the new funky word logo of Mozilla. This might be the last time you see the dinosaur. So, I didn't really tell you what this is about. I mean, what Project Lighthouse is about. Because initially, Project Lighthouse was about, we know how to use cameras, we know how to program this kind of thing. Can we do something not creepy? Because if you look at the use of cameras in connected devices in IoT these days, just about everything is about watching you do something and telling someone on the cloud exactly what you're doing. Which is, incidentally, one of the reasons why some many big companies are interested in this, for strategic reasons, for insurances. And there are many companies that have interest in knowing what you're doing at home, either to sell you things or to charge you for things you're doing. And that's one of the reasons why there is so much focus and so much money involved and invested in Internet of Things these days. It's not just because it's fun. It's also because being creepy is profitable. But we wanted to do something not creepy. So, what can we do with a camera, a Raspberry Pi and a microphone and a speaker that happen to draw more juice than the alimentation of the Raspberry Pi can provide? Which is pretty much why I cannot do the demo today. Not the only reason. And can we do something useful? So sometimes the solution meets the problem. Because there turn out to be some people who have interest in having a camera and something interesting done with the camera. Some people's life can be changed with the camera. So according to the World Health Organization, there are close to 300 million users, sorry, people who live with a sight impairment. About 30 million of them are entirely blind. But still, that's many people who have trouble in everyday life recognizing objects, for instance. This means that in shops these people have lots of trouble. They need help to find out exactly what they're buying. And since shops tend to, at least in big shops, tend to change the place of things regularly to force people to revisit everything. If you're sight, if you're visually impaired, you're in trouble. Because they are actively working against you, against all your brain or muscle memory. They are trying to force you to not know where you are anymore. There are basically, basically the solution is find help. So there are a few devices that I know of, essentially two specialized devices to help people in this kind of situation. They both are pretty much more than $2,000 for something that actually contains this kind of hardware. So the idea is, that's worth about $30, maybe $40, I don't remember. Plus the battery. So the idea is, can we do something open source, open hardware, and cheap? And we set out to try that. So more specifically, we wanted to help visually impaired people recognize household items. Because if you have two packs of the same size, it's pretty hard to know. Well, if you have two cans, one is Coke, and one is, I don't know, Fanta, you have no way of knowing which one is which if you don't see them. But in that case it's pretty easy for a camera to make the difference. Other things are much harder. But we tried to see if we could do that. So we actually got good results with the one above and pretty bad results with the one below. But we only had a few weeks of development. So how do we do that? So we get one of these. One of these, that's a small camera. A little software, two developers. Oh, sorry, yeah. Lots of open CV, Python. So two developers working for a few weeks time. And we actually managed to... Oh, yeah. And the final ingredient. Users. So I have pictures of actual users, but I'm pretty sure that I do not have the right to put them in an image. So I use supervillains instead. So we have 24 test users. So we actually tested something that was a slightly more refined version of this. This was prototype number one. There is a prototype number two that still is somewhere in California and probably going to stay there. So I could not show you. It would have been a bit easier to make the demo because it does not rely on so many outside stuff. And so I call this presentation lessons learned. So lesson number one learned. If it hasn't been tested by users, it doesn't work. Just because it's cold, just because it has code, just because we had fun developing it doesn't mean that it works. So we were lucky after relatively few iterations, we got something nice. So that was prototype two. It's a bit ugly, but not quite as ugly as that. And our lessons were yes, not all of these 24 users were interested, but enough of them were interested that it was worth doing. They thought it would change their life. Not everything in their life, but at least shopping, which is an important part of life if you need to, you know, eat. So we're supposed to work on making on keeping it cheap, making it simpler and have a better form factor. Okay, so that was making a prototype. How do we get from there to a product? So we put some designer to design some stuff. We put a factory and that's pretty much not how it works. We thought we could do that, but the truth is it's a bit more complicated. So I found a document on a blog that details a little bit more of the process. From the time you get the hardware prototype and a good industrial design to the time you can actually ship it. So forget the marketing on the top, forget the app development on the top. You still get all these three, four lines and our estimates were that they could take about six month time to get done. Could actually be worse. When I wrote this slide I was preparing to tell you that it's probably going to take more than six months. In fact, it's going to take forever, so I was right. So the question here is what do you do? Are you going to sit down during six months and do nothing? How can you not waste your time? So again, this lecture is about lessons learned. So we eventually came up with a solution and it's actually a simple one. You are building a device. You want to test it. So test things, if possible, with devices that people already have. Most people these days have a smartphone or a tablet. Some have a smart TV. So if you can find a device that your users already have, you can parallelize things. You can have someone, possibly you, working on the hardware, finalizing the hardware, getting the providers, getting the supply chain, getting everything together, testing actual hardware components, et cetera, et cetera. Something that's going to take you insane amounts of efforts and time. And you can also work on the software at the same time, which is the things that some of us do best. Me, yes. So in this case, we set out to use a device that blind users have. And this device is called an iPhone. It turns out that a huge number of, we actually have numbers of blind users, not vision impaired users, but for blind users, it turns out that many of them have iPhones. In the US, it's like 85% of those who have a smartphone have an iPhone, and many of them have a smartphone. I don't remember the exact numbers. So this means that we got ourselves working on a third prototype that was that. We're all foes them. We are in open source stuff. If you want to develop for open source for this, yeah, all right. Let's just say that I have seen many platforms on which developing open source software is nice, simple, easy stuff works. This is not one of them. Apple is going to extreme length to make sure that everything is locked down, and you have to go through them for everything. So part of the lessons learned. Yes, you need to go where your users are for testing. If you can avoid that platform, it's worth trying. We actually only got three weeks of work on that. Sorry, five weeks. Most of the time for me was spent actually getting Xcode to work, getting unit testing to work, and trying to actually release binaries. We have not succeeded at that yet. Everything is locked down really tight. So at this stage, it would be a great time to discuss the next steps. What have we done? We have an early prototype of that. Oh, I could actually do a demo of that. Do you want to demo of that? Let's see if it works. So you don't hear anything. Do you? Plenty of people behind the camera. Oh gosh, completely from the look. Okay, so a complete failure. So it was trying to find out which was the interesting object in the scene and apparently the interesting object might be somewhere on the floor. Anyway, it's not the interesting object. Well, I think this really is a very good day for me because also the microphone is dying on me. Okay, let's forget about this demo. So let's pretend that I have a very nice demo. Should the next steps be? Okay, fair enough. Make the demo work. That would be a pretty good next step. What should the other next steps be at that stage? At that stage, assuming that we have software that works nicely, assuming that we are still on track to get hardware. Oh, actually, we do have many nice hardware designs. I did not include them in the slides because I think they involve commercial partners. So we probably do not have the right to put them here. It probably doesn't matter anymore. But hey. So at this stage, our next steps will be... Okay, so make it work for real. The prototype apparently doesn't work at FOSDEM. So open an issue doesn't work at FOSDEM. So we have been working so far on making some basic computer vision algorithms. Obviously, making better computer vision algorithms would be helpful. But the nice thing is, since we have the ability to prototype on this ugly stuff, we can iterate and paralyze. So people working on improving the computer vision, people working with actual users on improving the usability and the accessibility, people working with users, not necessarily the same ones, on coming up with new features. We were also planning to go further because at this stage, this is an IoT session, I was talking about T, but there was no I. There is no I in team. So one of our plans was to make it social. Not in the bad way, let's add Facebook everywhere. But in the good way, as in sometimes, people need more help than just identifying something easy to identify. Sometimes people need help to find, oh gosh, I'm wounded. Does it look bad? One of my breasts feels large. I'm maybe sick. I mean, that's a pretty good one. And that's the kind of reasons for which you need more help than just what a dump machine can help. So we wanted to help people share their questions with people they trust. And possibly in cases that were more broader, share questions with just about any volunteer. So we were in parallel starting to work on a platform for crowdsourcing questions from people with site impairments, such as, what is that object? What am I doing here? And that is probably not going to be released ever because there is no single line of code written yet. And nobody paid to do it anymore. But that was the plan, in parallel. As I said, if you don't want to die, you want to avoid multiplying devices, real devices. But you can, again, fake stuff using existing devices. Once you have software running, I mean, one of our industrial designs, for instance, was actually a necklace, a relatively heavy necklace, containing, well, basically, this. And that's something that we can fake easily if we have software. We can easily make prototypes that don't have to go through a factory for that. You can pick a real necklace or anything that fits around your neck. Try it and attach stuff. That's not sufficient for commercialization, but that's more than sufficient to find out whether it's too heavy for users or whether users actually manage to point the camera in the right place because we actually got tests. And tests actually say that blind people point cameras very well. That was a very good surprise. They have difficulties in judging the distance, so sometimes it's too close, but it's centered. And that's good because that's something that needed to be tested. So anyway, if you want to avoid relying on having a finished hardware device before you can iterate, and one of the ways to do it is to have fake devices. So just something rigged together using a device for which you already have software. Also, one thing you do not need to forget if you are working and if you want to somehow release a hardware device is that it does not stop there. You probably have lots of legal things to cover that we are not used to in the open source world. In this case, people could possibly have accidents because of this, physical accidents. People trying to use it to identify medications, for instance, could end up with the wrong medication, or they could end up with rat poison, eating rat poison because we had told us that it was beans, red beans, something like that. That's pretty tricky. So of course we want to make sure that the software works. But many of these things have real-life implications, which among other things involve lawyers. In this case, we should also have been in touch with a number of people such as organizations helping blind and sight impaired people because that was the best way to get in touch with our users. And one thing that's often overlooked is the Kickstarter. Kickstarter, people generally see it as a way to raise money. The secret of Kickstarter is that Kickstarter is actually a marketing tool used by companies to measure interest in a product before they develop it. If people are willing to invest $10,000 or $10 million that depends on the product, this means that you can sell the product. I know of a number of cases in which Kickstarter has been used to sell a product that had no kind of existence in the real world just to measure whether it was worth trying to build such a product. Some of our potential partners did that, for instance, and we had surprises. So, again, if you're trying to build something and turn it from hobby to something that you can distribute, Kickstarter is a pretty good tool for that, not just for raising money but for to check if it's worth doing. That slide might be slightly deprecated. Okay, regardless, you can find all the software prototypes of Project Lighthouse on GitHub and we have a discourse forum if you want to, I don't know, sympathize with us. Thank you very much for listening. And if you have questions, I will be happy to try and answer. Yes? What's going to happen with the prototypes of all the work involved in the Lighthouse? Is there a release on the insert, the software, or is that...? So, what's going to happen with the prototypes of Lighthouse? Will they be released? So, the software prototypes are on GitHub already. I mean, we need to put a license because we just forgot that in the rush, but they are on GitHub. The hardware design prototypes, I don't know. I would like us to release them. I do not know if they belong entirely to us. If they belong entirely to us, I think we can release them. If they don't, we probably can't. Yes? I was just wondering, if so many blind people and I imagine that you could have a smart phone? Yes? Did you consider just thinking about it, not bothering you? So, the question is, if so many blind people and vision impaired people have smart phones, why not just make an app? We also considered that. Yes, that was a separate track. I'm an IoT session, so I was concentrating on the IoT stuff. But yes, we considered continuing the app track past the prototype phase. Great. Thank you very much.