 Thank you Hello, everybody. Nice to meet you My name is Dietrich. I work for Mozilla I've been at Mozilla for more than 12 years Which for in the tech world? Really long time to stay at one job But one of the reasons why I've stayed at Mozilla this long is because Mozilla is not just a technology company Mozilla makes a browser most people know us for making firefox More people know the name of firefox than the name of Mozilla. That's for sure a Couple of hundred million people use firefox very few of them actually know the name Mozilla, but they all know firefox Many people recognize it for me staying at Mozilla and working on firefox is very important because we have one job That's to keep the internet open and free You may get accessible by everybody That is a very big job Against the type of people that we're competing against and the type of technologies that would like to control most or all of your life so There's all kinds of technologies that wants total market control, but for us firefox is a tool We don't need to win a browser war We don't need to have half of internet users or even 25% We just need enough firefox users to be able to guide the development of open technologies And that's what we've done right now all of the browsers that are most popular in the world are standards compliant super super fast and Almost all of them are shipping new features to the web For us. That's a healthy vibrant web When all the browsers are fast and they're all shipping new stuff, that's a win for open technology Open standards and freedom. So for us, this is why we continue to build open technology tools and Why I continue to work for Mozilla after more than 10 10 12 years now but today I'm going to talk about something a little bit different the First thing I want to talk about is zero Zero that's the number of app installations per month that about half of Android users reported by Google had this is in the US last year But it signals a larger trend. How many people here have too many apps? How many people here how many people here have apps that you've installed and never used Almost everybody and they're estimating now that most people use a maximum of seven apps Seven if you are a technology startup, you're gonna build an app. You're hoping people install your app They they don't want to install your app They're tired People are tired of installing new apps. There's a commitment required to installing it out in many ways There's a bandwidth commitment. I Looked at my app updates the other day and it was Facebook Facebook messenger Was a Google inbox and there's another big one linked in maybe all of them were over 100 megs These are updates and I looked at the description of what's what's new in the app in the release notes each one said We update our app regularly to make sure that you have a better experience 167 megs for a Facebook messenger No details did not say what was in the update Nothing, maybe make you safer. Maybe it makes you not safer. Maybe it adds new features Maybe it takes away the features you liked There's no way to know so it's very complicated situation as a as a developer somebody offering a new service as a startup offering a new business If you want people to install your app, you're facing a really tough challenge very challenging world right now for applications So Now there's a new technology is coming around IOT IOT is gonna eat the world. They say it will be in everything a lot of people talk about IOT as a vertical No, IOT is it's a new layer in the stack. It will be in all of our technology These types of things They will be in our homes Microphones cameras in our homes many people don't have a problem with this some people have a problem with this It doesn't really matter and it will be ubiquitous. They'll be everywhere and That's something that that will come to terms will take advantage of for the most part, but For us, it's important that consumers have a choice That you can choose to engage in the corporate cloud ecosystem With a device that sits in the most intimate personal place in your life in your home But what if you had the ability to have the same choice as you have every time you open a browser? where all of the diversity and Vibrancy of the open web is available to you in a way that doesn't require you to Commit to an ecosystem to buy in to a corporate cloud Just to have very personal services in your home in your life on your body for example What if you walk into a room and that room is a blank slate? It's an empty room Like when you open your browser for the first time and you have no tabs open you declare tab bankruptcy you start from scratch You have Your choice to determine what websites you open what tabs you open how you interact with which services There's no gatekeeper at this point Nobody's making the choices for you you have personal agency You decide what you want to consume and you decide how and you decide where if you say something Where that recording of her voice is stored? What services you get to have access to not whether they're a partner of a corporation or? Not whether that service already has their authentication set up through that service or is paying them or whatnot But you have full choice over what's available to you What if you walk into this room and this room is the computer? You can interact with this room the way that you can interact with the entire web That is a future that I would like to see for IOT not locked up in a vertical Not belonging to one platform or another but a Physical computing world where all of these services can talk to each other the same day the way that we can right now on the web Without me having to commit So all these technologies coming soon Internet things keep saying it's going to be big it will but it's not going to be big in one thing or one product or one way It'll be the fact that I have an accelerometer in here. That's physical and a microphone These things is that IOT for me the common denominator of IOT is when Computing meets physicality When people talk about IOT the only difference between that and a screen device or something like that is the fact that there's more physical interaction You're interacting physically is it's detecting your presence. It's listening to you. Maybe it's responding Maybe it's got a camera with advanced computer vision capabilities Detecting your movements or something like that AR and VR again VR is going to be huge Everybody wait it will be right, but probably not in the way. We think it will again This technology will be commodified integrated into many different types of hardware and services and things that we experienced How many people have watched an entire movie in VR? Oh? No, I'm not going to raise my hand because I haven't in neither of you why they say it's going to be huge because right now It's a pretty terrible experience. I mean one Nobody has the hardware to watch a movie in VR very few people can afford an Oculus or or even if you can It's not like you want to sit inside this headset for hours at a time, right? Maybe you try something out you do demos have you played three hours of a game? No, no one's doing this now because it's really awkward and uncomfortable But the technology itself is really interesting. We can build really fun things, but right now fun short things For a couple of minutes right maximum nobody's hanging out for long periods of time I mean anything you start moving around bumping into tables and things like this, right? It's just not possible at the moment, but it is going to be very interesting The technology is going to get a lot better a lot cheaper Artificial intelligence People say we shouldn't even say AI. There's no intelligence there. It's just advanced mathematics machine learning Doesn't matter for the consumers It's intelligent seems intelligent and that's intelligent enough to people again. There's nothing not going to be one AI winner It's going to be in everything. It's going to be in all of our little devices that detect when we come and go things like this, but Seems futuristic Not really a lot of us big basic sensors A lot of it is built using basic stuff that we have right now Things that we have in our phones Like an accelerometer, right? Every phone has one your accelerometer is used in your phone When you pick it up to put it by your head They use the accelerometer in the OS on your phone to determine whether you should shut off the screen So your ear doesn't open an app, right? Every time you put your phone to your head The screen turns off on off on off using proximity sensors Accelerometer different gestures to know what you're doing very physical technology. You can do this from web APIs right now If I set my phone down Nothing's happening What if my phone moves? Well, it means I picked it up or it means there was an earthquake Or it means somebody else picked it up if I'm not holding it and my phone moves then maybe there's a problem Really interesting things you can learn from these APIs that you have on your phone right now Microphone the get user media API in JavaScript Allows you to access the microphone from a web page, which is is pretty interesting. You can speak into it You can maybe record speech. It can detect noises things like this. There's a lot to learn from that The camera you can take photos from the web page. You can take video. You can alter video I'll talk a little bit about that and the speaker gives you a way to give audio feedback Right your phone can speak to you There's a lot of stuff about asking Siri to do things or asking asking okay Google which apparently they couldn't think of a good name Maybe to do things, but It can tell you things as well, you know car car mode some people are using this All these are things you can do on your phone Speech I think is interesting But like if any of you right now started talking to your phone everybody in the room would go That's super rude To talk to your phone in the middle of a group of people Nobody's doing that There's a reason why because it's awkward social construct right interrupting somebody ignoring somebody right? It's not really okay, but noise. I think is really interesting Noise tells you lots of things noise Tells us besides my voice this guy just walked in the door. We could hear it We know something happens, right? Maybe we don't know what happened, but we know something happened With web APIs You can do things like Record you can know if something happened So right now this web page is using the microphone on my laptop To know that I am talking It knows when there's noise and when there's silence Everybody make a lot of noise Now be really quiet we know when someone's home The computer knows when someone got home from work or got home from school Or if you're at work, it knows that your kids got home from school Sends you a notification or it knows that the cat's hungry. Maybe you forgot to feed the cat today It knows by the presence of absence of sound that something's happened in a physical space So I was doing things like set up some code plug my phone in leave it running this all day long at home and And then I would get a notification over if TTT Anytime that decibel level would go above a certain level then I would know when things were happening inside my house Not what of course, but I would know that okay a big sound happened Maybe you could add a little bit of recording using media recorder API Sends you a five-second snippet of what the sound plus maybe or You use get user media and just take a snapshot every time decibel level goes above a certain level take a snapshot send it to you You have this personal bot that sends you a photo every time there's a sound in your house big sound anyway, right? You don't really care about the cat Depends on you and your cats relationship. I won't get involved there, but things you can know by presence and absence This room could greet you as you come in and tell you what the schedule of events is today just by Going from zero sound to some sound Camera we have camera access inside web pages with get using media for many years now people have done a lot of experiments but for me, I'm less interested in and You know just taking photos or making a video or something. I'm interested in what I can learn from the camera, right? Like it's pretty basic. You're like, okay. Hi everybody. You can see me. It's kind of cool You can do things like WebRTC. Maybe have a video conference. There's a lot of better apps for that There's some good websites for that that people are using But what if we can add new things to what the camera can see and do right so if we add some things 3d objects inside the camera space. I Wish I had the other slide back on with the noise because they don't have caught that But so now imagine augmented reality Augmented reality is where instead of virtual reality. We take the regular world that we see and we overlay objects onto it This is something you can do with the camera API Using your camera. I can overlay Pokemon flying around this room. I can add a duck into the space In this room, maybe we have something like a Bluetooth beacon with the URL to the schedule of talks in this room You don't need to install an app at this point. Maybe it just opens up the fos asia schedule page to what's showing in this room If you have google nearby on your phone installed Or any beacon app You walk into this room and you would get a notification You wouldn't even have to look for a url or type it in or scan it You'd walk in get a notification Open up a web page It's kind of interesting Now imagine you walk in get the notification click the url and open a web page which opens up an augmented reality view Using today's technology with no app install required You can do these things with the web today You can do these things with bluetooth beacons, especially as beacon technology is going to become more ubiquitous Baked into operating systems to be more available So we take the same camera view We add instead of 3d this time We add a little javascript and we try to do some we'll see if this will work detection It detects Where my eyes are you can detect a person You can use the camera to detect And add something to someone you could even do augmented reality like snapchat filters and stuff like this inside of a web page You can do things like With more advanced javascript you could put a little work in do some more recognition Types of people somebody's happier sand fun things like this But without having to install different apps right and in a way that people can take this code Share it with each other and develop the algorithms to be more and more accurate Right now these are some very basic examples But all of this stuff works today and the barrier to entry for developing these types of things is very very low html javascript uh mobile phone or you're even a laptop but Most people in the world, you know this this is their computer And going forward it will continually be the only computer that most people are using So finding ways to share and distribute this low barrier to entry technology to more people Means that they don't have to wait until they can afford to google home or alexa You can take your old smartphone and set up these types of things today getting a lot of similar features from it Now let's say we take the camera We add some 3d We also add a little bit of javascript And then you can start doing also more fun things Ways to interact the page that actually detect And work with you This one just experimental sometimes works Not so well sometimes so well But you can do things like track objects augment people That you experience Imagine something like this when you're at a larger event super fun A little bit of javascript to access to camera microphone These apis have been around a long time Not really a lot of mystery here pretty good support from Firefox chrome browsers on android on ios Can't access the camera Can't access the microphone But ios also you can't access 95 percent of smartphone owners in the world. So By building these things for android Using the web you still have access to 95 percent of smartphone owners in the entire planet Which is really really powerful And for me something that when I demo this I like to be able to say Here please all people in the audience try this but i'm sorry iPhone owners you're gonna have to lean over and see what the android people are saying Because your smartphone manufacturer has decided you are not ready for these features yet Sorry So the 3d tools that i've been using to build some of these examples This one is called a frame There was a talk on a frame earlier by this guy over here How many people were in the a frame and vr talk earlier? A couple of people If you haven't heard of a frame It's for me basically like how jQuery changed people's ability to build for the web Over half of websites on the planet using jQuery at one point or another For me a frame is that tool for building 3d content on the web It allows you to declaratively build 3d worlds Using simple html tags It has primitives like a sphere cube cylinder It has an extensibility system That you can build your own components in the a-frame to build more sophisticated things It can import obj files So you can import all of 3d models that are openly available right now into a web page without a plugin required It's built on top of webgl And 3js to make it very easy to build 3d content Makes it great for building virtual reality content We've been having a lot of fun building augmented reality content using camera view like this in webpages Really really really interesting tool and i expect you're going to be hearing a lot more about a frame Over time already in this conference many people and we've talked about a frame. They've already heard of it It's very very easy tool and by design It's very easy to integrate into whatever javascript and web framework that you're using today Integrating into react or angular these types of things because it's declarative makes it very easy because these are tool gets that Amit markup makes it easy to integrate a frame into them The url aframe.io Has a bunch of examples and documentation One of the things that's made a frame very successful is the fact that it has a very strong community If you have questions about a frame, there are many places to go ask and they're all listed on the website And it's a very responsive community. So a lot of people have had success building things on top of a frame This is an example. So the source code that I just showed you you can see cube cylinder You can use html attributes on these tags to be able to modify color Size position You even have a tag for the camera view So you can say where the camera point of view is in the 3d space You can determine where the light source is coming from or even multiple light sources These are very powerful tools Where a lot of the complicated parts have been abstracted away What has typically been restricted to very advanced gaming sdk SDKs Is now available in markup in a web page through a frame Which is very very powerful and easy to use And on mobile phones here, I can interact with it in this way a mobile phones A frame detects the accelerometer and if you come by the mozilla booth We have a bunch of demos of where we're showing vr in a web page that responds to your motion It can't do advanced position detection like oculus survive But it can detect motion through the gyroscope and accelerometer The apis I've discussed today all available on mdn If you haven't been to mdn It's the reference for the web api The reference for the web contributors from all over mozilla other projects many volunteers And it's probably available in the language of your country as well Because we have teams of localizers that volunteer their time to translate it If you're interested in learning more about some of these apis mdn is a website to go to So This is the new web that we're looking at As iot technologies start to proliferate Do we want to install one app for everything in our life? Do we want to have 18 apps to manage our house? Probably not Very high friction Convincing someone to install an app is incredibly difficult Convincing someone to open an url Very easy So the friction to onboard new users in say smart cities What if you go to a smart city and you have to install apps for all different parts of the city? Your user engagement in your smart city program or project is going to be pretty poor If you just need to open a url to engage with a physical space User engagement becomes much higher It's much easier to onboard new users without that type of commitment And using a universal deployment surface that every phone everywhere will support Which is very powerful So if we don't want to see a future where our choices are restricted To high end products To single company ecosystems And i think you're going to see a lot more of this even from those companies as they realize these frictions and onboarding new users The path forward is the universal deployment system that you already have The one that works regardless of any device which is using web technologies to bring advanced technologies To as many people as possible Instead of just the customers that buy in to a specific ecosystem So Thank you very much. I appreciate your time today. If you have any questions, I'll be at the mozilla booth for the rest of the day Please come by and talk Thanks Three minutes. All right Ah, so there are a number of different libraries for doing basic computer vision stuff on image data in javascript I think this one we use head tracker jazz or something there's so many of them and Actually using web assembly you can convert binary binary Computer vision libraries Into javascript and use them against the data that you imported through the camera data It's pretty trivial these days And one of the things that's interesting is that this demo even works on like middle range android phone This is fairly performant But I think as the algorithms get better And as more people start start using these types of libraries It's going to take Less computational power That it does now for sure to be able to do the same type of image processing and basic computer vision You won't get the type of really advanced computer vision that you can when you have Dedicated graphics cards and things like this, but useful enough for the types of common cases That really are in most iot use cases. Did someone show up? Is someone there someone not there did they move Say something This is the most common types of use cases where you're not using really advanced you don't need 25 points per hand Tracking right if you just need if a hand went up or not not too difficult to do with a reasonable amount of accuracy Yeah, yeah, so it it depends on device for one And depends on other system load like any performance problem, right? So there are things you can do to optimize it Right now you're also we're relying camera access in web pages Is usually going through another layer of abstraction than would be native camera access, right? On the phones it really depends, but The I think you really want to optimize for your use case So even though you won't get pure native performance Just like you can get close to native in some web applications for some purposes But for this it's even a little bit slower than that generally you want to focus on your use case if your use case requires Absolute fidelity Then maybe this isn't the right solution, right? But if you need Generalize approximate fidelity you need to know if somebody came in or left somebody's moving raise a hand Raise a fist this type of stuff You don't need, you know less 15 millisecond response time Do you have 200 milliseconds response time? It's going to be all right Also, this is not even optimized code some of these demos I put together just in the last couple of days So you can definitely do more optimization than this for sure But you know, you're never it's never going to be exactly on par with your native But if you can be a little bit behind that's fine Right now Then this is a question I get all the time Can it be faster? There's a question you get about everything Most of the native apps I use aren't that fast My Android phones, whether it's the high end one or the not high end one Can hang repeatedly multiple times every day, so That question applies no matter what kind of application you're putting it to really Any other questions? Yeah, yes, I'll put a link on I'll tag it with fosasia hashtag on twitter Yeah What was the beginning pardon? Oh web assembly 1.0 just shipped so Mature enough to ship in production, but brand new I think the the bigger questions there are what uses can we put web assembly to? I think the technology is there. It's not available in all browsers yet But I'm interested more to see with now that it is available What people are going to do with it? What type of applications we're going to be able to see in web assembly? But Absolutely a brand new technology. We're just beginning to see the uses of it And it's only shipped in production last week in firefox. I think All right, I think we're out of time. Yes Two more minutes Any other question? Yeah Ah, yeah Working on the desktop Yeah Yeah, some of the apis not implemented I feel your pain because just today I was using get user media one of the web RTC apis And you have a facing mode attribute You can choose whether to use the front camera or back camera Selfie mode or room mode, but On firefox, it just prompts you and you have a pull down you can choose which one or you can specify On chrome it doesn't do either. It only has selfie mode nothing else on android They just assume everybody's so selfish. They're just going to point it at themselves. That's it But for augmented reality, you really want to be able to point out of the world around you, right? Not supported on chrome and android yet. So it's really it's really difficult right now web RTC Especially it's a pretty large set of apis as you probably know Somewhat complicated. There's some good abstraction libraries out there for using it Kind of like a frame abstracts 3js extracts web gel, right? But web RTC also has abstraction libraries to make it easier But again capability support kind of comes goes up and down especially on mobile. Yeah, so it's Yeah, it really is and so like it depends on that again I think it depends on the type of use case what you're building and and if you're building a Production level video conferencing system It's going to be really challenging to do that on mobile without our web RTC without Specifying at least you know these two browsers and these versions and things like that Yes, it's really challenging right now, but hopefully it's going to get better More and more browsers are implementing more parts of web RTC Like we haven't even implemented the full web RTC spec yet and obviously with the facing mode stuff Chrome hasn't either right so there's still some work to go there If you want to unbox something, use a device, you need to be able to discover a device Lead you into the device, they give you access to it There was some work on the discovery of the R and the browser side But I think because of security constraints Yeah So there's there's a couple of different things right Bluetooth beacons that's something that's getting Uh, I I'm betting we're going to see it built into android sooner or later Right, they're pushing physical web and stuff pretty hard. They google right now Uh, we'd not be surprised if we see that i beacons on ios things like that a way to project a url onto A directly to a device that that's great because you don't need to be on the same local network, right? If you're not on the same local network that gives you a lot more flexibility for people to onboard onto something through beacons Uh, then local network discovery another thing like mvns discovery of things Mozilla had a project called fly web. We're looking into how can you Dynamically from the web detect mvns entities and be able to connect to discover services and then connect to them that way Which it's off by default right now in nightly, but check out fly web It's a way that we're looking into the possibility of trying to do something like that Uh, but I have another talk where I talk about just that bit about the onboarding of people from physical From physical space into digital space using things like like qr code is the is the the original physical computing Right the original way using a qr code that you can connect from a purely physical space into a digital experience It's awkward and terrible But it's everywhere, right and a lot of phones people know how to use it generally Uh, NFC is another one Tap to transport from a physical place into digital I think there are going to be more options as physical computing becomes built more into the operating systems themselves But right now. Yes, it's still pretty tough. Even if you're on the same local network to discover device. Yeah Okay, thank you