 Thank you Before I get started. I just want to thank all the organizers of this conference. This has been fantastic I've had a really good time these last few days All right, so I am Louis Montez Montez Lou on most Social network things Twitter and GitHub and wherever else that handle is still available Like Alex just mentioned I run ice dev in the US We do full stack JavaScript. So no JS React or whatever up front and we've actually gone a little further than that in recent years going past the browser and into Hardware IOT related projects. So we're an internet of things company mostly lately so Just this past consumer electronics show in Las Vegas There was a bunch of really really cool products released with internet connectivity In fact one of them you could at least one of them you can use your your iPhone well only your iPhone no Android support and You know search for their product install their app accept permissions Sign up discover on the network locally Device press a button on your screen And a button would be pressed in the physical world right in front of you. So What a time to be alive, right? So of course IOT has been mostly marketing, but there is actually some tech that's come out of this, right? So Recent years there there's been a few things that have helped us to actually realize some of these amazing products There's been a whole lot really inexpensive boards, right Arduino like over the last decade Raspberry Pi And these things are you know really easy to get started with so we can prototype things really quickly it also helps that you know half the people in the world have a Super computer in their pocket with you know an internet connection that helps out quite a bit for IOT as well but that you know one of the things I'm really interested though is just sort of the ubiquitous bi-directional communication right we've You know historically especially with the web have made requests got responses But we have been doing a lot more server push in recent years, and it's gotten a lot easier to do it So that's a that's a big part of this Let's start off with the embedded hardware. So we are front-end or at least since this is the front-end half of The conference there's probably a lot of JavaScript developers in here and anybody that wanted to do JavaScript and robotics or you know Little IOT things has probably started here So the first commit to the node serial port Repository was September 7th 2010 by Chris Williams and this sort of started the whole node bots movement So let's say well before I get there with With node serial port, I mean it's a very popular project now. It's got a lot of contributors But what I really like what they did is a really simple API. So this should look familiar to anybody who's done node It's an event emitter, right? So we can if we want to read data or at least data events as they come to a sort of asynchronously We just add an event listener. So on data and we get we get some data every time that comes across the wire And if we want to write data, we're typically sending like a buffer a node buffer or you went eight array and You know, we just have a series of bytes and you know we use express those as like hex codes so Or zero to 255 or you know if you want to be real specific there with bits You can do that as a as a literal in JavaScript So let's say I want to connect using JavaScript using node.js specifically to a Piece of hardware like an Arduino Uno. So node. So your port is perfect for this And the way this has been working historically as we have our node program fire up establish a connection to the serial port and First thing we do is we you know ask the Arduino What its version is and that version is the version of fermata. So this is a common sketch that we've been using to To automate these pins. So every one of these Arduino's are one of these boards have general-purposed inputs outputs So we can turn switches on we can listen for changes in current so that we can connect a sensor things like that But fermata is a really good way of doing this especially and it doesn't really have to be node But this is really easy to do with node and you know, of course This thing shoots us back a message asynchronously you know what its version is and Then typically the process is okay our Arduino. What can you do? So give me a capability I'm gonna do a capability query and you're going to stream me back all the pin descriptions Like you know pin 13 can do digital in and out and you know pin three can do Pulse with modulation like variable amounts of current going out things like that All right, sweet. We've got a description of the pins We've sort of bootstrapped this connection and we could do something like yay turn on pin 13. So The reason we always do this is the hello world of electronics, right? We need to see that something actually happened from our JavaScript code or whatever language to the physical world And that's why turning on an LED always seems silly that we're making all these steps to flip a light switch But it's to show that we have connectivity and that everything's working Of course, we don't want to send all those bites and know how to do like what the things are low in those last slide That was where the actual bites that are going across for formata, but that's a lot to do so fast forward a couple years after node serial port and Rick Walter made the first commits to the Johnny five library which is basically an abstraction for doing robotics And this has become a very popular project in over a hundred contributors. It's it's really good stuff and Instead of having to know what to send down the serial port exactly. We've got some abstraction like create a board When that board's ready, maybe create an LED and then blink it or you know have a sensor that gives us the temperature or Have a you know servo motor to move a robot's arm and you know, we have like high-level APIs for all this and fast forward a couple more years and Rick did the first Coupling of making it something specific to Arduino and allowed for other boards So he made the the Galileo, which was an Intel board He got that to work and then everybody else sort of jumped on board people all over the world were creating Plug-ins these IO classes that can go into Johnny five to support all kinds of other hardware And some of these boards were you know a lot more powerful than saying Arduino So we could run, you know Linux and no JS directly on the board So they wouldn't have to be tethered at all the robot logic is all happening right there on the on the embedded platform and Over these last few years The the movement of node bots Johnny five all of this, you know, there's been Robots conf there's been a lot of this stuff at JS confident and the movement has grown quite a bit with people doing JavaScript and robotics and Once a year, there's international node bot stay so all over the world people get together and Build robots and this is the event that happens over in Phoenix and there's been four of them so far So four years in row started off just doing the tethered Arduino thing to Firmata moved on to Wi-Fi and eventually Bluetooth So it's it's been progressing and you know, it's a whole day of robotics and at the end of the day We smash the robots into each other. It's kind of fun And you know, there's been a whole lot of creativity that's come out of that as well So, you know people start by putting the robot together and getting a thing to move But then they want to do something like oh, I want to put a web dashboard on front of this thing So I can have some buttons and things to you know to build a UI more than just you know command line node and so typically will Have a node server right like express or happy or something like that and that takes requests in fires up Johnny five You know you build a dashboard and then you're like, okay, but I want this really, you know interactive I don't want to just make Ajax calls from my front end I want to do like have sensors push, you know event source down or use socket IO You know some web sockets to do things in real time from my from my dashboard to my node process running Johnny five Or MQTT, which is another IOT type transport to do publish and subscribe back You know over a web socket or just a direct TCP socket back to that and But so that's a that's a great start and what I really like about progressive web apps Is you know, there's there's been a couple of great talks already on this So I'm not going to go too detail on to what the progressive web apps are but we already know at high level There's a service worker that intercepts things so that We all network requests can go through that so we can run the things offline I'll get by manifesting the thing so we can get an add-to-home screen button We can do push notifications so we can have like a server Send a message to our service worker wake it up Maybe bring up the page, but it could also send up an encrypted payload as well That used to be a thing We just send a notification and then the web page or the service worker would tell the web page to make a request To get data to come back, but we can just do that all as one thing now, and you know They're supposed to also be fast secure and responsive, but and this is really good And we can just add that right now to these existing things We've been doing you know with Johnny five and the IOT But that just kind of gets us to where native apps are in a way Right like that is not pushing it forward as much as like catching up to It's a native Android and iOS applications But there are some things that the web just does better at least in my experience here Not you know, not just the deployment and you know not being in a wall garden or anything like that But what's already been talked about as well is Web RTC now this is actually a really big deal and this why it's I've heard a couple of talks here about this already And the key is was really good and really detailed and this is not detailed but just real simple story here imagine we got we'll call him Peter and Peter wants to send something really important over to his good buddy Varys now Peter doesn't really trust like Facebook messenger or Or Google chat so he doesn't want to go through somebody else's network to send this important information over He wants to just send it directly to him It was discussed before There there are some things that have to happen. We can't just go straight You know peer-to-peer we have to go through at the very at the very minimum. We have a stun server Which is how you know our browser on one side of the internet figures out how to tunnel out And so he gets a response back from the stun server And what does that look like? How does he take that stun message? Use his browser to create something that he needs to get over to Vera so he creates This offer it's about 5k worth of JSON And I don't understand everything in this But what's important is he needs to get that over so we talked there was talk earlier about a signaling server It doesn't necessarily have to be a server in fact While a web socket like between two things two peers Could be a really quick way to get this 5k over to the other person It's not nearly as fast as a raven so a raven can have that message bundled up and go over to the other guy Various on this in this case, you know can can transcribe that message you could put that Right into his browser who talks to a stun server and gets an answer That answer looked very similar Another big blob of JSON that has to get back to Peter now this stuff could be all like said out of bound That was what was mentioned earlier, but it doesn't have to be an actual signaling server it could be anything and He doesn't like Ravens. He's gonna send this back over gmail so he could just copy and paste it and Yay, Peter is able to establish now a direct link and just like we had a sort of the bidirectional Communication with serial port like we have with web sockets On message very important either one of them could read events that come in and of course either one of them can Consend things out so so very helpful and very helpful for For IOT applications and sort of expanding what we're able to do with the web using using peer connections But okay, what's that got to do with putting browser? Robotics together It just think of all these things as a toolbox right to connect every little every little piece but can we just Take Johnny five and use Webpack or browser fi and just expect it to work and talk to Talk to a piece of hardware in Arduino Typically, what do we do? We are we have some problem like this with the web So we'd want to wrap it in some sort of thing and deploy it to a store to have more access for something, right? Well, you might not need to do that. You might not need a wrapper. You might be able to do these things directly from a browser Okay, so anybody heard of midi old school right midi is great and Midis for making music right midi is for playing notes or receiving notes It is for that But what you can do is emulate a midi device So there's a whole bunch of different Arduino's that can emulate any USB peripheral with just a very small amount of code Some of the newer maker Arduino's the older Leonardo micro They can all be any peripheral USB peripheral you want and there's a very easy to use Arduino library to To make USB devices, but also another layer on top of that to say be a midi device Okay, so but the midi protocol is actually pretty Pretty constrained right we know there are certain things we can do like play a note on this channel You know with the first bite Which key you know zero to 127 What velocity we're gonna strike that key so for like playing a piano It's on this channel this note this velocity But it turns out even though we're we're constrained to what those bites are allowed to be that range The the peripheral that we want to talk to doesn't really care But if you think about it if we've got like 64 values in the first bite that we can be that's six bits of arbitrary data Zero to 127 in the middle zero to 127 another seven bits seven bits We got 20 bits to play in because it doesn't actually have to be a musical instrument And what's really nice about this is Since this seems like such a benign thing to do to just send notes or listen for notes You can usually get away with the browser not even prompting you for access to midi ports, which is very handy and just like The other API is that I was saying where we have a read and write This is just as simple. It's not you know the typical event emitter But it is you know, we're putting a callback of what happens when a midi message comes in We got data and our output send takes yet another array buffer another uintate array or a buffer of Bites However, these need to be in that format right they need to be in those ranges or they'll throw an exception in the browser You know the thing about like doing that low-level bit manipulation is that I For years and years didn't ever have to do that It didn't help me to make an enterprise application to know how to move bits around but some people have Been doing this and they've made some really cool libraries on top of it So if you just take the time to figure out, okay Yeah, this like bit shifting and I don't want to do all that But if you just spend a little bit of time on that you might be able to enable some piece of hardware That is very constrained and that you really need to worry about just what's going into it So the stuff that you might have done in computer science that Didn't have a whole lot of use very useful if you're doing any of this robotic stuff Okay, so if you don't want to use MIDI although MIDI is super great for this right now because MIDI is available It's been for a few years at least in Chrome It's there's an open issue on Firefox and there has been some work actually just three weeks ago There was a commit to do input stuff. It's not it's not production ready at them web USB however as of Wednesday is Stable in Chrome 61 so we can use this so what what is web USB that sounds like super dangerous, right? Like just generic web USB access Well, there's a lot to getting to the web us to a USB port through web USB You have to be on HPS the You can't even start until it you know you answer a dialogue that's popped up That dialogue is prompted by the a DOM clicks. You can't just load a page and have it Have it connect you have to instantiate that from a user interaction You also have to put filtering criteria of just which devices you're looking for you have to be real specific and say it's this particular vendor and product ID so But once you get past all of that, you know is if you're building for a specific device This isn't too bad. In fact all of those devices that I was just showing that can do MIDI can also do this web USB and then you're not constrained to just certain bytes You just get to full blast bites back and forth to a device from the browser and This is a little bit more complicated because there's more options on how you send things Like there's bulk transfers and more transactional things But in the end really just reading data sending data input output just like everything else and we're dealing with With arrays of bytes just the same Okay, so Bluetooth another way of connecting Now Bluetooth historically if you've never done like Bluetooth for robotics Bluetooth has been about Like pairing your phone to your speakers or something like that, right? These are older Bluetooth profiles But as a Bluetooth for which has been around for for several years now We've got to the ability to do Bluetooth low-energy devices and this is actually very very different than the old way of pairing Bluetooth low-energy devices let something like go to sleep for a long time wake up do a thing one of the things that they Go to sleep for and wake up is to beacon So imagine There's a little tiny device running on this coin cell battery Sleeps most of the time wakes up once a minute or however, whatever interval you configured it for and I don't know shoots out a URL Like some people in the audience tend to do And that is called the physical web. It's using a bite format called Eddie stone Basically encoding a URL into this packet. That's being advertised every so often This is actually really cool in that it gives us a starting point, right? So if you've got this URL that's coming up and it just shows up on your phone out of nowhere that might be because there's a piece of hardware that we might want to interact with and We can take that a step further cool. We're at this URL. We've opened up this say progressive web app To to talk to something potentially directly so the Bluetooth low-energy stuff unlike the The the older style Bluetooth the you know Bluetooth for on up has what we're calling Bluetooth services So imagine something like a heart rate monitor. It's got a heart rate service. I can have multiple services as well now these Services have characteristics a characteristic is like where the data is accessed at so if a service is a heart rate monitor So we might You know ask for a subscribe to notifications on the heart rate changes So we're able to read like events in real-time or we could potentially have a character that lets us write something back And this heart rate monitor is just an example because it's one that's sort of already known We're using like service IDs that are common, but we could make these up these could be anything So this is like an arbitrary communications channel that doesn't have to be something that you know like some approved vendor Specified you can grab a Bluetooth Arduino and just say I'm gonna create a service that does this thing And if anybody's done a Bluetooth low-energy with a node They've likely been using Sandeep mysteries project called noble and Nobles great for for node very easy to use API What I like about noble as well is since a lot of node projects like The rolling spider the the little drone that's web Bluetooth all of these things that were written against the noble API There's now a shim So you can take your noble node.js Bluetooth code and get that to run on top of web Bluetooth using using like browser if I or webpack Okay, so So that's pretty cool. We can Establish a connection this also have when we're using web Bluetooth when we want to establish a connection we have Some similar constraints to what we were doing with web USB So with web USB. It has to be an SSL sir HPS There has to be some user interaction We have to specify just which services we want to talk to as well a dialogue will come up It will connect to and then we'll actually pair with the device and once we're paired it's a one-to-one thing a device connected to a to a computer to a to a phone and And Once we have one of these service connections and we you know sort of iterate through its characteristics We can ask for on-data or write just like the event emitter just like serial port And in fact a lot of these Bluetooth devices are using a common service called Nordic You aren't and that basically ends up being just like serial port except it's it's wireless And there is a Bluetooth low-energy serial port implementation that we could plug into Firmata and do the the robotics that way if we want to Okay, so here's another one and this one's super super early, so This is very subject to change I imagine but there is a W3C spec on top of it, and that's near field communication So this is like I said, it's really early, but yet another way for us to take a device and potentially Send arbitrary bytes between a couple of devices right so we can have our phone Which has an NFC reader and it could also project NFC messages What does that look like if we want to expose some bytes to something else that's going to pick those up? We're using web NFC push So Navigator NFC push hello world if something wants to read that so on the other side It can set a watch to listen for NFC messages that it gets close to right So NFC watch do something all promise-based And that that should work pretty well Okay, so I went faster than I should have but I'm gonna start doing demos it Hardware demos are the dumbest thing that you could possibly do as a speaker So I'm gonna do the dumbest thing possible as a speaker and do a hardware demo a little bit of setup here Those Leonardo's the those things that can do USB peripheral This is a knockoff a pro micro that was like a three dollar version of that, but it's able to use a library So that's pretty cool The top right thing that is a accelerometer from a Tesla Tesla is one of those other boards that supported by Johnny five what I really like about what they did is those modules Yeah, they're they like they fit really well into a Tesla So they're really great for a Tesla, but they're also using a common standard to communicate Something called I squared C. It's a inter Yeah, I to see look it up. Anyways, it's a way to connect just microcontrollers to microcontrollers integrated circuits to integrated circuits and we can take the accelerometer data off of that and Push it into this Arduino Now why do I have a QR code? Cuz there's You know, they're new sort of experimental web API that kind of want to show off as well Jen in her talk yesterday showed what what they were doing with faces And faces are really cool because you get you know the position of the face and potentially Where the eyes and mouth are at and like a bounding box around somebody's head What I like about the QR code part of the shape recognition API is you could have data in there, right? so that That's cool because we could have position, but we could also have data and if we If we're doing IOT things we typically have our data described as our devices described as a UU ID, right? If anybody's done anything with IOT, it's basically 16 bytes. This is big ugly number If you're gonna do that Don't put the dashes in it for some reason we do like 32 characters and four dashes and I don't know who came up with that or what the reasoning is but 36 characters is just past the threshold of cool QR codes and into the really like nasty high resolution ones So just take out those four dashes and then you got an easier to read QR code saver tip for the day and The face here this little robot So QR codes are kind of ugly UU IDs are kind of ugly But hashing things into cute Rob robots are really cool, right? So if we decide like these bytes mean this color and all that so there is an algorithm in this demo that If you put any QR code you get a somewhat unique ish Robot face out of it. Okay, a little bit more setup here Since this is front end Video tag everybody knows what a video tag is in HTML5 video Inline styles, I'm sure somebody will yell about me putting an inline style there But I want to put the video in the back Canvas in the front, right? Little bit more demo setup 3js was discussed earlier 3js is awesome 3js has this trick you can do where you can say the alpha on your renderer is You can make it transparent and you could set a clear color I'm not sure why it all works like this, but what this does is gives us a 3js without a background right so a transparent 3js And it's a little bit more setup The barcode detector API which I've gotten here can also run in a web worker So even if it's a CPU intensive task we can background that in a thread With Chrome 61 also canvas Offscreen canvas was added, but we don't need that for this all we need to do is get our image data into into the web worker and That that image data if we post to the web worker is processed There's also in this particular demo a fallback So I don't want to use the fallback though because the only good QR code scanner that I found that's completely JavaScript based Doesn't give quite the same effect. It gives us three points the one that's built in Gives us four points around a thing and that makes a difference for what I'm going to do here although this the other way still works but but the native barcode detection Experimental works better Okay, so that was way too much setup for for this demo, but Okay, so like I said video we could just use get user media to put into that video and We can use the QR code scanning now what this is doing is This is not a real AR framework right like ARJS go use that There's another one that just came out by by Google The the stuff that I've seen Jerome any do with ARJS is really cool, but what this is doing is If we're gonna use a marker like ARJS is doing maybe that marker should be a little bit more significant and just hey There is a thing here Well that thing could correspond to something that's actually a device that we can read and write data to right so If you have something say a simple Oh also This isn't a plastic box because I've done demos before without plastic boxes If you're ever gonna do hardware demo put it in a plastic box, or you'll totally screw it all up if We take accelerometer data, and then we encode it as MIDI bytes Then we can take sensor data's match that up with something in the real world So so just think of this is sort of the start of okay We can have IOT devices, you know either beaconing or you know Tethered or whatever that are emitting data that correspond to something in the real world that we're able to then change the display with augmented reality Alright, and all of this right here has been done without like any major toolkit other than 3js But if you don't even have to use that if you know how to do the like really crazy WebGL stuff without 3js so it doesn't really need much more than we're scanning a QR code and Finding that position there was a little bit of math here that I had to learn to do that right like we're taking four points but I'm able to get like Angles and stuff out of it because we can translate those four points into 3d coordinate space with something called a posit algorithm I didn't even know what to search for but I managed to eventually find that so Anyways, that's about the just of it. So just think about potentially some of the these These tools that we have this the sort of toy box of things that we have now In browser that we can we can just build things on top of and stitch together IOT AR You know web VR all of that. It's just been a lot of fun lately kind of connecting the the pieces that's all I got Thank you so much Louie. That was a really sick demo. I'm impressed nothing went wrong I have a question here for you. Are there APIs that you want to see happen in the browser that aren't there yet? There there is So a couple of things one so I had to sneak over the bytes with MIDI we can maybe use web USB It's available on browser I'd like to see just a web serial for the you know older devices also the one I think is super important is some way to do UDB broadcasting that also sounds like super dangerous like can I just broadcast but The the handshake that I did with the P2P thing like the you know sending it over somehow if we could be servers With either Bluetooth, you know if we weren't just clients from our machines if we could also broadcast something that another one can pick up We could potentially you know Build actual mesh networks. We could do more of this decentralized computing if we had way of broadcasting things out So as soon as that happens a whole new class of just browser base No, you know no extension applications open up Very very cool. We're going to break a little bit early to get ready for the lightning talks. Thank you, Louie Let's give him another round of applause