 So yeah, so today we're going to be talking about WebRTC in a doctor's office and some cool stuff We've done with IBM Watson. Who's heard of IBM Watson thing to kick jeopardy's ass Sorry, I have a baby on the way. I need to stop cussing, but So we're gonna be talking about a real live use of WebRTC in something that actually really matters there is a client in Houston, Texas that we're working with that tries to help people with cancer all over the state of Texas and They had issues with people driving six hours to the hospital come see some of the best doctors in the world So they asked to help for us to develop an iPad out for their doctors where they could video chat with Other patients and do remote teleconsults really saved us patients some time. So Chris is going to be my my wonderful doctor today, and we're going to go through a little quick demo So this is a so the iPad we're gonna go through an iPad app It also has Twilio's WebRTC running on their their video chat framework and also a blue mix app that has It's just a note app running in the browser that we're going to use interact with it Yes, it is so I'm gonna ask Chris to open up Open up the app and I'll be be the doctor for me really quick. Sorry. We can't show it up here on the screen anyways So it's this is the app the doctor and see every day They can see some of their upcoming patients and then they can see some of their vital information about the patient and see some Some CT scans stuff like that regular doctor stuff. So let's say I actually wanted to video chat with my doctor today So Chris the doctor could go online and Not good enough for you So he would click the little video icon to do a little video chat with me top in the top right There's a little and I'm going to video chat with them as well. So I'm gonna be doctor general We'll get Justin here. The Justin is here So earlier we also saw this wonderful user experience with WebRTC. So I got asked to use my Microphone and camera. Okay. So in a second, it's going to come up. Hopefully on the screen if the Wi-Fi cooperates Come on. Oh, here we go. Cool Okay So the cool thing here is if you notice at the bottom of the screen It's actually typing what I'm trying to say and we get close to this computer and try to talk Hello, Chris. How are you doing today? I would like to talk about the the cancer treatment we have been working on turned the volume down Okay, the transcription really wasn't that good right there with it works really well with a microphone But it's actually not too bad. What's going on here is Watson is transcribing this text in real time And then we're going to get through all the fun dirty stuff. I'm ready to nerd out Who else is ready to nerd out to see how this works? Yeah, so actually you just pointed out Sergei pointing out a really cool point So Watson will auto correct text on the fly it'll try to figure out what you're saying a context of what you're saying it and try to correct it on the fly So really quick demo. Let's go through how some of this stuff actually works now Go back to the keynotes. I guess I should have had this up here earlier, but my name is Jeff's lawyer I'm a developer advocate for something that IBM has called Bluemix We're gonna go through what Bluemix is really really quickly and then we're going to nerd out So what is Bluemix? It's basically a platform as a service that's built on cloud foundry. Who's ever heard of cloud foundry? It's really awesome So there's also Docker and VMs and I basically think of it as Your one-stop shop for a developer for everything they need from a database to using cool things like Watson Running your favorite programming language for a web app auto-scaling your web app. You can run go You can run node whatever you want you can run So that's really what Bluemix is and that app we saw earlier. That was a node app running because I really like node So let's go through the flow of how this kind of works. So in the beginning The patient would click to call the doctor what I did on my web browser the next And it initiates a web RTC call to Twilio and then the doctor also initiates a call From there from his iOS device to Twilio as well So we got a call established great basic web RTC stuff nothing that crazy there So this is where it starts getting cool. So as we're chatting The dialogue is going on the the audio stream is streamed from the web browser To a node backend over web sockets and then it streams to the node backend and where it's running in Bluemix That audio is taken from the node backend and sent over a REST API to IBM Watson So what does Watson do with it? It actually takes the text and transcribes it from the voice stream into Text and as Serge pointed out great point It'll try to figure out the context of what you're talking about. It'll go back and try to correct itself kind of cool So we'll transcribe the audio in real time real times to keep the key point here And we just like storing the the transcribed audio just for fun And then lastly we didn't go over this in a little quick demo in the iPad app But the the doctors also wanted a way a fun cool way of seeing the the mental psyche of their patient How they're dealing with their cancer treatment So there's another really cool service Watson has that basically well you give it a bunch of text They'll tell you the personality of the author. So if they're like upbeat sad, there's a whole bunch of personality traits I don't understand. I'm not a psychologist. So I don't understand a lot of them But we picked out a couple that you can kind of see the mental psyche of how a patient was feeling about the cancer treatment pretty cool and the last step Sentency transcribed audio back to the doctor in real time over web sockets as well as well all in real time So we're going to go through a couple points and how this works So the first part see the web sockets piece with web RTC how that works and then lastly the Transcription with Watson and how that works as well So I love this picture With with Twilio's iOS SDK and the JavaScript SDK it made this really really easy Combined with an app in Bluemix and made this incredibly incredibly easy to do this and I did this all in three days It was for Twilio's conference earlier this year and made us in three days underneath the knife A lot of pressure and got it done in three days and the point I'm trying to make here is This web RTC and using Bluemix together incredibly incredibly simple and makes your life as a devs so much easier So let's get into the code. I promised you code. Let's let's really nerd out now So I won't go through some of this so This might be I don't know how familiar everyone is with some of the audio streams and web RTC So basically what we're doing is trying to grab the the media stream Let's see if this laser pointer thing works grab the media stream from the the local audio in The web browser grab that media stream connect it to An audio processor basically what that does we'll go through that in a second remember this on audio process Function here, we're gonna see that in a second what that does and it's basically tying the the media stream to The to a function that allow you to do basically anything you want with the audio you can spy on it You can change it all around you can do whatever you want. It's pretty cool And a lot of stuff is really not documented. Well, we're gonna get some lessons learned later But a lot of the stuff is not documented too well on the web So the cool part with with this is really the the web sockets piece streaming the audio in real time to To our IBM Watson so we can get that transcription the doctor can see what the patient's saying in real time So the big what basically what it's saying here is opening a web socket connection to our node backend Then down at the bottom. I should have line numbers on this This mess of this function show result We're gonna come to that in a second again, but that basically any time there is a new Shunk of audio that comes through a new Message gets sent over web sockets new message is emitted over web sockets to our backend that has the audio So promise to you this on audio process function This is really the guts and the meat of everything how this works basically what this is saying is We're going to grab the just the data that's coming off of the audio stream ones and zeros And then convert it actually this is the dirty trick convert it into PCM 16 whoever who knows what PCM 16 is There's an audio file in the room. You'll know what it is PCM 16. It's a high fidelity audio format. That's all it really is but Watson wants the audio in this pretty specific format I don't know why but it just does So I did not put a function up on the screen. It's called export data buffer here on the screen It is a dirty dirty dirty Function that converts a web RTCs audio stream whatever the Whatever formats it's in to PCM 16 pain in the butt. It was not fun figuring it out. That was a lot of guessing and checking So this function This is what after we get that all transcribed after we could get it converted into PCM 16 We want to send it to our our node back in and this is traditional socket web socket code Just omitting a message with our audio to the back end So I know this is a really long slide. I'm gonna try to go through it. There's a lot going on here The first maybe 10 lines is this saying when a web socket message comes in It's just going to create a little json object to send to Watson It's just constructing that so remember that PCM 16 That's where it comes up again right there and all this other stuff is just needed for Watson So I know how to do it the most important line really really simple and this is a node There's an MPM package for Watson, which is really really awesome This this method recognized live. It's an absolutely really really cool method So you give it a json object and a function to call back Once Watson transcribes a text and you can get your your data in and out Watson real time over web sockets Really cool, and then what we're saying here the line below it is We're going to wait for results to come back observer's result once it comes back from Watson We're gonna want wait for it to do something And then at the very bottom where is we send it back to the back end So let's go into this observary result mess Observer results. It's really really cool how this works So what this is saying I'm just going to highlight one line here. That's really kind of cool So when Serge noticed this earlier Watson was renaming some of the audio context. What was being talked about? Watson gives a best guess of when you're done talking about something. They'll try to figure out when you're done talking about a sentence And that's really really cool in the fact that you can see what's doing And then you can go back and Watson will go back and show you something different So one last little Little code piece. This is just a JavaScript on the client side This was the cool thing that was showing the The audio in the web browser and what this is saying is this dirt. I'm sorry. This is really dirty code But basically what this is saying is trying to figure out a sentence structure For when you're talking so it shows periods and breaks in it and we'll go back and replace texts as Watson figures out the text that you're talking about and We're gonna go through some lessons learned with web's RTC, especially with Twilio and doing this in Bluemix So order for starting a call it can be quite buggy at least in my experience with with with this For the app that we did when I made Chris actually hit the button first to start the calls a doctor For for some unknown reason if the call was started first by the the patient of the client It would not work. So that was a pain about to figure that out There is lots of issues with web RTC and Chrome and Firefox Leading up to when I did this demo at Twilio Firefox this came out with a new version. It was like Firefox 38. Sorry if there's anyone here from Firefox I'm hitting on you right now a little bit fire Okay, sorry a new version of Firefox came out It was like Firefox 3802 or 01 or something like that and it broke web RTC for remote audio I was talking on the phone with a rob from Twilio and it was like we're going cancel this demo because of shit This doesn't work, but eventually found a an older version of Firefox fire 30 Firefox 35 that worked So to give Chrome some hate also you cannot spy on the remote audio I don't know if it's fixed yet in Chrome yet You cannot spy on the remote audio from a remote participant in what web RTC right now There's an open bug about it. I don't know if it's been fixed yet. That was a fun one to work through also Thank you. It has not been fixed. Okay, so you yeah, you have to use Firefox 35. Really. It's the best for this right now So multiple web socket streams are really really hard I was trying to set up this demo where the transcribed audio from the doctor and the patient You get both of them at the same time if you've ever tried doing that and node or JavaScript on the client side It's not fun. It's one of the most complicated things in the world to do if you've done it I would love to see your code because I hated my life trying to figure it out Getting the audio format was right The API docs for Watson never really mentioned you needed the audio and PCM 16 Since this demo it's been fixed the documentation says you need PCM 16 But that was a pain about doing that and then converting the audio and the browser on the fly to PCM 16 was not fun That was incredibly incredibly complicated. I eventually found some algorithm that kind of did it and it's over my head Yes, I like it Another Watson service to convert audio from XYZ to PCM 16 like it So personally the patient so this was what I was talking about the doctor trying to figure out whether or not their patients upbeat about their cancer treatment or not so In the demo it was the personality We could not figure it out until the call was over because we're waiting for the whole chunk of text from the whole Transcript transcription to be there couldn't be cooler and smarter about and send over Done a bunch of rest calls as the call was going on you could see the you could see the Person out of the patient changing as the text is going through that would be kind of cool But this didn't do it. This was too much work Scaling web sockets It's not fun. I mean you can do it with like Redis and stuff and really make it scalable But there's also issues like how do you deal with the results coming back from Watson and knowing which web sockets session The sticking on to send back to the client web sockets scaling. It's not fun with that type of situation Spy on the media stream. This was I mentioned the list a little bit earlier It's not well documented at all with web RTC really on how to do somebody's piping with the audience the streams and the Context and stuff that was really kind of complicated eventually like cobbled together something from Mozilla's Documentation and some other sites kind of found some things that kind of work The a lot of these things were not really well documented was kind of really kind of bummer Lastly the the web sockets to Watson dropped a lot for some reason I don't know why but I don't it could have been I don't know what it was But eventually wrote some retry logic in there to reopen the web socket connection, which made my life a little easier And then this is my wonder I couldn't have a presentation without a cat and any questions. I Didn't have a unicorn, but there is a cat No, it's a stuffed animal Yeah, right up here in the front There's an organization. I can't remember the name But they're using software to characterize the style of callers and on this second call from that caller They feed them to reps that are specifically good at handling that style Are you aware of that and you considered building something like that in I've heard of something like that I forgot the name of the company that was doing that but There's another API in Bluemix that you can use is by a company called alchemy API a really cool company that IBM bought you can actually determine the sentiment and You can pull out keywords of their talks so you could use do something like that route If take the sentiment of the conversation see if they're really pissed off or happy and then use that for rally them to the right Type of agent the level two agent the level three agent whatever you want So I think those guys were in Chicago if that helps and yeah, that would be that would be really fun Next step more homework First is it live yet? Yes, this is live all this was running live on the internet There was actually the website was twillio dash video dot my bluemix net that's that's you could actually run the whole demo from there If you go to slash provider, you can fake yourself as a doctor You don't have to have the iPad app But yeah, it's all live on the internet and the patient can be on the mobile app as well Or is it no so just the patient can only be in the web browser And the only reason we did that was twillio on this to highlight iOS to the web browser I mean just a I mean maybe from use case perspective and somebody is really in trouble They would have their phone handy then opening up their laptop and connecting with the doctor and all of that so no That's great feedback. Actually funny you mentioned that That iPad app we that we was up here the doctors actually hated it They want it on their phone now because the iPad was too big. They don't like stick it in their scrubs It's too big for One more just okay back here WebRTC for medical and HIPAA compliance So which pieces are already handled by webRTC which pieces do you need do we need us third-party developers to worry about? Can you repeat the question a little louder? Sorry? WebRTC for medical and HIPAA compliance. Oh, okay, okay I'll defer to that at least on the video the video part to twillio I'll let Rob or or someone answer that later, but For everything that was running in bluemix everything else that was the data was encrypted at rest bluemix is HIPAA compliant So it's fine with that. We actually spent a lot of time with this client going through How their application can be secured. They don't get sued and yeah, we spent a lot of time with that