 We're past time to start so I might as well start. Hi everyone. Welcome. My name is Andy Trice and I'm on the I'm so used to calling myself a developer advocate, but I'm actually not a developer advocate anymore But I'm on the product team for bluemix particularly working on architecture for local developer tools, but That's not always what I've done like I've gotten to do a lot of different things as far as cloud connectivity goes and Especially my role and previously in developer advocacy got to work on some interesting projects Some of them to kind of explore what's possible and some of them transforming those explorations of what's possible into real-world solutions across a variety of different industries and today I'd like to talk to you about specifically use cases involving IOT and in particular flying robots and cloud foundry and connect combining those with cognitive computing to come up with Very very interesting use cases that can make a difference and provide value in real-world situations I'm Andy Trice on Twitter, so if you want to heckle me or you know say, oh man, this guy doesn't know what he's talking about That's fine. I'm cool with that But let's go ahead and start with IOT and in the most broad definition of IOT What we're talking about is a collection of sensors and actuators that are interconnected and so they work together to achieve some kind of task or solve some kind of problem and You know, I started off talking about like flying robots But but now I'm kind of talking about more general like sensors and actuators when we're talking about sensors We're really talking about literally anything that's collecting information in the case with this we're talking about Telemetry data so information about the location of it the altitude the pitch You know how it's flying and we're also talking about the images that that are being pulled off of the the aircraft mid-flight but While I'm talking about this particular use case don't limit yourself saying oh, we're just talking about Getting pictures and doing something with them The approaches that I'm going to show you could be used for literally anything So whether it's a some kind of sensor that's pulling like moisture from the ground or temperature or infrared something I Don't know if you can think of a sensor that's collecting information there's ways that it can be used in similar workflows using similar services and When we talk about actuators, that's like the other side of it So sensors being data coming in actuators meaning we're doing something to perform an action whether it's an Action that you as a person are doing to change your business workflow, or whether you're using some sort of robot to go ahead and You know cause some kind of action Automatically based upon this that sensor information that's coming in And now we have to think about with all these different components So whether it's a flying robot whether it's a temperature sensor whether you've got an array of sensors Let's say an intelligent agriculture situation you have sensors all over the field Like you have all this information coming in from so many places like how do you collect that information? How do you manage it? How do you make it actionable and how do you as a user interact with that? Parts of it might be automated parts of it might be user driven and the way that you can go ahead and Build these systems can be built in a number of different ways You can use basically streaming analytics message cues or messaging services You can have asynchronous processing in particular You know for IBM we have open whisk, which is a serverless programming or serverless programming model, which is event driven Sorry, it's beeping at me up here So you have all these different options that you could use to build these kinds of an application But what being a cloud foundry summit? I want to focus on in particular iterations that I've had that involve cloud foundry and not just cloud foundry for the sake of cloud foundry but Using cloud foundry and no JS to drive innovation And if you're thinking about like, okay, that just sounds like a lot of BS. I can I can understand that But but there are distinct reasons that I've done a lot of iteration with no JS and cloud foundry No JS itself being it's a really easy language to get things done very quickly Is it going to solve every problem for every person in every situation probably not? But it solves a lot of them very well And if you're also in a situation where you need to do a lot of rapid innovation and everybody talks about like Oh, if you're going to do something fail fast, but let's look at the other side of it So not just feeling fast if you're going to do something prove its value you can do so very quickly I have personally found no JS to be a very effective tool in building these kinds of solutions to prove out concepts and say Hey, we can take this we can make a couple changes We could take it to the next level then all of a sudden we've got something that could be potentially transformative to an entire industry and I'm going to try and show that to you in a little bit and talking about cloud foundry in this mix Using the combination of no JS and cloud foundry I've been able to take my findings with These IOT or physical computing based solutions and then deploy them across multiple client engagements Across multiple contexts across multiple industries by just changing they're taking the same thing taking just a few different code changes some configuration changes pointing it at a different Training data set for Watson services. So that's the cognitive computing part Which I'll get to a little bit more and then it's got an entirely different use case and and you know That combination has been able to open my eyes to what's possible and open other people's eyes into building solutions that again are transformative to how they're they're doing their business and You can see up here. I've got the drone here and I want to focus on particularly an evolution of a use case That had many many many iterations So it all started well before I get into sort of how it started what we're talking about in particular is this kind of an aircraft Whoops my video did actually play The video is supposed to automatically play. There we go So what we're talking about are these kinds of an aircraft. There's actually two different ones there There's literally this one is the one that took off and then another one that I have at home the DJI inspire one DJI phantom These are commercial Off-the-shelf aircraft you can get them they have SDKs so you can build apps to interact with them so that if you want to do programmatic flight if you want to be able to collect information you want to respond to events like capturing media you can do all that and To some people they think that they look at those and the first thing that they think is hey That's a toy or hey, that's just for photographers. That's for videographers But I've been using these as a hobby for probably like in different iterations Four or five years now, and I did start with like that hobbyist photographer perspective because I was like let's get some really cool pictures of like You know the sunset over the ocean but looking down on all these different things like perspectives that you can't get any other way But it's a lot more than just a toy Here's a couple use cases like Some of these are from data that I've captured some of them are from other use cases that I'm Aware of across the web, but of how these are being used to to literally transform Both industries and and how it's being used to collect information and respond to events. So in the top left this is actually a picture of Wikipedia of drone operators using it in emergency response operations in after an earthquake in Southeast Asia I can't remember the specifics of it But I could look it up and tell you guys more if you're interested But they're being used to in search and rescue operations to be able to scan and find people in a More efficient way so you can find people faster But also safer way than sending someone into like a building that's been damaged by an earthquake because you don't want to send Your rescue workers in there and have the building collapse You can send a drone in there look if you find people then you take the steps to go ahead and rescue those people you can outfit them with thermal cameras and You know it's saving lives and it's transforming how people are Saving lives on the top right This is a digital elevation model that I created by scanning a building so doing what's called photogrammetry where you scan it from a lot of different perspectives and then it uses a photogrammetry software to extract information from all those the collection of images to create 3d models and Because all the images are geo-referenced based upon the metadata of the aircraft when it's flying you can actually create geo-referenced 3d models that can be used from everything from construction to You know different kinds of surveying Intelligent agriculture like a lots of different things and I'll get to more about one of those scenarios in a minute This one right here is a photo that I took of a construction site Not because I was working with a construction company because they're building a really big building not far from me And I was like this is perfect for one of my talks, but Even though this is a hypothetical use case where I'm doing this These types of techniques are being employed across industries particularly construction industry mining industries to survey this state of like different projects captured over time and Be able to document things in a way that if they were using manned aircraft would be much more time-consuming and much more expensive And then this last one you might be looking at okay That's the top of a roof and there's like circles on it And this is in particular the use case that I will show you later is using drones to be able to transform how Inspections of physical structures can be done and in particular inspecting grooves for the presence of damage from hail so hail storms So like a really bad storm rolls through using drones and cognitive computing to automatically detect damage to structures so this entire project began at IBM interconnect not this most recent one, but but last year and Actually, let me go ahead and so it started with IBM Interconnect I proposed a topic for one of my sessions to Build a cloud connected drone not doing any of the cognitive computing or anything like that But just to say hey look we have a drone we're capturing data We're pushing that up to the cloud as and just to see if it's possible. Well that session was accepted and then I started working on a content for the keynote and the keynote obviously took precedence and I Was working working working all of a sudden the key notes here the key notes gone. I'm like my sessions tomorrow and I haven't started yet But I was able to pull it off and let me go ahead and jump over and show you the application And then we'll talk about the process of how it was created and how it went through some oh come on Some very rapid iteration cycles that were empowered by this combination of no jazz and cloud foundry to transform the solution itself So I've got a web interface up here. Let me go ahead and turn everything back on that I turned off earlier They told me I'm not allowed to fly this in here Some people want me to fly it in here totally could insurance I'm not so sure about but I'll give you a live demo everything that I'm going to show you will be exactly the same way it would work if it were in the air so What I've got I mentioned earlier that DJI provides an SDK where you can build apps that communicate with the aircraft So here on my iPad home, let's make sure if that is a live feed everyone's not moving all that much So I couldn't tell but yeah, it's moved. So here my iPad I've got an app that I wrote which is communicating with the aircraft So this aircraft it will do 4k video 12 megapixel stills it can capture data pretty high quality and you can grab all the information about it So when you're flying it with its speed altitude to the pitch of the camera all this information And I'm capturing it and if you guys want to see it afterwards like up close I'm more than happy to show all of you so we've got all that information in here and I Love to show this one also when whenever you move the camera the aircraft around and why it's flying the camera is actually stabilized So it's like a rock solid picture as it's flying around so you can get really high quality imaging off of it I'm going to go ahead and so I just hit my button that I have here. It's capturing data so just took a picture is uploading it up to the cloud and It's once it gets up there It's actually going to perform image analysis on it using the Watson visual recognition service And if you're not familiar with Watson visual recognition service, it's a cognitive service on bluemix, which is our IBM's cloud platform That you can do general classifications to saying hey, there's people in here It can do facial recognition But it can also do specialized classification using what's called custom classifiers And that's where you can train the service to look for specific things that you're looking for So you can train it to say if I want to recognize certain patterns specific kinds of information I'll get to a more concrete example in a minute so in that time it captured an image and At this resolution my layout gets a little bit messed up So we can see that there's an image that was captured We can also see the results of the Watson visual recognition service and what I find interesting about this is it Recognized that we're in a building in a hall a conference room Assembly hall so it did a pretty good job actually at assessing where we are right now just by capturing that information And using the cognitive service to classify. What's the content that's in here now? This is kind of just a cool demo to grab people's attention and that's actually intentional I wrote this specifically because just to grab people's attention, but I also want you to think about Using this combination of technology how you can start doing this in more complex and like real-world applications So you could be out there you could be Surveying properties and let's say if you were just serving a bunch properties you're capturing the data You can be using that to classify What's the contents of all these images and build an index on all of that without a single bit of user interaction other than capturing the media and So if you're wondering what this weird box is on the right hand side I don't have a GPS signal if I did there would be actually a map there showing precisely where that image was captured but let's go ahead and look at some Images from when it was actually in the air So here's an aerial photo and you can see that it was tagged with blue sky beach And then it's an aerial image. It's not exactly a beach But there's sand and water in the bottom of the picture. So it's not that far off this was captured in real time and Set up and it's been pushed up for analysis and again if we want to go look at another example So if you stop by the IBM booth and talk to mark Who's got the intelligent Roomba robot that's connected to Watson? That's him But you'll also see that the Watson visual recognition service is able to extract information about people that recognize in there so we can identify people gender age and you know doing all this and Literally, it's just pulling data off of the the aircraft pushing it up to the cloud trigger analysis So let's jump back to my slides and kind of talk about How this evolved over time so I said it begins with a cloud-connected drone and The kicker here is I started to talk about this earlier is I built that the first version in literally 24 hours And so this is where I'm saying driving productivity with no jazz building quickly and iterating quickly You know I had to do it quickly because I had higher priorities It took precedence like the keynote for Interconnect But it was one of those were like this is one of the coolest things and I've been a hobby suzer for years So it intrigued me and I just got sucked in but the first version was very basic There's the app that communicates with the aircraft that talks to cloud foundry the web interface for consuming the data was on cloud Foundry and so it would do an HDD Post to send the metadata with the actual image binary and then it's storing it in cloud Which is a no-sequel database available on bluemix so that it's just taking it. It's it's kind of a Not that advance of a system taking it in you know content repository stuffing data in there we can look at it But over time so after I got through that quick 24-hour cycle and I proved hey, I can do this I figure We can do a lot more than just capturing data and showing it on the web And that evolved into what we see here, which is a bit more complex, but it's much more reliable and Before I elaborate on that, but let's talk about how it works now So what you just saw is there's the app that's running on the iPad, which is plugged in the controller talking to the aircraft We're collecting data that gets saved in Local storage on the iPad so this line here is identifying what's happening on the client app and what's happening up in the cloud and This is using what's called the cloud and sync framework, which is local storage for cloud and which implements the catch-to-be replication protocols so that it can automatically push data up to the cloud and Whenever I capture data the metadata about it is being saved in here The binary for the image is being saved as attachment that object because you might be out flying and you have no connection to the internet So in the case in those cases might other Example actually would have failed miserably because it required a persistent network connection in this case It's capturing the data and whenever you get within range of a network And you know you could be operating off of a cell phone or operating from a SIM card. That's in your iPad Whatever it might be It will sync up to the cloud The cloud and you know you could have multiple instances in your cloud So you could have replication across that But it's being synced up the cloud and cloud and is automatically triggering events for open whisk which is our serverless computing platform So all the initial processing that I was doing in cloud foundry I had moved over to an open whisk action, which is also based on node.js You can write actions in open whisk in swift node.js Python basically whatever you want But because I started node.js on cloud foundry I could easily take that put into an open whisk action So as soon as data is written to the database it kicks off the serverless action Which calls a Watson visual recognition service Which then does the image analysis and then it takes it puts it back into cloud it and again Everything is running off of a web interface and that web interface is actually the exact same one that I created for the first iteration now This is a case where? It's an interesting scenario That captures people's attention and we were doing some interesting things with it, but It didn't stop there actually not not in the slightest after I did this and The funny thing was I did that use case and showed Watson connected drones that actually got picked up by tech crunch and a bunch of people Approach me saying hey, can we use this for like actual real-world scenarios? And this is where I say we can use these The combination of you know node.js and cloud foundry Particularly with cloud foundry on bluemix where we have access to Watson and cognitive services to start transforming industries And I truly mean transforming industries I was approached by a company not gonna be a team with an IBM focused on insurance solutions And in particular they were working with a client that wanted to be no Can we train Watson to detect the presence of one particular use case because you with with particularly with visual recognition services You want to start small and build from there, and that was can we train Watson to build? Or not builds can we train Watson to recognize the presence of hail damage on single roofs? And if you look at this how many people can look at this and point out where there's hail damage Okay, a couple of you but to the untrained eye and if you don't specifically know what you're looking for You don't know what you're looking for. I know that until people explain to me what hail damage actually looks like I didn't know what it was but basically Okay, it's easier to see up there, too There is on the screen, but there's small black dots and indentations like right there. There's a bigger one There's a bunch right around here. There's a couple around here the darker spots are actually areas where hail large hail has struck the single roof and Single roofs there's a layer of essentially asphalt with a granular like particulate on top of it and when the hail impacts the roof it knocks off the particulate leaving these black indentations and It turns out we were able to train Watson to recognize these specific use cases and that's kicked off a bigger project We're saying what else can we train Watson to? To be able to recognize automatically But what's also interesting about this is why we didn't Watson didn't automatically come back and say Oh, there's these certain areas that have in incidences of damage in instead It was well can we train Watson with these small images like crop down images of just hail damage to say is this Damage or is it not damage and then well What if you have a piece of an image that has partial roof and partial like grass and shrubbery and all these things around it? We trained Watson to be able to compare the difference between all these different pieces and then you know Chopped up the images into smaller grids and they were using the smaller grids of images able to determine where within the image there's instance of localized damage and That all started as a very very very simple application where it was like drag and drop an image from the desktop browser Drop it on to the web interface that we have here that uploads it up to cloud foundry That runs graphics map graphics magic libraries on it from the no JS app to chop it up into small images We submit those to Watson visual recognition and we visualize the results and that's literally the interface of what it looked like when we started and that's the hail image that you saw just a minute ago, but Again earlier. I mentioned how cloud foundry can be used to take something and Like no jason cloud foundry you can use it for across many different use cases and it's Basically snowballed from there. So it's been used for detecting rust on cell towers rust on bridges rust on high power lines Areas where there are cracks in like underground piping like for water and sewer It's been used to identify places of damage on rail cars and these are all real-world scenarios that I've actually worked with client teams just to see hey Can we just take that? Retrain Watson take the exact same code everything that we've done iterate on it slightly because some of them needed They had different requirements different use cases But can we take that and can we make it work for all these other scenarios and answers? Yes, and you know how long it took me to go from this one to those other ones each one including trading the data Watson was about four hours. So You think about the impact that that can have on just having people visualize What it can mean for their particular use cases if you know it ended up being very powerful, but It didn't stop there. So if we go back to the insurance use case We started with that and said hey look we can do something really cool here. Oh, yeah so the train the train room was different, but Rust there's two use cases where I use rust and I use my data set for it, which was mine By default the visual recognition service doesn't say this is a rust classifier. This is a hail classifier This is some of the classifier each IBM client Owns their own training data. So they provided me with Images that are positive and negative of the condition they wanted to train on and said hey can we make it recognize this and so you Use the visual recognition service you train the service and the custom classifiers to be able to identify that the different types of content that's within there so That answer okay, so like if I'm an insurance company a insurance company B can't come and like start using all this stuff So I mentioned that like that's not where it stopped because that particular use case We built on top of everything that was done in that first iteration. It was just analyzing those images and turn it into an actual real world Implementation so let's go ahead and take a look at that now if I jump over to my browser here and This is kind of how it evolved from that initial first proof of concept to this is a much more mature version of it and this is actually a Essentially not just proof concept, but a functional prototype that we created with the specific clients in mind Where it's being operated from the perspective of an insurance adjuster where they're going to go in They're going to fly around a property after the insurance the homeowner has called and made a claim saying hey We need to check this out because I had a damage from it from a storm They're going to go they're going to fly the drone around take pictures 50 to 100 Images images of a property and then we're going to analyze those to see what kind of information can be extracted and this is why I say it can be transformative to an industry because Rather than spending half a day of an adjuster physically climbing on top of a roof and you know, which is inherently unsafe and also Time-consuming and a very manual process we can scan a property in 15 minutes use the Watson services to be able to Analyze for damage and then it can also be all kept historically so we can easily reanalyze so as we train our different pieces of The visual recognition service so they come in they upload their images from the inspection and this is where You know if this image might look familiar We're able to come in and we can view all the images of a property and examine for incidents of hail damage And if you look here at these little black spots are the ones that I was talking about But it didn't stop there because this was all based on that evolutionary set of code which is all no JS and On cloud foundry You know it just kept snowballing so let's say well What if we want to add photogrammetry which I mentioned earlier, which is 3d object reconstruction? We've now got 3d point clouds that are generated from those collection of images that are available here in the web interface and You know I cannot just look at a recreation of the structure but I can also come in here because we have all the metadata when these were captured and Start extracting measurements because they're Geo referenced Point clouds that because we have all that information captured and we're using our processes that were set up and all orchestrated through the no JS apps in order to be able to kick off these analysis processes So I'm I know I'm starting to run low on time here. So I'll try and speed it up myself How this works now it all started as that simple no JS app, but that evolved dramatically So we've still got no JS app running in cloud foundry That's where the user is uploading their images those images are then being pushed off There's cloud and instead of using the images as attachments to cloud it in which it was in the very first version Application it's being pushed to cloud object storage, which is essentially an s3 compatible API for Media storage basically whatever kind of file you want that's being pushed up there, but The code for analyzing images I actually took out of cloud foundry because I was thinking like we're doing heavy analysis on a lot of images and put That on to what's called a bare metal server on bluemix and that's instead of a virtualized Instance it's a physical dedicated box that I have provisioned for my account And I think I've got like 64 gigs of memory 3 GPUs because in addition to all the image Breaking up into smaller images, which doesn't require that much computing process We're doing the photogrammetry operations and that's where it's taking all those images so 50 to 100 images and it's starting to generate the geo reference 3d models from those and using that You know We're able to achieve the solution which I just showed you but it all started through that iterative process of just taking an idea Saying hey, wouldn't it be cool if we could do this as a node.js app and it literally just built from there and That node.js code was partially on cloud foundry part of it moved from cloud foundry to an orchestration server this running on bare metal Which is kicking off the wants analysis which is kicking off the photogrammetry and again We're pushing that back into cloud it and then everything is available through a web-based interface Which is still on cloud foundry and I can scale up or down instances. However. I need because it's all cloud foundry based and all the Resources that everything I've just shown you there resources for it here I'll tweet this out. I was Andy Trice on Twitter so that if you want to read more about it You can definitely do so particularly cloud foundry apps on bluemix serverless functions with open-wisk on bluemix bare metal servers Kubernetes clusters Lots in visual recognition that each one of these is actually a link the skylink demo Which is the first one I showed you which is a cloud connected drone I've released that as open source So that if you want to set up your own instance of it see how it works You can absolutely do so just by grabbing it off of github and then the insurance scenario if you'd like to learn more about that And how it works. I've got a video on there and I've got a ton of information so that if anybody wants to learn more I'd love to be able to answer any questions that you have so with that. I think that's my last slide Yes, thank you. My name is Andy Trice and thanks for sticking around