 Alrighty, getting started a little late, that's no problem. We're only a couple minutes behind, but I think we're in good shape here. Thank you all for attending this session. You know, when you do a session on industrial automation and industrial IOT, you wonder how many people at a Linux conference will show up for it. So thanks for coming. I appreciate you being here. My name is Benson, and I will be presenting on the advantages of embedded Linux for industrial automation and IOT. So a quick overview about this session. I'll talk a little bit about who I am and the company I'm with. We'll also define a couple industry terms that are relevant to this session. I'll describe some of the problems that our industry is currently facing and maybe some potential ways to solve those problems. Then I'll present a real live case study of these types of applications being put to real use in solving business problems. Now, if I have enough time, I'm going to wrap it up by providing you guys with a live demonstration of a device that is running in embedded Linux that is used to solve these types of problems and even hopefully a live demo where if you have a smartphone, an iPad, or even your browser, I'm going to let you control the kind of stuff I have up here on the screen. So quickly about me, my name is Benson Hogan. I am with Opto as head of product strategy and marketing for Opto. And for those of you who have not heard of us, which is possibly likely in an audience like this, we are a California based, privately held manufacturer and developer of industrial automation hardware and the software that resides within. We've got over four decades of experience with an installed base in the millions at thousands of locations, at customer locations all over the globe. Most people know us for highly reliable mission critical automation products that are backed by a lifetime guaranteed and all made right here in the US. But what makes us kind of unique in our space is that we've always adopted an engineering philosophy of taking like open IT type technologies and then putting them into like OT systems or operational technology systems to make them more applicable to a broader array of applications. So that's a picture from my drone above our factory that's in Temecula, California. So I'm only about an hour north of here and everything is made right there in the factory and has been since 1990 is when we moved into that facility coming out from Huntington Beach. Now we've had a lot of firsts in the industry. We're kind of known somewhat as an innovator or pioneer of new technologies. We were the first to come out with this idea of plug-in based IO that would go into a single board computer or PCs back in the 70s and 80s. We were a founding member of a technology called OPC which is pretty prevalent today in industrial automation. It's basically a software translation designed to connect different systems together. And we do a lot of course in hardware design and development from Ethernet IO to PACs which are programmable automation controllers. We get to that in a little bit. We've been pretty active in the MTEM space as well, machine to machine. But our most recent innovation is something we call an EPIC or an EPIC. And an EPIC is an acronym for Edge Programmable Industrial Controller. And this is the device where we've elected to use embedded Linux. So the kind of places and industries that our products get placed into are very, very broad from telecommunications to process control to water and wastewater plants. I mean if you wanna see Opto, you've probably seen in some way Opto at work if you've ever seen the Shamu show at SeaWorld or perhaps you've watched the television show American Idol or Dancing with the Stars. It's a film in CBS Studios up in LA. Flown in and out of any major airports across the world or pumped gas into your automobile. You've probably experienced Opto. If you ever made a phone call on your mobile device, Opto 22 Equipments on over 10,000 cell towers across the United States and abroad. If you've ever seen the Bellagio Fountains, that's Opto hard at work making that water push out of those nozzles. If you're at a fat tire beer from New Belgium Brewery, whether it's the brewing process or even the bottling and other elements of manufacturing, you've seen Opto at work. And then even this guy. This guy has a penchant for diving into the very deepest parts of the world's oceans to see what he can find. And indeed he actually built a custom made submersible that in Sydney, Australia to do just that. To break the solo record for going to the deepest part of the world's oceans. That of course is the Marianas Trench. And this guy of course is James Cameron from Titanic, Avatar, Terminator fame. So this is kind of his hobby, but he actually created a movie with National Geographic and did indeed break the record for a solo dive. And of course he counted on Opto to make all that happen. We did all the controls on the submersible. All right, so let's define a couple of terms. Industrial automation and the industrial IOT. The first one here is industrial automation. Pulled this from Wikipedia. It's okay. I'm not gonna read it. Don't expect you to. Instead, I got some help hooking up the speakers for a reason and it's for a quick video. What is industrial automation? On automation. Industrial automation adds intelligence to a process. If you're manufacturing solar cells or if your job is to control the quality of your city's drinking water or even if you are brewing beer, you're using a process to do it. Of course you can make great beer by hand, but if you wanna make a lot of beer or produce beer of the same quality time after time, you'll probably need to automate the process. Suppose you have a large tank of water you draw from. You could have a person fill up the tank when it gets too low, but a person might get distracted and forget to turn the water off. Or you could automate the process by installing a tank level sensor and a pump plus a controller that monitors the sensor and turns the pump on or off. The controller responds in milliseconds and works 24 hours a day. Automation is great when you need to do something a human can't like lift heavy things or work in extreme temperatures or monitor a huge bank of sensors without rest. At Opto22, we make automation hardware and software to help industries around the globe produce and deliver many of the products we use today from t-shirts to smartphones. Opto22, we make automation simple. Okay, so that's just a nice brief overview of what automation is about and the tools of the trade are these types of products here. You heard the video mention controller. Well, there's PLCs. That's a very common in the industrial automation arena. PACs, which are kind of like a super PLC to some degree. And in some more rare cases, industrial PCs. These devices were never really designed for the future of communications and IIOT. So let's go into IIOT. What is the definition? Again, there's one from Wikipedia. I had my own version. I did a TEDx talk on IIOT back in 2014 and distilled it down to it's the ability of things to sense, communicate, and control. Now, that's IIOT. When we start talking about IIOT, things are a little bit different. So I've got a quick video here. It's about 16 minutes and then we'll get an, I'm kidding, I'm kidding. I'm not gonna make you guys watch a 16 minute video. Let's just move on and go with the Wikipedia definition. And in this case, it's very similar to IIOT, the difference being now we're talking about sensors and instruments and devices and so on connected to industrial applications. And the reason for this is to collect, exchange, and analyze that data to better business processes, maybe come up with new business ideas, whatever the case may be. So that's also pulled from Wikipedia as is this. And I really like this diagram because it does a good job of kind of compartmentalizing or saying what the pieces and parts of a given IIOT solution is. Down at the bottom, you've got your hardware, that's your connection to devices, machines, sensors, and so on. Then you've got that communications layer, Wi-Fi, ethernet, cellular, whatever it might be. Then a services layer. And this is where embedded Linux comes in because there's a lot of applications that may be required to do certain things to the data. And if I can find a way of running lots of different software on there, embedded Linux is certainly a way to do that. The last one is the content layer. This is the visualization, how I can actually interact with the process. Now this as well is something to also refer to as HMI. I'll get to all of this whole model here in just a bit. Now, some of the industrial internet of things, trade tools are things like gateways or sensor systems, maybe firewalls, things like that. And then of course there's a big software aspect to the industrial IoT. It was represented by tools like Node-RED, which is part of the Linux Foundation, as well as the Ignition software, again some open source software I'll talk about a little bit later. And of course cloud platforms. If you have anything to do with IoT, you've probably heard about the various cloud platforms for Microsoft, IBM, Amazon, and so on. So what problem are we trying to solve? Well, the big thing for all of us is that the future is here. We're hearing about all these wonderful new software tools and capabilities like machine learning and artificial intelligence and cloud computing and wow, it's all great. We can just go and apply all this stuff and transform our business. Yeah, not so fast. Turns out that in our industry in particular, there's a lot of roadblocks in the way. And a lot of it has to do with legacy devices that are in the field that are continuing to do their job, proprietary networking, proprietary protocols, all kinds of massive security issues. And these are just all roadblocks and getting in the way of doing something pretty exciting. So what our industry has kind of been forced to do is to try to use some of those proprietary technologies and then couple them with other technologies to try to cobble together a solution. And frankly, this doesn't work. There's not only because of all the proprietary protocols and perhaps even proprietary real time operating systems you're dealing with, but it takes a lot of multi-domain expertise from both the IT side and the OT side. It's a security nightmare because every place where those two make a connection is another attack vector. And industrial automation applications, that's just not, you can't do that. And then the biggest problem of course is that this just can't scale and it's incredibly difficult to manage. So how do you take that kind of a problem and solve it? Well, what we did is we came down here, at this conference, it was in San Diego in 2016, our engineering team came down here and we were looking for a path forward and we decided we were going to embrace embedded Linux. So that was three years ago and we built this product as a result of it. And we did have a product previous to this as well that was based on embedded Linux, but this time we did it quite differently and did it from the ground up. I'll get back to that, but when you start talking about what an EPIC can do, it can essentially collapse or condense all of those functions and features that are up on the top row into a single manageable device. That makes it a lot better for deployment, for security and all those things I described earlier. But I also said that you can't just rip and replace, right? So you have to kind of make use of existing technology where it might exist. So Groove EPIC also, well, I should say EPICS as well, can front end existing devices in the field and start adding all that functionality but still making it manageable and secure. So that's kind of the key piece where those things fit. We've got another problem as well that we need to solve and that's this notion that applications are so tightly coupled to devices. This happens in IT to a smaller degree, but in OT in our world, this is prevalent and it is a mess. Not only is it a pole response mechanism and there's a lot of unencrypted traffic flowing all over a network and you gotta run to IT to try to protect these networks, it's also just riddled with security loopholes, kind of like what we saw earlier. This idea that I've got to open ports on devices or open firewalls, it's just way too difficult and very, very tough to manage. So we think there's a better way. One of the things that's been kind of on the scene lately, which is odd because it's actually a 21 year old technology, is MQTT, this notion of a published subscribe model. It does two things. One, it allows you to push data out on an outbound connection to a broker and two, all of your access control and accounts and security is done at one place at the broker. So this creates an architecture where I can move a lot of data around in a very secure way all over a Pub sub-model. Now one thing about MQTT, a lot of people think, oh yeah, Facebook, Amazon, Alexa, whatever. MQTT was actually created, was invented to solve an industrial problem for Phillips 66 back in 1999. So this was created to solve industrial problems, not used as a chat mechanism on Facebook. All right, so let's go to a case study. This is a situation where, I call it the tail of two turbines, it's actually tail of 2,000 turbines. The idea is that we've seen, if you've driven through Palm Springs or up in the North Bay area, you've probably driven and seen hills full of these turbines and they've been out there for a long time and they were created to take wind energy and convert it to electrical energy. And there's a control system on there that makes sure the turbine operates properly and so on. And basically, if the wind blew, you put energy on the grid and people purchase these or own these towers and operate these towers for one reason. They get paid to put that energy on the grid, which is a great thing. So they'll get whatever the price of electricity is that they're paying, you generate the electricity and you get paid. Pretty cool, until it became a problem. So California, it turns out, happens to love renewable energy. So they legislated that 33% of our energy will come from renewables from starting next year. This was done several years ago. So starting next year. Well, that seems like it's a boon, right? What kind of was? More turbines went up, more solar panels. And that was great until it wasn't. It turns out that all renewable energy is variable in nature. If the sunshine and the winds blows, you have energy. If it doesn't, you don't. And that becomes a big problem because there's an organization called the Cal ISO that manages our grid. And when it manages a grid, it has to take in a lot of information about the future and how much demand there'll be, how much supply is available. And they've got to create a balance between all these, because you can't just store the energy, the excess. You have to use the energy at the time it's been produced. So the Cal ISO has to pull a bunch of levers, essentially, to figure out what resources to pull on, pull down. But they can't do that for renewables. So they had to come up with a different method since they couldn't just take a renewable off of the grid when there was, say, too much energy or dispatch to produce when there wasn't enough. They created this thing called the spot price. If they need energy on the grid, spot price goes way up. Everybody's making money. When there's too much energy on the grid and not enough demand, the inverse is true. And so what they do is they essentially lever the market based on this spot price. And it is determined by the Cal ISO. So let's take a look at a scenario, and this is the business problem. What ended up happening is in some cases, electrical supply exceeded demand. So immediately they got to figure out how to dispatch or turn off loads. And of course they can't do that to a turbine. And if the turbine operator is generating electricity when they don't want it, they'll move that price to zero. Now you're operating a turbine at cost, at your cost, and losing money. But what's worse is there's times when they need to do this and this is for real. They'll move that to a negative number. And if you're generating electricity when you shouldn't be and they don't want it, you write them a check. So this became a big problem for them. So they had to come up with a way of trying to solve this, enter embedded Linux and devices that can make use of a lot of different software to make this happen. They still can do the controls, but now they have technology and capabilities to reach out and get the real-time spot price directly from the turbine and make a decision whether or not it should run all on its own without having to rely on any other kind of system. It can also go out and get its own weather. It can get all kinds of information to control but it can also move all its operational data up to cloud platforms. So in essence what you get is an autonomous turbine that suddenly is able to generate electricity only when it's profitable and is also sharing all its data with external systems so you can do predictive maintenance and analysis. And of course you can take an old turbine and make it new again. So these kinds of technologies with embedded Linux and this kind of industrial hardware has real value when applied to not only new systems but also the legacy systems. So how'd they do it? Yep, as you might have guessed. They use products from Opto to pull this off and essentially the system architecture looks something like this where you've got at the bottom a lot of network connectivity to existing PLCs or to IO systems and then there in the center where the Epic resides is all that software that can be run on an embedded Linux platform. Lots of different options based on what it is you wanna do and then securely move the data up to IT or to the cloud or to devices for managing, making sure everything's operating properly. So I get back to that modular architecture we saw from Wikipedia and indeed these types of devices cover every one of these layers. The physical interface, the network layer to communicate, the services layer to do something with the data or go get other data and then of course the content layer to provide visualization. So where does all this kind of stuff run? Well for us we ended up building this controller with the quad core ARM Cortex CPU and we chose to use Linux, embedded Linux with preempt RT real time extensions for the real time control capabilities of the system. And we put some RAM in there, some solid state disk drive gave it a really wide temperature range because these are industrial processors but the other thing there, the one in the middle Power Fail Safe File System. If you ever played around with the Raspberry Pi or Arduino or some other EMMC based device you know that when you yank power to that thing you cross your fingers and hope it boots back up. You can't do that with industrial devices. So we use Reliance Nitro's Fail Safe File System on our embedded Linux and we do do Yocto, we're part of the Yocto project for building this Linux. Then we put web browser technology and web server so we can display HMI screens, we provide ways of managing the device directly from the front screen or from a PC or whatever all over a network. These are some of the screens that are applicable to the turbine application and you guys have time permitting we'll get a chance to take a look at some of that in a moment. Now like I said, a device that is industrial and meant to connect to the real world it's got to have a means to do so and that's where IO comes into play, input and output modules for tying to sensors, to devices, to circuits, to pumps, you name it. And then it all kind of comes together on various sizes of chassis so you can, it's all modular and designed to just put in the pieces you want for the modules you need for a given application. And of course it's all UL tested and approved for virtually any kind of application. Okay, so quickly about the software that's where the secret sauce is in my opinion. There's lots of software there to do various tasks including managing the device, building operator interfaces, creating the real time control strategies to operate a machine or a process line, build your UIs and connect to legacy PLCs even connect to the cloud. So there's a lot of pieces and parts in there very quickly I'll kind of give you a sense of what the power of embedded Linux gave us. The first was to develop an application on Node.js using JavaScript of course and TypeScript and put that into our Yachto build and this is how we manage the device itself. So very touch screen interface. And then the second part is the control. We have to provide real time control capabilities because this ends up in operating machines and so on. So this is where the real time extensions come into play. Lots of different options there from flow charting to traditional ladder which is common in PLCs. But the other thing that's unique about this type of product is we give the user shell access. You wanna write your own application in Python or C sharp or whatever, Java, whatever. You're welcome to do so and run it on this industrial hardware. The third part of course is what's called the HMI or the user interface. Again, designed from a browser, drag and drop stuff onto a pallet and build your screens. And then the next one's kind of cool. We all have probably heard about Node-RED and IoT tool developed by IBM. This tool is really powerful and gives you a lot of capabilities and it runs right on the device. Again, because it has embedded Linux. So when I started thinking about the things I can do with Node-RED running on this industrial device, I can go out and get weather data, I can get that Cal ISO spot price and pull it back in. I can even take all of the operational data that the system is generating and move it off somewhere else. In this case, it's moving it to Amazon's RDS, a relational database service. Or it could be IBM Watson, Microsoft Azure, and it can do all of these simultaneously. So you don't have to pick a cloud platform. You just spin up the stuff that you need all within the Node-RED environment. Finally, the last one is that we work with a company in Northern California called Inductive Automation. They also embrace open source and develop software that runs on embedded Linux devices. And this is it, and it provides that drivers and functionality to existing PLCs that are out on plant floors, out on remote sites, wherever they might be deployed. So it's gonna cover all of those ways of communicating to legacy systems. Okay, with that, I've actually got a little bit of time. I got about five minutes for some live demos. So what I'm gonna do, or attempt to do, is I've got my phone right here, and I'm going to go up and get it connected to my laptop. Hang on a second, make sure I'm on the right network. Yep, and then let's make sure this guy is on the right network. And let's go here, six. All right, and there we go. Okay, so what I've got up here, let's move on to the next slide. I've got this Groove Epic, this embedded Linux device here, and it's got a little router on the back, a little cradle point router. It's connected to the internet over a Verizon network, just a Verizon SIM in there on an IoT data plan. I've also got a local little Wi-Fi here using the same cradle point so I can use my device here to communicate to that system. So I'm gonna pull that out. But this is also connected through those I-O modules I spoke of to a turbine, little model turbine from Lionel trains, and also a stack light here. So, and then it has some other things you guys are gonna get to control that. So what this does is this simulates the turbine application. I can turn it on and off that turbine based on the spot price. I'm moving all the data to Amazon, RDS, I'm moving it to cloud platforms. I'm also happened to be going to get the Netflix stock, and if it's high or low, I can turn on the lights. So we'll get to that in just a sec. Okay, so I'm gonna pull that up on my, now what I'm gonna do is I'm gonna attempt to take this over to that screen. Oh, there we go. Okay. And then I'm gonna go to my application called GrooVue, and click on it there. And I'm gonna click on my local device. You probably saw my home on there, my entire home is automated with this stuff as well. So now what I'm gonna do is actually bring this up on my device so you guys can see it. And this is again kind of that one, that content layer on that IIoT model. So it's gonna be a visualization for what's going on in the device itself, and it also allow me to control, turn things on and off, and so on. So we're just, I've got a very contentious network in here, so it's taking a little longer than it usually does. But see, you can see that I've got the temperature up there, it says 74 degrees. Okay, I've got a little sensor right here. It's called an ICTD, it's Integrated Temperature Device. And you see as I grab it, there it goes up. So we're pulling data from the real world and pulling it into the system. But I think what you guys wanted to see, or what I've been talking about is the turbine. So here's the turbine control, and I can see that I can run the turbine at $99. If that's what I think I can make money. The current spot price of electricity, however, is about 26. So let's say, you know what, I think I can make money if the spot price is 25. And as soon as I do that, and I go into auto mode, I can see my turbine starts to run. You guys are gonna get a chance to change the price later if you want. So now what I'm doing is I'm pulling back information from the cloud, from the Cal ISO API call, and I'm making a control decision on whether to run it or not. And then I can see the market price, I can see my other information all from my system. Simultaneously, it's moving all the data up to, as I said, SQL Server up to the cloud. But one other thing it's doing is really cool. It's taking all of its data, real time data, and it's publishing it to that MQTT broker. And the reason for that is because I may wanna be able, I may wanna see that data somewhere else. So I can have an application subscribe to that data so that let's say you could see it on your phones. Which is exactly what I'm gonna do next. So if you have your phone, if you wanna play along, you can scan that barcode. What that's gonna do is load up an application in your browser. And it's gonna connect you to some software that's running up on AWS. And it's gonna connect to the broker and pull information back. So if I come in here, I'm gonna actually switch to that as well. Let's see if I can get there. Okay. I just covered up your thing, so I'm gonna move this guy over to here so I don't move in case you haven't done it. Now what I'm doing here is I'm gonna go ahead and switch to that one that's running on the outside. And that's a live demo that's out there and it's running all the time. So when I connect to this guy, I'm gonna get a different color screen because it's coming from a different server. Somebody just turned off the turbine. Congrats. You figured it out. So or turn it back on. Well done. Now scroll down a little bit and you're also gonna see where you can see the stock price. I'm connecting on such a slow network that it's gonna take a while. But you guys can probably get there more quickly. Yep, there you go. Somebody just press the button called stock check or hitting the lights. Isn't that amazing? I mean, if you think about it, you're operating this system from probably the wifi here or over your cellular network because you're connected to a cloud application as part of the IOT. It's publishing data or subscribing to data from the broker and then this guy running embedded Linux and using these open source technologies from Node-RED to MQTT to the OS itself as making all these communications happen. And it's all 100% secure. You had to log in, you're doing it over an HTTPS connection. There's no open ports on this device. So I have no chance of getting in there even through the Verizon network. Even if I were to hack that, I couldn't get into this device because it has no ports open. Everything is on an outbound basis. So it becomes very simple and very straightforward to start building out these architectures both in industrial automation and IOT using open source tools on an open source Linux embedded Linux operating system. So if you guys have some questions, I've got about four minutes left of questions and I'm also given away these 100% authentic Yeti knockoffs cups. So I've got about six of those or you can welcome to come up after and grab one if you like. Question in the back. Yes, I'm very familiar with them. Yeah, that was, we are working with those guys. We're limited by what we can say as we are under NDA, but yes, indeed. We are working with those groups for sure. Good question. So that's a question we get often. Obviously, as soon as it comes out and I just saw Nick's email and or tweet this morning that they went to one of the second beta, immediately we'll be putting it into our firmware. And we get the question about scalability a lot. We are running Node-RED here. So it's not running in the cloud meant to do big widespread stuff. It's kind of contained to the system here. So we've had a lot of success with it and we have a lot of customers that have it in production now despite that it's a still a point to release. But a good question nonetheless. We are running, we have a long history with Motorola which then turned into FreeScale which is so i.mx6. So today it's a monolithic update. We update the entire system. The system is designed, it has actually three copies of the OS on there because if you do run into trouble we have a reset to defaults which mirrors the image back over. One for during the update and then of course the operating partition. So, but today it's a piece of firmware. It's basically you're re-uploading new firmware to the device. That gives you the new functionality. Yeah, absolutely. Absolutely. We, you're above my pay grade but the intent is that those threads get real-time priorities for the control engines if they're running and you select what you want to run and not run. For example, you don't have to run Node-RED. Therefore it's getting no thread and no priority. Anything that is running is run under the traditional OS if it's a real-time task and there are three of them on there depending on what you're doing those get a real-time thread priority. Two of those real-time control engines you can only run one at a time. You can't run two. So, but there's a IO scanner in there as well which also has a real-time thread. Yep, back in the... Using what? We're not, we're not. We've elected not to go container on this. That could change in the future but our first iteration was to not use a container. We're using Yachto. So we have the build. There's another important consideration there. Because of the way that we're using Yachto we actually manage our own repository. So any packages that ever could be put on this thing have to come from our repository. Everything is signed. And the idea is simply that it's an industrial device. We wouldn't want somebody to put rogue firmware on there. So it has to come from us. It has come from our repos. These are great questions. So there are some security aspects of IDOT MX6. It's not coming through IDOT MX6 because there's not a lot there to my knowledge. So there's a number of different other ones that we're doing through that interface. It's at the OS level down. I'm not sure which ones that we're using right now though. That's a good question. So like secure boot and all that kind of stuff. Right, right. So I think that's, we're using those elements or at least some of them. But again, a little above my pay grade for sure. A good question. And if you're, I'm happy to take your question and get more details on those answers directly from the developer. Any other questions? Great, well I got some mugs. I've also got one of my colleagues here. He runs our developer.optot22.com website, a specifics website just for developers that has all the libraries, all the REST API, everything you could think of to develop for this platform. Key manages. And he's here joining me. He's actually at the conference attending many of the sessions. And we have some brochures and white papers and whatnot also in the back of the room. And I'll have some up here. But thank you guys so much for your attention and appreciate you coming to this session. Have a great rest of the conference. Thanks.