 I'm Chad Furman. I'm a product manager on the Ansible side, but for the context for this conversation probably more interesting to you is I was a customer for a very long time. I used Ansible. I installed Red Hat Linux for the first time in 1997, so I've been doing this a very, very long time. But yeah, I came from a oil and gas company, so I know entirely too much about how things work in an industrial setting. So I think this will be a fun conversation. For those who don't know, the microphone doesn't, or does it actually project? Okay, they told us it wasn't projecting in here, but it's for the online stream. That's why we're taking turns on the microphone, even though you can't hear it. Okay, my name is Adam Miller. I am one of these software engineers in the Ansible organization, and lately I have been Chad's partner in all things edge and specifically industrial edge more recently. Okay, agenda. We're going to talk about all the things and talk about challenges. Get into what we've actually been working on in the Ansible space. It's all very community driven, of course, because that's where we start from the open source side. And then get into really interesting things we've done with common industrial protocols and actual real world applications, because I think that's what really matters is how do people actually use this stuff? So edge computing is a huge, huge pain if anybody's ever thought about it. A lot of people have been in the data center for a very long time, but thinking about how do you scale and manage and have the interoperability and consistency of all of these different things and tie them together and how do you make a thousand devices work the same? Or how do you push an application to tens of thousands of devices and manage them and know what their IP addresses are and what they're supposed to do or what they think they're doing and the telemetry that all comes back from that. I know there are some people here that I've worked with for many years on this and I don't think anybody's actually solved it yet, but we're getting there. So who here knows, I like to make this interactive, so what does industrial edge mean to you? What's that? Excellent. Anyone else? Yep, absolutely. Anyone else? Fun. Alright, cool. Literally anything that makes things. So agriculture, factories, cars, electronics, anything that could have robots or machinery and all of these things that are, if you've been in the IT space for very long, we always think about VMs and containers, but when you work with an industrial company, a container is something that ships things in. It is not something that they use for applications. So it's a different conversation and I have that conversation often like, yeah, containers. They're like, yeah, we're running Windows and VMware and we don't know what a container is, sorry. So it's fun. So what we've been working on, Adam and myself, is working with different companies on how do you bring some of the best practices of IT into the OT space, so OT being operational technology and that's what a lot of them refer to it as. So things like SCADA systems that control robots and SCADA systems that control roller coasters because yes, they actually do the same things from the same companies, which is really interesting because at first, whenever we started going down this, me coming from a manufacturing space, I was like, oh wait, yeah, you probably use the same motors to control a roller coaster that you use to control a robot in a car manufacturing plant. Kind of cool stuff, but they don't know anything about IT in this space. Like to them, IT is the guy that they call when their stuff breaks and it is a ticket and it is a waiting period. And now that they're starting to kind of talk to the IT people instead of them being the enemy, they're now becoming friends and starting to think about things like, oh, so automation isn't just about factory plant automation. Automation is actually about IT things too because if you say automation in the OT space to them, it is process control automation, not IT automation. I think I kind of talked about this a bit. So here are all the different things that you see a lot in this space. From devices and servers to sensors that are detecting things in the air. That was one of the big ones where I came from. They're always looking at particulates that went in the air because there are laws about what you put in the air when you're an oil and gas company, sometimes. But it actually gets all the way to like substations from electric. And of course there are central servers that are there that generally are ran by IT, but sometimes they're actually run by the OT people. And to them, it's just the servers. It's not something they really know much about. It's just something they have to have to run all of the other things that they need. So unfortunately, we couldn't bring this with us because it would have been really expensive. But at Summit, we had some really, really cool booths with different companies we had been working with. So we actually had the Schneider Electric water. This is how they do wastewater management in a lot of places. We had it where you could change the valve knobs, where you would change the water displacement in the tanks. And then you could actually rip out a server and continue to work. And then you could plug the server back in and it would reprogram the server and continue to work. So it was kind of cool to actually show real world interactions in this space because a lot of times we're like, hey, here's a computer and we put a VM and a container on it and it runs an app. But it's actually cool to see stuff actually doing things. Are we doing this whole day? Yeah. You can take this one. All right. My turn to talk. All right. So one of the things that we've been doing is market research and talking to different customers and potential customers and just finding out where their problem spaces are. And we've identified a handful of business challenges. So this being kind of a summary of the business challenges that we're seeing out there from a plant manager kind of dealing in this actual space, this is their day-to-day. So the ITOT convergence is still happening. And this is kind of almost like a culture shift that we're seeing in a lot of the industrial manufacturing industrial edge, transit, logistics, these kinds of markets where it's an adaptation of technology that has existed in the IT space for a while that we have to modify and evolve and develop new capabilities for to be able to address the OT space. Largely because the devices there are different. The protocols are different. The types of networks they talk on aren't IP based. There's a lot of legacy technology there that we have to interface with. You find different challenges for things like data gathering when your networking is either intermittent, bad, high latency, or low throughput. You find yourself having to design or architect the way that you deal with what you would traditionally do in a data center to adapt to these environments. So the Purdue model, which if anybody in manufacturing at all would be very intimately aware of this, effectively this is what all of the industrial, I don't know if this is like a requirement thing or if everybody does this, but the industrial manufacturing follows this model and this is a security model for networking and every zone may only talk to the zones at borders. Nothing can traverse from, like plant level, near-edge to operation level, nothing can actually go more than one hop. And this becomes problematic. You can only go from the bottom up. Oh, okay, yeah, sorry. And you can only go from the bottom up so you can't reach back. So this becomes problematic because you have to deal with your ingress versus egress and you have to figure out how to traverse these things to accomplish automation in a way that you might not have been able to otherwise or in a way that you would have been able to otherwise. It just kind of depends. So this is a space that from, I guess, a traditional IT perspective is very, very different or the variables in play are different, more limiting. So we have had to find different methods to accomplish this. I can rant about the Purdue model for like two hours if anybody wants to rant. Yeah, so this is kind of a marketing slide because Edge is hot right now as a topic. Gartner loves to talk about it. Forrester loves to talk about it. It makes its way around. It's not new, though. We've had retail stores for as long as any of us have been alive. Those of them that have compute resources there, they were doing edge computing. This is just kind of something that has become, I guess, more relevant in our space specifically from a technology perspective, from a software perspective, from an open source perspective because we are bridging the IT to the OT and IT is typically where open source software and the things that we develop are able to be addressing those problem spaces. Now we're extending it into other problem spaces. So, yeah, you do this. Yeah, you do this. So, who all here uses Ansible or knows what Ansible is? Hence, okay, some people don't know what Ansible is. And I asked that question so I could tell you what it is if you don't know what it is. So Ansible, actually, no, I'll give you that slide because that's the next one. So we built a platform around automation and what we've had to do with this, so actually, I'll let you do your slide and then I'll go back to that slide. Yeah, we did it backwards. Bad lining on the slides. All right. Ansible is an automation tool. It is also an automation language. The idea of it is for it to be simple, composable, human readable, and if at all possible, idempotent. Idempotency allows you to only make state change when state change is required, such that you can repeat and rerun that automation over and over and over again, and only inflict change upon any system or systems when required. The idea from the perspective of what we're building and what we have built in the open as an open source project is for it to be the automation language or the lingua franca of automation. The goal is for it to be universal. You should be able to learn this skill and then take it into network automation, security operations automation, industrial OT automation, IT automation, et cetera. We are doing everything we can to adapt the technology underneath and design it in such a way that it is pluggable, composable, and reusable, such that this automation skill set and tools and technology is adaptable and can be used throughout. I haven't gotten there. One of the things that we did here in light of bringing the automation technology of Ansible into the industrial edge is we implemented a connection plug-in for Ansible to be able to talk to programmable logic controllers over the common industrial protocol. These devices are not connected to an IP network. They are not devices you can traditionally reach out and get to. However, they are connected to a backplane. That backplane does interface with a device that has an IP address and does speak IP. We wrote software that has the ability to reach out to that controller. That controller then translates to the backplane and then we can traverse those SIP networks and automate programmable logic controllers. For those who are not familiar, a programmable logic controller, or a PLC, is the thing that controls the robot arms or controls conveyor belts or controls different elements of the actual factory floor. The thing that produces widgets and boxes and vehicle parts and the bits inside of our laptops and our cell phones. That can be interfaced using the same language that you can use to automate your Linux system via the Ansible Automation Language. Now I'll let Chad rant poetically about the automation platform. I'll add the PLC thing there. Where I came from, the way a PLC was managed before and think about how many motors it takes to manage a conveyor belt. The only way to go and program them was someone walked around a manufacturing site with a serial cable and plugged them into each device and figured correctly. Now you can just use Ansible to do that, which is kind of fantastic. Really short, sweet. We've created a platform around Ansible. Ansible is a command line language. There's also a platform where we built it and made a lot of the things. As much as I love doing things on the command line, if I'm going to have to do it against 10,000 devices, I don't want to do that on the command line. It just is not a good way to do things. We've created a platform to do all the things that you hate doing, like potential management, integrations, all of the self-service things, and you just go push the rocket to go run against those 10,000 devices. The most important part about this, though, is the execution plane at the bottom. Remember that Purdue model? You use the execution plate with a network mesh to be able to get into those different parts of the environment, so now you don't have to worry about 5,000 firewall rules get to every device, one firewall rule, one particular network segment. Give me, give me, give me, give me. Pictures. Okay, so the one thing that I want to touch on that Chad just said was about the automation mesh. So, automation mesh has either a bidirectional or unidirectional communication method. So, if you needed to function in an environment in an industrial space that does follow the Purdue model, you can set up to where certain zones only allow ingress traffic. So, by doing that that will put your automation mesh node in that space into effectively a pull versus a push. So everything from the Automation Platform still looks as though it's a pull, but everything will do store and forward intelligently based on the routing algorithm on down. And it's just Dijkstra. We don't have to be fancy about it. It's, you know, this is a tried and true algorithm. It does what it does and this allows us to deal with that space. So, as Chad alluded to before, all configuration used to be done manually with these PLCs. We had to just kind of trust that it was fine because there was no validation. We couldn't really do CI on it. It was just a thing. So over here on the, let's see, right hand side is a it's a Pelican case and inside this Pelican case is a Rockwell, Alan Bradley Rockwell PLC, a control node, an emergency stop button, a drive that actually will spin that little puck with the Alan Bradwell logo on it. And then that little blue thing down here that is actually an LED. So it could be any kind of a flashlight. And we set up a, again, we couldn't travel with this because we just unfortunately couldn't. But the automation to accomplish that was all written with Ansible as a play, Ansible playbook. So for those who don't know Ansible, the automation recipe, if you will, is called a playbook. All of this is automated from scratch. The second you plug it in, everything is completely programmed and it will send that drive spinning, it will stop the drive, it will spin up the it will light up the thing, the LED and turn it off. And that was something that we did using this open source Ansible collection that we wrote called community.CIP standing for Common Industrial Protocol. And we have the ability to read and write tags. So in a PLC language it's effectively how you set or unset particular boolean flags to inflict change upon a running system in a programmable logic controller and then we can do audit and validation of information. So this allows us to do our audit trail and do testing. So this is effectively what an Ansible playbook will look like to control, command and control those things. And this is, it's just YAML for those who don't know Ansible. It's just YAML. So everything that you have set up, you have a host and after the host keyword is your host pattern. So the pattern will be the set of devices you talk to and there's a thing called an inventory. The inventory devices are individually atomized either statically or dynamically by that pattern. We want to define whether or not we're going to privilege escalation. So become is to become a more privileged user as part of the privilege escalation. We have a set of tasks. Each of them has a name. I have an indention error. Please ignore that. We're going to call the namespace module of insure tags and this is going to be the thing that takes care of the task operation. So we have a name of the tag and a value and again that is an item potent operation. It will only make change when required. We want to gather fact data. We want to inspect the current state of the running system so that we can then validate it later if we would like and then also we're going to verify our firmware version and I just okay yeah, tag value for the sine wave. That was what that one was. So that is a sine wave generator which was part of a demo that we had that would actually start to articulate a sine wave generator. So we have a human machine interface. Okay. And that is kind of the goal is for it to be human readable, easily composable. And this down here this is a templated variable. This is part of some of the string interpolation magic of the Ansible Automation Language that there is plenty of wonderful documentation out there if anyone wants to check that out. Okay. So beyond that we also have for doing device edge. So if you have compute nodes at these environments or in these cabinets that are co-hosted with your program of logic controllers Red Hat has a operating system that is based on a technology called RPMOS tree. It is immutable. It is composable. It is easily distributed. It is a delta application of an update and it is atomic in operation and we created some collections to be able to automate the life cycle management of that entity. So OS build is how you actually build your operating systems. They are bespoke. They are artisanally crafted by you and your team. You will effectively pass in the blueprint or the desired outcome of your image. And then from there we can do deployment life cycle management. We can, I'm sorry, keep saying life cycle management like that means anything. We'll set up an RPMOS tree repository. We will actually automate the distributed update of systems. So to Chad's comment before about tens of thousands of hosts we can do this at scale of tens of thousands of hosts. We can do the full roll out. We can do the update system. And because of the nature of OS tree we can do it in these fractured environments with bad network with low fault toleration, things like that because I like to pose this question. For those who run RPM based distros or any traditional package manager, DPKG I don't know, package build pick one, if you're updating your kernel and it's in the process of rebuilding your in at RD and you kick the power out from underneath the server, will it boot? It's shroding our server, we don't know. We don't know until we try. Whereas with this type of a system that cannot happen because it's either all or nothing and if there is a failed attempt at a boot it will roll back to a previously known good state. And this automation that we've created allows you to do that. And then the next phase of that is micro shift. So for those who are familiar with Kubernetes who want to do Kubernetes at the edge there is a miniscule version of OpenShift which is a Kubernetes distribution called micro shift. And then we have the, again, the ability to run application payloads out on those devices build the images based on RPMO tree based technology for those do a full lifecycle management of that as well. A lot of stuff in one slide I apologize. Yes, what? Oh yeah, with these we can do Fedora, CentOS and REL with these. So we can do a CentOS stream nine, eight, eight eight and nine any current non-end of life Fedora and red enterprise Linux eight and nine. Yes. And we full CI test all three of those across each of their releases. So if something breaks, please let us know we'll add a regression test and make sure we don't break it again. Yeah, Chad. Oh no wait, haha. I already talked through this without diagrams up. I'm very sorry. Very quickly this is basically the flow diagram of what these, we have 10 minutes. This is the last slide. We can do this, I believe in us. This is the flow diagram of what the automation content that we created does and allows you to have your image builder host your Relfor edge image gets built there from there we have our device blueprint which will define the curated content that you would like to put into it that will generate it we will extract the contents to then serve up for your update system and then we can roll that out to your device edge images and we also have a boot ISO that can be downloaded and you can flash devices with it, that kind of thing. We actually also have a tool set to be able to do credential injection, post creation of the ISO so that way you can, depending on what your chain of trust or your processes are, those kinds of things. And then for the open the micro shift one the only thing that you add there is you have day two operations of doing the application deployment and life cycle management micro shift one he added fancy words did I? I haven't slept, I arrived in this beautiful town at half past midnight last night so bear with me I lost my train of thought Chad, it's your turn that's a sales slide, we don't want that questions because I'd much rather listen to you guys talk than me and Adam talk we talk a lot is he, is he being serious? no, God no Chad, Adam, Adam, Chad Adam Williamson is a fantastic human and a great A heckler I'm waiting for Peter, he hasn't heckled yet at all no, no, nothing, okay that's good no beer for you hey yes Peter, he needs fist mode please put an issue in for that and then I'll send it to Peter yeah, so the fifth excellent Peter's working on this it sounds like at the end of the day we just use the OS build bits behind the scenes yeah, so the thing that we build is the automation and I'm sure is that it's easily consumable and repeatable if you need core system technology added we will glad to refer you to our friend and colleague Peter in the back there Tosha we have a bi-weekly believe me, we talk okay, I'll repeat them yes, sure Tosha do we have a place to plug in the CI of the image that is built that's a him question yeah, I mean so literally the build that comes out the way that we do it is we just provide you an example playbook of how to do it and you just put your inputs at any point in that phased process you can just pause, extract the artifact and then send it to a pipeline and you can actually do that in your ansible playbooks you can have the CI validation occur and then allow the playbook to continue to form your CD for you if you want or you can then pipe that over to another system for that if you want to use Argo or whatever is hot this weekend all of our CI is in the GitHub yeah, I mean all of our CI is in the GitHub I don't know if it's particularly the best way to do CI the way that we built CI is integrated into the open platform that the ansible open source project uses like all the data is there everything is open and available the only thing that's not is our cloud keys because we don't want people mining Bitcoin on our on our ADBS bill but yeah, it's all out there we could definitely absolutely plug into any system there and that's kind of the goal is to offer an opinionated method of doing it but be flexible when needed any more questions? so the question is do we have one kickstart file per image and are we generating the kickstart file or can you add one after so okay, so if you want to change a config do you need to re-run the whole pipeline if the config can be addressed in a kickstart you can do either we will support either so the options are there any automation to add kickstart changes or post build you can just supply your own in the bootline like you otherwise would yes so the way we do that is with versions so versions are either provided or auto incremented and you have to actually tell our automation that you want it to auto increment and create you a build every time you run otherwise it will come back and say that that version has been built and is already cataloging indexed and my laptop is going to sleep it's fine yes role-based access control is your friend so this was that conversation I had doing things on the CLI for 10,000 devices is not very good so within the platform there is definitely role-based access control and you would assign different roles to different groups and coming from a company that was really 12 different companies that is a very tedious job but it's very important that you think about what has the access to what roles and the separation of duties of who can do what so definitely you would need the platform and do that within the platform architecting identity is a whole different thing that nobody ever thinks about until they are like oh wait we shouldn't let Steve do all the same things that Adam does there is also a centralized logging and audit within the automation platform so the automation platform is composed of basically 14 or 15 different open source technologies integrated together and one of the things that are in there is the SSO role-based access control policy management that kind of stuff yes if you granted one employee enough access in the role-based access control to destroy the factory yes you could find them it may be too late but yes so the question was how do we ensure that basically do you have a dev test type of an environment for this yes you should absolutely have development environments the question is how much is it going to cost to have a development robot environment in manufacturing and I can tell you because I've set them up it's really expensive but it's worth it if you can automate an entire factory floor that becomes hundreds of factory floors so there is a cost to benefit there but you don't want to test this do not test this in prod I mean I test my code but I only test it in prod except at a factory I think we have one minute if there's any other questions out of time so we actually just announced an AI platform to do that so we're working it's in the upstream in a beta right now but it's called Ansible white speed the question was is there a way to write helpers there's also a thing called that I highly recommend you check out it's integrated into VS code and the language we're out of time we're out of time the language server can also be used from other editors like Emacs and Vim if you're we gotta go also our friends over at steampunk wrote some cool stuff too check it out