 Hello there, my name is John Senegal and I'm a solution architect at Red Hat working with partners around AI and edge technology. Shovik and I are going to talk to you about a particular AI use case, parsing engineering diagrams using deep learning and graph search. But before we get into the use case, I'd like to talk to you about the technology that makes this use case possible at scale. Shovik, you'd like to introduce yourself? Hi everyone, my name is Shovik Mani and I'm a data scientist at C3AI. I work on building machine learning applications for industrial use cases such as predictive maintenance, fraud detection and process optimization. Very excited to share an exciting application with you on digitizing engineering diagrams and making use of them in the AI applications that we build. That's awesome, thank you Shovik. Alright, let's get started. So today I'm going to do an introduction on the technology that enables this particular use case that Shovik is going to talk to you about. And as part of that, I'm going to talk to you about why Kubernetes makes sense for AI workloads. And then Shovik is going to get into this very special use case that we'd like to talk to you about. So let's get started with, what are some of the problems that exist in operationalizing AI ML? We know that it's not trivial. We know that as part of this whole life cycle of creating models, you have to have business leadership that understands the problem space and what they're trying to accomplish. And then of course you've got to find and scavenge a hunt for the right kind of data, parse that data, clean that data, make it available to the data scientists who then build the models, who then you work with developers to integrate those models. And then those models have to be monitored and managed for drift and make sure that they're constantly updated and doing the things that you're expecting them to do. So taking a model, taking an idea and going from idea to in production is not trivial task. In addition to that, the problems with actually making that happen is talent shortage. There's not a lot of Shoviks in the world, data scientists who understand the space. And so some companies have difficulties finding these types of resources. Also getting tools and getting tools are available to them as they need them, when they need them on the environment that they need them. It can be difficult as well. And then being able to take these projects and operationalize them, as we talked about on the previous slide, can be very complicated sometimes, depending on the project, depending on how large, depending on the requirements. Enter Kubernetes. And this is where Kubernetes makes sense for AI workloads. First of all, Kubernetes brings agility. What does that really mean? It means that when you have these types of projects, Kubernetes gives you the ability for resource management. So as your requirements and your demand for resources seek compute power, for memory, all of those increase when you're deploying AI workloads, Kubernetes gives you the opportunity to request resources on demand, to scale up and scale down as required. The other thing is consistency and portability. Kubernetes is an open source project. And so with Red Hat OpenShift, the leading Kubernetes platform, we give you the ability to have a multicloud, hybrid cloud model that allows you to run these workloads anywhere. So now if you want to run the workload on-prem, if you want to run it in any cloud, you can have the common platform that runs these workloads anywhere the requirements are directed. You have flexibility. You have the ability, excuse me, to spin up new AI ML projects as needed. So if a data scientist has a bright idea about creating a new model, in this kind of platform, he has the ability to spin up projects as needed, get the resources, get the tools that are required to play around with his project. And most important is scalability. We know when you're running AI ML workloads, the ability to scale up and scale down based on demands, based on the requirements of the workload and the application is critical for the success of these kinds of workloads. In order to do that, you need a whole platform of things that can support this lifecycle of building and deploying models. So at the bottom, you, of course, need some infrastructure. You need a cloud, public or private, virtual. Maybe you want to run this stuff on the edge, but you need hardware, of course. You may need compute acceleration. So we support GPUs as part of OpenShift. And then the next layer is where Red Hat OpenShift is. You need a Kubernetes platform based on what I described before, because that platform will give you the agility, flexibility, scalability, and portability we talked about. And then, of course, you need data pipeline services. You need ML DevOps kinds of tools to build and deploy those models. So in order to execute this lifecycle, you need an entire platform like this. One of the things we did when we looked into this of why Kubernetes makes sense is when you talk to customers, they understand those issues that I talked about earlier. And 94% of them have decided that or have already decided that they will containerize the AI ML workloads within the next year. Either they've done it or they will do it within the next year because they understand the value that containerized workloads running on a Kubernetes platform has for them. And so in order to support that, Red Hat has the hybrid multi-cloud platform that I talked about that is the Kubernetes platform. But in order to complete the stack, we have to partner with other people. And so we partner with NVIDIA and Intel around hardware and compute acceleration. And of course, we partner now with C3AI. C3AI provides those features and functions we talked about before in terms of ML DevOps kinds of tools, data services. They complete the picture and allow the lifecycle to happen. And our platform supports them in doing that. What we bring to the table in this partnership is ease of deployment, consistency, and portability, like we talked about before. And Red Hat is the leading Kubernetes distribution in the industry. What C3AI brings to the table is the ability to deliver enterprise AI at a global scale, also rapid development and deployment, and the ability to operationalize AI. So when we talk about technology that will support the use case we're going to cover, of course, a reference architecture is a great place to start. But actually, this is where we're going to stop. We're not going to get any deeper into this. But I must say, you can see Red Hat in the picture as a supporting common platform across all of those environments, public cloud, VMs, Intel, NVIDIA, one platform to support anywhere you want to run your AI ML workload. And so I'm going to turn it over to Shovek now so he can tell you more about C3AI and get into the depths of our use case. Shovek? Thank you, John. Thanks for the excellent introduction of Kubernetes and how it is applicable to data science and data scientists. And at a high level, the thing I like about Kubernetes and containers open shift is that as a data scientist, you don't have to worry about hardware or spinning up a GPU or specific container. All of that is automated and you can get to work and do the machine learning that you love. So really, this Red Hat open shift powers our C3AI platform, the C3AI suite. And what the suite is, it's a model driven architecture consisting of features such as an integrated development studio to design and build applications, data integration capabilities, integrating data from disparate sources, operations and security. And finally, a whole host of AI model development, deployment ML model operations to speed up the model development and deployment process. So this C3AI suite, we expose this functionality using APIs. And these APIs are used by applications that we build on top of them. So that will be our focus for today. I want to tell you about a specific application that we offer at C3. And I also want to talk about a specific capability of the application, which is capability of parsing engineering diagrams and being able to extract structured information from highly unstructured and noisy diagrams. So let's start with the context and the application. So the application we're going to be looking at here is C3AI reliability, which is one of our flagship AI applications from C3. And what this is, is this is a system of systems anomaly detection application. So what do I mean by system of systems? Imagine you are the operator of some industrial facility. Let's say, you know, an offshore oil rate. You have a lot of systems at your disposal and there is a hierarchy, right? So you may have a compressor train system. And then in that system, you have multiple subsystems, right? So you may have a power turbine in that compressor train. You may have a gearbox. You may have a gas generator and so on. And as you can imagine, you have telemetry from all of these various subsystems, different levels of your hierarchy. And you know, you want a way to understand the health of your entire plant at both the system level as well as the subsystem levels. So what the C3AI reliability application does is it integrates all of these disparate data sources, such as, you know, time series telemetry data from pie sensors. It takes in work orders and maintenance logs. It takes an engineering diagram. And it first builds a unified image of your facility. And then, you know, using machine learning models, it estimates the risk for each of these system and subsystems. So you're able to predict when some anomaly outlier is going to occur and take action to fix that. So as I said, this application consumes a variety of data. And our focus for the talk today will be a class of data we call engineering diagrams and specifically piping and instrumentation diagrams or PNIDs. You know, these PNIDs represent a map of an industrial facility. And it tells you various features of a facility, such as where certain equipment are located, how equipment are connected to each other, and how equipment are identified. And what happens is we take this hierarchy and we use this to create our system of systems data model in the reliability application. But again, the engineering diagrams are key to creating that hierarchy because, you know, they tell us how these equipment, as well as the sensors, are placed in this hierarchy. So let's dive into the engineering diagrams themselves. And this will clarify how they're actually used by their reliability applications. So these engineering diagrams are, you know, crucial data source for any industrial facility. They're drafted during, you know, the design of the facility. They're used during operations and maintenance to identify where equipment and sensors are. And they're also used during safety checks. So, you know, in certain jurisdictions, there's actually requirements to keep these engineering diagrams up to date because, you know, they are the most up-to-date living kind of representation of a facility. And this is an example of an engineering diagram. This is called a piping and instrumentation diagram, as I said, or PNID. And as you can see, it's a collection of symbols and text and lines between these symbols. And I want to peel this apart and tell you what it means, what it conveys. So first of all, you see, let's go over here, yeah. So you see you have some big equipment symbols, like this E321 or this V329. So V329, this is a vessel. This is some other equipment. So it's a large, large symbols are equipment. Then you have these circles. These circles are called locally mounted instruments. So these are actual physical sensors, so like a pressure gauge or a temperature indicator, which you'll find on the facility floor. And then you'll notice that these circular symbols are often always connected to this infesting symbol, which is a circle inside a square. And these are called tags. And tags are the digital representation of the sensor. So if you can see this tag here, it says PIC3030. And what would happen is if you go to their Pi database or their time series sensor database, you can actually find an entry under PIC3030. And you can fetch all of the time series data coming from this tag here. So as I can tell, really, this is the map of the facility. And having a good understanding of this lets us recreate the systems of systems, asset hierarchy, which will power our C3I reliability applications. The final thing I want to note is often you'll see lines in these diagrams. So whenever you have solid lines, those are physical process lines, so lines that carry fluid. And sometimes you'll see dash lines. And those dash lines are electrical signals, which makes sense, right? So you have some sensor and you have some electrical signal to digital representation. So that's how to read these diagrams. The key scope of parsing these diagrams is to be able to detect these tags. So you want to be able to detect these tags in a diagram, be able to parse out the text. And then if you have a list of tags in a diagram, you have a sense of where everything is located, right? So for a typical facility, you may have thousands of diagrams. And if you didn't have a way to automatically parse them, you'd have some poor employees sitting through thousands of pages trying to manually construct this hierarchy. So the goal of this diagram parsing will be three steps. Number one, to detect these tag and LMI symbols. Number two, to detect the text, to detect and parse the text inside these specific symbols. And number three, to detect connections between various symbols so that we can link symbols to each other in this asset hierarchy. So let's get started. First, I'd like to give you an overview of our machine learning pipeline, which we use to digitize these diagrams. As input, this pipeline takes in the image of the diagram, as well as an optional set of manually labeled symbols, such as equipment. So these are one-off symbols, which we can't really show you in the Model 4, but the user can scaffold the automated diagram parsing process by providing these kind of one-off symbols as labels. So that's what the green boxes represent. Then we process it through a series of steps. Number one is symbol detection, which we use to detect the tag and the LMI symbols, as I said. Number two is text recognition and association, to detect the text. And number three, connection detection, to detect the connections between the various tags, LMI, and equipment in the diagram. And the output of this pipeline is what we call an asset hierarchy. So essentially, it's a structured representation of this highly unstructured image. So it tells you which symbols have been found, whether they've been manually labeled by the user or detected by the pipeline, what kind of symbol that is, whether it's an equipment, a tag, or an LMI, locally mounted instrument. The text, which has been parsed for that symbol, as well as other symbols, which it's connected to. So it gives you all you really need to know about this diagram and its contents. Before going into more detail on the secret sauce and the AI models, which make this all work, I'd like to go through a demo in a Jupyter Notebook. And specifically, this is what we call a containerized Jupyter Notebook. And this is C3 software, which can run on Red Hat in a container. And it lets the data scientists easily train and deploy models and collaborate with other data scientists without having to worry too much about infrastructure. And so here we have a C3 containerized notebook where we are demoing this pipeline for diagram parsing. So the first thing we do is we load example diagram. And you notice that a lot of these calls are made from the C3 handle. So this is a library that's automatically loaded in. And it contains the whole, it lets me access all the data as well as the functionality of the C3 AISuite. So that's how the APIs are exposed to me as a data scientist. So I take this diagram and I convert it to a parsable diagram object. Here you can see that diagram, it's the same diagram I was showing in the slides. As I mentioned, we let the user specify a set of manually labeled symbols. In this case, they pass it in as an XML file. And here we have a set of five manually labeled symbols, which the user has specified, as well as the text that the user has labeled. And so you see five of these manually labeled symbols, zero or zero, one, two, three, four. And these are equipment symbols, which our pipeline otherwise couldn't detect. And then we get started with our pipeline. The first step in our pipeline is a symbol detection step in which we use a CNN or a convolutional neural network to look at crops of the image and classify them as a tag symbol, an LMI symbol, or not a symbol. So that's why you have this one by three dimensional output at the end. And notice what we do here is we initialize a machine learning pipe called a symbol detection pipe. And the C3 AISuite, the way we think of machine learning, the way we do machine learning is through this concept of pipelines. So we have a pipe for symbol detection. We have some hyperparameters, some other metadata, the path to the model, which we want to use for this step. And once we have initialized this pipe, we pass the parsable diagrams. Remember, the parsable diagram is the object which contains the diagram itself, as well as manually labeled symbols. So we parse that diagram, we pass that diagram, and we process it using the pipe. And what we get is we get another parsable diagram. And now, if we display that parsable diagram, you notice we have a whole new set of symbols in red, which is what the pipeline has detected. So you see some of them are tag symbols, some of them are LMI's. So those look correct. You'll notice that sometimes the pipeline makes mistakes. Like, for example, this is not an LMI symbol. This is a completely different symbol. This is an oval, whereas LMI symbols should be circles. Sometimes it does make some mistakes, like overlapping symbol. But if you look for the vast majority of cases, this is really good detection. And we're able to isolate where these sensors are and where these LMI's are. And that's very exciting to someone who would have to otherwise parse each of these individually to create our systems of systems acid hierarchy. And the great thing is now we can take this diagram and we can convert it into a structured acid hierarchy. So you see these five were the manually labeled symbols we started out with. But now we have a whole set of detected symbols in this diagram. And that's done automatically. The next step is the text recognition step, where again, we pass in the same parsable diagram into now this text recognition pipe. And the output we get out of that is now you see this column has been filled in. And that's the text which has been detected inside or nearby those symbols. So for example, let's take a symbol like this one, maybe this one here. PT 83030, and we see the ID is 24. So 24 should say something like PT 83030. We can look it up in our table here. And we see sure enough, symbol 25 is detected as an LMI. And we see the associated text to that symbol. This is just another visualization of the text which has been detected in blue. The symbols again, detected symbols in red and the manually labeled symbols in green. And finally, we have the connection detection step in which we use a graph search to parse the diagram, to traverse the diagram and detect connections between symbols. Now you see this column has been populated. And you can see for each symbol, we give you a list of symbols, other symbols that it's connected to. And this can be used later in downstream use cases, to be able to know how symbols are connected to each other. And for example, say you had equipment in red. And you wanted to know what sensors to monitor, to do predictive maintenance on this equipment. Well, now with this connection detection, now you know that all these symbols in green are actually connected to the symbol in red. And so you could get the data, you can get the sensor data from these green symbols and use it to monitor this equipment. So that's a rundown of the pipeline as we have it. I'd like to go back now and I'd like to give you some details on how we train this pipeline. First, to train this symbol detection model, this CNN network that I mentioned. We first labeled a lot of diagrams. So we just sat there labeling the data. We labeled 18 diagrams in our training set. And specifically, we labeled all the tags. So we drew bounding boxes around those tags. We drew all of the LMI symbols. And we randomly sampled a ton of not symbol crops. And so this essentially created a data set that we could train on. Then we created this model and trained this model on the data set. So for each crop which we had labeled, we had a corresponding output for whether it's a tag or LMI or not a symbol. And we trained the network to predict on crops and give us their classification. Then we took these manually labeled symbols. And then on this diagram, we took a sliding window and we passed it over the diagram. And each sliding window, each crop, we passed it through our network. And using that, we were able to get the actual predictions for where all the sensors were located. And so this is the result I showed you earlier of all of the tag and LMI symbols in red being detected by the pipeline. So that's symbol detection. The next step, which is the text recognition. For text recognition, what we did is we used a pre-trained text recognition network called East. So this was just available off the shelf. And we used it to detect regions of text that you see here in blue. After that, we took these regions and we processed them using Tesseract OCR. And based on proximity, we mapped them to our detected symbols. So in this case, you see this LMI symbol here. And you see inside it, you have two of these detected text regions. So once you process that using OCR, you're able to say that, okay, this PD-3030 belongs to this LMI symbol. And that's how we do this association. Finally, we have this connection detection step, which I believe is the most sophisticated part of this pipeline. And a part that excites me the most as a data scientist. So connection detection, again, is this problem of finding which symbols are connected to each other via the lines. And the lines, again, they can be physical process lines, like you see here, like a pipe. Or they can be like an electrical signal, right? Like a sensor to a database. What we do for this connection detection is we actually take the image and we represent this as a graph. So you have a graph of pixels in this case. And you have symbols, which are, you know, sets of pixels. So you have four symbols here. And we do a graph search going from symbol to symbol, following the lines which you see as these black dots in this graph. You can imagine, right, this presents a difficulty, right, when you're trying to traverse these dashed lines. Because, you know, your graph search, right, it'll hit a roadblock. It won't be able to traverse these dashed lines. So we also have an additional step where we use a probabilistic of transform to fill in those dashed lines. And now suddenly your graph search can traverse through those dashed lines and solid lines and find connections along the way. So using this graph search process, what we're able to do is for each source symbol, in this case the red symbol, we're able to find connections to other symbols in the diagrams through these lines. So for example, you see here, and then the connections, the path traversed by the graph search is shown in blue. So here you see that from this symbol, you know, we follow a traversal down this path, down this path, and we're able to hit this connected symbol. You know, similarly, sometimes, you know, it takes an exhaustive search. It goes all the way down here. And, you know, poor thing doesn't find any symbols. So it comes back, and maybe it finds some symbol here. So really it's able to traverse this diagram and find connected symbols. And again, as the output, what we get is we get a structured asset hierarchy table, right? For each symbol in the diagram, we're able to tell you what kind of symbol it is, what kind of text is inside or around the symbol, and what other symbols this is connected to. This associated text here, this is actually key for our reliability application, which I introduced. Because often from customers, we get, you know, hundreds or thousands of these diagrams, and we get a list with, again, thousands of tag IDs, right? Pi tag IDs, which help us fetch the data. And we have no way of mapping between those two, right? We have no way of knowing, you know, how to get data for a specific symbol in the diagram, unless we, you know, manually parse those things. But now with this diagram digitization pipeline, we can automatically detect the tags and we can detect the associated text. And if we match this associated text against the time series data, then we can suddenly fetch data for any tags that we wish in the diagram. Finally, presenting some results of this pipeline, you know, in symbol detection, we had very good performance for tag detection in terms of precision and recall. And as you can see in the test side, we achieved, you know, essentially 100% precision and recall, perfect classification. Whereas for the LMI symbols, they're a little harder for all problem to track. So we had a little poorer, but still you can get in 90, almost 90% recall with relatively high precision. This is a sample of the symbol detection results where you see the, you know, ground truth labels, which, you know, we labeled in green versus the labels by the machine learning model CNN in red. And you can see they're pretty close overall. And finally, as a next step, as a taste of what we're working on right now, we're working on expanding this in future directions, including using RCNN, which is a region proposal, neural network. And this, as you can see in the bottom, so this is an RCNN, this will help us get better bounding boxes around these symbols. And it'll help us avoid a sliding window approach, which we had introduced earlier. So we're excited for this new development and for the improved results that it brings us. So overall, you know, at C3, we build AI applications. We build applications that work on a lot of disparate data sources from time series data to maintenance data to engineering diagrams. And oftentimes it turns out that these diagrams are, you know, key data structures to getting these applications to work and to work as the way you want them. So with this diagram digitization pipeline, we'll be able to power applications such as C3 AI reliability. And we're also able to create new generations of applications, everything from diagram search to equipment sensor mapping and eventually the creation of a facility-wide digital twin so that you know where all of your sensors and equipment are located, which time series data they're connected to and you have, you know, overall view of your facility. Thank you for your time. Thank you so much, Joe and Shovik. That was awesome. That was some good stuff. I hope the audience liked it. I'm glad you're so awake. Yeah, we'll be around to answer questions. So thank you for joining us for our session and ask away.