 Hello and thank you for joining us today for the webinar on real-time pipeline monitoring for the energy sector. I'm Grant Swanson, your host for the session, and I'd like to introduce James Stewart-Bruges, head of product for Clarion, Ben Cleary, CTO for Clarion, and Alex Mikolev, Senior Solutions Architect for Infineon. Welcome. Hello. Good morning. Good afternoon. So this is James Stewart-Bruges speaking. As Grant said, I lead the product at Clarion. Clarion is a British company, a young company based in the UK, and we do both data analytics and we do development of electronics and hardware that goes with it. And so what are we doing? Our main focus is around pipeline infrastructure. So pipelines, energy pipelines, oil and gas pipelines, also water pipelines. And our mission is to optimize that infrastructure, those pipelines. And so there are all sorts of different ways in which this is relevant. For example, you can think about pumps or compressors driving the fluids through pipelines, that the amount of power that those pumps or compressors use really needs to be optimized because that makes a big difference. Just a small, a small improvement in optimization can make a huge difference. We can think about flow and leak detection. These are things that we can monitor and measure and detect. We can do predictive maintenance and look at how the equipment is running and work out when is the best time to do your maintenance and your equipment with analytics that will see problems are likely to happen before they actually happen. So really optimizing that time between maintenance so that you don't do it too early and you don't do it too late. We also can support geo hazards. So around a pipeline, you have a lot of hazards from, from the hillsides or landslides or seismic activity or floods, these kind of things. Another area where our software data analytics and hardware can can really support. So what do we have, we have developed some really, really great software, which is designed from the bottom up to be really secure with the latest standards, which Ben will talk about a little bit more detail later on, probably. But the, the great thing about the system is it's, it's very modular and it's scalable. And what sets us apart from other companies that do this type of thing is that we don't just do software and data analytics. We also provide the hardware and provide our own edge device which we design ourselves. You can see on the right hand side, you know, this is a device which can go out into the field self powered work in a remote location. And do data analytics in the field in a remote location so that you don't have to send back very large volumes of data over the, you know, over a satellite link or over a low bandwidth connection. So we develop our own algorithms in-house. We integrate data from third party sources as well. So it's not just data that we capture from our own sensors, from our own hardware. It can be data that is already available in the system from sensors that currently exist. It might be sensors that are already connected to a SCADA system. If all of that kind of data can be ingested, it might be data which is manually collected and ingested, or it might be data that's a third party data source such as what the commodity price is today, or what the weather forecast is going to be, and how the temperature is likely to go up and down. All of these different sort of siloed data sets, when you combine them, pull them together, that's where we can really add value and provide some meaningful insights. It's not just about what's happening in real time here and now and monitoring real time, but also, what was the past? What was the history? What did that look like? And with the intention to be able to learn from the past trends, understand the present and being able to then predict what's going to happen in the future and can we get a better grip on being able to see what the best way of running an operation is. So that's enough on the sort of high level overview. If we start to think about how does this all work, then what we have on the right hand side there is a workflow where you start by capturing data, you record data. As soon as you've got data into a system like this, one of the biggest challenges is cleaning up that data. A very large amount of time can be spent doing this, and this is one of the areas where we really have good expertise to be able to automate that process, to be able to grab data from multiple different sources and format it, clean it, make it useful. So capturing all of the valuable insights in that data, and without losing any of the critical information because very often in data, there's some really valuable information that might not appear to be meaningful on the first glance, but when you understand it, look at it in more detail and there can be really meaningful insights there. So cleaning data is an important and big part of that. And then we go through the workflow of doing some advanced analysis, which I will talk about a bit more later on, but one of the things that's really, really key to recognize there is that we're not just doing simple analysis in real time, we're not just being able to do some machine learning, some artificial intelligence and some sophisticated data processing. In real time and on the fly as the data comes in, which is different from most systems and a really powerful way of being able to process, manipulate, understand and get value from data. As soon as it goes on, we reveal some insights, we provide the visualizations, the communication, the meaning in that data, with the objective of being able to understand how better to manage things in future, take actions, learn from it, optimize things and then go back, close the loop, going back to being able to keep looking and continuously improve. So with that, let me hand over to Ben to go into some of the more technical detail on that. Thanks, James. Hi, everybody. I'm Ben. I'm the head of technology. My role is obviously kind of building out the kind of software sides and data ingestion and I'll be continuing on from here. As James has mentioned, we ingest data from multiple kind of places. We have our primary pro-tree edge controller. We have our external APIs. As James mentioned, this could be commodity pricing or anything else. So we have also then third party data integrations to this could be existing kind of SCADA or field ops or control room or incident kind of management tools. Now, our existing architecture looked a lot like this. We were using kind of RabbitMQ, which we're able to kind of feed in a number of services. So each one of the services on the left, they have to be kind of written and maintained. And as I'm sure you're all aware, this becomes quite hard to kind of manage because at one point you don't just want to ingest the data. You also want to store it. You want to be able to stream it to a front end. You also want to perform the actions on their data in kind of transit. So the main kind of kind of challenges that we were facing was we had a lot of moving parts from protocols which are varying from OPC to Modbus to MQTT and HTTP2. We often work in a mixed environment. So we would work with on-premise and kind of cloud-based tools. The service deployments were not a simple task. There were lots of them. We had to have lots of supporting tools. All of these then required that all that kind of logging and monitoring and maintenance. And again, third-party data integrations often undocumented APIs, data formats that weren't their greatest and they required a lot of cleaning. We also found that this approach was not as flexible as we first thought. A data team spent probably 90% of their time to clean the data before they could actually get to work on it. And this is where we were introduced to E-Infinity last year. And we began to look at that open source kind of product and Alex. I don't know if you want to say a few words on this particular bit. Thank you Ben. Now, Infineon is a company behind the Fluvia open source data streaming platform. And we also provide managed services as a part of the Infineon cloud. So our data streaming platform is built from ground up using Rust, which provides low latency and high performance and programmable guarantees. We also have smart modelers which allow you to deploy data transformations and data cleaning into streams, which removes the need of moving data in and out of stream or message queue like in previous example given by Ben. We also provide immutable store so it's not a queue, it's a stream so you can read data from there up to retention period. And we provide clients native API for Java, JavaScript, Rust and Node.js clients. And we also, as a company Infineon, we support our developers. They have smart modeler development kit and connected development kit which we'll touch up later. Thank you Ben. Thank you Alex. So, yeah, so once we had been introduced, we began work on adopting this. The architecture that we now have kind of flows in this way. We have a series of connectors. These could be MQTT, HTTP. These will be managed inside the Infineon cloud service that we use. All of these then allow us to write and configure data to be transformed and streamed in. And we have an example of pipeline here where we will serialize to JSON. We may run some change point detection. And then we also want to store the data there. You may be asking what the difference is between the previous architecture and this one. That is that this is all kind of managed inside of the Infineon cloud. So our team just get around to writing the data analysis. So the real keys for us was it was a simplified architecture. There was an efficiency improvement. Our data team can add connectors, smart modules, and build out the pipelines that they want without really waiting on their dev teams. And it's extremely extensible and we'll jump into that in just a minute. The inline execution of code or these kind of modules is something that really drew us in. And we'll explain that there. We use it for our data ingestion site. So this is from our edge all the way to an existing SCADA. We use it also for real-time data visualization. And we also use it for our STL workflow. Our data team has moved away from the traditional ETL workflow and we're able to just tap into streams of data to begin the analysis. We also are beginning to explore the ability to apply AI and ML in stream. So the ability to, as James mentioned, perform that in real-time. That's a bit of a game changer for us. Now I'm just going to jump into another screen and we'll run through a demo of where it sits and how it works. Just a second. So our platform is an overview of the location. In this instance, we have a map view of the UK. We have a couple of edge nodes that we've configured for this kind of demo. We have the ability to view kind of tickets and incidents from the home screen. What we're able to do, and this is where part of the video includes, is we can actually stream kind of data straight to the map. So our kind of ops teams are able to get a snapshot of how things are performing in the field. This is just one aspect of where it sits. The other aspect of where we have focused is on our kind of dashboarding side. This is an area that we're quite kind of proud of, in that we're able to feed the data from the Infinity Unclad over web sockets into our kind of dashboards, and we can get a real-time feed, as well as a mix of historical and contextual kind of data, which provide quite a powerful view. Traditionally, our dashboards have been more based around kind of SQL and performing a query. As you can see here, the data being streamed in, and users have the ability to add their own blocks, which are made up of a query or a stream of data. Now, this is all good and fine, but I'm well aware that this kind of audience probably works more with the data, compared to other examples, which allows us to walk through how we can work with streams and data prints. So what we have is, first off, this is just a basic kind of Jupyter Labs environment, so you don't need any particular kind of tooling. All of this, we're using Python, and we're going to just basically install Fluvio, which then will allow us to begin working with the data. So it should already be installed. It's great. And we're going to look at three ways of working with this. We can work in a traditional kind of client-based way, which is where we're going to hook into the stream of data externally, work with the data and filter and parse it. We're going to look at how we can use WebAssembly as part of this. And we're also going to have a quick look at how we can apply WebAssembly kind of module, or this smart module to the actual stream itself. So we have a number of imports here. We're also going to load the environment file. This will enable us then to kind of log into the Fluvio code. Alex mentioned in Phineon how to have a cloud-managed service, and that is the service that we use. But Fluvio is an open source kind of product. So if you needed to run it locally internally, then you can do as well. So that's just kind of a sign-in to this. So we should get a message back to say it's all kind of linked in, fantastic. Now what we have is a couple of functions which we've wrote of, which allow us to grab some data from a topic inside of Fluvio, and we'll be able to set the amount of messages that we want to see. So in this instance, we're just going to grab data from the MQTT kind of topic on partition zero, and we're going to get the first 10. And then what we're going to do is just print the first one out. So that should run through, go grab the data, and then allow us to iterate through each one. And you can see at this stage their message is not that useful or friendly. We have the MQTT kind of topic and a payload, which is just a list of bytes. So what we need to do is we need to manually clean and process this. So what we have is we have a function, we wrote up that will clean the data, and then it will just spit this back out as a JSON string. And if we run this here, you can see that that message has worked. We now have all of the keys and values. And you can straight away just pop these into a data frame. And if you had more than just one in there, you'd be able to continue on working. That is all good and fine, but that is a very traditional way. And that was the way we're trying to avoid trying to work with this. And one of the reasons why Infineon and Fluvio were so attractive to us was our ability to leverage Rust to build out WebAssembly kind of modules. So I'll show you what that looks like. Here's a snippet of some Rust code. The Infineon team have been kind enough to build out a tool chain that allows you to build out modules based around the filter map, filter map, array map, and everything else. So it allows you to kind of quickly get up and running with that and to do everything by yourself. What we have here is a basic kind of map. So every kind of record that goes through this will be mapped. What all we want to do here is we want to pass, we've taken that kind of Python code, converted it to Rust, and that is what will be sent out. Where there's an error, we'd also get that kind of an array into the queue as well. So if we jump back to this, you can see that we've just imported two extra kind of imports. One is the consumer config and then the smart module kind, which is an enum. We just built out a really basic configuration here. And then we've modified the previous function. That allows us to pass in this consumer configuration. And that then allows us to call the function here. And if we run this one now, we should get, again, a nice kind of set of records here. That means we haven't had to do any kind of processing inside of Python. It's all done as part of the WebAssembly kind of module. That also means it's portable and shareable across your team or others. And it also means that it can actually be applied then at the edge. The other areas that we were interested in is exploring how we can use WebAssembly down at the edge and our edge controller to try and help streamline the data kind of cleansing and processing before that data even hits the cloud. So, again, what we'll do, we'll take the list of records. We'll put these into a data frame. And then as you can see, it is already there. This particular way of working means that your data teams don't have to rewrite the same code. They have WebAssembly code, which is extremely fast. It also means that the security of your code goes up. WebAssembly modules can't particularly do much. It's built for a particularly safe purpose. Now, those two methods are great, but they still require you to manage, put these on and share them around your team. One of the things I mentioned in the slides prior was our ability to move this into the stream. And that was the biggest self for us in that we can empower our data teams to take their kind of cleaning and passing and basically the 90% of their work, probably into a series of modules that can move this to where the data is. And this is how this particularly looks now. So we can configure a YAML file and we can say we want the NQTT type source here. We can configure the broker. Then what we can do, we can build out a series of steps here. That will be deployed with this configuration. So every kind of message that comes in on this kind of topic will run through these steps. So for us, we've used it to clean and pass these extra bytes that unfortunately will occasionally kind of creep in on time to time. We've used it to perform some threshold analysis. All of this is done in stream, which means our data teams don't need to do all of that kind of manual work. Now, just to show you how that actually works, if we literally run this now, these two lines of code, we get a data frame back of data on there, which has been passed. There's obviously still kind of a cleanup where there's not numbers and various things. But in the general gist of being able to get your data team there, in a particular way, has been a powerful feature for us. You can also then publish this onto what Ian Finney called their smart module hub. So that means all of your team can pull it from one place. So they don't need to kind of share kind of repos or binaries or anything else. It becomes quite a powerful tool that that enables teams to work faster. So that's where that's where we'll come back. And I guess, Grant, back over to you. Excellent. Thank you so much, Ben. That was very insightful presentation there. I had messaged, we have a couple of questions coming in. The first one's asking, did you have any Rust developers prior to using Fluvia? No. One of my first experience of Rust was actually creating a pull request on the Python SDK. I was really interested in using it and I wanted to have learning it. It seemed like a good time to, well, in a good place. So no, you probably don't need to actually know too much Rust to do what you want to do. It obviously helps, but it's also relatively, it's a good skill to learn, I would say. Excellent. And anyone from the audience feel free to continue to put questions into the question window. We have quite a few more here. The next one is where you talked about data security. How do you ensure data security in the platform? So from the edge all the way up to the cloud, everything is encrypted in transit. We also kind of leverage quite sophisticated security at the edge by leveraging TPM kind of modules and various things. We also kind of ensure that data is encrypted at rest. And, you know, we ensure that all our kind of vendors that we use, such as a vignan, they take security just as seriously as we do. So yeah, we've got no concerns there. Excellent. Thank you. The next question is what sets Clary on the park from other data analytics companies. The real kind of difference is that we have the ability to go go full and to to to end. So we have edge control of which is about the size of Raspberry Pi kind of thing that could be plugged in at the field. So there are data gaps. We don't just, we don't just say we can't get that data, but actually we have an answer. We can put our edge controller in, plug in the various bits of our kit. This could be a sensor, it could be a pump, it could be anything. We get that data and securely then transmit that up into the kind of cloud. And the other kind of differences that we have is actually the infinity on cloud. It sounds a bit kind of tongue in cheek, but actually been able to do a lot of the cleaning and processing in stream. It means that our data team is relatively small, but we're actually able to do a lot of processing again by leveraging those kind of modules and workflows. James, I don't know, do you have any other kind of answers there? No, I think those are the main ones you covered there. It's the ability to perform the data analytics and really powerful transformations in real time on the fly, which is quite special, quite impressive. And as you said, the combination of not just the data analytics, but also the hardware, the provision of edge computing and local processing from the field. So for example, if the data that we're trying to acquire is a highly dense amount of data, for example, if there's a bit of rotating machinery, a pump or a compressor, and you want to know what's happening with the vibration on that. So, well, you need to sample that vibration data, perhaps from an accelerometer, you need to sample that data fast, and it's a lot of data that's being acquired. It just doesn't make sense to send all of that data over a communications medium via satellite link or a 4G or whatever. It doesn't make sense. So being able to process that data locally, apply some insights up and be able to then update, upgrade and improve that analysis in the edge and transmit the intelligent version of it, the summarized version of it. For example, the frequency spectrum, that's powerful. And that's something different from most of the companies that are doing things with digital transformation and data analytics. Excellent. Thank you. So the next question, I know the demo that you showed with the map and the dashboards is a real live project. And this next question is, do you have reference projects in the oil industry. So I'm not sure if we can disclose that information of who that is or but is there any referenceable projects in the oil industry. I guess, James, would you like to take that one? So we do, and I can't say who it is, but we do, we have some of our equipment on a pipeline monitoring flow and it is feeding into an airport. And that particular pipeline that I'm thinking of is doing real time analysis of the data that goes by in order to automatically identify the volumes and the batches that are going through. It generates a report, which, which tells you, okay, this is the amount of flow. This is the volume and then this is the price, the value of the commodity and it does this all automatically monitoring what exactly what's what's going on in that we're working with another organization where the pipeline is supplying different product types and we are supporting efficiency gain improvements on that pipeline. So in other words, how do we make sure that the pipeline is running at the most efficient manner, looking at pump operations flow rates and what you can do to optimize that. This is particularly relevant in the context of electricity prices that are variable and going high because these are electric pumps and when you pump and how much you pump and how much power you put in your pumps is all very relevant to that. So there are a couple I can, I can mention but, but yes, we're also working with a water company with a similar type of a similar type of problem is the first step of that which is looking again at pump efficiency and optimization of energy going into that. Excellent. And for the, the, the person who asked that question, we can connect offline in a meeting and get more detailed into those projects. So we'll reach out to you shortly after the webinar. The next question that just came in is, have you reached a limit yet on how many different edge sensors. You can pull into your platform. No, no, we've, we've performed some virtualize kind of testing with ridiculous amounts. So the video includes has handled it all like a champ. It's gone through it. No kind of problems. So we're really kind of happy. There's the one thing we did have to do was move our MQTT kind of broke it to a clustered environment and low balance that to ensure that. We're running to problems on the MQTT side before the actual kind of streets. Excellent. Thank you. So we have just a few more minutes here if anyone has questions, continue to type them into the chat window. The next question I think is more relevant to Alex Mikolev can we, can you write smart modules and other languages and are you, do you have the ability to chain transformations? So yes, you can chain transformations on connectors. That's where the YAML file, which been demonstrated, have a keyword div. After that, you can buy multiple transformations. We provide JSON to SQL transformations and Jolt, JSON transformation, JSON language transformations supported by Intinian. And there is a walk on the way of being able to write smart modules in Python, but currently they only supported language for smart modules is Rust. Excellent. Looks like we have one more question here. Give me just a moment. So as I think this one's more relevant to you again Alex, can you republish topics from inside a smart module? Currently no. So smart modules. So the answer to that question is actually complicated. And not the way how it's described. You can't directly republish into two separate topics. But there is an option of creating different types of records and publishing it into the same stream. So the stream isn't the queue so it can contain multiple types of records. So I think whoever is interested in discussing what is the best way of architecture and data around streams. I'm quite happy to support that conversation. Excellent. Well, thank you, Alex. Thank you, James. Thank you, Ben, for the outstanding presentation here for the audience. We will follow up with a recording of the webinar. And we appreciate your time today. Thank you so much for joining us. Thank you.