 Okay, so, hey guys, I'm Venkat, I run the data platform for Predix platform, so we're both coming from Sandramon, so yeah, we're sitting here to actually talk about, it's actually a change in the topic a little bit, so we talked, we want to, we'll touch up on real time events as well, but we will do handling stateful applications on our past journey more, so it's going to basically cover more ground from Predix perspective. Hi guys, this is Zubala Ji, I run Predix's application and container platform for Predix. Okay, so what are we going to do in this talk? So we're going to bring you guys up to speed quickly on what Predix IoT platform is and what we actually do. We will talk about our past journey, we started building a platform about two to one half years ago, so we'll actually talk about like how we originally initiated and then where we are and what our challenges and we are actually hitting. We'll also talk about stateless and stateful applications in general and what our needs from both of these services are. We'll talk about our challenges and we'll do a demo on Q&A. So the first slide we want to talk about is here, when people talk about IoT platform, everybody gravitates towards a consumer IoT product like Nest. The first differentiator that we are building in as part of GE Digital Charter here is we're building in whole class secure industrial IoT platform. The biggest difference when you throw in the word industrial is just one word but the scale of what we're talking about, it just grows out astronomically. We're talking here about mission critical applications, mission critical devices like jet engines, MRI machines and wind turbines which are pumping in data at a much higher rate than that's ever been seen on any IoT platform before and all of these devices come in with their own inherent challenges and intricacies especially around security, around compliance and all this stuff. So that has been our biggest differentiator when we talk about an IoT platform. So right when the word GE has been around for, this is actually coincidentally a 125th year of GE. GE is being a well-known name across all households and we have presence all over the world but GE has been tagged as being an industrial big company. So there's not even a single vertical that exists in the world where GE's presence has not been there but when you take such a big company and you pivot around GE to become a digital footprint, that journey is not easy for one and that journey has been something that we willfully taken head on and to quote our former CEO here, we are in the business of dealing with information, we are in the business of dealing with data whether we like it or not, whether we want it or not and given the fact that GE has such a huge footprint in many business verticals, we are the biggest domain expertise of if you've taken any business vertical here and it is such a natural organic growth for us to become the leader in building up this industry IoT platform especially because of that of our inherent domain expertise. Okay, just a quick survey. How many of you guys have known our use predicts? Okay that's awesome actually it's the right audience we wanted to talk about. So like Balaji talked about, so predicts is our IoT platform. GE's industrial customers are pretty much everywhere in the world. They are in almost all the geographies because we span across like various businesses right from aviation to transportation to healthcare, energy and so forth. So from a predicts perspective in order to build a platform that's going to sort of help all of these different customers we have to be there in multiple areas. So predicts is right now in like four regions. There's one in US, East, West, there's one in Europe and Japan and it's only expanding pretty much and majority of our food plants there are in AWS today while we are also expanding it in other clouds. So we have about multiples of 10,000s of machines. We have thousands of developers using the platform today. We have close to about 30,000 applications that are deployed in this platform across various businesses and whatnot. And we have a number of software stacks that are actually running in predicts. These stacks are included from stateless applications to stateless applications like relational databases to no SQL source and whatnot. So from a cloud perspective, you guys got a sort of view of scale from a cloud perspective but there is also humongous data that's coming to our cloud. So G has assets or machines that are deployed in so many different places that all of these are actually generating a lot of data. So for example, this is one of the jet engines. This basically says a particular jet engine generates about a terabyte of data every day. So the way we actually build cloud is to make sure that all of these data from these edges are able to be ingested into the cloud for doing your analytics and everything else. So when you talk about how predicts is built in, we are trying to become a cloud agnostic IoT platform. So we depend heavily on automation to bring up our predict infrastructure itself. We use Hashicarp's Terraform for bringing up a infrastructure in any footprint today. That's Amazon and Azure eventually into any other public cloud that's available. So we depend on Terraform to bring out our underlying cloud infrastructure. On top of that, we use CloudFoundry's Bosch to bring up CloudFoundry. From the time CloudFoundry is exposed out to our developers, CloudFoundry is the only contract that our developers exist within predicts. CloudFoundry provides a very unified way of accessing predicts itself for our developers and for end customers' developers who want to consume predicts. The advantage of what CloudFoundry provided us when we started this journey was CloudFoundry has this robust marketplace service where you can have asset building models, analytic building models, and then CloudFoundry also provides a very secure interface to predicts itself. What you see here is a typical use case of predicts where we have predict edge machines that are deployed out to our customer site, which acts as a local aggregation point of sensor data, does some computation on it, sends back the learnings and the data back to the cloud. We also have a enterprise data connect, which is used by G-current, for example, to pull in data from local light sensors to do analytic jobs on the predicts. On the other extreme, we have northbound applications that customers build on top of predicts to consume that data to do predictive analysis. Awesome. So extending to that, we use CloudFoundry as our base platform infrastructure that we just talked about. CloudFoundry, for one, abstracts the developer experience in a way that's actually sane for them. They don't have to manage individual machines and infrastructure and whatnot. It's actually great for state-less applications. It gives you a nice marketplace that you can interact with to sort of create your service instances that you need, bind it to your applications and so forth. There's also CLI and API access to it, and Balaji also touched upon how we deploy CloudFoundry. All of these are fun, but although the biggest sort of shortage or limitation with CloudFoundry today is the support for state-full applications. Currently, many of our state-full services are deployed through automations outside of CloudFoundry. So this is either using Bosch-type deployment or Sheffield Terraform-based infrastructure. So we deploy these outside of CloudFoundry in a separate sort of segregated set of instances, and then expose them to CloudFoundry via a service broker. So this essentially started about six to nine months ago. We started this journey of looking at the next-gen platform for providing a seamless integration for anybody who wants to run state-full work program predicts on a container platform. We naturally gravitated towards Mesos, which provides a very generic orchestration platform for running containers with a very good robust two-level scheduler, great adoption among the open-source community and enterprises, who has done all... We have similar customers who are going through this journey, who cry about each other's shoulders, who understand what the pain points are going from one pass and other passes, how we provide a unified experience to our end customers as well. The stuff that we like today about Mesos is the fact that you have a first-class citizenship for state-full workloads and for persistent volumes. Okay, so now we talk about how do we bridge both of these worlds. I mean, we talked about state-less applications and how our customers use CloudFoundry today. And the goodness of what Mesos actually brings into this platform of state-full centric applications. So our intention is to see how to bring both of these worlds together in a way that our customers still have the same experience from how they are using and the predicts platform, like how they actually go and look at a marketplace and how they actually use the service instances and everything. So what we are essentially looking upon is kind of actually building a unified marketplace across both of these past systems, where you can still expose all of your native constructs that are available for state-less applications in CF, but also bring up the services, the state-full services that are available in Mesos into CF. So we're going to talk a little bit more about how this integration actually works and what we are doing there. So what you see here is a sample of a marketplace that anybody who logs into predicts using the CLI look like. So here you see about five catalog services here. The good thing about what this particular representation view gives you is, for example, you have our RabbitMQ cluster and a Redis cluster. These are orchestrated using our on-demand Chef Broker, where anytime a user interacts with either one of those brokers, Chef is our backend infrastructure which kicks in, which goes there and creates a on-demand cluster for the user. The anomaly detection that you see here is actually a native app in Cloud Foundry itself, which is running as an app in Cloud Foundry, and people can interact with that natively within Cloud Foundry. The last two here are our first cut of our implementation on Mesos, where people use these two clusters to go spin up a Mesos cluster to consume Elastic and Cassander. Okay, so how this happens behind the scene? So is anyone here familiar about Open Service Broker? Okay, a few hands. Okay, that's good. So a Service Broker is an API that the CF community came up with a long time before. So it's effectively a way to sort of integrate your platform's marketplace and your sort of service provision which is your broker. So the broker spec is like five or six APIs. One API is to basically return your service offerings, like what is your service, what are the plans that your service is offering, and what are its configurations. There are a couple of APIs around instance creation and deletion and a couple around like how do you get the credentials from this service instance and attach it to a particular application. So I think the community saw the value in having a Service Broker API that's common across different platforms and sort of came together. So the Kubernetes and the Cloud Foundry community have both sort of adopted the Service Broker API and they essentially have extensions for both. But the missing piece was a Service Broker for Mesos. So we went ahead and did an implementation of a Service Broker for Mesos. So Mesos DCOS has this API called Cosmos which is a package manager API. So when you go to, when you look at the Mesos universe to actually look at all of your individual packages, like even the stateful packages that you've essentially got, so there is an API behind the scenes that actually helps you manage the lifecycle of each of those packages. You can actually go and create an instance and whatnot. So we went ahead and created an implementation of Open Service Broker that actually talks to this Cosmos API to go and spin the instance, sort of make sure that the instance are completely up, sort of monitor each of those tasks and also bring back the credentials and everything the Service Broker normally does. So this actually sort of shows you about the different sort of service frameworks that we've got and how we've essentially integrated with different APIs on the Mesos side. On the other side, we essentially have the Provisioner or the Service Broker APIs on this side. So what you see here is a typical call flow of what happens when a user interacts with Cloud Controller API to instantiate a service. So the request is when a user goes logs into CLI, which is what we're going to be seeing the demo at the end of it under the deck is, a user logs into CLI and hits a command to go create instantiator service instance. That call goes to the Cloud Controller, which looks at which broker is registered for that instance. In this case, it would be a broker running in marathon itself. That call goes out of the broker turns around and then instantiates a cluster within marathon. The response back to the end user would be a environment variable, which gets embedded into their CF application itself. And at that point, the application then can consume the cluster that's in Mesos. What we're also attempting to do here is to create a seamless service mesh between Cloud Foundry and Mesos and also start down the journey of having a secure isolated single-tenant offering within Mesos itself. Cool. So before we go there, so we do plan to open source the implementation here. The implementation is about 100 to 150 lines of code pretty much. It's written in Scala. It does have a sort of open implementation that can actually talk to any package in Mesos, except for a couple of things, because there are going to be specific sort of nuances within each package on what configuration it exposes and what configuration that you have to send it to it. Those are easily extendable in the framework. The framework has written in such a way. So yeah, we'll probably write a blog or something at some point to sort of let you guys know. Okay, so the next important thing is persistent. We just wanted to do a double click on the persistent volume, because this is the whole missing piece in the Cloud Foundry infrastructure today. So two things that are super important for us is to have sort of a unified infrastructure where we can essentially do both persistent and non-persistent services. So for persistent services, of course, there's a couple of things that you need. You need to be able to sort of attach a persistent volume to a container. So the container actually moves from one place to another place. You want to be able to reattach your volume. The second thing is you won't also be able to do this dynamically. So you don't have to sort of pre-create this huge massive disk and attach it as a single volume to your container. So you want to be able to sort of shard and then partition these disks and then assign it to these containers. So when we started this journey, CSI was CSI as the container storage interface that was already talked about at that point. So we looked at it closely. There is still work that needs to be done across each of these platforms, including Mesos and Kubernetes and everything else. And there is also the sort of integration with the frameworks that's also missing. So we decided to sort of stick with the DVDI module and gravitated towards port to works as our solution for using persistent volume. So from an industrial perspective, so there are a few, I mean, like when Balaji talked about security and compliance, supporting encrypted volume, being able to take backups and snapshots are all very, very important characteristics for a data store that we essentially provide. So we were looking at a storage solution that actually has all of this and that's, that was one of our primary sort of requirement when we are actually looking at these things. Then the other thing is we also wanted a level of support from the community and with the frameworks that are available in the community because we didn't want to go back and reinvent every single thing across all of them. So DVDI came in very handy and the support on port works was also super useful for us. So when you talk about workloads that we want to run on the IoT platform, there are two prominent types of workload, high ops like Cassandra and Elastic, which is a bread and butter that as far as our platform is concerned, and a throughput intensive application, for example, Spark. The ability for a underlying persistent volume provided to quickly switch back and forth between either one of those and have a quality of service associated with each of those was very imperative for that. And that's where port works came in and helped us out. So the other thing is we also, when we started working at persistent volumes and the persistent data stores, we immediately started working with the Meso skies on the DCOS Commons SDK that they started building. So we work with them on a few frameworks including Cassandra and Elastic search and work with the port door skies and sort of switching them together to be able to use them. So we will talk about those learnings as well. So one particular use case that might be interesting for this conversation is sort of an additive manufacturing use case from aviation. So I mean jet engines, GEE has been one of the largest manufacturers of jet engines. They actually say for every few seconds, there is actually a flight that's taking off with the GEE engine. So one of the things, the first thing that the additive team started actually working on is sort of figuring out like what is the one that can actually make a lot of difference. And they picked up fuel nozzles as one of those initial sort of pieces that they wanted to reprint. So this, to give you guys a little bit more idea, these nozzles are like made up of 20 parts when they started. But when they were 3D printed, that's just one part. And it essentially provides you 5x durability and it's like 25% lighter weight. All of these are important attributes when it essentially comes to like building a jet engine, so to say. On the background from our perspective, from an outcome perspective, I think two things. One is the metal powder that are being utilized for manufacturing is like a highly expensive commodity. And these printing actually takes a long time. So some of these nozzles actually takes like hours and hours to essentially get this right. So what we wanted to do from a business perspective, what they're actually looking for from us is to essentially provide them an anomaly detection algorithm and a system behind that can essentially go and identify if there are any issues while it's actually printing. If there are, the need is to immediately stop it and sort of basically flag it to someone, say go and actually take a look at it. Because if you continue to print, I mean these metal powders are like super expensive and can be erased. So to help with such a use case, what we are looking at is a architecture that looks something like this where we are trying to push the computation of as much closer to the edge as possible with the chipsets that we have in business today. A lot of these edge devices are very powerful, but we still see the need of doing in near real time analysis on the cloud as well. Which is why we have a edge processing which pushes in data to our IoT event hub infrastructure. That's a running in Mesos as a framework, which then takes in two forks based on a record of time real time computation goes into work in progress with Apex and the Heron teams as a framework on Mesos, which then uses Cassandra for its tape or a micro batch processing that can be done on Spark, again using Cassandra in the back end of the finish line. Okay, it's back. This is our basic picture of how our real time architecture looks like today in progress. Okay, so we will sort of briefly talk about some of the challenges that we were essentially faced over the course. So cleanup of tasks wasn't automatically done when the service was done. So when we started using the new SDK for the persistent framework like Cassandra, I think one of the initial problems that we faced was everything was working for sort of creating a particular data store and then sort of managing it. But then the moment you actually issue a delet for whatever reason, all of these tasks are basically still running there like a ghost task, so to say. Which is an important problem for us is because when a customer actually chooses to delet his particular service instance, we want the entire thing to be cleaned without any traces. So that's one of the stuff. The good part is we worked with Mesos community and then that got actually fixed as a part of 1.10. Application metrics. So we sort of actually, we have our own monitoring system to essentially monitor our infrastructure. And PromiseDS is one of the prominent ones that we actually use. The application metrics API in Mesos is a completely custom API. So we've given the feedback and we're working with them to essentially see if we can get something as more standard API or maybe compatible with PromiseDS itself because it's a good show today. Few other things that are in Cosmos API is, so when you actually saw the create instance call that Balaji was essentially sort of helping us trace through, there is actually a call to Cosmos API that actually says here is all of the parameters go and essentially spin a cluster for me. But the sort of the pain point there is the JSON that we essentially send it to this API is not getting returned back to us. They're actually completely returned in a completely different format. So this is something that the community is aware, so that's getting fixed. We talked about a few other issues in the SDK frameworks along this. I'm going to skip that. Windows support. So I mean, GE's analytics are not new. I mean, GE has been running analytics for a long time now and some of these analytics are written in so many different languages across businesses. They go from Python to MATLAB to R to like very custom Windows-y analytics that are actually written. And some of these analytics are like so big and complex that it's actually not a trivial task to be able to rewrite the same thing in some of their language. It might essentially take time and it's going to be a gradual sort of migration. So for us, being able to support Windows and Mesa's DSCOs is pretty important because there is so much of analytics that are still in Windows ecosystem for us need to be able to run in this sort of unified infrastructure. So that's been something that we're working with them as well. Okay, let's switch back to a demo. So the first demo that we're going to essentially show you guys is sort of how do you essentially see the marketplace? How does this marketplace integration actually works? And how can you do a lifecycle of service? So I am logged into a CF environment, I hope. Okay, cool. So we're hoping it's all visible to you guys. Okay, so I am going to look at the marketplace. I'm actually using the CF-CLI, which is the CLI interface to interact with Cloud Foundry. So I'm doing a CF marketplace to actually get all of the services that are available in CF to use. I think it's the Wi-Fi network, which is okay. Okay, perfect. So there is a service that's actually available here, which is a Cassandra framework that we have essentially exposed it from Mesa's. So what I'm going to actually do is I'm going to essentially create an instance of this particular service. I'm going to actually go to my notepad and then I'm going to paste the... So what this actually says is it says, when a customer actually logs in, he's going to basically run this command and say, okay, I need a Cassandra cluster, and I need three nodes in this Cassandra cluster, and this is my name of the cluster. So the moment you run this, this actually talks to our open service broker implementation between the scenes, and then that actually talks to the Cosmos API for Mesa's. So at this time, I have essentially initiated a request to create an instance. So I'm just going to do CF service to actually see where it is. So you see the create is in progress at this time, and then if I do a dash dash GUID, it actually gives me an instance ID here. So now if I switch back here and then go to my browser, the connectivity is still fine here. Okay, this is a test Mesa's cluster that we put together for this demo. So going into this particular one, if you click on the services, you can actually see this particular service instance, which is the 778, is getting created here. So this is sort of spinning the clusters now. It's trying to basically get things ready, and if you do a CF service, again, in the CF, you can actually, it's still created in progress, so it's actually still in progress. So if you go to the Matathon API or Matathon console, you can actually see the same cluster that's coming up here as well, just as a reference for us. This actually takes like a couple of minutes to actually spin. While it's actually spinning, I want to actually show you guys a couple of other things that we actually did. So the service broker itself has an implementation for creating an instance. It also has an implementation for scaling the instance. It might not be super valid for every single data store. Depending upon the type of data store, you want to expose those APIs. You can do a bind, unbind, and you can do a data recognition. Talk about that as well. Okay, so if you actually see the status here, it's saying there is one node that's actually starting, and the two other nodes are pending. So this is still coming up. Oh, yeah, sure. Yeah, absolutely, yes. So Predix, okay, it's going to be a broad question, so I'm going to actually take a stab at it. So two or three things. So Predix is a platform, which is an IoT platform. Predix also offers a bunch of industrial product suites, like asset performance management. There is brilliant manufacturing, which essentially gives you a way to essentially make your manufacturing environments much better. There is operational optimization, which is also an add-on to brilliant manufacturing and so forth. So for customers, for industrial customers, you could use Predix applications or Predix products to essentially go and start monitoring your own assets, or you can actually monitor Predix assets and basically do preventive maintenance and sort of reduce your unplanned downtime for your applications. So those are at a product level, and we do have some customers that are essentially creating their own applications, extending our Predix products and using some of the Predix platform components directly too. So both of those are customers, and that's sort of our business model here. So let me see. I'm going to essentially, we looked at the GUID. I'm going to actually create a service key here, okay, because the operation is still in progress, we won't be able to essentially create it. Okay, this one is prepared, so it's going to be done pretty soon. And then, okay, so the other thing that I wanted to show you guys, well, this is actually happening in parallel is what we also did is we also created a package for the open service broker in the universe. There is a Cassandra OSB package that also will be open sourcing it. This is for folks who are actually using the like a CloudFont infrastructure to be able to go and say, okay, I want to deploy my own service broker for CloudFont, the open source open source broker for CloudFont. You can just do a deploy and then it creates a sort of the API compatible version of a service broker for Cassandra. So there's going to be documentation and stuff that are going to be published on this as well. And the network is a little slow, so it's just picking up its own time. So what you'll essentially see once this is complete is we will be able to create a service key using this cluster of Cassandra, and then that gets exposed as variable within the app. What is next for us as far as this is concerned is we want to then take that and create a unique service mesh between an application in CloudFontry and a backend infrastructure or a stateful service within Asus. So we're working with partners like Avi Networks here to help us out in that journey as well. So what that also leads us towards is to build a secure mesh as well on top of that. What we will also eventually do is to take all of this implementation right up close so that anybody else who wants to go on top can stand on our shoulders and make this journey more easier for the next ones to come in. Okay. So I think we can also open it for questions while this is actually happening, so absolutely. You can. So one of the biggest things that we have is a platform for anybody to write an application that consumes predicts essentially. So GE has in-depth knowledge in every vertical that's out there. For example, healthcare or aviation, power, oil and gas, transportation, lighting. So we would expose out analytics that we've learned out of our assets. That would be exposed out as part of an analytics catalog that a customer that for example if you would say Kaiser for example, Kaiser then consume that, consume any analytic that is pertinent to your vertical. And you can go start using that to build more models on top of that. Did that answer your question? Okay. So I think in the interest of time is I think the demo guard is not helping us clearly. So there is actually another service that was pre-created before this. So I'm just actually going to use that service to essentially go and show you guys what I wanted to actually really show. So I mean that is another cluster here. I'm going to show you. There's another one called BN as a service instance. I'm going to show you guys that. So it's a three-note cluster as well. It actually has everything running and the deployment is completely successful. It's a Cassandra service as well. And I can do a create service key. And I can basically say create key one. What it actually does is it basically goes and creates a key for us. And then we can look at the key here. So it was instance name which is BN. So when I essentially go and say, okay, give me my service key, it actually gives you all of the nodes that were essentially created on basis. And all of the ports pretty much that you have to essentially connect to this. And the credentials and username and password. This would only be available within that particular water container. And if you only have access to the arg and the space and cf, you can even see this and execute this command. This is protected like that. And we can do a very similar thing with binding the application. So we can actually say, okay, I want to go and bind a service with just a seed application with BN. This, what this is actually doing is instead of us actually manually creating a service key, we're actually saying, okay, go and take these credentials and give it to this guy, this application and inject it as an environment variable. So now it's actually injected as an environment variable for that particular app. Okay. And then in the environment variable, you get to actually see all of these credentials as well. So from an application perspective, you just go and look up your particular service instance and then get the credentials and start interacting with it. And what we are working on as part of this journey is to create this service mesh on the fly as well so that there's no zero touch deployment as when you go create on-demand services. You just got a five minute flag. So any questions? So for us, I think from a cloud perspective, all of our clouds today are on AWS. But of course, from a predicts perspective, we want our customers to have sort of a unified experience. So wherever they are, they actually get the same interface and whatnot. But from a storage perspective, AWS does offers a few types of EBS volumes, like including provision IOPS, standard volume and so forth. So in this case, with Portworks, what we are able to do is sort of create a storage pool based on the type of EBS volumes and then expose it as a class of storage to our frameworks. And the frameworks essentially is going to basically say, I need an IOPS heavy storage volume and it actually gets the storage volume. Yeah, but there is a bunch of performance testing and stuff that are in in in flight pretty much for us. Yeah, awesome. Thank you. Thanks for coming in. Thanks.