 Hey, everyone. Welcome. My name is Yu Chong Yu. I'm a senior technical staff member in IBM Cloud for financial services team. Krishna is our VPN CTO. I know some of you are expecting him. I hope I don't disappoint you. OK. So today, I'm going to use a real-world use case to talk about, you know, to demonstrate how AI is actually used or I mean how open source is actually used in the AI lifecycle in a hybrid cloud environment. OK. And this is the agenda for today. I will start with the trend challenge for AI followed by some real-world use cases. And then let's talk about a little bit, you know, about different open source tools, how they are used in the AI, different stages of the AI lifecycle. How did I build the secure training environment in the hybrid cloud environment? And I can give a little short demo and then we can wrap up. OK, I will leave a few minutes at the end for questions and answers. You can always catch me and, you know, you have my email on the color page. Feel free to contact me. OK. So AI is impacting every single industry. We all see AI almost in every day's life, right? So there's some interesting numbers here. As you can see, based on Mackenzie's report, 25% of the insurance industry will be automated in 2025 thanks to AI and machine learning technologies. Based on Dwarf's report, 37 of the financial services firm will globally adopt AI to reduce operational cost. And 33 of North American financial service firm predict AI will change the way they innovate. And based on PwC report, global GDP will be up to 14% higher in 2030 as a result of AI, the equivalent of an additional $15.7 trillion. Now, what you can see behind those numbers, you know, those are actually some of the areas that AI can help us. AI can help us reduce operational cost. AI can help us to improve the employee efficiency, to solve the big volume transactions. And then AI can help us to gain more insights, make better decisions, and eventually drive our revenues. So those are really the typical areas where you see how AI is being used today. Now, what are the challenges we are facing when companies try to adopt AI? One is that every company normally has something up and running already. How can you integrate AI technology into the existing system? That's one challenge. And then once you integrate that, how do you apply your AI at scale? So that you can, you know, let's say, apply, get some insights for every transaction. That's really hard. And then another thing is model portability, interability, you know, that's another topic. OK, so where do we want to go with this? You know, this is kind of the ideal stage, right? We want some agility from AI, which means, you know, the ability to train anywhere. And hopefully, you can also deploy anywhere, right? You don't have to stack with, you know, the data scientist environment and stick with that, right? And then you want AI at scale. You know, I will give you some real use case where you can see, you know, it's really important for some, let's say, banks to do in-transaction inference. And then you want some optimized AI ecosystem because, you know, when you talk about AI, it's really not just the model. The model is just one step in the process. You know, even if you have the model, you have to monitor it. You have to evaluate. There are lots of things. You have to do it iteratively, right? So you need a whole ecosystem to help you go through different stages of the AI stage. OK, I will talk about some... Those are really real-world use cases. OK, so as we all recognize that AI is being applied to lots of different use cases in a wide range of different industries, right? I just gave some examples here. Those are really just examples. They are not exhaustive at all. I highlighted a few here. I just gave some more information about those I highlighted in the last page. You know, those, again, those are real financial industry use cases, you know, things like fraud detection, clearing settlement, loan approval, claim fraud. Again, those are not exhaustive, right? So you can see that, you know, AI there is helping with different things, right? It can even go beyond this. So think about when you... I don't know how many of you have, you know, tried to chat with Webbot or tried to call into your bank and there's some, you know, robot, right, answer you. So the AI may have to deal with natural language and then do some natural language understanding. You know, you may pay check, pay bank a check, right? Those safer payments, the bank may have to process your image of the check, right? So AI may have to deal with image, right? Language, voice, all those, right? So they're really, you know, different data sources you may have to deal with in your AI model and you probably need lots of data to train your model and explore that, right? So let's drill down into one of the use case and see how exactly, you know, I'm just, I just want to use this as an example to show you, you know, how I went through the AI cycle, okay? And maybe you can apply that to some other use cases you're facing every day as well. Again, this is a real customer use case. So a large US-based bank, they want to apply the fraud detection model to their credit card processing application and they hope that they can do this for every transaction. Okay, so originally what they did is that, you know, they built a machine learning model. They deployed it off platform and then they try to invoke the model, right? From there, their transaction happens on mainframe, right? So when they invoke the off platform model, you know, there are different delays, right? Network or whatever, right? So they're seeing about 50 to 100 milliseconds, you know, the time may vary depending on the day of the time of the day and things like that. So what happened is that they couldn't really do that for every transaction. So they could only score about 20% of their transaction. So that means they couldn't really catch every fraud case beforehand, right? So for this particular customer, what they did is that they collocate the fraud detection model on mainframe so that, you know, the delay was reduced from over 50 milliseconds to under two milliseconds. And then they could invoke the model for every transaction and this could save them like, I believe 20 million is the number in risk deduction, okay? So again, this is just one example, right? But let's say if I ask you to build an AI model, what would you consider? What factors would you consider? What problems do you have to solve? Right, at least some of the challenges that you should consider, okay? Those are important ones. There may be some other ones, but let's start with those. So first of all, you probably would deal with hybrid environment in the sense that in this bank for example, right, their credit card transaction happens on mainframe already, right? Mainframe deals with about trillions of transactions every day. So, and then their data scientists probably would not have access to mainframe, right? Normally that's the production system. If you just want to play with something, you probably don't play on mainframe, right? They probably come to the cloud where they can easily access resources, different services. So naturally you are dealing with a hybrid cloud environment, right? That's one thing. The other thing is that, you know, think about what the data scientists. So, you know, put yourself in the data scientists shoes. When data scientists, they're trying to solve a problem. Probably the first thing that they want to focus is how can I solve this problem rather than how will this be deployed? Am I working in the same deployment environment? Probably that's not something they want to consider, right? At least not the top priority one for them. So how do you solve the issue? If the environment they're using is different from the environment, you will deploy eventually. Like in this case, the mainframe, they haven't even touched it. They haven't tested there, right? How do you solve that? And then we are talking about, you know, use cases in financial services, security. Security, security, security everywhere, right? Your data needs to be secured. Your platform needs to be secured. Your infrastructure needs to be secured. Everywhere needs to be secured. Otherwise you may have to pay a big dollar, right? For those, if the data is leaked or whatever, right? And then regulations. There are lots of, lots of regulations in financial industry, right? I'm sure you all know that, right? And there are even regulations for AI models. Like your AI model cannot be biased intentionally or not. That's if you use my voice to train a language model, you are biased because I'm not a native speaker, right? So you may not do that on purpose, but it is there, right? Model inference. We look at different use cases and some of the use cases really have low inference latency requirement. Some may relax a little bit. You may be able to do batch inferencing, but there are some you cannot, right? Model management. This is the ecosystem I mentioned, right? You need a healthy, optimized ecosystem to help you with every stage of the AI development. And even after you deploy it to model the whole life cycle and you need to monitor, if your model drifted over time, you have to retrain and redeploy it, right? So that's an iterative process. Okay? So how could open source help? Okay, again, I work for IBM. IBM is really serious about open source. Here are some numbers. You can see we are like 25 years plus in open source. There are like over 15K commits every month and we don't stop there, right? Open source software is really a major component of IBM's DNA. And IBM and Red Hat, we contribute, we're the largest corporate contributor to open source community, okay? So that's come back to the AI life cycle, okay? How did I use open source to help me build this AI model? Okay, what are the open sources I used? You know, I put something here. So first of all, when we talk about AI, right? AI modeling, what do we need to do? First of all, you need to understand the business problem. What problem you are trying to solve? And you also need to understand how you evaluate your model. Does it really help you, right? So you start with business understanding. That's the most important thing, okay? And once you understand the problem, you start to collect data. What data do you have? What data you need? Whether your data is good enough to help you to solve the problem, right? And then you gather all the data from different sources, you know, or you organize them together. Now you need to clean them. What if they're missing data? What if they're duplicate data? You need to, you know, clean up your data. And after you clean up your data, right? Your data centers come in, they look at different columns, different fields, right? They need to extract all the features, right? And they build the model, but sometimes they could be overfitting. So they may have to delete some features, right? So you can see that, you know, data preparation, modeling, it could be already a iterative process, right? And now you go down this path. Once you have the model, you evaluate it, right? How good it is. Is it in the quality that you can use it, right? And then, you know, if it's okay, you deploy it. But you see the big circle around it. You know, again, I want to emphasize, you know, AI model developing is really a iterative process. It's not something you can do it and forget about it. If you start it, you probably will never get over with it. You have to monitor it. You have to evaluate with new data coming in. You have to retrain it, right? But it's also, you know, how you gain insights for your company, right? Make better decisions. So it's a valuable process. Now we look at the inner circle, right? Let's talk about how I use the open source to solve the problem I mentioned before. First of all, you know, the IBM open source sample credit card fraud model. You know, I will share the link. The link actually is in the deck. You can grab the model. Actually, they open source the couple. You know, I used one of them for today's talk, but you can grab the GitHub repo and try it. And they also open source the sample data set. You know, the credit card data set is a little hard to get. So this is a thing to size the data set, but they open source it, so feel free to use it. And then for me, I use, again, I work for IBM, right? So I really used some open sources, but also some closed source tools for me to finish this cycle. But again, you know, based on your preference, you know, you may choose to use some different tools. For me, I do use Jupyter Notebook, but I use it inside, you know, Cloudpack for data, right? In that Watson Studio environment. It just make it easier for me to connect to different data sources. For example, I can just automatically ingest a code and things like that, but you can just go, you know, use Jupyter and handwrite some code, right? I use the Python packages, you know, I list a few examples here, there, you know, things like NumPy, Pandas, scikit-learn, those are, you know, very common Python libraries, right? When you build AM models. Onyx, why do I need to use Onyx? Now, again, let's come back to the data scientists. How can we separate them out from the deployment environment, okay? Data scientists, you know, they can use whatever environment they're comfortable with, right? To build their models. And then eventually, you know, our, you know, dev-opt pipeline could, you know, dev-opt engineer should be able to take their model and deploy to where we need that, right? Onyx is one way to do that. So what that means is that, you know, whatever the environment the data scientists build their model, they can convert that model into Onyx format, okay? And then the deploy, whoever deploys the model can take the Onyx format and then deploy to the framework, you know, the model serving framework they need. Okay, so that's why I use Onyx. And for me, again, if you look at the open source, the model is built in TensorFlow, okay? So that's why I use TensorFlow to Onyx to convert the model format. But I will also show, if you use TensorFlow and just save your TensorFlow model, you can still deploy that, let's say in mainframe, right? For my particular use case, I can show you how to do that. And I do use, this is Onyx, or not just Onyx, a model viewer. If you use TensorFlow, right, you can just view the model structure inside TensorFlow as well. So on the right side, what you can see are the open source tools I used when I built my model. On the left, those are the different tools I use, let's say when I deploy the model. Some of them, actually, you don't really need to know what is being used under the cover. For example, once I have my model, again, whether it's in TensorFlow format, or I can convert the TensorFlow format into Onyx format, right? If I keep the model in TensorFlow format, I can use TensorFlow serving, in this case, a container, and use that to deploy my model. When I use the Onyx, I use something called what's the machine learning to deploy my model? Okay, there is a no charge free version, if you want to try it out on Z. Okay, again, for the use case I'm talking about, yeah, well, this Onyx, so Onyx under the cover, if you use Protobuf format, actually, it's the same format as TensorFlow model. Okay, so if you look into the details, before TensorFlow 2.7, they have a bug. So when you train the model on, that's the x86, it's little Indian, and when you copy it over to mainframe, it wouldn't run. And 2.7, they fix the bug, so right now for TensorFlow model, you can just copy TensorFlow, you know, the BF file, I believe it's BF, to mainframe, it should run, but I don't think Pico file is Indian, independent. If you save the model in that format, you may have to do something with that. Okay, again, let's come to the right side. What I was saying is that I save the model in Onyx format and I use what's the machine learning environment to host the model. And when I import the model into WML environment, the deep learning compiler is already happening under the cover, so I don't really need to know, you know, a deep learning compiler is working and under the cover, deep learning compiler is built on the Onyx, you know, MLIR actually, right? You don't need to know all that. And the ZDNN is, you know, if you really want to make use of the AI accelerator Onyx 16, that provides your API to program, right? What, you know, how you use the AIU accelerator. Normally, you know, you don't have to use that. And I just mentioned that TensorFlow serving you can use to serve your TensorFlow model. Swagger API, you know, I listed there because after I deploy my model, I can make API calls to my model. And that, you know, WML can dump you a Swagger YAML file, you can just import, okay? Oh my goodness, time flies by. Hybrid Cloud environment, okay, I need to speed up a little bit. So what we really want to achieve here, right, is really once you organize all the data, that step, right, you have to do to gather all the data from different places and find out, you know, what data would help your business, right? Well, once you have the data, what we really want to achieve is tree anywhere, deploy anywhere, right? Again, for this particular case, I will show you how to deploy on Z, but it really based on your use case, okay? So when you build a training environment, especially in hybrid cloud environment, you know, you need to make sure it's secure, right? Especially for financial data, you know, that's one of the concerns, right, for some financial company to come to cloud because of data security concerns. So what you really need to consider is, you know, again, many things, network security, you know, I highlighted something, right? Like how do you isolate? Do you set up subnet? Do you set up VPC, right? I put some IBM services there, IBM cloud services there, but if you build this in other cloud provider, right, you can replace them with services with similar functionality, but again, you do need to consider all of that. Network security, identity access security, how do you authenticate your user? What access do they have, right? Even for services based on their role, they may need to see different information, right? Application security, how do you make sure your code is secure, your container image is secure, and et cetera, and data protection, data is the most important asset, right? How do you protect your data, right? At least some options, right? You have a confidential database, which means, you know, your database itself is encrypted, right? It's secure, confidential computing with your computation environment is also secure, and key management and other things, right? This is what I did. Again, I am IBM, you know, and I am from IBM Cloud for Financial Services team, if you remember, it's on the cover page. So what IBM Cloud for Financial Services provide is, you know, it has pre-conflict controls, like almost 300 of them, and then we have, you know, the architecture based on best practice, and we even have a Terraform template, so I can go into IBM Cloud, click a button, and you can choose whether you want to set up container-based, WSI-based, or classic. So for me, I just set up that environment, and then I have all the controls configured, I can go to my security and compliance center to do continuous monitoring. So this is kind of the flow. I'm just watching the time. So basically what I did is that first, I come and set up my secure landing zone for FS Cloud. This path two is, you know, you ingest your data, right? For me, I push my data into Cloud Object Storage, and that Cloud Object Storage is protected by my HyperProtect Crypto Service keys, okay? And then you see I use Cloud Pack for data to train my model. You know, I could choose to test on WSI, and then the model I bring back to mainframe to deploy. Okay, so once I have my model, right, I mentioned that there are different ways for you to deploy the model. In this particular case, if you want to keep your TensorFlow format, right, what I did is I just zip up my TensorFlow model, I copy it from my X86 environment onto mainframe, S390-based environment, and then I deploy it into a container, okay? Another way to do this is, again, if you have ONIX model, especially if you have really low latency requirement, you want to deploy that onto mainframe to utilize, you know, IBM Z16 has the AI accelerator, it's in-chip, right? It can really speed up the AI modeling, and speaking of that, right, I just highlight, you know, what model really take advantage of the AI US today. It's primarily for deep learning models, okay? For machine learning models, you can still use it, there are certain actions operators can still be speed up. The demo, okay, if you look at the deck I post, there is a link to the audio video you can take a look. I can quickly show you, I know I'm running out of time, but I think we also started a few minutes late, right? So if you don't mind, let me just quickly show you a few things. You don't see my screen. I can drag, oh, this is a different screen. I can drag the whole thing over one second, okay? So what you can see here is, this is the open-source model I mentioned, okay? Again, all the link, they are in the deck I posted, you should be able to access them, and there are two sample models in here. What I used is this one based on LSTM. There is another one based on GRU. You can try them out. And if you look at, right, this has the information about how you run this, and this link actually link to the open-source data set, okay? I also, I don't think I put it in the deck, but this is a very good red book. Basically, it covers almost everything I talk about, so feel free to take a look at this red book. Okay. And this is my Watson Studio environment where I train my model, right? You know, I think I'm running out of time. I don't know whether I have time to go through all of that, but basically what I'm doing here is that, let me come to the, I cannot see it on my screen. Let's just go to the top. Okay. So basically what you can see here is that, here I connect to my cloud object storage. Basically it use F3 APIs. You don't have to code, again, it's hard for me to see. So if you use Watson Studio, you can come here and inject your code. So that would generate some code for you to access different sources. And then what I want to highlight here is that this is the data loading, right? Basically I'm using Pandas to read my test data. Here's some sample of the test data. And then I do some preprocessing of the data. There's some functions defined to map the data. Like if you have strings, you want to map them to numbers and things like that. This is the data mapper. And then this is the data generator. This is where I create the model, right? You can see it has a few layers. And this, again, is the LSTM model I'm using here. And then here is where I train the model. This may take a while, right? Depending on your data volume, depending on the vCPUs you have, if I can ever score past this. Okay, model saving, you can see that, right? For one way, I just zip up the model and then I copy it over. The other way is after I train my model, I upload it into my cost bucket. So you can think of this as your devout process, right? After you build the model, you check it into some repository. So this is kind of like that. And then I test my model, right? I want to show you where I convert the model into Onyx format, model conversion. So you can see that this is a TF2 to Onyx model converter, okay? Let's see. I do want to come back to the slide just to wrap up. I see some people coming in already. So I think I highlighted those already, okay? Now, for your testing, right? I think for any one of you who have developed the AI model would know that when you test your model, right? When you do inference, you have to do similar data transformation just like what you did when you train, right? So this is an example, right? I have some test data coming in. So in this particular case, when you look at the Git repo afterwards, you can see that what we do is we have one transaction, we look up the past few transactions, we group them together, we pass it to the model. So based on a group of transactions, the model tries to decide whether it's fraud or not, okay? And the same data I can pass to the TensorFlow model, I don't have time to show, but basically, there is the container registry, you can download the TensorFlow serving container and this is how you run it with Docker run, for example. And this is how I import it into WML, right? You import the model, you can make API calls and think like that. And then I just want to sum it up. What we really talk about today is how did I at least use AI to use open source to train an AI model, right? And I do want to achieve that. I can train anywhere and then deploy anywhere, right? And again, the ecosystem is really important. And here are the different resources. It's in my deck, you can find out more information. And again, my contact is on the cover page. Feel free to contact me. Feel free to catch me. So let me stop here. I know I run out of time. Any questions? Thank you. Thank you. At least we fixed the technical issue, right? So you guys could see the slide. And thanks to Rajesh. Where do we see open source in 10 years? Personally, I think it will help us a lot, okay? If we all contribute, then we can make a bigger progress. So I do think it will help us. Yes. Thank you so much.