 Live from San Francisco, it's theCUBE. Covering IBM Think 2019, brought to you by IBM. Okay, welcome back everyone live here in Moscone North in San Francisco. It's theCUBE's exclusive coverage of IBM Think 2019. I'm John Furrier with Dave Vellante. We're breaking down all the action four days of live. Coverage of two great guests here, Elenita Elinan, executive director, quantitative research at JPMorgan Chase and John Thomas, distinguished engineer and director of the data science elite team. Great team, elite data science team at IBM and of course JPMorgan Chase, great innovator. Welcome to theCUBE. Thank you very much. Thank you, thank you guys. So I'd like to dig in, great use case here. Real customer on the cutting edge, JPMorgan Chase, known for being on the bleeding edge sometimes. But financial money, time is money, insights is money. Tell us what you do at the quantitative group. Well thank you, well first of all thank you very much for having me here, I'm quite honored. I hope you'll get something valuable out of what I say here. But at the moment I have two hats on. I am co-head of quantitative research analytics. It's a very small, SWAT, very well selected group of technologists who are also physicists and mathematicians, statisticians, high performance compute experts, machinery experts and we help the larger organization of quantitative research which is about 700 plus strong as well as some other technology organizations in the firm to use the latest, greatest technologies. And how we do this is we actually go in there, we're very hands on, we're working with the systems, we're working with the tools and we're applying it to real use cases and real business problems that we see in quantitative research and we prove out the technology and make sure that hey, we're going to save millions of dollars using this thing or we're going to be able to execute a lot on this particular business that was difficult to execute on before because we didn't have the right compute behind it. So we go in there, we try out these various technologies, we have lots of partnerships with different vendors and IBM is being obviously one of a few very major vendors that we work with and we find the ones that work, right? And then we have an influencing role as well in the organization, so we go out and tell people, hey look, this particular tool, perfect for this type of problem, right? You should try it out, we help them set it up. They can't figure out the technology, we help them out. We're kind of, like I said, we're a SWAT team, very small compared to the rest of the organization but we add a lot of value. You guys are the brains for us too, you got the math skills, you got the quantitative modeling going on and it's a competitive advantage for your business. This is like a key thing. A lot of new things are emerging. One of the things we're seeing here in the industry, certainly at this show is it's not your yesterday's machine learning, there's certainly math involved. You got cognition and math kind of coming together, deterministic, non-deterministic elements. You guys are seeing these front edge of the problems, opportunities for you guys. How do you see that world evolving because you get the classic math, school of math machine learning and then the school of learning machines coming together. What kind of problems do you see these things, this kind of new model attacking? So we're making a very, very large investment in machine learning and data science as a whole in the organization. You probably heard in the press that we've brought in the head of machine learning from CMU, Manuela Villosa. She's now heading up the AI Research Organization, JP Morgan, and she's making herself very available to the rest of the firm. Setting strategies, trying different things out, partnering with the businesses and making sure that she understands the use case of where machine learning will be a success. We've also put a lot of investments in tooling and hiring the right kinds of people from the right kinds of universities. My organization, we're changing the focus in our recruiting efforts to bring in more data science and machine learning. But I think the most important thing is, in addition to all that investment, is that we, first and foremost, understand our own problems. We work with researchers, we work with IBM, we work with vendors and say, okay, this is the types of problems, what is the best thing to throw at it? And then we POC, we prove it out. We look for the small wins, we try to strategize, and then we come up with the recommendations for a full-out, scalable architecture. John, talk about the IBM Elite Program. You guys roll your sleeves up. It's a service that you guys provide with your top clients, you bring in the best, and you just jump in, co-create opportunities together, solving problems. That is exactly what. How does this work? What's your relationship with GP Morgan Chase? What specific use case are you going after? What are the opportunities? Yeah, so the data science elite team was set up to really help our top clients in their AI journey. So in terms of bringing skills, tools, expertise to work collaboratively with clients like GP Morgan Chase. And it's been a great partnership working with Elinita and her team. We had some very interesting use cases related to her model risk management platform. And some interesting challenges in that space, but how do you apply machine learning and deep learning to solve those problems? So what exactly is model risk management? How does that all work? Good question. That's why we're building a very large platform around it. So model risk is one of several types of risks that we worry about and keep us awake at night. There's a long history of risk management in the banks. Of course, there's credit risk, there's market risk. These are all very well known, very well quantified risks. Model risk isn't a number. You can say this model, which is some stochastic model, it's going to cost us X million dollars today. We currently, it's somewhat new and at the moment it's more prescriptive and things like you can't do that or you can use that model in this context or you can't use it for this type of trade. And so it's very difficult to automate that type of model risk in the bank. So I'm attempting to put together a platform that captures all of the prescriptive and the conditions and the restrictions around what to do and what to use models for in the bank and making sure that we actually know this in real time or at least when the trade is being booked we have an awareness of where these models are getting somewhat abused, right? And we look out for those types of situations and we make sure that we alert the correct stakeholders and they do something about it. So in essence you're governing the application of the model and then learning as you go on in terms of... That's the second phase, so we do want to learn at the moment what's in production today. Morpheus is running in production it's running against all of the trading systems in the firm inside the investment bank and we want to make sure that as these trades are getting booked from day to day we understand which ones are risky and we flag those. There's no learning yet in that but what we've worked with John on are the potential uses of machine learning to help us manage all of those risks because it's difficult. There's a lot of data out there. I was just saying I don't want our quants to do two stupid things, right? Because there's too much stupidity happening right now we're looking at emails, we're looking at data doesn't make sense. So Morpheus is an attempt to make all of that understandable and make the whole workflow... So it's financial programming in a way that's come with a high whole scale of computing a model gone astray could be very dangerous. Absolutely, and it costs real money to the firm and this is all that you said. So a model to watch the model. So policing the models, kind of watching. Yes, I met a model. And you have to isolate the contribution of the model not like you say before, are there market risk or other types of risk? So you isolate it to that narrow component. And there's a lot of work, we work with the model governance organization another several hundred person organization and that's all they do. They figure out, they review the models, they understand what the risk of the models are. Now it's a job of my team to take what they say which could be very easy to interpret or very hard and there's a little bit of NLP that I think is potentially useful there to convert what they say about the model and what the controls are on the model or just something that we can systematize and run every day and possibly even in real time. So it's really about getting it right and not letting it get out of control. But also this is where the scale comes in. So if you get the model right, you can deploy it and manage it in a way that helps the business versus if someone throws the wrong number in there or in the class, we got a model for that. Yeah, exactly. And there's two things here, right? So there's the ability to monitor our model such that we don't pay fines and we don't go out of compliance. And then there's the ability to use a model exactly to the extreme, right? I mean, where we're still within compliance and make money, right? Because we want to use these models and make our business stronger. And there's consequences too. I mean, if it's an opportunity, there's upside, there's a problem, there's downside. You guys look at the quantification of those kinds of consequences where the risk management comes in. Yeah, absolutely. And there's real money that's at stake here, right? So if the regulators decide that, oh, a model's too risky, you have to set aside a certain amount of capital, right? So that you're basically protecting your investors and your business and the other stakeholders. If that's done incorrectly, and we end up putting a lot more capital in reserve, then we should be. And that's a bad thing. So quantifying the risk correctly and accurately is a very important part of what we do. So a lot of skill sets, obviously. I always say in the money business, you want the best nerds, right? I don't hate me for saying that. You're smart as people. What are some of the challenges that are unique to model risk management that you might not see in sort of other risk management? There are some technical challenges, right? So the volume of data that you're dealing with is very large. If you are building, so at the very simplistic level, you have classification problems that you're addressing with data that might not actually be all there. So that is one. But when you get into time series analysis for exposure or prediction and so on, these are very complex problems to handle. The training time for these models, especially deep learning models, if you're doing time series analysis, can be pretty challenging, right? Data volume, training time for models, how do you turn this around quickly? So that's how we use a combination of technologies for some of these use cases. Watson Studio running on our hardware with GPUs. So the idea here is you can cut down your model training time dramatically, and we saw that as part of the... Talk about how that works because this is something that we're seeing people move from manual to automated. Machine learning and deep learning to give you augmented assistance to get this to the market. How does it actually work? So there is a training part of this and then there is the operationalizing part of this, right? So at the training part itself, you have a challenge which is you're dealing with very large data volumes, you're dealing with training times that need to be shrunk down and having a platform that allows you to do that. So that you build models quickly, your data science folks can iterate through model creation very quickly is essential, but then once the models have been built how do you operationalize those models? How do you actually invoke the models at scale? How do you do workload management on those models? How do you make sure that a certain exposure model is not crumbling or it's not crashing some other models that are also essential to the business, right? So how do you do policies and workload management? And on top of that, we need to be very transparent, right? If the model is used to make certain decisions that have obvious impact financially on the bottom line and an auditor comes back and says, okay, you made this trade so and so, why? What was happening at the time? So we need to be able to capture and snapshot and understand what the model was doing at that particular instant in time and go back and understand the inputs that went into that model and made it operate the way it did. Can't be a black box. You can't not. Holistically, you got to look at the time series in real time when things were happening and happened. Happening and then holistically tie that together is that kind of the impact analysis? We have to make our regulators happy. That's number one and we have to make our traders happy. We as quantitative researchers, we're the ones that give them the hard math and the models and then they use it and they use their own skill sets too. What's the biggest needs that your stakeholders on the trading side want and what's the needs on the compliance side of the regulators? Traders want more, they want to move quickly. They're coming from different sides of it. Yeah, traders want to make more money and they want to make decisions quickly. They want all the tools to tell them what to do and then for them to exercise whatever they normally exercise. They want competitive advantage. They want that competitive advantage and they're also, we've got algo trades as well there. We want to have the best algo behind our trading. And the regulators side, we just want to make sure laws aren't broken. They're just auditing the whole. We use the phrase model explainability. So can you explain how the model came to a conclusion, right? Can you make sure that there is no bias in the model? How do you ensure the models are fair? And if you can detect there is a drift, what do you do to correct that? So that is very important. But do you have means of detecting sort of misuse of the model? Is that part of the governance process? That is exactly what Morpheus is doing. So we are, the unique thing about Morpheus is that we're tied in to the risk management systems in the investment bank. We're actually running the same exact code that's pricing these trades. And what that brings is the ability to really understand pretty much the full stack trace of what's going into the price of a trade. And we also have captured the restrictions and the conditions, like it's in the Python script. It's essentially Python and we can marry the two and we can do all the checks that the model governance person indicated we should be doing. And so we know, okay, if this trade is beyond, operating beyond maturity or a certain maturity or beyond a certain, you know, expiry, we'll know that and then we'll tag that in. And just to clarify, Morpheus is the platform name of the platform. Morpheus is the name of the model risk platform that I'm building up, yes. A final question for you. What's the biggest challenge that you guys have seen from a complexity standpoint that you're solving? What's the big complex of challenges? You know, you want to just be rubber stamping models. You want to solve big problems. What are the big problems that you guys are going after? So I have many big problems. The one that is right now facing me is the problem of metadata, data ingestion, getting disparate sources, getting disparate data from different sources. One source calls it a Delta, an IR Delta. This other source calls it something else. We've got a strategic data warehouse that's supposed to take all of these exposures and make sense out of it. I'm in the middle because they're there, probably have a 10 year roadmap, who knows. And I have a one month roadmap. I have something that was due last week and I need to come up with these regulatory reports today. So what I end up doing is a mix of a tactical strategic data ingestion and I have to make sense of the data that I'm getting. So I need tools out there that will help support that type of a data ingestion problem that will also lead the way towards the more strategic one where we're better integrated with this. John, talk about how you solve the problems. What are some of the things that you guys don't give the plug for IBM real quick because I know you guys got the studio. Explain how you guys are helping and working with JP Morgan Chase. Yeah, so I touched upon this briefly earlier, which is from the model training perspective, Watson Studio running on power hardware is very powerful in terms of cutting down training time, right? Now, but you got to go beyond model building to how do you operationalize these models? How do I deploy these models at scale? How do I define workload management policies for these models? And can I can do their backbone? So that is part of this. And model explainability we touched upon that. To eliminate this problem of how do I ingest data from different sources without having to manually oversee all of that? We need to apply auto classification at the time of ingestion. Can I capture metadata around the model and reconcile data from different data sources as the data is being brought in? And can I apply ML to solve that problem? So there are multiple applications of ML along this workflow. Talk about real quick comment before we break. I want to get this in. Machine learning has been around for a while now with compute and scale. It really is a renaissance in AI. It's great things are happening. But it's what feeds machine learning is data. The cleaner the data, the better the AI, the better the machine learning. So data cleanliness now has to be more real-time. It's less of a cleaning group, right? It used to be clean the data, bring it in, and wrangle it. Now you've got to be much more agile. Use speed of compute to make sure that your qualifying data before it comes in to these machine learnings. How do you guys see that rolling out? Is that impacting you now? Are you thinking about it? And how should people think about data quality as an input to machine learning? Well, I think the whole problem of setting up an application properly for data science and machine learning is really making sure that from the beginning you're designing and you're thinking about all of these problems. It's data quality. If it's the speed of ingestion, speed of publication, all of that stuff, you need to think about from the beginning, set yourself up to have the right element in it. It may not all be built out. And that's been a big strategy I've had with Morphosis. I've had a very small team working on it. But we think ahead and we put elements of the right components in place. So data quality is just one of those things. And we're always trying to find the right tool sets that will enable us to do that better and faster, quicker. And one of the things I'd like to do is to upscale and uplift the scale sets in my team so that we are building the right things in the system from the beginning. Well, that's math too, right? I mean, you talk about classification. Yeah. Getting that right up front. Yeah. Mathematics is... And we'll continue to partner with Elenita and her team on this. And this helps us shape the direction in which our data science offerings go because we need to address complex enterprise challenges, right? I think you guys are really onto something really big. Love the Elite program. But I think having this small team thinking about the model, thinking about the business model, the team model before you build the technology build out is super important. That seems to be the new model versus the old days. Build some great technology and then we'll put a team around it. So you see the world kind of being a little bit more. It's easier to build out and acquire technology than to get it right. Yes. That seems to be the trend here. Congratulations. Thank you. Thanks for coming on. We are Keeps Conversations. Here we're live in San Francisco for IBM Think. I'm John Furrier, Dave Vellante. Stay with us for more day two coverage. Four days will be here in the hallway and lobby of Moscone North. Stay with us.