 Oh and welcome. Yes, my name is Shannon Kemp and I am the Chief Digital Manager of Data Diversity. We'd like to thank you for joining today's designing a successful government citizen data science strategy sponsored today by Data Robot. Just a couple of points to get us started. Due to the large number of people that attend these sessions, he will be muted during the webinar. If you'd like to chat with us or with each other, we certainly encourage you to do so. Just click the chat icon in the bottom middle of your screen for that feature. For questions, we will be collecting them by the Q&A section in the bottom right hand corner of your screen or if you'd like to tweet, we encourage you to share highlights or questions via Twitter using hashtag dataversity. As always, we will send a follow-up email within two business days containing links to the slides, the recording of this session and additional information requested throughout the webinar. Now let me turn the webinar over to our new host Ron Powell to introduce today's webinar and speaker Ron. Hello and welcome. Thanks, Shannon. Welcome everyone. I am Ron Powell, an industry analyst and executive producer of the World Transform Fast Forward series. I am excited to be the moderator for today's webinar, Designing a Successful Governed Citizen Data Science Strategy. The webinar will address the importance of AI and data science for competing in today's digital economy, how to approach your journey into citizen data science, and it will also provide a wealth of best practices and lessons learned from successful early adopters of augmented data science. It gives me great pleasure to introduce today's speaker, Jen Underwood, Senior Director for Data Robot. Jen has over 20 years of industry experience and analytics. She has been helping analytics professionals learn how to solve complex problems with machine learning in the emerging area of citizen data science. Jen will provide insights into developing an additional roadmap, staffing, training, mentoring, and ongoing governance. Welcome, Jen. Thanks, Ron, for having me today. You're welcome. So essentially, yes, I'm going to review an introduction to the topic itself and the role that the analytics center of excellence can play with forming an AI center of excellence, the path to becoming AI driven, and how citizen data science can be effectively rolled out in organization. And then we'll take some questions at the end. It will be filled with demos so folks that like to see AI in action, we will have that as well. And I just want to open up with comment from Tom Davenport on competing in analytics. The decision in making techniques and technologies to automate will be that next competitive gale ground. Those are using business rules, data mining, analytics and optimization today are the shock troops of this next wave of business innovation. And what we're seeing is that's become a reality. The other thing that's become a reality is that folks that are now you may be data driven. However, you're not necessarily competing with other rhythms yet. You may not be machine learning or AI driven. In any case, the early adopters are taking advantage of that gap today. And here's a simulation by McKinsey. I have the link on the bottom of the screen there that if you want to look this stuff up and you want to experiment yourself with some of the numbers, you can do that. And looking at AI being one of those conversation pieces in all the different quarterly reviews, these are C level conversations. And you're looking at cumulative changes in cash flow. So again, these are high level stakes that you're looking at when you have competitors stealing, I think it was this forester that has said it's several trillion that folks can be stealing in this economy from a market share perspective. So where does citizen data science fit? Now, I feel like when I'm working with the early adopters, I'm having the same exact conversations that I had in 2010. And, you know, through about 2010 to 2014, when we were starting to adopt self service BI, it's just a bit different. Some of the similar conversations are we, we're not going to, we can only have data science doing this or we can only have the certain group doing this. But we are seeing in the market with augmentation and automation, as I'll show you today, there is a much wider base of what I would call you need to be data study and able to query data to be effective with these solutions. But to use them, you can, you can have any more people empowered effectively to do that. And to properly do that, you would not necessarily want those. We saw this happen in self service BI. We just have different units popping up. I don't know if someone's not muted, could they with themselves if they're not speaking? Thank you. And when you're looking at different areas of the organization that may be leveraging some of the tools out there, what you want to avoid is bottom up and it's not coordinated. There's a much deeper level of skills and experience and understanding what the output means. And you can easily find yourselves making mistakes that can be embarrassing mistakes for the organization. One of the examples was the Amazon last year and not to pick on them. There's plenty of other groups that that has these embarrassing mistakes of rolling out a machine learning model that accidentally was discriminating. Because the words inside of the model itself that was using were still using gender based words for candidates. And there's many different ways that models can go wrong. And rather make them may have to be a scary session. I want this to be a positive session. But I do want you to be aware that you will need training, you will need to properly approach these things to be successful. And not underestimate the learning curve for doing this. And you can do it. The tools are much better today. But just be aware that you'll probably be infusing this into existing centers of excellence. And when I say existing centers of excellence, you may have it's an analytics COE or you may have a cell service BI governance plan that now gets expanded to add machine learning into it, or another program like that within your organization that defines the vision for these technologies, creates a roadmap of use cases, you know, how are you actually going to solve problems with the technology? It's not a toy. You know, it's really fun. And it's really cool. But you ultimately wanted to deliver value to the business. And that's really important. Providing the technical architecture and partnerships, you don't want to have a bunch of siloed solutions. You really want to leverage, you start thinking 360 views of the customer, customer experience, all these different types of data sources that you've built through the years. Now we start taking analytics further with AI and you'll want to leverage those. So defining the right tools, the governance programs itself. And when you do have models developed, even monitoring that they're doing what they're supposed to be doing, if you were in a regulated industry, and I'll show some documentation today that we have for regulated industries, other industries can certainly leverage themselves from a governance standpoint, just understanding even what went into the model, how the model is supposed to be working and, you know, understanding what the output actually means. And of course, orchestrating this throughout the organization. So three steps to becoming AI driven and really the key message here is upskill your existing team. And you can use existing tools. You'll apply automated data science and collaborate with with export mentors. And they may be mentors from within some of your own organizations, they may be mentors or vendors that there's many different vendors that will provide mentoring for you. So there's a couple of different ways that you can do this to really upskill your existing team at Data Robot. We have a two day course that gets you familiar with the tools, however, there's also a process that folks will go through to learn how to identify everything from the right types of problems to solve. And in AI, we call that use cases to, you know, implementing a project, solving a project with machine learning and then operationalizing that and putting AI to work in the business. So there's a few different things when we think about this. So what is Data Robot? I've mentioned this a few times. We have it's a staff now of over 500. Most of them are data scientists. The most data scientists I've ever worked with in my career, which is it's a special experience in and of itself. We've built, I believe we surpassed a billion models. Now we will be having probably some PR around this quite soon. But but we're very well known. We're used in some of the top organizations in the world, banks, insurance companies, syntax, marketing, many different areas, and areas around the world. So I was in Tokyo two weeks ago already, you know, talking to them, Asia is an early adopter of a lot of these types of technology. For folks that are familiar with Gartner's Magic Quadrants, we are, we debuted this year, we didn't participate last year, but we decided to participate this year, we debuted as a visionary. And one of my quotes that I've got here is it's the standard for augmented data science and machine learning. So this is what enables us to enable citizen data science is easing some of that process responsibly with automation. So speaking of automation, when you think about scaling analytics, and you hear these words automation, it's going to replace our jobs and all these scary things about the data robots, or the data robots for the robots in general coming. You know, we have these same, you know, fears in that BI movement from traditional BI to self service BI, oh, what am I going to do, etc. I can tell you the folks that understood how to develop models responsibly. In that case, it was dimensional models instead of machine learning models with data warehouses. Those folks were the heroes and the leaders that could help the organization be successful with it with an easier tools and technologies. They're more productive and it really scales what we would call the value that you're delivering. So it's not a replacement of data scientists is not at all. In fact, it allows them to focus on more complex, more intellectually engaging work that they want to do while, you know, reducing some of the busy tedious annoying work that they wouldn't necessarily want to work on. So, you know, when you're thinking about automation, it's really about the scaling message of intelligence and really being able to do more. And then you've ever been able to do before. A common team and that you know, just emphasizing again that data scientists the key role that person plays. They're part of the team. Now you may have them as a mentor from data robot or another vendor. They may be a data scientist or a trained business analytics professional that understands how to use these solutions effectively. That's really mentoring some of the rest of the team on getting started and how to how to use them and interpret the results and what the results mean. But we'll have what we call citizen data science. They can be anybody not necessarily a PhD in data science. It's not their profession, but they're leveraging these technologies. And we are seeing that. So they could be somebody's a business analyst professional. It could be someone when I say data engineer, maybe it's a database developer, maybe you're developing data pipelines or software developers, you know, embedding machine learning into your software itself or different apps in the organization. How you think about all the different ways that you can leverage these technologies. I've seen some really neat easy buttons out there and we even have an easy button. I'll show you our easy button. But keep in mind that you'll want to democratize this correctly. And often when you're looking at some of the easy buttons in the BI tools, most specifically they'll have to mention no models that aren't necessarily the data design, the input going into the predictive algorithm has not been optimized for machine learning. So there's a lot of gotchas there. And what you want to really do is instead of going backwards from the BI tool and developing a model, you really want to develop your model, make sure it's been validated. It's been approved. You went through a process in the organization that said, yes, now we want to put this to work. The best way to deploy government models is through a data or ETL pipeline. Then you'll store the results. You can leverage a lot of the existing role-based security and governance and then allow folks to consume those predictive results within the applications and within the reports used. Now granted, there are other ways to use predictive models. When you think about scaling across an entire enterprise, that requires all of the auditing and chain controls and different types of things. You'll ideally probably want to deploy in an ETL pipeline. There's a few ways that you can use government predictive models. And I specifically put government predictive models here so that you know that you're consuming this from the approved deployment. Is that we can do things like tableau, what is analysis? I've got what is analysis and click. There's, you know, consuming predictions and Power BI and other visualizations. So you can take the predictive output now and put them inside of solutions that your business is currently using and to sell usually the concept within your organization. Folks don't typically understand what AI is. I think one of my favorite pieces of feedback and lessons learned from an insurance company in London that's been super successful. It said, we couldn't believe it, but our data science team was noticed for the first time in many years. And the executive, because we were making a difference on the bottom line, he wanted to meet AI. So he thought AI was a person. You know, he says his data scientists didn't even know how to react to that or what to say. But often the point being is your stakeholders that can get the most value, especially some of your executive stakeholders, they'll understand AI better if you show it to them in the world they already know, meaning maybe it's a dashboard, maybe it's a sales force application or some CRM, that type of thing, some pricing type application. It's those worlds that your consumers and your users and stakeholders will understand. And just to give you a feeling for some of the results that we've seen from early adopters, and these are the citizen data scientists. I really want to make this point clear. This first one's one of my favorites. It's a global retailer. They had an existing data science team that has been doing projects and research. And one of their one of their executive stakeholders has said, we don't want our talent working on theoretical data science research projects. Let's leave that to the big tech funders. What we would prefer is for our data scientists to be ruthlessly practical at delivering value to our business. And that's a key point when you think about all these data science groups that they really mean well, but no one really understands what they're doing. There's this big gap between the business. The beauty of citizen data science is that a lot of these folks already developed, you know, that are already engaged in the line of business developing reports. They understand the context of the data. They understand the business. You can get these types of results. In this case, there was a 400 million estimated savings from, you know, just making better demand forecasts because of all the supply chain. And this is from the last Black Friday. So just to give you an idea, this is from what November of 2018. And these results were achieved in less than three months. Another one, and this is one of my favorite stories, and I've met this business analyst. I had to meet him. I was one of those things because I needed to hear it from the person individually to just really understand what he went through. And when I show you DataWorld, I'll show you where this person, this particular data analyst, found the result. It was an anti-money laundering use case in a bank, and a business analyst or a data robot, and they found money laundering and it's estimated at least 150 million class savings for that particular finding. And again, the bank has certainly had data scientists, but having that business subject matter expertise, you know, looking at different charts, understanding the context, you can find much more value. Another one, and this one's another one of my favorite case studies that I talk about is TD Ameritrade. So historically, their data scientists wouldn't even touch a project without an ROI estimated at 875,000. So all lots of little, if you think lots of little quick wins that data analysts and business analysts or BI professionals could be doing, or just ignore. But with data robot and automation and augmented citizen data scientists, they're now able to quickly create models on all those little projects that they see, and instantly, you know, deliver value lots, lots of little quick wins to the business. Another one, and I picked this one because I like the fact that it's just a tiny bit of reduction. So if you think about, quite often, when automation is added to our early adopters, they're already coding Python or coding R, or using something like SPSS or SAS in an organization. It's not that they've, they don't use any sort of predictive analytics, it's that they've made an improvement with automation, because they can do much more faster and get smarter with it. And this is an example of just a small, tiny, 0.1% production in patient length stay for a hospital will deliver estimated cost savings. This is for, I guess, 8 out of 38 hospitals implementing this model was 10 million. And again, this group, it was not the data science group necessarily. It was, in this case, the software development group that did this. So not PhDs, but able to make a massive impact to the business. So I always tease when I see the slide, just hit data and start, right? It's that easy. Well, yes and no. We do make it very, very easy to get started. But we also make sure that it's easy to make mistakes when you have some of these new technologies. So in this case, we faked in and when you think about what are those 200 data scientists doing at DataRobot, they're cooking in their best practices and all the lessons learned through the years and how to build high performing models, provide warnings and guardrails and my example here with target leakage for folks that are not data scientists on the line. That means you may have accidentally provided the answer. Your model's too good to be true. And DataRobot throughout the development process will issue different warnings for you as we come across things that may not be quite right. Now historically, I was a wannabe data scientist when I was earlier and my first models were developed in 2003. I would have no idea other than my results not being so good that I might need an error. So it's these types of things when you think about the improvement in the quality and how far tools have come, it's much, much easier. But also a bit more robust from a quality control perspective and I'm going to talk about more about that as I go into the demo. So some things to look at as I give you a demo today. Some key capabilities that is baked into DataRobot to help non-data scientists get started. And even data scientists that may need some extra help because there is a lot to know, of course. It's easy to get started, yet I'm going to show you how flexible it is. And for people that are data scientists, we have about 50 percent data scientists that use our solution. 50 percent of data analysts and BI professionals and software developers, so folks that are not. Those best practices. Automated data quality detection. Some automated feature engineering. That's the data prep for making the data ready for machine model. Transparency and you'll hear a lot about this in compliance documentation, which is really important for understanding what drives the model and avoiding those mistakes that you don't want to make in a model. And making sure that it's trustworthy and something that you can make automated decisions upon. We'll also show you human-friendly model explanations. Some built-in auditing, health monitoring, and management. So once that model is built, making sure it stays healthy and, you know, if it is having errors, where is that happening? So let's see a demo. I'm going to go ahead and show you a bunch of different demos on building a model and also using a model. So in DataRobot, there's a few different ways that you can get started creating a predictive model. You can connect to a data source, so any kind of ODBC, JDBC, maybe you have a Snowflake data warehouse, those types of things, or Hadoop cluster. You can go ahead and leverage that. If you are Ralph, so if you're, say, an Amazon customer and you're using some S3 files as your data lake per se, you can leverage that. And just, you know, for the sake of, I'm working on my local laptop, I'm just going to toss up a file here and upload that. So essentially when you're working with predictive models, I mentioned earlier, dimensional models, you know, not necessarily optimized for machine learning, you'll want to feed it some data that has an outcome, what you want to predict, and columns, we will call features or columns or dimensions that will describe that. So let me take a look, let me scroll down and show you what I mean by that. So DataRobot right now is doing some exploratory data analysis on that file. And my outcome, in this case, this is a lending club. It's a, you know, the use case we're going to be talking about today is loan, loan repayment. Are you able to pay your loan back or not if, if lending club issues are loaned? So what we want to predict is going to be is bad. We're going to use that as a target. But when I talk about all these different things that, you know, formatting your files properly, we have employment titles, lengths of time of employment, and under each column I can now look at the structure of my file. And this might be structured a little differently, right, than we would have in dimensional modeling. So this is more of what we would call a flattened model or a flattened view of data. All these different columns that describe that outcome. So that is something different. And right here, if I wanted to, so most of my citizen data science folks, and even most of the data scientists, we'll just click the start button. Data robots, smart enough, it's, it's known that the optimization for this type of problem, which is a classification problem, is long lost. Now if I did want to override everything I did as a data scientist, we could go ahead and do all sorts of fancy extra things here if we wanted to do that. But I'll go ahead and I'm going to click this big easy start button. And there's a few things that are going to happen here. And as they happen, I want to describe how it used to be and how automations kind of changed the game for machine learning and enabling you to even attempt to do citizen data science. So what we're seeing happen right now, data robot is automatically creating cross-validation holdout partition. So it's taking samples of the data and splitting it out so that, so that you don't have to know that you would even need to do that so that it will prepare. It has provided this warning to me so that is something that in this case it's triggered a warning I should be looking at, loan status, might be the answer already. So that'd be something I'd want to look at. The other types of things that it will add, and this is something that's really changed the game when you think about predictive modeling. Now I'm going to be looking at, and I'm going to go ahead and bump this up to 20, instead of building one model at a time the way I used to have to do it, and I drag it up all these different steps in an ETL looking tool, and then see how I did and give it a try again and where did it make mistakes. Right now data robot has 20 models running at once. It's got all sorts of different open source. I believe it's over 100,000 packages have been pre-built with the feature engineering steps, and I'll explain what that means in just a moment. But we've got Python in here, we have R, we have TensorFlow, Valpal, Rabbit, all sorts of different algorithms will be automatically run at the same time scored in ranked for me. So as that's running I'm going to go ahead and show you to make the most of our time today some of the outputs of this. So here's an example of the results. We have 80 models now that within a matter of a few minutes have been generated. In the past that process used to be much longer to do, usually months, to finish, you know, running those many models. We have a few different things. We have some badges here. So in my citizen data science we have recommended for deployments. This is the one we're recommending in case you don't know what that cross-validation score means. We have most accurate tags and we have fast and accurate. And digging into the model itself, this is what I think you having done the hard way is these blueprints. Think of blueprints of all the automated feature engineering that's been done for you. And it's been done for you for each algorithm type. So here we have categorical variables that's been split out, numeric variables with the concept of one hot encoding of these variables. So for each one of these steps, the ETL per se has been done for you. And if you want to look at the documentation for what does that step mean, what did DataRobot actually do, and you still want to do it the hard way, you know, go for it and take a look at that. But each one of these algorithms has different recipes or different blueprints. This here, I don't even think in the 10 years that I was learning and trying it the hard way I would have ever figured out some of these pre-processing steps. So this is the beauty when you think about the busy work and some of that knowledge and augmentation in data science. This really helps you develop a much better high-quality model by leveraging the brains of 200 data scientists. Now you need to go in as a human and validate that what it found makes sense. And how you do that is with the Understand feature. So now we can go into Understand and we'll see what we call Feature Impact. Feature Impact will list my variables in order of importance of how it influences that outcome. So the top one on the list, Revolving Utilities, we have the title of the loan itself. How many inquiries were in the last six months? Those are how many creditors are calling and you have asked about your credit standing. So how many loans are you trying to take right now? Successfully or unsuccessfully? The description is the fourth one down and what we found in a lot of different models and we've had it was Scott from Farmers Insurance talk at Tableau's conference about this finding as well. Having these long text fields of descriptions of why you know something is happening, that really helps improve the performance of the model. So that's showing up quite high here and it's going to show up in a couple different places but can certainly improve. And from your business stakeholder standpoint when you think about communicating results and using results in the business, this is one visualization that's really helpful for non-data scientists or your executive sponsors to understand what really matters most. So in this case having a public record means maybe you know that person had you know criminal history or whatnot doesn't really affect whether they're going to pay that loan back. It's not really that important. You can export these or use them in the API to data robot and we can surface these up inside of the BI and visualization tools because they have really important information for the business decisions on. Taking it a little bit further, this is the particular visualization that the banking business analyst found that money laundering in. It's called partial dependence plot. The beauty from a BI analytics folks that might be on the line with this one is it gives you the features but it also gives you the value ranges. So if I click on for instance annual income what we have is all if what this means is if all the other variables are kept the same this is how much annual income will influence that target outcome by the values itself. So looking at you know 30 to 40,000 I'm a higher risk to be given a loan. That makes sense if you think about it. And then from 40 to 60 I've become reduced and then about 62, 63,000 I'm even lesser of a risk and for some reason between 85 and 90 I pop up a little bit but after 90,000 I'm a very low risk for you know not paying back my loan. But as you dig into some of these variables and you look at the pattern you can get a lot of really interesting insights with this particular visualization and again you can surface these in different solutions. Digging even further we have it's called prediction explanation. So really really going and looking at what matters in human language and now we can look up for the outcome of predicted to fail. We have a real tiny income here so that makes sense so we're validating that makes sense. Medical bills and there's been three creditor inquiries in the last three months so this is why for each individual prediction that one might fail. On the flip side we could look at you know why one might not fail or why why somebody's predicted they will pay back you know they're looking at the payments or they're looking at their income it's much higher 225,000. So this is how when you think about validating a model making sure things make sense opening up that black box to make sure that you understand what's driving those outcomes then you can explain it. This really important stuff. Some other things that my particular non-data scientist audience really also appreciate are things like maybe they just want to put this in a word cloud or they want to have maybe just a summary of all the ones especially things like fraud or just detecting you know irregularities and transactions and a summary of the anomaly detections or you want to do some deeper text mining maybe you want to have business rules hot spots are not a big list or a big table of business rules so if you think of things like decision trees this is the type of information you'll find in hot spots. So those are some other visualizations that you can get and again when you're explaining and using to the business that becomes really really helpful and once you have a model your business trusts and you say yep we can use this model I'm going to go back to the model screen let's go ahead and deploy that simply into production and this is another thing that used to be time consuming and quite difficult to do with deploying models and everything required custom coding and it was just really really difficult to get things rolling and a lot of a lot of models were never deployed they were just shelfware and in a presentation somewhere and to get value from the model you really want to deploy it and put it to work and to do that a lot of my my folks might use batch prediction so that's just uploading a new set to be predicted with the file most commonly what we'll have is just clicking this button here add to deployment and you'll have a deployment and I'll show you that in a moment deploying it to Hadoop so being able to do say in in database scoring a few things like Spark or Databricks and just being able to score and have those models left there prime is having outlaws you to download the code itself so a lot of my things we'll have to actually show the code type thing and we have other types of options here as well for downloads and prediction application and once you have a model developed now we can check the health of it in deployments and start using it in all those bi applications that I talked about now you have a model that you can put some governance around this and you can have different permissions and levels on this and track the history of performance over time and let's look at click let's just do the integration with us so I should have some data on here and we'll see a few things we can check the service of the health I can look at you know how many models have I run through here how often have I used this how has the predictions drifted over time and the greens are healthy the yellow is not so healthy so I want to take a look at that it'll also give you the field so in this case it's the revolving utility it changed and then we've got this red red one here so we can dig in a bit further data robot can also email you and let you know that there's issues you can automatically update it or you can decide to override it without changing your endpoints to your application but keeping and making that audit trail and to integrate in general we provide you code stuff with csv and json but we also have integrations with popular bi tools and that's really what I think about making this so much more accessible after you have a developed model to be governed and used this is an example in tableau of using a data robot model we have that same one one is bad i have some of these insights that we've seen so that feature impact per se i've built as our customer dashboard can we put this in full view and some things that we can do here right we saw those top default reasons that that feature impact again in here but now we can make it interactive and we can filter on it and that type of thing i can look at specifically we have default probabilities and predicted prepayments so these are predictions indeed a robot the other thing i put in here and i've done this differently in a few of my different demos it's just going ahead and put in the reasons why so this is the default reasons the service or name itself the nation the loan age and refinance i could look at a different one we've quickened loans the loan age again we have the current lifetime value but we can look at these different reasons why for each individual one which helps the business manager you know really look at what are the risks there and understanding the reasons for each individual account to take account level action another thing that we can do and talk about what can do with the model is there's an extension called the what if it's an extension for data robot within tableau itself that allows you to drag and drop and then point to the predictive models that you have deployed so i can go to let's see let's see if we can use this one we can show you know what's most important in here and we can start doing some what is simulation analysis on these so in this case i don't have to be i could be anybody in the business using some of the tableau solutions and again you know democratizing ai so that anybody can use this we can easily put in you know some different values in here let's do 40 000 home improvement and get predictions and we'll get predictions here and we'll also get some explanations so this one we have a good probability so if someone was just asking us some questions and we wanted to get a feeling for what that outcome would be we can experiment here with some different outcomes so maybe instead of that maybe it's a small business line and we changed that to 95 percent and 35 and let's see if that changes the outcome it's a little bit but not a lot so we can take a look at some different scenarios here add that to comparison and we could keep experimenting with different scenarios to see knowing when does something influence an outcome and not we have similar capability there's a few other ways that you can use governed models for folks that let's see if i can find there's another example and this is ones from click they've developed a solution that allows you to also do some what is analysis on the fly they've got a couple different things here that they've done they'll allow you to in their connection they'll allow you to create a model if you'd like you can explore the different results within click and use the same sort of green and gray explanation that you would like and click but you can also do similar what is analysis that i showed you in tableau and how they've done this is this call this is calling a deployed model within data robot in my screen and i can change the variables here so i can look at the different mode amount i think it was the income that really made a difference on this one and it should change on the fly as to whether or not let's see if this will work for me today the different installments and outcomes there we go okay yay so this type of thing again can be really powerful because the organization is already using predictions in their workflows another example that we have is from using power vi and i'm going to go to a public one here and this is a healthcare scenario and we have this published to the power vi public data story community this one's developed by Nathan Patrick Taylor and you know for as long as i've been doing predictive analytics and you know building we call smart reports and dashboards it's really just adding a column so when you think about what are you storing or storing those probabilities in report and then adding them into your existing workflows so in this case we have somewhat of a of what if here this is not an exact what if but we're able to search at what are the different outcomes for different nurses and you know should they be worried about anybody and who should they be most worried about they can dig in and look at they can look at by location which i think is very interesting when you think about you know readmissions for hospitals and again having the reasons why the factors for medical specialty is that we're hovering over there until that's this to the reasons why taking a little making it a little easier to see you know what are the high abilities what are the probabilities and then the other thing that we have here again and I showed you one way to look at word clouds and data robot but bringing those into maybe some of the solutions and there's many different use cases for this free text or under state and you know how this free text will influence an outcome so here we have heart failure is most probable you know a high reason for readmission rates where hemorrhaging and kidneys are not you know so just understanding this can be really powerful information within the existing flows of your business process and for folks that just love excel like mr. Underwood does I always tease about my husband he's what I would call as my design solutions throughout my career he's one of my users mr. Underwood loves pivot tables and if it's not in a pivot table he will not use it so influenced by mr. Underwood data robot partnered with a company called C data and they have it's an extension in excel that allows you to point to data robot and the models that I have permissions to view I can now consume and use and run batch predictions on so this is an example of one here in my predictions table and you can see on the bottom here this is really what I want I want to get that information but I also want the predictions and the numbers themselves in in the value and I can get that right within excel so then I can explore it mr. Underwood can now look at you know he works in the energy sector you know what particular in his case energy it's predictive maintenance what types of solutions should he be focusing on first to make sure that there aren't energy outages in that case and I could continue to show you many many more hopefully I've influenced you again the models I'm showing live inside of data robot they're governed they're managed and you can see if I can go back to that screen that has the deployment dashboard you'd be able to see the different results in here in the health event so going back to our deck and find my deck now after all these demos some other things that we'd want to look at and I'm just going to leave this partial strength I'm going to show you one other thing that's important is as you're going to developing these projects it's really important to understand the data source itself and making sure the data in its garbage and garbage out so you'll have an executive summary you think about governance and governance of a process who are the stakeholders why was this model developed you'll have an overview of the data source that was available and almost always there's things missing in the day so you'll say well these are the things we know are missing and this is the appropriate use of the model or inappropriate use some of the preparation steps some of the model methodology used and you change change over time you know your change in auditing and the beauty is in at least in data robots we've automated that for you so we have it's a document that's automatically created that will allow you to get a jump start on all of that documentation within a model so for a lot of our banks that still use this compliance I guess I can present this this compliance documentation so if I go back to the model and I say I want to have I can find the screen with this let's just go back to my model where the documentation is is under compliance and again we have a big warning because this hasn't been a fully validated model on all of the warrant but if I didn't have this model I could this morning I could go ahead and generate a full report of the high level summary that has all the documentation needed and then all the different steps that you would need to add to personalize that documentation for your scenario so for the governance standpoint this is really quite helpful to jump start your program and let's go back to presenting so how do folks normally get started with this typically what we'll find is we'll have people's call I mean I'm definitely one of those people I was skeptical when I first saw data robot and like well I need to play and then I need to see it and certainly hearing people that have been very successful in meeting with them the past two years has been phenomenal so now I'm no longer a skeptic but often in an organization what you'll find is you'll you'll pick a use case let's just be let's pick one high value use case and you do that with your account team to find what what makes the most sense for a nice quick win that I value and typically what we'll find is wow that that was amazing we never saw this before this model better than we had tried with manual python or manual our approaches what can we do next and then you'll start to embrace your data science team or maybe it's your citizen data science team we'll embrace what we call citizen data science and then you start democratizing it into the business itself and using some of the other techniques because you'll have more than one model now for folks to consume and use and then you'll start that training wider at the organization on how to take advantage of some of the advanced analytics and that you now have available and the process to get started responsibly always starts with training there is a learning curve even though the tool is that easy button that I showed you just load your data and go could you do it yep absolutely you could but you're really going to be missing out on how to properly do it how to properly solve the right problems how to properly you know interpret the results so you always want to start with training and even you'll want to train in this particular case your executive and that's a little difference in a lot of the bi and analytics projects that we worked on in the past where executives really didn't have to be part of it this is a c-level conversation and they do need to be on board and they actually have to understand so you don't get the person in the room that says okay I want to meet AI and they think AI is a person um you don't want to make sure that you don't have that situation you know you will get assistance from with an initial project develop a roadmap and an architecture and typically uh this is an example of what a use case looks like you're just going ahead and putting some value around what do we think we should get with this and having it be measurable and if you do this right you know you'll often exceed what you're expecting to do and and it can be very valuable projects and what we're finding that data robots folks are really doing very very well and then they'll stop even telling us how well they're doing they just they they don't want to they don't want to talk about it anymore it's a big secret so that's that's one thing you might run into is you do so great that people don't want to share their secret sauce of how they're doing it and here's an example of a proprietary list of use cases you know going through with your team or going through with with your own team internally and saying well what is it that we want to do and look at and you know prioritizing this and if you've ever done any sort of you know maybe see the data warehousing roadmap project this should look somewhat similar to you it we've been doing this or I've been doing this with most of my career when I was in implementation and consulting from that perspective and then building the AI driven enterprise so as you continue you will train again you're going to train new users so there you have an internal portal on how to use it here's all the ads that you can start consuming whether you're using to blow click excel maybe you have other processes that you're using with with machine learning so you're just continuing to train folks on it just like you did in the self-service bi roll out having those champions in your business unit saying hey check out what I've done and what what might what can we do setting some best practices you know having that documentation there so in case you know you get a better job somewhere else which really seems to happen a lot of folks will roll out very successful projects and then get a better offer from another firm that says hey can you do what you did at ABC company for us we're seeing a lot of that with our early adopters and then having things like change management in place ongoing mentoring needed support and always keeping things accountable so you know keeping track of your success metrics you can use that road map you know put put the estimated values and what you're really achieving again then you can go to your boss and say look what I did for for our company last year and you know be held accountable for your successes which should be you know celebrated for your success so next steps would be contact us to explore you know a use cases if citizen data science and automated machine learning sounds like something that you can use in your organization we can work with you to find the right use case I'm going to steal a quote from one of my favorite peers in the industry Donald farmer used to say when he was with clicks seeing as believing the same applies to data robot even if you have manual processes today it's been really fun to hear the stories of success of you know how much further automation took existing use cases and you know brought forward some of those ROI's that I was sharing with you early on and attend a robot university it's not a two two-year program I took it to your program 10 years ago before these types of things existed data robot the university is a two-day program it's really practically focused and allows you to understand you know the inputs and the outputs to make you successful since the in between steps now have been automated Jen a great presentation we have a number of questions I know we have a few minutes to go here what sort of training would be recommended for creating an AI center of excellence besides tool training so I think what's really critical and this is the same thing that O'Reilly media just had for strata they have a presentation in a downloadable booklet from February of 2019 so it's very relevant on research that was done last year on understanding AI use cases so data robot has it is training on for executives on the AI defining an AI use case that's the biggest thing is folks don't realize what is an AI project how do I define it what does it even look like and that seems to be the biggest blocker to success in a lot of organizations because if somebody can't figure it out you're really not going to go too much further so what would be the ideal make up of a team from the very beginning I know you laid out all the roles earlier in your slide but if we're just starting what would be the typical starting team so you definitely have an executive stakeholder that's in debt you know on board with this they have some funding to get you going you know they have some authority there and they have something that's high value that you want to solve so problem that's high value to solve so that piece is important that person is important because they want to be your hero to other executives in the board the other pieces I it is really critical even though it's citizen data science having data scientists when you get started is quite important I've learned a lot from working with 200 of them myself that you know even having played around I say played around for 10 years I thought I knew what I was doing a lot of data scientists out there right now you only have less than three years experience it really helps to have an experienced one help you interpret the results make sure what you're feeding into the system makes sense and isn't by seeing some of some of the output and having that guidance in that ongoing mentoring is really helpful then folks like me and I think it's like me you know I built dimensional data warehouses and click for reports and Tableau reports and Power BI reports it's folks that understand you can query data and flatten it into whether it's equal view and be able to use some of those tools you understand the context of the data you permission to query data into a flat if you do a little bit of data prep and cleansing on it to get it ready to feed into data robot that's like the key person so it's really those three people you know that executive and then those other two that can get this going and anyone else is really you know it's just added added extras you know another question that fits in with with with your answer here is do you find much difficulty in garnering buy in from executives with models that have low explainability yes and that is one thing I really love about data rollout I've learned a lot from one of my peers and I adore him you should follow him as Colin Priest he has it is a kind of like a cheat sheet for how to interpret models that you could apply to even if you're not using data robot quickly you can use some of the visualization type but data robot has more visualization types in there for explaining that's the key so once you understand it you find things like that one banking analyst I talked about that found 150 in money laundering by looking at the visualizations inside the model it's so powerful to be able to do that and it also prevents you from making ridiculous mistakes that's great you know what what are your sources of truth for foundational assumptions for your AI models that's a loaded question that's a loaded question I mean guys you know how to answer that one no problem no problem that's okay I'll admit when I don't know something will data robot prime provide a hundred percent how the model was created or is it an approximate oh it is an approximate it's an approximate per gen okay and can you provide more detail and automation of data quality and issue identification oh yeah so there's a few different things that we have so we have it is within the model itself or within you know as I clicked on the model there's different habits and I didn't show where it talks about it shows you the profiles of the data itself and you know the different values that are available in there so we do have some of that in the model itself I just didn't show that oh very good very good well we are close to the top of the hour so I'm going to turn it over to Shannon run and Jen thank you so much for this great presentation it's really been great and thanks to all of our attendees ring so engaged in everything that we do just love all the questions that have come in just a reminder I will send a follow-up email for this webinar by end of day Friday with links to the slides and links to the recording this session and we'll get you some additional info on data robot Jen thank you so much Ron and thank you I hope you all have a great day thank you they can send it to us perfect