 Hello and welcome, my name is Shannon Kemp and I'm the Chief Digital Manager of DataVercity. We'd like to thank you for joining this DataVercity webinar using machine learning to understand and predict marketing ROI sponsored today by Altrix. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them by the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet, we encourage you to share highlights or questions by a Twitter using hashtag DataVercity. If you'd like to chat with us or with each other, we certainly encourage you to do so. Just click the chat icon in the upper right-hand corner for that feature. And as always, we will send a follow-up email within two business days containing links to the slides, the recording of the session, and additional information requested throughout the webinar. Now let me introduce to you our speakers for today, Raz Neister and Scott Charlotten. Raz is the Chief Data Scientist at Keyris US. He comes from a background in the physical science where he learned to use computers to help people understand complex problems. And Scott joined Altrix in January of 2016 as the Marketing Director of Channel Demand Generation Strategy and Programs. And Altrix engages with partners and customers to address their challenges with self-service data analytics. And with that, let me turn it over to Raz and Scott to get us started. Hello, and welcome. Hello, Shannon. Thank you. This is Raz from Keyris. I hope everyone can hear me clearly. Nice to be here with everyone today. Please do let us know if you're having trouble hearing or seeing the presentation. What a great topic we'd love to share with you today. This is one of the hot topics in the market that certainly we've come across at a lot of our clients. And I think we can in about an hour really drill through and try to understand how we can leverage some of these new age and leading edge machine learning tools to do some really, you know, market shattering and a really cool analytics predicting marketing ROI. So with that, thank you for everyone for joining us here today. We'll go through before we even get to that. I'd just like to set the stage a little bit, talk about a little bit of who's Keyris and how we use Altrix in this sort of day-to-day. We'll take also a little bit of time to sort of set up the sort of landscape in the industry. We'll be focusing on the CPG or retail sector specifically, but these types of analytics that we'll see here today can really be used in multiple verticals. If there's one message that you should take out of today is that the time to really answer your big picture initiative is now. If you forget everything that we talked about in all the details, just remember that the time to really start pushing on your big picture strategic initiatives is now. And that's really like the fire. I'd like to light in everyone's stomachs and in your bellies here today. We've seen a kind of big shift in a lot of our sectors and a lot of our clients in the last maybe eight months or so where everyone's starting to use like these new machine learning platforms to really do some of the really cool and advanced analytics. Specifically here today, we're going to cover two kind of themes or two problems that have popped up a lot in our work in the CPG space. We'll talk about what CPG is in a second. But these are kind of like very fundamental problems. The first is what are my customers? If I'm a company that is producing goods or importing goods, who's actually buying my stuff? Do we even have a credibility on that? And if they are, and if I'm in a very competitive marketplace with very tight margins on my supply chain, what's the best way to market to people so that when they walk into a store, they take my product off the shelf instead of with competitors? These are age old questions that we'd like to sort of actually like answer today in about an hour's time. And I hope along with the messages of the time is now that you really should start start thinking about the intermission now. I hope you take away, you know, sort of the ease of use and the accessibility that modern analytics platforms allow. Just your regular business analyst to sort of leverage and use so that you can answer these big questions. A little bit of background on Keras. Who are we? We are big data and analytics consulting firm, primarily based in New York. We have various teams and we like to see we deliver end-to-end strategic initiatives for our clients. This could be everything from starting from the back end, building databases or setting up, you know, big data systems, through to the data discovery team which builds visualizations and engaging networks that people can use to track their business performance. And then, you know, most recently in the last year and a half, of course, putting the cherry on the cake of combining everything and all of our insights together using data science to leverage a lot of the machine learning and advanced algorithmic steps, packages that we can apply to every business problems. So we work on end-to-end projects and I'd like to, you know, kind of set the stage a little bit with what I said about how, you know, some themes that we've been picking up in sort of the market space relating to our CPG clients. If you type, you know, AI and marketing, there's like hundreds of millions of, you know, Google hits on this topic. It's a very, very hot topic. Typically, the marketing teams on a lot of our clients were probably the ones most underserved by analytics. But hopefully by the end of today, we can see how we can leverage an easy to use and accessible platform to really solve the big pictures. And this is how we would typically structure a strategic initiative for a client perhaps about a year ago. And this is like straight out of one of the sort of roadmaps that I laid out for a couple clients recently. We would start with like cleaning some data, bringing it into one place, okay? Building some, and supporting some standard reports, getting into the self-service, and agile visualizations using, you know, the modern dashboarding tools like Tableau, ClickSense, Power BI. And then, maybe then after we got through all of that in maybe two or three years' time in a very comfortable and generic way, we would say, okay, now we could start to do some of the predictive analytics or some of the optimization stuff. But if you look at what delivers value to the business, this Y axis, it's really like the questions that are going to deliver value are going to be answered by these techniques up here. So in the last probably year, what happened was all of our clients got their data somewhat in the right place or in the right format and started this big shift where it's using data science and machine learning to do the bigger stuff. And it's no longer, you know, we can't wait another six months, another eight months, another year to start initiatives. If you start a project in data science, it will still take time to ramp up, design it, you know, sell it to the business, communicate the insights. And then by then it's already too late, it's already a year has gone by. So the time to do everything is now. Moreover, we can't solve like these generic things and say we're going to do predictive modeling. So basically in business, we'll understand you if you say I want to do a data science project, we have to solve very specific problems that the C level management and sort of the DPs and everyone can understand in terms of business context. And that's what we are in the sort of CPG space. The CPG stands for consumer packaged goods or in the retail space. These are companies that make or import goods and sell them throughout the U.S. And there is many points along their supply chain and sales process that have really sort of big picture type of value questions that can be targeted or answered with advanced analytics. So last year partnered with Altrix, we did some really interesting webinars on supply chain optimization, modeling cost to serve, better forecasting, all of these things that's still like a very hot and ongoing topic. Today we're going to focus a little bit more on the consumer who is actually buying our products and where they located. I would be surprised at how, I guess, sounds like a simple question, but see how we can leverage some of the tools and some of the data within Altrix to really get answers to that very quickly. And then the biggest thing we've heard probably in the last year is anything we're doing like actually working, people spend a lot of money on advertising and promotions. I'm saying that the business doesn't have some directional sort of feel for what works with them, but there's certainly no real quantitative sort of implementation at a lot of our point. It takes a good look at what's actually delivering on marketing campaigns. And as we sort of seen a shift between sort of like setting up a brand, you know, selling or a sales strategy perspective, diving in now more to understanding who our consumers are than some of these questions need to be no addressed and answered immediately. How are we going to do this? Of course, we're going to use some data science. I don't want to spend too much time here. Even the audience, the data science is essentially just, I would see the art of using some algorithms and computers to solve business problems. There's a lot of different topics or areas that I would sort of bundle into data science. Some of them are very, very hot topic like natural language processing, analyzing tweets for sentiment and tonality. The majority of what we're going to talk about here today can either be done with like simple demographics analysis or applying machine learning. In terms of the data science spectrum, as we go from left to right to right in both complexity and I would also say hype, a lot of business processes really can be answered by sort of just living in kind of something a little bit more obviously than just basic business intelligence. But, you know, something very accessible again through if you have the right tools and platform and specifically here machine learning will see how that can be leveraged to really build some cool models. We're not going to get into the sort of AI stuff that you might see some really cool marketing demos where you just speak to a computer and essentially organizes your entire marketing campaigns for you. We're going to be a little bit more pragmatic here. So what is machine learning? I would like to think of it as nothing more than an advanced statistical algorithm or process that finds the best way to map a bunch of inputs to an output. So if you gave me all of these points here that you see in the charts highlighted here in black circles and you told me to find the best way to describe this data. Maybe I would guess, maybe it's like a square root, maybe it's a fourth root, kind of steep here, so maybe it's like a logarithm like shift upwards. That's me trying to come up with a way to describe the data. Machine learning uses advanced algorithms to sort of find that relationship for you at the cost of actually seeing what that function is. We're not going to worry about it, we're just going to use them and the machine will pick up the best way to sort of describe all the nuances and non-linearities in the data. So who can do machine learning? Of course, data scientists can do it. Who are data scientists? They're a mish-mash of all kinds of people from all kinds of backgrounds. I myself came from physics, so of course I transitioned to machine learning and analytics a couple of years ago. And I've been working at Keras for about three years where now I head up the data science team. To actually stand up a data science project takes a lot of different insights, knowledge, and skills. You have to be good at all the machine learning and computer stuff and programming. But the biggest thing we find is you have to have business domain knowledge. You really have to understand the business and the data. And you have to set up clear lines of communication with stakeholders with upper-level management to explain the results and to really think about how those results are used in a real-life setting. So that's like a, you know, a unicorn of skill sets. Not everyone has that. And not everyone can actually do the computer-based stuff. Luckily, we have platforms like Altrix these days that could really enable and unlock your analyst's true potential. We've seen regular business analysts, people very good in Excel, but not much more technical than that, capable of picking up the tool and applying some of these more advanced algorithms. To give you a little bit more background on Altrix, I'm going to hand it over to Saad. And he's going to walk you through some of the platforms we've delivered. Excellent. Thanks, Radz, for the introduction. And thanks for kicking things off here for us today. Appreciate it. For those of you who are not familiar with Altrix, Altrix really is a company and the software provider that we focus on providing a complete analytics platform and experience for data analysts, data scientists, and even for the folks who Gardner describes as the citizen data scientists, if you will. This platform is built around four fundamental principles. One, being able to giving analysts and data scientists and allowing them to create data sources that can be trusted and that can be shared throughout your organization. We find that for most folks that one of the hardest things or that first step of being able to trust your data and discover data within your organization that can be trusted and you can use from the offset, getting that data ready to go for advanced analytic analysis, that can be tricking itself. So that's where we provide a platform where you can discover and share data sets and also share insights as you move forward and you build your analytic processes. We allow you to prep and blend data to allow you to again taking data from various disparate data sources and allowing you to bring that all together to build a cohesive data set that you can form advanced analytics on. And again, we give you the tools again all on the same platform to do that advanced analysis and to do it in a code free and a code friendly way. And I'll show you a little bit more of what I mean by that here in just one minute. And then finally being able to deploy and manage advanced predictive and statistical models here. We may not touch upon it here today, but one of the things that we've seen from data scientists and from companies around the world is a number of you will build predictive models and you'll want to actually deploy them and put them into production but get them into use basically. And it can be significantly difficult to do that. We find that less than 20% of all the models that are built out there for the predictive models that data scientists build for their organizations less than 20% actually ever get deployed. Most of them actually just sit on a shelf somewhere. Well, metaphorical shelf, of course, because it's not something you can actually put on a bookshelf, but at the same time, most of them don't get deployed. So we provide you the tools and the platform in order to actually take data sets, prepare them for analysis, do the analysis, build the predictive models, and then finally deploy those models and share them with not only with your audience but also to share the insights that you have that you discover from your analysis throughout your organization. Next slide please, Ras. And we found that we'll talk about this today. Ras is going to show you a few examples of how we help with the data prep and blending step. And we found that most organizations, you know, as we move up what we call the analytics stairway to heaven, and we're going into that advanced analytics realm. Again, most companies are still struggling a fair amount with just the first step, which is the data prep and blending. We've always thought of this as this was the fundamental first step and most organizations here are addressing that now because it's, again, if you can't trust your data, if you don't know what your data is telling you and you know where it's coming from, that's going to prevent you from really moving forward with any type of advanced analysis and doing any type of predictive modeling, getting insight into the data. Next slide please. So what Altrix does, again, is we provide the next generation analytics platform. We take data from various different sources. It can be Excel spreadsheets. It can be your data warehouse that you use. You can get it from the cloud, whether you keep all your data on AWS or you're using apps or cloud apps like such a Salesforce or NetSuite. We take all, we can pretty much connect to any data source out there, even social media. So if you're tracking information and feedback from Twitter, from Facebook, from LinkedIn, you can bring all of that in. Take the different data sets and you can do your preparation of that data, blending all of those different data sets together in one single platform, getting you out of Excel hell, as people like to call it. We also have third-party data sources that you can get access to such as Experian, the Census, TomTom, to help you out with if you are interested in bringing in spatial, geographical, demographics, demographic analysis. You can bring that in. And again, all the same platform, enrich your data sets and perform the advanced analytics analysis that you are looking to do, whether that be predictive analysis or predictive analytics, spatial analytics. You name it, you can do it here on the one single platform. And then finally, being able to share your workflows and share the insights that you are uncovering, not only just within your group, but with senior leadership throughout the organization. We find that in order to create a true culture of self-service data analytics within a company and get that transformational change done, it's less about the technology and more about the people. And people need to share insight. That's when that information really gets powerful. And so whether you want to keep that all within the platform or you want to share that insight via a visualization tool such as Tableau or Microsoft BI or you want to load it back into the data warehouse or back into a data analysis and just keep it as raw data, you can do that all, again, all within the same platform. That's what we pride ourselves on. Next slide, please. And I spoke about code-free and code-friendly analytics. With Alteryx, we provide you with a platform that you do not necessarily need to have a PhD and need to be a full-fledged data scientist in order to do advanced analytical analysis on your different information here in your different data sets. We provide a simple drag-and-drop user interface that allows you to create workflows. Raz is going to show you how to do this here in just a few minutes. We provide a wide range of tools for you to bring data in, do the analysis, and even do a lot of advanced and predictive modeling. That's all within the platform here. Next slide, please. But I was going to say, if you are a PhD and, hey, you work hard for that PhD and you want to code, you can bring your code in. And again, Raz is going to show you a brief example of how you can bring our code Python, any of the, a number of the advanced analytical coding languages, you can actually bring them into Alteryx and perform the advanced analytics right there from the platform. You don't necessarily have to, you know, do that autonomously and keep building code in R. You can use, you can leverage R code and Python within the platform. And what that allows you to do is instead of having to start from scratch every time, it allows you to do a repeatable analysis, repeatable workflow that is consistent and you don't have to necessarily step back and start from scratch every time. And with that, Raz, I think I'm going to turn it back to you here to get started with some analysis of what Keras is doing with Alteryx to build some of these predictive models and identify marketing return sources here from the retail and the consumer package good markets here. Yes, thank you, Scott. So absolutely, let's dive into some, not only some examples and demos of using Alteryx, but let's actually solve some real world problems with this. The data is a little fake because we can use real customer data, but it is like exemplary of sort of what we've seen going on. Now the retail and CPG sectors are what I call a target rich environment. If you've seen the 1980s movie Top Gun, probably most of you have not even been born then, but I love the 80s. So this is like what Maverick says when he walks into a bar. I don't think he was talking about the enemy fighter jets at that point, but this is a sector here that has a lot of different data sources. And if there's ever been a need to be able to consolidate them, manage them and wrangle them in the proper way, it's really in this sector. And some of the things to give you an idea of how the supply chain works in the CPG retail space. I mean, you have the actual companies, the vendors and manufacturers, they have a ton of internal data, finance data, manufacturing data operations. Most of it that we've seen is Excel, you know, Excel Hell is still very much there. There are a few companies, some of the larger, for example, spirits and beer manufacturers that we've worked with on a global scale actually are essentially big companies. They have a lot of databases and movie pieces. They usually sell goods through a distribution system. Sometimes this is managed by the actual manufacturer like Walmart or Walmart. It sends all this stuff to Walmart warehouses and then to the stores in the alcohol and spirit space. They're third party companies that buy the product from the vendors and then sell it down to the accounts which you get on the level. And then from accounts, of course, that's where the consumers buy it from. And everyone, everyone really wants to know what's going on in the consumer space these days. There's a big shift from kind of tracking and developing sales strategies in the sort of brand focus way to become far more consumer centric. So who are my customers? What are they thinking? What are they saying on social media? Social media is like a super hot topic on this case. But throughout here, you know, very easily you could rack up 15, 20 different data sets on that in some way you have to piece together in sort of meaningful ways so that you can do cross-functional analysis. Because if you want to understand what sales returns from your consumers or from your accounts level are impacting your marketing and advertising spend, you know, there's a lot of like cross joining you have to do here. So like we said, when we work with our clients, we try to develop end-to-end strategic initiatives with them. We actually found that if we say we want to do a data science project and applies, so we have to solve like strategic big pictures for them. A lot of our clients are really large CPG companies. A lot of them are focused in the spirits and alcohol space. We're based in New York and there's surprisingly a lot of headquarters here in the northeast area that we've obviously worked with. A majority of the time they're basic, you know, sales models essentially boils down to they need to understand who the customers are and then they need to understand the best way to sort of reach them so that they can influence their sales. So some of these companies spend about over, you know, $100 million on advertisements and promotions. These are things like TV commercials, YouTube ads, Facebook ads. They could be focused things in the accounts like sampling events like if you're in a grocery store and you could try different yogurts or if you're in a liquor store and you could get a product there. There's a lot of price promotions. There's a lot of like sales and so there's just like a ton of different things companies do to try to influence consumers to buy the products. So we're going to actually answer these two questions. Who are my customers and, you know, what marketing campaigns are actually working using all tricks and, you know, in the 20 or 25 minutes that we have left, we'll see how to answer these questions. You can imagine the pace of delivery of actually getting to the insights. So for the first question, consumer segmentation, who are my customers? We're going to take sales data and tie onto that some of the demographics data that Scott mentioned. We want to identify regions of highest growth or regions where, you know, our products are selling. A lot of these data is offered as an add-on into the Altrix platform. And the neat thing about it is that consumers are already profiled by experience in what they call the Mosaic consumer groups and demographics. There are 75 types of consumers. I'm sorry. You are not individually enough not to be one of those. At some point you are going to fit into one of those profiles. And that's why people use these consumer groups to sort of bucket people into types of consumers. So let's take a look at Altrix. I'm going to tab over here for a second. As a platform, a short introduction and get into solving some of these tools. This is what you see when you open up Altrix. It's a blank canvas with a bunch of tools at the top. Essentially everything you'd like to do with data is represented by a picture or an icon. It essentially is the smartphone of, like, you know, analytics platforms. The very, like, sort of, you know, visual representation. So everything you'd like to do is represented by a different little picture. The learning curve of using Altrix is really kind of in place to, like, when you first got your first smartphone, how you had to learn what each picture did. So if I wanted to input some data, well, I just dragged the input tool down. If I wanted to select some fields from it, well, I just dragged the select tool down. I could then filter it and perhaps add a custom formula to it and then probably join it onto, you know, another data set that I'm going to bring in from a completely different source. If I wanted to, like, summarize the data up and then perhaps do, like, some distance analysis on that data, I could do that on the same platform. And then, like we'll see here today, has a, you know, a bunch of advanced tools built out for you in a sort of RBAC end where then once I got my data ETL then prepped, I can actually start to drag out some of the more advanced machine learning algorithms like random forest decision trees and neural networks. So this is what Scott meant when he said that it's a complete analytics platform. You can do a bunch of the ETL and data prep in the same canvas so you can do a lot of geostatial analytics and a lot of the advanced analytics. This is why we use it kind of on a daily basis at Keras. We use it all the time because it just helps us be faster at our jobs. We don't have to script and debug, like, kind of syntax of, you know, forgetting a semicolon here and there. We can build out complex logic sequences visually, which certainly helps. And then we have access to, like, all of these advanced analytics. So it almost becomes a question of, well, if you're doing all this stuff with data, why wouldn't you do some of the advanced machine learning? Take a crack at it. It's there for you and it's very low hanging fruit. So in summary, it's a visual program paradigm that really allows you to do things that you never thought would be possible before, but also makes you much, much faster at your job. I'm going to skip over some of the initial data prep stuff that I wanted to sort of go over just for the sake of time. And just know that we have used Altrix as an enterprise level ETL sort of platform. That's fine. We've used it there. You know, clients don't want to get into the advanced stuff. That's okay. But it really shines when you leverage the sort of machine learning and sort of analytics capabilities that it has built in, because it's so easy to blend and prep data together. So let's get into it. The first problem we said was we want to take our sales and sort of understand who our customers are. We're going to do this by taking demographic information that's available as an add-on to Altrix from Experian from Census Data and all that stuff. We're going to take that and join it onto our sales so that we can understand in certain regions who's actually buying our products and what type of customer is actually buying our products. So look at the data we have coming in. I sort of ran this flow for a second, but just bear with me here. So the data, our sales data is pretty typical. You know, we have like a data of sort of the sales. We usually, you know, ship through a distribution company. That's their name. We're talking about a bunch of products that were sort of in charge of these sort of made-up names. So we're selling like some fish stock, selling some coconuts and some French soup and stuff like that. We're actually selling, you know, through the distributor down to the account. These are the actual store addresses in the cities and the codes that they're located in. We've enhanced this by adding some census sort of anchors or identifiers of which sort of region these accounts fall into, primarily one called DMA, which are sort of like marketing areas. We'll see that out naturally. We have some latitude and longitudes. And then really the only like metric in our sales data, which is our volume, it's going to be like volume of pallets. It could be units of, you know, bottles of stuff that you sold or a case of beer or something like that. It doesn't really matter. Essentially it's the quantity of product that we sold through the distributors of this product down to the account level. Okay. We're going to take that and enhance it with some demographics information. Now, like I said, this is a third-party data that's available in Ultrix. And the neat thing about it is that, you know, you can have like a very granular sort of view down to a block level. This is like the most granular sort of geographic entity in this day. A block is like literally my block in New York City. All the demographics information about that region or about that geographic level. I'm going to do it at the DMA level. DMA is, again, I'll show you on a map. They're slightly larger than counties, but not quite states. So they're a little bit different, but you get the idea. There's a ton of data that you could pull in by this. So we're going to pull in some basic information. Let's see. I want the age and some educational properties of the people. I want their actual household family incomes, aggregated income per capita income. That would be interesting. Racist and ethnicities of people in there, male, female, split. Just like a ton of stuff that you could pull in. We could pull in actual information based off the survey data of their, you know, consumer and expenditure habits. And then, like I mentioned, the mosaic sort of clusters and groups of consumer types. And then for each DMA level, I want to pull in consumer type in that DMA. Because I want to understand, you know, on average, sort of a higher level. Who are really the customers in this area? So far, so good. So I'm going to do that. I'm going to pull in that sales data. I'm going to just kind of filter it and work on it a little bit. You could see I've added a filter for New York. And I'm just going to pull in some of this and add some custom congratulations to it. And again, the ability to get all the practice data is really, whoops, in this click. The ability to sort of join these complex data sets together really comes down to just writing note and visually programming your logic flow. So I'm going to join our sales data onto our demographics data by DMA ID, right? A little hex is hidden there. Okay. And I'm going to run this flow to give you a sense of, yes, this is a visual programming paradigm because everything you want to do is done with icons and pictures. But also it's visual programming because when I press play and I start running the query, I sort of see how the data flows through each step of the logic at each segment. So I could see I pulled in like about two and a half million rows of sales data. I pulled in about, you know, only 10 DMAs because I'm only concentrating on New York City area and New York State area. You know, so I could see at each point through my filter from two and a half million, I went down to 43,000. I picked only like my top seven products and entered down to 33,000 lines. And the neat thing about it is it'll, this allows us to like debug really complex logic very easily. I can't click on any of these input and output anchors to sort of see my data each step. So if I click on, if I click on this, a screen will pop up and this will show me a snapshot of the data at that step. Right. And after this join, I could click on this again and see the data at that step. So just because you, the fact that you could see how many rows are flowing through is like, this is like an incredible ability and saves us a lot of time in debugging. More so out of the joins, I could automatically see the right and the left orphans that didn't join. So if I have a problem with master data and somebody don't line up, I know this is my list of, you know, sales data that are going to join on. So I could either, you know, fix that here or call like, well, represent charge of this data and take it to them. So at the end of the day, all we've done sort of build in and sort of enhance our sales data that which went up to volume with a plethora of demographic information. Just endless, endless. And I know you're getting dizzy. Endless sort of levels of demographic information. You can see some of that is like what school data team and what percent are they male, female and all kinds of things. That's a lot of data. Let's take a look at what this looks like in sort of a dashboard representation. I want to focus on the tool, but this is like a neat way to sort of see what's going on. So this is a dashboard of that data, not just from New York, but the whole country now. You can see in the map in the upper left corner, these are essentially what DMAs look like. These are all the DMAs in the country. So they're not quite like states, but they're bigger than counties, I'm sure. Each DMA is sort of colored right now by year over year growth. If we wanted to see volume per capita, that's like kind of another way of looking at the data. Looks like I didn't have access just for a while to change the measure. In the top right corner, we see a sort of products chart where we are this year versus the line last year and our percent growth per product. And we're going to leverage some of the demographics information in the bottom two charts. In the bottom left, we have the ethnic or racial sort of population split. Of the 80% percent of our market is white population and we have quite a bit of other ethnicities. And then on the bottom right are these tags, these mosaic demographic groups, which we've mentioned, which are essentially, you know, pre-filtered to bucket consumers. So if I wanted to look at my fastest growing demographics group, which looks like somebody called Striving Single Seen, I can actually click on that, get an idea of what DMAs they're located in, take a look at my year over year growth, see that they're, you know, they have quite a large subset of Hispanic population. Take a look at two areas and say, you know what, in Phoenix Prescott, Arizona, I'm actually growing down at the bottom of this top at over 10%. But in Denver, I'm actually losing, I'm not growing at all, I'm actually down 3%. What's going on? This is the same population in two different, very close related DMAs. I should be able to come up with an explanation. This is something I took my marketing team. So very quickly, I can get a strategy to go, or a story, and now I can go and market to these people specifically. So that's some of the power of, you know, in a matter of like hours that you'd be able to sort of enter these types of questions by using the right platform. So let me go back to the slide very briefly, just highlight what we said that we've identified actually there's, if we went through, I want to skip over for a second time, but there's three demographic groups we want to kind of go after. There's these aging of Aquarius people. We want to maintain them. They're our biggest one. These stead facts, conventionalists, we're really losing them. So we got to go back after them. And these new ones that are coming up on the scene, striving single seed. And if I wanted to understand what these people or what these groups actually were composed of, I will have the PDF of what, you know, mosaic customer segments are sort of grouped as, and I could go to the striving single seed people and get all kinds of information of who they are and, you know, what sort of media and channels best speak to them. So that's really interesting. And that could help me sort of develop a marketing strategy to these people. And once we have a marketing strategy in place, we want to know if it's working. That's like, it's not even a new question. It's like an age old question. What works? You know, like we put so much money into different things. What actually works? And these are like some of the same questions that people have been answering in the 80s with like minor models in Excel. We want to be able to come up with a marketing model as they call it. We want to understand how much our distribution affects our total sales revenue. We want to understand how much social media contributes to our sales revenue. So we want to link a bunch of inputs to one specific output. This is like just ripe for machine learning applications. So we're going to do this in all tricks. We're going to do it relatively quickly, but I want to show you how fast you can actually build some of this stuff out. We are going to combine some marketing data with sales information. So again, we're going to ETL and prep our data. But then we're going to build a plethora of models on top of that to sort of get to this ROI. And the cool thing is, okay, we're going to build a model. That's really neat. Let's pat ourselves on the back. But once you have a model, you have essentially a little simulation that you can exploit to one, understand what's going on. Two, really cool what if analysis. And three, optimize, because if there's a way to spend the same amount of money in your various channels, but get bigger returns, of course, people are going to want to know how to do that. Now, like I said, traditionally, people have done this with like linear models. That's great. They said, oh, okay, we're going to do a mixed level model and we'll have some linear terms here. And then like an epsilon representative saying, you know, the unknown unknowns that we don't know. We'll say that's like three percent when the reality turns out to be 80%. But real life is nonlinear. Your response, how much money you put into a channel and the returns you get out of it is not going to be a straight line. It's going to peak and it's going to have very nuanced behavior. So we're going to use some more advanced algorithms like trees and random forests and neural nets. I don't want to get into the details too much. But we're going to use some of these more advanced algorithms to describe these nonlinear nuanced patterns and features in our input data. Before we really do that, we should ask, do we even need to? Do we want to do this? Because if I showed you the answer between a linear model and a neural network sort of performance where like a straight line along these dots is like the perfect answer, you can see like, yeah, the neural network does really well. It's getting the data, right? Predictive values versus actual. The linear model is not that bad, right? There's a small extent to it, but it doesn't do that bad. But really what falls down is when you look at how the response is captured, as you ramp up spend from zero to like, say, a $4 million in a particular channel. The linear model, it's a linear. It's not going to be able to do anything except describe things for you in a straight line. Whereas the more complex algorithms are going to be able to pick up these nuances and these really kind of neat features that the other models just not going to get. And if you look at the results, then yes, the linear model will actually tell you you don't need to spend like $3 million. You can always spend like a million dollars and get the same return. That is a million dollar question and a million dollar answer. So yes, you should be using some of this stuff, especially if it's so accessible as we see. So we're going to have here a mixture between some marketing data and sales. Essentially, we're going to tie our data on data, some products, and we're going to take a look at our spends, our actual dollar amounts that we spend in various channels, versus some sales metrics and KPI, like distribution and things like that. But essentially, we want to get to all of them. So let's take a look at this data. If I told you we could prep, use Altrix to prep this data. This is what a prep would look like. It's a little bit involved, but you know what? Anyone could take a look at this phone because it's laid out in a linear way. They can easily get to it. So if I wanted to start with some of this data, and I'm going to take a look at some of the metrics we have built in here. Some of the things from the media side are things like spend, these are actual dollar amounts and PRPs. These are like measures of how many people we reach through various channels. Some of the channels we want to spend money on are like cable TV, network TV. These are Hispanic population or demographic was important. We're going to invest a little bit into Spanish channels. And then from the sales side, we've got a whole bunch of stuff from this phone, Nielsen. Essentially like distribution and percent market share. But our target we want to predict here is our dollar volume. It's essentially our revenues. And that's what we want to do. So if I wanted to build a machine learning model using this data set, you could take a look here at the raw data, what it looks like. It's just kind of all pieced together in a flat sort of file structure all in the same row. So the machine learning model, I got to this point, I worked so hard to deal and prep everything. And it's just a simple matter of dragging out some of these algorithms and going in there and setting essentially what model name, but setting what variable you want to predict against and I want to predict against dollar volume and telling you what features sort of have to go into the model. You could take dollar volume out of the feature list otherwise it would be perfect. And that's it. That's essentially the amount of work it takes to build out a machine learning model in Altrix. I could run this and then we could see the model will run. And essentially we've kind of built out and coded a random forest machine learning algorithm to be able to map all of those inputs, marketing, spending, sales to our revenues. And once we have this model and have it built out, we can really start to play around with it and take a look at what's impacting our things in various channels. Just give this a second to finish and taking a while. The computer is probably slowing down and shouldn't take this long. Of course everything goes wrong when you run your demos, but there it is. It's finished and essentially we've just constructed a model. It's no more work than actually dragging out the linear model here, the linear regression model. It's the same amount of work. If you want to do more models like random forests and decision trees and neural networks, please feel free to do so. In fact I've taken the liberty of running six models including the linear one on this data set and I was a little bit more careful about splitting it into training and test sets and standardizing the data from neural networks and all that. But essentially I could get a performance metric out, which is R squared in this case, of the various models to see how they perform. And like I said, the linear one's not that, like 0.9 R square value, that's pretty good. It's not as good as the tree method, but that's fine. It's still pretty respectable. Gamma regression here, no offense to gamma. Gamma sucks, so we're not going to use that one. But like I said, the new responses and the ability to describe features in the data is what's really important. That's not going to come out of the linear model. So in a matter of like 10 minutes, you could build 10, you know, 5 to 10 very complex machine learning algorithms. That's the power and scalability of the platform. The really cool thing is you can then understand the patterns in the data. Add in the tree-based message, you can understand things like variable important plots. These will tell you which calls or fields are most important in the model, which get ranked very high as being important. So things like sales metrics like distribution actually comes up to be more important than any media spend that you could put into things. So you have to be careful how you structure your data and what you're putting into it. This essentially is your attribution model. And then when we look at response curves of how much money we put into a specific channel and how much revenue we get at, and you can see these really complex features that the other models will be able to capture. Since we have a model, we could do a lot of neat things. So one of the things is actually calculate the attribution model. So I want to see to what extent my cases per point of distribution actually affects my revenues. I could click into the models now that I have them. I could do what if analysis. I could essentially randomize like the values or take the values out of this column and see how poorly I do describing the data. And I could come up with metrics to say, you know what, 25% of our revenues is distribution driven. So before we even think about like media spend, maybe we should think about increasing our distribution in underserved markets. The really neat thing just to wrap up here is that once I have my model, the next logical question the CEO or CMO will ask you is to tell you, okay, how can we best spend our media dollars to increase sales? And we've built like another little flow and it's very accessible, nothing too fancy. But we essentially made an app that perhaps the CEO can access and publish this to the Ultra Excalibur. For example, we'll allow you to pick a product and pick a market. Let's just do the national pick a model and this app will take your input data for say the next month and tell you what your optimized spend should be instead of like what you were actually planning to spend. And it's going to do this by doing like a really kind of search on the model in the parameter space and kind of many different combinations of spend to actually give you, you know, at the end of the day a report that says your sales list, your sales can increase by 13% next month. So instead of spending this much of your total spend, instead of spending 55% on cable TV, if you took that down a little bit, and if you took down the Spanish channel cable TV a little bit and invested a little bit more in network TV and particularly Spanish network TV, then for the same amount of money, you would get predicted increase in sales and that's hugely powerful. With the type that's the end game that everyone wants to do and you've seen that just by essentially being able to piece the data together and then have access to these levels of algorithms that we can actually come back with a pretty confident, you know, estimate of these types of things, and it'll be far more confident and discussion worthy than sort of doing things in a linear model. So that's the type of thing we're dealing with. So no matter 20 to 25 minutes, we did a complete demographics analysis piece on marketing and sales data together, and essentially built a marketing ROI model. We understand the attribution of each of the channels, and we can actually optimize in a pretty simple way we're not to get too fancy doing Monte Carlo and all that actually just do simple grid search if that works for you. So instead of doing this, you should do this to increase your sales. That's the level of ability and capability that, you know, you can unleash in your analytics environment in your company. All tricks up as a platform is very easy to use. It's visual so you can see everything that's going on. The data pack is great for this customer segmentation and demographics analysis, and I think you'd see that some of the elements from some of the vendors are offered at a very disruptive price point for those. So what you miss in doing machine learning and doing the ROIs are very hot topics these days. I would suggest that you just do it. Remember, the time is now you have to start doing this because it'll still take some time to kind of end it up in your organization, and if you don't start now, six months, eight months will pass by, your competitors are doing this. So with that, I think I'll wrap it up. Open up the floor for questions, and thank you for paying attention. Thank you so much. And just a reminder and to answer the most commonly asked questions asked by attendees, just a reminder, I will be sending a follow-up email by end of day Thursday with links to the slides and links to the recording of this presentation from today. So just diving right into questions here, you guys. All tricks have an in-database support for DB2 Blue, BLU? Ooh, that's a good question. So all tricks, I'm not sure if it has for BLU. I think I'd have to get back to you on that. I don't want to say something that may put a foot in my mouth. We have used the in-database tools quite extensively, just not on DB2. I'm waiting. Scott, did you want to jump in on that? Yeah, so at this time, I do not believe so, but we can double-check and get back to you on specific in-database functionality there. So this particular questioner said they would like getting in, but do you have documentation or steps for each of the model types on best practices for selecting variables to avoid situations like multi... Well, I'm going to just slaughter that word, too. Keep people new to the space from creating bias conclusions. Yeah, that's a great question. So one of all tricks' strengths are, is essentially a lot of documentation. So if I go back to sort of a model, if I'm in the sort of random force model, I could always click on question mark and it'll take me to Altrix Community webpage that I could start to read through to learn about the model and the tool. Every tool in Altrix does this. There is a ton of kind of built-in sample workflows from everything from finding data to using demographic stuff to the machine learning stuff in the predictive analytics to show you examples. And then the best way to sort of understand and try to find holding narratives. There's like correlation analysis tools available in the data investigation tool set so you can use, you know, Pearson correlation to sort of weed some of those things out. But of course, the best way to learn is just try, try different things. The model, the platform is very scalable. You saw that if we wanted to increase our data set, like add on some more financial data, add on some demographics into these actual workflows. It's a very scalable and easy platform to do that. We're just going to join things on. And then I just kind of go into the model and click the right fields that I want to try. So just try things. That's the best advice I could give. And then use the scalability of the platform to try and fail. If you're going to fail, fail fast. That's the message. But don't just build one thing. Don't overthink things. Like just throw everything in the kitchen sink in there and take it from there. That's like the best starting point I think. So we could say, of course, you have to be very careful of telling the results. You have to be sure of what you're doing. And make sure you're overfit and, you know, have some statistical rigor to everything. There's tools in Altrix to help you do that. But part of the fun is just getting to it and trying all kinds of things. Can Altrix connect to a file system like HD Insights? We can. A lot of our connectors, if there's any questions on what we connect to and how do you do it and is it built into Altrix or do you need a special connector for it? A lot of that is kept in two different places. One is the Altrix Analytics Gallery, which is part of the product and part of the platform that Altrix provides. And this is a complete resource center of different starter workflows, different connectors, et cetera, that analysts and data scientists, users of Altrix can access while you're in the Altrix platform itself. You can also visit the Altrix community, which is, you do not have to be a user of Altrix to access. You can actually, if you're interested in learning more about Altrix, that's actually a great place to go to as our community. It's a very active community and what I learned about it is it's a chance to meet with folks just like you from around the world. They're struggling with data challenges and helping each other. Most of our, if not all of our Altrix certified experts, which are our aces, as we call them, they are with our end users and with our partners and they are very active in assisting with folks who have questions or are looking to learn more about the Altrix platform and learning and trying to address their different data challenges, whether it be on the Data Prep and Blend side, the Advanced Analytics side, the Deployment side, the Connecting to Data side. So that's all very active. So you can either visit the Altrix Gallery, which the Analytics Gallery, which Raz is showing you right now, or you can visit Altrix community and you can access that just directly off our website at Altrix.com. And you can find the top of the page there, the tab or connector to Altrix community. I love it. Thank you, Scott. So I think we have time to sneak in one last question here. How do you get buy-in from your clients when they don't understand how the models work? So that's a great question. If we can, we try to use some simple explanations like for decision trees, for example, we would run that and say, you know, this is just like a complicated or weighted if-else statement. But basically we instead try to focus on selling the value and the scalability and the reproducibility and, you know, the ability to kind of support this what-if analysis. And they will slowly come on board as they kind of, you know, sort of educate themselves or scale up on what exactly a neural network is. And of course they have to have trust in the data and in the partners and in the platform and in the people running the analysis to make sure that it's statistically rigorous. But that comes with time. So we instead try to focus on the value proposition. Raz, thank you so much. And Scott, thank you so much again. I'm afraid there's so many great questions coming in, but I'm afraid that's all we have time for. Just a reminder, I will be sending a follow-up email by end of day Thursday with links to the slides and links to the recording of this presentation. Thanks to all of our attendees for being so engaged in everything we do. We just love all the questions and love the engagement. Raz and Scott, thank you so much. And thanks to everybody. I hope you all have a great day. Thank you. Thank you, everybody. Hope to hear from you soon. Thanks, everyone. Have a great one. Thanks.