 All right, so please join me in welcoming Arian Ariyat, who is a senior data scientist from Gusto. It's a recipe kit company in UK, and she has been working on recommendation and forecasting models. She has done her PhD in computational chemistry from Imperial College of London, and will be talking about building personalized menus using Python today. Over to you, Arian. So yeah, first of all, thank you so much for having me. It's great to be here, with this weird circumstances. I'm Arian Ariyat, I'm a senior data scientist at Gusto, which I'll talk a little bit more about now. And I'm going to be telling you about how we've gone about building menus and parcelizing them. So hopefully the touch-up will make sense by the end of the talk. So if you haven't heard about Gusto, we are a kind of UK leading recipe kit delivery company. So basically you come to our website and you have over 50 recipes to pick from, and given the nice setup today, I thought I'd just quickly show you. So you come on the website, you have many recipes to pick from, and what you do is pick whichever ones you want. You pick a date, and on that date we send you a box full of pre-portioned ingredients. So if you only need one garlic clove, that is all we'll send you, or half a teaspoon of dried chilli flakes to your door with step-by-step recipe cards, which means that basically you get to do all the fun, cooking and eating and deciding what you're going to buy. But there's the more boring aspects like the supermarket and the planning you don't need to do. And quite importantly, it also means that you don't have to read waste because we send you the pre-portioned ingredients. So today, we're going to split the talk a little bit. So we're going to first be talking about building the perfect menu, and about midway through the talk, we will add the personalised aspect to it. So bear with if that's what you're looking for. And yeah, I think let's get going. So I want you to start with giving you a bit of an overview about what we mean with menu creation. So as you saw before, there's like 50 recipes on the menu that change weekly. So what we mean by menu creation is actually deciding what's on the menu. So we have a big recipe library, a thousand of recipes that are, the recipe developers have built. And what we do with menu creation is say, okay, well, these set of recipes is going to be live on this day, then the following week there's going to be this next set of recipes and so on and so forth. So as you can imagine, the menu is always like, the heart and soul of our product. It's what makes people decide, yes, I'm going to sign up to the service or for people who actually buy week two weeks, say, am I actually going to get a box this week or am I going to skip it? So it's really important for us that we get it right. So also just to give you a bit of context, just wanted to go through the menu size. So back in 2016, we had 12 recipes and we've been kind of constantly increasing them to now when we've got like 50 plus and the aim is not to stop there. So as a company, our kind of, the unique thing about us that makes us stand out is choice. So we believe that as a service, we want to offer meaningful choice. We want customers to come on our website and feel like they have a range of different recipes they can pick from, even if they have dietary requirements that might normally make that not happen. So whether you're plant based, whether you're gluten free, we want to make sure that everyone gets meaningful choice, which means that we've been scaling the number of recipes on our menu. Now, as you can imagine, picking 12 recipes by hand is okay, quite doable. And this is what our food team has been doing. So the food team has been manually deciding which recipes are going to go on which menu, on which dates. So when we actually plan for menus, there's several things that we need to take into account. So first of all, as I said, it's variety and choice. We want to make sure that we offer a wide range of really inspiring recipes. We want people to come to our website and be excited by the amount of things that they can pick. So that's like our first key thing. However, secondly, we are a business and not all recipes were born equal. Some are definitely more expensive than others. So cost is definitely one of the considerations that we need to have. So all the menus need to be planned to make sure that they're hitting the budget that is set from our finance team. And finally, we are actually a physical product. So we do need to account for several operational constraints and they range from, for example, you need to put tofu on the menu every couple of weeks because that's the agreement with our supplier. Or it might be that, you know, you can't put more than two or three recipes with broccoli or cauliflower on the menu because of a customer orders them all, you actually can't fit them in the box. So there's several of these things that we actually need to take into account. So as you can imagine, if someone's trying to do is with 50 recipes, keeping all of this in mind, making sure that all the operational rules are met, what ended up happening, I don't know if this gift will show properly, but it's a little bit of a whack-a-mole. Like, you know, this cat is having some fun, but, you know, the food team was getting to the point where it wasn't fun anymore because, you know, they were in the process of picking the menu and, you know, they'd be this recipe that actually, you know, violated one of the constraints so they'd stop it for something else. And then actually that in turn, you know, maybe was really similar to another recipe and that had to be moved. And it was just very hard to keep track of all of that at the same time. So we came together as a business and we said, right, do we think that it's actually time that data science can maybe, you know, try to help with things here? So hopefully I've described kind of the manual process and, you know, how kind of time-consuming it was becoming and how complicated to, you know, get right it was. So I'll tell you how we've moved to where we are now, which is data-driven with some manual touches and then kind of, you know, the absolute future is definitely like a fully data-driven process, which we're not there yet, but I'm sure we'll get there at some point. So, yeah, just wanted to start talking to you about the actual data-driven menu and how we went about it. So we decided to actually use genetic algorithms for this. So genetic algorithm is an optimization method that is actually based on evolution. So for anyone not familiar, I put a bit of like a glossary here, but it's actually really nice and intuitive to understand how this optimization works. So basically what we have is we have individuals, which in our case are menus, which are arrays basically of different recipes that create a menu. We then have a population, which is a set of all those menus. So we tend to use about 50. So they're all different individuals. And what you do is you randomly initiate this population and then start changing them, tweaking them randomly like what happened to, you know, genes. So it can undergo a mutation where like one or several of the recipes in an individual will be randomly mutated to another recipe. And you can have crossover, which is the combination of two different individuals to make new ones. So you take apart from each parent and then you create children. And what you do is for all these individuals, you have a certain set of objectives that you're trying to optimize for. And you keep the best ones that satisfy that and then undergo that whole process again. So during different generations, what you do is basically mutate and crossover until like the best genes, so like the best set of recipes, the best combinations actually end up appearing in the end. So I've also kind of summarized this in a chart. So what we do is initialize the population and now this one I wanna make like a really important point which is that actually we can already introduce all those operational constraints that we were discussing before, like the broccoli, like the tofu, we can already implement it here and make sure that any individual that we are taking into account already has all those parameters. So it means that, you know, something that the food team had to spend quite a lot of time checking and thinking about something that we already, you know, can very easily get rid of. So an individual is just simply not valid if it doesn't follow all these operational rules, which is already, as I say, like a big advantage. So what we do is initialize the population. And as I was saying, we mutate those individuals, we do crossover on them, we evaluate those individuals, we select the best and we start again. And this means that eventually the population becomes much better at those objectives that you're trying to get to. And after a few hundred generations is what we tend to use, we actually can make a final selection. So we have a set of really strong, you know, populations that then the food team can go and pick which one they prefer. So I'm gonna talk a little bit more about which objectives we used and why. But first I wanted to quickly say kind of what we've used to implement this. So we've used a library called Deep which is distributed evolutionary algorithms in Python. It's an open source library that has kind of like supports like a range of these. It's super fast to like set up and start prototyping. So it's all based on a toolbox. You can define functions according to each important, you know, mutate crossover evaluate. And then it kind of takes care of the rest. So it's, yeah, I really recommend it. And also something that we haven't taken advantage of but that I'm sure we will at some point is the fact that you can actually use parallelization which I'm sure will come in handy at some point. Yeah, so this is the basis of the way we've done it. Just wanted to kind of touch on the objectives. So basically when we first set out to do this we decided that two main things were going to be cost per meal. So as I said before we need to make sure that we're on budget. And then also we want to, as I said increase that average variety which we basically measure through a range of recipe attributes and how similar they all are within a menu. So we are working happily away on all this and then COVID hit and, you know, as you can imagine Gusto is away for you to actually get food to your house without having to leave. So demand kind of, you know, spiked and actually we started being constrained by our capacity because we physically could not deliver any more boxes. So in order to help operations we kind of again had a chance that well, this is already a multi-objective optimization and it's relatively easy to add extra objectives. So how about what we actually do is also introduce a minimization of the number of unique ingredients on a menu. The less unique ingredients there are on a any given menu that faster the boxes can get out the door so the more we could do. And obviously it was interesting because if you think about it these are two quite competing objectives. So, you know, it's harder to get more variety by having a smaller amount of ingredients but it was really easy to, you know, tweak the weights until we thought we'd got a balance that we felt comfortable with. So in terms of building the menu these are the results we've had so we've successfully reduced the average cost of the menu which is amazing. Really crucially we've also given the food team time to spend actually focusing the time on important things, you know, rather than like following loads of complicated rules that, you know, ultimately a computer can do better which was really important for us. And finally something kind of a little bit unexpected but also good is the fact that it allowed us to be really agile through a really challenging time. So the fact that we could add that extra optimization the fact that actually there were loads of proposition timeline changes because of all the uncertainty and we had to redo menus quickly and, you know, if we could only have the manual way that would have been very hard to pull off. So all in all it was great to have this in place. On the not so positive side we do still require manual changes. We know we're not like a perfect end state. This is kind of mainly due to need of a better definition of menu variety but, you know, we're working on it. So this is from, okay, thank you. So this is from like a menu building. Now we're gonna go to personalization. So, you know, obviously again, 50 recipes is quite a lot. So we want to use recommendations so way of helping users kind of navigate that large amount of choice. So what we do is we have an algorithm that orders all the recipes for each customer and we have a personalized collection where you land on. So the way we go about recommendations is quite simple. We actually use two types of recommendations. So first of all, we have a content-based part of our algorithm which is based on the similarity between different recipes due to their attributes. So we uploaded all of our recipe and ingredient attributes to Neo4j, which you can see here. Neo4j is a graph database so it means that everything, all the data is stored in relationships. So, you know, this pasta, well, both of them have cuisine. Italy, this pasta has protein four, this pasta has protein lamb. So this allows us to really kind of understand how similar any two given recipes are based on the attributes that they have that they share. And also, if we couple this to what customers have ordered in the past, we have a way of understanding which recipes are most similar to your past orders. So this is really valuable because we can kind of predict what you're going to buy but equally, you know, sometimes you want a bit more excitement and you don't want to cook what you've cooked before. So we also introduce collaborative filtering and this is more based on user similarity. So people like you also liked. And what we did for this was actually get all of our users and they're ordering behavior and create five distinct clusters. So you can kind of see what they are, like pescatarian, vegetarian. And so each of these clusters, we can find out what recipes they overindex on. So what did they buy off most? That differentiates them from the rest. So each customer is given, you know, a classification. So we understand which cluster they belong to and what they overindex on. So what we then do is kind of put these two together. So we have the content-based approach, we have the collaborative-based approach and we combine them using Board Account, which is a method to basically put together two different rankings in a fair manner, you know, that represents the two different ones. And what we create is a hybrid ordering. Now, on top of this, we add a bit of curation in the sense that, you know, maybe we are trying to, you know, proposition would like us to push a bit more for certain recipes because that's what we're talking about on Facebook or whatever. And that's how we end up with like a final personalized order that hopefully, you know, helps customers pick their recipes more easily. So in terms of personalization, again, like we've kind of successfully seen an increase in conversion when we applied personalization, which is great news. And we do see customers kind of consistently ordered from the highest fracks of the menu. However, we did do this a couple of years ago and I think we need to kind of take it, now take it to the next level. So we currently don't have dynamic recommendations, which we knew was super important. And we currently don't have a way for customers to actually give us feedback on how right or how wrong we're getting that personalization. And just to finish off, you know, I wanted to talk a little bit about what's coming. So we're going to be tackling recipe development. So as I said, we've got our recipe library, which our recipe developers keep adding new recipes too. So we're going to also try to take a disdriven approach to understand where those gaps are in the library and hopefully filling these gaps should be to better menus as well. But most importantly, I think is the fact that, you know, these are the two building blocks that we've got at the minute, but 50 recipes isn't the end destination. When we have lots of recipes and actually you can't scroll all the way through them, we're going to need to build much stronger links between these two because your recommendations is going to have to follow the principles of many creation to ensure that the recipes we're showing you make sense as a whole, make sense as a menu. So this is what we're trying to scope out, which is very exciting. And now to actually finish, I always like to kind of, you know, share I think learnings that we found along the way. And I take these three from this big project. First of all, good discovery really sets you up for success. You know, this was a big project with the food team, with analytics, with UX designers and really taking that time to, you know, all understand what different points of view we've had, what different learnings and what we are actually trying to solve. I think really helped us down the line. So even if you're like, you know, really excited to get going with something, sometimes taking a little bit more time to discuss, actually pace off down the line, I think. Secondly is plan for early value release. You know, I think sometimes you really want things to be perfect, but actually, you know, we don't need all the bells and whistles to really start delivering value. So for example, with the menus, we know we're not at the end, we know they're not as perfect. That doesn't mean that the food team, you know, doesn't find value in them. Even if they still need to do some swaps, the fact that that is there to help them is really good. So how can you plan your work to actually make sure that you can deliver early? And finally, I think this one kind of goes without saying, but I think it's always good to explicitly think about. It's ensure you're thinking about tomorrow, you know, make products and architecture decisions that, you know, you're not going to regret later that are scalable because you don't really know what's coming. So try to make it as agnostic to the situation as possible, you know, with the menu creation, we know it works for 50, but we know it also works for 250 or 4300. We know we can add different objectives. So I think hopefully that will pay off in the future. And yeah, I think those are my lessons and my story. So just wanted to thank you so much for listening. I hope you find that useful. It's very weird not to get any kind of feedback and not know if you're talking to yourself. But yeah, thank you so much. Thanks a lot for delivering such a great talk. It was really nice to hear you. Thank you for having me. Thank you.