 Thanks everyone for joining in. Today, I'm going to talk about this amazing framework which I've been working with for the last two years. It's called PyMu. It is an entire API unified framework for solving multi-objective and constrained optimization problems. I want to thank the EuroPython community for giving me this opportunity to speak with all of you today. I'll give a quick introduction to myself. I was there last year in Dublin at EuroPython 2022 because of unfortunate reasons I could not join in person this year. But yes, in the context of optimization, I'd like all of you to know that I've had real-life adventures with optimization. In my present organization at Polymerize, I'm working in the fields of material discovery where there are a lot of challenges with respect to objective-based experimental design. If you want to develop materials that have a certain requirement, for example, with respect to elongation, compression set, all the properties of materials, and you want to find out the right formulation that gets you the jackpot, that's where you need optimization, and we've worked extensively and used PyMu internally to make sure that our customers are given with the best optimization and optimized formulations. Previously, I worked for a company called DataPoem. It's a startup based out of in India, which is working in finding out the best marketing spend and attribution distribution so that our customers could maximize their return on investments based on their marketing spends. Before that, I worked at Sears in the US, where we were navigating different components for the supply chain management business of the company, making sure that the company has the right balance of inventory, the order times are appropriately managed, as well as the cash balances are maintained for a healthy cash flow. Apart from this, I've done a lot of consultancy projects with multiple organizations where I've helped in domains from manufacturing, fintech, and energy. Well, like all of us here, I'm also a Pythonista. I love Python as the programming language, and most of my career has been based on using Python to solve interesting challenges and problems. I've not just done optimization as a research on some notebook environment, I've actually pushed these solutions into production and its scale. A lot of challenges are completely different from the research environment to the production environment, and I'd like to discuss them with you today. Even though I've done a lot of research, I've solved a lot of problems with optimization, I still feel like an imposter because of the depth of this particular field. It's an age-old problem, and that's why it's still having so many complexities that we are yet to address. We have to solve for efficiency, we have to solve for peak performance, we want ingenuous solutions. All of these cases require some optimization. In general, optimization is actually a problem throughout all walks of life. Personal optimization as well as business optimization. We all want the best of all worlds, and optimization is a solution just for that purpose. Now, this core challenge is actually based from multiple businesses, and it has certain complexities that we need to understand. So wherever there is any kind of trade-off, there is always an optimization. For example, if you want to work on your health, you do a lot of good eating, you do a lot of going to the gym, but that also takes a toll on the amount of time that you have for other activities. So there is a trade-off. You can't be spending all your time in the gym. So if there's a trade-off, there is an optimization. The problems are usually very challenging than it may seem because the trade-offs also might not apply with the general sense checks that we have. I'll give you an example in a minute. Almost all of these cases have more than one optimal solution. So if you're working on an optimization problem, you don't have just one unique correct answer. There can be multiple possible answers depending on your priorities that you've set on the objectives. And in general, monopolies are rare, which is why multiple solutions always work better than a single solution. If you look at this particular diagram over here, I'm trying to optimize customer satisfaction and expenditure. If I spend a lot on my customer, I give a lot of refunds. I make frequent calls to my customer. I adhere to every need that they have. Obviously my expenditure of the business is going to increase. But if I don't do that, my customer satisfaction will be low. So the ideal case would be to have a balance of expenditure and customer satisfaction, which is why the two points highlighted here which are the extremes, for example, this is the lowest expenditure on the X-axis. And this is the highest customer satisfaction on the Y-axis. If I take this route, I have a high expenditure. So customer satisfaction, the maximum value is coming at a trade-off with an extremely high expenditure. Obviously we can clearly see that amongst this solution and this solution, I might have a trade-off. Similarly, in this particular case, where my customer satisfaction is actually becoming negative for the most minimum expenditure that I've done in the past. The idea again here being this particular point might be optimal in one way, in one approach, for example, in expenditure. But in a larger context of the business, both of these points are not optimum. We actually would be finding a point somewhere in this region, which would be a balance of nominal expenditure and a good customer satisfaction. To put it in perspective, this particular point that you see here, which has a significant amount of expenditure, but actually comes very close to the maximum customer satisfaction seen on this chart. Now, to understand the entire problem, let's first understand the universe of the problem. So in any kind of system, an optimization problem has four core components. The first component is all the elements of importance, which in machine learning terms, we can call as features. Everything that impacts the system becomes element of importance in technical terms in the optimization jargon. It's called decision space or decision variables. There could be multiple variables. There could be a single variable and that defines the complexity of the problem to a larger extent because the larger number of variables in the system, the system can have a lot of possibilities. These possibilities would eventually lead to multiple combinations in terms of values and that would lead to the larger decision space, which is difficult to optimize obviously. Then coming to realistic possibilities. Now, all of these elements of importance have some restriction, some kind of foundation. For example, if I talk about expenditure, I cannot spend $1 trillion. Maybe I get the highest satisfaction from that, but I will not be able to spend that kind of money. So there is a realistic possibility of how much I can opt for in each decision variable. Then there are system limitations. These are limitations that are not based on individual variables, but these are called constraints of the system, which could be as simple as the sum of two variables cannot exceed a particular value. This is a constraint of the system. Individually those variables could have any kind of range, but a sum of them has to be restricted by some margin. And finally, all of this is done keeping a goal in mind, the goal of optimization, which in technical terms is called the objective. The objective function or functions can be multiple and they also create the complexity of the problem. Hire the number of objectives to solve for, hire the complexity of the problem, more demand for convergence, more chances of suboptimal solutions, more chances of failure of the system because of the results generated. Let's take a quick mathematical example to set the stage for optimization. You see this mathematical equation here where we are trying to maximize a function fx. Whatever we want to maximize or minimize is called the objective function. There are three decision variables, alpha, beta and gamma. The relation for the function is here, alpha plus 13 beta squared minus 0.8 gamma, where alpha and beta are restricted individually, which is again the bounds of the system, by within this range, minus 12.5 to 18. Gamma does not have a lower limit, so it does not have an upper limit. It has a lower limit that it has to be positive. And finally, the constraint of the system is such that this particular equation should comply with the right-hand side. So square root of 1.8 alpha plus 3 gamma minus 9 should be greater than 45.61. So these comprise any optimization system. Obviously, the decision space can increase, there could be more variables. There could be multiple objective functions, not just fx, there could be another zx, gx, hx, et cetera. And finally, there could be multiple constraints. A feasible solution is the one solution or more than one solution that lie within the bounds of individual features that comply or agree with the constraints of the system and give a significantly maximized or minimized value compared to the existing function. Now, when you see this kind of function, it's a deterministic function where you can always find out the value by inputting the variables. There could be more complex scenarios where the relationship is statistical. Let's take an example. Let's talk about supply chain optimization. What are the different kinds of objectives that a supply chain management system optimization tool must meet? Obviously, the supply has to meet the demand. If a customer comes to my store and they request a product, if that product does exist in my inventory, if that product is sold by my organization, then I should definitely have that for my customer to buy. There should be no compromises on a healthy cash flow. If I order a lot of stock and my cash balance becomes negative, I will not be able to order anything more. Rather, I will actually come at a halt because there's no more cash to make orders. Even if there's a credit line, that's kind of possible. Then I have to adjust for the delays in logistics. There could be realistic issues in my logistics system where a plane gets delayed or a truck meets with an accident and my entire supply system might break for a day or two. So I should have an adjustment for those possibility delays. Then there's a cost of transportation. Now, I could transport everything from point A to point B, let's say using an airplane, but that costs a lot. But I could also transport from point A to point B using a truck, but that would take a long time. So I need to make sure that, depending on the timelines of customer demand, I have to choose the appropriate transportation method so that my product reaches the inventory, the store on time, so I can sell it. Then there are product expiration deadlines, then there's a balance of cost and quality of products I sell and a lot more. And this is just one example of supply chain. This could be similarly applied to finance, to healthcare, marketing, et cetera, et cetera. So let's translate this problem that we just saw into an optimization problem. What are my decision variables? I have my inventory capacity, I have my in-store capacity, I have my logistics cost, I have my supplier credit and so many other decision variables that are a part of the system. Then there are bounds. My inventory capacity should always be greater than 70%, otherwise I'm underutilizing it, but it could not exceed 90% because then I might be on a very red-linish kind of a situation where anything beyond that would actually lead me to put my products outside the warehouse, which is not optimal again. Then my supplier credit should also be in a certain range and so should be all of my other variables. Then there could be a natural constraint on the overall system where my net cash balance of whatever I have to pay, whatever I have to receive and whatever is pending in terms of checks, in terms of credit, should all lie between a minimum of 2 million to a maximum of 4.5 million. Any cash beyond that, my cash is being unutilized. Any cash below 2 million, I am actually in a risky situation. Similarly, my logistics capacity utilization should be greater than 90%. I want to use most of my logistical abilities. And finally, with all of these things in place, my primary objectives could be either a single objective, which is maximize profit, right? But that's usually very ambiguous and very vague because there are a lot of factors that relate to profit. So I might also consider multiple objectives, like maximizing product availability in my store, minimizing my delivery time for all my online orders, et cetera, et cetera. This largely is actually a translation of a mathematical optimization problem into a business optimization problem. Now, how do we define an optimization problem? Let's say you get a brand new problem and you are willing to find out what are the individual components, what is the larger idea in the optimization problem? So you first identify the problem type. Is it a single objective, multi-objective or many objective? Single objective, like it obviously means, is a single unique objective. Multi-objective is when you have more than two, close to three objectives. And finally, you have the most complex cases where you have many objectives, which is more than four, more than or equal to four objectives. Then between objectives, you might have a bias. While you want to maximize profit, let's say your priority of that maximization is 80%. The remaining 20% might be split into two other objectives. So you have a bias towards a certain objective and not all the objectives are equally weighted. Now, there could be add-on complexity to identification of this problem. Objectives may not converge. Two objectives might be completely inversely related to each other. So even if you find an optimal point, they might not be sensible for the business in general. Functions may have inflection points. If it's a statistical problem, there could be points where the differentiation cannot be happening, which is the case where certain algorithms like differential evolution might not work. So functions having inflection points can be a problem because anything to the right and anything to the left has a huge gap, which makes the convergence of the optimization very problematic. Then there might be undefined regions in the decision space which are areas that you've never explored in reality. So you don't know how the system behaves in those decision spaces. And lastly, in all the cases of optimization, we should definitely go for a fail-safe option, which is a foolproof method to make sure that if the optimization does not work, what do I do to give the best output that can at least be sustainable for the business? Finally, going to the next part of the problem, we evaluate the decision space. So we can't add all decision variables that would just increase the complexity of the system. We need to also synchronize domain expertise. For example, if I add four decision variables and three of them genuinely don't impact the objective, depending on a domain expert's knowledge, then I should, I reduce the complexity of the system by removing those. Finally, as much as possible, I should make sure that I reduce the size of my decision space. More the size, more the complexity, it's always better to have a great solution converged with lesser number of variables. And finally, these variables can have add-on complexity, not everything needs to be a number. There can be things that are Boolean. There can be things that have a certain precision. For example, if I try to work in a decision space where my entire real number is the space, it's very difficult to manage the precision of the problem. For example, to a business, a sales of let's say 1.5 million and 1.568346 million might actually be the same. So I should not increase the complexity of the problem by adding meaningless decision. Then there might be different variable types. There could be categorical variables, integers, real numbers, like I mentioned, and then there could be specific problems like mutual exclusion. Two particular things might not always work together in the system. So there could be a situation where if you're using this, you can't use the other decision variable. So that kind of input being fed to the optimizer is very difficult and we'll see some examples of it. Then after detecting the decision space and the particular bounds of the system, I will then go to constraints where I would want to consider the larger logical and sense checks. Here I would check for practicality, the relationship inconsistencies between my objectives and some meta-objectives that might follow along. And then I also explore larger limitations. For example, if I have not explored a particular decision space, I don't want to go to that decision space because then it will lead to some kind of inconsistency in the system and that loss of robustness might not be appropriate. Finally, my system should comply. It should be feasible. And in the non-deterministic regions in the joint decision space, there should be no value suggested. Then coming to the objective function, well, if you're lucky, your objective function would be deterministic. So it's just a math equation which you can plot, let's say, obviously to a maximum of three dimensions and try to solve it. If you can't visualize it, you might be able to put it into a system and try out different values and observe the different objectives so that you can point a particular optimal. But in most business cases, your objective function is not going to be deterministic. It's obviously going to be statistical, which is where you need to integrate the different algorithms in machine learning, deep learning, and other modeling tools. Now, an additional complexity could arise if you have a dynamic system. For example, in routing optimization, the decisions that you take are not some pre-created map. It's actually a real-time and agile system which is going to constantly monitor the process and take a decision, let's say, on every hour. Those kind of situations can be solved with reinforcement learning. There are some great examples out there for that as well. And finally, the entire objective function is the core driving factor behind the optimizer. So you have to make sure that the underlying pattern learned by the objective function, if you're building a machine learning model, has to be very robust and consistent. The dataset size should be large enough for the learning to be robust with any kinds of possibilities that the optimizer generates. Then there should be checks and balances for objectives. You might not allow the objective to go beyond a certain range, even though that's mathematically possible, that might not be practically possible. And then you could also have penalties for various kinds of strengths and high trade-off points where moving a small amount for a particular number might change the entire objective by a huge margin. Those are points where the optimizer gets confused. Welcome the Paimu API. The Paimu API is a state-of-the-art modular Python framework which has an object-oriented interface for the maximum complexity of an optimization problem, a multi-objective constrained optimization. Now, it has a lot of additional features apart from the core solution. And these features make it a great bet for you to try it out and use it for your optimization requirements. So you have the problems subclass in the Paimu class where you have single, multi, and many objective. There are ways you can choose gradients. You can paralyze using latest libraries such as Dask. They're building support in progress for Ray. And they also have the star map where you can use pool, threading, multi-processing in Python, et cetera, et cetera. Coming to the optimization part of the problem, you have custom sampling, custom crossover, mutation, mating, survival, et cetera. If you have not heard of these terms in the past, just to give you a quick brief, a larger set of optimization problems differentiate as the classical optimization problem and the evolutionary algorithms. Evolutionary algorithms have genetic algorithm, which have the NSGA2, it has differential evolution, et cetera. Genetic algorithms, basically, which basically work in the idea of how the human species evolved. So there was mating, there was mutation of genetics, there were crossovers between different pools, and then today as a human race, we are the best that has been seen from the past. Then survival of the fittest, all of these concepts actually arise from the evolutionary algorithm logic, I would say. And then there is the repair class where you can repair your results while the process of optimization continues. There is constraint handling, there's decomposition. There is termination of a solution, which is like early termination. You might not have to iterate the complete set. You can terminate early if some criterion is met. And then once you've done the optimization, Paimu API also provides an entire set of analytics on the optimization. So you can check what was the Pareto front, what was the kind of convergence that your optimization problem saw, what decision making and performance indicators you'd like to check, you can create custom classes for that, which will help you design a problem in a more realistic and post optimization analytical point of view. Now, there are some noteworthy features which are not easily available with the other packages that I've been to almost in the last two years. There is support for different data types. So you can create your own custom data type. You have mixed variables like binary, discrete, real, integer, et cetera. Then you have a large bouquet of algorithms for all types of optimization problems. You have performance analytics. You have executional optimization, where you can do hyperparameter tuning of these. You can create custom classes like I mentioned. And then you have different ways of adding constraints. So constraints might not always be in purely mathematical sense like greater than zero, equal to zero or less than zero. There might be cases where there is a higher penalty when you do not meet a certain constraint and a lower penalty when you do not meet another constraint. So between those two constraints, you are okay with not constraining your system with the second constraint, but you can definitely not cross or not follow the first constraint. Those kind of functionalities are available in PyMu. So lots of talk. Let's move towards the code and let me give you a quick glimpse of the kind of potential that PyMu has. So I'll just do a quick tip install, upgrade my tip and I will install PyMu, which is I'll import PyMu. Then I'll just quickly show you the version. This is the latest version of PyMu. If you are working with a previous version, that version does not have certain features. If you go to the PyMu website, you will actually find out all the new features that they flagged with the new keyword. So this is the latest version. It has most of the functionalities that they have available. There are some PRs that have some additional functionality, but those are gonna come up soon. I'll just quickly add my helper libraries and I'll start with the most fundamental standard optimization problem. So if I want to minimize x minus three square where my x can lie in the range of minus five to plus five, we can easily look at this system and say that to minimize this particular function, the value of x should be three, right? Now how do I transform this into a PyMu problem? So PyMu has something very interesting which actually helps in a lot of production level code where it has three ways of defining a problem. The first way is the problem class. The problem class actually works in a vectorized way. Why is that functionality developed? Because most of the multi-objective algorithms work with populations. You can relate the genetic algorithms evolution with a population size. Since a population size is obviously greater than one, it requires some kind of vectorization with NumPy, which actually speeds up the process of optimization. This particular class evaluates multiple solutions at the same time. From the problem class, there is also another class called the element-wise problem class. This actually inherits from the problem class, the only difference here being that it only chooses one single solution at a time. So there will be more iterations because you're only going through one solution. And finally, there is the functional problem class which is more towards functional programming ability. It inherits from the element-wise problem class, but instead of working like a class, it actually works more like functions. So objective functions are actually only functions in this particular class. Now, let's discuss the first way of designing a problem. Like mentioned, PyMu has a very object-oriented interface. So it's very easy to use. And it also has a very unified framework. So same problem can be solved with multiple algorithms. And one algorithm can solve multiple problems because there is a problem class and then there is an algorithm class. We'll see how that works. So I've just defined a deterministic problem where I initialize and then I take them at a class. I have number of variables one, which is coming from here. I have only one decision variable, which is x. Number of objectives is one because I only have to minimize this particular equation. The lower bound, which is notified by xl is the lower bound of x, which is minus five. And the upper bound is plus five. After defining the problem in the init constructor, I then move to the evaluate class. The evaluate method, sorry, evaluate method of the deterministic problem class that I inherited with the problem class. In this case, all of these variables need to just stay where they are. It takes an input x, which is coming from the sampler of the algorithm. And it takes an out, which basically goes back to the algorithm to see what results were generated after the particular sample that it created. Now, to show you an example differentiating between the problem class and the element wise problem class, I've actually added a print statement here to show the length of x in this class and compare it with the length of x in this particular class. And this only takes my out capital F. So capital F actually is the objective function representation. I just add to my out F, I add the x minus three whole square, which is my objective function. I initialize my problem with the variable name problem underscore P. A similar situation can be created entirely. The entire problem can be created with the element wise problem way of defining a problem. Everything remains same in the init constructor. In the evaluate class, everything remains same. The only difference what you will notice once I run the program is you will see the print statements where this would actually have a larger length more than one. And this would have an exact length of one because only one solution at a time. I initialize my element wise problem as well. And finally, the third way of creating a problem which is more functional. This is where I have my objective function written as a lambda. And then I define my problem where I take functional problem, define my variables lower and upper bounds and my objectives just come as the objective function defined here. There is no concept of the evaluate method. It's simply a functional way of writing the same problem. Finally, I have my problems list. I've put in all three problems defined above here. Coming to the algorithms part. So once the problem is defined, I can use the same problem with multiple algorithms. Apologies for the interruption. You have 10 more minutes. You have two more minutes. Got it. So then there are multiple algorithms. I have differential evolution. I have genetic algorithm. I have Neldermet. So these, all three of these are single objective optimizers. So S-O-O stands for single objective optimization. I initialize all three of these algorithms and put them in my algorithms list. And I've just run a simple loop where you can see I can run each problem type with every algorithm type available. You see deterministic problem with differential evolution, deterministic problem with genetic algorithm, deterministic problem with Neldermet. And I can run all my problems with all the algorithms available. This is a classical advantage over any particular package where I have to ensure that things compile with each other, which generally is not the case. You can see the solution defined for all of these was three. So my optimization did happen. And at the value of three, I'm getting a value very close to zero, right? It works for all cases. Now, let's add in more complexity. Let's have multiple objectives. Now, I want to minimize these two functions subject to these two constraints, which you can see here, and the two decision variables have this particular bound. Now, this is a classical BNH problem developed by Bill and Kohn. This problem can be clearly transferred from this mathematical equation to this particular scenario. I'm using the problem class again because I want to speed up the process of optimization. And since there are multiple objectives, I would need more iterations. So vectorizing the problem is always a better choice. I have two variables. I have two objectives. I also have two inequality constraints. A very interesting feature of PyMu is that it also allows equality constraints and inequality constraints separately. So every optimization problem in a lot of packages, you only have an inequality constraint. But PyMu also provides an equality constraint together with it. So you see here, I would transfer and rearrange these two constraints into these two formulas. So by default, the constraints are considered to be met when there is a negative value or less than zero value of the equation that has been passed. So I just take this 25 on the left-hand side and that becomes my first constraint function. In this particular constraint, I just take this entire equation on the right-hand side to make it again a less than zero constraint. And these would be my two constraint functions. Obviously, my two minimization problems, the F1X and the F2X can be written in this particular way. I'm basically getting access to array-like structures. So I am choosing all the rows and the first column here, all the rows and the second column here to act as X1 and X2. I'm putting my objective functions with a NumPy column stack in the out F, which is the objective functions representation, and the out G represents inequality constraints. So I'm stacking my constraint function one and two with this. I initialize my problem. I first try to plot my functionality. I just want to plot how does my F1 and F2 look like for multiple values of X1 and X2 in the linear space. So this is what my functions look together. Now to solve it with constraints, I'm going to use a very classical multi-objective optimization algorithm called the NSGA2. It is basically has the logic of non-dominated sorting using gentry algorithms, more on that in the references section. Then I simply use the minimize functionality of Paimu where I pass in the problem, pass in the algorithm, and then I can just run this. When I run this particular line of code, I get two parts. I get the result X, which is the input variables, and I also get the corresponding output. So result X and result F, I just plotted them. You can see X2 is constrained to the value three because the mathematical equation show here that X2 was to be bounded to the value three. And I was able to get a particular set of solutions using the optimization solver. Now, that's not it. Let's increase the complexity another notch. Let's take a statistical problem where I've actually taken a data set. I've modeled it with a non-linear regression model. There are multiple variable types and I want to solve it for two objectives, maximize and minimize. There are equality as well as inequality constraints. So I take this data set. This is a glimpse of the data set. I have these input features and I have two output features, inclusions and steel strength. Inclusions are anomalies in the final product of steel. So this should be less. So I want to minimize this and steel strength should be more and that's why I want to maximize it. I've actually created machine learning models in the back end. So I'm just loading these models and my current model looks like this. I have the scikit-learn pipeline where there are processors for numerical, categorical and I have the random false regress. I just want to do a quick check for seeing if my models work. To build the optimizer, I can use the pie move different data type options. There is the integer class, the real class and the choice options. Providing a bound for each of these, I can constrain my optimization problem by using multiple variables in their own personal space. This is what my mixed variables looks like. Then I define the constraints of the system. This is where I want the sum of two particular features, the cope percentage and the mineral percentage to be equal to 100 because that's what the logic of percentage says. If two things are in percentage, they should sum up to 100. And finally my cooling temperature must be less than the heating temperature. This is again a constraint of the system. I can actually transform this entire problem. First of all, by passing the WADS, which is mixed variables and specifying that in this particular case, I only have one objective. I can increase that to two in a minute. Then I have an equality constraint and an inequality constraint. In the evaluate function, I take the X sampled by the optimizer and convert it into a data frame and then use my predict method from the model to actually get the first objective, which I want to maximize. It is the steel strength model. I pass in my equality and inequality constraints in this fashion. I stack them together. Equality constraints are stacked in the out edge. And then I just use this line of code which considers mixed variable meeting, mixed variable sampling and mixed variable duplicate elimination to solve the problem using the NSGA algorithm. I add in my hyper parameters like population size, et cetera. And then this is what the outcome is. In the larger sense, I'm getting a higher steel strength, which you can see on the y-axis. It's towards the maximum. But it is not giving me the minimum inclusion. I transform this problem by adding two objectives and adding the inclusion model as well as a part of the prediction and then running the same piece of code. And finally, I get this kind of output where this is my steel strength and this is my inclusion. To put this in context, this is the final output of a multi-objective problem where I'm getting a high steel strength with a low value of inclusion. And these would be my optimal points. It's also called a Pareto front, which gives you a set of optimal solutions in the system. Because it's a random forest model, which is non-linear, you obviously don't get the best of things, but you can see the larger pattern is where I've been able to optimize my system. So we have a couple more minutes. That's it, I would like to take Q&A. Go on. That's it. Yeah. Cool. Thank you. That is excellent. We have five minutes for questions. If anyone wants to take the microphone to ask anything. Thank you very much for the presentation. Can you please give some example or share your experience of profit after solving one of these tasks by Paimu? Maybe there was an example about the steel, but is it was like from production or something, things? Right. So I've solved multiple problems using Paimu. The larger advantage of this is because of the object-oriented interface, it's very easy to put in production. So even if some problems are not converged with one particular algorithm, you have an entire set of other algorithms that might be able to solve the problem. Since we have parallelization available, you also have ways to optimize faster because there's support for dusk, there's support for threading and pooling. They're also in progress support for Ray, which is like a distributed hyperparameter tuning and optimization tool. It also has access to Optuna, which is one of the hyperparameter optimization tools. So in a larger sense, you have more ways to find out a solution in case one particular setting does not converge to an optimal solution. You can see here, there are hyperparameters like population size, there is number of generations, and there are a lot more, the mating, the crossover, the termination. So if in a particular setting, because a lot of this has randomization internalized, if a particular solution in a particular setting does not work, you have ways to find out other solutions using other algorithms or by changing the hyperparameters. That really helps in solving these problems in production. I've solved this for the material science industry at my current organization, Polymerize, where we can recommend new formulations based on existing data in such a way that your entire experimentation pipeline, your trial and error time actually reduces by a great margin because that helps you get products to the market faster. I've previously done this for the marketing and I've also done this for supply chain, so I can definitely vouch for the applicability of this API. Thank you. Any other questions? So I'm very a beginner in all that stuff, but I didn't understand the evaluate function that we saw because they received one possible value for X and they returned with the parameter out some things. How come they were only called once? How come the system immediately found the right X? I expected millions of attempts for different values of X actually. So maybe I missed something or is X magical a magical object? No. Okay. Sorry to confuse you here. Actually, I didn't run these cells. The code is here. It actually takes a lot of time to run, although there's parallelization which does reduce time by huge margin. This X is actually not a single value. It actually is generated by these mixed variables. So internally, the algorithm NSGA2 calls a sampler. The sampler first finds out all the variables which you see here is the mixed variable dictionary. Using these internal classes in PIMO, it actually finds a sample which is restricted by these bounds that I created here. So it is going to generate an integer value in between these two bounds. And with all these mixed variables, you're going to generate one single sample. I mean one single sample of the entire model. And that sample is what this X is, which is why upon taking this X, I transform it into a pandas data frame and then I call the dot predict method of the model with this pandas data frame. So X actually is a multidimensional array and it also is having different data types, in which case you can see here that I have a production type where you can see unit one, unit two, et cetera. So it also samples categorical variables which is another interesting feature. I hope that answers your question. Thank you. One more question. Take one. Cheers. Can you give some examples about how maybe deep learning can come into the picture of non-deterministic objective functions? Right. So my experience has actually largely been with deep learning models replacing this classical machine learning model random forest here. The advantage, I mean, the larger application is just that this dot predict, right? This dot predict, this model, instead of being a random forest regressor, it could be a deep learning model. It could be a stacked neural net. So those kinds of possibilities are also there. For example, if you have a stacked neural net that ultimately predicts the price of a house, one using the images of the house, let's say a convolutional neural network, another using, let's say, any kind of deep neural net, right, the ANN using the different attributes of the house. You can actually use them as two different objective functions and maximize both of them, right? The larger intention would always be to find a corresponding X that maximizes the output. So deep learning models obviously go from left-hand side to the right-hand side from the input to the output. If you want to go from the output to the input, you have a desired output and you want to find out the best possible input to generate that output, then you use optimization, right? I mean, then PIMO can help being, you know, having the deep learning model embedded here. Thank you very much. Thank you for joining from such a long distance. Thank you, everyone. Bye. Thanks, everyone. Have a good day. Bye. Please join us to the forum hall for the final keynote.