 Yeah, so I'm Stefano Armon. I'm an assistant professor in the Department of Computer Science. Here at Stanford, I'm affiliated with Stanford Artificial Intelligence Lab. As you might have heard, it's a really exciting time to be doing AI research. Thanks to recent advances in sensing technology, we are able to gather massive data sets in a lot of different domains. We have new techniques to store this data, to communicate it efficiently across networks, to process the data at scales that were not even imaginable a few decades ago. And what this means is that we are able to collect these massive data sets, and we can extract information from them. And these techniques have really revolutionized many subfields of computer science, and in particular, they led to some major improvements in my subfield, which is artificial intelligence. It really revolutionized the way in which we think about problems, the way in which we solve the problems, and it really bootstrapped this approach based on machine learning, based on probabilistic modeling, based on probabilistic inference. So in the past few years, we have seen an enormous success in a lot of applications. We are building machines that can imitate human behavior at increasingly more complex tasks. We have computers or even phone these days that can understand when we speak to them. They can answer simple questions. They can recognize objects, faces, and images, and tag them, say, on Facebook. We have automatic machine translation tools, like Google Translate, that can translate between different languages. Soon, we'll have systems that can translate automatically spoken dialogues between two different languages. You've probably all seen Google cars driving around and collecting data. Soon, we're going to have autonomous driving systems that are going to be deployed and are going to be on the streets. So there is really a lot of excitement about artificial intelligence. A lot of companies are investing heavily on AI. They're trying to recruit all the best people. What this means is also that there is a lot of emphasis. There is a lot of focus on applications that have market value that can be monetized in some way. This is great because, of course, they are meeting demands that are really out there. But I think if we take a step back and we look at the big picture, we'll see that there are other kind of needs, other kind of goals that we should think about. We should be trying to solve as a society. And these include things like ending poverty and hunger in the world or finding sources of affordable and clean energy, which is probably very dear to this audience, or preserving the planet for future generation or fixing climate change. And these are very important challenges that have been recognized by the United Nations. They've been adopted by over 150 countries. And my research is really here. It's the intersection between those two worlds. I was trying to build a bridge between these two worlds. It's a new field that we're calling Computational Sustainability, where we're trying to find ways to apply all these cool ideas, these cool techniques from computer science and find ways to use them to help solve some of these big, important sustainability challenges. And my research in this area is centered around three main themes. One is the use of very large data sets. And we call big data. We always use artificial intelligence and machine learning techniques to try to understand these data sets, try to infer new knowledge from the data, try to build models that we can use to predict, and also optimization. We want to use these models to make better decisions. We want to optimize systems. We want to figure out what's the best way to allocate limited resources. And I've worked on a number of projects in this space, ranging from finding policies to optimally manage network resources over time. Recently, since moving to Stanford, we've been looking at modeling groundwater systems and weather systems. Of course, we all know about the drought that is affecting California. These systems are not well understood. We have a lot of data from remote sensing. And we think that using machine learning and this kind of data-driven modeling approaches can provide some interesting insights into these systems. I worked in the energy space working on how to use AI techniques to improve the efficiency of energy systems and also helping out material scientists to speed up the discovery process of new materials. They are generating these very large data sets through high throughput experiments. And we developed some techniques to automatically analyze this data and speed up the discovery process. And recently, I've also been working in the context of development. How can we use machine learning and sequential decision-making ideas to understand poverty in the developing world? Can we use large data sets from satellites, for example, to automatically understand the images and try to build models of livelihoods and try to figure out the best ways to use the limited resources that we have to fight these issues? Now, given that this is an energy seminar, I'm not gonna be talking about this other project. Instead, I'll be focusing on the materials discovery work using machine learning in the context of material science and on the optimization of energy systems. So let's get started. Why materials? Well, materials technology has been historically a defundation of successful thriving civilizations. If you think about it, there are entire historical periods that are named after a material, stone age, copper age, bronze age, iron age. Historically, new developments in materials have always led to these major shifts, major changes in the economic aspects of the societies into the other societal aspects, the culture. So if we want to achieve clean and sustainable energy, I think it's pretty clear that we're gonna need to find ways to speed up, to scale up the discovery process of new advanced materials. For example, we know that hydrogen fuel cells or solar fuel cells are very promising enabling technologies for lots of applications like electric vehicles or the storage of renewable energies, but their performance is essentially dependent and critically dependent on the kind of materials that we use. So if we wanna push the boundaries even further, we wanna make them more cost effective, we wanna make them more widely used in practice, we're gonna have to find new materials. This was also recognized by the government a few years ago, the president launched the Materials Genome Initiative. It's a multi-agency initiative that is aimed at accelerating the pace and reduce the cost of discovery, deployment of advanced material systems. Now, what does this have to do with computer science? It turns out that this area of research is really opening up new interesting research opportunities for computer science research, as was highlighted here in the original slide from the Materials Genome Initiative from the government. Clearly, material scientists have been using computational tools for a long time, simulations, quantum simulations and things like that, and they are very good at it. Some of the biggest supercomputer centers that we have in the country are funded by DOE and are used precisely for this type of research, but I think there is a shift going on, there is a paradigm change. We're starting to see scientists that are collecting very large data sets through, say, high throughput experiments, and what this means is that it's really creating opportunities for using different types of tools, different type of ideas from computer science, not just simulations, but things like machine learning, things like optimization. Can we figure out, can we discover new knowledge based on data? Can we figure out ways to optimize the data collection process, or in figuring out use machine learning to guide the way in which we design experiments? So in some ways, this reminds me of the early days of computational biology. Scientists were collecting these very large data sets and from genomics and they had no way to process them by hand, so they had to come up with new computational models, new computational techniques to actually make sense of the data to actually do something useful with it. So my interest in this area started back doing PhD at Cornell. There I was interacting with Bruce Van Dover who's a material scientist leading the Energy Materials Center at Cornell and John Gregoire who was also a student back then and recently joined the Jacob, the Joint Center for Artificial Photosynthesis at Caltech. And what really got excited me about their vision was that they were trying to speed up, to scale up, to accelerate the discovery process of new materials by automating the process as much as possible. Essentially, this is a high level picture. I apologize to the material scientists in the room if it's not exactly accurate, but the idea is that they set up, first component of their pipeline is a high throughput materials synthesis component that is able to generate a lot of, quickly a lot of candidate new materials. Like John has a system that can generate a few thousand new materials per day. The way I look at it is some kind of inkjet printer that can generate a lot of candidate new materials in a single sort of batch experiment. Then they have a high throughput screening process that can be used to collect data about the materials that they generate. And this pipeline produces a lot of data. That's the challenge. The challenge is that we generate these massive data sets and at this point the bottleneck is really the human time to process the data, interpret the data and try to make sense of it. So what I'm interested is this part which is how do we actually process automatically? How do we scale this up by introducing some clever computational techniques? And there are sort of, there is a whole stack of things they can do with the data. It starts from simple things like data cleaning, things like quality assurance, trying to do anomaly detection, figure out if things are going wrong in the experiment. All the way to try to identify the crystal structure of the materials, which is something I'm gonna talk about. And hopefully this is something we haven't done but it's sort of like our vision, our dream is to actually close the loop, figuring out ways in which we can extract new knowledge from the data, we can use it to guide the design of new experiments, figure out what's a promising candidate material that we can try and or even in a simpler terms we can even maybe just try to guide the data collection process, figuring out whether we should abort an experiment because it's not promising or we should focus on a particular area because that's where the interesting things are going on. So it's a very exciting I think area for computer science research and what I'm gonna talk about next is one piece that's one step in this direction. So here is how their approach works again on the very high level. They start with a silicon wafer in a vacuum chamber and then they spatter three metals at the corners. And what they get is they get continuous composition spread where at different points on the silicon wafer correspond to different concentrations of the three basic metals. So say up to this corner you get 100% of metal A down here 100% of metal B, here in the middle you probably get 50, 50 and so forth. So you get deliberately homogeneous composition spread on the wafer. And the idea is that with a single batch experiment you generate a lot of candidate materials. Hopefully they're gonna have different physical and chemical properties and hopefully you're gonna find something useful by doing like this single experiment. So after you do this you get what they call a combinatorial library. It's made by two things. One is a set of samples, just points that you actually for which you collect data. And the other one is a distribution of crystalline phases within each sample. Here I'm showing them with this. I don't know if you can see the colors very well but these areas are pure phases. They correspond to areas in which the crystal structure doesn't change. So then the challenge here is that phases can be mixed. There is a distribution of phases over points. And so the phase map identifies is this map telling us at every point what phases are present there and what's the relative composition of the two phases. And yeah, the challenge is that we don't only get to see pure phases but we get to see regions where there is a composition of more than one. So how do we, the goal is gonna be to try to identify this phase map. How do we get some information about the phases? What the kind of data we were playing with was from X-ray diffraction. The idea is that you collect a diffraction pattern by hitting the point with a high energy X-ray, say generated by a synchrotron and you get a diffraction pattern, something that looks like this, that characterizes the underlying crystal structure. And if you integrate it sort of radially, you get a signal for every point that looks like this and the position of these peaks are characteristic of the particular crystal structure that we have at that point. Now the challenge is the following. The challenge is that when we have points that are where we get to see a mixture of more than one pure phase, the spectrum that we get, the measurement that we get for these kind of points is gonna be a composition, is gonna be a mixture of the patterns that correspond to the two pure phases. So it's gonna look something like this where we have a composition of this piece and that other piece. And the challenge is to disentangle these two factors and figuring out what are the individual components. This is what we call in signal processing is the cocktail party problem where you are in a room full of people, chatting between each other, you're hearing a signal which is the mixture of all the voices in the room and you need to somehow figure out and you need to be able to focus and try to understand what a single person is saying. So this is the phase map identification problem. We have our input which are the diffraction patterns for each individual point. The output what we wanna get is this phase map, the pure regions, the mixed regions and we wanna identify the X-ray diffraction pattern that characterizes each individual pure phase. And the challenge is that there are also a number of constraints that we want this phase map to satisfy. We know that these phases, these regions are gonna be contiguous in that space we know that there cannot be mixture of more than a certain number of phases in each point. Even points within an individual phase don't necessarily have all the same spectrum. The spectrum can shift by a little bit but in a somewhat predictable way. And of course there is a lot of noise, there is air scattering, there are all sort of things that are going on so we need to be able to build a system that is robust to that. Is the setup clear up to here? What's the problem? Any questions? So what's the basic idea? The basic idea is we're gonna do unsupervised classification. So the picture looks like this, we have a bunch of objects and they have different characteristics say color and shape. We have oranges and apples. Oranges and apples are objects, they have different shapes and if you look at each individual point and you look at the characteristics of that point in this feature space based on color and shape we're gonna get a bunch of points. And the goal is to be able to somehow cluster them based on these two characteristics. So we need to be able to figure out that there are two types of objects, one that looks like oranges the other one that looks like apples. That's gonna be the basic idea. Mathematically what we can do is we can formulate this as a matrix factorization problem. We build a matrix where we put in every column one of the diffraction patterns and essentially what we wanna do is we wanna find a lower rank approximation of this matrix. We wanna find k basis vectors that we're gonna put in this yellow matrix such that each individual diffraction pattern that we measure can be written as a linear combination of these basis vectors. That's the basic idea. Here we're assuming that we know k, we know the number of faces. The challenge is that the data is non-negative. It doesn't make sense to get negative numbers here. There are sort of electron counts. So we also need to enforce that this matrix is W and H are non-negative and true-wise. So that's why it's called non-negative matrix factorization. So you can formulate this as an optimization problem. You have your data which is in this matrix A. You look for new matrices W and H such that they give, it's a good lower rank approximation of A. Think of that norm up here is like Frobenius norm. And we have to enforce these constraints that the matrices are both non-negative. So if you run it on some data, it turns out that it scales pretty well, although it's known to be intractable in the worst case. In practice there are some reasonable heuristics to solve this problem. The problem is that it does not enforce all those constraints that we know the true phases must satisfy. So for example, here is a system with aluminum, lithium, and iron. It has six phases shown up here in the first row. The results that you get if you run non-negative matrix factorization and MF for short don't match very well the ground truth. These images on the bottom row should be similar to the one on the first row. What happens is that it produces phases that are disconnected. It produces mixtures of more than three phases in this case. So it doesn't give a good solution. It doesn't provide the answer that we were looking for. Yeah. Is that done with known good data? That they were synthetic. It synthesizes nowhere. Yeah. So it's not an error problem. It's really just the optimization problem we're trying to solve is not the one we want. It's not a good model, basically. You're trying to find for causes. Essentially, you can find a better fit, but that violates these physical constraints. So it's finding a solution that is non-physical, but it minimizes that objective function. Because we have the wrong objective function. Yeah. Can you say how you chose your features? How what? How do you choose your features? Full spectrum here. You don't think? Yeah. So our approach was to incorporate some prior knowledge into the model. Here we used what we call a satisfiability model theory, which is a way to combine logic with arithmetic. So you have a very expressive modeling language where you can reason about automatically about real, valid quantities, like the kind of measurement data that we have. But it's very expressive in the sense that you can encode complex relationship between the variables, like trying to, to some extent, at least to have first cut, trying to encode what we know about the physics of the system, those constraints that we were talking about before. So if we do that, we get a new optimization problem where we have some variables that represent the known basis patterns. And the idea is that we want to make sure that they explain the measurement data as well as possible, and they have to satisfy all these constraints. It's a complicated formulation. I'm not going to go through the details, but to some extent, you can view this in the following way. We have the same objective function as before. Try to find a good low-rank approximation. But it's going to have some extra variables, and it's going to be subject to more complex constraints on the variables. It's not just going to be non-negativity. It's going to be a bunch of other constraints that encode the prior knowledge that we have about the system. So for example, we might want to say that each measurement point, each sample, is a linear combination of at most S basis vectors, where S might be smaller than K. And to do that, you can introduce some extra variables that are binary, and you can encode the fact that they must be smaller than S, and this is encoding, in terms of constraints, the prior knowledge that we have about the model. So at the end of the day, we end up with another optimization problem, which is non-convex, mixed integer, non-linear. So it's a pretty nasty problem to optimize. The good news is that there is some structure. If we fix W, then the objective is quadratic in H, and the constraints are linear in H. So it becomes still not an easy problem, but it's a mixed integer quadratic program. And we have pretty good solvers to do that. You can relax the binary variables. You're going to get a quadratic program. You can use that to do some kind of branch and bound search. And these tractable relaxations can be used to prune the search space, and you can actually, in practice, you can solve pretty large problems reasonably efficiently. So the strategy is fairly simple. We can just do coordinate descent, block coordinate descent on the variables. So we start initializing the variables somehow, say random. And then we keep alternating between optimizing the H and the W. And at every step, the objective function is guaranteed to improve. And it also satisfies the constraints that whatever we know or the prior knowledge that we have about the system. And this turns out to work better in practice. It's still not perfect, but better. This is the same system as before. The dots corresponds to the points that we classify as belonging to a particular phase. The shaded area corresponds to the ground truth phases in this system. So it's doing a reasonable job in this case. It's still missing some points as the borders, but those have very little concentration of the phase. So it's understandable that it's getting the wrong answer there. So by incorporating some prior knowledge about the physics of the system, we can get significantly improved accuracy and we get better phase maps. We tried it with some real-world data. Here, it's harder to evaluate the results because we don't have ground truth. What we can do is we have some sort of way to verify that the results make sense. In this case, the system, we have like a nice user interface where you can plug in the data that you get from the synchrotron and shows you the phases and does all the plots for you. In this case, it recovers a phase. It's what it calls phase one that matches fairly well one of the ones that we found in the literature, which is sort of saying that, yeah, it's probably doing the right thing. We used it in other data. This is a tantalum rhodium palladium. I came up with this phase map. And this purple region here seems to correspond to a region with high fluorescence activity. So again, it seems to indicate that the phase map that we get kind of makes sense. But again, we don't have ground truth. So these are harder to evaluate. So now we recently started some collaboration with people at Slack, EGN and Apurva. They also have, they are sitting on a lot of data, this largely different type of data, but they have a pipeline that can generate a lot of absorption data. So we're trying to see whether we can apply the same. We have a master's student that is trying to apply this idea, these techniques, to the data that they have been collecting up at Slack. So any questions on this part before I move on to the next topic? Good. So this is, again, another collaboration with people at Slack. I think Daniel is there, so he might tell you more about the actual domain, what's really going on there. But the idea is that we have the LCLS, which is this Linux Coherent Light Source, which is the world's first X-ray laser, 10 billion times brighter than any other X-ray source before it. Very complex machine, difficult to operate. It requires tuning. It's currently done by operators, sort of in a pretty much ad hoc way. And given that the operating costs are about $1,000 per minute, Daniel approached me at some point and said, well, it would be nice if we find a way to make the parameter tuning better, whether we can do it automatically, we can make it faster, we can make it work better. And the good news is that they have a lot of data. They have an archiving system that is recording 200,000 independent variables once per second. And it goes back several years. So we have lots of data available, a lot of data points. And the question is can we use machine learning methods to come up with better ways to automatically tune the machine. So here we've been taking a Bayesian nonparametric approach. So these are very preliminary results, but they seem to be very promising. It's a pretty exciting project. So the idea is that we want to come up with a function that is able to map the parameters that we can tune and all other independent variables to the output we care about, which is how well, what's the energy output, how well is the machine performing. So we don't know what's the relationship between these variables and the output. So what we're going to say is we're going to start with a week prior that is just saying, well, whatever the function is, is going to be some smooth function. If you change the parameters a little bit, the output is not going to change by much. And then what we do is when we get training data, we have data from the past. We know how well the machine performed when the parameters were set in certain ways. So we can use this information. We can use this evidence to update our prior and get a posterior distribution over these predictive functions. So it will tell us, and by doing this, we are able to predict how well the machine will work when we set the parameters in some new way that we haven't tried before. And the nice thing is that we don't only get a single value, but we also get some measure of uncertainty. So we have an idea of how much we believe the prediction that we make about how the machine will perform if we set the parameters in some way that we haven't tried before. So the basic formulation is nice, but it doesn't scale to large data sets. It scales roughly as n cubed. So the number of computations that you have to do is roughly n cubed, where n is the number of training points. So we have to introduce some approximations to make it scale. Essentially, it all boils down to using low-rank approximations of some covariance matrices to make it scale. And you introduce some approximations, but we can scale it up to larger training sets. And it seems to be working reasonably well. It seems to be able to hear it. I'm showing the predicted energy versus the true energy. It seems to be doing a reasonable job at predicting on new data how well the machine will perform as we set the parameters in different ways. And yeah, so this is still preliminary, but it's pretty exciting. And the next step that we are looking into is to do some kind of Bayesian optimization. Here, the idea is that we want to be able to optimize this function that's mapping parameters to how well the machine performs in an online fashion. So we want to figure out a way to search this huge parameter space efficiently. And this is particularly suitable for settings where evaluating the function is very expensive. Like you actually have to do an experiment to figure out how well the machine performs at a given parameter value. So we can use a very similar approach. We have posterior over the predictive function that is telling us how well we think the machine will work if we set the parameters in a certain way. And then what we can do is we can try to compute, we can do some optimization on top of that to figure out what would be the next parameter that we should try in order to, and there are several objectives that we might be interested in. One is to say how should we set the next parameters so that we maximize the probability of improving or we could try to maximize the expected improvement. We could try to find the point, the parameter settings that will maximize the expected improvement. So we have what we call an acquisition function that will look like this. For example, in this case, it will tell us that the best point that we should try is somewhere around here. Because in some sense you want to be close to the points you already know that work well, but at the same time you want to be able to take into account uncertainty. You want to be able to make sure that you're going to be exploring areas of the parameter space where you don't know how the machine will perform. So this is a classic problem that we have in this kind of sequential decision-making problems. It's called the exploration versus exploitation trade-off. On the one hand, you want to be exploiting. So you want to try out points where you're near or very close to where you know that the system is going to work well. But on the other hand, you want to make sure that you explore, that you're going to try out parameter settings where you don't know, you have no idea how well the machine will perform. And this approach, this Bayesian optimization approach, provides a sound and a principled framework to deal with this classic trade-off in a principled way. And so we're working towards implementing this and trying it out directly on the machine. So questions on this part? Cool. So the final project I wanted to talk about is one project I did during my PhD on trying to optimize the efficiency of multibattery systems. As we all know, electric and hybrid vehicles are a very promising approach towards more sustainable transportation systems. One of their main limitation is given by their batteries. They have high cost, limited capacity. They might be inefficient. Maybe there are better ways to improve their efficiency. So what we were looking at was a way to see whether we can use AI techniques, similar kind of optimization ideas to improve the efficiency of these systems. And in particular, we were looking at multiple battery systems, where this was an idea that came out from CMU, the ChargeCard project, where they were considering these setups where you have multiple batteries with different capacities. Some are big, but inefficient, and some are small, like a supercapacitor, but very efficient. And the hope is to get the best of both worlds. You want to have large capacity, you want to have high efficiency, and you want to have lower, low cost of ownership. So the way I think of this is sort of like a hierarchy of the classic hierarchy of memories that we have in a computer system where you have disks with high capacity, low latency, high latency, and then you have a cache that is very small, but very, very, very fast. And by being clever about the ways you use this hierarchy of memories, you can sometimes approach, you can sometimes get really the best of both worlds. So the question there was to what extent can we actually exploit the fact that we have multiple batteries? Can we actually manage them in a more efficient way? So that was sort of the problem. The problem was take past driving behavior, look at current vehicle conditions, like position, speed, time of the day, everything you can think of, and try to figure out what we were trying to compute was how to allocate the demand between the different batteries and how to store the energy that we get from breaking. And it turns out that this requires a good model of driving behavior. We want to be able to predict sort of ahead of time high current events corresponding to accelerations or sudden decelerations. So what we did is we tried to learn a model of driving behavior using a large data set of commuter trips that were crowdsourced in the U.S., so they were volunteered by people across the country. And so we used that data, we used open street maps to have an idea about traffic lights and signs and speed limits and some elevation map from USGS to try to figure out whether people are gonna downhill or uphill. And we combined that with some optimization approaches and again some machine learning to try to generalize to new settings that we hadn't seen before. And what we got at the end was a new system and intelligent, what we call an intelligent management system that was able to significantly improve, led to some significant energy savings for the simulator, for the setup that was being considered by the people at the ChargeCar project. So yeah, this is the end of the talk. I guess you might have heard about this growing concerns that people are raising about the future, about the threats that AI is posing to the future of humanity. Elon Musk and Stephen Hawking, they're wondering whether robots will take our place and then we'll all be out of jobs. I think we can take a more positive attitude here. We can think about all these recent advances and all these new techniques and all these new methods that we're developing. We can think about ways to use them to have beneficial influence on society in the context of energy, sustainability, conservation and many more. So it's an exciting area to work on because it's really also opening up new interesting opportunities for doing CS research. It's really like a two-way street between computational sciences and sustainability sciences. Sometimes we come up with new algorithms. We figure out ways to apply them to sustainability problems. Other times we are encountering new problems that people have not looked at before. We have to come up with new computational models or methods to deal with these new challenges. So yeah, thanks for having me here. If you guys have any problem that sort of seems related to this where you think you might need help and you think you might be able to apply some of these techniques and some of these ideas, please talk to me. I'm happy to talk to people and see whether there are opportunities for collaboration and finding ways to apply techniques like this to your problem. Thank you. Question? In all of the applications you showed, but particularly, how do you go about example-wise? So I mean, given that you try to source across the country, presumably you have more opportunities to do that. So how do you go about that in your work? Well, you know, like we always try to validate. These other results are always validating cross-validation. So to some extent, we try to account for that. Of course, we always use regularization. We try to build systems that will, to some extent, generalize to new data as much as possible. And it's always hard to say whether the system will actually work on new data that you haven't ever seen before and it cannot do it in general, because if the system has not been trained on that particular situation, it's not gonna work. So I think what's interesting about, for example, introducing prior knowledge that you have, say, the physics of the system, then we know those are sort of constant, they won't change. So that's a good way to make sure that the system will work well, will work in the way you want, also in the new scenarios. But yeah, generally speaking, we are definitely, these are data-driven models. And so, you know, it is like garbage in, garbage out. That's, we cannot get around that. So if you were to, you know, do that analysis and then try to apply it to actually, you know, a better management system that a lot of people might put in a card of, what would be your recommendation? How to, you know, if you wanted to actually use that, you're not gonna have all the data. Well, the typical way these things are done is you can just collect more data and then just, you know, this, the way we think about these problems is not, we don't care about producing an output. What we care about is producing algorithms, such that, no matter what you put in as output, it's gonna produce, hopefully, a good solution. So what we are recommending is our algorithms and models that then companies or people that are interested in this can then apply on their data and I'm sure improve our results. I mean, we're always, in academia, we're always limited in the kind of data sets that we have access to. So there is no point in trying to compete with companies on that because we just cannot scale to that size. We don't have the resources to do that. But what we can do is we can provide algorithms, we can provide ideas and then companies can sort of apply them to their particular case. The last example, what's the parameter you're optimizing for? Is it the size of the different batteries in the car? Is it the management data? So we were trying to optimize for the efficiency, which basically, the one way to think about it is we were trying to keep the output as stable as possible. So we're basically accounting for Quaker's law and that was sort of the main thing we were optimizing. So that's your objective, so what can you change? What you can change is allocate the how much energy you take out from the different hierarchy of batteries, how much energy you move and where you put it back when you get burning breaks, the car breaks. That was basically what you can play with. So the demand is fixed, you cannot change that. What you can do is you can try to keep sort of as stable as possible so that the inefficiencies are hopefully smaller. But it was based upon the simulator. The simulator was telling us what's the efficiency that you're getting at the end. Yeah, yeah. I have a question about the earlier example you showed, the material science example, I think it was a Rotium ternary alloy where you showed different fluorescence intensity. My understanding was you did this simultaneous, the position of three elements and then a robot or just an XY stage was measuring fluorescence on a coordinate system. Also X-ray diffraction and fluorescence. Also the X-ray diffraction. Where exactly does now the machine learning aspect come into play in this particular example? In that example was to identify the crystal structure automatically, just purely based on X-ray diffraction. And then the idea was to always sort of try to relate the phases to the properties that we care about, like fluorescence. So, but we were just looking at the X-ray and trying to see whether we can come up with the phases, the phase map. That's something we're actually thinking about with John is how can we incorporate different sources of signal like optics or other type of high throughput screening sort of pipelines that he has in place. Can we combine multiple sources of data and come up with better phase maps? Yeah, Mattel. So related to this question then, it may be a silly question. You have stated that they are able to produce countertowel. Do you have a combination per day? No. How does the rate of your machine learning process compare to this rate of production? Is it faster? It's pretty fast. It's maybe takes a minute or so to get a result. So it's getting there. It's way better. If you do it by hand, it takes them a week to process a sample. So it went from one week to a few minutes. It's still probably not good enough. We need to scale it up more, but at least it's better. Like people I'm talking at a slack, they are sitting on piles of hard drives with data and they just can't keep up on analyzing it. So they would really help them if we had ways to process the data automatically, at least at first cut, analyze it a little bit. But still, room to improve this procedure. Such that it's almost live. You take the data and analyze it. Ideally, yeah. Because ideally what you would want to do is you would want to, as you get the data live, you could try to come up with a face map, try to figure out where the boundaries are, then you can sort of change the sampling so that you sample more often at the boundary and you try to figure out what's really going on. We never got to do that, but ideally that's what we want. So what's that? No, just to try and ask a couple of questions. So what would be the rate at which room to really analyze this data, or what's the improvement that you need to add on or out of the loopers to get there? Is this something that you can actually achieve? I think we can achieve it by making some simplifications, just maybe like right now we're looking at an entire spectrum or not, but potentially you can do some kind of PCA, you can do some kind of random projection, or reduce the dimensionality of the data to scale it up, then it's sort of like a trade-off in seeing how much accuracy you lose, versus how much computational advantage you get. I think there is space to explore. The thing is we never got to actually have access to the experimental pipeline and get it to actually plug in our stuff. They would never let us do that, so we never even tried to scale it up, but at some point it would be nice if we could really do it. Can I ask a comment? I don't know if you know or if you would be interested in the topic of mathematical engineering. My name is Mara and I'm going to be an arrow to see the impact of mathematical history on the engineering materials, but it doesn't make sense to talk about it. I'm not familiar with that. I'm a computer scientist. I always rely on my collaborators on the sort of domain expertise, but I'd be happy to talk about that. How do you quantify the error rate? There are some classic measures when you have... It's a good question. There is not a unique, clearly defined measure of when you try to evaluate clustering algorithms. There are a number of ways to do that, based on entropy, based on accuracy, precision, recall, and it's not a single best one that all our peers have agreed on. We evaluated on a number of them and it seems to perform definitely better than MF on most of them, and you've seen visually it seems to give the right answer, but in the papers we've done some experiments, like pre-extensive experimental results on synthetic data, so you can see all the numbers there and it compares favorably to other techniques.