 Hello, everyone. Welcome to the talk. My name is Karan and I'm a data scientist in the AI Center of Excellence at Red Hat. So, let me start by giving you a brief overview of what we're going to talk about. So, first I'll briefly mention like some of the challenges that we face with data storage at scale. Then I'll mention one possible way in which machine learning can help mitigate some of these issues and maybe the title of the talk gives away what this way is. Then I'll show you a brief demo on how we can use the failure prediction stuff today because all the stuff that I'm going to talk about today is upstream and you can check out all of that. And finally, I'll talk about how the open source community that you guys can get involved in the project. So, what was the motivation for this project? Just by a show of hands, how many people here are data scientists? Anyone? No? Engineers? Software engineers? Alright, architects? Okay, sweet. So, how many people work on getting something useful out of data or provisioning data for others to use? Okay, some. So, essentially the thing is most projects these days involve around doing something useful out of data and making it efficiently available to others. So, and these projects usually involve a lot of data and because data is so central to these projects and because it can literally drive like company decisions, you want to make sure that this data is not corrupted and not lost. So, with that in mind, most of the solid solutions like SAF or maybe some variants of RAID, they use replicas or redundancies to ensure that you get that kind of fault tolerance. But when you're dealing with data that is at a large scale like hundreds of terabytes or petabytes and beyond, then doing this can get really inefficient or maybe even expensive. So, the idea was instead of that, is it possible to monitor the state of our hardware somehow and then identify by some sort of magic that, hey, now is the time when something is going to go wrong. And then based on that information, you proactively add another redundancy to basically solve the problem before it even occurs. So, when you would do that, you'll get a bunch of benefits like you'll have a more efficient resource utilization because you'll only use storage as and when needed. You'll have an increase in the fault tolerance because the probability of losing data is generally related to all of the machines failing and if you add one more machine in sequence, then basically you can improve it by up to an order of a magnitude. Another thing is you will get an increase in the performance because now you have less instances of failure recovery to deal with, so it'll be a more performance. And finally, it will be easier on your pockets and it's going to save you some money. So, the way that we choose to do the prediction of when a hard drive is going to fail, we use machine learning as our predictor tool. So, to build any kind of machine learning model, there is one key ingredient that you need and that is the data. So, what kind of data can we use to learn what does failure mean, what does OK healthy drive mean? So, for this project, we use the smart metrics for our data. And these are essentially a set of attributes that tell you things like read error rate or temperature and so on and so forth. So, it's sort of like the vitals of the hard drive and you check the pulse of the hard drive to say that whether it's OK or not. And the way that you can get these metrics is by using a tool called smartmon tools. So, you can just install it by EM installs smartmon tools or something. And basically how it works is you can just type Sparkadel and then the name of the device for which you want to get the smart metrics. So, when you type that, when you run it, it's going to spit out a list of all the metrics and that's going to tell you how the hard drive is doing. So, that is the data that we use because most of the modern hard drives expose this kind of data. So, it's readily available and more than that, when you're training machine learning model, you need like a lot of data. Like just using one or two instances is generally not enough when training at least a supervised machine learning model. So, the great thing about this was that someone out there had already done the hard work of collecting all of this data for their hard drives. So, Backblaze is I think a cloud backup company that basically collected smart metrics data for their hard drives over a span of like six, seven years or so. And they put it in a very nice format in a CSV format and it sort of looks like this. So, they have like the date for which they collected the metric, the serial number of the hard drive and then the list of all the smart metrics. So, that's great because they have it in a nice format and it's open source and it's readily available for use. And even more than that, what's really great about this is that it also comes with failure labels. So, they took their data and in addition to the smart metrics, they also said that hey, this is the day that the hard drive did not fail. So, zero means it did not fail and one means that it failed. So, if this hard drive with the serial number foo failed on January 3, then this will have a label one. So, this is really useful because now it unlocks like a wide variety of machine learning models that we can try in the supervised domain. So, yeah, so that's the data set that we go with. So, instead of using the raw data set, we tweak it a little bit and we pre-process it. So, the first thing that we do is, so if you notice here, there's only two kinds of labels like zero and one, like fail or no fail, right? So, even if I have the best model in the world that is like super accurate, super precise, that can only tell me that my hard drive is going to fail today. And that itself may not be super useful. You might make it work, but it won't make your life easier. Like imagine if a doctor tells you that, hey, you're going to die today. Maybe at lunch, maybe at dinner, who knows? So, that's not very useful information. So, instead of using like a zero one label, a better way is to use a more fine grained labels. So, you have normal that is like a hard drive has at least six weeks or more to live. Then there's one called warning, which says that the hard drive is going to fail within six weeks. So, it's not super urgent, but keep an eye on it. And then there's critical or bad. So, that means that this hard drive is going to fail very soon. Do something. And of course, if none of that work out, if the model fails to detect, it's not going to crash your system. It's going to just say there's insufficient data or unknown output. So, this is the format that is more useful. And also that's the way the SEF ecosystem is set up for the prediction module. So, that's why we decide to go with this kind of labeling. So, we take the original data that used to look like this. And then based on when the hard drive failed, like using this label and the date, we essentially change the labels that we want to predict on. So, we make this into warning, good or bad. So, that becomes our raw dataset. So, this is great. Like we have our raw dataset up and running. And now we want to build the model on it. But as most data scientists, I think will tell you that you cannot just take the raw data and simply shove it into a model and train the model to do something with it. So, the most of the hard work or the smart work lies in taking the data and making it into a form that is more meaningful to train on. That gives you meaningful outputs. And another thing is that you want to design a pipeline that can accommodate training this kind of data. So, in our use case, let me show you the kind of training setup that we have. So, there was a bunch of things that we have to consider while setting up the training pipeline and the pre-processing pipeline. And some of these for this particular project were the dataset size. So, the back place company has been collecting their data for like a long time. And like approximately one year of data corresponds to roughly 10 gigabytes of CSVs. So, if your resource starved or yeah, if your resource starved, then this is not a great dataset size to work with and it might give you problems. So, you might still be able to do it, but it will be super slow and inefficient. So, one way and probably the most obvious way to deal with this was to change the way that it is stored. So, as it is, the back place dataset comes as CSVs. And that is pretty to look at, but it's not the most efficient storage format. So, we converted to Parquet. That's for faster read writes. And if you're using like a deep learning model like PyTorch or TensorFlow, then instead of using Parquet or CSVs, you can just serialize the data and use like the .pt for PyTorch and similarly for TensorFlow as well. So, essentially make sure that you can do the reads, writes and efficiently. Another thing that we had to do was because the data is so big, the compute power may or may not be enough, especially when you're working in Python and you have libraries like Pandas. So, a library like Pandas is mostly single core based, so that would not give you very efficient, like that would not be very efficient and it would take some time, especially when you have such a large dataset. So, we used a library called Dask and that essentially takes your data and all the tasks that you run on it, the preprocessing, the cleaning and everything, it essentially divides it into workers and runs it over all the cores of the system. And if you have multiple machines, you can also schedule it to run over multiple machines. So, that was a very important design choice. And next, perhaps the more important one is the class imbalance. So, what I mean by this is in most systems, like failure or anomalies is probably not going to happen as frequently as things being okay. And while that's a good thing, like from the perspective of the world and society, et cetera, as a data scientist that kind of annoys you a little bit because this means that you will not have enough data to show your model what failure means. So, if I have a million data planes, maybe for this case you have maybe 100, 200 or something like that drives with failure and the rest of them are working okay. So, it's going to be highly imbalanced and biased. So, to deal with that, there's two main things that we can do. One is to down sample the abundant class. So, if I have 50 million working hard drives and only 10,000 failed hard drives, then what I'll do is out of the 50 million, I pick only a smaller subset like maybe 10 million or so that are well representative of all those 50 million data planes. So, that's one way to down sample the abundant class. And another way is to over sample the minority class. So, and this may and may not work for all models, but essentially you can use the same data point from the minority class again for each batch or something like that. So, these two things were what we did for the class imbalance. And finally, something that has to do with metrics. So, smart metrics as it turns out can be very manufacturer specific. So, for instance, you can have a Seagate hard drive that exposes smart 51, let's say, but a Toshiba hard drive doesn't have that at all. So, if your model uses only smart 51 or something, then it will work only for Seagate and not for all the other manufacturers out there. Or maybe it is the case that what one smart metric means for one manufacturer might mean something else for another manufacturer. So, let's say smart 51 being at a value 50 means like a Seagate drive is doing great, like it's the best drive in the world. But the same value for Toshiba hard drive, let's say, could mean that it's basically trying for help. So, there's a lot of variation in the interpretation and definitions of smart metrics. So, to deal with that, we thought because these metrics and because the data is so specific to the metrics, we should also make the models specific to the manufacturers. So, what I mean by that is basically we have one model to predict stuff for Seagate, one model for Toshiba, one for HGST and so on and so forth. And this is still kind of doable because there aren't as many manufacturers in the world. So, this is the design plan that we decided to go with. So, at this point, we have the pipeline sort of set up and we want to take the data and start feeding it into the pipeline. So, one of the most important parts of trading the model is to featureize the data that you have. So, first of all, you have to clean the data because like in the wild, the data is going to have a lot of holes. It's going to have inconsistent values and things like that. So, I'm sort of just hand waving it right now but like that's a big chunk of the process of training a model. So, you have to first clean the data set. Once you have a clean data, we selected only a small subset of the metrics to work with because there's a lot of smart metrics out there. And so, with that kind of high dimensions, it becomes very resource intensive to train the model. And even so, even more than that, it becomes more difficult to find models to train on because the concept of distance in high dimension is not very well defined for some metrics. So, having a lot of features can be a problem. So, that's why what we did was we read some previous research, we read on the internet about the domain. And we also looked at some of the analysis that Backplace had done. And based on all of this information, we took a small subset of the features that we want to train on. So, now once we have the features, we want it to go beyond just using the features as is. So, as it turned out in the CEPH, this prediction module, we have up to six days of like exactly six days of data available at inference time. So, instead of looking at just today's data to predict the health, why not look at the sequence? Why not look at the whole six days of data and look at the behavior of how it changed over time? So, we tried to characterize like this behavior over the six days over time. So, essentially for each time point, we have like a window and we look at like the six days before it. And we tried to characterize how it was. So, we look at the mean value over the last six days, the standard deviation of the metric and the coefficient of variation. That's sort of like just a ratio of the two. So, instead of looking at just a metric, just a number, you look at the behavior of that number over a period of time. So, now we have our clean data. We have down sampled the number of features. We have down sampled the data and we have featureized all our metrics. And at this point, we are sort of at the stage where we can just start feeding it to different kinds of models and try to see how it does. So, this part is what I think is the easiest part of machine learning to actually fit the model. Because here is, so this doesn't involve that much work from the data scientist side necessarily. Because this is just something like from a library, you just call model.fit or something like that. So, that's the easier part. So, for this particular task, we tried a bunch of models starting with like the defecto or like the go-to models for this. So, things like SVMs or decision trees and random forests. And then we also tried some fancier models that build beyond this. So, something like a XG boost, gradient boosted decision tree. So, it's like a tree but fancier, made in a fancier way. We also slightly explored deep learning techniques. So, something called long short-term memory. And yeah, but we didn't go too much deep into it and in a minute it will become clear why. So, with all our experiments, we found that what worked the best within what was possible was just a random forest. So, a random forest is essentially a collection of decision trees. So, a decision tree is essentially a set of questions the model asks to arrive at a conclusion. So, like is smart metric 10 less than 5? Yes, then go here. Is smart metric 50 more than 70? Yeah, go here. And then sort of trickles down into a decision. So, that's hence what is called a decision tree. And a collection of those is a random forest which we found to work the best with some limitations. So, the limitation that I have been talking about that what prevented us from trying other or going after more fancier Python libraries and things like that is that at the end of the day you want this model to be integrated upstream, right? You don't want it to just sit in some repository and sit there and look pretty. So, the thing was with upstream integration, not all Python packages work well for that. There's a limited set of packages that we can use to build models that can easily integrate with what already exists in this ecosystem. So, because of that we didn't experiment too much with things like deep learning and so on because it's a little bit more difficult to get it upstream. So, within that drill we found the random forest things to work the best. And yeah, so one more thing is the results at least from my personal opinion it's not something that is out of the world. It's definitely, it can definitely get better, but it outperforms the sample model that is already upstream. So, it's already an improvement over what we already have. So, that's one key win and another key win was that all of this, like all the data set, the model making and everything is completely open source as opposed to the one that is upstream which is sort of like a black box to us. So, with that said, there's always scope for improvement on the results. So, speaking of results, let's speak of results. So, usually when you have such an imbalanced data set and an imbalanced problem of failure when you have one class dominating the data set and a very small percentage forming the rest of it, you cannot use accuracy directly as your performance measure. So, we use, you use something other than that called recall and precision which I'll get to in a minute. And so, the recall is basically that of the data points or of the hard drives that were actually good or that were actually bad, how many did my model correctly identify as being good or being bad? So, in that respect, we were pretty close to profit store and in some case better than profit store. And yeah, so this is done on a subset of data that neither of the models has seen before. So, it's as fair as it can get with the data that is already available to us. So, that was our metrics on this. And another metric that we used to evaluate the model was precision. So, when my model says that, hey, this hard drive is going to fail, then how likely is it that that hard drive is actually going to fail? How precise is my model? So, in that respect, our models were almost always better than the profit store models, like you can see here. And in some cases, like the warning class, it significantly outperformed the profit store models. And the thing that is something to highlight here is that it worked especially very good with the warning and the fail, the bad class. So, essentially, it does very well where it matters the most. So, with that said, I want to show a small demo on like how you can get this set up on your self cluster, how you can use a predictor model and run it today. Is this sort of visible at the back? I think it won't, I don't think it will actually go full size, it will just add the black. But, oh well. Awesome. Contributions. All right, so here, first thing that I was just cloned the repo as is upstream. And because I was testing it out on my local machine, I did not, obviously my local machine is not a self cluster. So, self comes with a utility called VSTART and that essentially builds a deployment fake cluster for you to test out your developments on. So, the first few lines is just me setting up the fake cluster for testing, just make your build and then make restart. So, that's all you have to do, just two lines of code. And then it's going to go through making it, but I'm not going to go through it because it takes a very long time and this is a small talk. So, once it's ready, you want to start the cluster locally. So, before that, they also have a script called VSTART.sh and we just have to call it and that's going to set up a local cluster for you to test out. So, this also takes like a minute, but it's a super useful utility too. Thank God for the skip button. Okay, so now we have the local cluster set up for testing and the first thing that we want to do is to enable the disposition module. So, that's just SEV manager module enable and that will enable the disposition whether it's professed or whether you use these models. This is sort of a requirement for you to set up. So, one set is set up, you want to type, yeah. So, SEV has two kinds of this prediction available. One is locally, so it uses the local models that it has in the system to predict. And another one is cloud that essentially takes all the smart metrics and sends it to a server and then that server runs the prediction and gives you the result. The models that we made are only available locally and not on the cloud at the moment. So, the first thing that you want to do is ensure that it's running the disposition locally. Right, so that's there. So, once that's active. So, the next thing that you want to do is although we are confident in the models, we don't impose that on the user. So, by default, the models that are going to be used are the ones that are already upstream, the profit store models. But if you should you want to change the models that you use, you can just set a config variable in SEV that is predictor underscore model. So, we can set that model to Red Hat for the lack of a better name, which was Red Hat. And then once that is set, it's going to use the models that we built and then use predictions from those. So, that is pretty much all the setup that you need. So, once you have that set up, you're free to run all the predictions and have fun with it. And the way to do that is, so, SEV and then disc, is it disc predict or device life expectancy predict. And then the name of your device or the name of the hard drive that you want to run the prediction on or something like that. So, in this particular instance, it did not give a very useful prediction. It's unknown. So, you're already here, you're back where you started. But I tried both the models and both the models give the same prediction, which is unknown. And maybe it's not so much interesting running locally. But yeah, it's out there. You can do it and hope that it works. So, that is the end of the demo. And if you're doing it locally on the fake cluster, then you also have to do one additional step, which is to make sure that you shut down the cluster, otherwise it's going to eat resources. So, it also has a script called stop.sh, just run it. And yeah, I think that was all for the demo. All right. So, as you probably saw, it's a great model, but it's definitely not the end goal that we want to have. And in fact, at least as I see it, it's more of a starting point from where you want to go. And the goal of the project was to enable development on this and iterate again and again on the models. So, let's talk about how you can contribute to this. So, a core part of the project was a bunch of notebooks, Jupyter notebooks. And if you're not familiar with these, it's a format in which you can run, in which you have these code blocks, which have Python code, along with some fancy outputs, pictures, and things like that. So, we have notebooks that are designed for every step of the training pipeline. So, you have one for exploring the data, getting used to it, and getting a feel of what it's actually like. Then there's a notebook for cleaning up the data, which is a very, or at least in this case, it's a very hard thing to do. So, there's a notebook for that, and then you can take the data, clean it, and then there's a notebook for pre-processing that clean data, and featureizing, extracting the features, and then feeding it to the model. And then finally, there's also some notebooks that take all of this and put it into one thing. Even beyond that, we have something called a Kaggle Setup. So, my colleague Michael Clifford took the effort of taking all of that notebooks and essentially packaging it very nicely into this one Kaggle kernel, and essentially what that does is introduces the problem very gently, if you're a first user, and then tells you, describes the problem, describes what kind of models you could use, and basically handholds and takes you through the entire process. So, if you want to quickly go through this, there's like a nice description, the feature selection, and the missing data handling that we do in the repo put very nicely in this. Once you go through that, so like just making one function that does all of that previous steps for all other instances of the dataset or other manufacturers. And once you have that, there's like some simple heuristics that he wrote that helps you better understand the data and compare how machine learning models would have done against a simple prediction, like return one. So, that's all of this part. And of course, it has a lot of pretty graphs. And yeah, so finally, we compare the ML models with some simple heuristics and see how it does. So, that's all up here in this repository, and it helps you basically get set up with the problem and like the name suggests, getting started. And finally, if you're a deep learning enthusiast or like me, we also have made a set of tools, utilities, dataset classes that essentially help you get set up with this problem, but in a deep learning context. So now it opens like a whole new door of things that you can try on. So, what this includes is like a set of tools to take the raw dataset and make it into a format that is better to work with PyTorch. So again, this is going to be very PyTorch specific. Because of personal bias, but at the moment, this is what's available. And essentially it takes the data, makes it available in a format that's easier to use with PyTorch, and it has some dataset classes that you can use for very easily loading data and then training model. So, to give you a very brief overview. So, all you have to do is like from datasets import dataset and then, excuse me, give it the path to where the dataset is stored and it's going to do the splitting and everything for you. And yeah, and then in this line you define which dataset you want to use. So train dataset equals this dataset that I imported from here. And yeah, that's pretty much it that you need to load the data. And once you have this data loaded in like three lines of code, you can go ahead and define the model. And please don't define it like this. This is just a toy model. It will do very poorly if you run it, if at all. So yeah, you can define the model very simply. And then all you need to do is write a for loop, like for sequence in train order, prediction equals predict. So in like very few lines of code you can get started with this problem in a deep learning context and go have fun with it. With that said, I do want to mention one thing that because the dataset is so highly biased because there's like 99% data is of the successful or like working drives and only 1% is of the failing drives. So deep learning models usually don't do very well on that kind of data, at least not when you use a very simplistic loss function. So this may or may not give you great results if you use it as is with the default loss functions and everything. But with this basically you don't have to worry about getting the data and getting it set up. You can focus on things that matter more, I guess. So yeah, so whether you're a data scientist who wants to go deep into the problem, we have notebooks for that. Whether you're just getting started, we want an easy path to the problem. We have a Kaggle kernel for that, too. And if you're a deep learning person, we have that set up for you, too. So yeah, that's pretty much all I wanted to talk about today. And yeah, I'll be happy to take any questions. Yes. The general gist of it, for me at least, is that smart is a very bad predictor for failure. It's a very good signal to tell you, yes, I already failed. I see. And what I've noticed is a good predictor of failure is when the performance of the drive is going down. Like, a time for the drive requires multiple times of additional delay, like if you have a 700 RPM drive. If it takes like 40 and then 21, like 36 milliseconds to reach a given sector. And if that happens for multiple sectors, then you know that that drive really will fail in a soon-basic. I see. Do you have any plans to add this kind of monitoring and also data for self so that it can have better prediction? As you saw, there are around 30% failure for that prediction. Sorry to interrupt, but I'm really happy to succumb to that. Because everybody in self support has seen a lot of customer drives that before failure when they get really slow. Like you have slow ops just on normal SD. And this is a very good indicator that the drive will fail. But we had a lot of cases where there was not ever seen a smart on the control or something. It's still the drive that's already quite slow. Yeah, so did everyone hear the question? It's kind of too long to repeat. Okay, so to answer your question first, I think, but I may be wrong, smart metrics do have one of the metrics that measures the spin rate. Correct me if I'm wrong, but I think there's a metric that measures how fast the thing is spinning. Yeah. Normally, if you are reading just sequentially, you get blocks by blocks in under millisecond time. If the hard drive needs to read the block again, because the CRC correction failed or something like that, then you will see a multi-millisecond delay when he did that to that sector. So that's the predictor for the drive really has a problem because it doesn't give you the data. Oh, I see. Can you say the last part again? It doesn't give you the data? If you ask this to give you the data, you are asking a sector after a sector. You will get the sector spinning one millisecond, one millisecond, half a millisecond, one millisecond and so on. Yeah. Okay, I get it. So I don't think there is... So at least with this data set, I don't think they have that information and I don't know if how you could get that metric, essentially, because we were not doing the data collection itself. Part of the difference is you can't... Yes, I agree. But the thing was we didn't actually do the data collection part. We were using the data that people have already collected and open sourced. Yeah, that's my question. Do you plan to add that? Yes. So I personally don't know if SEF is planning on doing that. So I work in the AICOE. So I'm not necessarily on the SEF team. So I don't know if they have plans to do that, but that is a very good suggestion and I'll probably trickle up to the people. Yeah, thank you. I just follow up the challenge. One of the things I wanted to do was enable working with data like this. So that place here was the only publicly available data set where we could do something like this and SEF adds this prediction. It was already built in and you replace it with an open source data set. That's basically the first step and then we put the feedback from the community or guys that are actually running large entrusters and then play with that data set and then you come up with a better way. When it's the first punch or hold into this problem, somebody told me that basically serial numbers are the best way to predict what the driver will pay. So you see monitoring that batch and just moving forward with the others on that batch. But the office needed one of all serial numbers. Maybe that was the best way to do it. Yeah. So the question was are you planning on adding the number of LBAs written or the number of bytes written as a feature on the data set, right? Yeah. Yeah. So we tried half of what you said. So we tried to look at, I think you're mentioning power on hours, right? That's the lifetime of the, like how long the drive has been on for. So in one of the notebooks that I showed before, we looked at that metric for fail versus working drives and we basically found no difference in like when they fail. In fact, like if you run one of the notebooks, you will see that most metrics, how they work is they're usually like doing okay, okay, okay. And then suddenly they fail. So we tried looking at that and we looked at the power on hours too, but it did not provide any significant value. But that being said, we did not look at power on hours by day. So I'm trying to think if the model would, Yeah, exactly. So let's go on. If that's more, that would be more of the people to SSDs than to string it up, string it up. So that's an interesting point because we've got all these type of specific variations. It's thinking about this spindle, it's grasping at a point, you might see a flash drive storage which doesn't emit a single smart drive, also the smart lock is completely empty or doesn't receive any new entry. But the flash is accessible very soon because the garbage collection and the devices, the internet, so you don't have any evidence of who are egos or something like that. It used to come with their slow access. Yeah. So, yeah, sure. Just before you go, so about the LBA is written, right? So I think smart 241 or something, that is exactly that number of LBA is written. So that is already in the, yeah. Oh, yes. Yeah, it's on the ratio. Yeah, we did not do that. So I'm obviously not the domain expert, so I didn't know if that would be okay to do or what features should I be taking ratio of or playing around with. So did not add that and just took the metrics as is and featureized it. But yeah, maybe what you're doing could be a more powerful indicator. And the second question. So this was essentially written as a model to be used in the SEF ecosystem. But at the end of the day, it's just a model. So it's just a serialized pickle file. So if you can run Python on the system that you want to use this on, then you can essentially, yeah, so you can take the model, feed it the same data that we are feeding. And if the data is same, then yeah, the model will work. Yeah, any other questions? Of course. You know, I'm a father. You had a split second power outage and the firmware couldn't quite help with it and the device went offline and never came back after racing. We set that up to the vendor, they do their, you know, black magic and return. It's completely okay. That kind of thing because I wasn't poor. This is, of course, only a very small subset. I hope that, right? Because the fingers were quite hot. So are you planning to go all that also? Which goes back to the general framework I already had, like feeding in that vendor information. You won't probably get that kind of vendor information that will help ease the chance of outage. Can you say the last part of the last thing, though? Yeah, the last thing was, you know, vendor information. That really both, you know, like the partnership with the vendor allows us to have these kinds of, you know, storage of widgets. I just don't want to play with power and it went offline and never came back. Even though the physical media was intact. You see that kind of information needed to be fed into the system also along the lines of, you know, TPW and you know, so at the moment, because the models are manufacturer-specific, I'm not sure how you would feed, so that's already kind of meta-data. That's already kind of like an assumption that model has to do only with that manufacturer. So that should already be able to deal with what you're saying. Like if that happens specific to vendors, to manufacturers, then that model should be able to deal with that. Does that make sense? Sure. Yeah, I'll be happy to. That is true. The second model. Yeah, the second model. The second model. The model has been trained in this deploy. Can you add it to the model? So, I'm not sure. So there's two answers to your question. So one is does it get data, field data from the customers? And the answer to that, I'm again not sure of because I'm not sure what the SEF team is planning for their instances. So I'm not sure if it gets from the customers, but as long as, as far as the retraining goes, yes, the models do need to be retrained and there is no automated way of doing that right now. Like you're just required, like someone to fetch new data from back place every quarter and then retrain it. Exactly, yeah. The SEF team do that on a community basis and say what's the better things about collecting that from the customer side, but it's still in the really early stages punching the first holes into SEF that we can work on that data and collect that data and setting up systems that will collect it from a community standpoint. So far it's just people's head and we need to implement it so that anybody who is working in the SEF community can drive that forward. Basically, our team tries to bring AI tools to the IT industry and storage also just an example in the virtual side of it. All right. Any more questions? Going once, going twice. Okay.