 Okay welcome to the last session and we have the pleasure of having here David a wrap up from the Atlanta Fed and the discussion will be Michel van der Beek okay thank you very much can you hear me yes okay yes thank you very much enjoyed the the conference of course it's been it's been wonderful it's an honor to be here it's an a special honor when I realized I just learned that it's official ECB conference policy for the best paper to go last so yeah I'm I again I'm more I'm deeply honored so thank you for that okay so this is joint work with a number of people from our who's including let's see Daniel Bora Christian Monteshuta and sender Schwank Nebe and then Philip Gulley Coulomb from UCLomb the usual disclaimer of course so stating the obvious here you know these days large data sets and machine learning right there trending we're all doing it they're growing in importance in macroeconomics and finance when it comes to out of sample time series forecasting as we've seen in a number of applications of course at this very conference large data sets provide greater capacity right to incorporate relevant signals so if I want to forecast a particular variable the larger my data set although sequel you know the greater chance that I'll find relevant signals machine learning well if I have a large data set right I'm going to be keenly worried about overfitting if I use conventional techniques and of course machine learning provides numerous tools to guard against to guard against overfitting and this of course will help to improve out of sample performance with many predictors machine learning can also accommodate general nonlinearities so if we think nonlinearities nonlinearities are an important component of the data generating process then a number of machine learning tools are quite helpful again we've seen these at work here at this conference familiar examples random forest boosted trees neural networks some recent examples and macro forecasting right and again this is just a partial list might be a few people here but again applications with inflation forecasting output employment growth unemployment rate initial claims recession let's see I think fiber might be here maybe yeah and gray patches here financial forecasting as well it's becoming quite popular in finance especially in with respect to stock returns both at the aggregate level and at the individual stock level and in other aspects too I think I think Danielle Bianche right on returns yeah with machine learning so again this is this literature is constantly growing in addition to out of sample forecasting accuracy the interpretation of these fitted prediction models is important even crucial so just such basic questions as you know which predictors are the most relevant for determining the forecast generated by some fitted machine learning model and actually at the heart of this paper how do the predictors contribute contribute to out of sample forecasting accuracy interpretation helps users to wrap their minds if you like right around forecasting models so they're not so called black boxes right that's kind of a standard criticism of machine learning methods right you know what the heck's going on it's just a black box you know from an economic standpoint a theoretical standpoint gaining insight into empirically important economic mechanisms in our forecasting models by interpreting them can help to guide the assessment and development of theories and as a recent someone who recently joined the Federal Reserve System yeah you know understanding these forecasting models helps us to provide more comprehensible advice to policy makers right I don't think you want to say well you know Mr. President of central bank it's what my black box model says so no worries of course there are a variety of existing model agnostic tools for interpreting for interpreting fitted models I'm sure many of you familiar with these primarily along two dimensions so if you want to examine how the predictions or the forecast made by a model vary with an individual predictor or perhaps a couple predictors if you want to go into two dimensions this is there are a number of tools for this you may be familiar with so-called partial dependence plots Shapley values will lean heavily on Shapley values tonight or this afternoon whatever ice plots limes ales so again this is when we want to study how a model's prediction varies with a predictor variable in or feature importance also receives a lot of attention right so just measuring you know how important is a given predictor in your fitted model how important is it for generating forecast in your model and so we have PDP based measures permutation that can be used to measure feature variable or feature importance as well as Shapley based measures now existing tools are more appropriate for cross-sectional data for reasons that I'll describe in more detail as we proceed what we do in this paper is we focus on developing Shapley based metrics for time series data and so we'll develop basically three metrics actually we have a fourth that we're going to add that I might be able to talk about time permitting but if it's the last session I could go on for yeah yeah that's okay yeah let's do it okay so we're gonna develop what we call I Shapley VI and O Shapley VI so this is the this is an in-sample Shapley variable importance measure designed for time series data you're estimating a sequence of forecasting models to forecast time series data we'll develop an out of sample Shapley variable importance measure O Shapley VI and then our main methodological contribution is what we call the performance based Shapley value and it's going to measure the contributions of individual predictors to the out of sample forecasting accuracy of your actually sequence of fitted models that you use to generate a sequence of time series forecasts of say a macro variable or say an asset return what have you and it's going to identify the predictors most responsible for a models out of sample performance and in this way it's going to allow us to basically decompose the model's loss function so we can look overall what's the what's your RMSC for your model we'll be able to decompose that RMSC into the contributions of each of the predictors and in this sense we anatomize out of sample forecasting accuracy hence the catchy title if I do say so myself PBS V applies to any fitted prediction model it can be linear nonlinear it can be parametric non parametric just need like a predict function it applies to any loss function we're going to have an empirical application we'll forecast us inflation we'll come to that in a moment I'm gonna have to move it at light speed here okay so I'm not to skip over a lot of details if you have questions about the details you know please reach out to me afterward of my email whatever we can have zoom calls but let me just try to set up some notation in our time series context so we're going to index individual predictors that you're using in your model by P and so cap S will be the index set of predictors going from 1 to cap P we'll collect the periodicity vector of predictors okay and the the vector xt here's the prediction model our target will be some you know cumulative growth rate or cumulative return and then we're going to have f of xt this is our the conditional mean or the prediction function and then we're going to have an additive error term right and so this is our this is our target right we want to so we want to fit this we want to estimate f t to make forecasts when we yeah so we wanted to find the the training sample or the the estimation window that we use when we fit a training model so for a given model I w I will just be the set of time series observations used to fit a particular model your training sample or your estimation window and then your fitted prediction model in this time series context will be f hat you're going to evaluate it at instance xt so you plug in xt in your model and then we have w I reminding us that you fitted the model using the training sample the window w I and then H is the forecast horizon we interpret as I mentioned before fitted prediction models with shatley values so shatley values essentially exploit the analogy between predictors and players in a cooperative game earning payoffs and in the context of making forecasts the payoff is a predictors contribution to a model's prediction so you can imagine the predictors kind of they're playing a cooperative game and the payoff to each predictor will be again its contribution to the model's prediction and so we can think of shatley values as fairly allocating the predictors contributions to the prediction shatley values have some nice properties I don't have time to go into details on those but in general I think it's safe to say if you consult a number of textbooks on interpretable machine learning shatley values seem to be the preferred metric so this there's a couple of nice papers by strumbling Kononenko and they use shatley values to interpret prediction models so shatley values were shatley invented shatley values way back in the day 53 I think and so then for you know game theory and then again strumbling Kononenko apply shatley values to interpret prediction models and so we're basically going to modify their approach for time series prediction and so the aim of a shatley value in a time series context is to we want to quantify the marginal contribution of this of predictor P right to this prediction or forecast f hat xt window wi horizon h conditional on the presence of all the other predictors so all the predictors are in cap s and this is set minus p okay so we just want to measure the contribution of predictor P to the prediction that's actually a tricky issue right it's it's trivial almost in a linear model right okay because there are no interaction terms but if you have interaction terms nonlinearities what have you then to try to decide well exactly what was the marginal contribution of predictor xp right that's quite tricky and so shatley values the logic of shatley values is going to allocate the contributions of each predictor to the prediction fairly if you buy the logic of again shatley values this is the formal definition of the shatley value for predictor P for instance xt and again I'm just emphasizing here this is for a model fitted with window wi in horizon h so again there's so much notation to go through here and we could spend again an hour just making sure we have a good handle on on shatley values but basically what's going on is you're going to form you're just going to select a coalition denoted by cap q of predictors suppose I predict I select predictors one and two and I want to measure the contribution of predictor three to the final prediction well what I would do is I'd say okay what is I condition on predictors one two and three that's this value function here and then I subtract the prediction made by my fitted model just conditioning on one and two right the two predictors in this coalition and I get that difference well do this for all possible coalitions that exclude P and then take a weighted average of that okay that's how you measure the shatley value okay so it's using these coalitions right to control for the effects of all the other predictors and you're looking at this difference in the value function yes in the interest of time right again a nice property of shatley values is so called efficiency aka local accuracy so if you have a given instance a given xt you plug it into your prediction model it gives you a prediction of say 3% you can basically take the shatley values for each of your predictors add them together you get 3% okay so they perfectly add up okay that's a suggestion right yeah good okay thanks I respectfully accept your suggestion of 10 minutes okay there are other nice properties missing this symmetry linearity again we won't go into the details now it's the formula that we saw before where you form all possible coalitions that exclude predictor P to compute the shatley value that's infeasible to do in practice so we need to use some sort of algorithm and again scrumbling kind of NCO development a nice algorithm for estimating shatley values we use a refined version of it again can't don't have time to go into the details but we can rewrite the definition of the shatley value in this manner and then we can basically come up with an algorithm where we're going to make a random draw of an ordered permutation of the predictors and if we make a reasonable number of draws we can approximate quite well quite accurately the shatley value corresponding to a given predictor again this is the all the details again I'm going to move quickly any interest of time for for the algorithm that we use to estimate the shatley value the efficiency property holds so that goes through which is a nice a nice result to have so if you use our algorithm again you fit a model you take a particular vector of X's you plug them into the model you get a forecast of 4% you use our algorithm to compute the shatley values they'll sum to 4% now what we've described so far is a local measure right it's saying for instance XT and predictor P what's the contribution of that predictor to the prediction right so it's a local measure well we'd like a global measure of P's importance and here we can compute a variable importance measure right so what do we do well we just look at all instances that are in our training sample for each instance right we can compute the shatley value corresponding to P we take each of those shatley values take the absolute value and then average them and that gives us a variable importance measure based on shatley on the shatley value this is a popular metric for measuring variable importance now this is straightforward to compute if there's only a single training sample right what do I do here's my total data sample here's my training sample maybe hold this out as a test sample and so just for every instance in my training sample for predictor P I compute the shatley value take the absolute value average them and voila I have my variable importance measure for my training sample that's typically what's done that's what you'll see in a textbook often okay but what do we do in time series right in time series you know we typically estimate a sequence of models right we don't estimate a model one time we estimate a model regularly we regularly retrain refit the model as additional data become available right so we just want to expand this so now imagine we're gonna have a set of training samples in the set of training samples could be based on a rolling window or an expanding window and just collect the the set of training samples in this cab w okay and we're interested in the importance of a predictor for the entire sequence of fitted models used to generate the forecast right and so what do we do well we just talked about how you can take one training sample and compute the variable importance for predictor P well just do this for each training sample that you have that you use to generate your sequence of out of sample forecast and then take the average and that gives you that gives you our eye our eye shapely vi measure we could also talk about measuring the shaft shapely value for your out of sample forecasts right so we take the X's that you plug into your fitted model generate in the model was fitted using a particular window we compute the shapely value for a given predictor for that instance they do this now for each out of sample forecast okay that's painful each out of sample forecast take the absolute value and then you can compute what we call the out of sample shapely value variable importance the oh shapely vi so then you're staying you for my whole sequence of models how important was a given predictor in generating the forecasts and now what do we really care about and this is how we modify the algorithm this is the main contribution to the paper again we're interested in the importance of a actually let me go one more we're interested in the contribution of a predictor to forecasting accuracy and so what do we do well it's it's deceptively simple just when computing the shapely value instead of focusing in the value function on the prediction made by the model per se f of f hat of xt we're just going to wrap a loss function around that we're going to take into account the realized value of the target as well as our prediction but otherwise we just use the basic logic of shapely of a shapely value and we can compute for each out of sample observation for a given predictor p we can compute a shapely value associated with a loss function okay we can do this again jointly over the entire out of sample period and that allows us to use the logic of shapely values to exactly decompose say the RMSE or your model so if your RMSE is again 5% we'll be able to take the set of predictors you have could be 5 10 100 and measure the prediction made by each of your predictor to that out of sample performance okay so in the interest of time I'm going to jump again there's a an algorithm that does this one of my co-authors Sander Schoint Nebe who I think is from Germany did a magnificent job of creating a Python package called anatomy where you can implement all of this so we do we undertake an out of sample forecasting exercise for inflation the usual stuff Fred MD use some Michigan University of Michigan survey of consumers here's the initial in sample period here's our out of sample period we use a rolling estimation window so we have well over I think well over 120 predictors we're going to use a number of different models principal component regression linear E-net model we have some nonlinear machine learning models and some ensemble forecasts so here's the benchmark AR the RMSE these are the RMSE ratios below one means we're outperforming the benchmark so you can see we're doing quite well especially at longer horizons for the nonlinear models don't okay and so what we do is for each of these different models I'll do one example here let's take the forecast based on the E-net right so we generated a sequence of inflation forecasts using a rolling window and a large number of predictors using again a linear E-net model we computed the I Shapley VI O Shapley VI and our PBS V our performance based Shapley value so the the the red line is the I Shapley the black line is the O Shapley they match up quite well the green line is the PBS V and so what we find is in many cases perhaps not surprisingly the I Shapley VI matches up with the PBS V so I predictor that's deemed important in generating forecasts according to the more standard in sample Shapley value measure right matches up well with a variable that's important in generating an accurate forecast however even though our models perform well overall we find some important discrepancies for example you can see this is negative one here this means this is the predictor that contributed the most to detracting from out of sample forecasting accuracy so according to the in sample Shapley value it's one of the 25 top predictors but it actually contributed the most to again detracting from forecasting accuracy and it is the this happens to be industrial production for materials so you can see when these green lines are what facing south and they have numbers like minus one minus two minus right that means these are predictors that are contributing the most to negative performance okay but yet they're deemed important by the Shapley the in sample Shapley value so the warning the potential warning message is just because something looks like important through the lens of the in sample Shapley value right when you actually examine whether it how it contributes to out of sample forecasting accuracy it could contribute in a in a helpful in a positive way contributing more accuracy but there's no guarantee and even if your model does well overall you can find some discrepancies I consider that well done on my part okay thank you for your generosity okay great thanks very much so thanks to the organizers for asking me to discuss this very interesting paper well try to be quick given the time that we have yet that we go so just a one-page paper summary well I think David it is well mentioned it as well there's a lot of interest in machine learning and this conference has also mentioned that but a caveat of course is that well yeah it's subject to a black box criticism and why I think it's particularly important that we talk about it at this venue is that it of course limits the use of such kind of methods for policy analysis say you want to get a forecast but you also want to motivate a forecast what is driving a certain forecast and that's exactly what this this paper is about in short what they do is they develop Shapley based metrics for interpreting the models and I have three of them two are well to have the importance of individual predictors for predicted target values and that's in and out of sample and a new one PBSV metric where you look the contribution of individual predicted for the loss in sequence of a given model I'll talk a little bit more about it on the next slide because this is really the core of the paper but what's also in the paper is an empirical study of forecasting the inflation for the US and what they find is you you well the indicators or the predictors that you get they make some sense it's oil it's various components of CPI but also and I think that's also nice to see from the result is that these measures they do give some different kind of insights so you do need a variety of these metrics it's not one that that generates all the results that we have now because it's so crucial for the paper let me spend one slide on Shapley values but I think it was a very well explained essentially what you're dealing with is a certain model that we have over here a linear model a time series model a predictive model and we're predicting the future X is us to mention P so they use the Fred MD data set so that's well something like a hundred variables in there so that's large and we collect these indices of all these elements in X in this set S so that's the one over here and what we do for the Shapley value we look at it one predictor at the time so we get the Shapley value five for the P predictor P lower case the idea is very simple what we're doing is we're going to compare the prediction of all possible sets of indices where P is included so that's this first form and we compare that to the prediction you would make if you have such a set where P is not included and you do that for all possible sets that you can compile and you have some appropriate waiting for that in place so that's the whole idea of a Shapley value here it's already implemented with a prediction in mind I'm a bit sloppy in rotation so that's why I well I could fit it in one line instead of one slide but but that's the whole idea of that and you can change of course well what you want to evaluate for the Shapley value so it's a very general kind of thing now that's the idea that days back from 53 and what the paper does very cleverly and I think that's something that's really to be applauded for is adjusted for a relevant setting for well more time series econometrics where what you have is more than days a large number of predictors you can imagine you have a lot of those sets if you already have P of these variables so to have a clever sampling algorithm for that we have samples that keep an extending you make a forecast one time out you get a new observation you make a new forecast and not only that in that you have re estimation every time so this is all taken care of in the paper and you get this e o Shapley and you can also consider rather than the prediction that you get you can also look at well maybe the loss function that you have in mind and that's this pbsv measure that they do now for a few comments for the paper I've summarized them in three categories the first one deals with the empirical findings I hope this figure is well readable I think this is the same figure that you that you also showed in the presentation so it essentially ranks all these explanatory variables from most important to least important the first thing I noticed when I looked at this this figure in the paper it's for 136 and I think 12 months out is that there seems to be some instability in what what seems to be important for at least one of the dimensions so if we look at the one month ahead forecast you can see her PC EPI that's the least important if you look at the six month out forecast it's suddenly the second most important so it seems to be quite a big kind of kind of job I think it would be interesting to to touch upon that as I was thinking I've played myself with this threat MD data set in my own research and there's a lot of very similar series in there if you look at the yields there's six different yields in there and we know they're highly correlated for inflation there's a lot of similar kind of measures in there so I was thinking what do I expect to get from this Shepley value do I expect to get some grouping that all of these variables are equally important though I didn't click a button so let me go back that they're equally important or do we actually expect the contrary if we have six yields let me take the yield this example I take out one yield I expect that to not make that big of a difference so you would have it less important than Shepley but then again is that fair I mean if there's a lot of variables in this threat MD data set that's likely a sign that this is an important kind of variable so this is a consideration that I think is worth spending some time on what can we actually expect in such kind of data set not in the empirical application some choices are made relatively early on in the paper I'm interested about the robustness so rather than predict well steps ahead like age periods ahead it's the average of the age periods ahead predictors are included and also well not lags of predictors but a moving average a three month moving average of the predictor some curious why don't you just include the previous two periods as well I know it will be bigger but then you can let the model decide what you need for example and there's some selections in there so that's just I would say maybe somewhat minor kind of point another point is the benchmark the benchmark uses the ARK kind of model we've seen in this figure that I've just showed it's always very important so that is really an important feature that you're picking up but I personally would also be interested in looking at smaller models we've already heard to use the SV model a few times you can think of a limited number of macro variables also taking into account a little bit that the improvement that you get for the short horizon at least well I would label it somewhat modest it's seven percent it grows very rapidly it goes to 20 percent if you look at 12 periods ahead but I well I supervise a lot of master visas on these machine learning methods and then seven percent is rare usually you see something in the vicinity of 30 40% the kind of outperformance so press that as something to do with a benchmark maybe the benchmark is already very good so I would like to see some robustness there as well as some alternative choices and then another point that I will get back to perhaps at a final kind of slide is that you also include a more traditional kind of econometric model a PCA or you call it a PCR principle component regression some curious for these regression coefficients if you would look at significance would we make the same kind of conclusions as we would make based on the Sheppley kind of kind of values final I also have a paper where you Sheppley values I do that for finding out the driving factors of sovereign credit rating and I thought maybe it's helpful to share the kind of feedback I've been getting from the referees there so it's free writing maybe it's even lazy but I do think it it can can offer some value and some of the comments I've received is what would happen if you omit some variables so you found something is not important what if we take it out what would happen in that particular case okay maybe extend the paper with more models I think that's a risk if you work in a machine learning literature it's a very fast-moving kind of literature by the time the ink is dry there's a new method and you can go back to the drawing boat already ensembles were mentioned in our case you have them and I do think you have a very nice selection of methods already panel nature we were asked to take the panel nature into account we didn't we just do a fairly basic kind of analysis you do so I think you really make a big step in the literature for for doing this for really taking that into account in these kind of metrics there we go some of these these these Python packages that they already spit out some sort of variable importance kind of metric so I think this relates a little bit little bit to the audience you have in mind that's one of my final comments but yeah you will probably be challenged to compare to those kind of off the shelf kind of techniques as well and a final thing in that's something that I alluded to as well in some of my previous comments is to to have a closer comparison of your findings with the existing literature so how does it line up with what other people have found in this kind of setting so the final point is then indeed what kind of audience I think the paper right now is very much in between of a machine learning and an econometric kind of paper and that's fine I think that's where a lot of papers are nowadays I was tempted because you do include at least the principal component regression and you have a very nice paragraph where you look at the linear model to see if you could take that a little bit further and to see what's well what would be econometric implications or what can we learn econometrically about what you have hard to say in machine learning but perhaps in the linear case you could do it what about a simulation study if we simulate data from certain data generating process we make variables important not so important maybe we can figure out how strong is the Shapley the information that we extract from the Shapley values how strong is that information actually and perhaps you could even look at some sort of statistics for the linear case and see if there's some connection between significance and between these the Shapley values and that's it to conclude I think it's a very nice paper it opens the black box very cleverly done adapting it to the time series setting and I recommend you to read the paper questions from the floor hi I'm telling you from the Austrian Central Bank I'm great talk thanks I've actually two questions so the first one is whether you think or it might be that it's a bit distracting to just take the mean because what I was thinking is if you take the mean across all the Shapley values across the holdout and across maybe the rolling window when I got that right that it might be that for example in us to production it is a good predictor I don't know in the 2000s but then we had the corona virus and then the observations get crazy and now it's not a good predictor but then it's kind of we would like to know whether it was one before the pandemic maybe and the second question is whether we could even use your tool as a kind of model selection or variable selection tool because you showed that when the values are negative I hope I got that right that they are disturbing kind of so would it be better than just to kick them out yeah thank you hi so we were talking before so I go in line with some of the questions and jump at it is from DCB so one thing that that when I try to think about it is like the in sample is like the residuals know when you construct your model your tree you you you fit the tree based on some errors and some you know and some lost function in this case and so that will give you the in sample Shapley values while in the your idea also in the out of sample Shapley values you on top put the error that is the what we think in linear models about the out of sample errors which you know we always try to test our models with the out of sample errors better than the in sample error of course because this is the fit okay so my question would be is there maybe it's in the selection in line with the ghost can we learn from the out of sample errors from the out of sample Shapley values what not to use in the in sample or in the construction of the tree because in the tree we don't use that we just use the in sample errors thanks thanks for the very nice presentation I really enjoyed it I have just a couple of questions one is related to the discuss some point would it be possible to exclude groups of predictors that perhaps are correlated because clearly in the Fred M.D. that gives you you know some of his fiscal some his interest rates some his prices so maybe you can get over the comment on correlation and then related to the previous question how expensive it is if you have to track over time the Shapley values maybe you have in the paper I'm sorry if is a so big question but can you eventually look at teach 90 how the ranking changes if changes at all thanks thanks thank you also for the interesting talk I just like to know whether the the Shapley value isn't is in the end and distance measure can you also associate with this distance measure some statistical distribution so we can basically formally test whether they are important or not I think that will really complete the whole thing thank you thank you so Ignacio from the from the ECB and I wanted to clock your thoughts on on using these these method for for variable selection so more or less on on this line because if if you use if well if you include one one of the variables that according to your motto is not actually not contributes to reduce the error so increases the error if you include that variable on your on your random force given the random force in the end is going you're going to estimate it in order to reduce the error or to minimize the error that variable is not going to enter in in your tree as a as a potential one for in this pleats so what's what's what's your thought on these do you think that that makes a difference that using your approach to to other approaches or not at all not reducing the variables should I go ahead okay okay well thank you so much especially to discuss to discuss them but these are excellent questions so I'm just going to kind of go down my list and cover your cover your point some of your points and address the questions as well so one thing I should mention we this might be helpful you know we have those those plots but we're actually developing what we're calling let me see if I can remember the name a model of cordon score and MAS so we're going to provide you with just a single measure between zero and one of how well the in sample Shapley valuable value valuable in sample Shapley variable importance corresponds to the PBS V so we think that'll be quite helpful okay so yeah you know instability of the results especially over the different horizons yeah you know there's some commonality but there are certainly or some differences I think we can tell some reasonable stories for some variables just that made if you think about the nature of inflation and you know sticky prices but yes it seems somewhat idiosyncratic to me so I don't know it must apps this is somehow indicative of noise in the data that we inevitably encounter when we have to generate forecasts but that's something we're going to need to think about some more exactly how to interpret that but you make an excellent point so I think with Danielle and you mentioned about grouping you can absolutely do this actually we have a we're doing a sort of a finance related spin-off on this and we have you know we're into the literature on firm characteristics and we have just a tremendous number of firm characteristics and it becomes very costly to compute these Shapley values because we're just exactly what we're doing and we actually do group variables there so you could certainly do everything we have here and you just group them and so everything would still go through if you take your groups and they're exhaustive and non-overlapping if you add the PBS v score you would get the prediction and actually you would get the out of sample loss so absolutely I think grouping is a is an excellent idea and maybe will aid in interpretation and maybe help to avoid some of those weird results that we saw let me let's see let me move to yeah so a number of people I think in various and sundry ways have said well can mean what can you do with this information right so this of course is exposed right so I'm saying you know you've generated this sequence of forecasts and then after the fact I can we can use the metric and say oh these predictors you know they really help they really improve the accuracy of your forecast but these other predictors are not so much right they detracted from accuracy so we are thinking about trying to come up with a way I mean just I guess I'll tell just don't tell anybody else I'll just tell the people in the room the plan so you can imagine having like a holdout out of sample period right so you could for that holdout out of sample period you could compute the PBS v's and then say okay the ones that perform poorly maybe you drop them so you don't include them when you build your next random forest and you can kind of do this over time now it's a little tricky right maybe I want to reintroduce them after a while so I think one would have to do a good bit of experimentation but I think there is some scope for using this not just to say after the fact you know what caused things to go south for you but to try to actually improve out of sample forecast I think a number of people again I maybe Karen and Daniel asked about sub-sample analysis absolutely actually there's a if I do say so myself a nice figure in the paper where we have non-overlapping 12 month periods and we say for these periods which dictar was the most important which predictor contributed the in a positive sense to add a sample forecasting accuracy during that 12 month period and which can which most attracted you can and you can literally go down to one observation you could do it month by month day by day if you want and I think that can be we find some interesting patterns and we actually talk about them in the paper so statistical test yeah thought about that from the beginning yeah not sure what one would do I mean we can scratch our heads about trying to make some core set of assumptions just to get started I'm certainly open to hearing ideas one could you can imagine doing some sort of in a maybe in a crude way some sort of bootstrap or simulations but the problem is it gets quite computationally costly to calculate these things so I think of I mean maybe if I had you know one heck of a cluster I could pull it off but it would be it would be challenging I don't know if I'm missing anything I appreciate PCR like you know looking at the significance of predictors in the PCR and then comparing them to the what's happening in terms of the the PBS fees on an out of sample basis this could be this could be interesting as well I think I've covered most of the questions if if you have others you know feel free to approach me so I'm not leaving till tomorrow so and my social calendar is wide open tonight so thanks again okay thank you