 Looks good, Frank. Great. Yeah, so I'd like to give an introduction to protection predictive variable selection. So first, an introduction to the introduction. Protection predictive variable selection here abbreviated by PPVS is a special variable selection technique for Bayesian regression models. It's implemented in the R package approach grid. And apart from the variable selection itself, it also allows for at least approximately valed post-selection inference. I've added approximately here because it is not validated for the selection of a sub-model size, but typically the overfitting induced by the selection of the size is small. A key component of the PPVS is that it requires a so-called reference model, which is the best possible model in terms of predictive performance that one can construct. Often this is the full model with some shrinkage prior for the regression coefficients. I've added some example papers down here which illustrate this. But sometimes, and again, example papers down here, it can also be necessary to perform a dimension reduction for the predictors. For example, by computing principal components from the original predictors, then using these principal components as predictors in the reference model, but then still performing the variable selection in terms of the original predictors. Then the aim of the PPVS is to select the smallest sub-model achieving the best predictive performance with respect to the reference model's posterior predictive distribution. In other words, we aim for a trade-off between sparsity and predictive accuracy. And as you can guess from the name, the projection is a key step of the PPVS. And by projection, we mean the projection of reference model's posterior distribution onto the parameter space of a given sub-model. Typically, this is done after clustering or thinning the reference model's posterior draws simply for speeding up the computations. And then the projection itself consists of the minimization of the Kalberg-Liebler divergence from the reference model's predictive distribution to the sub-model's predictive distribution. I'm simply saying the predictive distribution here, the details are a little more complicated, but that's not the point here. Frank, really quick. I'm sorry to interrupt. Are you advancing your slides on your machine? We don't see advancing slides. Oh, sorry. OK, then it didn't work too. There we go. OK. OK, sorry. Oh, no problem. Thank you. OK, so the prediction problem, in general, is not easy to solve because it involves an expectation over the reference model's predictive distribution. But often it can be shown that the projection is equivalent to fitting to the fit of the reference model. And this is also the key idea behind it. Yeah, so now we want to head over to an example in R. And for this, we'll use the body fat dataset from the th.data package. And here I'm centering and scaling the predictor variables. At least the scaling is necessary for the R2-D2 prior, which we'll use afterwards for fitting the reference model. And here you can see the resulting dataset. It has 71 observations with 10 variables. DEXFAT is the response. All others are predictor variables. And all of them are continuous. So we continue with fitting the reference model here using the BRMS package. And for the model formula, we use LOCKDEXFAT as response. And all other variables as predictors, including their two-way interactions. We use the Gaussian family and the dataset we just saw. For the prior, we're using the default priors for all parameters except for the regression coefficients for which we are using the R2-D2 prior with some customized parameters. I've hidden some details here that provided in the full code that I uploaded to the website. So this was with BRMS. For Projpred, the reference model usually comes from either BRMS or R-STAN-ARM. But in general, the reference model is not restricted to be a model that is based on the STAN software. After fitting the reference model, we need to go through the typical steps after fitting a Bayesian model. First of all, we need to check the MCMC diagnostics to ensure that the Markov chains have converged with the desired efficiency. And we usually want to inspect some basic posterior quantities. And this is important, especially for Projpred. We usually want to conduct some predictive checks. I've skipped these steps here because they're not the focus of the presentation. But they're still important. So now we can head over to Projpred, which has two main functions, VICEL and CVVICEL. And I'd like to describe shortly what they do. First, VICEL consists first of a so-called heuristic search, which is either a forward search or an L1 search. And thereby, we're fitting to the fit of the reference model, so we're performing some kind of prediction. An L1 penalized projection in the L1 search case, and otherwise, the usual projection in the forward search case. And the output of this is then the so-called solution path, which is simply the best submodel for each size, or in other words, a ranking of the predictor terms. The second step is then performance evaluation, which means that we evaluate the predictive performance of the submodels along the solution path, for example, using the mean log predictive density MLPD as a performance statistic. The internal procedure of CVVICEL is basically the same as for VICEL, but with a cross validation around the whole procedure, either K-fold CV or Pareto-smoothed important sampling leave one out CV. So this piece is LUCV is an approximate but quite fast LUCV. The extent of the cross validation within CVVICEL is mainly determined by this argument Validate Search, which when set to true means that the search is included in the cross validation. And when set to false, we simply run a full data search, so not fold-wise searches. And as you can guess, Validate Search true is recommended over Validate Search false because it protects more rigorously against overfitting. And both CVVICEL variants are recommended over VICEL. Validate Search false is currently only possible with pieces LUCV, but yeah. And I also want to mention that VICEL also offers the evaluation on an independent holdout dataset, but this is something that I won't illustrate afterwards. So now we continue with Projpret with the code. Here I'm using version 250. And the main step is to call CVVICEL. Here with a k-fold CV, which as I said implies Validate Search true. And I'm setting n terms max to 10 here. n terms max controls the submodal size up to which the search is continued. Here this specific value of 10 comes from a preliminary run of CVVICEL. The main vignette of the package shows how such a preliminary run can look like. And after running CVVICEL, we can then plot the results. So here we're using MLPD as performance statistic. And I've also set deltas to true, which means that we're not looking at the MLPD, but the submodal MLPD minus the reference model MLPD. So the dashed red horizontal line, which corresponds to the reference model, is at 0. And then for the MLPD and also this MLPD difference, the higher the values, the better. So with help of this plot, we then want to pick a submodal size for the final submodal. And here a natural choice would be 6, at least if the focus is predictive accuracy and not sparsity. But we can see this more clearly here in the next plot, which is simply a zoomed plot where we see that submodal size 7 leads only to a minor improvement in terms of predictive performance compared to size 6. So we continue with size 6. This is our manual choice. And Projpet also offers a function called suggest size, which is able to automatically give a recommendation. But this is a rather heuristic decision. Here we continue with size 6. Yeah, then for constructing the final submodal, we need to retrieve the corresponding predictor terms. So first of all, we retrieve the full solution path from the CV var cell output. This is simply a character vector of ranked predictor terms. So the most predictive or the most relevant one is first. And then we cut off this character vector at the selected size of 6. So we pick n through 4 to knee breadth as the predictor terms for our final submodal. Then for post-selection inference with this selected submodal, we need to project the reference model onto it once again. This is done by applying the project function and passing the selected predictor terms to the solution terms argument. And then we can apply the yes.matrix function, which gives us then a matrix of projected posterior draws, which can be used like any matrix of MCMC draws, but with some cavities. First of all, in case of a clustered projection, we need to take the cluster weights into account. And we need to be cautious with regards to the interpretation, because the projected regression coefficients do not reflect isolated effects in general. Yeah, and with this matrix, we can then continue as with other matrices of MCMC draws. For example, we can apply the posterior package to compute basic quantities for the marginals of the projected posterior. For example, median, MAD, and some quantiles. We can also visualize the marginals of the projected posterior here using the base plot package. And then we can also make predictions based on the projected posterior. There are two options for this in ProjPret. The first one is ProjLinPret, which is similar to posterior LinPret methods from ISDNR and BRMS and so on. And the idea here is to calculate linear predictors, but also to possibly transform them to the response scale by applying the inverse link function. ProjLinPret also has an argument called integrated, which when set to true will average the possibly transformed linear predictors across the projected draws. We will see this in a minute. And the other option is ProjPredict, which is similar to posterior predict methods, which gives us draws from the posterior projection predictive distributions. And it encompasses the uncertainty not only from parameter estimation, but also from the observation model, which is the Gaussian observation model with its residual standard deviation in our case. And yeah, this is useful for what would have to be termed a posterior projection predictive check, PPPC. And we will see this also in a minute. So first, ProjLinPret supposing we have the following three new observations for the predictor variables of the selected submodel. We can then compute the expected values of the corresponding posterior projection predictive distributions. Yeah, so the key step here is to call ProjLinPret. Here we're using integrated true, which gives us a single value per new observation. And yeah, appended then simply to the new data set here. As I said, ProjPredict is helpful for PPPC. Here I'm using the base plot package for visualization. Yeah, so we call ProjPredict with no actual new data. So we calculate in-sample predictions. And yeah, these in-sample predictions are these light blue kernel density estimates. And the observed response values are the dark blue kernel density estimate. And from the fact that they mostly agree, we can conclude that at least in this regard, the final submodel is a reasonable one. Yeah, now I'd like to say something more about the supported models in ProjPret. In terms of the response, ProjPret has the so-called traditional prediction, which works for the Gaussian family that we just saw, the binomial family and the Poisson family. Recently, the augmented data prediction has been added, which adds support for the cumulative family from BRMS, which is an ordinal family, and the same observation model as STENPOLR fits from RSNARM. And the augmented data prediction also adds support for the categorical family from BRMS, which is also known as multinomial logistic regression. Yeah, recently, the latent prediction has been added as well, which also adds support, native support for the cumulative family from BRMS and STENPOLR fits from RSNARM. But the latent prediction is a much more general principle. So it also supports many more families in a custom way. So in that case, a little more input is required from the user. On the side of the predictors, ProjPret supports conventional or population-level terms, multi-level or group-level terms, and as an experimental feature, additive or smooth terms. For further information, you can take a look at the main Vignette. It has a section on supported types of models. Now here, I've appended some slides concerning the history of the PPVS for those who are interested, but I'm not going through this in detail now. And yeah, the remaining slides are references. And Ben, I'd like to thank you for your attention, and I'm excited to hear your questions. Thank you so much, Frank. OK, we'll give everyone a minute to put some questions in the chat for Frank. None quite yet. We made sure that we were linking everyone to the materials for your talk, Frank. So can validate that those links are correct. We'll give it a few seconds for people to chime in. Yeah, I'm sorry for the beginning where the slides were not advancing. So the slides are available on the website. Sorry for that. Excellent, no problem. OK, we have a question in the chat from Ben. How does SuggestSize work? What is heuristic involved? Yeah, simply, basically, it looks at the plot that we saw, the predictive performance plot, and then it picks that size where the error bar, the upper end of the error bar, is above the reference models dashed red horizontal line. This is basically the idea behind it. So it's more focused on sparsity. Excellent. All right, if there are any other questions, we do have a few more seconds, but we can work on getting our next speaker lined up.