 So welcome to the Biaxcel webinar. The webinar of today is on non-bonded libraries, and BLIP, a performance portable library for computing force and energy of multi-particle systems. The presenter of today are Joe Jordan from the Royal Institute of Technology, as Sebastian Keller from CSCS Lugano. I'm Alessandra from the Royal Institute of Technology, hosting this webinar together with Julian Singh from the University of Edinburgh. Today's presenter are Joe Jordan. So Joe Jordan got his PhD in biochemistry and molecular biophysics at the University of Pennsylvania, and he moved to Sweden in 2018 to work in Stockholm in the group of Erin Lindl. He was doing a postdoc of a driving molecular simulation with his experimental data. Then he moved to the Computer Centre of the Royal Institute of Technology in 2019, PEDESSE, and his actual position is a research and software engineering. Sebastian got his PhD in theoretical chemistry at age in Zurich, Switzerland. Then in 2018 also he moved, but he moved to Stranford University for a postdoc on GPU-accelerated electronic structure simulation. Then in 2019 he went back to Switzerland and he started to work at CSCS Lugano. So now I will give the word to Joe so he can start the webinar. Okay, so today we're going to be talking about INB-Lib, a performance API for force calculations. So, okay, there we go. First, I will say who are the people who have been working on this project. So myself and Sebastian have already been introduced, but in addition, there is Prashant Kanduri at CSCS in Zurich, Victor Holanda, who is also at CSCS in Lugano, where Sebastian is, and Artem Zemirov, who is at PDC, the Computer Centre of the Royal Technical Institute of Sweden, as I am. Okay, so the outline of the talk looks like this. I'll talk for just a few minutes about what INB-Lib is. Then I'll give a little bit of background that'll hopefully make the talk a little easier to understand. And I'll start talking a little bit about the implementation, but Sebastian will do most of the talking there. And then at the end, it'll come back to me and I'll go through some example workflows that hopefully will be used cases that people are interested in. Okay, so what are the objectives of the INB-Lib project? So I've highlighted the most important one, the one that requires most of the work, and that's to implement an API that permits existing performance portable grommacks non-bonded force calculation routines to be called as a library. So there's another number of sub-goals or sort of related goals, like being able to have nice C++ on Python code. We currently don't have Python, although we may in the future, and to make the code easy to get your hands on. So in terms of it being easy to get your hands on, it's a version of INB-Lib is actually shipped with Gromax 2021. So you should be able to use it on compute clusters that you have access to. But I should go back for a second and talk a little bit about what Gromax is just so we're all on the same page. So Gromax is one of the most popular codes that are run on high-performance computers, counting for about 5% actually a lot of plots on HPC system. And it's a code for performing molecular dynamics and it's been developed for a long time, almost 30 years and a lot of effort goes into making sure that it's high-performance, but until fairly recently, not so much effort has gone into making it easy to use the code. And so there hasn't really been much of an API and so that's why the INB-Lib project exists. Okay, so another reason why it is relevant especially now to have a project to try to expose the performance aspects of Gromax is because we're moving rapidly into a situation where there's a massive proliferation of different kinds of hardware that are coming online or have recently come online. It's in the high-performance computing world. And this is a good thing if you're a scientist because more competition means that the computers theoretically can be cheaper and that there can also be more competition over speed. But if you are the person programming these computers to try to get the maximum performance, then you have a little bit more work to do because these different machines that I have listed here have different CPU and GPU architectures and this can be very challenging because for instance, you can have to write an entirely different back end for each GPU type as an example. And so again, this makes something like INB-Lib very important because then you can imagine instead of having to go and try to for each new machine, figure out how to get the best performance, you can just say, oh, well, there's some other people who have already put a lot of work into getting the best performance for this particular type of operation. I'll just call their code as a library and focus on the sort of scientific things that I'm actually interested in as opposed to the low level performance details. Okay, so now I'll give a little bit of background. So hopefully we're all on the same page at this talk on what a molecular dynamics simulation is. But basically, you have a bunch of particles or molecules and you compute the forces of those particles or molecules and use that to update the positions and basically do that over and over again to get a dynamic picture of how the positions evolve over time. And so what are the types of forces in a molecular dynamic simulation? Well, there's things like bonds, angles and coercions and there's electrostatic repulsion, you know, a charge interaction and there's also van der Waals attraction. And so the reason that the project is called in-belive is because it is a non-bonded library. And so when we say non-bonded, we typically mean the van der Waals attraction. And to a lesser extent, you could also imagine non-bonded refers to electrostatic repulsion, obviously. And then you've got bonds, you know, other kinds of forces. But the van der Waals interactions are the most computationally intensive. And so the Leonard Jones interaction is basically one way to compute the potential of van der Waals forces. So it looks, you know, the equation for the potential looks like this. So then you have this graph which I've taken straight from the Gromax manual. So at the lowest point on the line on the graph is the energy minima. So that's, you know, where the particles are sort of happiest to be. If they move too close together, they'll get a high force. And as particles move farther apart, the force will actually go to zero, basically. And the fact that the forces go to zero is important because, you know, if you've got a big system and you need to compute all the non-bonded interaction, if the forces didn't go to zero at some relatively fast pace, actually this would be an in squared problem and it would be computationally very challenging to make much headway. But fortunately, it's not really in squared in the total number of particles but only in the particles within the radius. So you can actually sort of divide up, divide and conquer, so to speak, the non-bonded interaction. So what does that look like? Well, you could imagine going down the arrows here, you know, taking your system and dividing it up into little chunks and just computing the non-bonded interactions in those chunks. And this is basically why Gromax gets a very good performance is because a lot of effort over many years has been put into developing this algorithm and also making sure that it runs efficiently on a wide variety of hardware. So really what we're trying to do is to expose that work that's already gone on. And so now I'll move to actually talking a little bit about how we've made it possible to access this performance. Okay, so, excuse me. All right, so in order to do a simulation, you need some particles. And so we actually define a topology API that you can use. And so particles have a name and a mass and these are the only two parameters that particles have. So you can imagine the particles being atoms or if you do coarse-grained simulation beads or, you know, some other thing depending on your domain of application. But so in the example we're gonna talk about today to sort of walk through the implementation. What we're actually gonna be building is a system with methanol and water. So here's the particles in water. You know, you've got an oxygen and a hydrogen. Not so surprising. And here are the particles in methanol. You've got the methyl oxygen and the methyl carbon. And in this case, we're using a United Atom methyl carbon. So there's no hydrogens explicitly attached to the carbon in this scheme. Okay, so the non-bunded interactions are flexible. You can define different kinds of interactions between the different particles in your system. And so in order to do that, you need something called a combination rule. And so this is actually sort of the last background that I'll show. So remembering the formula we had for the potential, the Leonard Jones potential between two particles. The geometric combination rule is exactly what it sounds like at the geometric average of the C6 parameters of your particles and the C12 parameters of your particles. So that's all we mean by a geometric combination rule. So, well, what do you do? You take this particle type interactions object and you add the particles in your system to it. So first here, we're adding the oxygen in the water and giving it C6 and C12 parameters. And we're also adding the hydrogen and actually giving it zero for C6 and C12 parameters. And you can do the same thing for the methanol, oxygen and carbon with, as you can see, different parameters that are from some force field or from some experimental validation that causes you to trust that these numbers are correct to your specifications, to your needs of the problem domain. And so in addition to being able to, so for here, we're basically showing how do you define the generic things that will show what the interaction of particle I and particle J will be for arbitrary particle I and J. So the combination rule is just telling you how to combine the C6 for, for instance, the oxygen in the water and the oxygen in the methyl. But you actually might not want to use the oxygen in the water and the oxygen in the methylgeometric combination rule. You might have some more experimental data that basically says that there's some other number that is gonna give you a more accurate interaction potential. And so if you want in the INB-Lib interaction, API specification, you can do that. You can add explicitly non-self interactions. Okay, so now we have some particles and we have some interactions between our particles. So the interactions, again, are for computing the non-bonded interactions, which is the performance-sensitive ones that we really care about. And so we're gonna add our particles to some molecules. So we're gonna build a water molecule and particles in a molecule must have a name and they may have a charge and a residue name. So in the case of water, we're adding our particles and giving them name, you know, oxygen H1 and H2, sort of obvious names for a water, and we're giving partial charges, which again are coming from a force field, you know, some kind of experimental validation. And then you're just adding as the last parameter the actual particle. And you can do the same thing for methanol. And the one interesting thing that I'll note here is since the particles are themselves kind of uncoupled, you know, they're independent objects, you can actually add, you know, your hydrogen particle to both your water molecule and your methanol molecule. So this is the hydrogen on the oxygen in your methanol. And you can use the same one, but just for the different partial charge. And now you have the different, you know, now you have your two molecules basically. Great. So another thing that is useful to add is, you know, in these kinds of simulations, there's something called intramolecular exclusion. So if anyone is not familiar with what this is, this is basically a performance optimization. So this is saying, you know, in a minute, I'm gonna show you how you add bonds. And so anything, any two particles that are connected directly or, you know, closely at least by bonds, the force from the bonds are gonna be much higher than the forces from the non-bonded interaction. So using exclusions, you can actually say, exclude particle i and particle j from the non-bonded calculation because it's a waste of time to calculate that force. And so this is what that looks like for water. You know, it looks pretty similar for methanol. So all you need to do is keep track of the gains of the particles in the molecule that you're adding the exclusion to. Oh, and I should mention that these self-exclusions are added automatically. So yeah, it doesn't try to compute non-bonded forces with itself. Okay, so as I mentioned, you probably also want to have bonds. If you wanna simulate, you know, anything useful without bonds and angles and such, you can simulate things like argon, which is very interesting, but perhaps not interesting enough for everyone. And so, you know, you can add bonds. So what does that look like? So we have a harmonic bond type and the parameters are the distance in the fourth constant. And so you make your OH bond and you add this as interaction. So obviously in water, we need two bonds, one for each hydrogen. And that's what this looks like. And then now we have a similar setup for methanol. So we want a different OH bond, which I'm gonna call OH3, since it's the third H on the slide. And it has, yeah, whatever parameters it has. And so you add that interaction to your methanol and then you also add an O-methyl bond to your methanol. And so I'm not gonna walk through the addition of angles and dihedral because they look very similar to bonds. So now I'm actually gonna hand it over to Sebastian, who is gonna talk more about how to, you know, finish building the system and actually run using the, yeah. Right, does that look good? Okay. So I'm gonna pick it up right where Joe left it. So he described how you can create molecules from particles and bonds and angles. And before you can actually calculate forces, the remaining information that you need to supply as a user is how many molecules of each type you wanna have in your topology. And so we have a topology builder here that you need to instantiate and add those molecules. And it looks like this. So you add your molecules and you supplies a second argument here, how many of each you want. And the reason why we need this builder is because we cannot efficiently translate the string particle named to a sequence ID. So in order then to finally get the topology, you call topology builder dot build topology when everything has been added. And you also need to add the C6 and C12 parameters that Joe described earlier. And you also need coordinates and velocities which you might define in your program or read from an input file. And you need a buffer to hold the forces. And you need a box to specify the bounding, the coordinate bounds of your coordinate buffer that you just defined, right. And at this point, we basically have every information you might need about your topology and you're in principle now ready to call the force API. So it's perhaps useful to recapitulate quickly what the data flow looks like. So the particles and the bonded interactions, they only go into molecules and then the molecules, they go into the polity. And so what happens next now is that the polity prepares for you the actual inputs to our non-bonded force API. And so those inputs are the particle types. That's just an array of in teachers and then the non-bonded force parameters. So that's the C6 and C12 parameters for each type. And then an array of charges for each particle. And then there's an array of interaction flags, which specifies for each particle, whether the Coulomb electrostatic or the Fonderwall non-bonded interaction should actually be computed. And then there's another array, actually not a two array of in teachers describing what particles are excluded. And so what does that look like in actual code? You also need a small data structure with some options. It mostly has sensible defaults, but it for instance controls parameters such as pair list cutoffs. And just as the graphic illustrated before, you can now instantiate the non-bonded force calculator itself by requesting all the input data that it consumes from the topology. And then why is it actually an object? Why can't we just make a function? That's because the force calculator needs to have a pair list inside that determines which are the pairs of particles that actually are close enough to each other to have Fonderwall's interaction. And because it's actually fairly expensive to compute this pair list, this is done only every say 100 steps or so. And for this reason, you don't want to update this pair list each time you compute forces for a given set of coordinates. And so once you've updated the pair list, you can then finally call compute. You have to give the bounding box the coordinates and then you will obtain the forces. So the forces are going to be added in this buffer supplied here. And it's important to note that actually the whole topology API that Joe and me just described is actually completely optional. So if you're totally empty handed, it might be the easiest way to actually assemble the input data in order to be able to call the non-bonded force API. But for a lot of users that might not be the case. So you might already have a TPR file from another Gromax calculation and that TPR file already contains the exact same information such as the particle types and the non-bonded parameters, et cetera. And it also would also contain a coordinates. So the only thing you'd have to do is to use the TPR file reader and supply your TPR file and then you can directly use the data from there to call the non-bonded force API. And it's even worth mentioning that all of the input data that the non-bonded force API consumes are actually just elementary arrays of int and real or so this is float or double. So it's completely imaginable that you would actually be able to prepare this data yourself using a third-party molecular dynamics code and then you'd still be able to call the non-bonded forces from Gromax. And so that's what the alternate data source would look like in code. You'd create this TPR reader and give it your TPR file and then from there instead of the topology you'd be using this TPR reader to calculate to create the force calculator and the coordinate and force buffers. And then from there you'd be able to call compute and obtain the forces. We do also provide actually listed forces which is Gromax non-bonded for bonded forces. So that's everything from bonds angles to dihedral and the way that this works is exactly analogous to the non-bonded force calculator. You also construct the object with data that you can request from the topology and then the force calculator or the listed force calculator also offers the same the exact same compute signature. And at this point you're actually already in a position to write the complete simple MD loop. We do have a very simple leapfrog integrator but you can of course provide your own two and then you can just simply write a for loop over all the integration steps. Then you then start with computing the non-bonded forces. I mean, this is assuming that here the apology box coordinates and forces has been set up as illustrated before. And then you can add a second call to compute the bonded forces on top. And here in this example, if I call it like this the bonded forces just get added on top and then you can use those forces to move particles ahead in time by one time step. Here in this setup I'd actually have to zero the forces between every step because both the bonded and non-bonded force calculator add forces on top. And you might of course imagine all the stuff that you're doing to the system in between such as add your own kind of forces on top or whatever else you might imagine. And with that, I'd like to hand back to Joe who's going to cover some example workflows in more detail to illustrate cases where the capability that we just described might be useful. Yes, so I'm gonna talk about some example workflows. So the first one I'll talk about is about computing a subset of interaction. And so there are a number of cases where this would be interesting. So I'll go over one that looks a little bit like drug docking and sort of give an idea hopefully of what it would look like to do multiple time-set bangers or quantum mechanics, molecular mechanics, hybrid. And then after that I'll give an example about multiple states. Okay, so for computing a subset of interactions, what would that look like? Okay, so for this example, imagine you have two polymers called in polymer one and polymer two and you have some water and you've already added your particles and your fusions and your interactions. And so now you can build a topology which represents your full system. So for this example, you can say that the full system has one molecule each of polymer one and two and 100 water molecules. And you can also imagine having a system which is just the two polymers. And then you can build each of those topologies respectively the full system and the polymer only system. Okay, and so while you set up your non-bounded fourth calculator and your listed fourth calculator and your leapfrog integrator and that looks similar to the way Sebastian describes or won't belabor the point. And then you again, as Sebastian described basically do some steps. So you can imagine basically if this is a drug docking scenario that you have some other method maybe some other program that gives you trial coordinates basically that you think are going to be relevant for computing the interaction energy between these two polymers or for some other kind of system basically. And so once you've, but it might be the case that you don't trust this other software completely and you kind of wanna relax the system a little bit give it a chance to move around. And then after you've done some a few time steps with your full system now you actually only wanna get the forces for your polymer system because you've updated the coordinates a little bit to your specification. And again, you don't have to use the leapfrog integrator you could implement your own stochastic gradient descent minimization integrator for instance, but at the end of the day you want to only get the forces or perhaps the energies between the polymers that we're interested in here. So right now work is ongoing to actually expose the energy as part of the non-bonded API. But okay, so just sticking with this example just for a moment. So this is an example of how you could do a sort of docking thing where you wanna compute the interaction energy between two particles which are a subset of your whole system. But we also have multiple time stepping and QM in them which are examples of subsets of interaction. So for multiple time stepping instead of having one topology which is the full system and one which is some sets that you're interested in the interactions then you might for instance, divide up your system between everything that's not hydrogens and a topology that's just for hydrogens and then you could actually have a double loop where you integrate the hydrogens a few times and then have a longer time step than you might otherwise be able to for the rest of the system basically. And for QM in simulation that actually kind of like is this example but turned inside out. So in a quantum mechanics, molecular mechanics simulation generally you have some other program CB2K for instance that generates forces for the quantum particles. And so basically what you would do there is instead of defining your system to instead of basically defining two systems one of which is the full and one of which is the only part you would have one system of which one part actually has all zero interaction parameters. And then in between the force calculation and the integration steps you would just add the forces from your other program for instance, CB2K and through this method you can get a decent quantum mechanics, molecular mechanics hybrid simulation. Okay, so now we'll talk about using NBlib to do a workflow with multiple states. So there are a number of types of workflows that involve multiple states. So I'll show an example which actually encompasses both forms of trajectory or replica exchange. And then I'll just kind of describe what it would look like to change the topology during a simulation. Okay, so in multiple states in this case we actually just pretend that you have your topology builder as we've illustrated it before and that you wanna actually build the same system twice. And so I'm gonna call these systems first and second. It might actually be more canonical to call them state A and state B but when reading code state A and state B are a little too close together. So I thought that the example was clear with first and second. But in this example that I'm showing you're actually the same topology. And so what do you do? Well, you get your coordinates and your velocities for your first system and you get your coordinates and velocities for your second system. If they could be the same coordinates they could be different coordinates. You know, it's up to your imagination and the specific type of workflow that you're interested in what this would mean. And so while you define your in B force calculator your non-bonded force calculator as before and your integrator as before and I'll leave out the listed force calculator for reasons of space because it doesn't really add anything in terms of clarity I think to the example. So you do that for your first system and you do likewise for your second system. And so now you can write a loop just like the loops we've been talking about over however many steps you're interested in doing and basically you compute your forces for system first and system second and then you integrate system first and system second. And this is actually already a sort of swarms of trajectories type of workflow right here. So you don't have to limit yourself to two. You can go so wild and do as many as you would like and you can even imagine using MPI to run each one of these actually on a different rank on a supercomputer and get that embarrassing parallelism that you don't have to do much work in order to actually get so that you could more quickly explore phase space since you're interested in by basically running in copies of that system in parallel as opposed to running them one after the other. And so what does it look like to do replica exchange? Well, all you basically need to do is add a little bit of code that gives you in this example every 10 steps lets you swap the velocities. And so, well, why do people like to do this? One example is that you might have different temperatures and that's different velocities for your two systems. And you might want to periodically switch them because you're interested in the statistics at a lower temperature, but you want to increase the exploration of phase space. And so you occasionally swap the velocities with coordinates of your system in order to facilitate this. Okay, and so just to give some idea of how the last type of workflow that involves multiple states, which as I mentioned is having different topologies. Well, how would that work? Let's go back. So changing the topology during the simulation would be as simple as basically building first and second and then having this loop and then after a first and second in this case were actually different and not the same topology twice. You could imagine doing, instead of every 10th step swapping the velocities you could actually write a different loop which was 10 steps of system A and then build a whole new calculator and 10 steps of system B or system second. Okay, so those are the examples that I wanted to talk about. Hopefully they were at least somewhat clear. So I'll talk a little bit about what our future goals are. So one of our main goals is this talk and one of the things we're really hoping to be able to do going forward is to find some users. So we have a pretty good idea of the things we can build and we have what we hope are good ideas about what people will find useful, but the only real way to know what people will find useful is to have users reach out to us and basically say, hey, we're interested in this use case. Can you prioritize getting such and such feature in? And so we are quite open to this. Our emails I guess should be shared as part of this talk and if anyone is interested in reaching out and having a conversation with us about some kind of workflow or some kind of features that they would need for workflows that are very interested in, these are conversations we're really excited to have. So please do reach out in this regard. Okay, and so other things that we're sort of already working on and thinking about are mostly about performance. And so one thing we haven't mentioned is actually the non-binary for stuff that are actually already works on GPUs, which is nice because since Gromax already has basically backends for all the different kinds of GPUs, whatever kind of system you have, you should just, it should just work and you should just be able to run on whatever sort of CPU, GPU setup you have. But one thing we don't currently have is domain decomposition, which is what you would need in order to have a coupled parallel simulation. So I mentioned how you can easily get an embarrassingly parallel simulation where you just run one simulation per MPI rank or per node, whatever. But if you actually wanna run one simulation across nodes, you need domain decomposition. And so this is a little more complex and so this is something we're actively working on. So another performance aspect that's fairly important is actually having nice handles for the data transfer. And so what I mean by that is both from the CPU to a device to GPU and also sort of at the API boundaries because as a user, you may wanna add your own forces or get in the middle of the steps of coordinates back and to do some other kind of thing. For instance, check if you are at a stopping point. And so we need to come up with, yeah, we just need to do more work there. And so this again is a place where hearing from users is extremely useful so we can have an idea of basically which things to optimize. And one sort of longer term goal that we're looking at is task scheduling. So we've been showing kind of naive simulation loops where you just compute one thing and compute another thing. But in principle, this may not be the best possible performance. And so we're hoping to extend our various APIs in order to allow you to overlap these various tasks in order to actually be able to play around on whatever kind of hardware you have in order to try to write a custom schedule to give you the best performance. So that's all we have to say today. So I'd really like to thank Bear Kess who is one of the original developers of Gromax and is in Stockholm at KTH as well and has really been very helpful, sort of pointing the way and helping us understand some of the algorithms that were necessary in order to expose the in the API that we've talked about today. And then in addition to all the people on the IndyLib team that I mentioned at the beginning, we've had a lot of productive discussions with other Gromax developers, namely Christian Blau, Eric Gurgeng, Eric Lindahl, Mark Abraham, Paul Bauer, and Suede Paul. And yeah, so now I guess we'll go to a session and answer session. And I really wanna thank everyone for taking us on to listen to our talk. Thank you, Joe and Sebastian for the very interesting talk. Lucy's asked a couple of quick questions. So the first of those quick questions is, can other integrators be used besides LeapFrog? The integrator is totally independent from the fourth API and you're able to use whatever you can imagine there here. Yeah, Chris, this is one of the things that makes the API so powerful actually is that any kind of workflow you can imagine since the components are completely decoupled and all you need is that the data have some certain sort of minimal amount of data, you can do whatever you want. You can even imagine getting an integrator from some other MD code if you wanted to do that. Great, thank you very much. The other question that Lucy had asked was kind of function be included within a loop to view partial results or is not possible? Partial results as in a subset of the forces or? I can take a guess at what is meant here. But yeah, I mean, so this is what I was saying about data transfer basically. So with all the loops we've shown, if you wanted, you could, put in a prompt that allows you to enter in some extra data or allows you to do some other kind of check. And if the check succeeds, do one thing and if the check doesn't succeed, do another. I mean, so you can look at the forces now and you can look at the coordinates now. The only real caveat is that if you are using a GPU, there is some cost of moving the data back from the device to the host and you have to manage that yourself. But other than that, there's only performance limitations and not limitations in terms of like when you can look at the data or what you can do with it. Great, thank you very much for that answer. The next question we have is from Patricio Barletta. Patricio, I have unmuted your mic if you would like to ask the question you're more than welcome to, otherwise I can read it for you. Cool, thanks. Can you hear me? Yes. Sorry, thanks for the talk. Actually, you have just answered my first question about uploading to the GPU. So it is available, so that's cool. I was also wondering, it's a very specific question. Do I have access that the API exposed the Gromax cell list for the current frame? Oh, be nice. So the Gromax, what for the current frame? The cell list, you know, for the add-ons? Yes, no, it does not. So I'm very curious to hear what your use case is for wanting to look at the pair list. No machine learning kind of force field. Okay, yeah, very interesting. So this is something we've considered and we've even looked at some grant applications that, you know, for writing follow-up projects. So this is an area that we're very interested in. So I would be very interested to talk to you about specifically what you think, you know, would be needed there. But in principle, we can expose any kind of data in practice, it may not be so useful because the data structures, you know, are some more and some less sort of human interactable and intelligible. So it really requires some further specification of like what data is useful, basically. But I'd be happy to talk more about it. Right, yeah, it's a very specific use case. Okay, thank you. Thank you very much for that answer. The next question is from Henry Whitler, whose microphone unfortunately doesn't work. Henry says, interesting work, thank you very much. Can one use NB-Lib test to make molecular specificity, for example, binding affinity in joules or to quantify the force in Newton of a binding interface for EG protein ligand interactions? Yeah, so this is kind of what I was trying to get out with the first example where, you know, in that case, we were just computing the forces between two polymers. But, you know, if you imagine polymer one is a ligand or small molecule and polymer two is a protein, then you're already basically in the situation that I think is being described by the questioner. Great, thank you very much. And our final question for today comes from Jonathan Barneau. Jonathan, I do not seem to be able to unmute your microphone, so I'll ask your question in your stead. Jonathan asks, are virtual sites supported? Virtual sites are not currently supported. We do not have it on our roadmap to support virtual sites, although in principle, they could be, it's just that virtual sites are a lot of extra overhead, and especially since multiple time stepping is, you know, easily facilitated by the API we have outlined and, you know, furthermore is like, I don't wanna say more correct because actually in testing, it has some instabilities because some of the force fields have not been developed for multiple time stepping, but you can imagine as force fields evolve that basically just integrating the fast interaction as much as needed for one period of slower interaction that this would mean that virtual sites can go away. So unless there's a really compelling case, I don't think that at any point in the two near future, we would investigate virtual sites, but I'd be interested to hear what the specific use cases that Jonathan would think are enabled. So Jonathan replied saying that virtual sites are heavily used in the Martini force field. Ah, yeah, of course. Yeah, no, I should have known. Yeah, all right. So it's something we can look into. Yeah, we would have to revisit our topology API in order to figure out what it would take to support this, but it could in principle be compatible. Great, thank you very much for that answer. So with that, we come to the end of our webinar. Thank you very much. Thank you very much, Sebastian and Joe, for coming and giving this very interesting talk and answering all of the questions. Thank you also to all of our attendees for coming to this talk. And I wanted to finish by mentioning that there is another BioXL webinar happening on Tuesday the 30th of March titled Applying the Accelerated Weight Histogram Method to Alchemical Transformation, which will be given by Birk Hess and Mathis Ludborg. It should be a really interesting webinar and we hope to see as many of you there as can come. Thank you very much, everyone. Have a good day.