 Let's start by the long term, or what we identified because I don't know what's going on. So minimizing self-delusion. But delusion. Right? So actually it came from, I think it's right. So it was like how to minimize the fact that you tweak your own protocol. And it goes together with having this, and you see that we kind of build up on this first idea of having a well-defined workflow. Then make it fully automated. So for that, define it with very amount of cerebellum. And finally, let's follow some benchmarking. On the benchmarking as a sign-out, you might have noticed that post-conveyor has now a specific issue about benchmarking. So it goes in this direction, which is nice. So minimizing self-delusion. So how about tweaking yourself your simulation to get what you want, and rely on defined workflow. And rely on defined workflow involved. As I said, find relevant observables to define and ensure that we agree on the same standard and parameter to be defined. So last year, actually it could go out almost everywhere. Okay, to do that, we already have some tools. I'm going to repeat myself a little bit because yesterday we already talked about them for the workflow, biophysicist base, empty trash. Biophysicist code didn't show up till now. So it's a work from Joe. So it's the name of the GitHub repo, right? Biophysicist code. So I would have one button and two buttons. But if you're not sure, you're at the end. What's that code after? Biophysicist code. It's just GitHub, biophysicist code. But it's a Python wrapper, particularly to ground access, which you said, run and analyze simulations. Okay. And a GUI. So we talked about reproducibility of protocols. But that's also something important. It's reproducibility of analysis. Because actually you might get the same trajectory as a protocol, but actually two ways to analyze it. And then coming to different conclusions. You could have a problem in your analysis or whatever. So reproducibility of analysis is really important. On tools like MD analysis could be the solution to kind of have a common workflow to do. So, reproducibility of analysis. Yes. So something important as well. When you compare your data on what you got from your simulation and you want to compare with some biological data, don't compare with interpreted data. Compare with raw data. Because we cannot challenge you on raw data. Raw data, raw data. You compare with raw data. That's fine. Interpreted data. Yeah, that's coming back to the reproducibility of analysis. You should know how it has been interpreted. So should we try to then reproduce diffraction patterns? Yeah. And only intensities, rather than distances. But how do we get experimentalists to share that? How do we get experimentalists to share the raw data? Oh, they have to. They have to use the QRRD. They've worked at all the information. It's now a strong meeting for a week. I agree. It's about the same as re-sharing trajectories, right? Yeah. But you can imagine if you do that, you will have also, you may put that in some databases and you'll only difference between the experiment and the result. You just understand. It's probably different experiments to be like. It's about linking data, right? Yeah. You can have experimental data in one repository and you can have simulations and so on. All these two are connected to each other. Maybe some people will understand what you are doing. Maybe you can understand. Yeah. There is some, like raw data available, which is just the problem is that it's difficult to read. Personally, I can definitely have. So for the NMR, I can read those things. But I remember when I was not able to read them. So you have to understand the experiment quite well often. That's a good idea anyway. If you want to simulate an experiment, it's a good idea to understand the experiment in the first place. Yeah, but then it's like you don't have time to learn how the air of each experiment. Because sometimes it takes that too. But yeah, maybe NMR is just, because that's probably the most difficult one. I agree. But there is data available. Should we say we should reach some kind of satisfactory level of understanding, maybe not be the expert in NMR and please have some idea about it. And we should learn to talk with the experiments a bit. Yes. This is the key. You have to be able to understand, like talk with them. I am absolutely for communication because I think that's what we're lacking. I mean, I don't want to go in the lab and do the experiments by myself. But for some reason, all these isolated companion groups and then we have experimental groups and then we try to locate each other's data and I think they should be probably more... Symmetry. Symmetry. Yeah. One thing, for example, I thought that some parameters NMR people extracted and I wanted to compare with my simulation and they didn't agree. I wanted to learn about the tasting assumptions which they made in deriving these parameters. So I asked them, what did they assume? And they said, nothing. What? Yes, it's called a model-free approach. It's actually a model. Yes. And so I said, there is no model-free approach in the world. There is a model which I'm doing. So let's talk about it. And in talking to them, I learned about their assumption and the funny thing was, they also, they were aware of their assumption. And so that was very fruitful. And it's terrible branding. It's so illiquid. One reason. So, what we consider as an easy win on the kind of work is if you want to make things reproducible, it should be accessible as well. So accessible repository, keepers and although we're naming it, that's not going to exist in tools, actually. And then luxuries, because it's complicated, we already highlighted yesterday, full interoperability of MD tools. So if everything can be just switched and what you are, you can just use, together, yeah, you will achieve an important disability quite quickly. So I put it there. Yes, since we are talking about data, just before, reliable experimental data for benchmarking. Because sometimes, so benchmarking is another thing that you can see if you want to compare our methods, I think we need to define a reliable benchmark set, which is related to biological questions that we want to answer. So not to need tests or whatever, or speed, but you know, delta, delta Gs or all the parameters which are in different data. You probably don't want to do after realization rates because that's going to be hard. But yeah, so that's, it's not something which is in our hands. So that's something we need experimental people also to do one. But it's often, especially if you do free energy kind of stuff, you are collecting all bunch of data from different groups and experimental methods and that's what the benchmarking is. And the D3R is only a good one. It's a good way of doing that because at least the data and D3R are coming from the single groups and they are all being measured in a consistent way. You all. So that's probably somewhere in the middle here. I think it's very important for the field. It's not too difficult, but we need to find what we can do and what to do. Link to that. We have more benchmarking. That's the standard benchmark set we have it. Benchmarks related to biological questions. You don't want to say I can't run half a nanosecond and I'm getting the same results. What is the question that you're answering half a nanosecond is your relationship. So that's probably that one. Annual benchmarking by a third party. Some of our course field developers are some good developers. I see here. So what you see by your space is going to do that because you can't run everything. I don't know. But you can't compete to do this. Yeah. But you want something then we need to define what you want to benchmark. You're going to do that but you can say every year there's a new version then we're going to put the benchmarks and publish it in some public site and that's it and people can do better. The major issue is that of course the developer will say but you didn't use software in the proper way because you had to tune that thing. So that's what you're doing. So is it important? Is it difficult? That's probably not so simple to arrange to make everyone angry. What do we have here? Understanding MD parameters. Mental study on MD parameters. So that's that was yours actually and we put it somewhere here in the middle. I think it's important. It's for sure important for the people I guess as developers you should understand what the parameters mean but the people using it do they understand what it means. So that's You're pushing a bit more to the way. Yeah. Very important. So, okay. And a bit lower? It's not that high it's a bit higher. You're still not proud. You're still not proud. You're still not proud. You're still not proud and probably this is also about so link to that I would say tutorials. So if you want people to understand things we should also provide tutorials so that's probably easy to do and I think it's important. It's not, I think it's important. Provide a list of sanity reality checklist. Check. So if someone set up so if you think of this, this and this probably before they run their microseconds that's not important. So that should be, I would say easy to do and it would say maybe wasting a lot of resources. Having obsessed many students it seems to be pretty difficult. Yeah. I guess maybe we should come with a list but that's so is there a good this is a checklist before you run your simulation? Making people to use it maybe. Yeah. But if I can but there are a lot of things here that are not simple because people say I've modeled the 30 my example I've modeled the 30 missing residues from that loop and then I'm going to run the microsecond simulation probably you're going to get garbage because you cannot model reliably 40 missing I mean. I was just going to say that like yesterday everything is important and difficult so that's what you had stuff that was more on the left on your chart because you had scaled it too. Yeah. So there was something about killing features so that was a video as well. I think So there is a lot of stuff that we have but how much do we move by now? So essentially killing features it was like killing things that we know are not working software. No, no, no. I think it was awesome by just reduced complexity even if things are working even if we know the physics is correct even if this isn't part of software even then you kill it if it's not used by the community and understood and so on. It's abused. Yeah. Yeah. So I guess that's easy to do on that importance as well. But there's also a known flaw with price call it shouldn't really still be used and especially if that's a price call without that price. So you flip up the safety cover right? Yeah. So there was someone in Twitter yesterday when some people are following the Twitter feed that we have and say okay we're speaking about formats and things that you say okay don't define a new format for simulation unless you remove another one. So all that new feature until we remove one. Two. Two. Two. Yeah. Let's see. Regression tests. Yeah. It's important that it's easy. If you get it you can get it two or something. Two or something. But that doesn't mean that it's realized that you will reproduce your simulation. That's okay. Education in statistics. I think that goes with tutorials. Multiple starting points. So just teach people that they should not run a single simulation but they should ask for that. That's all about tutorials and education I guess. Minimum standards for reporting results. I think that came on yesterday also as well. So what is the there is no defined standards that the journal require. If you report a simulation provide this table with these parameters in there or these statistics. It's there for structure. It's there for crystal structure. Probably as well but there is nothing for standard situations. I think that's quite easy to do and it's also important. It will go in the training part here. Existing tools. So policy of space and the trash and the analysis they are even to facilitate things it doesn't mean that people are going to do less mistakes probably but they also for example there's a primal plug-in for relax. I don't know how today it is but and one question so there are examples for is it existing or is it probably there but do we need to store the software the compile software and the OS in a VL to be able to go back to that. I know. There is an initiative in the US called NMRbox.org where they actually doing that. So they got funding from NIH to keep software versions with a particular OS system and if you want to say I want to have version X running on this system then there is a copy of that VM available which you can start. So it's not only about archiving the software but actually executable and the OS. Of course for MD this will not allow you to re-run a microsecond of simulation if you only have a VM but you might at least try test a small bit of the code and that's the question mark. Challenges or it concerns difficulties what is the question which will lead us to that clear question that you want to answer before starting simulation is not always the case. I think there are still a lot of people running simulation but what can I do with this data that was mentioned. Can I add to that one because the word hypothesis was product really easy. I think that's a better way to put it. What is your question then to demand that one has to have a hypothesis. No, yeah, I agree. I think that hypothesis is somewhere in there already? No, no, no. People just mentioned it. Yeah, questions. I agree. The time for biologists you always have to have a hypothesis. It's psychology and now it's even worse. But now they're doing these pre-registrations which I think for us may not necessarily have a little bit of difficulty. Concerned difficulty available resources linked to repeats also for benchmarking so if you want to make it the recurring thing so do we have resources to do that and one which is probably linked to understanding parameters so it's a concern difficulty coupling of force fields software and MD parameters so it's in there but that's also a concern. So force field is often parameterized with a given version of a software on some given hardware and it doesn't mean that you're going to get the same results with the same force field when you move to the next version of the software on a different hardware. So how resilient are the force fields to the details that are there when you use them? You should really just ship the machine you use to do the studies. Yeah, right, that's impossible to do but we have to accept that there will be statistical differences but then how much do you accept? And maybe if there are a good benchmark you can say it doesn't matter because you still get within your 0.2EV error point. Actually you can very much go next. Keep really happy and use the tape. Give you some tape. I'm glad that we are reducing difficulties in today's session like in the previous session everything was mostly filed in the upper right corner and now I think we are making our a little bit easier seems that we are in the sphere of possible today. So we of course identified the usual set of existing tools for doing things that are relevant to the questions of how do we have reproducible analysis to be obviously how the analysis tool gets that are out there reasonably useful for their purpose. Daniel identified that there's an initiative called Popup to do with trying to build more general reproducible platforms for all kinds of computational sites not just the molecular simulation community. So make sure we don't reinvent those wheels go and use at least the ideas if not the infrastructure that's already available for us. Hopper? Hopper, yeah. That upsets me. EOPPR. That's awesome. All right, yeah. We identified that we need to have force field validation suites so that we understand whether the simulation tools we are using are actually fit for what they are designed for. Very often people produce a force field that's like, hey, here's a set of files go and use it with some piece of software and you'll get the results written on the page. People update these over time. We need much better understanding of whether our tools are actually doing what they were supposed to say in the box. Very often they haven't said on the box what they should do either. I think these are a good example of where a little force field fails too much. Sure. So you need to identify that the manufacturer tries to work on this set of prawns using this sort of experimental work on QM data on the ball with the protein. If the force field validation suite doesn't have examples of value of intrinsically disordered proteins, you have a much higher burden of proof in designing your study. Should we kind of aim for more specific force fields or maybe something that is more transferable? We need to know what we have and currently don't. People just go, ah, I used to use that. I think we need to identify where the problems are and then kind of think how to fix them. But I think I really don't even know where the problems are. So we identify that people would like to be able to do the same kind of data analysis on other systems, perhaps tweaking some of the parameters either in the simulation or in the analysis. That of course requires that people have shared the scripts that they use so that you have the opportunity to do that. Obviously there's a bunch of related things here about where that actually gets pushed. It's already up there somewhere. Yeah, I'm just looking for it. So there's interoperability, but we want to be able to run the same analysis again for studying products. The chain. Fully automated workflow, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0. I'm going we need to be able to share our input files so that people can actually see what it was. Very often there are defaults set up in the simulation packages or in the analysis packages. They change over time. We need to be able to go back and see the output of those packages to say, OK, the default now was this. If we don't have that available, all we have is one sentence in the paper that says, yes, we wrote Python scripts to do our analysis. Your work is essentially fabulous. Nobody can reproduce it. This should be rejected at review time. Speaking of which, individuals who are reviewing papers should be much more strict. Yes, that's a diplomatic word to you. I was thinking of a lot of what. We need to be strict with each other and accept that we do have reproducibility, reproducibility crisis in our community and part of that relies on policing each other. When you review a manuscript, have a look to see whether you could possibly reproduce this work with the data they have shared, but they haven't got their simulation input files but they haven't shared their analysis scripts. What they have done is to invent a bunch of numbers that could be random for all of you. I would argue that's the only thing we should be doing in reviews. That literally is our only job is to make sure that another competent person could reproduce what was in the manuscript. This is an easy win. We should all be doing this. Getting the whole community to do it is a bit hard, so we acknowledge that there are two parts to it. But of course the more frequently you get a review response saying I could not possibly reproduce what you have made available to better then people will start building that into their experimental designs and the planning of their projects and hopefully those people will grow up and then do the reviewers and they will do the same sorts of things. Community change takes time. The best ways to change people's mind is to wait for people who can't change their minds to die. Can I ask you to point to something saying here's what you should be doing instead. Yes, that's that also. As a community we need to put together best practices and publish them in journals like live columns. We know what we should do. We need to have both experimental designs that permit us to gather statistics and to understand how to analyze them appropriately. That connects to something we already had. That of course connects to how we got documentation and training available for people so that we understand that just doing a single simulation for 200 microseconds and then it's like, oh look, this really isn't enough. Can they work in force fields or do you have to get how do you assess whether your experimental design was potentially competent. So let's go into the next one, are the tools we're using actually fit for the experiment we're trying to use them for? Most people adopt the methodology they're going to use from a high. They're going to do what the postdoc tells them, they're going to do what the PI tells them, they're shifting labs, they don't understand what the expectations of the simulation community are around. The particular kinds of simulations they're thinking are running actually operate. Very often people design an experiment without understanding it. What would I require from my analysis package or my simulation packaging order that I can actually conduct in my experiment? What data should I require from that particular version of the particular hardware with the particular parameter choices that I'm making? How can I know that this combination of choices and software that I've been using, how do I know that they are fit for any particular purpose? One way this problem manifests is that when we update versions of our analysis tools and software packages, nobody takes them up for the first couple of years because they're all, I'm worried about there being a bug. One of my responses to that is how do you know you don't currently have a bug when it's affecting your results? Most people have actually grapple with other tools I'm using likely to produce in some sort of reproducible or validated way the ones that I'm actually looking to sample. If you already have that, then it's very straightforward to run them again on the updated version of the software and take advantage of the new hardware, new performance, and the mixed bugs in the end time. If you ever thought of that with your tools of fit, you just generated random numbers. Should we maybe put that like, what is the question? Because I think it's also, what is the question and again, what are the best tools to address this question? And careful design of experiments is something that connects on from that we need to understand what we are trying to observe. It is valid to have a hypothesis free walk in the park but it is relatively unlikely that your look just ran on trajectory to see what would happen. Very unlikely to be able to defend the conclusion that the dynamics of this particular membrane prudence diffusion is this. You need to design some other experiment. Having worked out that these things may be interesting that you need to do a broader study to actually say something about the time scale of the dynamics that you saw hints on in your first exploratory simulation. I mean to know in advance whether you're an exploratory or something that's designed to produce some sort of quantitative result. Both the value that you don't just get to the right directory and then you make the other. So, everything's doing good. I'm going to counter that because we have a really, really interesting discussion at the beginning of ours which really focused on why we run it in simulations. What is the value we're trying to extract from the simulation? And actually the value we're trying to extract might be a question of is an experimentalist coming up to you and saying I have all of these possible mutations I can perform to this protein. Which one of these mutations are the ones I should look at first? Or I have all of these different ligands, modifications to this ligand that binds well. Which one of these modifications should I try first? Synthesize to actually go. And the computational chemist basically has to make that decision relatively quickly. They're not going to invest a huge cost to actually give that answer because the value is I'm going to choose these versus these. And actually that is a combination of simulation. It's to provide inspiration and it's actually a combination of simulation plus intuition. So what you tend to find is that real computational chemists working in the field, the best one I saw was a med chemist who basically was great at designing drugs in her company. And what she would do is she'd run very quick simulations of dynamics. She'd look at the binding sites. She would see where the protein was moving and then because she had so much inbuilt she would say, aha, that looks like that might be a pocket about to open. I think you should begin functionalizing the ligand on this side. And that was the result she gave back. Functionalizing on that side more often than not gave drugs which found better and she did a fantastic medicinal chemist who was highly found by her company. But actually that decision of going here or another decision of make these mutants in this residue, that decision is not a reproducible decision but it's extremely valuable decision because it actually helps the experimentalists get to where they're going. And what this really came back to for us is what can you really expect from a simulation? So there are simulations you can perform and there are real calculatable observable properties which you can link back to experiment and which you can use as benchmarks to move the field forward. But actually that range of things we can calculate is so far less than what the experimentalists are asking us to actually give them. And that's why there is this goal for the reproducibility. Because if you're trying to do what the experimentalists want you are really running hero simulations where the thing you're trying to calculate is so far beyond the auto-correlation time of your simulation there's no way you can statistically converge it even if you run a thousand replicas. So actually these things you're being asked to do to run from the experimentalists maybe we should stop trying to justify our intuition based decisions by running these very complicated simulations but we should actually step back and say let's use some quick and dirty methods for that and accept that we can't calculate those things. So the thing we actually mentioned we've all got condensed down into one thing which was we need to actually set out best practices for planning simulations and be able to detect poor conversions. And actually work out what is it we can calculate. And so let me put this kind of in the middle and then the other thing we said. So what is actually by what is the thing you're able to calculate. So we basically have a protein where we know the motion of this protein is on the seconds time scale. I'm sorry for using the example but if this motion is happening on the seconds time scale and the only thing you can simulate is on 10 nanoseconds or 100 nanoseconds on the seconds time scale you need to go through several motions probably 50 of those motions to be able to get any statistical convergence of that. No way can you run a 50 second simulation of this protein and use a field to actually have the confidence to go to experimental colleagues and say what you're asking me to do I cannot rigorously give you. But if you're asking me for inspiration and say what things out of this selection which one would I go to using my intuition and my knowledge of simulation I can give you an answer but it's not going to be a great answer. And I think we need to then actually say go back to the quick and dirty methods and say actually this isn't reproducible but this is the combination of intuition plus experiment. It helps a lot to go to single molecule experiments because it's easier to simulate single molecules than to simulate the same single molecule. And the single molecule field they are very much aware of these averaging and problems and challenges because they have the same thing so it's much easier to talk to them. And then experiments contain more information. I feel I've kind of started with protein and they're going down to single molecules which I've discussed in one because it's also a lot of fun but at least you can converge things. And actually one of the things you learn in the single molecule field is you learn how to autocorrelation times. And you learn about system thermodynamics and you learn how to actually calculate your errors properly and understand that this thing you're doing if the simulation autocorrelation time is longer than your simulation then you're still in the calibration. And nothing you can do no matter how many times I run the simulation I'm still in the calibration. And so what we need is whether or not it's two objects automatically calculate the autocorrelation time that you're looking at. So as the simulation can tell you sorry there was no point in running that simulation which is a whole thing safer. It's just the way it is. In terms of easy wins we do need to trust our tools and through so many times in this workshop our tools don't actually give us the same answers. So different MD packages, different simulation tools they give us different energies for the same force field in quotes and actually a force field is a combination of the package plus the implementation of that force field. And so a really easy thing to do is basically if we did collect a benchmark suite of molecules which have been parameterized in lots of different force fields and we made it easy in our packages so there's a little option that just said calculate energy and print the energy out Amber doesn't do this it's really annoying to get that single point energy because you've got to do one time step then back it up and everything else but if we can actually make it to every single simulation package you could give it a molecular file input in its own format it just prints an energy out we could then do energy benchmarking and at least be able to say all of our packages will give the same energy for this molecule parameterized with channel 35 so they get the same energies for this molecule parameterized with Amber that would be a really quick and easy thing to do we kind of begun that with bias in space so I think it's something we could actually help to develop and that was automated comparison of serious snapshot energies and forces for codes, for different force fields we then went moving on from there actually we need to do automated evaluation of the methods for physical problems once we have defined things we can actually calculate in the code so in the single molecule field it's hydration for the energies if there is anything like that in the protein field and we could really come up with something that you could actually calculate that would be useful but if there's something like that we could automatically run the simulation and collect that observable out and then just see how different codes or different methods how close they could get to the experimental observable that would be really useful and it would help us feel along so it's easy to automate it and one thing that's difficult to actually work on what it is we're trying to get that then moved on to this one which is basically the automated sensitivity analysis to program settings because we have chaotic systems so we know we make any change in the simulation parameter it's going to change everything in terms of the simulation and then statistical dynamics will mean actually that's okay because statistical analysis takes care of the system and it will give us physical observables but actually how sensitive are our simulations to those changes because the problem with statistical dynamics basically if you run it for an infinite time you'll get the average and it will work but if we change the settings of the program how long is the infinite time change so it will converge more slowly with some settings it will converge more quickly with other settings and so the average you calculate really will depend on what settings you use so sensitivity analysis will be really useful Can I be here? Yeah, I will say this defined reproducible observables so actually defining them would be really good automatic benchmarking of force fields because of the properties I kind of think we've already said that one as well we need to define automated workflows for reproducible computing observables we keep hitting the same thing we need things we can observe then almost like challenging the field and give us workflows where you're calculating it once we use the workflow we're going to run them basically validates that we get the same answers as you and then automatically run them lots of times but to really make all of this work we need to facilitate data sharing and discovery to enable the comparison of different simulations so actually people sharing their data is the first step and that publishing everything so the end of my life this was brilliant, it was a great tour because actually just people publishing everything live into the field we just start publishing our simulations as we're running them rather than three months or six months after the paper was published I think that would be a very easy thing to do I would have high potential investment and then existing tools D3R sample blind challenges blind challenges where somebody goes out and says I have this thing, I want you to calculate which is a real experimental thing and here is some basic input everyone, I don't care how you do it calculate it, blind and we'll see who wins so this is what the D3R challenges for we're approaching the binding for energy sample challenges as well more of these challenges would be good maybe as a community you could work out what is your challenge from ElectroDynamics I come from the binding community so I don't know what it is but what is your challenge for every day what are you trying to reproduce and then make the blind challenge a competition out of it and then make sure everything is published and then my life is just fantastic Well thank you all Any comments? Can I? The calculation of the correlation times you put on the easy side Where is that? I'd like to challenge that and I think you cannot calculate the calculation times but we'll even find out whether you are converged or not in MD because you may leave a complete large mention of phase phase unexcored, you will never detect that in the MD, you have no chance to detect that, that's the dangerous thing It's better than nothing But it's not as good as having multiple starting points which is very easy on the scale Okay so I think what we got again in this session unlike other sessions is that we have an easy and important quarter of the course built I think that the thing is that a lot of things we can already change right now without actually implementing models in any particular software or something what kind of comes across we need to define actually best practices I think that's what we are obviously sorely lacking and to define what are we doing and how do we want to do it essentially and for that we need probably better training, more tutorials we should let people know what are the minimum standards I guess when you are trying to analyze data what is expected of you because at the moment I don't think there is a clear consensus that we can calculate RMSD but why? why did that become a particular variable that everyone is calculating so again going back to the challenge why did you calculate this why did you run this simulation what are you trying to do what is the most appropriate way of doing that and that brings us to the more complicated thing which is again obviously there is a need of automation whether it's workflow or benchmarking but I think the clear goal is that we need to automate as many things as possible because then it removes this probably I think it removes a possibility of introducing more errors because we always all make mistakes and if you leave to people it's probably more likely there will be more mistakes to introduce and I think probably if we automate things it will be easier to actually track down mistakes and errors because then we can go back and actually accurately track down every step that was done and try to figure out where things went wrong speaking of that it's also clear that we should actually treat simulations and I mean in a way we shouldn't treat force fields and software packages separately we should actually similar to what QM when people report we use this level of theory with this basis set I think we should use this force field with this software version but that should actually be really clear that it's one thing that gives you this particular result I mean we do report we use this software package version but we should really make it even like reporting standard like Ember 99 FSB slash Ember 12 0.0 I don't know something like that we can think about it and I guess if we make these things automated it will be challenging probably some things will be easier some things will be harder we definitely need to start sharing the data if we want to get any kind of useful information in terms of what kind of parameters people are using to run simulations what values do they choose for thermostats, for barestats what kind of values they use for their electrostatics so on and so forth how to do it I guess maybe we came closer a little bit maybe we didn't I guess we'll see I think for the time being we can only use the existing repository we can all dump at least the input files and maybe at least the initial coordinates and maybe even last and maybe even like 5 frames from the entire simulation that's not really too heavy in terms of data it's really easy but no PDFs I mean again going back to QM they also share their minimized geometries the PDF which is not perfect but at least they share it and we don't share our PDFs we share stupid figures and movies I mean considering how much time we spend staring at the simulation and looking at things from various angles this is horrible we should really use the ability of computers and these interactive interfaces if we can and we can but we are not doing it again going back to the most difficult side as always community how to actually change people how to make people to do things better how to train people to review actually papers there is really this interesting initially called pre-review where postdocs and PhD students are trained to do peer review and they are done online but they are actually peer reviewing I think bio-archive papers what to focus on when you are peer reviewing papers and how to be constructively critical rather than offensively or something else so maybe we can also create guidelines how to review an MD paper what are the minimum requirements what are the minimum reporting standards what is the data you can provide is it what all the analysis creates what is this minimum package that every MD paper should have and ship together with your main publication we have tools to analyze and maybe we can also ask if someone is willing to pay the responsibility of maintaining MD trench we have challenges and we also have some experimental data but maybe we could also for someone team up again maybe this is more for this collaborative approach and ask experimental people to design some kind of experimental set that would be actually really easy to compare to MD maybe we can come up with a different set for different functions providing free energy maybe some other structural properties NMR or FRED or DR there are many things that are out there and maybe we can select a bunch of proteins for which we have structure these things on multiple residues or something I don't know there are probably ways to do it but someone has to do it and it's not again not exciting if one plan your nature papers so how do we get there so NMR lipids database content also experimental data but now it's on the lipids but I'm not working to find the correct parameters to use for proteins NMR lipids is like trying to go but there's lipids and then you can have proteins and you can have DNA and you have RNA the data set probably explode pretty quickly and that's just NMR but we probably want to test against multiple so we have NMR scattering that but the most important question is that you have to have good data like what is good data so for lipids we have this too and for the proteins I'm working on so we are trying to find what's the best I think that's probably the challenge so you said something very important you won't get a nature paper from that and that's something which I think really clicks our field than the other fields because none of those things which we all seem to agree on are very important to get as a nature paper so how can we go around that how can we break this dictatorship I should almost say of having to publish in nature and science and so on that's the only thing actually the nature papers are not important what is important is to get money it's important to get salary so I don't care if I get paper to the nature probably I care no this is what people think that's perfect unfortunately no it's not I checked five years of very good funding to do NMR with this project I didn't have any time for papers without any vision to see that I would publish time with paper I'm not saying I have a set correlation but if you have two nature papers in a row that enhance it's a change it's about queuing the probability is this the fact is it like do you have data for this I have heard I have heard people claiming that there is data which shows opposite it's not necessarily us that's the problem maybe not even funding but it's the trainees so if we have young trainees that are embarking on academic careers right now they will lead a nature seller science paper to get a decent R1 job but this is I have seen people arguing with the data opposite and I just want to say this is what everybody thinks and also one thing if everybody thinks like we are the scientists okay we are the ones who review the applications if we think like that it becomes self like it if we think like that it's gonna be like that if we stop thinking like that it's not gonna be like that so we are the ones who decide in that so we have to remember that once you enter in that state of mind it's really difficult to get out we still congratulate each other when we publish in high because it's a great work see on twitter oh this person published here and everyone said like oh amazing congratulations when you publish people tell you there is no this euphoria when you publish in a lesser impact I mean it exists it's present it's not in our heads but we are designing the scientists who are designing that our senior colleagues at our universities are the ones who are designing it we have to persuade them but we are going to fail let's retire first maybe we find some changes because that's a hot topic of course but if you are in a search comedy or an appointment comedy or in a fund agency comedy you've got a pile of 100 series for example with publicists and you have to sort out 10 for a short list in a cover of weekend and at that point it becomes very hard not to pay attention to these higher chance because you are very much tempted you cannot read all the public pensions simply not that's a problem doesn't make any sense this means that the editors of nature and science are actually the people who are designing that's the best part I was interested in the last part let's move away there is something we can change and we might fix scientific publishing can I just say again the UK is better but the UK most institutions have signed up to the San Francisco Declaration which says they do not use impact factor to judge academic progression the next thing we've done is in our research council now exploring blind peer review where the actual person your track record is now not seen by the reviewers of the grant it can't be far enough it can't it's anonymising as much as possible we actually ran a trial of it and they were not funded in Oxford and so there is a strong movement going on in the UK and I hope in the rest of the world where we basically get rid of this thing we've signed up to the Plan S which is the method which basically pushes open standards I think it's a really important thing San Francisco Declaration Plan S and a move to blind refereeing and blind re-referring of grants it only comes in the UK because it was demanded by the research community because the research councils have to follow the research community and they want it from your funders yes so there is power but people are unwilling to voice their opinions we could change things but we are not so so I guess what are the concerns and difficulties I guess in a way who will do it there is a lot of this is not glamorous for when you do some and let's face it it's boring for some people but there is a lot of benchmarking a lot of repeating it's a lot of repetitive work we can try to automate as much as possible but at the beginning it could be a lot of also just like digging in the tunnels all these sorts of so who will do it one research group take the bridge two, three who will do all this work maybe one more comment on that one I sounded maybe already negative so it may be true that you don't get a science on Asia paper but what is also true is most of these work create citations which last very long and are very high in numbers and I think to get nature and science out of business much more important is to pay attention to citations rather than to journals and there we have a benefit and that may be a very good sense of us to invest time in this yes so I think this should be also maybe a really a distributed project so we invite many people to actually contribute so if everyone just does like a little part then it becomes also easier and less taxing so you don't have to come a lot of dedicate all of your time and I think probably live comms is also a very good place to publish these results so we don't have to I think live comms is a really great place to publish these like best practices that we are like we can update, we can change we can modify and we can add things that are actually important to us and I think methods are really important because that's something that's transferable proteins will change, you have to behave in a different way and sure if you capture that for your event it will be amazing and interesting to see but at the end of the day I think what we need to do is not make the methods reliable and trustworthy so when you have a problem you can say I have this method and it's really good to solve these problems but now I think we are still agreeing with dark a little bit so maybe we should turn on some white and see where the holes, put holes not be folding to them