 Hi, yw Ynw'r Ardys Halffol, mae hwn yn ymweld i'r 3rdd yng nghymru ar y sgolol Southampton, ac mae'n ein sgolol yng Nghymru 2017 yw'r Institute deallu Ffwrdd Sfasurau. Felly, rwy'n gweithio i'r ddweud o'r ddweud o ddweud o ddweud o ddweud o ddweud o ddweud o ddweud o ddweud o ddweud. Felly, ymweld i'r ddweud o ddweud o ddweud o ddweud o ddweud o ddweud o ddweud. Felly, in my research I model explosions in space. Or, if you want a more technical title, it's the effects of including general relativity in models of type I X-ray bursts in neutron star oceans, that basically explosions in space. So, a bit more detailed. So, I'm looking at modelling these explosions which happen on neutron stars. If you look at the top left here, that's kind of an artist impression of a neutron star. Dyma'r rhaid â bod ystod oes yr oedol o'r ystyried yn ymgyrch yn fyrdd amser. Felly y gallwch ar 10 km o raddau, ac yn ymgyrch ar yr oedol y penderfyniadau. Felly mae'r penderfyniadau yw'r gwaith yn ymgyrch yn ymgyrch. Felly mae'r penderfyniadau yn perthyni ar y penderfyniadau. Mae yna ymgyrch ar ddim yn ymgyrch yn y penderfyniadau, ac mae'n gwybod yma'r cyfrifyniadau o'r ymgyrch ac mae'n gyd-ynghyrch yn gyfathio'r cyfrifyniadau Until eventually you get an explosion. And these explosions, they actually happen every few hours. And we have about 50 known sources of these. So, we have lots and lots of observations, unlike the most astrophysical phenomena. But unfortunately neutron stars are really... We don't really know much about them. So, it's really hard to try and understand what is causing these explosions and all the different phenomena we see within these explosions. I'm trying to look at including general relativity in our models to try and understand them further. So I said I started my PhD about three years ago. I did a physics undergraduate degree so in this I actually learned about experiments and stuff. I kind of found it a bit crazy when I started my PhD and started reading all these different papers about really exciting codes. I looked them up and I found lots of them aren't open source so I can't find them anywhere to download them and run them myself. They have next to no documentation so there's a really big code in my field and I went on the website and all the instructions for this supercomputer in Germany. I had no idea how to install on my own computer and I had to find a postdoc to sit down next to me and tell me how to get it to install. There's also kind of some questionable approaches to testing so it's very common in my area to have kind of standard results and people will show that their codes can replicate these standard results but there's no real unit testing or integration tests or anything like that. And personally I don't think this is very good science. I mean it kind of goes against the scientific methods so yeah. I got a bit fed up with this. Fortunately my supervisor, he's quite close to these things and he pointed me in the direction of the Software Sustainability Institute. So this is an institute that was set up to really promote better research software so I'm kind of helping to spread their message today hopefully. Okay so here's an overview of what I'm going to talk about for the rest of my talk. So first I'm going to talk about what is software sustainability and why you should care about it. And I'm going to go over why scientific software is different from other types of software. And then I'm going to present a kind of suggested scientific software development workflow. So kind of a load of different steps you can take to really make sure that your code is sustainable. Okay so what is scientific software sustainability? So I was speaking to one of my colleagues about this. I don't do any coding and he was like oh is this just like printing your code on recycled paper. Software sustainability to me is will your code still work in 5, 10, 20 years time? So if you publish a paper with results from a code chances are people are going to look up your paper in a few years time and they're going to want to try and run your code and get your results again. So kind of there's two main issues here like can you still find your code? So if you post something on the internet websites are often rearranged so you need to really make sure that someone's still going to be able to click on a link or type a URL into the browser and they'll still be able to find your code. There's also the issue of can they still run your code? So if you write your code and it depends on a library will the library still be there in 10 years time? And kind of things like this. And I feel like if someone can't find your code or run it in 10 years time then that harms scientific progress because people are going to have to kind of, yeah people are not going to be able to base their research on previous results they're going to have to kind of recreate the wheel every time. And I don't think this is very good. Okay so what makes scientific software different? So with lots of software it's made to solve a problem where you know what the solution is going to be. You kind of have a good idea of what you're trying to create when you write your code. But with scientific software you don't necessarily know what the result is. You're probably writing the software because you can't solve this problem analytically. You can't do it using normal mathematical techniques. There are scientific codes that are often developed over long periods of time. So there's a big code in my field again, I'll mention it. And it's been developed since the early 2000s. There's codes which go back to the 70s. Hence why there's still a lot of Fortran 77 floating around. And they can involve a lot of collaborations so it could be with hundreds of people across the world. And finally they're built by scientists not software engineers. So lots of people writing research software don't have any formal coding trainings. These are people who are self-taught so they're kind of maybe are not aware of proper coding practices. And I think this is one of the main reasons why you get kind of issues with software sustainability in science. Okay, so the scientific method. This is something that is used in the experimental science community. And it's the principle that you don't trust a result unless there is full testing of the apparatus that you've used in the experiment. And there's full documentation of the method that you have used in your experiment. And you do this to show that your results are accurate, they're reproducible and they're reliable. If someone is reading your paper you've done everything you can possibly do to convince them that they can trust the results that you have gained are accurate and probably true. In computational science you're essentially doing experiments but the computer is your apparatus. So to me I think that it means you should still follow the scientific method. And we really shouldn't be trusting codes unless they've been properly tested and properly documented. So this is a nice picture of an experimental in experimental sciences. So this is quite a well laid out lab. I expect she's probably following proper scientific principles to make sure that her experiment is as reproducible and reliable as possible. If we go to something that's more common in computational science from this webcomic from PhD comics. We've got like a million of different versions of a file. It's probably very disorganised. I doubt this person is using version control. So yeah, this is not great. Okay, so now I'm going to try and suggest a developer workflow that you can use when building research software to try and make it as easy as possible to implement some good software sustainability practices. So fortunately there are lots of tools out there that will help us automate things to really make it as easy as possible for us to do this stuff. Okay, so first version control. I expect almost everyone, if not everyone here, uses version control. But from a scientist's perspective I like to think of this as kind of a computational science version of a lab book. Because it kind of is a way to keep a log of all changes to your code. So kind of all the things you've tried out, maybe if they worked, maybe if they didn't work. So this is what lab books used to look like. Nice picture from Wikipedia I found here. And I think this is one of my repos. So this is kind of what I think lab books look like now. If we look at kind of commits, these are maybe not the best commit messages, but they kind of at least show different things. So they kind of show the changes of the code and how it's developed over time. Okay. So another thing about version control is it really aids collaboration. So you're not going to overwrite your collaborators changes. And it helps you hack without fear. So you can develop everything on a branch and there's no danger that you're going to irreversibly break everything. Which I definitely did many times before I was implementing proper version control practices. Okay, so testing. I've said that we shouldn't be trusting our results unless we've got proper testing. So we need to show that our apparatus and our method have really been demonstrated to work properly. And we need to make sure that we understand any limitations of our experiments. So in experimental science, we'll always measure the systematic errors and random errors and all these sorts of things. And we'll try and understand all the possible errors that could occur in an experiment. In computational science, kind of the equivalent of this is understanding where we could get numerical errors due to floating point computations. Or because we're using a low order algorithm in our simulations. So, well, this is a bit cut off here. But I expect most of you have probably heard about the LIGA experiment. So this was the gravitational wave detection experiment that you might have heard about last year. And when they published the results from this experiment, they published a whole series of papers giving really detailed information about this discovery. And one of the papers that they published was characterisation of transient noise and advanced LIGO relevant gravitational wave signal GW 151014. But in this paper, they really go through and they characterise all the possible sources of noise. So all the possible limitations there were in their experiment. And it's so careful. It's like there are so many pages in this and they have lovely plots really detailing all the possible sources of noise and error in our experiment. And this is just one out of like 10 papers they have on this. So we really should be doing something similar in computational science to understand our errors like that. So unfortunately, scientific codes can be quite hard to test because they're often very complex. We're solving some really difficult equations and they're investigating unknowns. We don't really have a benchmark quite often to test them against because we don't know what the solution looks like. But this does not mean that we can really excuse ourselves and give up and be like, ah, it's hard, can't test it. Okay, so there's a series of steps that we can go through to kind of do our testing. So first thing, we can break our code down into unit tests. So you can't really trust that your entire code works unless you know the individual parts work. It also makes testing your complex codes a lot more manageable. When you're building these tests, it's really important that you cover your entire parameter space and you also check that your code breaks when it does. So have a little example here. So I've got a very simple function here. So I'm just taking in some inputs and I'm squaring them. And then I'm using unit tests just to build a simple set of tests here. So my first test is just a very simple one. So this is maybe a common normal value that we all get when we pass, when we use this function. I've then got an edge case. So I put up a really large number in there to check that my code is still working. And then I'm checking that it breaks when it does. So if I pass in a string that it breaks properly. So if I run this, it tells me, ah, I've got an error. So what was my error? So if we look here, we can see on this line, the square of minus 2 is not 25. So change that, run it again. It works. Hooray. I can do maths. OK. So, sorry. If we go back. OK. So next step. So once you've checked the individual parts work, you then need to check that they all work together. So an example I like to think of is if someone had got all the parts for a car and they assured you that they tested them properly, they all work individually. And then they hand them to you and ask you to build a car. I expect most people in this room have never built a car from scratch. So if they try to put all the pieces together, the car is probably not going to work properly. So just because you've got unit tests does not mean that your code will actually work when all these parts are interacting together. Unfortunately, testing is a lot more difficult here because your code is a lot more complex. But this is a really important step. And then finally, you need to monitor the development of your code with regression tests. So regression tests are where you get individual versions of your code and you compare the results against each other. So ideally, the performance of your code should always be improving of time or at least not getting worse. If it is getting worse, then this is a really good indication that a bug has developed. This is a good way to catch bugs where they don't completely break your code but they kind of introduce some more subtle errors. A good bonus here is if you have lots of people using your code while you're developing it, then this can help you enforce backwards compatibility for your users, which is always a nice plus. Okay, so some science-specific issues with testing. So I said that you can have unknown behaviours. You don't know what your results are going to look like. So a good thing that we can do here, again with inspiration from experimental science, is we can use controls. So a control is where you put in really simple input data into your code and you check that it's still producing the correct solution. So if you think a really simple example that we can do is if we have a balance of scales and you want to check that your scales read zero when you have nothing put on them. If they don't read zero, then you have some kind of systematic error. So you have kind of some offset in your scales. So we can do this with code as well. So I quite often build simulations where systems evolve with time and there's quite often a solution where the systems just stay static and nothing should change. And if I put in the input data and it does start changing, then that's a really good indication I've got some errors there. So for lots of simulations in particular, this is something that we can do. Another big problem that we can get in science codes is there can be some randomness. So we could be processing some data with some random noise in it. We could be looking at something where it's going to be a chaotic system. So stuff that we can do here is we can try and isolate the random parts. So we can use unit tests to check all the other parts work and then isolate and check the random parts individually. Also we can do things kind of by looking back at the equations we're solving themselves. We can use this to test averages and we can check the limits and in particular in physics we can check conservation of physical quantities. So another example for this. Okay, so here I'm just going to define an array of some random numbers and I'm then going to apply a function to it. So I'm squaring it and then multiplying it by the sign. So if we plot that, then we can see we get some random stuff out. If I run it again, it changes. So we've got some random data. So if we look at this function, we can put some constraints on what we think the answer is. The answer is going to change every time. So we can't check that all the numbers in this array are equal to another array. But we can check. So we know that the input is between zero and one. So we can check that the output is between zero and sign of one. We can also check that the mean value here is equal to around 0.223. So if we run this, then hooray, it works. So I've just tested that we've got the limits we expect and I've checked that the average is what we expect. So that's the way we can deal with randomness, well, a way that we can deal with randomness in our code. So for simulations, what we often do is we take something that's continuous and we discretise it. So we solve it instead of solving it at infinitesimally small points, we solve it on a grid. But this leads to errors in our solution. But it means if you make the grid finer and finer, you should be getting a more and more accurate solution. And the accuracy of this solution should improve the order of the algorithm that you've used to solve those equations. And if it's not, then it's a good indication that your algorithm has not been implemented correctly. So if we look at another example of this, so I'm going to use something called the Trapezium Rule to solve this integral. So you might remember from secondary school maths, Trapezium Rule is where you divide up the area under an integral into a load of Trapeziums and then you sum their areas together. So I'm going to solve this integral here on an array of different resolutions. So I'm just going to iterate over that, and then I'm going to calculate the error compared to my known solution. So I'll run that. And then by then, here I'm now going to plot the errors and see what they look like. So here, so the errors are the blue points here. And what I've plotted in this red line here is I've plotted the resolution squared. So I know that the Trapezium Rule is an algorithm which is order H squared accurate. So because my errors have the same slope as H squared, this is a really good indication that my algorithm has been implemented correctly. So yes, we can also have issues with numerical error. So because we're doing floating point computations, because we're trying to approximate some exact maths using non-exact maths, it's likely that we'll get errors building up. And so if we use simple equalities in our testing, then it's probably not going to work very well. So we can do things here by kind of doing approximate equalities. So functions like non-ply is close and non-ply all close for arrays. Okay, so if you've built a load of tests and stuff, and that's great, but you need to make sure that you run them frequently to make sure that your code continues to be well tested. And we can do this by using tools such as TraverseCI and CircleCI. And there are many others out there. These are just a couple that I quite like for Python. And so these continuous integration tools will regularly run your tests for you and they'll report back your results, which is really great. So here's a little example for my code. So just very simply, I wrote a little script telling it what I wanted to run. I said I wanted to run these Python versions. And then it's run on a virtual machine for me. And it's told me right at the very end that all my tests passed. So really great. I can set up these tests and pretty much forget about them. And every time I push changes to GitHub, these will be executed. So that's really great. So continuous integration is really good because we can find out when bugs occur much sooner. And therefore we can fix them when they occur rather than waiting months down the line when the bugs are likely to be much more entrenched in our code. So this is important because, oh yeah, so there's a danger here. So I said we can pretty much set up the continuous integration and then forget about it. But if you're developing your code, you're probably going to add new functionality to your code all the time. And outdated tests are pretty much as useless as no tests because they're telling you whether your code is correct compared to an old benchmark. And so this is not great. And yeah, so if your test only covered 20% of your code, there is no guarantee the other 80% is going to work. And so a tool that we can use to really check that we are testing all of our code and not just a small fraction of it is code coverage tools. So one that I quite like is CodeCov.io. So here is what CodeCov.io looks like for my code here. So if we look into one of my functions, we can see it highlights all the lines that have been executed during the tests. And it shows me here that there's a little branch in my tri-except block here, which has not been executed in testing. So this is important because this exception was probably written for an edge case. And edge cases tend to be where your code breaks most often. So this is a really good way for me to find out what tests I should write and how I should make sure that all of my code is being tested properly. OK. OK. So documentation. So we need to make sure that someone else in our field can set up our code and run it without us having to physically sit down next to them and explain it for them. Because this is a massive barrier if you have new people entering the field. Because it means they're not going to be able to run, like, get started. And it's from experience, I can say, it's like quite alienating. So we really need to make sure we include comprehensive installation instructions. Not just instructions for a couple of supercomputer systems, but instructions that will run on standard Linux systems. So anyone with the right hardware can run your code. We need to make sure... So the code itself should be documented. So we use sensible function names, variable names, comments. So we have self-documenting code. So this means that if you've got a really complex code, people can look at the code itself and understand without having to read through pages and pages of documentation. A really good thing in science codes is having user guides with examples which demonstrate usage. So something that I found for my Python codes that's really great for this is having Jupyter notebooks. So something that I did when I wrote my code I've been using as a demonstration here. We wrote Jupyter notebooks which kind of explain what the physics behind the code is. And then we then run the code and we show what results should look like from our code. And then you can then actually use these in read-the-docs documentations. You can use MB Sphinx to convert this into Markdown and then you can then render it in read-the-docs. And so yeah, this is a really great way to kind of build up user guides with example cases for your users. Okay. So finally, distribution. So I said that at the beginning you might remember that we need to make sure that our code is still, we can still find our code in 5, 10, 20 years time. So first step to making your code findable is making it open source in the first place. So there's a little caveat here. I understand if you're working in a sensitive environment, so maybe if you're working in the medicine industry or something like that, then you might have proprietary information that you can't share. But if you're modelling black holes you're probably not going to get rich of black hole code. So it doesn't really make much sense to me for this code not to be open source. You then need to make sure that it will always be able to be found. So having a DOI, so digital object identifier, to provide a permanent link to your code which won't disappear with time. We also need to make sure that we can reproduce the runtime environment for our code. So this is some stuff like making sure that when people run your code they have the right libraries installed and the right versions of the library. So if your code only runs in 2.7 you need to make sure that people are aware of that. So an easy way that we can do this for our users is by packaging our code. So packaging it in things like docker containers, producing condor environments and things like this. Basically your installation should be as painless as possible. So it shouldn't be that people have to write lines and lines in the command line. They should just be able to write make and everything builds. And also it's really great to try and limit reliance on open source libraries and material because this again creates a barrier for your users. So it's not really very good practice. And also you have no idea that this non-open source material is still going to be there in five years' time. So to conclude we need to make sure that our software is future proof because if it's not future proof then this is going to harm scientific progress. We need to apply the scientific method to software development and computational science is the science so we should really treat it like it is. Codes should only be entrusted if they've been demonstrated to be reproducible so anyone in your field can go online download your code, install it run it themselves and reproduce your results. They need to test it to prove to you that any new results they get are actually to be trusted and you should document your code so that people will actually understand what it's doing and can potentially hack on it and develop it themselves. So if anything I've said here today is interesting to you then I really recommend checking out the software sustainability institute website because they have loads of guides up there which really cover all of this stuff in a lot more detail. So thank you for listening. Thank you Alice. So we have time for a few questions if they are interested in asking something. Thank you for the talk. It was very interesting. I'm worried how do you convince people to adopt this especially older generations because for example my advisor doesn't care of sharing the code or doesn't even care of reading my github logs so how have you been you say you were lucky that your advisor somehow pushed you here but how has been your experience with other people? Yeah that is like a really challenging thing so I think just nagging them and really just trying to put pressure on them I mean it's very hard definitely as a less senior person to try and encourage the more senior people that these practices are good but putting pressure maybe on things like journals to make sure they have higher standards so that they only accept codes that maybe open source which have proper testing to actually show that the codes are working properly but I think awareness as well because I think a lot of these people because they're self taught and they've always done it this way they're just not aware that such things are really bad practice but yeah it's definitely challenging to convince maybe more senior people that they should change their ways. Any other questions? Hi, thank you for the talk. How much overhead do you think it will add to your project this approach? I mean in terms of time? I mean for me I feel like actually doing this sort of thing saves me time because by having tests I can catch the bugs early on and so I mean in my master's project for example I hadn't heard a version control or anything like this and I was working on a code and for months and months it didn't work and two weeks before I had to hand in my thesis I found that I had hard coded a single value when testing and if I had some kind of testing or version control in there I would have caught that and I would have saved so many months so I think there's an initial outlay of time definitely but I feel in the long run this will save time. What would be your advice for sharing code or things like that on articles on research articles? For research articles when you write a paper how would you share code or would you put a link to the project or share snippets or what would be your advice? Definitely having a DOI to make sure that your codes it will always be able to be found in the future. I think that's definitely because having a GitHub repository is great but we shouldn't really rely on the fact that GitHub is still going to be there in 20 years time. One last question. Congratulations very good presentation and I would like to know about sometimes when I read the same papers the computer scientist don't share the code and the data but the paper using a good previous for example but it's not possible to reproduce the code and the paper and have any place to put the data and the authority and the source code is in GitHub but about the data do you know anything about this? Sorry what was the question? Do you know where it's possible to put the data and the results about the paper because sometimes don't read in the paper and it's not possible to reproduce the research on the paper for example? Of the top of my head I can't remember I'm afraid I don't really produce vast quantities of data I know there are definitely I can definitely look it up if you ask me and there are definitely websites out there for scientists to dump their data on and make it open source I know that ACM is starting an effort to try to reproduce the code for the papers so you can get a batch if you submit the code in a working style but it's not generally used yet so hopefully that will I think it's definitely starting to improve so thank you again Alice