 Yes, so I'm Simon Ramford from the University of Birmingham, but today I'm talking as an EasyBuild maintainer about our plans for EasyBuild 5 and how that will impact the community and how the community contribute. This talk will cover changes, policy changes, yes we're looking, well policy changes might imply in some places that we had policies this might actually be a good indication we're going to write something down so we can work with them. It's also going to give you background to why we're making decisions and changes and like I said tell you how you can contribute. What you'll also see bottom right hand corner as I've nicely highlighted here, there'll be an information box or information boxes. Yes as I've said at the bottom right hand corner there'll be a box, this one's nicely labeled information, another slide it will be labeled other things such as, sorry I'm giving myself back here, okay I'm sorry yeah red one breaking changes that will be a change to how EasyBuild works. These are ones that will like well might impact you, they might not impact you at all and which features you're using and not using. I will get to a few of those enhancements, a new feature those are entirely designed to make using EasyBuild better or easier and the bottom one, green one's the code cleanups, code enhanced or changes and you shouldn't see any impact on those well maybe not quite, if you contribute it might make your contributions easier and it might make EasyBuild run faster if we clean up a bit of code in places. Status types, so we've got some things that we have ideas about when we, sorry I saw a message pop up on the screen, those will be things where we're desiring feedback, we will talk a bit more about or I can talk a bit more about how we want feedback, there's plan changes, these are things where we already know what we want to do, we just need to get on and do it and the really good one is I've already been working on some things and there are some things that've already been implemented, that's available for testing and I have built something with EasyBuild 5, the development branch thereof. As Kenneth mentioned, EasyBuild major versions, 2012 to version one, you can see about two and a half, three years between versions, so we do look a little bit overdue for a major version and we've had a new logo in there but Kenneth will talk about that. So general information, yes there will be breaking changes, we're going to try and minimize them as much as possible but yes some things will break, hopefully not too many. We're going to give you as a deprecation warning so we've already been deprecating things that we're looking to remove for EasyBuild 5 and similarly some of the changes we will make will be, this will be deprecated in EasyBuild 6 to give you time that by the time we get there and you can probably guess that's another three years into the future, you'll need to have made a change. So most of the things that are going to be breaking changes in EasyBuild 5 are going to be things you will be warning you about and there will be documentation, we've already started to draft PR with what we are changing or what we have already changed, there's not very much detail I don't think in it yet, Kenneth shaking his head, I haven't contributed yet to that, it's on my to-do list but I had to write this talk instead. So some things change default configuration, we're going to enable trace by output, sorry by default, this is for the output from EasyBuild, what we sort of talked about as maintainers is nearly every maintainer I think has this enabled by default and realised that whenever we try and debug some of the ounces, who isn't using it, we always look there, look a little bit confused as to what the output is, why it's not getting so much information. There's some changes to Python package installs which is basically some things that you'll now have in Specify and EasyCon, things you'll be able to take out now if they're the default and so the default will change the other way round. Another one is we'll run the prepare step before the source step, this will allow you to use some forms of build tools or something similar to build tools to do unpacking, this will be fairly much for non-standard, so you'll still need a tar untara, probably an unzipper but say some like something to unraw that probably be able to supply that via EasyBuild instead. There is sort of bootstrapping thing there, you have to have at least one method of unpacking to be able to unpack the first few minutes of build and we're changing to slurm as the default end job engine and one that's you know a definite enhancement, don't run the sample checks in the installation directory, make sure they're running a completely separate directory so you don't leave a bunch of artifacts in there. Code clean up, we've already, this is one of the implemented features, the support for YAML easy config, YAML based easy configs has been removed, it hadn't had any updates, no real use, nobody asking for features for it, nobody looking to implement features, so we've removed that and the other one is because our item is moving distutils at some point in the future, we are removing, we're obviously having to remove that use directly and so for the loose version we've moved that inside EasyBuild, the relevant bit, for other bits of distutils we still need to do exactly what we're using and what we need to change, so this one that Kenneth has already spoiled, Python 2, this is entirely for what Python is used to run EasyBuild itself, as Kenneth said and this is based off the user survey, Python 2 use has gone down, Python 2, the route, how problematic, Rival has gone down, we deprecated early this year, we've been making warning noises for longer than that, that we would like to get rid of it, there, also what versions of Python 3, we worked out from the user survey that it seems like nobody's using Python 3.5, we've been at the end of life for two and a half years, the problematic one obviously is Python 3.6, it's already the end of life for a year, but mainly we would say because we've seen quite a lot of Red Hat base systems, that's the default at the moment for released version 8, so version 8 also sort of Red Hat base systems, 53% of our users are on that, so this is, next slide is probably not going to be too much of a surprise, we're going to drop for Python 2 and 3.5, so we will be able to use features that only in Python 3.6, all greater, this will allow us codes identification, in fact this is one of the things I already implemented and it does, no more indirect use of this Python 2 versus 3, at least in the central repo, it makes things a lot easier, you now know exactly what Python feature you're actually calling rather than this indirect method of not really understanding why ED build has this feature, get the actual Python library, you will not be able to use Python 2 as a valid Python command considered by ED and because of what is used by various systems, that Python is often still actually Python 2, we're going to think that Python 3 is the command over Python, that's particularly relevant for that large chunk of central 7 users where Python 2 would be what is actually labeled as Python or what you get with Python, this is where we start going into an idea and we're trying to work out exactly what we want to do, it has some impact from what Kenneth talked about from a number of open PRs, it's mainly about easy configs but I'm going to talk about it in terms of what tool chains we're supporting centrally, one of the things that we thought to mention but we haven't, we never really say exactly what it means if we provide the central repository of easy configs as a sort of a reference library, we're not intending it to be the one-stop shop for every easy config that exists, we're expecting sites to build on top of it, there's quite a, that link there tell you one of the ways of setting it up and one of the things we're sort of going to look to going towards is trying to go back to what we talked about every now and again is that inside a tool chain trying to keep one version of a piece of software in there not have multiple of the same software in a tool chain but I do say there will be exceptions but we're probably going to try and make them more exceptional than we are at the moment there. What we're also considering and this is one of the bits definitely has the idea flag on it is that for the central repository we will only accept PRs for recent tool chains. I'm using an example here based on 2023 and I'm the first suggestion we have is that it will be two years back so if 2023 we'll go back for 2022 and 2021 tool chains we will deprecate one further back than that so 2020 tool chains both the A and B and anything older will get archived which will then be the older software that we haven't got to merging PRs from we would then just start closing at that point. One of the reasons we're going through this is while we do use semantic versioning for easy build versions the tool chains are very much a gear based concept and we're sort of separating out the two that we're going to deprecate tool chains in some way that is based on the year rather than waiting for a major version because as we've seen major versions three years apart maybe four years apart we don't really want to force out a major version just to deprecate tool chain. So there however we inside the maintenance we had a discussion about this have at least two alternatives if not more possibly and you'll see I already talked about maybe basically on 2023 but actually there's no 2023 tool chain out yet and we haven't really started one I'm really considered it yet so maybe it will be referred to as the last however many common tool chains so maybe six maybe seven you know numbers at the moment are partly mainly where the idea is here we haven't quite decided or maybe that we'll go one more back on the year so it will be 2023 minus three so back 2022 is being the last one we're supporting and then like I said yeah we'll then close we'll also then discourage PR back porting new versions of software into older tool chains particularly when all that I need to do is just change the what tool chains mentioned and dependencies. What we might do with those things like this is encourage it still to be a PR but to PR they get closed quite quickly with a we're not going to merge this but we'll you know it's useful for it to be here because some other people I wanted there and I'll sort of mention that a few times is it's to do with balancing the maintainer effort and how much time we are taking and making the process quicker which is something we're sort of conscious of which will help us then merge more particularly if you see that you know all those graphs of how many software packages that we were talking about earlier that there's just more and more and more. The next idea and this talk about module tools kind of touched on this quickly. We've deprecated in easy build four our mod six and six is deprecated might be seven as well no did just six. I couldn't quickly work out when six was last released any any six versions last release or when it was first released I got too lost looking through release loads trying to work it out but I can tell you seven was last released in 2019 so we can probably guess that six was a long time before that. If you're on sort of a red hat base then Apple has 8.7 something for all of those all the supported versions 7.8 and 9 however Ubuntu doesn't. We're just having a discussion in the room because yes we're hoping that this will get solved at some point you know there are packages that can be installed I think but they're not there by default and we will put this out as an idea but one of the things we're not quite sure about is we're not quite sure there's no feature reason for us to drop support for 6.7 it's just motivating community to use a newer version that generally works a bit better than the older versions it's generally a bit faster there's a couple of new features that are quite nice to have if you maybe want to use them but they're not required and we work around them so we're not quite sure we're going to discuss and open that one up for you know the community to talk about. In things that are planned we are going to change the run function which is something that sits behind how easy build actually does the installation and having over time these have just basically you know anything like this it was nice there was a reason for it but then somebody goes I need to do this extra thing and bolt something on and then they you know something else gets bolted on and then after a while you work out that the default thing that everything's not doing looks nothing like what it first did so everything seems to be an exception to how you're trying to use this function and it always seems backwards to what you expect it to be so the aim is we're going to push out write a new one that hopefully works better hopefully works more directly hopefully allows us better access to error logs which I'll come to and talk to a bit more but the aim is here that you'll still be able to use the old versions but that will deprecate them in the easy build five so by six everybody will need to have moved over to you new one but there'll be some method that makes it not painful when you get to easy build five if you're using it directly sorry so who we want to call run cmd a 20 headed monster it might well be the case of calling a 20 headed monster I know I've tried to debug a debug how to well I've tried to add better error reporting to easy build or better error passing of the logs and I got so lost as to where I needed to add it into the mix that I had to stop and go well actually I need to revert back and start again so yeah something that's better and the more of us understand more of us can use all more of us can edit sorry and that's you know back to maintain the rest but if nobody really understands it who can actually edit the code as I said error reporting and logs these are sort of improving the log files is actually a function of probably we want better error reporting because you're probably only looking yet in the log files because the error report is not always great unfortunately builds will fail I wish they didn't but you know that if we could solve that that would actually help it but that's what sometimes it looks like trying to chase down errors um untangling the tape um so we are we have definitely have planned here definitely have plans is make it more obvious what the log files were make it more obvious what command was being run when something went wrong um and hopefully build in and this was the bit I was trying to get to ago it's context of where reporting that you know if you're doing configure make it will go and look in the configure log if um figure dot log if it fails and also pattern matching because there are tools out there that already do some of this pattern matching and we just need to hook them in better or hook them in so yeah john so john has raised the that's just uh basically that's just upload the report yeah we're considering that um we want to balance yeah well it saves a step the question becomes who's going to debug it you know we don't want every single report uploaded because there'll just be so much noise that we never look at them so it's that balance between making sure we it's only done at a certain point and that's one of the questions we have is that balance as to when and I'm far raising yeah so Bart's talking about the if it fails then we want to be in the shell so I've sort of left this out of this slide and actually the previous slide it was in our my earlier notes but yes the being able to drop into a shell if it fails or being dropped into the shell and when it fails yes it's a very good idea I'm not sure how we get that to work and I don't want to be promising it in terms of edbuild 5 but I would definitely love it but I would let you know need to find somebody who can work out how we would implement that yeah um but Kenneth's comment was yes we're going to you know build that as a possibility you know it's if it's in the port when we're designing in front of function and hopefully it will be possible to work out how we do that or how we got that functionality in um idea um and this one this is something that sort of we seem to go back and forth with about us to going in different directions as to whether we want to add more and more stuff to the base art and uh by an easy configs or whether we want to make them as bare as possible and put everything into a different easy config and different sites I now have different solutions for this um um why we will talk with um community one of the reasons we were considering the bear is having this uh bear as the base is particularly for python and actually I haven't mentioned pearl on here but probably there is we have them as build dependencies in several places and having this not having a separate build dependency for pearl and python as the version that you then actually use so you just reuse it at some point um but there are bits there as to you know if we are going to do this how do we decide particularly if we are which is now over a thousand packages in the um central repository may not easy helping who gets to decide how we split them up or who is going to decide how we split them up um but probably and one thing well one thing I want from it if we do do it is the how do we uh I want one package that still looks like the old one but that's just my lazy list because I just want to tell my users just going to lay back now um easy configs there's a few um ideas for sort of improvements when specifying the toolchain we currently specify name and that I will specify the name that it's false and version that is there we're proposing or I'm proposing at least a simpler option of just having some sort of people of well you just said false and 2022 a you know you don't need to say that the first thing is the name the second thing is version because everybody writes them in that order as far as I know and and particularly also is we can probably then add the toolchain ops in there as a dictionary or something similar for the times there um there this one I definitely want um and there partly because whenever somebody comes up this is the when you have a python module and it but it's let me eat sanity check for import is it might be called well PyTorch you know PyTorch is the module name but the you import it as Torch um I can never remember that syntax and whenever anybody asks how do I do this I have to go and you know search the repository find an example and then go you need to do this um so something simpler and more direct for that um there are some other options and ideas one is to make I'm going back at the top as I got on the top module class not required so you can decide whether you actually want to use module classes um this is probably coming from me because I know my site will be completely ignore module classes um because about 50 percent of our easy configs are by in bio um so it's not a great useful category to us so um you know there another thing and we think we should be getting better at is having easy config specify what life sport software license is like you know is a gpl or whatever else and one version and we think that would be quite useful particularly then also so sites could decide if sites are this way minded that I'm not going to install anything isn't a certain type of license or certain permissiveness of a license um and also potentially a method to specify conflicts um we see this sometimes with components that you can have two modules that actually should be saying that they're conflicting because they provide different versions of a default library but actually because of the way that's hidden inside somewhere it doesn't tell you quite that they uh you know you can actually load them at the same time and then not know which version of that library you're actually going to get work or not work or both of them can't work um the documentation site can as mentioned this isn't really easy build five directly but just that you know we've looked it done that look about the yearly review and the integration of the tutorial into it so make it a bit more joined up so how can you contribute we're going we're forming a working group and this is for people who want to get involved with the coding the reviewing the testing writing documentation there is a slack channel for it in our slack e5 um we're going to be having monthly meetings first wednesday of the month at two central the european time nine my time zone so i need to translate it already and those are obviously the first four meetings and more if more depending on the progressers to help while we're doing progress and timeline we are aiming to this year for a release um we'll see how there there is a github project board for it um you'll find get easy build five labels and milestones across the repositories they're already there that as i said the five five dot zero dot x branches are there you can test from them i have used them to install many stuff i've done the language don't have a look and see what's being done um as i said we will be plating feedback on ideas it's likely to be in the form of these ideas that i've raised so i'll specifically ask don't try and talk to me during this easy build use the meeting please when we ask for them when i've sort of coherently done it add the feedback then this is mainly because if you talk to me in the next couple of days while i'm busy organizing i will i will remember that somebody talked to me about it but i will have forgotten what you've said um so what what will happen is one of the maintainers will post on uh one of the github repositories we haven't quite decided which one an issue probably a github issue with this is the proposal we'll then notify you from the mailing list and the jones channel in slack and say for the next probably four weeks this proposal's open please go and discuss this as to where you think we should go for these things so there'll be one for the toolchain's poll policy there'll be one about how mods there'll be another some others and we need to decide what we're going to you know which one we'll write up but we will let people know and then ask for feedback um one observation i do have the survey is our main way of knowing what our users are doing or how they're using easy build so hopefully it's accurate hopefully is a good snapshot but please do fill it in because that's how we know what system we're trying to support um there and questions and that's where that's a list of where all my pictures have come from not very many questions though because it's nearly time for the next talk Casper has a question I think we're going to make it a clearer policy because I think it's there's only the maintainers this is about what Kenneth talked about so the question was about over one version of the toolchain so Kenneth mentioned things like there have been 15 new contributors this year I know it is maintained you know it's maintained and probably several of our long-term contributors know it but I'm not sure we've written it down anywhere in the documentation um and so there are things like that yeah where we want to make sure it's clear there's a policy and why you know rights and justification as to why and explain it so we can go this is why we're doing this and this is the reasoning and then let people know where to get their efforts and things like that