 Okay, welcome everyone to this special session on on easy build. Yesterday in November 13 was exactly 10 years ago that we published easy build 1.0. And I didn't want to let this pass without doing something special. So here we go. So the purpose for this talk is to look back a bit at the previous decade. Where do we come from? How did easy build actually came to be? I'll go through a couple of major easy build features that you should know about and hopefully know about already. And if not, you really should look into them a bit about the easy build community and the lessons we've learned throughout the years. And then also looking forward. So what are we currently working on? What is coming really soon. And also some exciting opportunities that we see ahead for easy build and related projects. And then at the end, it'll be a nice surprise as well. So, over the decade of easy build is actually over the decade because if we look before easy build 1.0, it was already there. The first public release of easy build was 0.5 in April 2012. And before that, there was a lot of in-house development. 10 years is a long time. And over those 10 years we've gone through different eras or different phases. So the first part was definitely the in-house development where nobody outside of our team and the Fleming Schubert Computing Center knew about easy build. At some point we did the public release 2012. We started getting first users outside of our team, getting lots of feedback, doing some promotion. That kickstarted the easy build community. So that's the third era. And then we matured the project a bit and had a big spike in the number of supported software packages. In 3.0, we added support for specific features like R-path and hooks. And that was also the start of the easy build maintainers team. And then easy build 4.x, what we're currently playing with is the first. The first easy build version was compatible with Python 3. It was easy build 4.0. And from then on, lots of enhancements and software updates have been added, of course. So that's a lot of releases were getting close to having 100 easy build releases quite soon. Maybe in 2023 we will hit that. So that's a lot of work that has gone into this. Easy build started in-house mainly to scratch our own itch. So to deal with the problems that we were having as a new HPC team. So the team was really started in 2008. And our first central supercomputer at Ghent University was installed in 2009. And pretty quickly it was clear that dealing with incoming software installation requests was going to be a big problem. The manpower back then was very limited. It was two technical people and a manager. And they had to do everything basically from the hardware problems to the system administration, OS updates. And also the user support and providing software installations to the researchers. That was going to be impossible if everything had to be done by hand, which is apparently lots of other teams were doing. And so we wanted to do or we had to automate that effort a bit. Now we didn't really knew what we were doing initially. We didn't have a lot of experience with software development, let alone publishing open source software. And we sort of learned things on the go. And yeah, you can definitely see that throughout the years. What made a big difference is that we had a couple of very good student interns who helped a lot with even redesigning easy build a couple of times. From scratch essentially just pulling things apart, putting it back together in a better way. We had students learning us about tests and why we need tests. We didn't have any tests at all. We figured we didn't need any because we knew the code base quite well. But they showed us that just by writing tests you can very quickly fix, find bugs and fix them. They also helped out with developing a couple of features and so on. So the original code base was so bad that the the pilot code style checker gave a negative score. So you get the gives you like a score from zero to 10 or something and if it's really, really bad it goes below zero. So initially the code was way below zero and the students helped us a lot by cleaning it up and improving it. But just for some context this was back with scientific scientific clinics five biting two dot four since it's a very long time ago. At some point, we started to wonder if this could be useful also to to other people. It looked like it was working for us pretty well. And we couldn't find solutions out there that other people were using so we figured maybe this could help others as well. So we wanted to make it available, mostly to get feedback, because we still felt that maybe we were missing something maybe other people were using this fantastic tool that solves all your problems when installing software. And they were not telling others about it. I don't know. And before we could do that we needed a logo so we we started playing around with some ideas, which you can see on the slide. And in the end we drew something on the whiteboard after having a couple of beers, which looked like a good idea and then literally this screenshot or this picture of the the logo on the whiteboard became our final logo. And it has served us well over the last decade. So then in 2012 you were finally ready. We were confident enough about the code base to make it available publicly so we did. We created a GitHub repository in March 2012. So switching away from our private SVN repository. And we quickly figured it was a good idea to split up things across for different repositories, as you know framework easy blocks easy conflicts, and also the documentation. We did a release or at least be tagged the version in our GitHub repository in April 2012. This was very, very early was like a, again a cleaned up version of our internal code base and all it supported was building HPL with the, which was then called to the go off to change so GCC open MPI Atlas, lay back and 50 W. So you can only build HPL with it nothing else. There was some documentation on the rekey page but we only copy pasted some stuff from our internal documentation it was not very user friendly. And there was not even an EP come out yet so there was a script that did some stuff but there was nothing as clean as an EP come out. So the screenshot shows how the repository look back then. It looks pretty empty everything is of course in the easy build directory. But it's very different from what it is today. So it was about 6000 lines of Python codes 23 easy config files and nine easy blocks that was it. So very, very lightweight first release. Then the main reason to push out that release is because we got a talk accepted at the Hapix spring workshop so Hapix is a high energy physics conference about easy build and of course we wanted to have some code to show to whoever was interested. I found the slides and I didn't actually do this talk this was a colleague back then Jens Timmerman, who gave the actual talk the slides are still available so here's some screenshots. We mentioned that we had support for over 250 software packages already back then, but we didn't actually include all that code and easy build five zero dot five yet. We were still cleaning things up and putting it in the public release. Pretty shortly after doing that we got some feedback on the easy build IRC channel, where people were saying or reinventing what gen two has done with prefix and yeah why are you actually doing that that doesn't make sense. So they didn't really understand what we were doing or why we were doing it the way we were doing it. But at least we were getting feedback so that that was good. We also made the easy build mailing list public and the first person to actually subscribe to it outside of the HPC again team or at least the Flemish HPC sites was focus was then I think working in Cyprus at the Cyprus Institute. So he noticed probably the talk at Hapix and was looking for a tool like this. And he was really the first person outside of our team who picked it up. The list has about 300 subscribers, but most of the activity in terms of discussion discussions and asking questions is actually going on on the easy build Slack, which has over 600 670 members so that's that's quite good. Then, still that same year. We wanted to make easy build stable in the sense that we froze the easy build framework API so if people were developing their own easy blocks for it. We could give them some kind of guarantee that things wouldn't break over and over again with every easy build release. So we gave some some certainty about it being stable. The easy build release 1.0 itself was done on November 13. And this was hours before it was going to be presented at the pie HPC workshop at supercomputing 2012 and the paper in the slides of that are still available. So this included support for 148 software packages about 21,000 lines of code. 76 easy blocks and 300 something easy conflicts. And the feedback we got from this was was pretty positive so people. Some people like the anti tarrell, where they show the tweets of saying that this is the best thing he'll take or the coolest thing he'll take home from supercomputing and that was a very nice feedback. And to be happy and we seem to have found a hole in the market where nobody else was working on a tool like this, which is good. And of course this this this motivated us to continue with the development and making sure that whatever we could share publicly was being shared public. We really started the easy build community so more and more people started picking up on easy builds and started to use it, not only use it but also actively helping out with fixing bugs or adding support for more software adding features and so on. We also started organizing hackathons where we wanted to brainstorm about easy builds like what should it be able to do how can we implement this. What should the focus be, and how to make it bigger make it make it make it support more software and so on. We shortly after we also set up the now well known easy build conference calls every two weeks we have an easy build conference call anyone is as welcome to join in. Usually discuss recent developments and usually we have some time for questions and providing some answers as well. It's very rare that we actually skip one of these calls so you can, you can rely on them being there if you need any help with easy builds. We also looked into other ways of promoting easy build a bit indirectly, so we proposed getting scientific software installed birds of a feather sessions and both supercomputing and ISC. And of course we wanted to give ourselves a platform to promote easy build a bit more and on a more regular basis, but we also looked at at other tools related so lmods was being discussed there as well. We also invited top gambling, when SPAC was created to talk about SPAC at the sessions and so on so we were very open to also promoting other tools and and making it very clear that we are not the only ones in this space. The rooms at FOSDEM the HPC dev rooms were initially created sort of to have an excuse to also be able to present easy build at FOSDEM. But these dev rooms have been going on pretty much continuously since 2014, and have been have been quite successful as well. Later, we started doing easy build user meetings so basically stepping away from the hackathon model and going more towards user meetings and related to those user meetings we also did user surveys on the yearly basis since 2017 to get a better, a better view on the easy build community so how people were using easy build what to define important on what type of systems they're using it and so on. And more recently we finally got a tutorial accepted at ISC we tried multiple times and and the COVID year 2020 was the first time we actually got the tutorial accepted. The tutorials were actually canceled at that year at ISC for for the due to the pandemic, and it only ended up being presented in 2021, but we did get our foot in the door there. And also this year we had a tutorial there and we hope to have up next year as well. Now one thing that's very difficult when doing open source software is to get a good view on how many people how many HPC sites are using it since everyone can just pull it down from the web and doesn't have to tell you that they're playing with it. You don't really know who's using it so the best way we've come up with is looking at the visits to the easy build documentation, where you can at least see from which cities, people are accessing the documentation and how frequently they're doing that. So the screenshot of the world map shows you the cities from where easy build was visited in the last year and how many times so the bigger the the sphere, the more it was visited. And currently we have over 1000 page views per week in the easy build documentation so that's definitely not bad it's getting lots of lots of reads. The logos at the bottom show you some of the bigger or more active sites that are using easy build today but it goes way beyond these universities and these institutes of course. So then back to the hackathons. People who picked up easy build early on also were very keen in getting involved with the development for this was definitely very, very active there in the early years of easy build and actively helped out with setting up these these events. The first one was actually organized in Ghent itself and the only outside visitor was for this he came in I think from Luxembourg. And yeah just wanted to discuss with us what easy build can currently do. And what should what we should work on next in terms of support and and software packages. You can also see Todd gamblin so on the picture on the top right. Todd also joined us for a couple of these hackathons and to exchange ideas with stuff he was working on in SPAC and things we were working on in easy build. And all of the notes and the slides from all of these hackathons are still available on the easy book wiki. Then of course we also continue to do promotion about easy builds. Whenever we got the chance and lots of other people did the same thing as well. Our Twitter account has been quite active over the years we now have 6166 followers and I didn't gain this in any way. There's also some interesting quotes on certain talks where George from the Cyprus Institute for example said that easy build saved his life so without it it will be very difficult to do the work he's been doing. We handed out stickers we handed out coffee mugs and we will keep doing that. When if we see the chance as well. I've been interviewed for a podcast. We had easy build cake at some point so when there was five years the five year celebration of easy build 1.0 was celebrated at supercomputing 2017 so somebody ordered cake with the easy build logo on it and that was a lots of fun to see. And the picture on FOSDEM so the middle sort of bottom middle is the first time that we had seven easy build maintainers in the same room. We actually have over 20 easy build maintainers today so maybe at some point we will do better than that. Then the user meetings this this kind of evolved from the hackathons where we saw a bit of a shift into what people wanted to discuss or wanted to talk about the hackathons. It was less of hands on development and more discussions and showing each other how they are using easy build or in what context and how it was configured and for what reason and so on. So this is what the user meetings still are today. These have actually been very well attended so the first couple of user meetings had 20, 30 attendees. But the last physical one in Barcelona early 2020 had over 50 people actively joining with some people flying in from the US or Australia so that's a, that's quite unexpected I would say and that's quite a good sign of the, of the maturity of the easy build community. What we've always done from the very start is at the very least record all these talks and also stream them and all of these, all of these talks are still available on the easy build YouTube channel so we hope we can keep this up. And at the very least make this hybrid events where if people can travel or don't have the time to make the trip, they can still follow along. Now of course that the previous two sessions or the previous two events were fully virtual through zoom and slack due to the ongoing pandemic, but we think it's, it's now safe enough again to have a physical easy build user meeting again next year. So we are going to move away from from wintertime in Europe, just to hopefully avoid some of the trouble. So we're currently looking at April May or something to organize the next meeting now. This is nothing is set in stone there. The date or the location is not fixed yet so we're currently looking into this but you'll hear about this soon. And of course hopefully will. We will break the record of attendees, at least at the physical events will see how that goes. And the major easy build features that were developed over the years. So all of these are are documented either in the easy build documentation itself or if not, they are covered in the easy build tutorials. Of course, lots of features were implemented over the years and easy build. A lot of these are driven by the community either by suggestions or questions by by people using easy build, or quite often also people who actually implemented them themselves and then sent us pull requests to get them integrated into easy build itself. There's lots of examples here. I won't discuss all of these. But if you're not aware of some of these things on the list, they're definitely worth looking into. One of the major features that was implemented by people outside of the HPC again team of support for our article module naming schemes. So the unique super competing center was a big driving force behind this, especially Marcus Geimer spent quite a bit of time on this. So the intent is to organize your modules in a different way that you have very short module names and you can only load modules which are compatible with each other. So as the picture shows, you have a GCC module and only if you load that GCC module additional modules become available that are compatible with this version of GCC. And then the same thing for the, for the MPI module if you load a particular version of open MPI, then more software becomes available that is linked to this MPI library. And now this was a lot of a lot of effort to get right because there's lots of tricky things to keep in mind here and communication with Elmod or putting things in a certain way so Elmod is happy. When consuming this module three so this was this was not easy to implement. But it was done in a way that you can actually define your own hierarchical module three if you want to so the classic one is has three levels as shown in the picture. A core level compiler level and MPI level, but you can make this three as deep as you want to with additional levels if you want to. As long as you're careful enough and the implementation of the naming scheme so all you basically do here is implement a small Python module that describes where the modules should go in the module three and then easily takes care of the rest. Then one of the key features and easy build is definitely the integration we have with GitHub. So it was actually implemented out of need we were starting to get too many contributions there was not enough structure in the contributions. Many people wanted to contribute but we're not familiar enough with kids or GitHub. And to some extent we still see this happening today. We felt we had to do something about this we wanted to avoid that people had to do all the, all the commands to rename easy config files put them in the right place start a branch. Push it to GitHub and then go ahead and get up and open the pull request in the right way, targeting the develop branch and putting the right information in the PR title or the PR description. So we ended up automating that whole process. So you can straight from the easy build command line open the pull request, and you don't have to give the file name, the correct name or something easy build can figure out how it should be named and how where it should be placed and so So this saves a lot of time both for the contributor but also for the, the easy build maintainers who review the pull request because everything is very raw structures. And we know there are certain things that we don't have to keep an eye out for like the naming of the easy config files the location, the target branch should be developed all of this is taken care of automatically. If the pull request is opened with with PR. So this was a big help. And next to new PR there are various related features as well like pulling easy complex files from a pull request or uploading a test report. When you're doing an installation from a pull request, updating pull requests. So there's lots of small variations or related actions here that you can, you can do straight from the easy build command line and it's really, it has been a game changer I think, in terms of getting contributions to a small thing but I think a very useful thing and maybe something we should even enable by default in the next major version of easy build is the trace output mode. So by default easy build will not tell you a lot of what's going on in the background, it will just give you lines like preparing or installing or building without actually showing you what's going on in the background. In the enable trace mode you will get a lot more detail, but without it flooding your screen with lots of access information. So when it does the repair step and when it's setting up the build environment it will show you which modules are being loaded for example, when it's doing actual installations it will show you which command is being run, and where the output of that is being stored so if you want to you can tail that log file and see what that command is actually doing and how it's progressing. You will get timestamps as well so you can check on things if something is taking too long. You can quickly spot them. So this is very, very useful. And I definitely strongly encourage you to configure easy build to always do this because, at least in my experience it doesn't get in the way it only helps. Another mode is the dry run mode and this is very different from trace because this will actually just look at how easy build would install something without actually doing it. So this will always run in a matter of seconds because it's not actually installing anything it will just show you the commands that would be run if easy build does this installation. You will get an answer as shown on the slide so it will show you during the build step method which type of commands or which commands will be run and in which location. And for example for the sanity check it will tell you what easy build will check for before declaring success on the installation. So this is very useful it opens up the black box that people were. Yeah, the people felt that easy build was to them so they didn't really know what was going on or what easy build was going to do. With the eb dash x dash dash dash x option. You can actually get that information straight away and very quickly. So this is I think a very good idea if you're wondering what easy build will do or how a change to an easy block or an easy config file will affect the installation it's very easy to tell if this will actually be picked up or not. Then another key feature, and this is this was implemented relatively recently is the support for customizing the functionality of easy build with hooks. So throughout the easy build code base we've, we do callbacks to hook Python functions. If they are defined, and in these Python functions which you can fully control yourself. You can do any additional actions you can run checks or you can run additional shell commands, you can even manipulate internal data structures of easy builds to make changes to what easy build will do by default. And so this is relatively easy to use and it's a very powerful feature, because you can really change anything almost to what easy build does by adding a small amount of Python code. So the example in the slide is shows you how we can customize the configure options for open API with a pre configure hook. So you're basically telling easy build if you're installing open API and you have with verbs in the list of configure options, then you should replace that with without verbs instead and then just continue with the installation as you would do normally. So yeah, this is a very easy way to customize easy build and keep track of, of things in your own version control setup. The slides are really using this quite extensively. There's a presentation at the last easy build user meeting by the University of Brussels that show how they build open API and all the MPI libraries for their heterogeneous setup where they have a mix of different interconnects in their system. So they do lots of detection and tweaking of configure options there through their hooks. Perhaps the most recent feature, big feature worth mentioning is the richer outputs that easy build can give you. So for this we leverage the rich Python package. It's not a required dependency but if it is installed, easy build will use it to give more colorful outputs or to give progress bars, whenever it's downloading files or running an installation. When it's installing extensions which could be over a thousand, for example, for recent R easy conflicts, it will tell you how many extensions are done, how many are still left, how much time has been spent up until now on the installation and so on. This gives you a concise and a colorful overview of how easy build is progressing. This is also something I strongly recommend. So all you need to do is to install the rich Python package with the same Python version as easy build is using. And then this will be enabled automatically. Okay, so next to all the features that we have implemented over the years, a big part of easy build is of course the software that is supported by easy build so what you can install with easy build. This has been increasing over the years, of course, we even see it increasing more quickly in the last couple of years so we're adding about 400 extra software applications and libraries per year to easy build so what it can support. This does not include version so this is unique different software projects. And this also doesn't include what we call extensions so things like Python packages are libraries are usually installed as extensions to something else. So those are not included in these counts. So very soon we're going to break the threshold of over 3000 supported software packages. And if you would count extensions you'll have to add over 2000 to that counts we're going to break the 5000 mark there in terms of different support and software projects. Now there's a small decline there in the graph. So this is when we archived very old easy configs in easy build 4.0, which we're only using very old tool chains and this is typically that software that nobody is interested in anymore so that's why we have a small decline there. We're probably going to have a decline like this again in the next major version of easy build. So overall this is a, this is good. So we keep adding support for more software to easy build, but it's also scary because this trend is not slowing down so it seems like there's a bottomless bit of research software that will have to keep adding a new new packages keep coming up from new domains things like bioinformatics machine learning AI so so new projects are being created probably every day. And I don't expect this effort to slow down anytime soon. Now all of this software support doesn't come automatically of course this is this comes in through the contributions we get from the easy build community and all the contributors that do a lot of hard work. In opening pull requests and making sure those installations work and go through the fairly strict review process that we have before we include something in easy build so we're quite extensively testing contributions before we can actually merge them in. So we're currently seeing about two and a half thousand easy conflict pull requests per year. So if you think about this per working day or something that's over 10 pull requests per working day which is quite a lot. And that's beyond that's without the framework and easy block PRs that often go along with it as well. In the last decade, we have seen over 25,000 pull requests being opened to easy build so that's that's quite big. The requests were opened by over 375 different people over the years and over the different repositories and per year we're currently seeing well over 100 unique contributors so 100 different people in. 2020 2021 and also this year we've already broken our previous record in terms of unique people so we're probably going to have over 120 unique contributors this year. Now, there was a big impact here by the GitHub integration features so the support for opening pull requests from the easy build command line. This support was implemented because we were getting too many pull requests and there were not structured well enough, but we're probably shot ourselves in the foot there a bit by adding support to make it very easy to open pull requests we've been getting way more way more pull requests than we were ever before so you can see the support was implemented in 2016. And since then it has been increasing quite a bit. This is mostly okay because the pull requests have way better structure and also for the easy build maintainers it's way easier to process these pull requests so even though we're getting a lot more. We're still quite happy with the GitHub integration that we have. And all these contributions are reviewed by easy build maintainers. So in 2016, we were starting to get too many contributions for just the HPC again team to handle. So we started asking for help by experienced easy build. And people who are very active in helping with already maybe reviewing pull requests or testing pull requests who started to reach out to some people and see if they will be up for helping out with processing contributions on a voluntary basis. So to be clear, nobody is really being paid to do this so they're doing it as a part of their job. And they're just trying to find some time left and right some hours in the week to help out with reviewing pull requests and getting them tested and merged. We also have a rotating what we call maintainer of the week role. We actually have two. So two maintainers are responsible for looking into incoming issues and pull requests that week. And this is also on the best effort basis. So the guideline is sort of to try to spend one hour per day that week to look at stuff that's that's incoming. And whether they hit that hour or not or spend more time that's absolutely fine. So they just do what they can. Last year we also had the first easy build maintainers summit so we're all or as much as possible easy build maintainers were getting together to discuss on the short term which changes should be implemented in easy build how we can more efficiently process contributions and so on. We haven't planned the next maintainer summit yet what we probably should sometime in the coming weeks or months. So the easy build maintainers have been very, very important to easy build without them. There's absolutely no way that easy build would be would be in the current state that it is. So whenever you get the chance to thank an easy build maintainer. If you run into them if you know them. Please buy them a coffee please buy them a beer, or or thank them in some other way so whenever you get the chance to do so please do. Then there's a couple of things that that we are certainly I have learned over the years like I mentioned we have, we had pretty much zero experience with releasing software in the wild or doing open source software development. So these are a couple of things that come to mind that we've learned over the years so first of all and I've mentioned this multiple times. So if you want to release open source software. You think that's a good idea. Maybe it is but be careful because if you start a community then things will explode quite a bit. And you'll have to somehow deal with that and there's. You can find multiple stories by people that have burned out because they, they released some software out in the wild and it became somewhat popular so you may want to think about what you're going to do if that happens. We're not prepared with this but somehow we found a way to deal with it by asking for help in time, implementing features that make it easier to deal with income income and contributions, and so on so we're still quite busy. But I think we're more or less keeping up, and we're spreading the load well enough that we can keep doing that. In terms of promoting your project, the, the best way, the absolute best way to do so is word of mouth so if you would like to get others to use your, your, your project give talks about it at every possible opportunity and different ways, podcasts, classical presentations, giving demos, so everything you can do to show people the value of your work is something you will have to do. So even then, even if you're doing this for over a decade like we have. We still run into people every now and then which are in the HPC or the computational science community, who have never heard about your tool before this this will keep happening, no matter how many talks you give or how many promotion you do. So people will still be surprised and still tell you why didn't I know about this even though you've been screaming from the rooftops for, for 10 years. I've heard about your project it doesn't mean they've actually tried it or they are convinced to really spend the time to start learning to use it so people really have to be convinced before they start using your project and the best way to do them is to somehow getting them to try it hands on themselves, even if it's only for 15 or 30 minutes. Let them try it let them play with it and let them be amazed about what it can do and how much time it can save them. So making clear to people what your tool is actually doing is not easy. So they may have some high level idea or they may say oh this is just like Gen2 prefix does. And then why should I take, why should I take a closer look at it and of course when they look underneath the covers or when they start playing with it themselves they realize there's there's more to it or there's a different aspect to it. So that's immediately clear so that's also, yeah, something that's very difficult to get across. If you're implementing new features, very often they will get overlooked, maybe because you didn't make enough noise about them or maybe the documentation is not there or not detailed enough. So you have to keep pointing out these features over and over again when people ask questions. And you shouldn't be surprised by that so it's very easy to to overlook that you're developing with this this tool every day or using it every day, while other people may maybe only using once or twice a week and then of course they are they are going to overlook features or not know about recent developments. Without incoming contributions we've we've seen this multiple times that somebody comes up with a very small pool request initially. And it's very important that you're you're trying to help them or try to explain them what they need to know to get that pull request in, because these people may grow out to be really, really good contributors really frequent contributors or people who have skills that you're are directly missing in your contributor pool so that there are examples where people are fixing bugs in GCC or complex libraries like open blasts. And they're only still around in the project because we were very welcome that welcoming to them initially. And if we wouldn't have been if they would have probably walked away and we would have missed out on getting all that experience into the project. The experience and the skills you have yourself and get and get have definitely comes to mind here. Also don't don't take this for granted so many people are not familiar with things like this yet. And maybe do want to make contributions but you'll have to be patient or you have to be helpful to get them to the point where they can make the contribution that they want to do. So something that's very important and something that reflects as well indicates of integration that we have where we were trying to get rid of that hurdle. All right, so that was the last decade so we're to since today we're starting a new decade for easy build is the project has grown quite a lot in the last 10 years, and hopefully it will continue to do that and the next 10 years it's possible to tell whether this will happen or which roadblocks we will hit. But I think we have a good, we have figured out a good, pretty good approach by now on how to how to deal with contributions and how to get a project of this size to keep it on the on the rails. One thing we've never been really good at and I guess the lack of dedicated funding for the project is, is definitely part of that is that we're not very good at setting a clear road back. So things we want to implement by a certain time, or that we want to change or support. We've never been able to set a very strict timeline to that we could make promises like by the end of next year we'll do this or that. But it's probably very unlikely that will stick to those promises so there's no point in making them. So long term long term planning is very difficult. At least today. We do have short term plans. So in the next months, weeks months and, and let's say next year. So one of the things that we will be doing, or are actively working on already. One of these things is supporting the easy book documentation to mark down. And I'll have a separate slide on that coming up. Then the next easy book release is probably going to be 4.7. So this is, let's say weeks away maybe by the end of the month that should be there, depending on whether we're hitting any issues or want to get in a particular feature before we do so. So let's think about the release of easy build 5.0 so the next major version of easy build, and have some more specifics on that as well. And of course the next easy build user meeting is something that we're actively discussing on when that will happen where it will happen. In the longer term we have lots of ideas, small and big, but one thing we should definitely spend some time on is significantly improving the error reporting that easy build currently does so whenever an installation fails, easy build does keep track of all the information, the build error is going to be somewhere in the build log, or maybe in a separate file in the build directory. But I think that easy build can do a lot better job in catching those actual build errors or pointing out where they are, and maybe even make some suggestions on how to fix them. And then another long term project is making easy production ready and I'll explain a bit more about easy as well. The thing that we're currently very actively working on, and that's an effort that has only started a couple of weeks ago is improving the easy build documentation. The main thing we're doing here is porting what we have currently, which is in restructured text and rendered restructured text and rendered to its things as we're moving this to markdown and mk docs. And in my experience mk docs is a way better tool than Sphinx's for writing documentation it lowers the bar to making changes to the documentation quite a bit. The markdown syntax is also the same syntax that we're using in GitHub issues or GitHub wiki pages, get to pull the quest so that the jump to that format is a lot smaller than it is to RST. The format is really holding us back in different ways we're currently not maintaining the documentation well. And I think part of that is because of the format so we're moving away from that and hopefully that will help. The first part of this effort is really porting to the new format without really reviewing the contents of the documentation. At the later stage once all the porting is done we also hope to set up some kind of review cycle, like one page every week or something. Where we look at that page in detail and see if there's any updates needed or if it's still accurate for the current easy build versions. With mk docs you can very easily get a local preview of the documentation while you're editing it so as soon as you hit save. If you're using PMAX or whatever editor you're using the render documentation will automatically refresh and that's, it's really helpful when writing documentation you can see how it will actually look. And also when the, when the documentation is rendered with mk docs, the search feature that mk docs provides is also way more powerful way more instant as soon as you start typing something you'll get instant results, and better search results as well in my experience, it's very welcome. So all of this effort is being done in a separate repository we're pulling out the easy build documentation from the easy build repository into a separate easy build docs repository. And one thing we will, we are doing and are trying very hard as to make sure that existing links for docs.easybuild.io will not get broken so even though we're going to move everything to a new format and then new documentation has will have a slightly different structure, the existing links that you'll find in emails or God knows what should keep working. And for this ongoing effort I would also like to thank Simon and James who have been spending quite a bit of time on this in the last two weeks, and will hopefully continue to do so. But anyone who's willing to help out with this effort, either by looking at what has been ported already, or by actively helping out with porting what still needs to be ported is very welcome. So for this please take a look at the easy build docs repository, where there are some clear instructions in the read me file on how to get started, then looking ahead. A couple of months, let's say, is we're starting to think about the release of easy build 5.0. So this effort has not started yet really at least not the implementation part of it. But we do have a pretty good view on what should happen by then or what breaking changes we will make an easy build 5.0. The biggest one, which is hopefully going to be of low impact is dropping the support for running easy build on top of Python 2.7. So Python 2.7 is end of life since January 1, 2020, which is now almost three years ago. So we think it's time to pull the plug on the support of Python 2.7 at least in terms of running easy build on top of Python 2.7. And hopefully the impact of this will be very small or nonexistent. So just like an easy build 4.0, we're also going to archive the ancient easy config so anything using an old tool chain. And that's anything using GCC older than 8.0 or a common tool chain version older than 20, 2019 a is going to be moved into an archive so it's going to stay in easy build, but we're not going to test it anymore and you should expect that it's going to be broken quite soon. This is maybe going to be the biggest change in terms of impact is changing some defaults in the Python package easy block where we're going to enable the use of pip and running of pip check and sanity check will enable those by default. So that's the biggest backwards incompatible changes there may be additional ones where they're probably going to be small and the impact of those is going to be limited. And in any case, we'll have very clear documentation of what has changed and how you can mitigate those changes if you are affected by them. Other major changes which we have in mind, which may only be implemented with easy build 5.0 but may also happen earlier, because we think we can do these and without breaking backwards compatibility is a new a cleaner function to run shell commands. So the current run command function that API is quite cryptic I would say you can do things like not fail when the shell command fails or get the standard out and standard error separately. So if you know how to do so but it's very, it's very unclear how to do that with the current API so we're going to implement a new function for this with a cleaner API we have lots of examples lots of experience on how to run shell commands. So we can probably now design a better function for this, and also with with some new capabilities so there are ideas when a shell command fails for example that easy build can drop you into an interactive shell and then you can start figuring out how the shell command failed in the in the built environment that easy but was running it. So this could be very helpful. So one thing we will also do is basically copy the code of the loose version class which is now part of the Python standard library we're going to copy this into easy build, because we're using it heavily throughout easy build both an easy build framework and easy blocks. So basically have to copy either copy it or do something else because this is disappearing in the Python standard library. We want to be ready for this, so we can keep supporting the future versions of easy build of Python three as well. There's several other ideas here, which are listed on the wiki page mentioned on top of the slide. We discussed among the easy build maintainers, but also if you have any additional suggestions or ideas for breaking changes that you think should happen in the next major easy build version now is a good time to raise them. So open an issue and get up or mention them on slack or on the mailing list because now is a good time to do so we don't do major versions of easy build quite often our, we're quite careful with making breaking changes. If there's something that you think really should happen then now is the time to bring it up. Then, this is again, longer term I guess beyond next year and in the years to come. We see a bunch of challenges here for easy build so, as you probably have noticed the landscape of computational sciences is changing in various ways, and changing quite fast. There's a division in available software projects that are relevant for researchers is already been happening quite a while and that's why that line of supported software keeps going up for easy build. There's more additional domains people are being pressured a bit more to publish their code and then other people are picking up on it. And there's also wider adoption of HPC in general like GPUs for example. So all of this has an effect on the software that researchers are using. They're interested in using the cloud AWS Azure. So all these commercial clouds, or even private cloud setups are being used more and more. And this also has an impact on how software is being installed. So, I think one thing we want to enable here is that people don't sacrifice performance when they're using these platforms but they can still run binaries that are properly optimized for the hardware on which they are going to run. That's basically what doesn't happen with containers and conda so their people are sacrificing performance, because they want to be able to run anywhere, and in our view that doesn't really make sense. And I think we can avoid this. What's also definitely happening is an increasing variety in processor my micro architecture so in the last decade. Certainly, until a couple of years ago Intel was still King and HPC basically every HPC cluster had Intel processors. Changing changing fast. The last couple of years AMD is back in the game, which is already a bit of a head scratcher in terms of compilers and compiler tool chains that you're using and compiler options as well. But then also we see arm coming quite rapidly. There are lots of HPC sets that have an arm cluster, and this is only going to increase probably in the future because it's going to be in terms of price performance way too attractive, and it's going to be impossible to ignore. Power is still there but it's pretty niche and this is probably a dying breed in HPC. The risk five CPUs the open source CPUs, if you will, which are coming quite fast as well so then very capable risk five CPUs are being actively designed and will be on the market quite soon. And there's a lot of Euro HPC money going into the design and the research on this so and then in the next couple of years this will be something else to worry about. And then next to the CPUs there's lots of diversity in GPUs as well. Next to Nvidia there's indeed GPUs there's Intel GPUs, and even beyond that other accelerators like tpus and ipus and God knows what. So this is, yeah, we're going back to a way diverse system system stack and of course this will directly affect building software for for all of these variations. And then also what we want what you see is a big shift towards clang based compilers so the new Intel one API, the MD compilers are all based on clang. So in some sense that's more uniform but all of these are small variations as well. And that's definitely going to be a headache to deal with to. So these are all things that whether we like or not we will have to worry about we will have to somehow take into account into the easy book project as well. Now, one thing that we are already working on and to mitigate some of these problems is the easy project so the easy project somehow evolved from the easy build community to some extent. So easy here stands for the European environment for scientific software installations. And if you're wondering whether this is gained so we can spell it easy. Yes, it is. So that's not an accident. What we want to do here is to build a central stack of optimized scientific software installation so like you are currently installing your software on an NFS mount inside your cluster. We want to give you a file system that you can just mount on your system, and that has all the software already pre installed and properly optimized for the for your hardware. So we know how to do this. We're following the lead of what compute Canada is doing there with CVM FS and having a compatibility layer and all these things. We want to make this a community project that basically grows from easy build that this is not going to replace easy build we're using easy build as one of the tools in here to make this possible. It's still even though it's a project that started in 2020 it's still in the early stages we have a proof of concept setup which we call our pilot repository, which is publicly available so you can you can get started with playing with that straight away if you follow our documentation. And we have a long paper and open access paper where we explain this ideas and how we can build this up in detail. This is a high level overview of the project I'm not going to try and explain this in detail because there's a lot going on here. And as you can see from the logos there's a lot of open source projects involved in here from CVM FS to Gen two to Elmont and easy build, and special libraries like Archback were using singularity or obtain her to build the software installations. And of course this is heavily inspired by what compute Canada has done for their software stack. What we are doing here is we're from the ground up taking into accounts that we have an increasing variety in types of CPUs Intel AMD arm, and also risk five in the future. We hope to support all of these and also the different generations of these so different generations of Intel CPUs will provide separate software installations for so they are properly optimized. The core idea here is that you, you as a researcher can basically go from zero to doing running your scientific workloads in three easy steps the first step is getting access to the easy repository. So for this, at least currently you'll have to install CVM FS. And this requires admin privileges so it's not for everyone. But if it's is admin is willing to do that and doing so minor configuration. They can get all the older users access to easy. If you don't have admin privileges you can use a container that includes CVM FS itself, and basically, at least inside that container and single nodes you can very easily play with easy straight away, as long as you have singularity or obtain or install. Then once you do get the access you source the in its scripts which is shown in step two here in the, in the example below, and that's enough to start playing with all the software that we provide in easy. So then you can start running grow match or open foam or TensorFlow whatever, and the in its script is smart enough that it will detect what type of CPU you have whether it's an Intel Skylake or an AMD Rome or a risk five or whatever variation it will pick the right part of the software stack so you get optimized installations for this. So I really recommend people playing with this so go to our documentation check the pilot section and clone our easy demo get a repository where you get a couple of scripts to to play with a small TensorFlow and open foam workload to to see how this works. Hopefully you will learn a lot more about easy because one thing I didn't mention here is that we have some euro HPC funding through the, the multi scale project and a big part of this is making easy ready for production and making it grow in terms of supported software and platforms, but that's going to start that work is going to start early 2023 and it's going to run for four years so we think we can get a lot, a lot of stuff done in those four years. And then we're ready to to wrap up the presentation I did promise a small surprise. So we've been playing with easy build for over a decade now. And sometimes you have to make some changes to a project sometimes it's time for for some refreshing and one of the aspects of easy build is definitely up for refresh. We're quite we've been quite happy with this aspect for now, but we're very aware and I think everyone will agree that it can improve a lot. We do, we did try to make an evolution of what was there before. And hopefully it looks a bit more professional. And maybe it will cause some confusion initially but hopefully not too much. But I think it will grow in us eventually. So I'll wrap up the presentation, the new easy build logo that we've created, which I think looks a bit more professional. And, and hopefully people see it as an a way forward and an evolution of what was there before. And with that, I'll, I'll wrap up, wrap up the presentation here. And if anyone has any questions. I'm happy to take them. So if you have any questions, maybe raise your hand in zoom or post them in Slack and then hopefully Simon can keep an eye on those and raise them. Or if nobody has questions as well. That's fine. Yep, we do see a question. Go ahead. Go ahead and unmute now. Hi, hi. Hi, Kenneth, thanks for the thanks for the presentation and thank you for thank you and the rest of the community for for developing easy builds and I'm more of a user than a, than a contributor, although I, although I do have done quite a lot of my own EBS over the years. One of the things that I that that I suffer with or struggle from is oftentimes I want to update one software package that has a long dependency chain. I have to go back and, and, and update a lot of a lot of easy builds. Sorry, scripts. Easy configs. Easy configs. Yeah. And, and sometimes it's straightforward. Sometimes you can use the try function that sometimes doesn't really, really work very well. And yeah, for example, when you want to just update to new to newer compiler to newer tool chain. And that can really be a massive time suck. You have some, some, some software that has a lot of dependencies like, like GDAL I use I do a lot of GIS on HPC and then yeah so something like that has a lot of dependencies and then you end up spending a day or more updating easy configs. And I know that I know that maybe there's no sort of easy way to make it easier. But, but that I would say that's the only that's the only and they're probably still the easiest thing that's probably still the easiest way to do it compared to any other method for maintaining software on my, on my cluster but still it seems like it takes a lot of time. I don't know. I don't know I just want to throw that out there as kind of thing I do. Yeah, thanks. Yeah, there are a couple of things here that I think could help so you've mentioned try but do you mostly do try tool chain or are you also using try update depths for example. So the try update depths methods is, and maybe you've muted again but the try update depths methods is fairly new and is an experimental feature it's definitely not perfect. But I think it does help a lot. Yeah that's that's a good idea and actually I didn't know about that so I will I will try that's a good that and so when you so just a question on that if you do a try like that does it create a new easy config that you could then put in a PR. Yes, when you do a try you can actually well if you do a try will do actually try the installation as well if that's successful. You get the easy config file in the installation directory, but you can you can combine try with copy EC. So that's also that's just copy EC is just to copy an easy config file to somewhere. So the try will generate the easy config file and the copy easy will just copy it and that way you're not trying to do the installation. When you when you combine try tool chain when try update depths, it will not generate one easy config file but multiple ones so all the ones that it needs for the dependencies, and either it will. I think it will either pick up new easy conflicts that are already there. And if not, it will just assume it will have to use the same, the same software version for the dependency that was already there. And that gives you a stack of easy config files to start from to try with a new compiler tool chain and then see what breaks and what doesn't. Yeah, I will try I will try more with try. Yeah. Yeah, that's one thing there's another thing which is currently not there yet, but which has been brought up recently is usually what easy build does is, when it's installing a whole bunch of things. When one of them fails, it just gives up there straight away that doesn't continue with other things that it could be right still trying to install so it basically stops at the first breaking dependency. We could have a mode there that tells easy but just keep trying and all the easy conflicts you find. Try installing them until you've gone through all of them and a lot of them will probably fail because of a missing dependency, but then others may not and it can try and install that it can, and at least that way you have a more complete view on what is working and what doesn't. That that would already be helpful. And there was something else. There's there's some kind of an alternative to try as well, which is a feature that was started but never finished. There's maybe a bit more intelligent as well and there's, I think an open poll request and framework there that we should we should pop up again, where you can, you get a bit more fine grade control over how easy built is generating a new easy config file based on an existing one. And that's probably a feature we should pick up again and and and see if we can get that a bit more mature and get into easy build and maybe an experimental implementation. Thank you. It's great feedback and I agree this is one of the key pain points currently. But like you said there's probably no no magic bullet here so no no easy solution, but there are some things we can do to make it a bit less painful. Yeah, the one other thing as well I'm still not muted I just would also say that the documentation is helpful but needs more examples. And I'm always thinking like yeah that's probably one thing where I could maybe try and contribute myself some stuff because they just, I learned it you can learn a lot just by seeing how somebody already did something and so having more examples in the documentation would be great. Yeah, any contributions that are definitely welcome or just ideas of where we could add examples. I'm not sure have you looked at the easy build tutorial at all, because there we do try to have some examples and even exercises with solutions along with them. Yeah, a long time yeah I could go back and have a look at that yeah yeah because I think that has improved a lot like the tutorial we did at IC 22. At least goes through the basics we have a separate version, which was done for the Lumie support team which also covers more advanced features like hooks, and have some very clear examples and even exercises on playing with these features. And there we probably need to merge the tutorial and documentation a bit more or at least. Yeah, get things a bit more organized and structured so people don't miss, don't overlook these things. Yes, thanks a lot. Great feedback thanks a lot. Okay, York, I think you're next. Yeah, thanks. Two things first of all a big thank you for making easy build definitely makes my life easier and also very big thank you for all the help I received over the years with my sometimes probably trivial but I didn't understand it problems and sometimes they are like. It's not an issue now please raise an issue. So thanks a lot I really appreciate that, and it cannot be said too often. Thank you. And one question I'm having is you mentioned IP use and TPUs do we actually have access to these hardware. I currently don't I know these things are around there. I also. I have experience in playing with these I hope it's just installing another library and then this thing knows how to talk to this special purpose hardware but I have no idea how these things work. That makes that makes two of us. I was trying to get my, my fingers into RPU at the previous workplace. And then of course I've moved and the PI I was talking to then became apparent and so things were a little bit stored, but I still have contact to the company. So, I'm going to see them in December at the conference. So if you like, I can ask them if they would be interested in supporting us here. Getting access to the hardware is only a small, a small part of it right so having a needs to actually play with that hardware isn't another thing. And then having time to actually develop what needs to be developed to support this is yet something else. But yeah I guess getting access to additional hardware doesn't help at least to play around or to see to see if there's easy, easy changes we could make in easy like for for risk five that's really minimal support for risk five currently in easy builds by getting some access to emulated setups and things like this. But that's already helpful to maybe help other people to get started on that so there's I think a couple of easy things we could do if you know where to where to look into or where to make changes that could help others to to start playing with these things and anything we can do there I think make sense. Yeah, I was, I was thinking a little bit more in terms of collaboration because they are porting hardware, sorry, they are porting software. I can't promise anything. But I can ask, if you see what I mean. Yeah, that could be interesting just to see, see what these things look like or what kind of work would be needed in easy build to support them, just knowing that will already be interesting. I mean my, my crazy ideas and you know I've got plenty of them. My crazy ideas if they, if I can convince them that using easy build would make their life easier as well. And they can see benefit here selling it to customers. I think that would be a win win situation for everybody. But it is a crazy idea I mean it could well be they say no, thank you. No. There's a lot of things in an easy build that has started out as crazy ideas and now everybody's using them. So, I think the GitHub integration is a good example like, I haven't seen a feature like this anywhere else I think. But for easy build that has make it a really big impact. So yeah, sometimes crazy ideas are good. Yeah, cool. Thanks a lot. All right, next question Satish. Yeah, you should be able to unmute. Yeah, I hope you can hear me. Yes. Hello. Hi. I just had a question regarding the easy build framework itself. So the framework has a lot of features, a lot of variables are used a lot of functions are used within the framework itself and to write some some sort of easy block. So this is a bit more involved. It's not just easy conflict. So writing easy blocks to make it easier or to make, I typically found that there is less documentation regarding the framework itself. Rather, so I would have to search for a little bit newer easy blocks or compare them with somebody some some other easy blocks which were either written by you or someone else just to know what was changed in the in the framework or if there was a major change in the functions or even something just as simple as like say loose version or something like that. Yeah, so is there going to be so you already shown that they're going to be a new place where the documentation is going to be. Is this also going to be addressed over there. Yeah, we can definitely do that so we already have documentation and writing easy blocks from scratch with with examples even so in the easy build tutorial. I think it's only in the Loomi version. There is a separate part in this tutorial that explains how to implement an easy block and it gives you a dummy software package to write an easy block for. And it tries to guide you to doing that essentially from from scratch. Now, that's a pretty basic easy block this it's not something very advanced but this could this could be a good step. Another thing is we can definitely add some more documentation on on easy build framework itself like what kind of functionality is in there that could be useful for easy blocks because not everything is, or having separate sub sections on if you want to do something like this in an easy block, what kind of functions are available or what kind of approach could you take with what framework supports today. So running an interactive commands for example something that needs a Q&A kind of thing that's that's one example, how does that work currently. If you need to do something different depending on the software version that's being installed, how would you do this. So I think some very small examples also coming back to what Jet mentioned. You can definitely do a better job and in the documentation. Yeah. And yeah, this is this is welcome in any form of the documentation also in the current one. But I think that after the porting to markdown this will become a lot easier to start to start adding things like this also because the easy build tutorial is now written in markdown. So the crossing things between the documentation and the tutorials is going to become a lot easier once they use the same format. Okay, and also some sort of like an like an index pointing towards all the all the safe functions may not be like a detailed index, but just to just to see what sort of functions going go in some because we can see we can see these kind of things in an oxygen kind of a framework where you see where you see the function the inputs that go into it, and hopefully the output that comes out of the function and kind of documentation. So, since the framework itself has a lot of these. Maybe some some sort of an index will be also helpful I think, I think it's just a suggestion. So the current documentation we already have an API section. So if you go to the docs.easible.io and you look for API. There's a link there that has an overview of all the functions, but searching through that is currently I think a small nightmare. And that is something that will will definitely keep an eye on for the new documentation that the search functionality is a lot better and MK docs. We're going to make sure that when searching for things that you should hit and the API part of documentation which is something we still need to figure out. We're going to try and make sure that you're getting decent search results there so if you look for a function to read a file for example that you can very easily find it through the search box. So it's definitely something we'll keep an eye out for when wrapping up the porting efforts. Yeah. Yeah, and just, yeah, just to echo something I'm not sure if Alex is in the call here but Alex one of the easy build maintainers has asked for a more advanced easy build tutorial, working through the framework from the ground up so what is there how is it structured. And how do you make changes to framework but also how can you leverage framework as best as possible. That's probably something we should we should create a dedicated more advanced tutorial for. And this this would be I think very useful as well for people who write easy blocks. Okay. Yeah, so hopefully we'll we'll find some time for that. Thank you very much. Okay. We have any other questions. Pablo. Hi, my question is not directly related to easy build but to easy. It's like what, what do you think is missing to consider easy production ready. Or what are the main. That's a good, a good question the main thing that's missing is manpower. And that's actually a solved problem now because we have funding so starting January 1 next year. The multi scale Europe Euro HPC project will start. And a big part of that project is providing a software service like like easy does so we're basically going to make easy mature in there. We'll have a couple of full time people working on this on just the service side of the project so four or five full time people for four years. And that's mostly people who are already active in the easy project so that that's very good who are already know what needs to be done and already agree on on how it should be done. So I think in the first, let's say six months to a year into that project will, we should be able to make easy really production ready. And what that involves is a couple of things we have to change our, our CVFS setup a bit. So it's set up in a more secure way. We have some dedicated monitoring for it so if something goes wrong, we know it quickly enough will set up some kind of rotating support role so if something goes wrong there's always somebody around to to try and mitigate those problems and fix them. So that's what we'll be able to do with any production setup. And another big part of that is before we want to make it production we want to automate the way in which software is being added into easy. So right now we're using easy build for that but we're manually setting up the built environment with a container and then running easy build in that environment. And then we make a tar ball of the installation we upload it somewhere and then it gets ingested and all of these is to manual work. So humans are doing this shuffling this this tar ball around and setting up this build container. And we're creating a bot that does this automatically for us on the different types of CPUs. So that automates the whole process that's first so humans don't have to do it anymore. And it also opens the door for people outside of the project to open a pull request and say I want to add, let's say bow tie into easy. And all they do is to add a line with the bow tie version and the tool chain that should be used into a file. They open a pull request for that and then the bot says okay I know what what this means I will go ahead and build this on the different CPU targets. And then if a human tells me to deploy I will deploy it into easy so we're basically doing the same thing like we do with easy conflicts, but then to actually deploy something into into easy. And that's an effort that's already ongoing and I think we're quite close to finishing that maybe this that will still be finished this year. And that's another, let's say requirements before we can make easy production. So, we have a very good view on what needs to be done, we know how to do it where we don't have the time yet to really do it the way we want to but early next year. We should have the time to do that. So hopefully by let's say summer or definitely the end of next year things will look very different for easy and you'll be able to rely on it way better than currently. Maybe Alan has something to add here. Yeah, just just to mention I mean there's some some technical stuff as well right so we have GPU support in the works and for the CUDA stuff. Also we've been considering how to do how to deal with MPI implementations at different sites. So some people have vendor MPIs right that they have to use and there are potential solutions here but they need their. Yeah, they need work right so I've worked in something called MPI trampoline a bit and try and use that, which looks really positive for C and C++ but it's problematic with Fortran. And so basically that just gives you the same kind of approach as Flexiblast it just gives you the, you give you the standard and then you can have whatever is behind the standard and use that and any implementation. Yeah, and there are limitations there, and maybe we can get around them so I again had contact with that developer last week, but it's probably this kind of things that are also in the way of adopting at a particular site. And, but if you if you're already using open it if you're already using open MPI you're in a pretty good place I would say, but what's there. You'll definitely hear more about this Pablo when, when next year when we think easy is ready for production, we will be very, very vocal about it and you'll definitely hear about it. Yeah, so keep an eye on it, let's say keep an eye on on the easy slack and maybe the easy Twitter account and we'll probably hear about it in the easy building list or slack as well. But we can unmute you again if you have something else. No, I just wanted to say thank you and that maybe in my opinion maybe it will make sense to make it once the infrastructure parties in place, which I think is one of the main pieces. This is just my opinion maybe I'm missing information because I'm not very involved in the play but maybe it will make sense to make it available for people. Even if not the full GPU support is there and not full MPI support is there because you know, until you don't open to real end users, you don't get you need the feedback from the user. It's just my my view. That's definitely what we're going to do and that's also why we have the pilot repository already there, which has some things like bio conductor and things like this so things that we know work out already there you can already play with these. And that kind of structure or the way you use it is not going to change much once we switch to the production setup. And maybe something interesting for you there was a bio hackathon last week in Paris. There's somebody from SIB actually, Sebastian Moretti was adding a bioinformatics pipeline to easy. I were hitting some small issues there mainly on arm, but on x86 Intel and MD it's basically just working. And we're now very close to ingesting all of these installations into easy as well. And then, especially I think for for you and your users there's stuff in there that they can very easily play with. And you can see whether they like it or where they hit any, any roadblocks. So I will probably will try to give it a try just to let you know I'm trying to push easy across because we have a project to collaborate across different Swiss facilities and what my view on that is that we should help on easy. Yeah, so but for us the main point will be once the contribution part is is ready so we can really contribute to easy. And of course I can only speak for myself and I am not a manager but from our side from my side I will try to push our teams to contribute as much as possible. Sounds good and I think we're very close to opening up that that contribution workflow and having documentation on it and being able to provide help on it when people are playing with them. Yeah, all right. Thank you. Thanks a lot. Okay, do we have any questions in slack, Simon, or are we good to wrap up. I think we're okay. There's been some talk but I don't think there are specific questions. Okay, then unless somebody else has a question here I think we can wrap up. Thank you very much everyone and don't forget to have a beverage of your choice to celebrate the start of the next decade of easy bill. See you around. Bye bye.