 Hi, everybody. I'm Matthew Miller, the Fedora Project Leader, and this is our monthly Fedora Council video meeting. We normally conduct our business on mailing lists, tickets, IRC, and try not to depend too much on video meetings, but it's nice to have the high bandwidth sometimes, and we basically, once a month, we focus on an important topic and have a guest from some part of the project to talk to us about, you know, what's going on there. And today we have Daniel and Jan, who work at Red Hat on the – what is the name of your group? Packaging Tools? No, it's Software Management. Software Management? Awesome. That's the thing. So that's DNF, RPM. Do you have GNOME Software now? Just Modularity. Okay, and just this little thing called modularity as well. So, and recently this team took over the modularity project from a different team in engineering that have been focused on modularity, and so we're going to learn a little bit about, you know, how that transition is going, what the plans are, and then we're going to talk a little bit more about future modularity in Fedora, which is going to be exciting. I see we have some people from Fesco on the call here as well, the engineering steering committee. In general, a pretty wide audience. So I think mostly we'll let Daniel and Jan do the presentation first, and mostly we'll hold questions to the end. They'll have some questions, and then we'll take general questions, and then we'll see how things go from there. Does that sound good? Good. All right, and I'm going to mute myself because although it's quiet right now, there's train line construction one block away from me, and it's been very loud this morning, so I'll let you do the talking. Thank you very much, Matthew. So I will start and first I would introduce myself and my colleagues. So my name is Jan Brun, and I'm manager of software management team. Daniel is team lead in this team, and he is also expert of topics which are related to software management and also release management from past. And Kerasov Graczek, who is our developer, and he is the key person in the NF team who has been involved in modality. So any technical questions which will be raised, these guys will help me to answer everything. So I will start to share my presentation. Can you see the initial page of my... It showed up and then it went away again. Yeah, it was there for just a second. Does it work? Nope. Nope, okay, so just a gun, I don't know what happened. There we go. It's there now. Great. So we did a survey in April. The purpose of that survey was to collect feedback on modality. We published the survey on federal level and internal red hat mailing list in April 3. And it was also shared in other mailing list of federal and appell level. And during three weeks, we received 193 responses, which is actually great. Can I ask a quick question? Do you know how many of those... Oh, the graph there shows me my questions. See, I should wait till the end for questions. Please continue. Okay. So there were four sections. I don't know go... I don't go in details. There were some also regarding privacy that we do not share any data from that, just the statistics. And this data only a limited number of people can see them to prepare the evaluation. Regarding the structure of respondents, there were 22% of people from red hat com. There were approximately the same number of others and more than half people were anonymous. Most of people were contributors of federal. So 162. There were also contributors in rail people and so on. Also combinations. We also asked about the role of respondents because it's important what is the audience. So there was 33% of developers and the same amount of packages. So it means that 66% of people are anyhow involved in Linux development or the packaging. Some of them were also in other prime roles. On the other hand, we know that the people combine their roles. So if somebody is primary developer, that person can be also packaged and vice versa. We also asked about the contribution, how it is funded. So 43% were red headers and the same amount of people were contributing or are contributing in their free time. So which is interesting. Another question, which is key one, how the respondents are satisfied with modularity if they like it. So yeah, this answer was not positive. Only 12% answered that they like it. 39, their answer was neutral and almost half people answered that they dislike modularity. Another question was if modularity solves their problem. So for 15%, the answer was yes. For 70%, the answer was no. We also asked about problems and because it's important to understand what problems are there for the users or packages. So the main portion was the user experience. It was 62% of people facing that problem. The other topic error was system upgrade, 46%. Problems is model RPMs overriding, normal RPMs, 44%. And some others. So you can see it here, sorted based on the percentage of respondents for each particular issue. So we investigated based on the text in other questions, what they put other problems or some details people are facing. So for 42 respondents, modularity is too complicated or over engineered. Another big group was that people can see problems with installation upgrades, dependencies, and 11 respondents answered that they face build problems. You can see also the other answers here in this slide. We're also interested about understanding of terms which are used in modularity. On that graph, you can see the statistics of understanding that they are the answers that are correct or partially correct or incorrect. And the baseline is number of respondents who answered at least one question, which is 107 respondents. Here, what is I think important is the average on the bottom, which means that only 20% of answers were correct and 16% partially correct. The same numbers but calculated based on, based on answers of each question, what is the percentage, and it is sorted out based on the term of correct or partially correct answers on that graph. Again, you can see the average, so approximately 50% understand at least partially the terms which are used in modularity. We know there are some, we identified there are many confusions, what the terms really are. We ask also about problems which are solved by modularity to the respondents. So 24 answered that providing an alternative content is the feature that they, that is solved for them. For improving packaging and distribution, we decreasing support and maintenance costs. Certain people answered that in opposite way that there are some problems which are created by modularity. We also saw some other answers from the text, what are highlighted benefits for some respondents was parallel availability for four of them life cycle. So we were focused on each area and would like to present plan for the main topics, what we see as a problem. So the first one is related to understanding of modularity, the terms, the documentation gaps. So we are focusing on that and working on that. And also we use the experience from the survey to create grocery, which includes the terms and description what it really means. The feedback from the survey was used as an input so we believe that it can help to qualify better the meaning of each term. Currently, there is one guy working on that. He already finished, let's say the first initial steps. And now we are working on a review of the documentation and we expect that it will be available in the end of this month. Of course, we should continue on that and any help from the community is welcome. Regarding system upgrade, it is another issue which is very important. So we expect to solve that in Federal 33 and also in Federal 34. It is necessary to add some metadata and also to progress infrastructure to solve all of that. But this error is the understanding. It is very important and to improve the experience with system upgrade. Another error was model RPMs which was already fixed in Federal 32. And problems regarding overriding with non-modal RPMs with model RPMs. Honestly, it works as it was designed. However, we understand that this can be an issue. And we would like to ask people that they file a bug if they can see anything that doesn't work as they expect. We can discuss what to do with this issue. Another error is building, managing and distributing models. So it means tools which are needed for example to build models locally and so on. We are working currently on that. There is a guy from another team who is just progressing with that. And we expect that something will be available very soon during this month. Another issue was related to switching model streams which is not straightforward. And we plan to fix this issue in Federal 34. So it will be in our plan to cover that and fix that. And there is source control management like this, for example. So it's related to policies. We believe that the policies should be reviewed and it is not the technology issue. Modulity was designed to do many things. But there is a lot of freedom to do what to do with packages and in modules. However, there can be conflict. So we believe that it is necessary to think about some rules or restrictions how to use that. And to eliminate some conflicts by policies and guidelines. So this is something that people from the community also are welcome to work on that. If you can see an issue, you can see here links where you can file either issues in Modulity Tracker or use bugzilla for filing bugs. And also there is a reference to Modulity documentation with details how you can report bugs and issues which you can see. So that's all from my side, very brief overview. You can see this presentation with some additional data in the blog which was already published on Federa blog. And now it's time for questions. Yeah. That was very interesting. Thank you very much. I especially like the charts and graphs. Kind of the thing. I'm actually curious. Do you plan to rerun this survey? It makes an interesting baseline. It would be nice to see a year from now or even six months from now like if how things are changing based on the work that planned. We have not considered yet, but you are right. It can be interesting to do such a survey again. But for that information, we agreed also with the head with product management that they will do a survey for in our customers that we can see also the feedback regarding and user experience. Yeah, yeah, it was basically this was 100% contributor developer feedback or 90, 95%, something like that, right. So, which is very interesting and super important. It's obviously an area where we've got some serious problems with, you know, people don't like it. So that's not good. We want to be building something people like. But yeah, that user feedback would be really great. But I also would love to see, yeah, this redone just to kind of see if things are progressing in the right direction. If we can provide something that people are happier with and that, you know, solve, you know, as we do these things to solve the issues if they're actually solving them. I guess there's been some discussion in Fesco about the future of modularity and the scope of it. I'm really happy to see the, I think there's some fear that, frankly, Red Hat tends to get things to like 60% done and say this is too hard and then cut their funding. I've seen it happen over and over again. And I'm glad to see your plan. It has some concrete steps that are really going to address some of the big outlying things rather than just dropping it because I think we do have a lot of potential here. In the Fesco discussion, there was talk about use cases. And I think it's an exercise that we went through like, you know, five years ago or earlier on in the modularity life cycle and things have changed a little bit. But there are definitely some use cases that I want to see covered that we are not covering super well right now. One is the, I'm an end user and I would like a different version of such and such an application. And I think we have the technology there pretty well to do that. We don't have the content there because the packaging experiences is not making it easy to do that. But the technology for I want to switch between end user applications, that works pretty well with the upgrade problem solved and so on hand waving. One of the second things is I would like this to be a bridge between CentOS Stream and Fedora packaging with Apple. And that's because this again relates to those content streams. We actually have two streams of content that lots of people are working on all the time. And they're very, very similar and they can live side by side. And so it would be really nice if someone working on a piece of software could easily build that for both CentOS Stream and Fedora without really thinking too much about what the underlying operating system is. Obviously dependencies and things need to be managed. But I want to be able to build this for either one because that theoretically should make my life better as a package but also provides users with those different streams of content that we're already working on. And then the third one is this use case of our various spins and Fedora offerings for different use cases. So the example I gave in the Fesco ticket, which I think is a pretty good one. At a large university I used to work for, they had a Fedora based image that they used for instruction for their intro to computer science class. One of the most popular computer science class at the university, very huge. But they would basically take a version of Fedora and use that version for like four years, even though it passed its life end of life because they didn't want to update their curriculum material to match the newer version of Fedora, which is understandable but frustrating. So what would be nice is if they could, basically that's you don't want the compiler, you're using changing too much underneath you, you don't want new version of Python going from one version to the next. So what I would like, this is a long way to say it, but for example the Python classroom lab to be able to update every six months, but to have its primary version of Python maybe state the same for an academic calendar year or whatever that's based on the needs of that protection. So that's a very particular use. Maybe the tools that the robotics lab needs for robots, they want to have a consistent version of it, even as that software, the mainline version of that gets revved to newer versions in Fedora is like to hold to a certain version for a certain amount of time. So I would like our different offerings to be able to do that. That's the basic request and mostly that's the use case is going to be language stacks, but I can also see it for applications and some other things like that and it's possible. I don't know, but I would like to have the possibility for other things that could be different because I want these spins to be able to explore different options and make their own choices. So I see this as enabling technology for that. There might be some other use cases, but those three things are the main ones that come to mind. Does that make sense and do those things fit with where you see things going? So people in the chat are commenting that you can already do this without modules. Well, in some ways you can, but when you update your Fedora version for Python, for example, if you want to run an older Python command, you might have to tell people, you know, use this version number or whatever, and you may have to update your materials. But we don't right now have a way for spins to provide a different version of the stack than is on, say, we could maybe use SCLs or something, but that has its own set of problems. And we certainly don't have an easy way for the CentOS and Fedora packages to be built across both bases without this. I mean, you can basically, you can copy the spec file back and forth, but then you end up basically with fork spec file rather than the same thing. So I don't know if it's okay to jump in to say something. Yeah, sure. This is this is Bishak. Actually, maybe I can do this. As mentioned in the chat, it is certainly possible to do this kind of thing already. I mean, even the thing that you mentioned with the executable name, right now we have a version Python package that provides a sim link. And if you're going to host a class for a university, it is certainly within your technical scope to do a package that does any kind of thing like this for your students. But this is this is just one side of the argument. The other side of the argument is that modules are a particularly bad way to solve this issue because of this overrides. Right. I mean, everything is, the idea is to stick hundreds of package, Python packages into this, this single module, and then you deploy this on a system. And how do you know what gets replaced? You don't know, right. And I mean, yes, it is a solution that is one of the possible solutions, but it's not appealing technically in any way. And it's going to generate many, many conflicts and many, many hard to solve issues. And so, and this is just not a good example for this feature. If you try to replace a stack like this, it's going to be extremely painful. So I guess part of this, you don't know what to be replaced. I think that's might be a user experience UI problem in a lot of ways because certainly the tools know what package. And part of the idea of having a module rather than just a collection of packages that are basically compact packages is that you have the metadata about what this module is, that what the set of packages is where they come from and that they work together. And so you should be able to actually list directly. This is the contents of this module we're getting, right? In a normal distro version, you have packages that depend on one another. And now you're sometimes even implicitly, right? Because people update a few packages and don't always specify that a specific sub package got a bug fix. And this gets, I mean, those issues get discovered and fixed as the distro is updated over its lifetime. And now you're going to stop this and you're going to put in overrides for specific package versions from, I mean, with versions that don't bear any relations to the ones that you have in the distro. And this is just going to break in many, many small cases. I mean, that's my expectation. It's like you took a package from Federa 31 that you can possibly install on Federa 32, chances are that it will not work as expected. Yes, this is part of the design when you actually select the module which replaces the system packages, other packages in the distribution might break. But you must expect that when you actually install alternative software. The idea behind the module bundle is that the content of the module works together. So you need to replace packages to ensure that the module itself works together. There's no guarantee that everything else in the distribution of 25,000 packages will continue working with the replacement. That is, that is correct. Yes. Yes, I'm exactly this is part of the design. And even at the small scale that we had this in Federa, this was a source of problems right trying to do it. Yes, but it's also a key feature of that. So it depends on what you want to achieve. I completely agree with you. Yeah. Say something. Yeah, Neil. Let's hear what you have to say. So I kind of get a little bit of both sides here when it comes to like the idea of having modular content to provide self contained sets of alternative functionality or content or whatever. But providing alternative language stacks only would work in a modular context. If you essentially made sure that the entire set of package of packages that depend on that language stack are included. So for example, Python is one that I tend to work with quite a lot and Python is actually the most painful language stack to deal with. If I look at from the rail side because every alternate version provided in as a language stack variant is incomplete compared to the default one. And part of this is also that the tools that we have for this are just not they're not self resolving so like let's say I make a module and say I want a different Python language. What should be happening is that the tooling should say okay let's look at the set of things that depend on this. Here's all the stuff. Do you want to include all this so that you can build a new module that like slots right in perfectly. And then it should cycle figure all that stuff out like I, I'm not going to go particular in the details but the idea is that if you make a module of a language stack you should expect to make the module as functionally complete as what you provided before. In a either in the non modular context or in a different module, because otherwise you get weird expectations and weird inconsistencies and I think that is the, the ultimate problem that leads to the poor user experience whether it's a module or a separate repository or alternatives or whatever like I don't really care like the problem is that the content has to be functionally equivalent, so that things are still useful in all the ways that people are using them, especially with modules being fully conflicting and shadowing other content. Right, so I think the stream expansion feature modularity is meant to address that, whether or not that's a complete solution or not, you know, details. It is, but it requires the maintainers of all the packages to depend on your language thing to also modularize their content. Right, so it's the technology is there but it requires it makes more work for people. It also requires infinite computational resources. There is that to there is the whole everything's got to be rebuilt every single time without sharing anything. Yeah. Right, it gets complicated very quickly. So I think some of this can be addressed by one of the things we don't have which is if there can always be a base stack that is put that is you don't need to have infinite parallel install ability, but if you can make it so if if there's always a base language stack that is always there and like you know system Python stack basically that can be depended on by other distro packages and then they have the modules be or whatever whatever they are be independent of that. But that needs this needs some working on and working out obviously. I mean at that point you're talking there's also there's the policy question of what should modules be scoped to. And then there's the technical question of how to discover and automatically construct a functionally useful module. Like we don't both of those pieces are missing today. We have tooling to build modules. We have the functionality underneath to make such things possible. But we don't have the the I guess in language content is in programming context like the sugar to make it so that people can do the right thing easily without pain. Tony as it happens Steven Gallagher is working on a new modular policy. I'm not sure if he shared the link in the fest go ticket I can I can put it here in the chat. Thank you. Terry do you have something to add there you're making some faces. Yeah, I was going to ask if it would be OK if I back up and share with some with everyone some of the original goals and visions that that led to where we are today. Would that be OK. That would be awesome. Why don't you also introduce yourself for the recording for people who don't recognize. Yes, I was going to do that as well. My name is Terry bowling and I am on the rel red hat enterprise Linux product management team. And I work closely with the DNF and other system management tooling. That's my area of focus. I was not part of the earliest stage. Discussions that led up to modularity. But I've been caught up to speed. I joined the team about three years ago. And so these discussions had already been going on for a year or two prior to that. So. The problems that led to this discussion were. We were seeing both rel and fedora having the challenge of how do we provide our users. Access to more content, but still provide a curated tested stable distribution environment. And the use case needs of fedora is a bit different than than rel. But we also saw the ongoing continuing trend and adoption, particularly in the cloud where Ubuntu LTS and centos were just infinitely more popular. Or or used then. Then fedora part of the problem is there's a whole other set of problems where. Pre built fedora images are not available in the cloud. I don't want to confuse the discussion. That's that's an entirely separate set of meetings to discuss. Plus, I need a lot of whiskey for that conversation. Yes, you and me both. And so I don't mean to open that can of worms, but we have to acknowledge that fact. And so rel is deeply concerned. About the adoption and growth of the average fedora user, particularly in those university scenarios, like you described, which I have lived. I've been a fedora user for. Well, since it came out and red hat Linux 9 before that. And I used to teach at a university and I experienced that pain. If I wanted to stay on top of the. The latest fedora core release. I had to rewrite. Much of my my lab and curriculum. Every semester in order to keep up with the change because inevitably something would would change. And now I teach a robotics class at my daughter's school and. The tooling to interact with the VEX robotics, which is extremely popular in the US and used for a lot of university stem. Curriculums. My choices are either Ubuntu LTS or Mac or windows. So I can't use fedora to teach my, my daughters and my students this robotic stuff and I'm not a developer. I'm just your average. Less than average dumb user, right? So. So those are some of the problems that the growing fast adoption of. Ubuntu LTS and when we have sales. Enterprise. Conversate sales conversations with enterprises. We increasingly hear a trend of. They've got things newly developed in their business and they're turning into production because they work. And they were written either on centos or Ubuntu LTS because that's what the students were using in college. So it directly impacts financial and sales conversations when we talk to our real customers because it's a growing trend. Another can of worms that. We don't want to open right now is. You know, windows, whistle to and in the growing adoption of Ubuntu in that space. Again, Ubuntu LTS. Is extremely important to the average user. Who is either a developer in like a business setting. Not a developer as a Linux distribution maintainer. Those 2 developer personas are very different. So, I wanted to highlight that when we saw and interpreted the fedora modularity survey that you on presented earlier. That was. A very obvious thing that we noticed was. Matthew, you mentioned this as well that. A high, a large majority of the participants of that survey were fedora distribution. Package maintainers or or developers or both. Which is important. I do not want to minimize the. The pain that we have inflicted on our fedora developers. It is real. You are correct. We hear you and we want to solve this with you. But we have another user base. As well that we see is really important both to rel and to fedora. We'd love to see more average users using fedora. And quit using Ubuntu LTS instead. And so that was one of the hopes. Rel product manager didn't. Rel product management did not ask for modularity. What we asked for is the ability to provide and curate at the distribution level. More options for our users to help our users. More easily transition to a next major release like rel 7 to rel 8 without being forced to switch their. PHP or. Or postgres stack or whatever is important to their application needs. As well as how can we provide more choices. To make fedora. More usable without forcing that technical debt of upgrading. And everything on users give them more options so that they can update their applications when they're ready. But still be able to use the latest fedora release. And the engineering implementation that was delivered to us to solve that was modularity. So rel product management didn't ask for modularity itself. We asked, please solve this, these user problems and help us grow both. Both the fedora and rel communities. So that said, we have been watching the, the pager threads 21. 2114 I have that number memorized. I've, I've read every single comment. We're very interested in working with you. And open to the idea and, and yawn. I can't remember if you mentioned this in your presentation earlier, but you're having discussions about if and when we can start work on. Or plan. An evolutionary design change based on all the discussions with with the. People on this call. So we are open to that. However, rel 8 has a 10 year life cycle and we have to fix. We have to stop the bleeding and we have to, to fix our patient now and we're asking for your help. If you can be patient with us and help us fix some of the problems that we have now. We're listening to you and we want to take your input and work with you on. What you recommend as a better solution. To solving these user. These user experience problems. And the user experience problems. I mean, back to the original. We need to provide more options to our users. Some of that was around. Yes, I'm sorry. Yeah, Peter's been trying to get your attention to ask a question for a while. Well, Terry can finish. This is just a question about bureaucracy and whether you still intend to formalize your development plan as a new federal modularity objective in the future, or. If this is just going to be happening. Without any form of schedule and plans. Are you asking me that or the modularity software management team in general. John, would you like to answer that. Yeah, so currently, the team is working on the goals or tasks which I described. Regarding the objectives. The question is about this thing or to distinguish shorten and long term goals. I would suppose that we should agree long term goals. In the objectives, not just on the short term. So the objectives should be actually kind of paste placed in the middle there they're meant that they're not necessarily eventual goals there was things that are meant to be achieved like a 12 to 18 month time frame. And that's why part of the problem we had with modularity and objectives originally was that it turned out to be a much bigger thing than the 12 to 18 months. And so we kind of had to do it in objectives and phases and that that wasn't entirely successful as a bureaucracy process. But yeah, if I think from a federal council point of view, it would be really helpful to have this as an objective, whatever we whatever we name it, because. We definitely have a problem and we've got we've got some things that we want that we aren't at yet. So I think an objective would be helpful. Going to what Terry was talking about a little bit. Wait, let me let you get it word and edge wise and I have more to say. What are you going to say on. I just, I fully agree. And our plan is, let's say for the timeframe at the end of this calendar year, let's say, or in federal up 34, not for longer time period. However, I think that we should think also about some if you are, if you are talking about 18 months. And in that case, we should also think who will cover that, because currently, we are ready to do what we would like to send it. But there are discussions in the community about other options. And I do not see till an agreement, what is the best way regarding morality. And I would like to see that. I think it is this meeting is also the great opportunity to discuss that. Because if we are considering to open something like a new project, like, because there are such proposals. This team can't drive that it is impossible. On the other hand, it doesn't make sense to create a new project which will not collaborate with the current project, because if we are in the real world is some conditions and we can't just cut or kill this project we should continue we have some obligations. So I believe that we should find, we should find some agreement how to proceed further and based on that to set up the objectives. I kind of come from the same point of view as Terry. I have these problems. I see that I want Fedora to offer something unique to our user base that says, if you use Fedora, you have these options that you don't with other free Linux distros. That's compelling. And modularity was the technology developed to answer that question. It may seem to some people on this call that I'm weirdly loyal to this, even though it's not popular in the community and not working. I think my loyalty comes from the fact that we had a team of people had Red Hat commit actual engineering work into Fedora, and we had a team of people work in Fedora. Obviously we had some problems with the rail eight time frame where a lot of the work ended up being rushed behind inside Red Hat instead of being developed in Fedora. It would have been awesome if we had another year to bake this before rail eight came out, but that's just how the timelines worked out. But we have a team of people who worked really hard in doing this in a community way, and sometimes things are not 100% successful. We need to give people the opportunity to not have to do everything perfectly in order to be able to progress forward. And if things aren't working, sometimes you throw it out. Sometimes you say, okay, what can we do to move this forward in a better way and just as we go on. I think the project has seen a lot of adjusting as it's gone on. We went from the everything as modular approach to the base repo, the thing we have now, which is very different. We have some comments from Neil in the chat here, which I think will not get into the video. One of them is that modularity as it is actually seems a long way toward solving these user problems, even though it's difficult, even though the technology is almost there, but that we have this problem where we're not getting the content because the user experience of the packaging side and the tooling is rough. So I actually still have some optimism for this current technology, and I think that we can actually get something that does provide us with what we need in Fedora. On a technical note, one of the things that I think it is important about the approach taken, and maybe the architecture may be too confusing and complicated, but I think the basic concept of having the module metadata be separate from an RPM spec file is really key to working on this. It's the same thing we had when back in the dawn of time, RPMs were grouped together by a group tag, and that's what was presented to users, what group they belonged in, and that turned out to be horrible, and then COMPs was developed as an alternative thing that was basically the metadata for groupings lives outside of the packages. Now COMPs isn't perfect and could have used some updating over the years, but it's clearly a better approach to have this outside of the package. And so I think that's fundamental to whatever we need, and a lot of the complexity design kind of comes out of that, so that might be going into the weeds. But again, that's something I'm pretty attached to, and I think a lot of the other solutions proposed don't offer the flexibility that it has. Well, there was also a small piece of that up until about five years ago. Native RPM, like without patches like without the SUSE or mandrieva patches, didn't support weak or conditional dependencies and without those pieces, like the functional installable groups concept that we have with composition groups or COMPs groups as a lot of people call it, or even with modules was literally not possible. Like without without that it was literally not possible. And the old RPM philosophy back then was that this functionality didn't belong in RPM that has obviously changed. I don't but whether that means anything for solving all of the problems is a different question but like that was like some of the underlying background to why this wasn't why we had like this externalized weirdo metadata thing for groups instead of like putting it in the package spec. Well, the big thing with COMPs is that it also lets packages be in multiple groups and RPMs that whether or not that's a good thing. I since I have the floor now I want to get a few words and Terry go for it. I appreciate that as always. I think we're missing one key thing that Terry mentioned is that the need for Fedora LTS needs to be considered since the Ubuntu LTS is really popular. I don't see why we don't do Fedora LTS. But back to the modularity point. I do see the use cases as described they do make sense to me. But I do have concerns around the developer experience side of the house. If it can't be explained to a developer or a package maintainer succinctly then it's probably not a complete implementation. I'm not saying bad implementation is just not fully baked and I do think focusing on that is important for us to see any success with this. If the package maintainers can't wrap their heads around it, it's not going to go anywhere. So you've seen my comments, the scope of this, the, you know, what is expected of modularity maintainers, things like this. These are questions that are unanswered right now and they just modules just sort of exist as they exist in the state and rally and that's how they landed. So I would put importance around the developer side of this thing. And if that takes us in other directions, fine, but that seems to be the key missing piece. And I will shut up now. Thank you, David. That's super helpful. So we're going to run out of time in this meeting, but one of the things I think that is important at this point. I see there's a red hat commitment to working this out and I think we've provided a space in Fedora where that can be done with the ELN. And I know there's an ongoing conversation about whether default modules can be used in ELN in order to work on some of these things. I would really like to encourage Fesco to find a way to let that happen because otherwise that work is going to have to be done outside of Fedora or not at all. And that would be a fairly, it's not that red hat doesn't have to solve this red hat has to solve it. So let's make sure that there's a space to work on it that's in Fedora so that we kind of keep that that collaboration going and keep the energy also looking at Fedora needs as well. And not just end up solving a well internal problems well internally and not actually working on the Fedora community side of things as well. I know that. So the stuff that Terry said about what people want and what the general goal is this is very much true. And we also see this in the modularity survey that basically the most common answer that why people like alternative versions. And I think we should think more widely about how to achieve alternative versions and not how to solve problems with this one particular technical implementation, because this this constraints us a lot. And this push towards having Fesco accept this solution because realm is the solution and it meets it soon. This is setting again a bad bad precedent, like with, like you mentioned, the push to merge modularity before real aid. Now we are kind of fears later and we're repeating this story. So I think that we and the council should say set high level goals and the technical details should be figured out in a in mainly threads and places where there is stuff to, you know, to write up a 10,000 words description of some idea and not in a video meeting. Yeah, may come in. Hi everyone so I want to question. Why do we compare? Like, why do we consider alternatives to modularity to be to be replacement of modularity? Why don't we consider alternatives as alternatives which can be developed in parallel. Very much like the idea of like researching different solutions for the same problem. I think this is a good goal to have like to investigate different ways of what if if the goal is versioning yeah we can we can try to address it with modularity. We can try to address it with different thing. I feel like we need to give space to anyone to experiment with this technology. I don't see why you put it like as a requirement to remove modularity to start working on something new. Like, if there are people who want to work on something new. Let them work on something new if someone comes with the idea like let's create a totally different approach. Let's do that. But we have people who want to work on modularity. And we want to keep going in that direction as well so it shouldn't be one or the other it should be let's try to do us and see how it works. I think there is a need to. Basically, I agree with Alexander that this might be possible but you also need to look that look at that from a different angle as Terry already mentioned we have an obligation to maintain the current implementation for next 10 years which keeps us busy. And that will also require for my team I mean DNF team mainly to support both implementations in DNF and if there's multiple implementations, it would also support them. So the users could eventually choose which one to use. I'm not saying that both technologies should be supported by the same people. Yeah, I'm saying that we have defined group of people who want to keep supporting one side. We already have those people. When you say let's do it completely differently and isn't you, I'm not saying no, but I'm not I'm saying like, yes, let people if there are people who want to do that, let them do this, but let the people who work on the older solution keep going because we know that this will be still supported we will still have members in the community in redhead who will be supporting that. Here goes my tricky answer if you find someone who would take care of existing majority for me so I can focus on the future I will gladly do that. Well, I think Alexander's point is that 20 seconds. I think this point is about not stopping others, but about doing the changes right like the stop energy that just investing time into just preventing someone from fixing things is probably not the right thing. And yeah, if there's a group of people that just has this goal that we need to let them finish it right we can't just let them stop. You just can't stop them all the time. Yes, it is a strong enough to implement something better, I guess. There's also another side to this that there's nothing stopping people from working on what you're right here right now. So, let's, and there's, it isn't and Well, it's Yes, there is stuff stopping people from working on modularity right now it is the fact that every time anyone comes to fesco with a with a request to improve or change or test something in modularity. People say no. Well, okay, so let's, let's, if this was a different case if we're talking about NTP implementations. The way to do this is that you provide another implementation you prove that it works. And that's at some point you flip the default. And here, for some reason we are talking about flipping the default first. And then as we are going on, adding the missing pieces, this is just the order of things. I agree with your point, but I think, like, we may have talked about it before, but what we are proposing currently at this very moment is exactly what you describe. We don't like, I think no one in this room currently proposes enabling of model or models in fedora right now at this point in the state which we have right now. So what do we have to, Steve what is doing that. Hello, Miro here, I'm finally, I can speak. If we agree that we don't want that default module streams right now then we everybody would be happy, but every time that's something that's going to go on. We proposed and what led to this conversation. So we propose couple of weeks ago that we have ELN, which is a build route of for fedora height packages, which has no impact on the main fedora release. And in that build route we propose to fast co that we will be enabling default models to try and and see if it works if it can work together with fedora height packages in the same environment. So this was our proposal. And then you say, we cannot do that because we need models in fedora policy first. So this is how we came up to discuss in fedora models again but what our original intent was exactly what was described like build it on the side, don't have impact on main fedora, let it grow there separately. And then if it works there, we can come up with some idea how to make it work for fedora as well. This what what is proposed, we were not proposing modules in fedora right at the state of things. Right the only reason the policy I wrote generally speaks generally of fedora is because Miro specifically said I don't want a different policy for ELN and fedora so I just wrote it so that it would be applicable to both and we can only apply it to one at a time. Yeah, at the same time, there was a proposal to say we don't want default module streams at all, because we think they are wrong. And it was said that this is not the way to go forward that we actually need default module streams and nothing of the original needs for modularity actually says we need anything like this. And the problem here is that developers of modularity and developers of ELN and developers of model packages think that they want to experiment with default modules. We don't push you to agree with default modules in fedora yet but we want to experiment with this in the area which we maintain and we are responsible for. And this is not decision of fedora to say default modules are impossible. It's a technology, modularity is a technology it may use default streams and default modules we want to try that on the side. Before we make any decisions on the future of this idea we want to play with it and we have people who want to invest their time like my time, Steven's time, maintainers time, model or packages time to try this technology that's what we were proposing. And now you say like you say default modules don't make sense. If it don't make sense for you you don't have to work on this with us but you shouldn't blocking us from working on that. Go to the NTP analogy this feels a lot like somebody who has a new NTP implementation and Fesco saying that implementation is a terrible idea. We don't want that in fedora and then not allowing a spin with that implementation to be created because they don't like the color or the programming language it was in or something about it. Whatever just not letting this is not about color or programming languages this is about problems that exist problems that have been proven problems that packages had and nobody fixed their problems then we banned it and now you want to enable it again when there is nobody who would fix the problems. The maintainers of this technology would like to know the maintainers of this technology says they don't want default module streams in fedora because they think that will only create problems. Why are we pushing it so hard. Let me let me clarify this. So, my opinion is that the default streams really create some problems in federal but on the other hand ELN is a completely different project, basically different distribution, different target audience, and if they want to build that using modules. I don't think there's any reason why they shouldn't. And in this case, they can potentially share some policies, but if they decide that, for instance, they want to modularize everything and this is the goal. The users or consumers of ELN will probably have to live with that and if this turns to be not working, they will definitely change the policy or change the distribution accordingly. Furthermore, there were two, two main issues with default streams. One was the upgrade path problem that is being addressed by the module ELN obsolete. The other one was modules in the default build route, which was addressed by the Ursa Prime proposal, which works, but none of those are currently implemented because they were also stopped by Fesco or they are not. They were stopped because each one had significant third coming. Right. It's not that they were stopped because they were being perfect. They were stopped because it was clear that they, despite being very complicated, don't actually solve many of the issues. I think that with ELN, we are facing the same problem that ELN was advertised and approved as a way to have a real flavor of building packages in Fedora. But now, two weeks later, we are switching. Well, this wasn't mentioned at all, right? And for me, it didn't appear. There was a long discussion and this idea wasn't floated during the discussion. But to me, it's the providing modular packages to provide stuff for for ELN is a completely different way of delivering packages and it's not a real like build of Fedora anymore. And it seems that enabling default streams in ELN will cause the same problems that it's causing Fedora. And this, we shouldn't just ignore that issue. If you're right, then this, then they will discover that quickly. I really strongly agree with Alexandra that we need to provide the space for that experiment to happen and for the problems to be solved. If you're right, you can get a good I told you so in, but I think you got a good I told you so in the previous discussions and previous votes on modularity. Everything that happened in modularity was told you so. And that the you say we need to listen to Fedora, while at the same time, nobody's listening. There was so many things wrong. Okay, can you at least let me finish the sentence please. You said ELN is a place where we build Fedora content for in real like environment. And now you're saying ELN is a different distribution. I think we need to get the story about ELN right. And if ELN is Fedora content, we should build Fedora content. And if Fedora doesn't want modular streams because it's a bad thing to do for Fedora if then ELN shouldn't have a deal. If we want to build something completely different in ELN than Fedora, if we want to have modularized ELN and demodularized Fedora, it's no longer what was proposed as ELN. It's a completely different thing. Regarding different distribution. It's not what I said. It's what Daniel said and the wording is different here. So please don't treat it as contradiction. So ELN, as I proposed it and as we voted on it, is a place where we experiment on how Fedora height content can be packaged differently in a way which resembles how Enterprise Linux does it. This includes build, compile flags. This includes also, for example, the structure of a compose. And it was the intent of the ELN to create a different layout, different set of comps files, different layout for repositories which we create from this Fedora height content. For me modularity is that part. It's not the content of Fedora height which we're changing. It's the structure of the artifact which we get out of this content which we're changing. So for me there is no contradiction. I agree you may see it differently, but the idea is that default modules is basically a question of which packages we include in the ELN build route. And for me it's the same question as how comps files for ELN are configured, how compose structure for ELN is created. And this is what we promised we will be experimenting on. And when you say we have proven that in Fedora modularity doesn't work and default modules don't work, I think we clearly have on this meeting the understanding that not everyone believes that this is true. So what we are asking right now from Festco and from Consul is that to give us a space where we can prove to ourselves or to you that this is actually true or if it's not true then prove it, it's not true. So you say you're convinced default modules won't work. That's okay. We don't try to convince you right now that they will. We want to work as our separate group in ELN to see for ourselves if we can convince ourselves that it will work or not. This is what we are asking for. I have a question about giving us the opportunity to do that. Yeah, go on. I have a question about the did not work. Is it that it truly didn't work or we didn't quite finish it and we just need to finish fixing a few things and then we believe it can work? I think that that is the question that is up for debate. I think there are people that believe both of those statements to be true very vigorously. And so that's why I really strongly think that we need to have the space for experimentation and I think this is what Fedora, whenever we have a thing like this Fedora should provide these spaces because that's what we do. We already did. I would like to see at least a proposal for how turning modularity on for ELN will address the problems already seen with modularity. That's been like a simple ask and then what the what the scope is of modularity because as I stated in the last meeting. The viral nature of modules forcing other packages to become modules is a concern. With the developer experience like what what is the scope. Neil made a point earlier that modules have to be fully self contained. We keep talking about them as as modules is their standalone entities, but they're not really they're linked against the entire OS like shared libraries are still at play. And when we if we're going to allow packages that are not in modules to depend on packages that are in modules now we get into this sort of weird territory. So that's why I keep bringing up what is the scope and what is the what is the expectation of modules. Like if it's going to consume a develop package that's used by something else. I mean this is this is a developer experiencing and we've turned this into a fesco meeting, but this is the kind of stuff that we talk about because it does. Really impact developer experiences quite frustrating and that ask has been there and all we keep getting all we keep hearing back is yeah, but just let us turn it on. We just want to play with it. And I have a question. It's turning it on if we have a plan for how to address those things we've already seen. I have a question. Have you seen the proposal of a policy on default models. Stephen has posted. Have you had a chance to read it yet. No, when, when was it posted. Okay. Okay, very recently, as apparently like a couple days ago, he linked it in the chat so you can pick it up again if you don't if you want. So. I just wanted to come in. So, I think your point is good one that that we probably need to formalize more of what we want to try. So I think Stephen made this proposal. It actually addresses this concern you have. So what we currently have is a very restrictive policy on default modules. We're not even sure that real modules actually can work with this restrictive policy. So if we try this policy right now, this will be a challenge for well to fit in as well as it will be a challenge for fedora to fit in. And I really want to have this very restrictive policy to see in action if we can make at least one model to work with this. That would be a challenge on its own. That's what I would love to see in ELN first before we even go further in the development of more modules and more more stuff in there. The policy that I'm going to have to read that. I'm going to have to find that and read it. Sorry. Yeah, that's okay. I'll go and read that and at least get up to date on that information. I will say one more thing about the concern about what ELN actually is. Having having been involved with fedora and rel for a very long time and I've seen multiple. What we call rel rawhide tries come and go the farther away that ELN diverges from rawhide. The more likely it is not going to be useful to serve as a testing area for what becomes rel. So the concern I have there is with and Alexander, I think I mentioned this to you with creating branches and just get things like that as soon as we start doing that for ELN. It does become more and more divergent from rawhide, which makes it harder and harder to maintain. So anything we can do to keep ELN from doing that. Well, I think, I think we'll lead to ELN's success as a area to test things intended for rel instance to us. Thank you, David. I'll let you go last time. My turn. David. Yeah. Yeah. No, it's okay. Thank you for that. I think that's really good insight and then also I think your request for a plan and what's going to happen is a reasonable one. I have, and since this is a fesco meeting now, I have a technical suggestion that I think will help address a lot of these issues if it's possible. And that is the thing that was part of Steven's revised modularity plan initially, which then got dropped, which is for packages built in the modular system as default modules to get actually tagged into the non-modular repo as non-modular just regular RPMs. And the reason that didn't work was, as I understand it, because that modules can conflict. If we want default modules to not conflict, that actually should be possible again. And also actually helps enforce the they can't conflict thing by making those conflicts obvious. Peter is shaking his head at me because he doesn't think that there are other technical bits that make this impossible. So, yeah, there's a whole issues there. Steven says we can do it though. So that's the thing I said that the major reason we do this in the past was because we had, we had as a design philosophy that modules could have dependencies and that the dependencies could change. The current highly restrictive policy eliminates that major problem. That does not mean that there are not other problems waiting in the wings. That was just the one that simply prevented it from ever being considered. That's something I would like to see experimented with because it does it removes it removes the quote viral nature of it because these base packages can just be used as non-modular packages. And then if you enable the module, then suddenly the module the metadata is available. People. Yeah, those rpms are specially flagged by nbs so that rpm knows that they are modular, even if the modular metadata gets corrupted or is missing or something. So if you just put them in the non-modular repo, they will still mess things up. Also, dnf will refuse to install packages that are marked with this label because of because of it. These are problems in you know, you see, so all these problems you mentioned, they will appear in and we will have to solve them with a redhead help with. Or we will have to cancel this idea. So we actually can try this in eln fail or win depending on the environment, but we won't damage anything in fedora by trying that's that's the main point from eln seek here. So I think that it's actually perfectly fine and I want to see this actually happen in fedora through eln because because we have boarded so early we couldn't it was a lot harder to figure out the full scope of the faults in the existing in the existing infrastructural design. I'm not going to comment on like the other total design on the other part of it, but the infrastructural design, as I've seen it experienced with intent to try to work with is we haven't fully fleshed out what we need to make that work. And I think eln is a good place to figure that all out. However, one of the constraints that I'm really I have some very deep misgivings about is that the dis the constraint that we must that we cannot change. We cannot change the underlying user experience for modules both developer and user side, because that's what they are in rel. That that restriction needs to be removed for us to be able to succeed in making this a useful project. The last few times that I have given particular feedback and even I've talked to Daniel mock and and Yaroslav and like started working on patches and stuff. The result of it was, we can't change this because rel says we can't that we need it to work this way. And that is not acceptable for evolving a solution for future rel or even fedora like I'm not like, I'm going to ignore the fact that this is about the door I'm saying that for future rel, it is not okay that present rel blocks future rel. Like that's that's just not okay. So as long as that constraint goes away I think modularity as a solution as an initiative has actual hope for success. But if we if we anchor ourselves to rel a we're screwed. I am preparing my sourdolos to get them ready to go into the oven. Excellent. That was my my lunchtime plans. Before the skull ran over. I hear you loud and clear. The red hat software management team has been focused on fixing current problems. In particular, the fedora 32 upgrade problems. So we have a challenge we know rel eight is going to have some needs like we want to deliver more module. We want to deliver more application streams, which means, you know, newer modules like pearl five dot 30 and a few other things. We also have some other layered product things like the vert team needs to deliver multiple versions of the vert client tools, so that multiple client tool versions are available to end users to choose from, depending on what you're looking for. So we have a number of of important needs, and so the team has been focused very much on the context expansion problems, the module upgrade problems and things like that. So it's not that we're saying you can't work on something new. What we're trying to do is we want to make sure that we're able to deliver more applications. And we can't just introduce radically new module metadata constructs or a total redesign into rel eight to fix those things. So we have a number of major problems that need to be fixed as soon as possible with the current design implementation. And we can't just introduce radically new module metadata constructs or a total redesign into rel eight to fix those things. So we need to find balance, I guess. So like, simple, very simple example that I was told no multiple times when it kept coming up as a problem was I need a way to enable switch and distro sync in one action from one module stream to another. Like, I can't just have things break, because I need to request content from one thing to another and I don't control the base layer. There's an RFE for it and things like that. And it's just, I don't want to say let's burn everything with modularity and start all over from scratch, because one, I don't think that's realistic and two, it's not even necessary. Like, functionally the metadata is mostly fine. The tooling just kind of stinks, but that can be improved. The user experience constraints are the ones that I'm the most concerned about. Because the user experience constraints are what are 90% of the negative attitude to modularity. Like, having spent a year in re implementing a lot of this infrastructure myself at work, like, I am keenly aware that this actually solves major problems can be made very useful. But it's but we have to be able to reconsider the restrictions that were imposed in the user experience. I'm not saying we change everything else, but we need a better user experience at every level. The stream switching isn't that one of the items on the list at the end of beyond survey. It is. Yes. So, so we listened to that we agree. I had to do a little bit of work on my end to say, yes, the developers need stream switching and I think regular well users will be too. So I did a little bit of work to fight that battle on my side. Unfortunately, it didn't come in time for. The door 32 and I think it's targeting for door 33. Is that right, Daniel? We are going to make it because we need to start working on the end of life and obsolete first. Yeah, both of those things are actually going to be both of those are going to be tied to each other to some extent. But like, I can't build images layered on top of other images like we're talking container things, because if the base layer chooses a stream and I need to switch to a different one. And they're compatible software wise and dependency wise, but because of this locking mechanism, I can't change anything like that. That's painful. And that's very broken. So we are, we are running up to 90 minutes on our 60 minute meeting here. Thank you, everybody. This has been a good, good discussion on a heated topic. And I think we need obviously the discussion needs to continue back on mailing lists and things. It might be useful for us to have another video meeting like this sometime pretty soon, because I think the high bandwidth conversations are helpful and the closer to personal interactions also can be better than mailing lists. Not that we need those discussions to the whatever 20,000 word ranks that were mentioned, but I think this is was also good. So thank you everybody for coming and participating. There are a huge number of comments in the chat here as well, which I was not able to keep up with. So if there are important chat comments, please bring those to the mailing lists and so on. I will try to get this posted on our YouTube channel as soon as possible as well so that people who weren't able to make it at this time can join the fun. Again, thank you very much, everybody, and I'll see you later. Bye. Bye, y'all.