 Good afternoon everyone, my name is Sian Zeleny and this is the Replacing Cam with the NF Talk. So I will start with the question that I actually got earlier from Matthew. This question was didn't we already do that? Yes we did and that's why you can see the retrospective back there. So what I'm gonna do is in the first part of the presentation I'm gonna tell you something about what happened, how we did things, why we did them, stuff like that and then we're going to move forward to what's ahead of us because the migration is still not completed. So this might be opposite to what the description of the talk said but you will see that the first part is actually useful to understand the second part of the presentation. So anyway I have a lot of stuff so if you have any questions please say them for the end of the presentation and with that I should probably state what are the goals of my presentation. So first of all I hope this presentation will help the audience to understand why things are the way they are. Second of all this is about self-reflection because not everything went well and we want to show you that we acknowledge our mistakes and we want to learn from them. We also accept feedback so if there is anything you don't like about the presentation just come and grab me afterwards and tell me what you would like instead. And most importantly this is generally not about individual bugs or RFEs that you might have. I trust there is a few of those few of you as well who have individual problems with DNF. Again I can discuss those with you after the presentation if I'm knowledgeable. I should state that I'm not actually on the development team I'm just the manager so I can talk about the high level stuff but I'm not sure about the team time details. So let's take a look at the topics so retrospective that's gonna be about that's gonna go way back like when the development started. DNF 1.0 that's more about recent history like what happened right before Fedora 22 and GA and these two will hopefully give you the background to understand what is planned and if there is time for that I also prepared a couple slides with a bit of controversy. Okay so let's dive deep into the history and start with the positive stuff. So what went well in the DNF development getting early adapters was definitely one of the things that in my perspective we did very well because we managed to get out the early announcement that this is going to happen the announcement went out almost 15 months before the target release Fedora and thanks to that after many many flame wars on the level list we are able to pinpoint some specific problems that were major and should be addressed and we managed to address them. Protected packages functionality is one of them and I'm sure I could come up with a couple more at least. Problem with that was that the community of early adapters covered only a small part of DNF users and therefore covered only a small part of the DNF functionality. I will talk about that in a little while and we didn't succeed to reach out of this rather small group of users who are like enthusiastic about DNF and wanted to try that. I'm quite happy about that. Most of the people that tried DNF for the first time encountered a problem they were just done with it which I can understand as a user just that we managed to get them on board and that could have gone better. So what else went well? Use case driven development that's something that we do today and basically bottom line is that we do not automatically implement what single user asks for. We try to listen to our users but only as a group and the reason is that some requests are contradictory so what we try to do is we try to figure out what is the reason what is what is the reason for the request what is actually the problem that the user is trying to solve. That will usually give us a clear indication as to what is important what is not important and why it is important it's the most important part. And thanks to this approach we already managed to identify and merge some duplicated functionalities. For example in YAM there are some plugins we're doing partially the same job as other plugins so we managed to merge it into one piece of code one place where you can call that functionality. The problem with this approach is the development process takes a bit longer because in some cases especially those cases that we don't understand or the development team doesn't understand they need to figure out what the problem is they need to understand it they need to make themselves wear shoes of others right so yeah and that takes time and what also happens from time to time that the development team can be perceived as unwilling to cooperate because instead of just getting the code in there just merging the patch just getting the stuff fixed they ask you a bunch of questions and when you answer they will ask you another bunch of questions and the reason is exactly for that they want to understand you they want to understand what you're going through they want to understand if there is a way how to solve a problem differently maybe maybe in a more efficient way. So another positive so these are actually three aspects that from my point of view differentiate the DNF development from the YAM development. So first of all unit testing DNF has an extensive unit test suite and I mean really extensive and the policy is that no code should go in that is untested or that is not covered by test suite so if there is a new code or if there are substantial changes in the old code unit tests are always required that's why if some of you send patches to DNF those patches might have been rejected because they didn't include unit tests. CI this actually shows that the testing is really important in the development process of DNF. Currently our CI is directly connected to GitHub so every pull request that comes in is tested first and then it's reviewed and if the test suite fails it's up to you to fix the pull request before anyone can actually review it. Test suite is like I said it's running every single time and it runs on all the components because it's not just DNF there are other libraries beneath DNF and they have test suites as well. Documentation we try to stress the importance of documentation as much as possible because from our point of view the documentation defines our API. You probably know that in Python you can basically take well almost anything that you want and you can you can use it you can use the internalness of DNF but when you come to us and you tell us like guys they stopped working fix that. We'll take a look at the documentation and we will say yeah okay so this is part of the documentation we will fix it or this is not part of the documentation you should not be using it. If you want to if you want this functionality file an RFE we'll take a look at the existing interface and maybe we will just take an internal method and make it a part of API or we will implement something else if it's a better choice. But the point is that the documentation is really important for those consumers that try to connect to the Python API of DNF and documentation is also important for users because DNF is a little bit different than YAM and documentation is also important for developers who want to join. So we have a nice guide how to get started with DNF and Nick over there filed an RFE to extend this guide to make the development on top of DNF even more simple and not just on top of DNF but the development of DNF itself for new coming developers. So let's take a look at some negatives. Poor stabilization goals. So even though we announced the release of DNF and the replacement of YAM quite early people didn't test as much of the code base as we had hoped and they led us to making wrong assumptions about what is necessary to make or to declare DNF stable. So originally we had only five goals five major goals which is not that many and we said okay after we do these five goals after do these after we do these five enhancements we're going to go we can we can drop it there and everything will be okay. That obviously was a huge mistake and it reflect badly on us as users got mad about important things we did not encounter before or we did not consider important because things didn't work for some of the users so this should have been should have been improved better. And another thing is that some parts of DNF were implemented quite late like the migration plugin for instance which led to problems in federal 22 alpha or beta was it I'm not sure at this moment. Basically these fun these functions did not work like at all because they were untested we are under a lot of stress and we didn't manage to get them done on time. Okay some other negatives outgoing communication and this is very important one this is basically why I'm doing this presentation. So we figured out that this needs to be improved in general but mostly in terms of transparency of the development process for the community because we in our development process we do a lot of decisions and some of those decisions are not clear to the community and to the users so we need to come forth and we need to explain why are we making decisions the way we are. So examples there are examples throughout the last year like what is the reason for for the YAM and DNF change what are DNF's goals and priorities what that it is actually going to happen because to the very last moment some people didn't believe that the YAM was getting replaced and yeah I mean some of the design decisions were not communicated properly which led to all sorts of confusions I will cover some of them later. So let's take a look at DNF 1.0 so this part will be about what was the development all about around a 32 GA. So first of all it was all about migration of application so this was one of the things that we actually started pretty late in the development cycle because we did the initial inspection in January and it turned out that there is a lot of stuff that depends on the YAM big surprise huh oh look what I've known we should have I know. Anyway so our our idea was to first of all migrate all the stuff that is in the base installation of Fedora and by base installation I mean all three products. So we identified all the components in all three products including including the base and yeah three products and the base and we told we told ourselves okay let's start with these let's make sure these use DNF instead of YAM and if they do we're pretty much good to go so the goal was to not have YAM in the standard installation of Fedora. The second step was to migrate all the YAM plugins. Turned out there is a lot of YAM plugins some of them are not maintained anymore some of them might be doing some stuff that no one no one cares about I think there's two of those so yeah that was the that was the second step the first step the third step was and still is to migrate all the applications that are heavily used but how you identify those well turns out our users know and our users tell us that something should be migrated and as a yeah as a fourth step the idea is to migrate all the applications with active and approachable maintainers and obviously the rest of the application the rest of the applications we need to solve that on a case-by-case basis if it's possible to solve I mean those applications with that upstreams and like maintainers that don't care much it's not really much we can do about those please not in the foreseeable future so YAM deprecated to this day I think that was that was a good idea to have this one even though some users on Fedora levelists didn't think so the idea of YAM deprecated was to have basically a simling from yam to DNF similarly all those like all those tools like repo query and stuff like that should still be there in case you need them for some reason because you know DNF is not 100 percent compatible with YAM and it can happen that someone relies on some functionality that is not yet migrated DNF and it's not going to be for maybe for some time I don't know so the idea was to keep YAM a little longer in Fedora and see what is critically missing in DNF the way we did it you probably you guys probably know we basically warn users that the old YAM commands are deprecated and DNF should be used instead if possible and then we try to redirect to DNF so yeah this is because we want to push users towards DNF a little bit because well who knows how long YAM is going to be around well I know I hope I know but I will share with you in a little while so yeah and the plan is to keep the YAM deprecated only as a backup not to add too much functionality provide some critical fixes I mean since then we implemented a bunch of features but only only those that made sense and yeah part of the part of the YAM deprecated yeah a part of this plan is to have a migration plugin and one page YAM to DNF basically the migration plugin is supposed to migrate your YAM database to DNF because there might be some data that is valuable to you and the man to DNF man page YAM to DNF is supposed to give you some ideas to what is the new command that replaces the old command that you were used to something like that or cover some config options I think as well and yeah one thing we did to improve the situation we created a special set of provides in DNF plugins basically these provides are supposed to give you an idea if you're trying to try to use a command that doesn't exist because some commands were moved from core YAM to one of the plugins in DNF DNF will give you a hint like try to use this provide try to install this provide that might give you the functionality that you requested okay so YAM versus DNF major differences I guess the message here is that still still there are some you some users who think that DNF is supposed to be a drop-in replacement for YAM I'm not sure why do you think that we have never said anything like that we're trying for the CLI interface to be as similar as possible but not always the same so DNF is not dropping replacement for YAM again we should have been more clear about this and yeah basically we should have we should have made clear that DNF is not YAM and it's never going to be but on the other side on the other hand we have a great documentation we have a FAQ you have the main page I original I already mentioned and we have a web page with all the differences or well most of the differences that are between DNF and YAM and all these are supposed to help you with migration towards DNF if you ask why is YAM different than DNF for most part it's because of the use case orientation of the DNF but yeah there are some other reasons but this is the main one so let's take a look at the let's talk about the incoming box so Fedora Alpha was basically Fedora 22 Alpha was basically the first point of contact for most of the users so between Fedora Alpha and Beta we got 93 new box for DNF and related components between Beta and GA we got 115 of those and since GA we've got a hundred more and we have closed since the alpha released we have closed more than 400 box zillas either with fixes or as duplicates or as notabucks or as won't fixes but we try to maintain this number as slow as possible so here is graph if you're interested this is the this is the amount of box that we that we got this zero right here that's the first week that's the first week of this year so here you can see how many box were open against DNF every week of this year including the trend so I guess because of the holiday the trends the trend is now trend is getting down here is about September October hopefully it will not get worse yes that's across all releases all right despite that's that huge spike that's F 22 GA okay there was a there was a huge problem there because in preference to GA there was a release of DNF that was like a month old and we got a lot of duplicates so that's the spike you can see so what do we plan finally getting to the interesting stuff so the big topic obviously is going to be migration or the rest of the migration and then I will reveal some other plans as well so what do we plan removing am big surprise we want to remove him yeah so this is likely something that you are the most of you are interested of interested in and why are you attending this presentation so yam has been in maintenance mode for more than 18 months now that means only a small number of bug fixes are getting in and only a small even smaller number of new features are getting in currently we have one active young developer and maintainer who basically maintains mostly real stuff some of the federal stuff as well and James over James over there is contributing from time to time as well so it is not completely that but it's very close to so let's take a look at three important aspects of the migration to DNF so first of all it's going to be about migrating the users so yam deprecated is staying in the door 23 even though we originally we're all originally thinking about getting rid of it but yeah yam deprecated is staying in federal 23 and if everything goes well it will be removed in federal 24 completely if everything goes well I mean there is about 50% chance that this is going to happen in federal 25 but obviously the goal is for us not to have yam in rel 8 by that time not sure what which time it's going to be but by that time we want to get rid of it at least it can still be there but not as a dependency of anything else so there are some important issues that we are currently tracking the new DNF release was actually built today earlier today so it fixes about 20 20 important bugs and it has a few enhancements as well and we're hoping to get all these important issues we have there is about 8 of them I think resolved by federal 22 federal 24 alpha so the second aspect migrating the applications so currently there is 20 applications to go there is a tracker bug in Godzilla out of these 20 we send patches to 10 so we are hoping that the maintainers will pick them up and merge them upstream in Fedora and two more are likely to be dropped I mean two depending packages that leaves us with eight applications remaining that need to be migrated from yam to DNF that's likely going to happen this autumn migrating plugins that's an ongoing effort I think we are pretty much done but there might be one or two plugins that still need to be migrated obviously some plugins are missing the full compatibility so we work on that as well like we're talking before the presentation and repo query is one of them it's one of the more essential plugins and then there are non Fedora applications which we obviously can track that cannot track at least for the most part so I know for sure that we send patches to Ansible to migrate to DNF and rest of the applications and their developers we are willing to provide an assistance they require and like I said before we have great documentation for them third aspect migrating the infrastructure and you release engineering guys over here none okay cool so bottom line I will I will talk to you guys about the plans for migrating to DNF currently the plan is to migrate to well yeah to migrate in Fedora 25 time frame not sure why exactly that's what I need to talk to you about basically the building infrastructure so Mock is already ported to DNF then there is Koji which uses Mock but that is not ready to adopt DNF but that should be fairly simple if I understand it correctly and then on top of Koji there are some other tools like Pungy I'm not sure what is currently used if it's if it's Pungy as well but the release engineering has all these different sorts of tools to make create composes and stuff so yeah so that's about migration let's see about some other shortened plans so API improvements for plugins so as the as the site of plugins that we have grows we discover on a weekly basis that there are some parts that are just not good enough for plugins like how the configuration is processed I can remember a conversation between the original DNF developer and James over there they were basically arguing about how the configuration management should be done obviously the DNF the original DNF developer did it his way and surprise surprise we discovered that it might have not been the smartest idea so this is one of the examples of things that we need to improve in the future and like I said currently we're quite okay but if we if we want more plugins to come out of the way you need to work on the API downloading file lists so who noticed how much data DNF downloads every time okay at least one person yeah the reason for that that the young was optimized not to download file lists by default and DNF does that so this is something that we will we will definitely work on because there are still some some deployments where bandwidth usage actually matters I mean it makes the DNF dissolving faster but at the same time like I said there are some some places on the planet that this is actually a bad thing so we will work on the bandwidth optimization yes probably I mean I can I cannot say that with certainty but the differences I don't have any stats but my first guess would be that it's about twice as much because the file lists are oh right no I don't have any stats on that yeah I mean the thing is that when these metadata are downloaded from the servers DNF needs to compile them into some sort of format that the dependency solving library uses so that's that's the reason basically oh rich dependencies that's something that should be available in federal 23 even though I should not advertise it loudly because it was not sanctioned by the FPC but theoretically rich dependencies should work what are rich dependencies about basically they allow you to use boolean expressions in your requires suggests recommends you know so those are in now to the point where we actually got because it wasn't that long ago that they weren't in so yeah so the last asking what the syntax was yeah okay yeah so I mean no that's for week dependence yeah yeah I mean the syntax is like in its alpha phase that's why we don't recommend to use them there are some expressions that are still missing like negation negation and yeah other than that it works but it's still like in an alpha development phase which we hope it should be done by by federal 23 GA but it's possible to start experimenting with them yes we can we can definitely meet and talk about this so subscription manager search disabled free pose plug-in not sure how valuable is that for Fedora if you're not using spacewalk or yeah that's not even in yeah it's it's possible anyway this started as a row feature but it's open source in the arms so we plan to port it to to DNF as well the thing is that what can happen sometimes is that you have some enabled and disabled repositories and when you install a package from one of those is enabled with repositories it has a dependency that leads into one of those disabled repositories so what is plug-in yet what this plug-in does it basically asks you if you want to enable all the repositories that you have in your configuration and start a dependency solving again and if it satisfies the dependency it will ask you if you want that particle repository enabled by default yeah basically it's supposed to make your lives easier if there are broken depths so some long-term plan long-term plans software database and integration with cash the cash integration I'm still not sure if that is a if that is a short-term or long-term plan because yeah we basically want to improve caching yum has improved that recently and we want to bring that to DNF as well the idea is to have more versatile caching mechanism that will allow you to store your packages without the need of like proxy server or anything like that that is especially usable in in Docker environment when you're building machines using the same packages again and again and so for database that's something we have been planning for quite a while the idea is to separate the database component of DNF create an API on top of that and basically make it available to other applications like package it because package it currently doesn't share the data with DNF okay oh yes that's that's one of the plugins that we are that we are currently working on the security plug-in and I was under the impression that's already in Fedora maybe with one of the latest updates I'm not sure about that but we definitely know about this feature and there is already work underway to make this available in Fedora so yeah for the recording the question was that one of the big missing features of DNF was the security update a security plug-in sorry so let's take a look at some very long-term plans these are mostly ideas at this point not no work is being done on those and yeah addressing the three different software management stacks problem so currently we have young we have DNF and we have package kids slash non-software which basically work well independently so what happens each of these each of these applications will download its own metadata so all the metadata that you download you to download them three times if you use all three of those there was a huge threat on Fedora deval list about this about two or three months ago so that's something we definitely want to address but it's not just about a metadata it's also about the business logic so we have young and we have DNF now that's gonna be that's gonna be resolved sooner or later but then there is package kit which basically re-implements everything that we do in DNF and GNOME software is running on top of package kit so DNF and package kit are not exactly like what you would call compatible for example you cannot use DNF plug-in package the business logic might be a little bit different even though the core pieces the libraries like the depth-solving library and the library for downloading data are the same the business logic on top of them is not the same so we want to address that we want to merge some of the functional some of some of the functionalities that we have and preferably utilize the same library that will contain the business logic transformation of the code to C or C plus plus there is actually a project called tiny DNF that was started by VMware they their idea basically was to take our core pieces the depth-solving library the library for parsing comps the library for downloading the data and writing their own command line utility on top of those libraries so again they duplicate our business logic well footprint it's all about footprint because VMware wants to use that for their container solutions and you know with DNF you still need to drag Python with you so that's why we consider transforming DNF to C or C plus plus in time like I said this is very long term plan and yeah demonization that's another idea floating around if DNF was running as a demon say I don't know like Docker it might be possible to do all bunch of stuff that is not possible today like scheduling tasks for later for example or I don't know managing non-root users and their access rights so non-root users might be able to install stuff oh yeah I mean yeah I guess it is but there might be other other use cases I mean I always hear people talking about non-root installable packages so yeah I guess something like that I mean currently well I'm not sure what's what's the deal with package kit but I'm pretty sure that for example minimum software which is I understand correctly our default way to install stuff on federal desktop only only displays those applications that have these descriptions which night which might not be all of the packages so but again this is this is just an idea and we're still like thinking through if it's really a good idea or not okay so how much how much time do I have like five minutes something like that okay so I was going to get a little controversial but let's see if there are any questions before we go there okay okay so here's here are some topics that we can talk about so first of all the name people keep asking why is it called the NF is just a fork of young so why don't we keep the name and just release young for my answer is that DNF is not young that's why it's called differently the API's are are different the CLI is a little different and at this point it's mostly about managing user expectations the major counter to that right is that all the users that understand how it's not young are all the ones who whatever it's being as well understand when it's not young when all the ones who don't know or care that it's not young will never understand and have different letters right so the comment was that the users that don't know why it's not called young will never understand why it's not called young right yeah and they don't they don't care so yeah you're definitely right and that's why we are currently keeping the sim link and the sim link will still be there for the foreseeable future so because they just wonder you know young commands to work they don't care about the name that's not just that follow descriptions and things from websites that's never exactly I would like to encourage you to think about making that young sim link not tell people to use DNF and to make it smarter about if it's something that it's pretty sure it's just gonna work just do it don't tell them yes that's like and I'm a good example of the system in for years and lookup years and years told you and that's lookup is deprecated you should use digger host and the only result of that was a bunch of system in angry about that message and I think that's lookup and complaining more often and I think right yeah same result here so I think we should just right so that's so the comment right yeah so that's so the comment was that also the funny thing is you probably should switch to host but and that was one of the things I thought this is a little bit intended but it's like a usability thing people saw that switch to digger host and switch to dig and then the plane that dig was way the wrong command and it never noticed the or host apart so so yeah for the recording besides a lot of off-topic discussion the comment was that we should not do just the warning to migrate to DNF we should do like automated translation from yum commands to DNF commands instead where possible and question to that is yes we know and we plan to do something like that because that's I mean if the use case is as simple as installation of one package there's no need for the user to you know learn a new command so I agree with you and we will be working on that so yeah the question was about Ansible yum module and I don't think we're going to work on that but feel free I mean you're on the team so oh right okay so yeah right yeah so anyway we're we're out of time so one last question it will try to redirect you to DNF in terms of like taking the arguments that you specified and calling DNF with those arguments but that will not work every single time there are times where you will end up with an error or something okay so anyway if you guys want to talk some more I will be available just grab me walking the hallway and other than that thanks you thank you for listening