 Hello, everyone. Thanks for joining in. So here goes the penultimate talk for the day in the UX track. So the talk is titled My First Year as a Fedora Packager by David Duncan. And the talk is pre-recorded, so I'm going to share my screen here and also post a link to the talk in chat if you want to view it on a separate browser screen. David is here with us today, so please feel free to post your questions in chat. And I'm guessing David would be happy to take it at the end of the session. Hello. Welcome to my talk here at DevCon for US 2020. My name is David Duncan, and I'm going to talk to you today about My First Year as a Fedora Package Manager. Now, it's the end of my first year, and I still feel like I haven't gotten anything done that really makes me feel like I've become more advanced. I'm perfectly willing to try packaging on pretty much anything that you bring to me and to do the work that's necessary to get the help that I need. And when I decided to do this, I was at Fedora Flock in Budapest, and I was talking with some of the people there and was super excited about the idea of getting more involved with Fedora and the Fedora engineering side specifically so that I could, one, grow my skills and two, be a beneficial member of the Fedora community outside of just being an evangelist and letting people know that I was excited to have Fedora on my own desktop and my own cloud systems. So what did I sign up for? I was talking to Kevin Fensie, and Kevin said that he had rolled his own version of the AWS CLI package, and he, well, it was derivative of someone else's, but he was working on it, and it was something that he was keeping up updated so that he could use it. And he said that I'd really like to have something, you know, like to have it off his plate. And this, my years picked up because I work at Amazon and I said, you know, this would be great. I have other high visibility requirements around leveraging the AWS CLI in the fencing agent and would love to, you know, to bring that into line. And so I was envisioning this space where I'd already had a lot of the groundwork laid by a senior member of the engineering staff and would be able to sort of guide this through in the process and learn a lot more about packaging. Super excited about packaging, and that was something that was a big deal. So with Kevin's help, I set out to become the the packager for AWS CLI. Now, this is my first package, and so I looked at it, and I thought this is something I depend on. And it's something that is an important part of my daily work schedule. So it's something I felt like I could, I could really get a much better understanding of very quickly. I wouldn't have to worry too much about what I had to do or how this, how it worked or where the, where the developer sat because I know that they sat sort of in the same company that I was sitting in and that we would be able to have a close communication or I'd be able to, you know, reach out and bend their ear and in short order. But what I didn't realize when, you know, when Kevin handed this over, he said, oh, yeah, this is kind of in lockstep with the BodoCore, and so I should probably just give that to you too. So one package became two packages because you couldn't update the AWS CLI without also updating the BodoCore package that it depended upon. So now I had two packages that I was working with, and that was a great place to start. I felt like I had this kind of understanding of how the overall I was, you know, had sort of a microcosm of the ecosystem where everything fits together and we understand each other. So here I have this very simple sort of controlled environment, and I thought that was super exciting. So getting started, the first thing that was important was finding a sponsor. You can't just jump on the packaging mailing list and say, hey, I'm just going to, I'm just going to package this guy. Okay. You need to make sure that everybody understands that you're you're interested in packaging and you want to be a part of the part of the Fedora community that's doing that work. And they will provide you, someone in the devil world will sponsor you for your participation. And usually that's someone who's interested in helping you to get a little bit better at what you're doing. So that's so they become and I would encourage you as someone who's looking into packaging or thinking about packaging to think about this as someone who's an important member of your team, your goals, your responsibilities are something that they're saying that they will take some degree of responsibility for. I mean, not in a way that any of us are in super trouble, but in a way that makes sure that we are getting what we need to get our work done. So that means someone who can can really help you. So choose carefully. In my case, I did I feel like I did choose carefully I had someone who was encouraging and part of the Fedora engineering team and, and had been doing this for a number of years, but also was successful beyond the years that he was that he had been packaging. So it was an opportunity for me to learn from someone who really rocked a lot of the component parts in ways that helped to build a bigger understanding of the ecosystem. So, you know, you'll learn a lot. You'll be amazed at what what tools are available to you. You'll see some of the things that you know that sort of behind the curtain in ways that you didn't see before. How the builds are done and how you're how the reporting tools talk back to you. But then at a certain point, you know, the focus comes off of your sponsor and your sponsor kind of leaves you to grow on your own. And that's not an important part of the growth period. I'm not saying that they're just just abandoning you but but to say that they're giving you an opportunity to seek out and and apply understanding as you as you see fit. And this is an opportunity for you to engage with the with the community overall. And I encourage you to be transparent about what your abilities are because there are lots of people out there and in the community who are willing to help you. And I think that it's important that that we as members of this community go on helping each other to be at our most vulnerable and to learn as much as we possibly can in a safe and environment as we can create. So I'm challenging you to be a part of that and make yourself, you know, extend yourself out to the edge of your understanding. So, yeah, the first experiences that I had around packaging happened in in a rapid fashion. I was not expecting to take on when I wasn't expecting to take on two packages but when I did I thought yeah that'll be good and I'll get a gain a better understanding of the ecosystem. And to that that meant that I would be able to take this opportunity to to understand more about that microcosm that I that I was participating in. Um, and so very fast very quickly that my chosen applications were starting to reach out into other requirements right and so I looked to the documentation, specifically the packet guidelines that I found I found to be absolutely a necessity in building my experience and understanding around making sure that I was doing this in a way that was going to be accepted by the fedora community had already been accepted by the fedora community. But the software that I was working with had a deeper reach than I really understood and there were a lot of other packages a lot of other components that were dependent upon some of the internal requirements that I had. And the AWS shell for was necessary for the for the AWS CLI but then the AWS CLI was also necessary for some other components like the AWS shell and we had there was another gentleman who was who was packaging the AWS shell and and he had specific requirements. And now I was starting to see that I had some downstream collaboration requirements here and needed to really understand better the collaborative requirements that I had to find those associated software and engaging requirements and to ensure that I was not only providing what I needed for for making sure that the AWS CLI was fully functional, but also the dependent projects were capable of taking those changes without having any kind of adverse effect. That led to me getting a lot more familiar with the downstream process but then kind of behind the scenes, there was more going on like I said there was a there was some go to market strategies around the fencing agents for high availability that were being integrated into the redhead enterprise Linux so the AWS CLI was starting to grow in its in its enterprise level or enterprise Linux level integration behind the scenes and and so I was listening to this sort of the heartbeat downstream from where I was but then upstream there were these other things that were happening that were more specific enterprise goals that needed to be paid attention to. And one of the things that I realized early on was that I didn't have a I didn't have the ability to just blanket build these packages in a simple fashion. And when I started asking around about automation and processes for automation, the answers that I got the responses I would I received back from other packages other people who were doing work was that they had their own bespoke process for tooling to ensure that their packaging pipelines were successful. And that meant that they injected specific types of manual process in where they needed to but they didn't necessarily have didn't necessarily have a process that fit into a neatly bundled stack that they could then hand over in a way that made a pack another package are capable of more successful. configuration. So, a lot of a lot of time and effort went into building these packages in a manual way early on, and building that tooling out took time. It took me a lot of time because this is when this was not somewhere where I had a tremendous amount of deep deep dive experience, but I was willing to learn and I was super excited about that. And this was a, this is a picture that frightened me right off with the AWS CLI so if you know anything about the way that the Amazon teams do development they really love to land sort of tip of the spear, which is great when when you're running something that has a very short life cycle, but for things that have long, long term life cycles like red hat enterprise Linux and the, and the multitude of versions of fedora that may be available and same with the, you know, being able to support the the CentOS packaging in the, in the, in the long term, you're required to kind of create this, you know, this branching process for each one of those within the context of the, the, the disk it server. And, and so just get became something that was really important in a, in a big part of the, the, the practice so. And so now, as, as I'm finding more and more responsibility for what's upstream and what's down, and what's downstream. I'm starting to feel like I have a responsibility to do something that had been entirely different before so when I first started working on this package and the AWS CLI. One of the things that was that was necessary was that it be available but it didn't necessarily have to have any kind of static existence in a specific just get branch. So for fedora 30 or 29 or whatever, I didn't have a raw hide. I didn't have anything besides the bill for raw hide, because there wasn't any requirement to really reach out and do significant amounts of testing it just needed to be available so that you could, you could use it in the standard systems. So now with, with hat with a with a requirement to have full support for for the longevity of a of a of a red hat release. Things became a little bit more difficult and there I started to get more feedback on the process and to hear more about what was required. In learning about how to build these packages. There's a large amount of documentation, I feel like this is out of order. There's a large amount of documentation that needs to be needs to be concerned right like so once you're kind of left to your own devices by your sponsor, you're, you're, you spend a lot of time reading and and concerning yourself with how these things work. I spent a lot of time reading the documentation around rpm and around different tools in the fedora infrastructure. And what I found was that many of them were either retired or they were task specific or a lot of the a lot of the details had been had been related from one sponsor to sponsor and in a way that was very specifically help beneficial for what for what they were working on at the time. But it didn't necessarily translate into something that was a an overarching mandate. So a lot of the protocol a lot of the decision process became this sort of living document for how how you interact with the product with the other developers, the other packages, the, the build systems, all of that had some metrics some reporting a lot of a lot of feedback, but there wasn't a whole lot of there wasn't a whole lot of guidelines for how that how it's supposed to work. So I recommend if you're if you're doing this work that you spend a lot of time looking through the RPMs that are available within the distribution and looking at sort of, you know, looking at the source upstream, but then translating that back into how it relates in the in the package maintenance for longer term support. What are we doing in Apple to make sure that this is this continues to be available and that we're fixing bugs. And to have a package that is in Fedora and has a has a division of the versions, and yet has some consistency in the operations. How do I get that and what does that look like. There's a, there are lots of those questions that are more difficult to answer than you might think, especially if you're not a strong developer in the languages that are important that are sort of core to the to the work itself and you're more looking at the operations and trying to apply with help, what needs to be done. There's a. So, so obviously, when you're doing that, you're looking for your that collaborative effort and you're, you're working on those, you're working on building up that body of knowledge you have to you have to go somewhere to where things are the way they are when you don't have a clear picture from just a, you know, one or two line changelog. You, again, this, this is a very welcoming, very, very inclusive community and it will continue to be that way. And so I want to encourage you as someone who is learning to put a lot of effort into, into making a good understanding of, of where, where you're falling short and how they might be able to help you move forward in that direction. A lot of times there, there might be some, you know, there might be some big gaps and you don't have to worry about that. You don't have to start in an area where the gaps are, are, are small. You just have to start. And then from there, your commitment, your experience and the direction that you take as a result of that may not be in the direction that you originally thought. But eventually you'll take on the responsive, you know, you'll take on responsibilities and in a way that is, that is consistent with what your commitment is. And I encourage that. Don't, don't think that you have to suddenly become king of the hill. So there's lots of places where the feedback comes from and first, firstly, there's bugzilla and this is somewhere where I was most familiar with interacting with the engineers who were working on packages was in the place where who file, file an issue and they return a response and request more assistance with stimulating failure or creating the exact complication in, in a controlled way so that they can help, you know, they can help to identify how to fix that. And now here it is your turn to be the person who is asking for that. So you can look for those consumer concerns, ask for those reproducers or build the reproducers yourself in ways that are, that are beneficial to you and to your, to your audience. And then the new hotness will, will is a, is a, an alert system that is built around monitoring the upstream community, the upstream tags and, and repositories with or the tags in those repositories to find, find out where you are in terms of the latest version and how, how you're building that, you know, gives you an opportunity to, to have an associated build process. Now, one of the great things about all of the bugzilla is that a lot of the build system and the tags that that you use in the change logs and whatnot are capable of executing tasks or stimulating tasks in the, in the build system to provide more detail around the, around the state of a specific associated bug. So like the new hotness will file a bug saying you've got version 1.18.2. And then you can create a change, a change set that is associated with that bug, and a change log that has the bug identifier in it. And then you can build that in the build system, which will then tag that system to tag that ticket to identify that the bug has changed state. And once you have that change state, you, you can, you know, it'll change state on QA and then once you have enough karma to release that package into support, then the, the, the bug will be kicked over to the, to complete it. So those are some great, that's a, that's a great kind of automation that's available to you and I want to make sure that you know that that's out there and, and possible for you to take advantage of. Another thing that's, that's great is the automatic bug reporting tool, and that files automatically on behalf of consumers who submit those crashes, so that they can, they can provide better and better detail around, around your, your problems and that aggregates really well into, into sort of a base metric to tell you where you need to focus your energy. Lots of things go wrong when you start to try and, and target specific versions, specific releases in this, in that, in this crazy disk at world, and this is an opportunity for you to get some strong constructive feedback very quickly. And then my favorite form of guidance, and early on came from a couple members of the community and one particular I think is a great leader in this approach is, is Justin. Justin Florey and Justin is obviously a council member for fedora and he's someone that I think has been in this community for a long time and built up a strong concern and a strong governance around code of conduct and making sure that this is a place where we can all participate without being clobbered. Because there are various skill, skill levels here and, and it is important that we think about, you know, we consider those skill levels when we look at how it is that we interact with the program. And then there's the proven packager. So, some of the reason I talk about that sense of tolerance, a sense of outreach and that that concept of having a pull request associated with something. So early on in my experience around the around building packages. I made some basic mistakes and one of those mistakes was was the way that I had formatted a specific date in in the changelog. Well, it was a mistake and it needed to be corrected and need to be corrected over over a series of the changelog entries. And one of the proven packagers took the package and reworked the the information that I had incorrectly put together, and then, and then submitted it, then pinged me in IRC and let me know what I had done. I went back to my sponsor we talked about and then he showed me a couple of things that I was that I was missing about some tools. One of those being our spec. And so some automation was born around that and I, you know, that really led me to a great experience. But like I said in the, you know, in the early part of this conversation, there were other things that were going on too. I'd gotten some bugzilla reports from from downstream use right like the people like one of the member who was who was packaging AWS shell. Basically moved on to well came to me after he had determined that I was moving a little bit faster than and was was possible for the for the shell team turned out they had stopped most of their development so it wasn't wasn't surprised, but he had some specific requirements around the version of pi ammo, I think. And, and that that requirement was that that it be that we loosen the constraints that the AWS CLI upstream team had placed on their their packaging. So he submitted a he had submitted a poll request to with a patch to relax those dependencies in the in the requirements configuration for the pi pi package. And that made it possible for me to apply the patch relax the dependencies and then make the Python build using the standard setup tools in the in the Python build. And then he could use it. And then of course that set off a chain reaction because one of the reasons that they had constrained this particular package was because it caused an issue in the documentation so customers who are consumers who were using the AWS CLI and then relying on the the use of growth in their in reading the documentation were disappointed because now they could use the AWS shell but they couldn't read the documentation. And that made it very difficult for other people so there was a lot of a lot of conversation around that and we made some modifications. Meanwhile, the rail engineering team took up access or took up the support of the Boto three in in rel. And as a result, they were dependent upon the version of Boto core. And this did not trickle down to me after I mean it didn't trickle down to me while it was in in test. And I want and then of course they filed a ticket to close off the the access to Apple. And when I looked in the Apple, I remember looking in the Apple package and finding a change in the in the way that the package was identified and and the the disk it branch was closed. And and I didn't really understand why but I talked to Kevin about it and and we had a nice chat saying okay yeah there was a bug filed by the red hat team and they were they're planning on having this as a part of the standard rail so it doesn't need to be in the extra packages any longer it will be in the standards. The source source RPMs that it's that it ships with so this was a great experience but then I found that you know of course I was continuing to build in the way that I was and I had built out some tools and now I was getting a little faster at doing the process, but there were a lot of things. I think, well, I'm sure that there were a lot of things missing in what I was doing because one of the proven packages made a decision to to execute on rebuilding the AWS CLI and rebuilding the the Bodo core together, along with Bodo three. And so all three of these packages were being consumed in in rail engineering. And there was a tighter coupling of the process that was that had that had started to take place in terms of their ingestion so they wanted. They really wanted. It's very clear to me that although I've never talked to him about it that's very clear to me that the that what they were looking for was a consistency that I was not providing in the way that I was doing doing the general packaging. So I found one day that the build system reported back to me that one of their builds was was completed. And in doing so, I found myself looking at a package that didn't look or looking at a spec file that didn't look like the ones that I had been creating it was much more you know it had had a level of sophistication that I wasn't was he yet capable of and so I was looking looking at it and I was interested and so it was like what are you you know what are you doing on what are you doing on my package I can't believe you're working on my package you know it was sort of what I was thinking immediately and then when I looked at I started looking at the package and I thought I thought oh wow that's oh that's smart yeah oh I would have never known to ingest that specific patch to fix this specific bug. The way you did this here this is interesting and it comes directly from the upstream and it looks like you have a one to one match and there's a seems like a close communication wow yeah I could really learn a lot here from from watching how this works. And so, you know, a lot of that effort on my education around building these packages started to and and being sort of closely connected to these these packages. was starting to dissolve into this passive relationship with the work that was being done around the packages that I felt responsible for but seem to be. But seem to be watching as other people more senior than myself started to push real, you know, interesting changes into and and making Megan headway. So, so I felt like that was the right answer in the under those circumstances was for me to kind of accept all this help and to see it and I hope that you know in in the same experience. You would feel the same way that that are, you know, I think, I think this is a positive experience right in the sense that that at first I thought that maybe I was missing something in my and they were they were missing something in my guidance. And then I realized that there was a long way away there was a long way in between our body of knowledge that they had in terms of packaging and my body of knowledge for packaging. And then it might be very difficult for them to spend a lot of time trying to push this through for me when they had goals and deadlines that were associated with go to market strategies, far beyond what it was that I was trying to do and you know in the, in the context of of the, the community experience. So, but, but still, all of the work that they were doing was still being reported back to me so I was able to, you know, still able even today I'm still able to, to maintain and manage the, the reporting and ensure that if there are corrections that need to be taken, then I'm taking those if I, you know, if I need to make modifications to package itself to ensure that things are happening like suppression of the v1 information versus v2 I can I can do that. And they also can take what I've done and remove the parts that are offending right that are not passing the quality control. And they can, they can rebuild those packages so that they're in their consistent with the requirements that they have for upstream for downstream quality. So, this is an interesting question now right because now I've got, I've got sort of a looking glass into these two packages that I that I started with the beginning of the year, and I was super excited about dealing with them right in the in the in that first section. My sponsor, you know, has been sort of off but then now I have many more I mean there's many more people that I talked to you on a on a daily basis doesn't. There's not one person who needs to be responsible for my efforts on it on on any given day. But I'm still looking in on the packages that that are, you know, I originally felt like I was responsible for and and I'm not seeing the work be done so. Guidance of governance started to become something that I did have kind of a sense of and the AWS CLI was a perfect, a perfect spot right so so what was going on was we were, we were working on the AWS CLI version one and version two came out. Well version two was exclusive from version one so there's there were some component parts that were shared in terms of the abstract components but then the version to itself provided for a lot more flexibility in terms of the of the build process for the participants within the context of the greater AWS ecosystem. And what I mean by that is that it was no longer in Popeye. So, this version two was no longer available to customers in a way that was consumable so I turned my efforts there, and then started to make a lot of those. So, the effort that I pushed into building relationships with the upstream community and the upstream maintainers towards helping them to make decisions about how they were going to build out their open source program and make this possible to have a better developer for their customers and and consumers I mean in this sense we are we are a strong consumer of the of the package itself right so or the application itself and so we want to package it in a way that is consistent with the distribution experience, but if it's not open source or the component parts are not open source, then suddenly we're just doing the framework and the plug ends don't really matter because they they don't come from somewhere independent they were coming from a single source. So it can't separate out separated out into what's open source and what's closed. When they're when systems are being delivered in both binary format and non binary form and just in time compile for. So, trying to work on that availability became a big focus. And you know there was a little bit of a struggle there to I mean I was looking for other packages and I just kind of reached out and different directions and tried to figure out if I could do things that were mildly interesting and as and and related to the environment that I was most closely supporting which is the Amazon EC2 environment and other other utilities. And yeah I still own these two packages but that maintenance is an ongoing problem for people who are more directly involved in the in the in the day to day maintenance of the of the rail engineering. And so now I'm building my relationship with the next generation tooling groups and hoping to create a scenario in which we can better we can better consume those into the into the story distribution. If you're not a package maintainer now and I thought I'm I fully attended to write this in a way that was encouraging for people who aren't package maintainers start with this wiki and and walk through what's going on in the wiki but don't make it. Don't make it necessary for you to understand every inch every component part, every skill, every exception that's listed and sort of detailed there. Just jump in let's get started let's have some conversations. If you feel like there are people who are way more advanced to seem to think that there's a lot more ground that you need to cover. There are people like me out there who are intermediate and can help more, you know, along those lines. I'm more likely to be on IRC than I am to reply back to the appropriate devil list. So I would say reach out in both directions depending on where it is that you're, you know, that you're that you're most comfortable and then wherever it is that you're most comfortable. Make sure you reply so that you're replying in an appropriate manner so that so that you can take part in that. And then the other thing is as an upstream contributor you have there's a lot of things that are being done in terms of the packaging that we want to make sure have an upstream first kind of goal. And that upstream first goal means taking the testing frameworks that are that are necessary for ensuring that the packages are fully functional and getting those into the the systems for the for the application itself. So as you can see that in terms of code coverage and you can verify that those switches and the appropriate parts that are there for supporting the distributions that are supported. Are there in the package in the upstream application itself, not just somewhere down below with no concerns over over how that affects the overall deployments so I think yeah that's the that's my advice on getting started. And I just want to say thank you for coming to the talk and thank you for being here at DevConf. I think this is one of the greatest experiences that that I have had is being at DevConf and I miss the camaraderie and the hallway discussions and the ability to have that close interaction but I'm grateful for the hop in experience and this makes things super easy for us to enjoy some time together. So I'm really grateful that you're all here and glad to talk to you about anything that is interesting to you about packaging or about getting involved in Fedora and just in general. Even if it's just to become an ambassador and to talk about Fedora in a way that is consistent, I think that's very important. Thank you so much David for that amazing talk and I think what you said at the end totally resonates with how we are all feeling right now. Thank you for being there virtually with us. We really appreciate it. Okay, did that work? Yes, that totally worked. I can hear you now. Yeah, did anybody have any other questions? It's definitely time now. I don't think it was very difficult. In fact, just asking on the mailing list, you'll find somebody who's willing but I think it's important to find someone that you enjoy working with, right? And that was something that I thought was difficult was finding the right person and the right balance of not being too annoying to someone and just because of general familiarity. But then them having the same, you know, like an ability to help you bootstrap. And it really is right. You don't want to be, you don't want to make someone responsible for your, you know, for your learning experience for the rest of their lives. Ben, I would pick the documentation. It's really difficult to figure out what tools really exist and what tools don't exist. That totally makes sense. I think Ben has a question on the chat. If you could fix one thing about the process, what would you fix? Yeah. Documentation. Yeah, right. Yeah. I mean, documentation is engineering. So I fully take responsibility for that. So, pound action, dab dunk will. All right. Thanks everybody. Thanks so much, David. I will also post a link to the breakout room if anybody wants to continue the conversation with David. Thanks. Thank you.