 Welcome to Design Up Front, a panel discussion on socializing ideas with enhancement proposals. My name is Adam Kaplan. I'm a software engineer at Red Hat and a maintainer of Project Ship Rate in incubating open source project under the Continuous Delivery Foundation. Our project uses enhancement proposals called SHIPS to discuss new ideas. With me are open source leaders who also use enhancement proposals in their projects, Christy Wilson and Kirsten Garrison. Hi everybody, I'm Christy Wilson. I'm a software engineer at Google and I lead the Tecton project, which is also a project under the Continuous Delivery Foundation. We have an enhancement proposal called Tecton Enhancement Proposals or TEPPS. And I've been working on the project for, I can't believe it's been actually three years now. Hi, I'm Kirsten. I am a senior software engineer at Red Hat working on OpenShift and I am an enhancement sub project owner upstream in Kubernetes. And I was also in the release team many times and I was an enhancement lead. So we're all using this thing called enhancement proposals and many folks watching this may not know what we are talking about when we are saying an enhancement proposal or an EP. So Kirsten, since you've been using CEPPS for a long time in the Kubernetes project, can you explain what these are and how Kubernetes uses it in its software development lifecycle? Sure, so the Kubernetes enhancement proposal is basically derived from Rust and Python. So we didn't go up with anything by ourselves. It's the process to create a standardized structure for proposing changes to the Kubernetes project. Often this might be a feature but it also just might be some operator facing change or a refactoring or some other sort of non-trivial change that you need design and discussion around. Proposals live in GitHub and they are versioned and source control. They consist of two parts, the readme.md which is the meat and potatoes of the proposal, your design, your motivations, all of the questions that we have, the production readiness review, graduation criteria, test plans. And then we have a metadata file which we use for tooling which helps us track the state author ownership whether it's Alpha Beta or GA and what release the enhancement is targeted for. So enhancements for us are part of the greater release cycle to encourage communication between all of the stakeholders pre-implementation. It also helps us make a roadmap and plan our work. So it's a tool that we're using to facilitate the best Kubernetes release possible and also have active development and conversation regarding any major changes going into the project. So Christy, I've noticed that Tekton uses a similar process. Can you maybe explain how enhanced proposals work for you? Sure, so for Tekton, we based our proposal process on Kubernetes. So a lot of the process is similar. One big difference is that we don't target enhancements towards releases specifically. For us, the enhancement proposals are more about agreeing as a community to what features we think should be in Tekton. So even if a proposal is accepted, it might be shortly after, it might be quite a long time until you see it actually implemented and released. The process for us is very similar. We also use GitHub as the source of truth. If you want to open an enhancement proposal, you open a pull request with the details. We have processes for assigning reviewers. We make sure that we have reviewers from at least two different companies do a bunch of back and forth. And then ideally, eventually you'll get to the point where everyone signs off and approves and we merge it. Yeah, and I say we kind of learned a lot from Tekton when we added enhancements proposals to ShipWrite. We more or less copied the Tekton and kept process when we created ours. But I think one thing that's maybe different for ShipWrite is that we are a very young project. Our APIs are alpha and therefore we're comfortable making lots of breaking changes. We want new ideas coming in. We want to iterate on ideas pretty quickly. The other big thing for us, and this is also an artifact of us being young, is that we're pretty small. And so our process is very ad hoc. We don't have necessarily like regular reviews per se, although we want to improve that in the long run. And also we aren't necessarily targeting things to releases as well. It's more of we have an idea. We agree on it. And then when people have time to get to it, they actually do it. And so you and Christie are primarily using enhancements for features, not just for any change to your projects. Yeah, I'd say that's a correct characteristic. We're not really about managing changes with our enhancement proposal process. We are really more about just getting the ideas and vetting those ideas. I think for Tecton maybe one, we do enhancement proposals, I guess whenever we feel like you want to kind of show multiple people what you're up to. So there have been some enhancement proposals which are for refactoring from time to time, but mostly it's for features. So maybe this is a good time to ask both of you, kind of like the deeper background of why you went forward with this approach to begin with. Can you explain a little bit about why you adopted this enhancement proposal process and created not just the template, but also the things that go around the template to get these ideas agreed upon? Previously we were doing design proposals that originally lived, I think in Google documents. And I think that one of the key motivators was having one source of truth. It should be the entire state of the cast and everything that you need to know about this feature is going to live in one place in our Git repo. All the conversations and changes go in as pull requests so that history is preserved. And that gives us a lot of transparency, not just into the state of the cap and what you can expect to get out of it, but how you kind of arrived at that state. And that's really important as an open source project to really value transparency. It's also about creating a common structure for all changes, not just changes that you feel like writing the cap for, but for all changes and sort of have a baseline of quality for the code going in and sort of discussion and also a baseline level of discussion. So you have to talk to the state, you have to talk to the API reviewers, you have to talk to all of these other involved parties and sort of creating that baseline level of quality for everything that was in one place, I think is really furthering just the general goals of community and transparency, especially in a large dynamic growing project like Kubernetes. For Tecton, I think having a single source of truth for this kind of information was a big motivator. I want to also give all the credit for the fact that we have this process at all to Vincent Demy Starrer, who's the one that proposed this process. I actually was kind of opposed when he suggested it. I liked the process we had before because it was pretty ad hoc and we kind of had lazy consensus. So basically as long as nobody objected, people could kind of go ahead with what they wanted to do, but I also had some cognitive dissonance because I also wanted to review all the changes that people were proposing. And I think the worst thing about our process before TAPS was we didn't have any kind of consistent format and you kind of had to be attending every single working group to actually keep up with what was going on, which makes it hard for people to become involved with the community, especially if they can't actually attend the working groups. So Vincent proposed that we take the Kubernetes enhancement proposals as a starting point and we introduced Tecton enhancement proposals. So now we have this one single source of truth. And even recently, I found it really useful to look back on, I think it was our second TAP, TAP two, and look at the specific details of the proposal and reference them when I was talking about something else. Another thing that I've personally found really useful is all of the back and forth. Well, it can take a really long time. I think it also makes all of the proposals better. And I personally actually have more confidence when I create a proposal now because I know that I'm not the only person who's going to be thinking things through and people who have knowledge in areas where maybe I have gaps can help fill that in and just help make the proposal better in general. And selfishly, I can now fulfill my desire to be involved with most proposals because I have one place to go to find them. And if I want to add my two cents, I can add myself as an approver. And I'd like to say from my experience, the whole notion of having one source of truth for your ideas and your designs has proved to be really useful, even more useful than I anticipated when Shipwright kind of formalized our ship's process. We started more using just like more loose markdown design documents. Shipwright was originally kind of a Red Hat sponsored project per se. And it was initially just most folks at the company. And it wasn't until we started getting contributors from outside Red Hat in this instance, IBM, who are interested in using it for their own needs that we decided to first just start documenting our design decisions in markdown. And that's how we kind of eventually, we then took the cap and proposal process and use that to formalize our own process. And just having it all in one place proved really helpful. As a matter of fact, all the original design documents lived in one repo at one point, we moved it to another GitHub repo. I initially wanted to just kind of start fresh, but the community actually really wanted it all in one spot and we ended up moving all of those old design docs into this other community repo that we set up. And then we had a numbering system and we renumbered everything so that everything was kind of in chronological order so we could see the progression of the project over time. For us also, I think the source of truth is important because somebody might propose the enhancement, but they might leave the project. So you need a way to track this so that once you've committed to a feature coming in through Alpha, Beta, NGA, anybody else in the project is gonna be able to pick it up and push it towards the end. So this all sounds great upfront, but those of us who have actually done this and tried to manage an enhancement proposal process, it's all not sunshine and roses. So can you perhaps talk about like what have been the more painful things that you've experienced in your EP processes and maybe talk about things that you've experienced in the past and how you have addressed them and evolved your EP process. The biggest thing that continues to be on my mind with our enhancement proposal process is getting feedback to people quickly enough and regularly. I think there's nothing worse than putting a lot of time into creating something and just to have it be ignored or to have it be initially ignored and then find out some long period of time later that you actually have to make substantial changes to it. I think we've made a lot of improvements to this. We've set some SLOs around how quickly people should at least get initial feedback, but people still miss those and I miss them all the time. And then at that point you kind of have to rely on humans being involved and going and pinging each other and saying, can you please review this thing? I really need feedback. I think we could improve this by adding automation and the fact that we have centered everything around GitHub and this one source of truth, I think that opens up the possibility of adding automation that we couldn't have if it was a whole bunch of Google Docs just kind of floating around, but I think there's still some improvements we can make there. And the other thing is that I would really like to have more folks internally who are able to contribute to Tecton even if they're not primarily working in open source, but in order to make that happen, usually if they want to work on a feature, the first thing they have to do is open a TEP and then they have to deal with the feedback and they have to deal with the back and forth and they're not even necessarily dealing with GitHub on a day-to-day basis. It's this whole other set of tools and this whole other thing they have to be aware of and check. And I think that that can be a pretty big stumbling block and the few times that folks have tried, sometimes it's worked, but sometimes the TEPs get stalled, they often get closed because there's not enough attention given to them because people had to move on to other things, sometimes they get picked up later, but I think that it can be a big barrier and even if this kind of process was more normal internally, I think there's still this other layer which is that with an open source project you have all these different companies who all have different priorities and you're not kind of, you're sort of aiming towards the same goal of having the best project you can, but each company has its own priorities and it can be kind of hard to manage those things. We used to actually have a similar problem with new contributors sort of screaming into the void like opening their enhancement first before they talk to anybody. And we actually started mandating that people have to go to the special interest group, the SIG, who owns that portion of code or functionality first before sort of sitting in their wheels and thinking a lot of time into an enhancement. And it means that most of the enhancements now that are open, the SIGs have already agreed to now. So they said, hey, this is a good idea. Hey, think about this or hey, no, we already thought about that, we're not doing that. And I think that looking at this as a tool for discussion and as a meeting of the alliance and starting that sooner rather than later has been really helpful and successful. The pain point that we've had is one new one is now that we have this process, we have this template, is it applicable to everything, right? Can we make it better template for maybe a process change as opposed to a code change? It's very code-centric right now. So that's something that we're exploring is adapting the template for even more use cases to better tailor the experience for the authors as well as sort of the classic open source problem of reviewer bandwidth. We have a lot more people wanting things in the repo than we have people who are committed to reviewing and approving things. So that's a big challenge that I think every project has but a lot of SIGs are really working hard to onboard new reviewers and approvers and we're trying to sort of just manage expectations that this is a community. So you can't just have it your way. You can't have everything on your timeline but hopefully if it's a good idea and it's good work it's going to get in. One thing I come back to is Tim Hawkins talk at a CUBE Contributor Summit about code reviews and the one thing that he says is do not be a rubber stamp and I think that that's also like kind of baked into our review process but our review process takes time. No one's just going to read it and come in and say it looks good to me and then you're off to the races. So like trying to set the correct expectations for contributors as well is important because you're going to get a lot of thoughtful reviews and that might not be what you expect. So a little bit of patience and it's kind of excitement like Chris was saying of like having a bunch of people giving you feedback is actually a good thing because you're not going to waste your time later. These are all things that we're working on but I think that the process every release is getting better and better and we're really taking feedback and comments seriously and it's like an iterative process. We're not done. So. And I guess for us with Shipwright since we are so young I think we too commiserate with the bandwidth problem and it's actually brought us into having bigger conversations about project governments trying to set up a good contributor pipeline and establishing how someone can go from first submitting the code to being established to themselves in the organization to working their way up to being an approver either for a subcomponent or in the case of the enhancements we now have the dedicated repo and we can add people as an approver there. So it's definitely helped us as a project start thinking about these things and maturing because we ultimately want to grow in scale. Now I also want to talk to both of you about that feedback cycle and getting people involved because there's definitely a lot of work that needs to go into an enhancement proposal which can be very intimidating for someone who is coming at it new. So I'd like to ask you what have you found to be helpful to make the writing and reviewing of EPs more welcoming and less intimidating to contributors? Yeah, I think I mentioned it in the previous question a little bit but I'll explicitly mention it now. We used to have the problem of people just opening enhancements like with all of their new contrator energy and then the enhancement just sits there because nobody knows that it's there. Nobody, they haven't talked to anybody and that was a really, I think, frustrating experience for contributors. So the thing that we did is to say, hey, talk to the SIG early, get that dialogue started not just to vet the idea but so that you know how to reach the SIG so that the SIG knows that you exist so that they know that you're serious about it and get that relationship going so you're not just a ship floating in the sea somewhere hoping that somebody sees you and that's been really successful for us. I would also just say at its core, having an enhancement, having this template, having this process to get something into a Kubernetes release is very defined. We're not hiding the ball for a new person it applies to everybody, there's no secret and I think that that sort of, it's fair. Everybody has to do this to get it in and we're telling you what you have to do and we want you to do it. So part of it is be clear and set authors up for success with your template but it's also set them up for success by being open to communicating with them early. I've never heard new contributor energy before but I really like it. So for that process, when people have to approach a SIG before they open a proposal, do they have to do that? Do they have to attend a meeting to do that or how does that work? No, absolutely not. So we are very community oriented and we are a global community. Like there are people everywhere, there are people who, they're living in the future, they're living in the past depending on their time zone. So you don't have to attend a meeting, we're very async. You can send things to the mailing list, you can go to the Slack channel, you can put something on the SIG agenda as well to discuss. So we have a lot of options there, not just for time zones, but for personality. Some people are more introverted, they might not wanna be on camera. Some people are English as a second, third, fourth language. They might be more comfortable just writing down their ideas and communicating that way. So we're very flexible and it's really just about that that communication happens, not the way that it happened. Yeah. One thing that for us were kind of, because we're smaller, I guess we're more relying on the template. And so one of the things that we try to do with the template is make clear what is expected, particularly at each phase of the life cycle since we have this provisional implementable, implemented kind of phasing. So when someone has like that new contributor energy and they want to submit an idea, we want to be able to say that this is the things that you need to fill out so that we can have a good discussion about whether or not this is a good idea or not. So we have like a summary, the motivation, a section on user stories, goals, which helps us figure out what is in scope versus what's kind of out of scope, what are the things that you're maybe not going to address in the proposal. So then once you get that agreement on, yes, this is good idea, then you can start investing that energy into actually thinking through the design from an API level from which components are going to be affected to even going into how you are going to test it, which infrastructure you might need because particularly for software that can be deployed kind of anywhere, you can have lots of different requirements that your CI environment may not be able to replicate easily. So knowing that upfront is actually really useful information as to whether or not this is something that the project can actually take on and accept and maintain over time because ultimately, especially with open source, you are not the owner of the code, like the community is the owner of the code and someone is going to be, at some point might be picking it up and taking it from your hands because you're going to be doing something else. So that maybe leads me to this next idea and because we were involved in open source, but we also work for big companies and we are taking the open source projects and converting them into projects. So is there a compatibility or maybe a disconnect between the two? Can you use this process for a product or is this something that maybe, it's great for open source, but maybe not something that is good for proprietary software? I think that if you take the idea of what is an enhancement, it's a way to clearly articulate your design along with potentially answering questions that your project thinks are important and sort of having a way to track the status of them. I think that that general idea is totally legitimate, whether it's a project or a product. Now does the Kubernetes enhancement proposal process and release process work for a product? Maybe not. We don't have people who are allowed to come in and say, hey, I want the feature in right now, two weeks before we cut a release. Because I think that our process has a little bit more friction in it and our deadlines are a lot harder and our exception process is more rigorous. But I think that the success of all of our projects, it can't really be denied. So I think that there's obviously a lot of takeaways here, like facilitating conversation, getting a lot of disparate groups of people on board with an idea, which can happen inside of a company, memorializing those ideas in a clear place where they live, not in 50 Google Docs around the internet. I think that those are all really great lessons for anybody to take away and to kind of take what works and bring it into your process and kind of ignore the rest. But be open to the process that a lot of different projects have because this is all through trial and error and you can learn a lot based on the work that's come before you. So I would say be flexible and open-minded and see what's out there and see what resonates with you. But yeah, absolutely it could be your product. I think that it's a really good point that you want to be flexible because I think that a mistake that gets made in software a lot in different areas is to assume that because an approach is working that's done in some particular ways working for one group, it's going to work exactly as is for another group. And I think the truth is that even what works for your team today might not work a year from now. So I think it's really important to kind of constantly be examining how you work and trying to improve and tweak it because another project is different from yours. Another product is different from yours. A team is different. You're a different person. Your team is different a year from now. So I think that being able to, I wouldn't take a process, try it exactly as is and then reject it. I would try to examine what did you like about it? What didn't you like about it? What changes could you make? And I think as Kirsten's pointed out there's obviously a lot of value that's come from this and Kubernetes has been, I think I'm going on a limb and saying it's the largest open source project. I think there are other huge projects but just the volume of contributors that are involved in Kubernetes and the success and kind of having all these people with all these different goals actually contributing towards one thing. I don't think that that can be denied. And Tecton is much smaller but we've seen a lot of similar benefits where we do have people with different interests and different priorities but we're still able to kind of come together around this process and use it as a place to try to be upfront about what we actually need and represent everyone's interests and find solutions that work for everyone. Yeah, and for me, one thing that might, one of my initial reactions to like the EP processes it feels like this is like waterfall all over again where you've got to like do all the big design upfront and then you got to work it through and then you're not releasing for another six months later. And so taking that, if you view it as a waterfall then you might be setting yourself up for failure but they're at least with the teams that I'm involved in we all often use it in like an agile process for like a spike story in particular. We found EP's to be a very useful mechanism to have like a concrete deliverable that we can agree upon versus like what might be a more looser free form spike where it's just, okay, there's this thing that we need to investigate and research. Well, often when we are scheduling spikes we will decide, okay, these are the things that we need to investigate these are the decisions we need to make these are the things that we want to have an idea of how we're going to do and we are going to write the enhancement proposal get feedback and get it to an implementable state. And that we find that it's helped us kind of focus what it is we need to do and it's helped drive those discussions and get the stakeholders that we need involved and get them involved early. Yeah, I mean, I would say that the biggest difference is probably that those sort of last minute things that get dropped in a lot of the features for products they go in as GA. And in Kubernetes, our features go through alpha, beta and then they're finally promoted to GA. So we have a slower process. And that's, it's really baked in like the enhancement supports the release and we're trying to get the best release possible and the best signal on release and the best ability balance with number of features into a release. So I think that that's probably like where our process might feel really heavy to other people in a product. You just want it in, get it in, fix the bugs later. But also things that there's like a real valuable lesson there like being able to clearly communicate with users like, hey, we're giving this to you, but it's not GA yet. It's, this is what it is. Like clearly communicating the state of your feature not necessarily over promising but really being able to deliver something that's gonna work the way that they can expect it to work and then incrementally improving it but in a very clear structured way. Also does have some value, I think. There's a lot of pressure on engineers and products to just straight to GA, fix the bugs later which can become pretty expensive like with time after the fact. So, you know, there's a lot of good things that I think the enhancement process can bring up that teams and products might want to consider and discuss. And I know Adam was like a consultant before so he has a lot to say about cost on the front versus cost on the back. Yes, I have actually been involved in consulting projects where we've had different budgets for features and bugs and the features budget is the one where we were just billing our time by the hour and those were more, that budget was more lucrative whereas the bug budget was fixed. And if we went over it, if our software was so buggy that we still couldn't use it my company was the one that was eating the cost. It wasn't the customer and the client who we were working for who was paying for that. So, in the long run it was probably means it can make sense to try and get things right and ship less buggy or stuff versus getting it out and then addressing the bugs later. And I think even when you're not directly billing someone for the bugs, I think that if you look at how much it costs to make a change in general at different phases in the software life cycle it's the most expensive to make the change once the code is already written. The EP process might feel like it really slows things down but I think once you kind of get used to it you'll find that or at least I've found that I feel like the part where you actually write the code is not the hardest part of the process. That's kind of an afterthought because you've been forced to sort of think things through so thoroughly and having to do that actually gives you like really rapid feedback time where you can have an idea, think it through, talk it through, realize the implications, realize you need to make an adjustment and make that adjustment. So you actually have these really quick cycles where you don't have to go as far as writing the code. Maybe you're writing proof of concepts along the way but you can write those a lot faster and a lot more cheaply and you can just kind of throw them away after versus getting to a point where you're actually ready to ship something or even worse, you have shipped it and then you realize that there's a huge problem and you need to change it. Like that is the most expensive time to be making that change. So I think that having processes like this can actually speed things up and make them cheaper. Yeah, I mean we're all engineers so we know that like features don't just go in with one PR, right? It's always like multiple PRs and having to back out PRs because you've got something wrong when you could have just had to have a couple iterations on a document, the document is the way. Backing out PRs that are half done features that are half merged is expensive and extremely disruptive and can be dangerous. I would love to talk about this more but unfortunately we are running out of time. So I'd like to leave these last few minutes so that we can share our takeaways, what we want the audience to take back either to maybe open source projects that they're involved in or the projects or companies that they work for. So Christy, why don't you get us started? Sure, I think the main thing that I'd like to get across is if possible, I think it really helps to try to shift your perspective from viewing these things as kind of arduous and painful and realize that actually having to take the time to communicate your ideas clearly and then getting feedback and responding to that feedback and incorporating it, it makes your ideas better and I think it can actually help you grow as an engineer and create better designs and proposals in the future. Yeah, plus one to Christy. I would also say that this is an inherently flexible process, there's no one right way to do it. So you really have to do some reflection and decide what's important to you and your project and use this process to reinforce those values. For us, it's transparency and communication and having this source of truth. For you, it might be guaranteeing stability or whatever your priority is, but think about your core values and then tailor your process to reflect your core values and I think you can't really go wrong. And I think my main takeaway running an EP process is that you really need to invest in it and make sure you're getting the necessary care and attention both from the reviewers but also the authors who are contributing their ideas. So it's almost like a plan if you've got a vegetable garden. You need to make sure that you're watering and checking in on them regularly. If you're not, then they're just gonna weather on the vine and fade away. So that concludes our panel. I wanna thank Christy and Kirsten for joining me and thank you all for watching this session. We hope you enjoyed our discussion. You learned why and how we are using enhancement proposals and maybe are able to take and use these EPs effectively in your own workplace. So thank you and enjoy the rest of Kukan.