 Good morning, everyone. Good morning. Good mic check, Duffy. I'm only ever 50% sure it's going to work. We'll give folks a few more minutes. Amy, do we want to wait another minute? Nope, we've got about 20 folks coming on in, so we are ready to get started. All right, let's go ahead and get started then. Good morning, everyone. Good day. Thank you so much for joining this public POC meeting as a reminder. We are all subject to the LF anti-trust policy notice. Meeting logistics, if you're here, you figured out where the meeting is next. And we have several TOC members here present today as a quick reminder. This is a public meeting, so we are welcoming community members feedback. We're going to be talking today about the approach to inactive CNCF projects. There is an open issue on our repo. Thank you so much, Dawn, for filing this so we can have this discussion today. So for those of you that haven't been following some of the TOC dialogue as a recent, we've been asking a lot more around a lot of questions around project health, their status, checking in with them, because the current process says that we have annual reviews for sandbox projects, but we don't have the equivalent for incubating and graduated. And as projects move between levels, they experience a lot of changes. Some of them take off, they see widespread rampant adoption, and then in other cases, some of them don't. So what do we do with projects that reach in an active state? It could be any number of reasons. Maintainers could have taken some time off to go deal with some personal matters or change companies or organizations, but we might expect them to come back. So I want to open the floor to the discussion. We've got some initial questions here that I think can help drive the conversation a little bit in this space. So I wanted to first make everyone aware that we do have several dashboards that exist that the foundation puts together for us. That from dev stats, we also have project health table that lets us know, kind of like, what are the most recent commits of a project? How long has it been since they've had them? But these are not always good indicators of a project's activity. There could be things that are going on outside of the actual Git commits and their repos. So I'm going to silence myself for a quick and kind of like open the floor for discussion. If you have something that you would like to mention, come off mute, raise your hand. We'll try to do a good job of making sure everyone has the opportunity to speak. Richie. Yes. So two thoughts at the same time. On the one hand. I want to lead in with that. I know that certain projects don't have to have a lot of GitHub or other visibility. Open metrics is the obvious, is the obvious example there as a standard. There is, it would actually be a bad thing if there was activity all the time. It's actually a good thing that there are long pauses in the activity. And of course, again, with that out of the way, I do think that we as CNCF would benefit from, from being a little bit more assertive in, in asserting the, the basically continued development of projects. Of course, I feel historically we, we have been very inviting to pretty much everything and everyone. And it just, it's overhead in a lot of places and part of the burden we as to see our feeling is a lot of projects to, to handle and, and being a little bit more assertive about both allowing projects and archiving projects. It would be good. Thanks, Richie. Josh. Have we discussed having incubating graduated projects do annual reports if they haven't had a due diligence. This year. We have talked about that, not in relation to this issue, but as a result of other conversations and challenges that come up. So I, I believe in any TLC member, please correct me. I think we're all relatively in favor of annual reviews for incubating and graduating projects. However, we want to ensure that we're not increasing the amount of burden on reporting coming from the project. So identifying which portions of an annual report are indicators of change on a project in an automated way as much as possible, where those engagements with the project to check on their status are slightly lighter weight, whether or not that's a TLC member checking in with them after we've received an automated annual report or annual review from them, or perhaps extending that capability to the tags to check in with those projects and leverage that report as a function, a functional driving point in conversation around some of those activities. Yeah, I mean, I was just thinking about annual reports both as a way to you know, indicate activity, particularly in the case of projects where a constant stream of commits is not the best gauge of activity. The second thing is also annual reports as a chance to check in on potentially other looming project problems. Like for example, an exodus of maintainers, which we've had as you know, with a couple of projects, the it's just we have a lot of projects now and keeping track of all of them by a dashboard is would be challenging even with the best dashboard in the world. And we haven't yet figured out what the best dashboard in the world looks like. Yep, I agree. Richie. So yeah, I mean it's maybe weird for me considering the work as I fully agree just having a dashboard is not going to be super effective because it's the case of putting expensive humans in front of a screen. Automated reports alerts and such would be much better. I with my project head on or two different project heads on a I fully agree with Emily that this must be automated as much as possible and only if certain if certain things are not being met, should we even actively reach out and really introduce much of a process because most of the projects are really resource and time strapped already and putting more burden on them is is not precisely why use a project join an umbrella. The other thing is tracking will not be always possible like with my open telemetry head on it's it's it's very hard to track relevant contributions which happen in other places and are still relevant for the project and this is also true for the for the life speed of a given project. So we can never fully automate all of this. But we should as much as we can. Yep, I agree. So I Richie use you just mentioned something that's interesting and I would like to kind of tease this out because we've had a recent discussion around this is when things are not being met. So right now the criteria for archiving projects is more or less kind of clear. It basically states that it the project has to continue to meet the criteria for CNCF acceptance and the criteria changes depending on what level it is that you're looking at for the most part we see projects that come in at the sandbox level. We have some criteria that is already defined but it's not very robust and when the TOC is evaluating projects for sandbox inclusion we cover. We look at a lot of different things. We look at how many maintainers they have, how active they are, what kind of documentation exists, what plans do they have for the project, what's the direction that they're heading in, and those projects provide us with a lot of information about what's going on with them so that we can help make those decisions. But for incubating projects that criteria looks a little different. Projects need to be ready and what's ready for one project doesn't look the same as another. We go through the due diligence process. We do have criteria that are defined at the incubation level. So my question is, is the criteria that we have for archival enough of a low bar for determining whether or not we need to reengage with projects if they come up as inactive? And how could we potentially programmatically identify indicators that would allow us to have that conversation? Because I don't believe based off of the criteria that we have, we can do that today. I think there is at least three and a half substrands in answers to that question. Maybe on the high level first, I personally believe that yes, there should be more scrutiny on, let's say, a graduated project than the Sandbox project due to the simple fact that it is people are more likely to trust it. So there is also more risk of sorts if that thing goes still. I lost part of the track of the, as you asked me directly, at the latest when we trigger whatever mechanisms or whatever threshold we have, we must be engaging with the project, but it's kind of tautologic anyway, but we should give them a fair warning before we do so and at least ask about the health of the project and things like these. And I forgot the rest, sorry. Sorry. For the archiving, we've talked about making sure that that process is a little bit more clear and to your point specifically engaging with the project to kind of understand first what's going on, see if there's something that we're not aware of that we haven't considered. Perhaps they are holding regular meetings to kind of figure out what their long-term planning and roadmap actually looks like and they might have paused work on the project until they've gotten a clearer vision and that's entirely reasonable and that's something we can learn when we re-engage with them. But generally speaking, we want to ensure that projects have the opportunity to let us know what's going on, but I want to ensure that they also feel empowered to reach out to tags or TOC members if they're struggling. Josh, is your hand still up or do you have something to add? I mean, I do have something to add, but it kind of takes us in a different direction. So let me wait for other people have more to say at that point. Okay, Ricardo. I just had a quick question on the second bullet in this slide, which is about health and activity of a project. We're talking about archival. Is this also counting on kind of demoting a project or do we expect it to move to archival and come back if it becomes active again? Or are we considering like incubation back to sandbox or graduation back to incubation, things like this? That is an excellent question. I don't know that we've actually talked about that or had that presented as a potential alternative. Richie, your hand is up. Craig, so I'm going to let's do Richie and then Craig. I'm pretty certain TOC has never discussed this. I know I was in several discussions about precisely this topic, about like demoting projects and having a way back. We always came out at A, this being super confusing for end users, this back and forth. And also about this, like if you are graduated and you're idle enough to need a warning, it should be a strong warning and not basically a soft cushion of, oh, I can basically just as soon as I ramp back up everything will be fine and dandy. I think there should be more of a cliff to honestly keep people motivated at that point where things are already pressing. It also makes it easier internally. If you are working at a company which used to sponsor something and you're being told, hey, this might fall off of a cliff, you have a much easier time internally and you have a lot of resources and everything. Craig? There was a comment that I put on one of the issues when you were talking about adding more requirements to graduation. And I think it was more like, let's say there needs to be a current diversity of maintenance. In the event that the project otherwise stayed the same in terms of adoption and no longer had that diversity, it technically wouldn't meet the criteria but because it had at some point in the past would remain in that bucket. So I think it was a little bit different from what I thought it would be in terms of how the project was going to be used. But I thought it was a broader idea which is what if there were just CNCF projects and then there were tags in the Kubernetes label sense rather than the tag sense applied to them that said this project is diverse maintainership. This project has a security badge and so on because right now there is such a requirement that it's going to be true that they meet all of the requirements. And Richard, what you were saying there there will be times where a graduated project it's not going to need archival, it won't want archival but it may no longer meet the requirements that otherwise a graduated project could be set out. And so its options effectively are demoted back to incubation which requires a huge effort then if it meets those requirements again. Or just have an approach whereby you don't meet this requirement and it makes me a bit more fine grained as to what the parameters are for a project. So I think badging is an interesting concept. The motion I think is also however I want to ensure that we understand what the value of doing that would be for a project because there is a lot of benefits and value that projects get at certain levels from the foundation and the motion would be achieved in what is going on. I think badging is very interesting. It allows us to provide better visibility and transparency to potential adopters around the state of the project. There may be some adopters that don't necessarily maintain our diversity very high. They just want to ensure that it's active and it's moving forward. For us in the TUC we want to ensure that the project has enough maintainership both in that they can collectively and collaboratively drive the direction forward so that there is no single entity that can change the direction of the project on their own and that's kind of defined in governance and we allow projects to do that. We also want to ensure that if something were to happen, if maintainers want to take a vacation because everybody would love to take a vacation in open source, they can do so and step away and the project can move forward for an extended amount of time and they can still receive commits and they are not relying on a single person or two or three individuals to carry the entire weight of its progress moving forward. I think there are some elements that are useful. I think the annual report concept is excellent but what would we consider kind of like health and activity for those? Dawn and then Josh, I think I got that order right. One of the things we should talk about with respect to archiving projects in particular is security vulnerability. This is one of the things we focus on in archiving projects. That VMware for example is if the project has a bunch of security alerts that they are not fixing and not releasing fixes for, then we look at whether a project should be archived. I think that the security status of a project is pretty important because what we don't want is that they are sitting out there with security vulnerability as a CNC. I think you've been in your business for a while. That's what you do. That's what we think about. I want to make sure I heard there was background noise that was coming in. The security vulnerability. To make sure I understood correctly, your consideration for security activities associated with the project, their ability to make sure all their dependencies that are reported to them have a good response window. Did I understand that correctly as a consideration? Yes, absolutely. Okay, thanks. I think that is definitely that hit some for me. Josh? Since we started talking about what the motion would mean, I think we need to really look very differently at incubating sandbox projects than we do for graduated projects. I feel like for graduated projects the CNCF has an obligation to our users to the users and stuff to try to keep graduated projects stable and current and available. Unless they are sort of naturally obsolescent, if you follow me. Let's stop using them. Sure, let's talk about archiving them. But as long as there is a user base that depends on that graduated project you don't generally get to graduate unless there is one. The first effort of the CNCF at the graduate level needs to be how can we rescue this project not demoting it to incubate it. I think that should be a truly exceptional turn of events for that. For a project going from incubated to sandbox, I think that is something we can talk about. Because it is after all incubating and that's kind of a possibility. Adam had asked in chat what is the criteria for making a project inactive and looking at the dashboard. There are five projects that are currently inactive which is fairly accurate. It's essentially as Amy stated anything that hasn't had activity for 120 days. Again, that's looking purely at GitHub and Git commits. It's not always a strong indicator. I was spending some time this morning actually looking at one of those to try to understand what's going on with the project. We do lose a lot of history when we're relying solely on Git and Git versioning and all those commits that have been happening over time. We need a better indicator there. Richie. I would phrase it maybe a little bit differently. I think having X amount of time in 120 days seems good as one of the possible triggers for deeper scrutiny is probably a reliable approach. Not as the sole trigger but basically if you do not meet X amount of metrics or if you do meet X amount of metrics that's when or a mix of positive and negative statements is when we trigger a more manual approach which would initially probably be TUC and or tags reaching out and just saying hey are you okay. But if someone has plenty of activity and contribution graphs are always stable over years and everything or even growing maybe we can just not have a deeper review of that project. I mean Kubernetes we all agree probably we don't need to do an in-depth analysis on if the project is going to go away next week or Prometheus or whatever like we know we don't have to. There was some discussion in chat. Don identify that open metrics as a standard and we've talked in the past about how depending on what kind of project it is if it's a standard or not their activity is going to look very different. So that might be an occasion where having badging or some other labeling associated with project is enough of a good indicator that this one might be fine but we should still look. Craig also followed up in chat with when external parties or adopters or other entities who are not the maintainers of our project are seeking to have a project marked in active but the maintainers don't and there are probably occasions where and it's not just this particular case where the maintainers just are moving at a different pace than what we're used to seeing and this for me I would expect the tag or the TOC to re-engage with the project to talk to the maintainers because they're the ones that are responsible for owning the commits and moving the project forward and engaging with them to understand kind of where they're headed regardless of any of the external factors then that kind of brings us back to the discussion of what is considered healthy and active what sorts of indicators do we have in place for that. Ricardo asked around having an annual report be a requirement for graduated projects I think that is still on the table here what other thoughts do folks have because I'm hearing a lot of very good ideas but we haven't actually talked about what some of those indicators are we're talking more around the process and that's fine but I'm curious for individuals that are maintainers or have been involved in the maintenance of projects how do you view your own activity and health? Do we have any maintainers on the call? That's a good question. Yes and I've been quiet in thinking there's lots of maintainers here so does a different maintainer besides me have thoughts on this? I mean I would say that for the most part looking at the health of a project I would look to understand issues and resolution of those issues and look to understand releases like are we actually moving those issues or things that are found in the project toward a release and releasing things that are fixed. These are the things that I usually judge the health of a project on. I would also say it has to do with where the the maintainers are in managing their future it's one thing to manage for the code but you have to wonder who's coming behind you because as I look at the CNCF over time if I go back four and five years and I look at who maintainers are in general now versus back then you'll see some people who are the same but you're going to see a lot of transition and so how do they for example manage that transition and change in addition to how are they managing and some things you're going to get more issues and stuff coming in than you're ever going to be able to react to look at Kubernetes and actually I'll echo Don's statement about another health metric is the security impact if somebody opens a security issue or if they're dealing with some kind of a way of dealing with security issues that's also an indicator of health because it usually comes after the project is actually already in a reasonable place for the most part in a security response the mechanism is really important Kathy and then I have a comment I think actually I think Ricardo raised the question whether we would like the project to have annual report actually I think that's a good idea you know and if the annual report includes some indicators health indicators of the project right you know like security vulnerabilities which was raised by Don before and also some like the activity level maintainers etc and then you know if that report we see some issues I think it's better to combine that with the annual review or we can have I think we need to give a chance for the apartment maintenance to explain if we see some issues why those issues and then we can decide what to do I also agree with John's comment we need to treat the graduated project differently than sandbox or incubating because once we approve those project to be a graduated project I think there will be quite some users using it so we need to put in extra I think payer effort to make sure those project should not go to the archive you know that status there was also a question in chat around whether or not how long it takes to review and close out PRs or issues is a good indicator from a recent discussion I had with one of the companies that does security projects for projects it's not always a good indicator of activity there are some vulnerabilities or design flaws that do exist within projects that just take an extremely long amount of time to get resolved and require a lot of individuals involvement so leveraging kind of the time that a PR has been open isn't always a good indicator same thing with issues it could also be that the project set their roadmap and they're going to work on those issues that are defined first and foremost because it's entirely up to them to prioritize what work it is that they would like to do Nikita had a good suggestion or observation around what Kubernetes currently does and potentially re-leveraging some of that prior art I like the idea of a maintenance mode for projects that are they've kind of capped on the amount of voracious activity that's going on within the project and so we will see kind of activity start to stabilize so having maybe a badge associated with maintenance mode for projects is also a good indicator to help reduce the amount of potential false positives that come up in an annual report Ricardo had a good suggestion on coming up with indicators for each level yep I think we were going to we're going to go down that path as well as looking at automating the detection of those indicators so I think we're on agreement around that one Josh added around metrics or what triggers the TOC member investigating I think that's still up for discussion so we still need to get into the specifics and I think we can probably do that maybe offline or if we have time later on the next question that I have for everyone here is who's this information going to be for and who's responsible for doing something about it so I think we have a combination of a lot of really good ideas that need to be pulled together into some different options that we can start exploring but I want to know who so in addition to the TOC looking at the activity of projects what about tags what about other maintainers what about adapters they're going to be looking for different sets of information Richie three points so far for who I think TOC and project maintainers primarily maybe also tags who does it I somewhat strongly believe CNCF staff the reason why I believe this and why I don't think this is something which should be at least led by the tags as the unfortunate one who did three full due diligence as part of a little bit of a test balloon within type observability there is a lot of strong opinions and a lot of politics and sometimes even perf and promotions and raises and everything attached to projects that is to moving levels and in particular to being removed I would dare say I don't think it is realistic to to expect all tags acting completely the same and as such you will have some element of tag shopping for the more lenient ones or the more aggressive ones and all of those things I don't think it is healthy and I don't think it's fair to to offload all of this onto the tags at least unless it is supportive and I don't think it would be as much administrative in which case I somewhat strongly believe CNCF staff would be better suited okay and to ensure comment on that is captured Craig had mentioned that that very point around there's a lot that's tied into moving levels that impact a lot of people's livelihoods and their abilities that's why he had suggested removing them or at least having a conversation but that's scheduled for the fall Ricardo so I was just I think that this should be for adopters and in users I think this information is very very useful I had a question for Craig which was if he was suggesting to add badges in addition to the maturity levels or just drop in the levels completely but he already answered in the chat and the concern there we should also ask end users what they're using these maturity levels for because it's kind of a stamp on the project and it's quite kind of useful to have these levels right now if we have a lot of indicators like browsing through the project might be a bit more complicated so yeah maybe maybe we need to ask the end users as well what they see the benefit of the levels and adding badges is what they could bring I think it's a great idea definitely has value for end users the current levels right now and I think that's a worthwhile conversation to have in that working group that we've requested a tag contributor strategy so if you're interested in the moving levels discussion and how potentially that could be changed and shifted to provide a more healthy ecosystem with projects and a positive experience for all of our project adopters and tags as well please check out that if someone has a handy idea I'd appreciate you dropping it in the chat for everyone to follow up on so I'm hearing that badging has a potential value add here for understanding what's currently going on with projects hearing definitely a yes to an annual reporting mechanism that needs to be driven by the level that a project exists in whether or not it's sandbox incubation or graduation and that graduated projects definitely need to be treated different and I think that Josh's point that it should be a rescue effort for them is pretty spot on and that's kind of the behavior mechanism that the TOC has been following with some of these projects as we are alerted to them but that is a reactionary mode we need to be a little bit more proactive on some of those indicators I think my next question for everyone is one, do you have any other observations or any of the gotchas that we're not seeing here and then two what do we want to do next with this we've had a lot of really good ideas we need to start moving them together thanks Don for posting that in the chat how do we want to move forward we'll provide a quick reminder to everyone that I understand that we are all busy but if this is something we really care about we do need an individual or several individuals to kind of step up and champion this work Richie? I as part of my TOC responsibility I have a project which I'm a little bit doubtful about so it might lead to archivo so I'm also happy to to look at all the policies and provide or do a PR and lead the public discussion Richie that's interested Josh what's the definition of the work I mean we've had a lot of discussion here but I'm not actually clear on what the next steps are yep that's a good question and kind of driving a little bit more of the one that I was trying to get at so in my mind the way that I see this right now is I've taken several notes I have a Google doc that I'm just consolidating things into and I will eventually clean it up and share it in the TOC channel for everyone to observe the next steps for me are taking all of the feedback in the discussion that we've had today some of the suggestions and ideas and figuring out how they can work cohesively together once that is done identifying what some of those indicators are and what's currently possible today with the tooling and the resources that are available to us as well as potentially identifying any modifications to the existing tools such as dev stats visibility kind of workflows how this all is intended to work and then from there going through a public review and comment on whether or not the process that we're going to define here works whether or not the indicators is also good that may require engaging with the end user and the adopters to understand what they're looking for it may require engaging with all of the tags to make sure that they are invested in understand what their roles and responsibilities are in this process if we need to define that for them and also making it available and helping maintainers understand kind of what are their expectations for this how do they go and view that information this is not a small project I'm happy to help on the end user part as well to make sure that we get feedback we need okay so I've got Richie for kind of like the process and the policy bits I've got Ricardo for user engagement who's super passionate about indicators we've got 24 folks on this call and I know that there are a lot more individuals that couldn't make it I mean I'll be working with somebody on indicators but we need other people on this and ideally a TOC member who's interested in that yep I agree so it's the definition of this just to look at the different indicators for the health of a project and figure out how to derive them from the project itself again I'm still struggling to understand like I feel like we need at least a version we need at least another rendition of the document or the notes that you're putting together so that we can break this up into things that could be taken on because at the moment it feels like it feels pretty nebulous to take on any part of it without having some structure around it that's fair come up with a list of indicators of the tasks that's one of them so since we're kind of stalling on what it is that can happen we can start with dons and indicators as kind of the starting point what I can do is I will go through and I will type a ball of my notes come up with an outline for what needs to happen and the individuals that are interested or have expressed interest in these areas I will go ahead and tag them on the issue hopefully I have your GitHub handled because this will modify this issue to keep track of this kind of activity and don since you're the one that filed the issue I want to make sure that I have your concurrence to modify it to reflect all of these changes yeah please absolutely okay so what I'll do is I'll take the action I'll go ahead and write up the summary for what it is that we're looking to do since I've been around here I'll tag the individuals that expressed interest on it and I'll make sure that this is available in the TOC channel for anyone to stop by as well as the tag chairs channel for any of the tag chair members that were unable to participate today if you want to pair on working through that I'm happy to do that that would be fantastic Duffy thank you I will take you up on that offer and Ricardo I see that you are happy to help provide feedback on the list of indicators moving so I'll add you to that group as well alright any other thoughts opinions perspectives on this we've got a few more minutes left we don't have anything else on our agenda so I will tentatively and cautiously open the floor up to any last minute questions that folks may have do we have an idea of how the badges thing would work I'm actually kind of interested in understanding would that be what do you think or are we looking at how that would integrate with tags and tags would associate if they pass a particular set of criteria they would associate a badge with a project or how are we looking at that I think that is so yet to be determined okay it sounded like maybe there was already some work there but I wasn't sure Craig is the one that had suggested the badging sorry when I said tag before I meant tag as in label as opposed to tag as in technical advisory group I wasn't trying to associate as in you may have more than one of them whereas today you currently only have one label that says okay um Richie had another thought make them key value pairs not one-dimensional labels alright if there's nothing else I will let everyone go I really appreciate you all taking the time to join us on this discussion hopefully we can begin to move forward on it but I understand that I'm the roadblock for that right now so I will take the time go ahead and get this written up and work with Duffy one other admin's trivia note we are extending sandbox voting for another day to be able to make sure that we're capturing votes there's been some kind of some drift between where votes are actually going so it'll be open through Wednesday cool okay come by if you've got any questions about that happy to be able to help on that but we're extending for another day just to make sure that we are capturing everything it's a question when is the next sandbox application review meeting probably like June ish okay I will follow up with like the more direct like notes for when those are coming up okay thanks thank you thanks so much everyone we really appreciate your time today we look forward to seeing you next time thank you thank you