 Thanks everyone for coming. I think we'll get started now. So just a quick disclaimer that this is all our personal opinions. We are not speaking on behalf of our employer or the Kubernetes project or anything like that. But most of our experiences are derived from the Kubernetes project itself, so you're going to find a lot of references to it. I'm a Kubernetes, oh, I've got to do the intro. So I'm a Kubernetes maintainer. Kiran's also. Yep. I've worked on a bunch of Kubernetes projects and I'm a maintainer of a couple of sick projects, but Nikita forced me to become a manager. I did not force him to become a manager. So I'm right now an engineering manager at VMware helping like the open source engineers continue to contribute to open source projects. And he's one of the best managers I've worked with and probably the only one of the few managers who really understands open source. Hi. So let's talk a little bit about open source, why we are all here and all that. So every every product that we built uses open source. So it's a no brainer that open source is a way to go forward. Keep going here. I think she's holding the controls. So why does it matter? So what's open source is pretty simple and why do open source matters every app or like, you know, we have to, if you have to stay competitive and lead the industry towards a certain direction, doing it together is the only way to do that. Like Tim says, like, you know, one of our principal engineer, if you want to go fast, you go alone. But if you want to go far, go together, right? So now this all looks like it's a little bit of a standard one of the standard open source. He talks with like boring templates and everything. Yeah, who doesn't contribute to open source projects here? Great. And how many of you folks who actually contribute to open source projects do it as part of your day job? Okay, that is quite a few folks. So this is not yet another open source talk where we just try to, like, you know, tell why and what of open source. What we really want to do is share our journey as engineers working on the open source projects, right? How many can relate to this slide? Right? So this was like right out of like the keynote today, right? There are a lot of intent to contribute or like, you know, fund the open source engineers. But when it comes down to it, we still are like struggling to do this right. Right. And we hope it gets better. The keynote was optimistic, like it keeps us optimistic that there are companies who are willing to invest in upstream. So we hope that it becomes a reality. And just wanted to give a quick shout out and thank you to all the community members. We've been friends with so many of them, even like we've been speaking to so many folks who work on open source as part of their day jobs, who don't work on open source as part of their day jobs and the struggles that they face, the wins that they've had. And we've tried to compile them into sort of patterns and if we've tried to present them in the slide. Yeah. And also thanks to all those people that work behind the scenes in an enterprise, taking the projects that we build and converting to a product. So it's not really about like a project or a product, both have to go together. Right. So what we want to establish or like, you know, share with you are the patterns where we can co-work and help each other out. Right. So this is like the standard in and down kind of a thing. So we can't do this alone. We need the support of the product teams and product teams can't do it alone. They need the support of the project teams. And the whole thing I just wanted to plug in like the Indian context of this, like the Indian Yang is like a popular one. But we also see that manifested in almost like every other culture. This is one of those in the dancing picture here is Shiva and Shetty or Ardhanarishwar as it is called. It's a, you know, fine balance between power and how do you actually control it? Open source gives you the power and product teams control the business. Cool. So as open source projects, right? Like I think we all understand the projects that we maintain and we are really good at it. We are the managers and the product managers, everything about that open source project. But that project alone is not something that a customer needs. Right. They have to put together a lot of things. So the product companies that we are all employed in have this need to assemble all of them together and make a solution out of it. And it's not an easy problem. Right. As many project contributors we have, we also need product developers for that one. Right. And this list is just growing a part of the growth in the number of projects that we are seeing is also a problem that we created. And we'll come to like, you know, how we are creating that problem. Right. And how we could probably potentially address that. So from the 2021 CNCF annual report, and I think this is almost also public knowledge. We're seeing that Kubernetes has crossed the chasm. A lot of companies are adopting it. They're productizing it. So you've got a lot of users building platform services and building products on top of Kubernetes, which also means that Kubernetes experts who were originally contributing to the Kubernetes project, they are required by their companies, their employers to work on the products, because that is where the customers are interacting directly. Now, what this also means is that the project level is losing contributors. And if you are a new contributor, even if you are an existing contributor and you create a pull request and you are looking for reviews, you just don't get that right now. Like it has happened with me. It has happened with a lot of Kubernetes contributors who are sitting here. You create a pull request, maybe you'll get a few review comments and the and the reviews disappear for a few days and they'll come back again whenever they have time. Now this is okay, but this slows down the project in terms of like how the PRs are merged, the commits are merged, which also means that this is a problem to companies and businesses. This is just not a hypothetical scenario. It is a problem to you as well if you ship a Kubernetes product. If you have a feature that you want to get inside Kubernetes, it's going to take longer and longer as this problem gets worse and worse. So it's going to affect your release timelines and it means Kubernetes are at risk, is at risk and your businesses are also at risk. Yeah, just to add one more thing about the less visible part of it, right? I think it's used in so many other applications than it was originally intended to. So now making any change to Kubernetes core requires a lot more thought and discussions. The Kubernetes 2021 annual report from the steering committee also says that a lot of contributors are not working on upstream Kubernetes 100% of their time anymore, which was true a few years ago. Now it's okay if contributors are working say 80% of the time 50 60 whatever percentage of their time, but it's starting to affect. Like I said, reviewers another pattern that we've also seen is that a lot of senior engineers, they've been asked to work on the products and so on, which means we're also losing senior engineers from the Kubernetes project. We've got a lot of new contributors showing up. We've got a lot of students showing up, which is great. Like I don't want to say that's not good. But we also need senior engineers to move the project forward. And that's a problem that we're facing and we'll talk a lot more about it as we go forward. So with senior engineers going out of the picture, right, like or like, you know, there could be so many reasons, maybe they have gone gotten a career growth, and they've taken on a bigger role. They just got burnt out, which is also like a real case that happens with most of the senior engineers that work on this project day in and day out, beyond like, you know, their day job, right? So it results in an interesting pattern. Now companies already start using these projects. Now those companies want to enhance that with some new features that the new customers are asking. When those requests are coming back to the project, the project maintenance are not able to respond to those requests, which is actually seen happening. You know, some of these requests take longer to go and like, you know, get introduced into the project. This means product companies can't wait for like that long for a open source project to show the roadmap or when that's going to be done. So they end up building like parallel projects or they could be actually maintaining force, which becomes like a dangerous situation where like you'll be pulling more and more engineers away from open source. And also like, you know, increasing the complexity in terms of like number of projects that are like coming back to the CNCF or like that we have to deal with for integrations. And one of the biggest challenges that I've personally faced that we've all personally faced is just convincing executives and leadership to invest in upstream to actually hire upstream engineers. So when we first proposed it to our leadership, I went, we went in with a proposal which said, okay, for SIG node for whatever SIG, like, here's what you need to do. Here are the problems that the SIG is facing. Here are the features that we need to deal with in our product and blah, blah, blah, blah. And all I got was blank faces. One thing we've understood is that executives won't understand your SIG, the minute problems that you're facing. What you need to talk about is numbers. What they understand best is numbers. You need to talk about customers, the customer impact that they've been creating. It actually reminds me of an analogy that Jace had once shared, who's also a Kubernetes project contributor and maintainer. Think of it as a city park that needs to be clean. And the park is too large for any one company to clean it up. So if you as a company know that you're going to have a picnic in one part of the park, then you go clean that area first versus all the other areas. That is a logical thing to do. And that also makes you good citizens. And it also solves the needs of your company. And open source projects are no different. So if you as a company are looking to invest in open source, you need to build a strategy which impacts your customers, and also does good to the community as well. The other thing that we found it hard to convince was, like I talked about senior engineers, we need senior engineers who can invest certain percentage of their time, even if not 100%, I know that's hard in upstream. It's still a hard problem for us to solve. It's not like we've solved it completely. But what has helped in the past is we talk about okay, these are the customer issues that would can get resolved if we have certain senior, like for testing, okay, like we have this certain engineer who works on testing employed in upstream and so on. And also talk about how this would expedite the resolution process. Matrix is a hard problem. I think like there are like some data that's available. Look at that. Another huge, huge problem that we've seen is, like say we solved the staffing problem, the hiring problem. So companies will hire engineers to work on open source. They might hire engineers who are already working in upstream and they might even give them time to develop upstream expertise. But then they get tasked to solve customer issues. And now, again, this is not a bad thing. It starts getting a little unwieldy when it's one customer issue, then another, then another, and then another, and back to back customer issues. And the engineer just doesn't have time to work on upstream anymore. This has happened so, so many times that like, I can't count it, but I'm sure you have a lot of experiences on this as well. I'm a manager and I get pulled into these discussions. So the pro it takes a long time for any feature to get like actually into the product or like into the release of the open source. So they'll be looked at like, you know, this process is free. They're not working on anything urgent, but this customer issue is urgent. Let's pull them and make them work on this, right? So here's some data points to kind of what we are all talking about. This graph shows the average monthly commits in the Kubernetes slash Kubernetes, so the main Kubernetes repo. So we saw a 32% decline in 2020. It was likely fueled by the pandemic, but there were a lot of other reasons burnout, the world's been going through a crazy time as well. But then what's interesting is that this is a consistent trend. So we've had approximately 45% decrease in 2021, and an almost 41% decrease in 2022 now. So like the number of commits has been, it's not a small number. It's like when you look at the graph, it's it's going down like crazy. And what happens in the future is the big question. Now, again, why might this be happening? So this graphs from depth stats shows company contributions. And this has been decreasing over time. We've not included any company names because the point is not to like name or shame anyone, but it clearly shows and it like companies have been contributing less. The coming in going down as well. The contributions here are not just about code and they're like a lot of other things, right? So overall, like the commits to the projects have definitely been declining. Yeah, and we have almost 70,000 or odd contributors in the Kubernetes project with 10,000 contributors getting added every single year. 10,000 is not a small number. We are not increasing our reviewers or approvers. We call approvers as folks who can actually sign off on changes to any pull requests. So we're not adding that number. So the ratio has been getting unbalanced as we move on. So 10,000 new contributors adding every year, the number of reviewers actually declining. So the project is like it's going to be hard to get us to working sustainably. The other thing I want to quote mother here that he pointed out was so there's this thing called chopwood and carry water if you aren't familiar with it. It's basically the unglamorous work that you do it be something as triaging issues or answering questions and slack or mentoring a new contributor and so on. Now as the projects been growing, and this happens not just to the Kubernetes project, any open source project. As the project grows, the maintainers need to do a lot more chopwood and carry water work. And this takes a significant percentage of their time, which also means that if they work on open source as part of the day job, and they have performance reviews and they want to apply for promotion or whatever, they need to justify this chopwood and carry water work. And this is the hardest thing to measure ever. Like metrics on this, like it's very hard to get any metrics on this and DevStats just counts the GitHub events. So how do you really measure mentoring or you're answering questions on slack? How do you measure all of that? That's been really hard. The other thing that takes up not just my time or folks working in my employer, but everywhere, we've seen this everywhere, is justifying it, right? And this takes up a lot of our time. So it leaves us less and lesser time to actually write code and work on things that we want to work on. Yep. So I think chopwood, carry water, people that come to this conference, they get called out in like the keynotes, they get awarded in like the contributor summits. But then who are the audience that are actually seeing this, right? So we do need to carry this information and have the conversations at a senior leadership level, where these are important. If you are not going to credit these people that are doing this CWCW work, then the investments are going to be pulled out, right? So we have to have like some way in which we can quantify that the ratio that we have between like the number of contributors that are coming in to like the number of maintenance of that project that's pretty bad. One way to look at this is imagine like if you are running a product, enterprises have like really good metrics around like, you know, how many people in a team should be like junior versus senior. Why is it different for an open source, right? It should be something similar, right? And the other problem also like with respect to like the getting new contributors has been senior engineers are already like burnt out. So we end up like when new contributors come, they don't know how to navigate this entire thing and get to contributors or like maintain a status and then they just run away, right? Or like they're just disappearing. But I also want to talk about some positive aspects here. So if you look at the number of unique PR monthly, unique PR authors, it's slightly decreased maybe, but it's almost remained the same. So people are creating issues, they're creating PRs and so on. If you look at the average number of monthly PRs opened, it's again, so that one was about issues, this is about PRs, it's almost remained the same. But the number of reviews, they have been going down like anything. It's again almost a linear graph. So that's the point that we were trying to make. Like there is interest in the project, but the number of reviewers and reviews have been going down. So how do we bring this graph back up, right? I think the solution is probably like very easy. It's right in front of us. We need more senior engineers working and also training more new members to become like the maintainers. But how do we do this? That's probably where this is. And I think some of these graphs have helped to some extent, but we do need to build like a lot more data around what the senior engineers in the community are doing to encourage more investments into that field. So this is an interesting graph. Yeah, this one is a little interesting. So this graph shows the number of review comments. One thing we've noticed is the number of review comments have been increasing. Anecdotally, what we are like correlating it is one thing is that the project is becoming complex. So the newer features that you add, you need to like the reviews are pretty complex. The second thing we've also noticed is that new contributors try to jump in and help out with reviews as well, which leads to quite a bit of review comments too. Yeah, I think like the talk that we attended earlier about note shutdown, it took like literally four to five years to get to where we are, right? Yeah, the best way to solve all of this is mentorship, but like you said, almost all senior engineers are burnt out and if they're not employed to work on open source, they just don't have time to mentor. And a project as large as Kubernetes, a lot of context is undocumented, it's just lost. So like I was saying, I've had instances where you create a PR, you need someone who's the original author of some part of the code base to actually show up and approve that PR, otherwise that PR is just not getting merged. So unless we have mentorship, we always need to ping the original authors and we're just not moving the project forward. So mentorship is definitely the right answer and hiring senior more engineers. Yeah, I think like a related problem to this is like I've spoken to some people where their companies are allowing the engineers to work on open source, at least like 20% or 40% of the time, but they don't know how to do that. So it comes back to like us, goes back to like the complexity of the project. It's pretty important to like have some programs in these open source projects to get new people to a maintenance level. So this is like just a quick recap, but we could probably escape. I think everybody understands this. So we'll get on to our journey on how. Yes, so how did we create the team? So one quick way to find out like you know if we are doing the right thing or not is like a quick poll on people who are contributing to the open source. Are they doing it for their company or helping someone else, right? If the answer comes as like I'm helping someone else, we have to drill down on that, right? That's not a very sustainable model. There could be like a handful of people that can continue in that mode for a very long time. This model becomes sustainable when we have more maintenance and contributors that are doing this for their company or themselves. Yeah, this one is like something that we can all relate to. We have to make sure like we put the mask on the project or the product before we help the other. So what do you think Meera, which one is the project in this case, mother or the child? I think that could be the mother. Like the open source, the project is like the foundation layer and that goes already like what happens to the product? Like the foundation needs to be strong. Yeah, and I don't think it's like role that's like fixed. Sometimes it's like the product that becomes like the thing that needs more care or like the project that needs more care. So this we have to identify which one needs the more attention and fix it right now. Yeah, it's a balance. So one other thing we like you can see from here is that we need to always align with business, right? Like we business alignment is the biggest thing here. Like companies aren't going to find upstream engineers or let us work on projects in upstream unless it's aligned to something that is bringing us revenue. So the way we see it is it's a very basic success metric. So the project that you're contributing to must be thriving and the products that you're building should enable your customers. And one thing to call it is like I think like most of the organizations know the importance. There's actually like a central body that helps with all the open source strategy and like identifying the projects. But what we are talking about here is each product needs to have somebody in that BU or like the product team or the portfolio team that's constantly advocating for the need for sustained investment in the project. And this is not something that we can do as an engineer's day in and day out. So we need like somebody at a senior level that's constantly talking to the leadership to make the case for this growth. So we've been like lucky I would say not like really bad than other organizations where we have like a VP director and like a product manager that understands the problem. It's important to have that kind of people in this case. The other thing that I just wanted to bring out here is it when these things are not there the burden comes down to like the maintainers and what should the maintenance be doing right. I've had this situation this I'm just quoting in my own team right like there was Arnau who was trying to help out with the recent patch release and they were like something that came up I had to pull him out but if I keep doing this again and again then that's not a sustainable model right. So there has to be a solid reason for pulling somebody out of a commitment in the open source. In this case we had to pull him out because there was a CVE related stuff that was going on to figure out like you know if we should be doing something more into the patch. So it resulted in a good thing but such instances have to be like minimized as we go forward. I like that you call yourself out here. I don't do a good job all the time. So no this is a doing a good job. Yep so this when we don't have that senior leadership and in fact like there was a real case with us right like it was like the maintenance of the project. We're just taking the example of Kubernetes here but this has happened within our organizations with almost all the other projects. So the senior engineers or like the people on the project who are the maintenance right they have to do this extra stuff like act as an engineering manager act as a product manager or you know be a mentor for the new people right. And also constantly work on giving small tangible wins so that the team is motivated. It's very easy to get into this thing that okay things are not working good I'm going to go out of open source project that's not going to solve anybody's problem right. So we've been lucky to have them on the Kubernetes project who helped us navigate this thing. Yeah and not even so Dims no longer works at VMware but even when he was at VMware he was such a strong voice for upstream contributions and pushing for it and like fighting with executive sometimes fighting the good fight of getting like the team that I work on it wouldn't have existed if Dims weren't here. And it's not just within the company by the way it's also within the community as a former steering committee member like working with the CNCF working with the Linux Foundation working with other companies trying to get more contributors. We need more people like this I think we are like lucky to have like a lot of people like this already in the community but this has to go on and to this effect we're not dwelling to this topic a lot but the next talk that Paris and team are giving is exactly on this right in the same room so we should. Yeah so this is more about what a company should or can do and Paris's talk is going to be more about what you can do in that company so please like make sure to add that one as well. All right so I think one of the biggest challenge that we have had with open source project was like creating silos around it. So the pattern that we have observed is we are pretty good as an open source community serving our users. This is pretty much like a great correlation between like everything that happens in an a project open source project and if you consider users as a customers we figured that out we do great job you know great products are coming out but the moment it goes into customers right we kind of like have like a different team called a product team so we don't like work together really it's context switching it's not always like going to be easy to do this right and we are not suggesting that like people who are contributing to the project should be contributing to product teams but there has to be some kind of a synergy here right so listening to the pain that our downstream for lack of better word people are like facing and trying to see if we can address anything in the upstream will be like a great way to create this balance right. So just like how users get into a customer's persona and like you know users are like the early adopters but when customers with their like a lot of people like Kubernetes has crossed the caste and kind of thing these customers have a totally different mindset about how to use the project and the burden of like enabling them falls on the product teams we shouldn't let product teams say like you know you're like someone else you deal with the projects and we are like only going to talk about like the community right we have to have those conversations we have to have channels where we engage with each other again just to be clear we're not talking about the same person working on two things it's about like having the conversations with each other right. So here's what our team charter looks like I'm going to like and when she says our team it's a upstream team right and just to be clear yeah sorry I should have been pretty clear about that so we mainly care about three things I'm going to go a little quicker here because we've got just five minutes left so one is actually writing the code so that's the implementation part the second part is we call enablement so that is actually enabling other internal teams at VMware so we act as the feedback loop between the community and the product teams and finally engagement so it's growing both VMware folks and community members mentoring them and trying to grow the project itself. So I think we all want to grow both as a project as a product but also as individuals right and it's not always going to be a straight path there are times it's a sine wave that's what we have realized working on this and they both have to go together if one one of them like you know falls apart like it pulls down the other one that's really important for us to like understand so this is like from the Detroit streets things aren't as bad everything is going to be all right it's a little blurry you can't see through the shade the trees but we're getting better right yeah and things that have worked well for us is one is customer trust so if any upstream like Kubernetes or any other open source project escalation comes in like customers would trust us to solve that issue because we have expertise in that project and also even if like just because we're participating in the Kubernetes project like we build friends we build a network so we build karma in the project for example I don't work on security but I know Pushkar who does who's also a good friend so I'm just going to go bug him until I get my issue resolved so that's how you build your network and you can like develop customer trust as well I think I've also been in some customer calls where some of the times like it becomes easy like to say oh that feature that you need oh yeah my team already is working on it this is where you can see how that is going on right so that adds a lot of value in terms of trust yeah so one again like work-life balance for sure I want to spend time with my family and friends I don't want to do pull request reviews like on my personal time so I've kind of self-explanatory so by the way this slide had like 100 percent full time OSS developers we tried to change it like because that's the reality we can never be 100 percent full time open source engineers it will be like almost full time yeah that's probably what's the reality at this point that's possible some and the biggest thing that so all of this like it doesn't really matter to be honest at the end of the day the what really matters is how we feel when we are contributing and the biggest thing that's worked out for me and like you can see in all of these quotes from Nabarra and who's also a steering committee member is the global impact that you create the impact that you create both internally externally and how you see and meant like you mentor other folks and how you see them grow from just to be a contributor to a member to a reviewer and so on so that's the thing that keeps us going and from a manager perspective right it's you know when when you work in an enterprise product you have like some customer success and all those metrics that you can pull in but it's also important to get your managers to attend events like this especially the contributor summit or like you know ask them to like see the Twitter there's like so much that your team is doing that's out there the appreciation is all out there you know I think like somebody has to act that you know put that bridge on bringing that back into the company I think like CNC or like Kubernetes committee does a great job of like if you're already contributing you reach out to them and tell that like you know we want to use this for promotion of like an individual of things like that and that works right you actually bring up a good point this is like present in the slides but I wanted like bring your manager to these events I was just speaking to someone this morning and they were telling how their manager mostly attend like customer meetings and so on because that's really tight of their revenue and they're not active in the contributor summits or the open source maintain our sessions but like bring your manager here so they can actually see the impact that you're creating and hear the shout outs and everything they're putting super I think we just flip through because we only have one minute so we just wanted to leave you with like call it commandments or the principles that we have to work together as a product in the project so if the product's goal should be around like you know minimizing the internal patches which basically means we have to support that by making the upstream more extensible the product's goal should be not to replicate and create a fork but rather like build a secure pipeline and the long term support around the product the option yeah and the upstream side like build forward looking features do the chopwood and carry water work your Kubernetes you're not gonna get a Kubernetes release without all the chopwood and carry water efforts from the Kubernetes contributors and maintainers yeah I want to just touch on the last one though I think companies understand like sending individuals to customer events or like company events that's pretty much given like you know they're growing but it's also very important to send the people that are working in the project to the community events that's where you build the karma and like you know you I know funding is hard but this is also this also should be a priority because it is important that maintainers attend events like these collaborate with each other and I'm happy that we are where folks are here both managers and individuals like cool that was the last one that we had